On Tue, Jan 01, 2019 at 01:14:54PM +0100, Walter Alejandro Iglesias wrote:
> On Fri, Dec 21, 2018 at 06:59:58PM +0100, Gilles Chehade wrote:
> > On Fri, Dec 21, 2018 at 06:56:57PM +0100, Walter Alejandro Iglesias wrote:
> > > Hello Gilles,
> > >
> > > In article <20181221145201.GA90310@ams-1.poolp.org> Gilles Chehade <gilles@poolp.org> wrote:
> > > > On Fri, Dec 21, 2018 at 07:41:41AM -0700, Gilles Chehade wrote:
> > > > > CVSROOT: /cvs
> > > > > Module name: src
> > > > > Changes by: gilles@cvs.openbsd.org 2018/12/21 07:41:41
> > > > >
> > > > > Modified files:
> > > > > usr.sbin/smtpd : smtp_session.c
> > > > >
> > > > > Log message:
> > > > > start simplifying log lines, they're no longer intended to be parseable, we
> > > > > have a reporting API for tools that want to analyze events, maillog is just
> > > > > for us, hoomans.
> > > > >
> > > >
> > > > that was not the best way to phrase my commit log ... sorry
> > > >
> > > > i meant they're no longer intended to be friendlier to scripts than to
> > > > humans: there will still be in a format that's easy to quickly script,
> > > > but they will hold information easily readable by humans, not a lot of
> > > > unrelated context infos so tools can generate dashboards out of single
> > > > lines.
> > > >
> > > > logs for humans, event reports for tools.
> > > >
> > >
> > > Since long I've been greping IPs from spammers and attackers from
> > > /var/log/maillog, /var/log/authlog and /var/log/daemon using a shell
> > > script I wrote that automatically includes them in a file read by a pf
> > > table. In the case of maillog, it relies in the address="" and host=""
> > > info currently included.
> > >
> > > Will it appear sender's IP and hostname in /var/log/maillog after this
> > > change?
> > >
> >
> > yes, you'll still be able to grep that information from maillog
>
> You selected carefully the words in your answer. :-)
>
not really, I don't know what your scripts do and how you wrote them.
the sender IP and hostname appear in the log, they are just not repeated
on every single log line but that shouldn't prevent scripts from keeping
track of them.
anyways, as stated in the commit log and my follow up message:
"we have a reporting API for tools that want to analyse events, maillog
is just for us, hoomans"
"logs for humans, event reports for tools"
the maillog format is going to go through many changes to simplify it,
remove redundant information, add missing information, etc... basing a
script on it is not recommended as we'll break them with every change.
> Indeed, I still can grep "IP" and "host" in maillog, but they are alone
> in a first line and the only way to associate them with the following
> lines containing the from= to= and result= (to know what "happened" with
> that connection) is by using the connection id, what will *painfully*
> overcomplicate my scripts.
>
As you imagine, I can't take into account individual scripts.
Other people have asked that the port or listener tag appear in lines.
Should these appear on all lines too ?
And the cipher ? and the authenticated user ?
Why is the IP/host information more legitimate to be repeated than other
information on every single line ?
What about the fcrdns check which will appear on connect lines, does the
check have to appear on every line now ?
What about the spf check when it is added at some point ?
maillog is not a context-free format, where each individual line carries
all of the information so you don't have to look at previous lines. Line
should describe an event and carry informations related to THAT event.
The only guarantee I make on the format is that you can always find what
you're looking for with at most 2 grep, one to find a session id, one to
find the event you're looking for.
That being said, there's a new reporting mechanism which is intended for
scripts and tools. It comes with a format that's easily parsable, that's
going to be stabilized, versionned and which actually provides more info
than maillog. It doesn't solve your context-free issue but it can easily
be used to script an output that repeats the info you need on all lines,
to be fed to your existing scripts. I have such scripts myself.
If you describe how your scripts work, I can probably help you.
> I don't know what's the opinion of the rest about this change. I'd
> highly appreciate you to include again the IP on each line of info as
> before. :-)
>
I didn't put this change to vote :-p
A lot of people had a bad opinion about the new config format but I knew
it was an improvement and ultimately it has unlocked so many issues that
we have had more commits in the last three months than in the last three
years.
I know you would prefer that I didn't change the log format but what you
want is still doable, so I won't revert unless there is a good rationale
that I actually made some use-cases undoable and unfixable.
Fixing your scripts to not be context-free only requires a few lines for
them to catch connect/disconnect events and map that to a session id. It
is also possible to not fix them but write a script that produces output
they want from the reporting mechanism.
--
Gilles Chehade @poolpOrg
https://www.poolp.org tip me: https://paypal.me/poolpOrg
No comments:
Post a Comment