ChangeLog for pflogsumm.pl [Note: Let me know if you would like to be notified as new versions are released. The latest released version can always be found at http://jimsun.LinxNet.com/postfix_contrib.html.] rel-1.0.4 20020224 Added "--no_*_detail" options. (Suppresses some of the "detail" reports.) Added "--version" option. (Thanks to "Guillaume") Improved handling of "[ID nnnnnn some.thing]" stuff (Thanks to Blake Dunmire) Repaired and optimized some of the "reject" parsing. Added processing and report of smtp delivery failures. Added --rej_add_from option: For those reject reports that list IP addresses or host/domain names: append the email from address to each listing. (Note: does not apply to "Improper use of SMTP command pipelining" report.) rel-1.0.3 20010520 Minor re-work of "reject: RCPT" parsing to account for Yet Another Change in the logfile format. (Semi-colon changed to a comma in "blocked using rbl.maps.vix.com,".) rel-1.0.2 20010519 Took another whack at "verp" munging. *sigh* Added code to summarize "Improper use of SMTP command pipelining" rejects by client. rel-1.0.1 20010518 Modified to catch "reject: header" log entries changed as of postfix release-20010228 (?). Prior versions of postfix had the string "warning: " (where the qid normally is). Thanks to Glen Eustace , Len Conrad , Daniel Roesen , Milivoj Ivkovic and j_zuilkowski@hotmail.com (Jon Zuilkowski) for reports and/or patches. Fixed a couple of "uninitialized variable" problems. Committed (actually starting with 20000925-01beta) to CVS. 20000925-01 Added a line to compensate for (new?) "[ID nnnnnn some.thing]" sub-strings that appear in logfile entries under Sun Solaris 8. (At least. Others?) Note: Upon being committed to CVS, this became rel-0.9.0. 20000916-01 Forgot to add "--problems_first" to the "usage" output and in the synopsis at the top of the comments. 20000908-01beta Re-did what 20000907-02beta was *supposed* to be! To wit: replaced missing "--ignore_case" bugfix, "panic" entry processing, improvements to "fatal" and "warning" message reporting and missing "--mailq" option. (Obviously: 20000907-02beta was derived from the wrong code base.) 20000907-02beta Fixed bug in ISO date formatting that caused the month to be off by one. Thanks to Kurt Andersen for the report and the patch. Fixed overflow of connect time reporting into days. (Can happen during weekly summaries for sites with large volumes of email.) Thanks again to Kurt Andersen for the report and the fix. Improved "rejects" reporting *again*. Thanks to Thomas Parmelan for the patch. Added "--problems_first" option to print "problem" reports such as bounces, defers, warnings, fatal errors, etc. before "normal" reports. 19991219-02 Fixed bug in code that prevented "--ignore_case" from actually doing anything. Thanks to Nadeem Hasan for reporting this and supplying the fix. 19991219-01beta Added the following caveat to the "Notes" section of Pflogsumm: ------------------------------------------------------------- IMPORTANT: Pflogsumm makes no attempt to catch/parse non- postfix/vmailer daemon log entries. (I.e.: Unless it has "postfix/" or "vmailer/" in the log entry, it will be ignored.) ------------------------------------------------------------- Added reporting of "panic" log messages. This was missed until now! Increased reporting detail of "fatal" and "warning" entries. (Actually, "warning" detail was increased in 19991120-01beta. Neglected to note it then.) 19991123-01 (unreleased) Added "--mailq" option. (Convenience factor.) Runs Postfix's "mailq" command at the end of the other reports. ------------------------------------------------------- NOTE: If Postfix's "mailq" command isn't in your $PATH, you'll have to edit the "$mailqCmd" variable located near the top of pflogsumm to path it explicitly. ------------------------------------------------------- 19991120-01 Tried once again to improve parsing of reject log entries. Specifically: those associated with "RCPT" rejects. 19991016-01 (not generally released) Added --smtpd_stats. Generates smtpd connection statistics. --------------------------------------------------------------- NOTE: Support for --smtpd_stats requires the Date::Calc module (available from CPAN). If you don't want to go to the trouble of fetching & installing that module, and you don't want smtpd stats anyway, *carefully* identify all of the code sections delimited by "# ---Begin: SMTPD_STATS_SUPPORT---" and "# ---End: SMTPD_STATS_SUPPORT---" and remove them. --------------------------------------------------------------- 19990909-01 (not generally released) Added -i and --ignore_case options. Causes entire email address to be lower-cased instead of just the host/domain part. Added "use locale". (This means that the sorting order within reports may be different from before--depending on how you have your machine's locale set.) 19990904-03 Improved "reason" parsing and reporting for bounced and deferred messages. Added parsing of "cleanup" reject lines to catch PCRE/regexp rejects. Added "reject" stats to per-hour and (on multi-day reports) per- day reports. Improved "warnings" report to show details. A single message deferred multiple times showed up as multiple deferrals--implying that multiple messages were deferred. Now shows "how many messages were deferred" and "how many deferrals" as separate stats. Changed display of "Grand Totals" to make it a bit more readable (IMO). Added "automatic perl finder" line for those systems that don't support the "#!" notation. By popular demand: added note to comments as to where pflogsumm home page could be found :-). 19990413-02 Fixed problem with last octet of IP address getting truncated in reports when IP address used in place of unknown hosts. Changed the way a few internal variables were handled to be compatible with Perl 5.003. Don't run it under Perl 5.003 with the "-w" perl switch, tho! It will issue lots of warnings. All tests I performed indicated that it produces the correct output, however. ------------------------------------------------------------ NOTE: While this version was tested to work with Perl 5.003, I recommend that you upgrade to 5.004 or later. I will not guarantee that I'll remember to do the full regression- testing that I usually do with 5.003 as well. ------------------------------------------------------------ 19990411-01 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is GONE! (As per the previous notice.) Added "--help" option to emit short usage message and bail out. Added "--iso_date_time" switch to change displays of dates and times to ISO 8601 standard formats (CCYY-MM-DD and HH:MM), rather than "Month-name Day-number CCYY" and "HHMM" formats (the default). Added "--verbose_msg_detail" switch. This causes the full "reason" to be displayed for the message deferral, bounce and reject summaries. (Note: this can result in quite long lines in the report. Also note that there have been a couple of subtle changes in the "reason" parsing/reporting in the default mode (no "--verbose_msg_detail".) Added "--verp_mung" option. The problem this addresses is "VERP" generated (??? so far as I can tell!) addresses (?) of the form: "list-return-NN-someuser=some.dom@host.sender.dom" These result in mail from the same "user" and site to look like it originated from different users, when in fact it originates from the same "user." There are presently two "levels" of address munging available. With no numeric argument (or any value less than 2), the above address will be converted to: "list-return-ID-someuser=some.dom@host.sender.dom" In other words: the numeric value will be replaced with "ID". By specifying "--verp_mung=2", the munging is more "aggressive", converting the above address to something like: "list@host.sender.dom" Which looks more "normal." (Actually: specifying anything less than 2 does the "simple" munging and anything greater than 1 results in the more "aggressive" hack being applied.) Added "--uucp_mung" switch for consistence with "--verp_mung". 19990321-01 NOTICE: As of this version of pflogsumm.pl, versions of VMailer prior to 19981023 are no longer supported. Sorry. Pflogsumm-19990121-01.pl will be made permanently available from now on for those with out-of-date versions of VMailer prior to 19981023. NOTICE: As of this version of pflogsumm.pl, the "-c" switch is DEPRECIATED. This version is transitional and retains it. The next version will not have it. Subsequent versions may re-use it for another purpose. Use the "-h" and "-u" switches instead. Added "-h" and "-u" switches to provide finer-grained control over report output. Depreciated "-c". Added "deferred" and "bounced" to "Grand Totals", "by-day" and "by- hour" reports. Added "by-host/domain" reports. For sent (delivered) and received messages: lists message count, total size of messages and host/domain. For delivered messages: also lists number of deferred messages and average and maximum delivery time. Both reports sorted by message count in descending order. Grand totals also now list number of recipient and sender hosts/domains. Re-wrote "by-user" data collection storage to reduce memory consumption by in-memory hashes. Moved "credits" from pflogsumm.pl to this file. 19990121-01 Now accounts for forwarded messages. Side-effects of the above: . Total messages size now broken-out into total bytes received and total bytes delivered. . Count of forwarded messages now reported. . Postfix-internally-generated messages (e.g.: Postmaster notifications of bounces) are no longer counted as "received". (They do, however, show up as "delivered".) . Forwarded addresses no longer show up as "recipients" (just as with aliases and mailing lists). Note that "delivered" will exceed "received" when messages are forwarded because of additional header lines. 19990116-01 Added processing for "reject" log entries. Expanded detail of "deferred" and "bounced" log entries to include "reason". 19990110-05 Added "messages received/delivered by hour" and "messages received/delivered by day" reports. See the "Notes" section in the documentation for details on how these behave. Broke-out total message count to "messages received" and "messages delivered". (For the above two enhancements: "postfix/pickup" and "postfix/smtpd" lines are now processed. They used to be discarded.) Renamed "summary" report to "Grand Totals". Added code to parse date & time stamps from log entries. This was needed, in part, for the "messages per-hour/day" reports. It would have been necessary for future enhancements in the way of date- & time-based processing anyway. Added "Notes" section to docs at top of code. 19990109-01 Improved display of large integer values. 19990107-01 Bugfix only. Data for "extended detail" listing was being built even if "-e" not specified. This resulted in unexpected excessive memory consumption when crunching large amounts of data. Added warning about memory consumption when "-e" option specified. 19990103-01 Further improvement to "accuracy" of by-domain-then-logname sort. (Presently used only by "extended detail" listing). For comparison purposes: mungs "machine(s).host.dom" into "host.dom.machine(s)" so sort is keyed on "base" domain name before machines within the domain. Does *not* attempt to reverse the order of the "machine(s)" - so within a particular "base" domain, may not come out in quite the right order. ("foo.bar.some.dom" will come out before "sales.acme.some.dom", for example.) Also works for 2x2-style domain names. (I.e.: "some.do.co") 19990102-01 (never released) Added "mung UUCP-style bang-paths" switch (-m). Improved performance and "accuracy" of by-domain-then-logname sort used by (only at present) "extended detail" listing. 19990101-02 Added "extended detail" option (-e). At present this includes only a per-message detail listing, which lists per-message detail sorted by sender domain, then sender username, then by queue i.d. Improved docs a bit. 19990101-01 Replaced warning message when message size unavailable in favor of producing a report of these, sorted by queue i.d. Unlike the other reports, this report header is not emitted at all if there are none of these. (Always acts as if the -q switch had been specified). 19981231.01 Added experimental code to lower-case all domain names so that "user@foo.dom" and "user@FOO.DOM" will come out the same. Added test for existence of message size value when "to=" records are being processed. This was necessary for cases in which the logfile entry containing the "status=sent" record is not processed at the same time as the logfile containing the "size=nnnn" record. Note that this will produce a summary that will show recipient counts without matching recipient sizes. The only way to cure this would be to create a separate disk file to "memorize" message sizes. (Which would introduce a whole new raft of problems.) Added warning message (emitted to stderr) when the situation above is detected. Fixed "usage" message to indicate you can specify files on command line Wrapped a couple of long lines in the comments and code. Added (temporary) version numbering scheme. Started this log. Other changes/enhancements since previous un-version-numbered versions: deals with log entries for VMailer as well as Postfix, more robust parsing of "to=" and "from=" fields (now handles spaces in these), eliminated double-counting of message sizes (happened when delivery was deferred), re-structured parsing to be more robust (not- to-mention correct!), added "grand summary" report at top (total messages, total size, number of senders and recipients). Bugs UUCP-style bang-path handling needs more work. Particularly if Postfix is not being run with "swap_bangpath = yes" and/or *is* being run with "append_dot_mydomain = yes", the detailed by-message report may not be sorted correctly by-domain-by-user. (Also depends on upstream MTA, I suspect.) To Be Done (Maybe) date ranges, "lastweek", etc.? (options for?) break-down by local vs. non-local?, further "drill-downs" to sender/recipient domains? Separate reports by-domain? (Would require that pflogsumm write files instead of emitting to stdout? Or maybe that would be an option?) Separate options for sender domain and receiver domain? Or perhaps sender address and receiver address regex "filtering" options? lower-case domains (experimental code in place) don't k-ize msg counts? (Or do so at a higher boundary?) (done) implement proper version numbering (interim version numbering in place) (use SCCS or RCS?) add changelog (rough one in place) Expand docs (in code?) Re-do docs to POD format? Improve UUCP-style bang-path handling Add option to use disk-based hash files instead of in-memory so processing logfiles with lots of postfix log entries doesn't run the machine out of memory? (This would really whack performance!) (Maybe only needed with "-e" option.) Add another "step" to integer processing: "g"? (Display formatting will presently mess up at values exceeding 999999m.) Internationalization? Necessary? There's English-language month abbreviations hard-coded in pflogsumm. I'm admittedly real weak in this area. (Covered by ISO 8601 option [below]?) Add option to lower-case entire addresses, rather than only the domains. (Or make that the default, and add an option to make it just domains?) (The RFCs say username parts shouldn't be touched!) (Done) Add percentage-of-total to all (?) reports? Add ability to handle compressed files within pflogsumm? SMTP logging by host (msgs, connects) (add to new "by-domain" reports?) (Done) UUCP logging of some sort? (At least what is sent to what host/gateway. Since rmail receives incoming, don't know as I can do anything there.) Option for ISO 8601 standard date & time formats. (done) Option for specifying "host" (for multi-host logfiles)? Credits [Note: The credits reflect suggestions and code contributions that have actually been added. If your contribution doesn't appear here, it may simply mean that it hasn't been added yet. (In which case it should be on the list above.) On the other hand: if I failed to credit you for something that *has* been added, please let me know!] Paul D. Robertson For much testing and patience and many good suggestions on how pflogsumm could be improved. Simon J Mudd For the following code contributions: Add "deferred" and "bounced" to "by hour" reports. (I also added these to "by day" reports and "Grand Totals".) "VERP" (?) address munger (less-agressive version) Suggestion for "by domain" delivery delay report. For the --smtpd_stats suggestion. Anders Arnholm For pointing out the problem with forwarded messages. Walcir Fontanini For pointers to changes to make for Perl 5.003 compatibility. (Added to 19990413-02beta.) (Which I will *try* to keep in mind!) Eric Cholet For the --ignore_case patch. Kurt Andersen For the ISO date formatting month-off-by-one patch and the connect time overflow fix. Thomas Parmelan For improved "rejects" reporting patch. Glen Eustace Patch to fix "reject: header" matching after Wietse changed the logfile format.