June 30, 2012
amavisd-new-2.8.0 release notes
Contents: COMPATIBILITY BUG FIXES NEW FEATURES SUMMARY NEW FEATURES - 0MQ NEW FEATURES - OTHER OTHER
COMPATIBILITY
removed an old compatibility measure: default value of @banned_admin_maps was changed from: @banned_admin_maps = (\$banned_admin, \%virus_admin, \$virus_admin); to a more consistent: @banned_admin_maps = (\$banned_admin); The previous default value of @banned_admin_maps tried to maintain compatibility with versions before the setting was separated from its companion @virus_admin_maps. Now this compatibility is no longer considered necessary and contributes to some confusion, so it was dropped. See 2.4.0 and 2.2.1 release notes for previous changes to this setting.
quarantining to an mbox format file used to include a local time in an mbox separator line, which differs from RFC 4155 and common practices of using an UTC timestamp; a time zone of a timestamp in separator lines is now changed to UTC;
BUG FIXES
fixed initial evaluation of dynamic (i.e. per policy bank) values of $enable_dkim_verification, $enable_dkim_signing and $bypass_decode_parts across all declared policy banks; these policy bank entries may be scalars of references to such;
finely adjust a message size for de-stuffed dots according to a size definition in RFC 1870; avoids occasional message size mismatch when using an antispam interface module SpamdClient (implementing client-side of a spamc/spamd protocol);
updated LDAP.ldif to match LDAP.schema; provided by Quanah Gibson-Mount;
updated AMAVIS-MIB.txt and amavisd-snmp-subagent: changed type of SNMP variables MsgsSize in the group amavisStats 7 from Counter32 to Counter64 for consistency with other MsgsSize variables in groups amavisStats 3 and amavisStats 9;
See also the bug fixes section of 2.7.1 and 2.7.2 release notes. All fixes applied to 2.7.1 and 2.7.2 are incorporated in the 2.8.0 code.
NEW FEATURES SUMMARY
For monitoring and statistics gathering purposes a new set of utilities and service processes is available based on a message passing paradigm, using a 0MQ (a.k.a. ZMQ, ZeroMQ, or Crossroads I/O) library. This replaces a functionally similar set of utilities based on a shared BerkeleyDB database, with a benefit of avoiding lock contention altogether. This can bring sigificant speedups, most pronounced on a host with many busy amavisd child processes.
Applied numerous fine-grained optimizations based on a NYTProf profiler results. Optimizations include a reduction in a number of generated Perl opcodes and similar micro-optimizations. This accounts for a large amount of small changes in the code.
Our current statistics (Q4 2011) shows that 80 % of messages are below 30.000 bytes, and 90 % of mail messages are below 100.000 bytes in size. As an optimization, messages below 100 KiB in size are now kept and processed in memory, including passing them more optimally to SpamAssassin 3.4.0. Some file activity is still there, but is much reduced. If $TEMPBASE also resides on an SSD disk (or a RAM disk), observed speedup between 2.7.2 and 2.8.0 was 3 to 8 percent on a busy host (with monitoring disabled, so as not to skew a measurement).
Use a module IO::Socket::IP if available, instead of dealing directly with low-level modules IO::Socket::INET and IO::Socket::INET6;
choose more appropriate defaults if running on an IPv6-only host (like connecting to ::1 instead of 127.0.0.1 which may not exist);
amavisd-release now also supports connecting to amavisd over IPv6;
as a debugging aid it is now possible that a late event triggers full logging of earlier events that occurred during processing of a current mail message;
$enable_ldap setting is now dynamic, i.e. can be changed by a policy bank, which makes it possible to selectively disable LDAP lookups per policy bank;
optionally avoid persistent connections to SQL and LDAP servers;
it is now possible to disable calling an external file(1) utility but still have MIME parts decoding enabled;
added support in Amavis::SpamControl::ExtProg for an external spam scanner Bogofilter;
added locking options to @spam_scanners entries, to be used with external scanners which need but do not implement locking of their resources by themselves;
added a global configuration setting $sa_userprefs_file, which is passed on to SpamAssassin as a 'userprefs_filename' parameter at initialization;
added a subroutine iso8601_weekday(), potentially useful with partitioning;
added several new macros available to logging and notification templates;
NEW FEATURES - 0MQ
New files in the package are:
amavis-mc is a master supervisor process ( master of ceremonies :), to be started at boot time as root, or as a user vscan/amavis. Currently its only function is to spawn three instances of amavis-services processes with dropped privileges, to monitor and restart them in case they fail, and to terminate them when itself if being terminated. Preferably this process should be started before amavisd and before amavisd-snmp-subagent-zmq, although things would eventually catch up even if this is not the case. This process must run on the same host as amavis-service processes.
amavis-mc_init.sh is an example FreeBSD-style startup/shutdown shell script for starting/stopping the amavis-mc process;
amavis-service implements three services, chosen by a command line argument. It should be running as user vscan/amavis (not as root!). All its instances are typically started/stopped automatically by the amavis-mc process with dropped privileges. A note for manual testing (started from a command line, not by an amavis-mc process): make sure to run amavis-service under the same UID as the amavisd is running. If 0MQ cannot write to a socket due to privilege violation, messages are silently dropped. Service processes as implemented by amavis-service must run on the same host as amavisd for two reasons: they communicate with amavisd child processes through a Unix socket, and at least some of these services need visibility of amavisd processes through signals (kill). At least the forwarding service must be running when amavisd is operational with $enable_zmq at true, otherwise amavisd processing might eventually stall when their message queue fills up. Preferably amavis-service processes should be started before amavisd is started, although things would eventually catch up even if started late or restarted during operation.
amavisd-status is a user utility program, similar to amavisd-nanny, which connects to amavis-service 0MQ socket and displays a status of running amavisd child processes. This program communicates with amavis-service processes through an inet socket and can in principle run on a different host (in which case sockets must not be bound to a loopback interface). The program can be started and stopped at any time, can run under any UID as long as it has access to a 0MQ socket $outer_sock_specs, and may run in multiple instances if necessary.
amavisd-snmp-subagent-zmq is a SNMP AgentX program, functionally equivalent to amavisd-snmp-subagent. It collects information from amavis-service processes and passes it as a MIB to an SNMP daemon. This process communicates with amavis-service processes through an inet socket and can in principle run on a different host (in which case sockets must not be bound to a loopback interface). If access to the amavisMta MIB (1.3.6.1.4.1.15312.2.1.3) is desired, the amavisd-snmp-subagent-zmq must run on the same host as Postfix in order to have access to its queue directories. In principle there could be more than one instance of amavisd-snmp-subagent-zmq running at the same time, although this hardly serves any practical purpose.
The old amavisd-agent utility does not currently have a 0MQ equivalent; use snmpbulkwalk with net-snmp and amavisd-snmp-subagent-zmq for similar functionality.
Please see comments in amavis-service for details and configuration of sockets.
To enable amavisd child processes to start sending their status and statistics information to amavis-service services, please set a configuration variable $enable_zmq to true in amavisd.conf: $enable_zmq = 1;
Optionally a 0MQ socket can be changed, it defaults to: @zmq_sockets = ( "ipc://$MYHOME/amavisd-zmq.sock" ); The @zmq_sockets is a list of 0MQ sockets, so in principle amavisd processes can report their state to multiple instances of amavis-service.
Both the 0MQ-based ($enable_zmq=1) and the BerkeleyDB-based ($enable_db=1) monitoring implementations can coexist: use one or the other, or both at the same time, or turn off both if monitoring is not needed.
Required Perl modules are either: ZeroMQ, which interfaces with a version 2 of a libzmq library (in case of FreeBSD that would be ports net/p5-ZeroMQ and devel/zmq), or with a Crossroads I/O library libxs, which itself is similar to a version 3 of libzmq, but provides a zmq 2.1 compatibility interface; or ZMQ::LibZMQ2 and ZMQ::Constants modules with a version 2 of a libzmq library or with a Crossroads I/O library; or ZMQ::LibZMQ3 and ZMQ::Constants with a version 3 of a libzmq library (FreeBSD ports: devel/zmq-devel). Although Crossroads I/O library is natively equivalent to a libzmq version 3 library, the ZMQ::LibZMQ3 perl module does not currently support interfacing with Crossroads I/O (libxs).
Tested combinations of a Perl interface module with a message passing library:
with 0MQ ( http://www.zeromq.org/ ): ZeroMQ 0.21 + zeromq 2.2.0 ZMQ::LibZMQ2 1.01 + zeromq 2.2.0 ZMQ::LibZMQ3 1.00 + zeromq 3.1.0
with Crossroads I/O ( http://www.crossroads.io/ ): ZeroMQ 0.21 + libxs 1.2.0 (zmq v2.1 compatible) ZMQ::LibZMQ2 1.01 + libxs 1.2.0 (zmq v2.1 compatible) ZMQ::LibZMQ3 1.00 + libxs 1.2.0 (native) (no, XS not supported by LibZMQ3)
PERFORMANCE with 0MQ
When scanning messages for spam is enabled (using SpamAssassin), a spam scan takes most of the processing time and resources, so replacing a BerkeleyDB-based monitoring with a 0MQ-based monitoring brings some speedup on a busy server, but the change is not dramatic.
But as an extreme counter-example: when DKIM signing passed messages, with most other checks disabled, a speedup can be by a factor of 10. (Synthetic benchmark: 7 KiB messages, 8 child processes, log level 2, CPU Intel Core i7-960 (4 cores, 8 threads), $TEMPBASE on an SSD disk, result: 19 mail messages per second with BerkeleyDB, over 200 mail messages per second with 0MQ, and still 130 msg/s with all checks but spam scanning enabled).
SECURITY CONSIDERATIONS with 0MQ
0MQ libraries (zeromq or libxs) do not provide any application-level security beyond what is available with standard Unix or INET or INET6 sockets. This means that Unix-style inter-process sockets are protected by the usual file system's ownership and protection bits, and access to INET/INET6 sockets is only protected by interface binding and system firewall mechanisms.
Communication between amavisd child processes and the forwarding service (amavis-services msg-forwarder) goes by default over a Unix-style socket, owned by UID vscan/amavis. Communication between utilities and service processes goes by default over an INET socket bound to a loopback interface, and as such is accessible to any process running on the same host, but is not accessible from other hosts. If access to these sockets from other hosts is desired, their binding should be changed to all or to ethernet interfaces, making them accessible to any host in the network, so host-firewall rules should be implemented if access needs to be restricted.
Having said that, currently information passing through 0MQ sockets is limited to statistics and health status only, and does not affect operation of amavisd child processes, nor is any sensitive information passed around, so access to these sockets from unauthorized sources is not expected to pose a high security risk.
0MQ and IPv6
IPv6 is supported by zeromq library starting with version 3, and by libxs (any version). Because the application needs to pass information to a library about a type of sockets needed and there is no universal and backward compatible (with v2) way to do so, currently amavisd does not offer any configuration option to choose INET6 over INET on 0MQ sockets. This restriction is expected to be lifted in the next version. Currently on an IPv6-only host one can choose to use Unix-style sockets, or patch amavis programs to turn off a ZMQ_IPV4ONLY socket option (these are commented-out in present code).
NEW FEATURES - OTHER
if a module IO::Socket::IP is available, amavisd will use this module to create its client-side inet or inet6 sockets, instead of using the low-level modules IO::Socket::INET and IO::Socket::INET6. This delegates some of the dirty details handling to IO::Socket::IP, such as using the getaddrinfo(3) system service to resolve host names, and dealing with dual-stack multihomed host names. If IO::Socket::IP is not available, the IO::Socket::INET or IO::Socket::INET6 are used directly instead, to preserve compatibility. Please use a fairly recent version of IO::Socket::IP, testing was done with versions 0.08 and 0.16.
added a subroutine read_cidr() which can read a Postfix style CIDR file, with a syntax interpreted according a Postfix cidr_table(5) man page. The subroutine returns a ref to an array by default (but can also produce a hash, and is able to add data to an existing array or hash).
Typical use: @mynetworks_maps = ( read_cidr('/etc/postfix/mynetworks.cidr') ); @client_ipaddr_policy = map(($_,'MYNETS'), @mynetworks_maps);
or: @mynetworks = @{ read_cidr('/etc/postfix/mynetworks.cidr') };
For details and more complex usage see leading comments in the read_cidr subroutine;
Maintaining a temporary capture log is enabled by setting a configuration variable $enable_log_capture to true: $enable_log_capture = 1;
Enabling a log capture costs a little bit of resources as amavisd needs to assemble and format all log messages regardless of their log level, not benefiting from early pruning of log entries not reaching the $log_level. Nevertheless the small overhead is quite acceptable when troubleshooting some rarely occurring problem and keeping $log_level permanently at the max is not acceptable due to sheer volume of debug logging.
The captured log is read from a temporary file and copied to a regular log as log level 1 entries (i.e. at LOG_INFO syslog priority) if a dynamic variable $enable_log_capture_dump is true by the end of mail message processing. A chunk of captured log entries is preceded/ended by a log line: CAPTURED DEBUG LOG DUMP BEGINS CAPTURED DEBUG LOG DUMP ENDS and each such log entry has a prepended timestamp (hours, minutes, seconds with milliseconds) of a capture time.
The $enable_log_capture_dump variable can be turned on directly by some debugging patch code, but is more conveniently loaded by activating a policy bank, e.g.:
$policy_bank{'SLOW'} = { enable_log_capture_dump => 1, };
$policy_bank{'GOTCHA'} = { enable_log_capture_dump => 1, };
which can be loaded for example by a custom hook, e.g.:
sub after_send { my($self,$conn,$msginfo) = @_; if (Time::HiRes::time - $msginfo->rx_time > 5.5) { Amavis::load_policy_bank('SLOW', $msginfo); } # or perhaps: if ($msginfo->sender =~ /some-regexp/) { Amavis::load_policy_bank('GOTCHA', $msginfo); } }
Btw, the only purpose of having two different policy banks in the example is to be able to see at a glance in the log which one was activated.
Example: [['zip','kmz'], \&Amavis::Unpackers::do_unzip],
which previously needed two entries: ['zip', \&Amavis::Unpackers::do_unzip], ['kmz', \&Amavis::Unpackers::do_unzip],
support an external decompressor lrzip for a .lrz format. Thanks to Jernej Porenta for a suggestion;
$enable_ldap setting is now dynamic, i.e. can be changed by a policy bank, which makes it possible to selectively disable LDAP lookups per policy bank. The LDAP code is loaded and a connection to an LDAP server is established if at least one policy bank has enable_ldap set to true (e.g. enable_ldap => 1 ) or a global $enable_ldap is true, but queries are disabled if currently active enable_ldap is false. Suggested by Tomislav Mihaliček;
Example:
$enable_ldap = 1;
$policy_bank{'GUESTS'} = {
enable_ldap => 0,
};
or the other way around:
$enable_ldap = 0;
$policy_bank{'INBOUND'} = {
enable_ldap => 1,
};
The behaviour is controlled by a setting $database_sessions_persistent: when true sessions remain open even after a SMTP session (from an MTA) has closed; when false sessions are closed after each SMTP session closedown. The default value is true for compatibility with earlier versions. Problem reported by Jernej Porenta;
it is now possible to disable calling an external file(1) utility but still have MIME parts decoding enabled: $file = undef; This may save some contents classification time, at the expense of losing results of a file(1) utility (i.e. short file type information) for banning checks. Disabling file(1) checks can be useful when most other checks are disabled too, e.g. in an amavisd instance whose only task is DKIM-signing, like after a mailing list manager fanout;
added Amavis::SpamControl::ExtProg support for an external spam scanner Bogofilter. An entry in @spam_scanners list for invoking the bogofilter program can be something like:
@spam_scanners = ( ['Bogofilter', 'Amavis::SpamControl::ExtProg', 'bogofilter', [ qw(-e -v)], # -u mail_body_size_limit => 65000, score_factor => 1.0, ], # ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin' ], );
The bogofilter interface code assigns a hard-coded score +5 to bogofilter's result status 'Spam', -5 to 'Ham' and 0 to 'Unsure'. This score is multiplied by score_factor (default 1 if not given) to produce the final spam score which is summed up with scores as contributed by other spam scanners in the @spam_scanners list. The 'X-Bogosity' header field will be inserted into forwarded message, unless prevented by a corresponding %allowed_added_header_fields entry.
Based on a patch contributed by Stephen Davies.
added Amavis::SpamControl::ExtProg support for auto-learning on external spam scanners; experimental: works, but may change in future versions; *** to be documented ***; Suggested by Jernej Porenta;
added locking options to @spam_scanners entries, to be used with external scanners which do not implement database locking by themselves. Options are: 'lock_file', 'lock_type', 'classifier_lock_type' and 'learner_lock_type'. The 'lock_file' specifies a file name on which a flock(2) is acquired. A lock type can be 'shared', 'exclusive', or 'none'. The 'shared' acquires a LOCK_SH (shared read) lock, the 'exclusive' acquires a LOCK_EX (exclusive write) lock on a given file. A default lock type is 'exclusive' if the lock_type option is missing. If either a lock_file is absent or empty, or a lock type is 'none', then no locking is performed.
Option 'classifier_lock_type' can override a generic 'lock_type' option when a scanner is requested to classify a message. Similarly, a 'learner_lock_type' option can override a generic 'lock_type' when a scanner is invoked for auto-learning.
Example:
['CRM114', 'Amavis::SpamControl::ExtProg', 'crm', [ qw(-u /var/amavis/home/.crm114 mailreaver.crm --dontstore --report_only --stats_only --good_threshold=8 --spam_threshold=-8) ], learn_ham => [ qw(-u /var/amavis/home/.crm114 mailreaver.crm --good) ], learn_spam => [ qw(-u /var/amavis/home/.crm114 mailreaver.crm --spam) ], mail_body_size_limit => 65000, score_factor => -0.20, lock_file => '/var/amavis/crm114.lock', lock_type => 'shared', learner_lock_type => 'exclusive', ],
added a global configuration setting $sa_userprefs_file (undef by default), which is passed on to SpamAssassin as a 'userprefs_filename' parameter during its initialization. If 'userprefs_filename' parameter is nonempty, SpamAssassin tries to load a file with that name as user preferences configuration file (overriding systemwide settings), otherwise it tries to load a file '~/.spamassassin/user_prefs' if it exists. Suggested by Quanah Gibson-Mount;
added a subroutine iso8601_weekday() which takes a Unix time as an argument (seconds since 1970-01-01T00:00Z), and returns a weekday number based on local time: a number from 1 through 7, beginning with Monday and ending with Sunday, as specified in ISO 8601 (EN 28601).
May be useful as a partition_tag for short-term cycling of a logging database storage (e.g. used by a pen-pals feature): $partition_tag = sub { my($msginfo)=@_; iso8601_weekday($msginfo->rx_time) };
added a macro 'weekday', which expands to a weekday number of the current message reception time, as provided by a call to iso8601_weekday($msginfo->rx_time);
added a macro 'secret_id', which expands to a secret counterpart to mail_id, such that: b64_encode(md5(b64_decode(secret_id))) == mail_id. It is encoded in base64url (RFC 4648), e.g. laL-rCJ6MBTm (with a counterpart mail_id: XlZbJeFhn4OE). Typically used to authorize releasing from a quarantine. Suggested by Antoine Nguyen;
added a macro 'mail_id' as a synonym to a macro 'i', which is a long-term unique mail_id on this system, possibly used in log and in quarantine names, encoded in base64url (RFC 4648), e.g. XlZbJeFhn4OE (with a counterpart secret_id: laL-rCJ6MBTm);
added a macro 'log_id' as a synonym to a macro 'n', which is an internal log id (also called task id, am_id) as shown in the log and by amavisd-nanny, e.g. 58725-05-2;
added a macro 'hexenc', which encodes its string arguments as hex digits, high nybble first;
added macros 'b64enc' and 'b64urlenc', which encode their arguments as base64 strings, removing the final null padding '=' characters. The 'b64enc' encodes into a character set [A-Za-z0-9+/], while the 'b64urlenc' encodes into a character set [A-Za-z0-9-_] according to RFC 4648;
added a macro 'body_digest', which expands to a digest (a hash) of a body of a mail message as computed by the algorithm chosen by a setting $mail_digest_algorithm (defaults to 'MD5', can be 'SHA-1' or 'SHA-256'). These are raw (non-encoded) bytes, not suitable for direct display. It is common to encode it with one of the macros: 'hexenc', 'b64enc', or 'b64urlenc', and possibly truncate it by a macro 'substr', e.g.: [:substr|[:b64urlenc|[:body_digest]]|0|9]
The result of: [:hexenc|[:body_digest]] is the same as the result of a legacy macro call %b.
The generated digest may end up as part of a quarantine file name (%b in templates), or via macro 'b' or 'body_digest' in notification templates or a main log entry.
OTHER
quarantining to a mbox format file was using mboxo rule for protecting a "From " line in a mail body, which made an original ">From " line indistinguishable from a protected From; now a mboxrd format rule is used, see http://en.wikipedia.org/wiki/Mbox
make MIME::Parser use $TEMPBASE as a temporary directory for scratch files instead of its default (which was /tmp, or failing over to a current directory, disregarding a TMPDIR environment variable). This can bring performance improvements if $TEMPBASE resides on an SSD or RAM disk and /tmp resides on a HDD;
distinguish an absence of an SMTP response from a negative SMTP response in an SMTP/LMTP client code for improved logging/debugging purposes; report delay time in case of a failure;
a default value for $inet_socket_bind now reflects the availability of socket protocol families INET (IPv4) and INET6 (IPv6):
if only inet6 available (IPv6-only host): '[::1]' Previously a default was always a '127.0.0.1'.
$forward_method, $notify_method and $requeue_method now default to an IPv6 address of a loopback interface ::1 instead of 127.0.0.1 when INET6 support is available and INET is unavailable (IPv6-only host);
remove an existing Authentication-Results header field only if we are capable of generating our own: keep it if $enable_dkim_verification is false or if $allowed_added_header_fields{'authentication-results'} is false;
add a field "Source-Port:" to "Abuse report format" (ARF) messages as per draft-kucherawy-marf-source-ports;
Avira SAVAPI av scanner: only log a warning instead of aborting when a QUIT command at the end of a session fails;
load all (both) applicable policy banks when %interface_policy contain both a "SOCK" entry and a Unix socket path name; and similarly when it contains both the "IPaddress:port" and a "port" entries. Previously the "SOCK" policy bank was not loaded when a socket path name entry existed in %interface_policy, and similarly a port-only -based policy bank was not loaded when a more specific "IPaddress:port" entry existed;
make use of a new SpamAssassin 3.4.0 option "skip_prng_reseeding" (description in the SpamAssassin Bug 6690);
no longer pre-load a module Mail::SpamAssassin::Plugin::SpamCop to avoid unnecessarily dragging-in modules Net::SMTP and Net::Cmd;
a spamd client code in Amavis::SpamControl::SpamdClient now obeys an option 'mail_body_size_limit' in a @spam_scanners entry and truncates a message passed to spamd (like other spam scanner interfaces do), instead of skipping a call to spamd. This interface module is mainly intended for testing spamd, or used with third-party software which uses the same spamc/spamd protocol.
modules Convert::TNEF is now made optional, instead of being required; do not load it if @decoders list is empty;
avoid a warning issued when encountering an empty ehlo-keyword in a response to an EHLO command (like on testing with a smtp-sink utility);
some fine-grained reduction in a number of generated opcodes and similar tiny optimizations; this accounts for numerous small changes in the code;
avoid some warnings issued by Test::Perl::Critic;
just in case: make sure that our SMTP responses at the incoming session are truly flushed to the socket and not stuck in a perlio I/O buffer;
updated 2.7.0 release notes, documenting that a policy bank may also be loaded based on a path name of a Unix socket receiving a connection;
updated and clarified schema and instructions in README.sql-pg based on suggested changes by Tim Howe;
fixed spelling mistakes in comments;
internal incompatible change: changed arguments and a result of a subroutine write_header; also, now it rewinds a message file by itself;
June 30, 2012
amavisd-new-2.7.2 release notes
BUG FIXES
a generated Received header field was missing the 'IPv6:' prefix in the TCP-info component of a 'by' subfield (as required by RFC 5321, section 4.1.3) when amavisd received a message over an IPv6 protocol; (btw, the TCP-info component of a 'from' subfield was correct);
changed data type of an SNMP variable LogRetries from C32 to C64 for consistency with the MIB;
updated AV entry 'AVG Anti-Virus' to consider status 403 continuation lines when searching for a virus name; suggested by Ralf Hildebrandt;
OTHER
reduce a log level to 5 on a log message: Amavis::IO::RW: Error flushing on close: ... to avoid an innocent but sinister-looking warning when a pipe to a virus scanner is broken and needs to be re-established; reported by Stefan Jakobs;
updated an AV entry for 'F-Secure Linux Security' to version 9.14; options updated by Mika Ilmaranta, a patch by Tuomo Soini;
fix a Unix socket compatibility issue with Net::Server versions 2.000, 2.001 and 2.002, where a method NS_unix_path no longer exists. This method was re-introduced for compatibility reasons in 2.003. Reported by Paul MacKenzie;
April 29, 2012
amavisd-new-2.7.1 release notes
BUG FIXES
prevent rmdir() from failing with 'Invalid argument' on Solaris 10 when deleting a temporary directory: current working directory must not be within a directory which is about to be deleted; reported and diagnosed by Maciej Uhlig;
forwarding or quarantining through a 'pipe:' method failed with "Insecure dependency in exec while running with -T switch" when a sendmail command-line option -N was needed; reported by Andreas Schulze;
when multiple sockets are specified (e.g. in $forward_method) as a redundancy/failover mechanism, and SMTP session caching is enabled, a failed forwarding session does not clear a cached session, so all further attempts are stuck with the failed server, instead of picking a different server from the list; discovered by Michael Storz;
on establishing a SMTP session when multiple sockets are specified (e.g. in $forward_method) as a redundancy/failover mechanism, the random choice never picked the last socket in a list; discovered by Michael Storz;
fix defanging by mimedefang, it was failing with perl 5.10 or later due to an unhandled "Insecure dependency in sprintf" while logging the result if the $log_level was 2 or higher, or when debugging was enabled; thanks to Steve Scotter for a problem report;
fix defanging by Anomy::Sanitizer, it was failing with an error message: "mangling by anomy failed: replacement size 0, mail will pass unmodified";
fix the 'xz' entry in a default @decoders list (in files amavisd.conf, amavisd.conf-default and amavisd); the first two variants ('xzdec' and 'xz') were glued together, so the xz decoder was only available if found under names 'unxz' or 'xzcat';
provide a workaround for a bug [rt.cpan.org #64642] in a perl module Encode, which gratuitously untaints a string when encoding or decoding it: https://rt.cpan.org/Public/Bug/Display.html?id=64642 (still unfixed in Encode 2.44, perl 5.14.2); A module Scalar::Util is now required, which should not be a compatibility problem, as this module is a Perl core module since perl 5.8.0.
avoid the use of Encode::is_utf8 due to a bug in a perl module Encode as bundled with versions of Perl 5.8.0 to 5.8.8 (fixed in March 2007):
Perl bug tracking: #32687: Encode::is_utf8 on tainted UTF8 string returns false https://rt.perl.org/rt3/Public/Bug/Display.html?id=32687 also referenced by #37170: https://rt.perl.org/rt3/Public/Bug/Display.html?id=37170
This is a re-manifestation of the same problem we had back in 2004, with a workaround provided by amavisd-new-2.2.1. Forgot that people are still using Perl 5.8 :) Reported by Peter Dieth;
fix a warning: _WARN: Invalid conversion in sprintf: "%a"
write informational messages during a stop/start/restart to stdout, instead of to stderr, avoiding unnecessary cron job messages; thanks to Cristian Seres, Sandro Janke and John Griffiths; also: https://bugzilla.redhat.com/show_bug.cgi?id=561389
fix a syntactically incorrect 'Avira SAVAPI' av entry (missing closing bracket) in a sample configuration file amavisd.conf;
minor: get_body_digest incorrectly logged 8-bit body as 8-bit header;
no longer insist on a minimal version 2.22 of a module Digest::MD5, the 'clone' method is no longer needed since amavisd-new-2.7.0;
do not call $parser->max_parts($MAXFILES) with some old versions of MIME::Parser which did not yet provide this method;
pre-load a module File::Glob even with perl 5.8.0, otherwise autowhitelisting in SpamAssasssin may fail with "Insecure dependency";
documentation: (files README.sql-mysql and README.sql-pg): fixed a field name "policy.unchecked_lover", previously incorrectly specified as "policy.unchecked_lovers_maps"; reported by TimH;
documentation: fixed the two SELECT examples in files README.sql-pg and README.sql-mysql, the field 'select' needs to be qualified with a table name: 'msgrcpt.content' to avoid ambiguity; reported by Gary V;
documentation bug in amavisd.conf-default: 'ESMTP' is not a valid setting for $protocol, just use 'SMTP' instead; reported by Pascal Volk;
COMPATIBILITY
commented out the LHA entry in the default @decoders list and in do_executable(). The program seems to be unmaintained, was seen crashing and as such it may pose a security risk; pointed out by Thomas Jarosch;
due to popular demand, bring the 'spam-tag:' log line back to log level 2 (version 2.7.0 dropped it to log level 3) to retain compatibility with some log analyzers. Caveat: 'spam-tag' string is now entirely in lowercase. Suggested by Stefan Jakobs;
OTHER
if a message is quarantined to more than one location using different quarantine methods, the SQL field msgs.quar_type indicates only the type of the last one. When archival quarantining is enabled this choice is unfortunate, as the primary quarantine type is more interesting than the permanent archival quarantine type. This is now reversed, the msgs.quar_type field now reflects the first quarantine type. Suggested by Patrick Ben Koetter.
SMTP session caching now no longer re-uses old sessions which are in use for more than a minute since their establishment; suggested by Michael Storz;
having the archive quarantine enabled should not be a sufficient reason to store information to SQL when $sql_store_info_for_all_msgs is off; Suggested by Patrick Ben Koetter.
ClamAV-clamd and ClamAV-clamd-stream av scanners: changed socket name in a sample configuration file amavisd.conf to /var/run/clamav/clamd.sock (previously the socket name was /var/run/clamav/clamd); this makes it compatible with a default socket name under several Linux distributions and under FreeBSD; suggested by Oliver Schinagl;
documentation updates;
July 1, 2011
amavisd-new-2.7.0 release notes
Contents: NEW FEATURES SUMMARY GENERAL COMPATIBILITY WITH 2.6.4 / 2.6.5 / 2.6.6 BUG FIXES SINCE 2.6.6 BUG FIXES SINCE 2.6.5 BUG FIXES SINCE 2.6.4 NEW FEATURES OPTIMIZATIONS OTHER CLEANING
NEW FEATURES SUMMARY
significant improvements affecting a pre-queue content filtering setup (time limiting, warm/flying restart, ...) - requires Postfix 2.7.0 and SpamAssassin 3.3.0, or later;
new daemon amavisd-signer makes it possible to sign mail with DKIM signatures without requiring amavisd process to have access to private signing keys;
added support for the Sophos-SSSP, Avira SAVAPI and ClamAV clamd streaming protocols allows amavisd to communicate with these antivirus solutions;
allow specifying multiple (fail-over) back-end mailers for resubmission of messages from amavisd back to MTA;
support for Postfix 2.8.0 XFORWARD IDENT, passes a local message identifier (queue id) downstream to a post-queue content filter and back to Postfix;
speedup in data transfer rate on receiving large mail via SMTP/LMTP sessions by a factor of 3.9 for plain text sessions, and by a factor of 11 for encrypted (TLS) sessions;
recognize and insert header fields as prepared by SpamAssassin 3.3.0 or later through its 'add_header' configuration option;
a new setting allows a forward_method to be chosen based on a message content type and/or recipient address; this may be useful for outgoing mail routing purposes or to implement sender reputation schemes;
per-recipient (or per- policy bank) SpamAssassin configuration files or SQL configuration sets are supported (@sa_userconf_maps), and per-recipient SQL Bayes database usernames (@sa_username_maps);
new macros: client_helo, client_addr, client_port, client_addr_port, mime2utf8, rusage, ADDEDHEADERHAM, ADDEDHEADERSPAM, banned_parts_as_attr, actions_performed, new arguments to macros dkim, header_field, HEADER, YESNO and YESNOCAPS;
@listen_sockets setting offers a unified configuration of listening sockets; it may be configured directly, or the traditional way: the $inet_socket_port, $unix_socketname and $inet_socket_bind just add their entries to the @listen_sockets list;
lists of lookup tables (the @*_maps variables) can now contain explicit SQL and LDAP lookup objects as their elements, instead of (or in addition to) the implied SQL and LDAP lookups;
a new configuration variable @virus_name_to_policy_bank_maps allows loading of policy banks based on a virus name;
a new configuration variable $mail_id_size_bits allows setting the size of randomly generated mail_id and secret_id codes;
a new configuration variable $sql_store_info_for_all_msgs allows storing information on mail messages selectively just for quarantined messages;
added SNMP counters InMsgsStatus* which combine the final mail checking status with a direction of a mail flow;
optional transparent archival quarantine, retaining envelope recipient addresses on delivery to a dedicated SMTP server;
GENERAL
With a synergy of four solutions, using amavisd-new in a pre-queue filtering setup became a sensible / better behaved solution:
the "smtpd_proxy_options=speed_adjust" Postfix option, available since Postfix 2.7.0 (20091101), improves decoupling between SMTP clients and a content filter in a proxy setup, reducing the number of content filtering processes needed for the same mail load. With this option turned on, a Postfix SMTP server receives entire message before connecting to a before-queue content filter;
a master_deadline option and its API equivalent, available in SpamAssassin since version 3.3.0, allows for time limiting on lengthy rules checking, while still providing results when a time limit is exceeded; this makes it more suitable for time-sensitive setups like a pre-queue filtering setup;
reworked sub-task time limiting in amavisd, along with its counterpart solution in SpamAssassin, makes it better suited to a real-time nature of pre-queue filtering setups where one has no control over how long SMTP clients are willing to wait at the data-end stage;
a re-purposed command line option 'reload' now does a warm restart, keeping sockets available to an MTA client at all times, thus reducing a chance that an MTA would even notice a content filter's warm restart.
Provided that required minimal versions of Postfix and SpamAssassin are available, on can try amavisd in a Postfix proxy setup. The $child_timeout setting needs to be radically reduced in this setup, matching the longest time most SMTP clients are willing to wait, and must be less than Postfix is willing to wait (smtpd_proxy_timeout), which by default is 100 s. A sensible value is somewhat less then a minute (e.g. 45 seconds). Even though RFC 5321 (section 4.5.3.2.6) recommends that clients SHOULD be willing to wait for 10 minutes at data-end stage, it is not uncommon that this recommendation is not adhered to.
Note that a pre-queue filtering setup (along with its benefits) still has all its drawbacks, like the need for more filtering processes to accommodate mail arrival rate peaks (instead of averages), and much shorter and unpredictable (client-dependent) time limits. The new features of the three products only rise the thresholds where trouble starts, and make the whole setup better behaved.
COMPATIBILITY WITH 2.6.4 / 2.6.5 / 2.6.6
due to popular demand to reduce undesired and unintentional backscatter, defaults for the settings $final_spam_destiny and $final_banned_destiny were changed. Previously they both defaulted to D_BOUNCE, new defaults are:
$final_virus_destiny = D_DISCARD; $final_banned_destiny = D_DISCARD; $final_spam_destiny = D_PASS; $final_bad_header_destiny = D_PASS;
Please adjust to will. If you have these settings configured explicitly in a configuration file, this change of a default value does not affect you.
For a pre-queue content filtering setup (smtp proxy or milter) a suitable value for undesired content is D_REJECT. For a post-queue filtering setup preferred choices are to tag-and-deliver (D_PASS), or to drop (D_DISCARD) and quarantine.
It is still possible to use a D_BOUNCE setting, but please limit and monitor your backscatter. Due to a default setting of @viruses_that_fake_sender_maps the backscatter on viruses has been fully suppressed since amavisd-new-20021116 even with a D_BOUNCE. Backscatter on high-scoring spam has been controllable since amavisd-new-20030616-p8 by a family of settings @spam(_crediblefrom)_dsn_cutoff_level(_bysender)_maps,
several ancient configuration settings were removed or deactivated, see section CLEANING below;
a command line option 'reload' has been renamed to 'restart', corresponding to a shutdown followed by a normal (cold) start; while a command line option 'reload' has been re-purposed to function as a warm/flying restart. See below for details. Protection of some files may need to be examined (configurations files and DKIM private keys should be readable for group vscan/amavis and not writable by UID vscan/amavis);
a failure of all virus scanners no longer automatically tempfails the operation, but flags a message with a CC_UNCHECKED contents category (just like a failure of decoders/dearchivers), and allows the usual controls (_destiny, quarantine*) to be used to choose behaviour. The $virus_scanners_failure_is_fatal=1 reverts to previous behaviour, see below;
a default value of $hdr_encoding and $bdy_encoding has been changed from 'iso-8859-1' to 'UTF-8' which better suits reporting of banned parts;
default encoding for reading text templates from the tail of a file 'amavisd' has been changed to 'utf8', which allows replacing a default text by a non-ascii Unicode template, encoded as UTF-8;
when using SQL for logging/penpals: three fields need to be added to a table msgrcpt: msgrcpt.content, msgrcpt.rseqnum, msgrcpt.is_local, and one to a table msgs: msgs.originating .
Semantics of msgrcpt.content is similar to msgs.content, but reflects individual recipient's settings (e.g. when a message is both banned and spam, a recipient with banning tests disabled will see a message as spam, while other recipient of the same message will consider it banned). The added field may also simplify queries by third party applications.
The field msgrcpt.rseqnum uniquely identifies/enumerates recipients within each message, typically by assigning them sequential numbers starting with 1. The only purpose of this field is to make it possible to define a primary key for the table msgrcpt, which may be needed for some clustering/partitioning purposes. Amavisd itself does not require a primary key on this table.
The field msgrcpt.is_local should be considered a boolean, its value can be: 'Y' ... yes, recipient is local, i.e. matches @local_domains_maps 'N' ... no, recipient does not match @local_domains_maps ' ' ... unknown - this is a default field value; amavisd always sets this field to either 'Y' or 'N';
The field msgs.originating should be considered a boolean, its value can be: 'Y' ... yes, message is originating from inside or from an authenticated roaming sender (the flag $originating was true); 'N' ... no, message is not submitted by our user ($originating was false); ' ' ... unknown - this is a default field value; amavisd always sets this field to either 'Y' or 'N', reflecting the $originating flag;
A combination of msgs.originating and msgrcpt.is_local tells a direction a message is traveling: originating is_local N N open relay (probably misconfigured @local_domains_maps or $originating flag not set) N Y inbound message Y N outbound message Y Y internal message (inside or authenticated -> inside) Note that a direction is a per-recipient property, a multi-recipient message can be outbound for some recipients and internal for others at the same time.
The following SQL directives can be used to add these new fields: ALTER TABLE msgrcpt ADD rseqnum integer DEFAULT 0 NOT NULL; ALTER TABLE msgrcpt ADD content char(1) DEFAULT ' ' NOT NULL; ALTER TABLE msgrcpt ADD is_local char(1) DEFAULT ' ' NOT NULL; ALTER TABLE msgs ADD originating char(1) DEFAULT ' ' NOT NULL;
If a primary key on table msgrcpt is needed for some reason, try something like the following: MySQL: UPDATE msgrcpt SET rseqnum=1+floor(999999999rand()) WHERE rseqnum=0; ALTER TABLE msgrcpt ADD PRIMARY KEY (partition_tag,mail_id,rseqnum); PostgreSQL: UPDATE msgrcpt SET rseqnum=1+floor(999999999random()) WHERE rseqnum=0; CREATE UNIQUE INDEX msgrcpt_idx_primary ON msgrcpt (partition_tag,mail_id,rseqnum);
If keeping a possibly customized copy of %sql_clause in a configuration file, entries 'ins_rcp' and 'upd_msg' will need to be updated accordingly.
To facilitate transition from 2.6.6 to 2.7.0, it is possible to configure amavisd 2.7.0 to supply with SELECT and INSERT clauses a subset of parameters as used by 2.6.6. A configuration setting $sql_schema_version controls this backward compatibility. Its default value is 2.007000 . By setting it to a value below 2.007000 (such as 2.006006 or 2.006004) a subset of parameters as was used with a version 2.6.6 or 2.6.4 is selected. SQL clauses in $sql_clause{'upd_msg'} and $sql_clause{'ins_rcp'} need to be adjusted according to a chosen version of actual parameters. Below is an example of a required setting compatible with both amavisd-new 2.6.6 and 2.7.0, which lets amavisd 2.7.0 use an SQL schema of 2.6.6, which lacks the four newly added fields:
our($sql_schema_version) if $myversion_id_numeric < 2.007000; $sql_schema_version = 2.006006; $sql_clause{'upd_msg'} = 'UPDATE msgs SET content=?, quar_type=?, quar_loc=?, dsn_sent=?,'. ' spam_level=?, message_id=?, from_addr=?, subject=?, client_addr=?'. ' WHERE partition_tag=? AND mail_id=?'; $sql_clause{'ins_rcp'} = 'INSERT INTO msgrcpt (partition_tag, mail_id, rid,'. ' ds, rs, bl, wl, bspam_level, smtp_resp) VALUES (?,?,?,?,?,?,?,?,?)';
Note that this is only provided to facilitate transition. Please add the new fields on an opportunity, then remove the above settings from your configuration file and restart amavisd.
SQL fields msgs.content and msgrcpt.content used to encode a content type CC_SPAMMY as 's', and CC_MTA as 't'. With default case-insensitive queries on a data type CHAR it was not possible to distinguish between lowercase 's' (= CC_SPAMMY) and uppercase 'S' (= CC_SPAM), so the CC_SPAMMY is now encoded as 'Y', and CC_MTA as 'T' (just in case). Please adjust your management tools if necessary.
please check SQL data types on fields msgs.mail_id, msgs.secret_id, msgrcpt.mail_id and quarantine.mail_id in existing databases, these must be treated case-insensitively - see details further down, please search further down for "must be treated case-insensitively";
SQL clause $sql_clause{'sel_quar'} no longer uses a coalesce() function (introduced in amavisd-new-2.6.2) which attempted to deal with NULL quarantine.partition_tag or with undefined $partition_tag, when releasing a message from an SQL quarantine - but payed the price of not using an index. If releasing from an SQL quarantine is desired, either ensure there are no (old) records in a table 'quarantine' with a NULL partition_tag (e.g. replace such fields with a 0, and don't leave $partition_tag undefined in amavisd.conf - set it to 0 for example when partitioning is not needed), or assign a former clause to $sql_clause{'sel_quar'} in amavisd.conf :
$sql_clause{'sel_quar'} = 'SELECT mail_text FROM quarantine'. ' WHERE coalesce(partition_tag,0)=coalesce(?,0) AND mail_id=?'. ' ORDER BY chunk_ind';
Thanks to Michael Scheidell and Thomas Gelf for pointing out the inefficiency.
if using Petr Rehor's amavisd-milter to call amavisd, please update it to version 1.5.0 (or later), as earlier versions did not accept a new attribute 'log_id', which is now included in a response from amavisd;
a sample configuration file amavisd.conf-sample was removed from the package - it hasn't been seriously updated for years, and it contained lots of aged or distracting information;
old helper programs amavis.c and amavis-milter.c are no longer distributed with the package, along with the entire helper-progs subdirectory. As a milter client please use the more modern 'amavisd-milter' package by Petr Rehor, available at http://sourceforge.net/projects/amavisd-milter/
old AM.CL protocol is no longer supported; it was provided for compatibility with versions of AMaViS pre-dating amavisd-new, along with its client programs: old helper programs amavis.c and amavis-milter.c. Handling of release requests and milter requests through AM.PDP protocol remains unaffected;
a sample AM.PDP client program for mail submission to amavisd which was previously distributed as 'helper-progs/amavis.pl' has been renamed to 'amavisd-submit' and slightly modernized. It provides partial functional compatibility with a very early AMaViS client program amavis.c . It takes a message on stdin, copies it to a temporary file, passes its name to amavisd daemon using AM.PDP protocol, and based on the response adjusts its exit status value so that an invoking script or program may decide whether to deliver the mail message or not;
mail_id and secret_id are now composed of characters from a character set [ A-Z, a-z, 0-9, -, _ ] instead of [ A-Z, a-z, 0-9, +, - ] (i.e. now uses underline instead of a plus) to conform to RFC 4648 base64url specification, thus making it potentially easier to specify an id in various GUI/API interfaces without a need for quoting a plus. The change is also reflected in a choice of quarantine file names. Compatibility with releasing of old quarantined messages is retained;
relationship between mail_id and secret_id has changed and is now: mail_id = encode_base64(md5(decode_base64(secret_id))) (i.e. md5 is applied to 9 raw bytes of a secret id) while previously it was: mail_id = encode_base64(md5(secret_id)) (i.e. md5 was applied to 12 base64 characters of a secret id) Releasing from a quarantine still understands old relationship and old base64 encoding character set for compatibility, so no problems are expected even when releasing a mix of old and new quarantined messages. The change may potentially affect some third party application.
caching of virus and spam check results based on a mail body hash has been removed. It was very beneficial years ago when virus storms were common and spam was not personalized. Nowadays (2011) the feature barely pays for itself (savings are comparable to additional processing needed), and is incompatible with per-recipient spam checks (as introduced with this version), and incompatible with DKIM verification on locally originating and signed mail being returned from a mailing list. Rather than trying to fit a square peg into a round hole, the feature is now dropped.
Associated configuration variables are still declared for compatibility, but have no effect: $enable_global_cache, $virus_check_negative_ttl, $virus_check_positive_ttl, $spam_check_negative_ttl, $spam_check_positive_ttl
a default value for $lock_file is now undefined instead of the former default value "$helpers_home/amavisd.lock"; an undefined value lets Net::Server choose a suitable temporary file (POSIX::tmpnam) for 'flock' serialization on socket accept();
updated (rarely used) AV entries 'Sophos SAVI', 'Mail::ClamAV' and 'av_smtp' in an incompatible way (they now use ask_daemon interface instead of a dedicated subroutine), please update your AV entries according to the new sample file amavisd.conf;
internal: spam_level() and spam_tests() are no longer properties of a message but are now a property of each recipient, which makes possible per-recipient spam checking settings (e.g. rules, bayes username, ...);
internal: a delivery_method() is no longer a property of a message, but is now a property of each recipient, which makes per-recipient forwarding possible;
internal: a load_policy_bank() takes one additional argument $msginfo, which is passed on to any policy bank's ACTION routine if a policy bank has one;
BUG FIXES SINCE 2.6.6
BUG FIXES SINCE 2.6.5
All bug fixes that were developed during a 2.7.0 developement cycle have been backported to the 2.6 branch and released as 2.6.6. They are all documented in release notes of the 2.6.6 release.
BUG FIXES SINCE 2.6.4
All bug fixes and some compatibility measures that were developed during a 2.7.0 developement cycle have been backported to the 2.6 branch and released as 2.6.5. They are all documented in release notes of the 2.6.5 release.
NEW FEATURES
A downside is that during a restart existing sockets are closed, so until new sockets are re-established an MTA client experiences connection failures, which is particularly disruptive in a pre-queue MTA setup.
Sockets remain open and available to clients during the whole warm-restart period, requests are queued by kernel (queue size is configurable through $listen_queue_size, defaulting to SOMAXCONN, or there may not be any queues at all when an IP stack is using SYN cookies), so apart from a delay in connection establishment, an MTA client will not notice a restart as long as the IP stack is willing to accept new sessions (as controlled by listen queue size or SYN cookies). This makes the reloading method particularly suitable for pre-queue filter setups.
A downside is that a HUP-ed daemon has already dropped root privileges during its first start, so it must restart as a nonprivileged user (typically 'vscan' or 'amavis'), which rules out its ability to chroot, and requires that configuration files, DKIM signing keys files, and perl modules must be readable by this GID or UID, otherwise a restart fails and a daemon process no longer exists. Depending on a version of perl and operating system in use, it might be necessary to specify an absolute path to amavisd on the initial start. To debug warm-restart problems it may be useful to first try a warm restart on a non-daemonized process (started manually as: amavisd foreground, or: amavisd debug), so that potential errors on stderr are visible.
A sensible protection of configuration files and files with DKIM keys is to set their group ownership to vscan (amavis) and UID ownership to root, and mode to 0640 (u=rw,g=r,o=).
A need for non-root accessibility of DKIM signing keys can be avoided by using a new signing service daemon included with this release (see further down: amavisd-signer).
One additional feature of a warm reload is that SNMP counters in a database (visible through amavisd-agent or amavisd-snmp-subagent) are not reset to zero, unlike the restart which clears them.
on stop, restart or reload, currently busy child processes are left to complete their current task instead of being abruptly stopped. This minimizes a disruption experienced by MTA.
added a client-side and server-side support for the IDENT attribute of a Postfix XFORWARD smtp command (available since Postfix version 2.8.0). The attribute allows passing of a local message identifier (MTA queue id) downstream from a front-end MTA to a post-queue content filter and back to a back-end MTA. Amavisd makes this information available through an existing macro %Q (which was previously non-empty only in milter setups), and as such the information appears in the log when using a default amavisd log template. This information is also passed back to a re-entry MTA if it announces a support for this attribute (enabled on a back-end smtpd service with an option smtpd_authorized_xforward_hosts), so the log entries are now easier to correlate in a post-queue filtering setup:
back-end MTA: postfix/smtpd[72995]: 553261D1CB0: client=localhost[::1], orig_queue_id=2F5971D1CA3, orig_client=...
post-queue content filter: amavis[20341]: (20341-15) Passed CLEAN ... Queue-ID: 2F5971D1CA3, queued_as: 553261D1CB0
front-end MTA: postfix/lmtp[73130]: 2F5971D1CA3: ... relay=127.0.0.1[127.0.0.1]:10024, status=sent (250 2.0.0 from MTA(smtp:[::1]:10025): 250 2.0.0 Ok: queued as 553261D1CB0)
support Postfix 2.9 long queue IDs (enable_long_queue_ids=yes) as available since postfix-20110321 by adjusted default values of $log_short_templ and $log_verbose_templ templates;
improved support for pre-queue content filtering setups: reorganized time limiting on processing to obey more strictly a deadline time, which is the sum of $child_timeout and a timestamp at the moment of a reception of a complete message (SMTP data-end time). The deadline time is also passed to SpamAssassin, which since version 3.3.0 supports a 'master_deadline' option and can gracefully terminate its processing on a time limit, while still providing results collected so far.
The setting $sa_timeout is now retired: the variable is still declared for backward compatibility, but has no effect. Instead, the time available for spam scanning is automatically determined from $child_timeout, taking into consideration the actual time left till the deadline;
$child_timeout and $smtpd_timeout settings are now dynamic, i.e. can be changed by a policy bank, which makes it possible to support (on different ports) both the pre-queue and post-queue (e.g. fallback) clients by the same amavisd daemon;
a new configuration variable $soft_bounce (also a member of policy banks) turns rejects, bounces and discards into a temporary failure when true; this is potentially useful as a short-term safety net when testing configuration changes on a low-traffic server;
added an AV entry and supporting code for Sophos-SSSP, implementing the client side of the Sophos SSSP protocol, talking to a savdid daemon (a replacement for Sophie) using its native protocol;
added an AV entry and supporting code for AVIRA SAVAPI protocol, implementing the client side of the protocol, talking to a savapi daemon;
added an AV entry for clamdscan which can serve as a useful backup scanner, connecting as client to a remote clamd; the supplied alternative config file should specify the host IP and port number where clamd is running using TCPAddr and TCPSocket options; suggested by Michael Scheidell;
added an AV entry for ClamAV clamd streaming which can serve as a main or backup scanner, connecting as client to a remote clamd. The client side implements clamd zINSTREAM command, batched in a zIDSESSION / zEND group. This approach is comparable to what is implemented in clamdscan and is somewhat less efficient than passing only a directory name to clamd, but has an advantage that the clamd daemon need not have direct access to amavisd temporary files, and may even be running on a remote host. The cost of this flexibility is additional data transfer. Suggested by Michael Scheidell.
lists of lookup tables (the @*_maps variables) can now contain explicit SQL and LDAP lookup objects as their elements, instead of (or in addition to) the implied SQL and LDAP lookups.
A new configuration setting $lookup_maps_imply_sql_and_ldap controls whether the SQL and LDAP lookup objects are implicitly prepended to list in @*_maps variables (when true), or not (when false). The default value is 1 for compatibility with previous versions.
Regardless of the $lookup_maps_imply_sql_and_ldap setting, the @*_maps lists of lookup tables/objects may now contain explicit lookup objects for arbitrarily named SQL fields and LDAP attributes. This provides more flexibility: the order of lookups is now configurable (previously SQL and LDAP lookup objects were prepended to lists and thus always looked up first), and the names of SQL fields or LDAP attributes can now be specified as arguments to SQL and LDAP lookup objects (previously field and attribute names were hardwired into code).
Three shorthand functions are available for creating SQL lookup (query) objects: q_sql_s, q_sql_n, q_sql_b, and three for creating LDAP lookup (query) objects: q_ldap_s, q_ldap_n, q_ldap_b. The _s, _n and _b suffixes imply a data type of the expected result: a string, a numeric value, and a boolean. Due to Perl's forgiveness a string data type can in most cases be used as a number or as a boolean and may be used when data type conversion and value normalization is not necessary or when a data type is not known.
Here are some examples:
@spam_kill_level_maps = ( { # a hash-type lookup object 'user1@example.com' => 8, '.example.org' => 7.5, }, q_ldap_n('amavisSpamKillLevel'), # an LDAP lookup object q_sql_n('spam_kill_level'), # an SQL lookup object $sa_kill_level_deflt, # a constant-type pseudo-lookup object );
@spam_subject_tag2_maps = ( q_sql_s('subject_tag'), );
In addition to simple scalar arguments (a field or attribute name), these six lookup object-creating functions can take as their argument a listref of field or attribute names, or a hashref where hash entry values are SQL field names (or LDAP attribute names), and hash entry keys are the result data names. Lookups resulting from such lookup objects will return a hashref of key/value pairs instead of a single scalar result. This is currently only useful in the @dkim_signature_options_bysender_maps list of lookups which expects such hash results (sets of data names and their values, i.e. entire records).
The listref argument is just a shorthand notation which can be used in place of a hashref when field names (or attribute names) are the same as the desired result data names. The following alternatives are equivalent: q_sql_s( { 'd' => 'd', 's' => 's', 'ttl' => 'ttl' } ) q_sql_s( { d => 'd', s => 's', ttl => 'ttl' } ) # perl shorthand q_sql_s( [ 'd', 's', 'ttl' ] ) q_sql_s( [qw(d s ttl)] ) # perl shorthand
Example (artificial, not necessarily useful):
@dkim_signature_options_bysender_maps = (
q_sql_s( ['d', 's', 'ttl'] ),
q_ldap_s( ['d', 's', 'a'] ),
q_ldap_s({ d => 'sdid', s => 'amavisSelector', a => 'amavisDkimAlg' }),
{ 'postmaster@example.com' => { a => 'rsa-sha1', ttl => 7*24*3600 },
'.' => { a => 'rsa-sha256', ttl => 30*24*3600 },
},
);
a new configuration variable $sql_store_info_for_all_msgs when turned off requests storing information on mail messages selectively just for quarantined messages. At the same time turning this setting off also disables pen pals lookups. A default value is 1 (true) as before, indicating that information on all messages is to be stored into tables msgs, msgrcpt and maddr when @storage_sql_dsn is enabled, thus ensuring long-term uniqueness of mail_id and proper operation of pen pals lookups;
a new program is included with a package: amavisd-signer. It is a DKIM signing service daemon for amavisd. It uses an AM.PDP protocol lookalike to receive a request from amavisd and provides two services: choosing a signing key, and signing a message digest with a chosen DKIM private key.
Amavisd uses this signing service when a $dkim_signing_service setting is defined and nonempty, and $enable_dkim_signing is true. For each mail message meeting the basic requirements for signing (originating, nonspam and not infected), the first request sent to a signing service passes some information about the message (its author, sender, recipients) and expects the service to choose and provide a suitable signing domain and selector (and optional signature options) when a signing key is available and the service considers it appropriate to sign the message. If the response does not provide a signing domain and selector, amavisd falls back to consulting its own settings (a dkim_key() set of signing keys, and @dkim_signature_options_bysender_maps).
The second stage of signing occurs when a signing key has been uniquely identified during the first stage, i.e. when its signing domain and a selector have been determined. Amavisd computes a message digest according to DKIM specifications and passes it to the signing service, along with the signing domain and selector name. The signing service computes the signature and returns it as a 'b' attribute (corresponding to a 'b' tag of a DKIM signature), from which amavisd assembles the signature header field and inserts it into a message. The signing service may still choose not to sign at this stage, e.g. when a private key corresponding to the requested signing domain and selector is not available. If a signing service is not available or cannot sign, amavisd falls back to its own configured list of signing keys ( dkim_key() ) for backward compatibility.
The main reason for separating the signing act from the main amavisd daemon is to make it possible to do the DKIM signing without letting amavisd have access to private keys - following the minimal 'need-to-know' security principles. For example, amavisd may be started as non-root or restarted from a jail, while the independent amavisd-signer process remains the only process with access to private signing keys (by running as root or under a separate UID or GID or with an access to a crypto device).
Additional benefit is that more complex decisions on which signing key to use for which mail message can be delegated to the signing service, which can be customized (through code changes or replacing it altogether) without touching the main amavisd daemon.
To let amavisd use a signing service, specify the signing service's IP address and TCP port number in amavisd.conf, e.g.: $dkim_signing_service = '127.0.0.1:20203'; matching the $inet_socket_bind and @listen_sockets settings near the beginning of the 'amavisd-service' file, then start the signing service and restart amavisd daemon. Currently all the settings for amavisd-service are contained in its file, no external configuration file or command line options are available at present.
a constant D_TEMPFAIL has been added to a set of allowed final_*_destiny values; mostly intended for completeness and testing;
a new setting @listen_sockets offers a unified configuration of listening sockets. This list may be configured directly, or the traditional way: the $inet_socket_port and $unix_socketname just add their entries (if any) to the @listen_sockets list, and $inet_socket_bind provides a default binding IP address for inet or inet6 ports. Each socket specification may either be a unix socket path (as in $unix_socketname), or an inet or inet6 socket specification (a binding IP address as in $inet_socket_bind, combined with a port number $inet_socket_port, delimited by a colon, e.g. '127.0.0.1:10024', '[::1]:10024', '10024'. When only a port number is specified without an IP address, the binding address defaults to $inet_socket_bind, which in turn (if left undefined), defaults to all interfaces.
An 'unspecified' binding address '0.0.0.0' implies any socket of the inet family (IPv4), while an 'unspecified' address '::' implies any socket of an inet6 (IPv6) family. Depending on the operating system and its settings an inet6 socket may or may not be able to also accept inet connections. To be able to listen on inet6 (IPv6) sockets requires version 2.0 (or later) of the module Net::Server, or a patched Net::Server 0.99.
Example: @listen_sockets = ( '10024', "$MYHOME/amavisd-proxy.sock", '0.0.0.0:10010', '127.0.0.1:10012', '10026', '9998', '[::1]:10028' );
Typical configuration variables where this feature is available are: $forward_method, $notify_method, $resend_method, $release_method, and $requeue_method, but only when the specified protocol is smtp: or lmtp:, (not pipe:, local:, sql:, bsmtp:).
Example: $forward_method = [ 'smtp:[::1]:10025', 'smtp:[127.0.0.1]:10025', 'smtp:*:10025' ];
$notify_method =
[ 'smtp:*:*', 'smtp:192.0.2.10:10025' ];
It is assumed that the protocol specification scheme (e.g. 'smtp:') of all entries in a list is the same. Mixing different protocols in the same list of alternatives is not allowed;
Note that the purpose of password scrambling is only to prevent an accidental or automatic opening of an attachment. It is not intended to be a strong mechanism for keeping messages secret. There is no point in providing excessively long / strong passwords.
A template for this first plain-text MIME part can be changed as before by assigning a new text to $notify_release_templ (or modifying a default template near the end of the file 'amavisd').
Three new configuration variables are added, all three are also members of policy banks: $attachment_email_name, $attachment_outer_name, and $attachment_password. Their default values are: $attachment_password = ''; # no password and no ZIP wrapping $attachment_email_name = 'msg-%m.eml'; $attachment_outer_name = 'msg-%m.zip';
The $attachment_email_name is a template for forming a name of a file, which is then inserted into a zip archive. This name will be seen as a filename containing an original mail message when a recipient unzips the archive. The supplied string may contain placeholders, the same placeholders are recognized as for filename templates used to control quarantining. For the record, here is a complete list of placeholders currently recognized: %P => $msginfo->partition_tag %b => $msginfo->body_digest %m => $msginfo->mail_id %n => $msginfo->log_id %i => iso8601 timestamp of a message reception time by amavisd %% => %
The $attachment_outer_name is a template for forming a name of a ZIP archive which will be attached to a message. This name will end up in a MIME sub-header field of the attachment, and as such will be used as a filename when a recipient saves the attachment (without unzipping it). The supplied string may contain the same placeholders as above.
The $attachment_password setting can be:
. an empty string, in which case no ZIP wrapping will occur and no passwords are applied; also the settings $attachment_email_name and $attachment_outer_name have (currently) no effect, as the attachment is not a ZIP archive but the original message itself; this is a default setting for compatibility with earlier versions;
. a fixed static string, in which case an original message is wrapped in a ZIP archive and the archive is encrypted with this fixed string password;
. an undefined value ( $attachment_password = undef ), in which case a 4-digit random password (PIN) is internally generated for each quarantine release, the rest is the same as with a fixed string;
. a subroutine reference, in which case the supplied subroutine is called (in a scalar context), passing it a $msginfo object as the only argument; the subroutine is expected to return a password as a string, or die if it cannot do its job; the returned value is then treated as one of the three cases above, i.e. an empty string disables zipping, an undefined value invokes internal PIN generating code, and any other value is taken as a password for encrypting the archive.
Example use: $release_format = 'resend'; # choices: plain, resend (default), attach
or: $release_format = 'attach'; $attachment_password = ''; # no archive, just plain attachment
or: $release_format = 'attach'; $attachment_password = undef; # internally generated 4-digit random PIN
or: $release_format = 'attach'; $attachment_password = 'fooBAR'; # fixed password
or: $release_format = 'attach'; $attachment_password = sub { my($msginfo) = @_; my $str = qx'pwgen -N 1 -n -B -s 6'; die "pwgen failed, exit status: $?" if $?; die "pwgen returned empty result" if $str eq ''; return $str; };
As the $attachment_password is a member of policy bank, it is possible to configure amavisd to listen to release requests on two TCP ports, for example using one to release unencrypted false-positive spam messages, and the other to release possibly problematic infected messages.
updated amavisd-snmp-subagent and AMAVIS-MIB.txt by adding ten user-specifiable 64-bit counters and ten user-specifiable 32-bit gauges; Counters are placed into OID tree under 1.3.6.1.4.1.15312.2.1.1.17 and named UserCounter1..UserCounter10, whereas gauges are placed under 1.3.6.1.4.1.15312.2.1.1.18 and named UserGauge1..UserGauge10. A custom hook or a policy bank ACTION hook can be used for adjusting their values by calling snmp_count64() routine, e.g.: Amavis::Util::snmp_count64('UserCounter3', 'UserCounter9'); or: Amavis::Util::snmp_count( ['UserCounter4', 1234, 'C64'] );
updated amavisd-snmp-subagent and AMAVIS-MIB.txt by adding the following counters, all placed under 1.3.6.1.4.1.15312.2.1.1 :
.19.1 InMsgsStatusAcceptedAll same value as .2.7 .19.2 InMsgsStatusAcceptedInbound .19.3 InMsgsStatusAcceptedOutbound .19.4 InMsgsStatusAcceptedInternal .19.5 InMsgsStatusAcceptedOriginating .19.6 InMsgsStatusAcceptedOpenRelay
.20.1 InMsgsStatusRelayedUntaggedAll (no equivalent) .20.2 InMsgsStatusRelayedUntaggedInbound .20.3 InMsgsStatusRelayedUntaggedOutbound .20.4 InMsgsStatusRelayedUntaggedInternal .20.5 InMsgsStatusRelayedUntaggedOriginating .20.6 InMsgsStatusRelayedUntaggedOpenRelay
.21.1 InMsgsStatusRelayedTaggedAll (no equivalent) .21.2 InMsgsStatusRelayedTaggedInbound .21.3 InMsgsStatusRelayedTaggedOutbound .21.4 InMsgsStatusRelayedTaggedInternal .21.5 InMsgsStatusRelayedTaggedOriginating .21.6 InMsgsStatusRelayedTaggedOpenRelay
.22.1 InMsgsStatusDiscardedAll same value as .2.9 .22.2 InMsgsStatusDiscardedInbound .22.3 InMsgsStatusDiscardedOutbound .22.4 InMsgsStatusDiscardedInternal .22.5 InMsgsStatusDiscardedOriginating .22.6 InMsgsStatusDiscardedOpenRelay
.23.1 InMsgsStatusNoBounceAll same value as .2.10 .23.2 InMsgsStatusNoBounceInbound .23.3 InMsgsStatusNoBounceOutbound .23.4 InMsgsStatusNoBounceInternal .23.5 InMsgsStatusNoBounceOriginating .23.6 InMsgsStatusNoBounceOpenRelay
.24.1 InMsgsStatusBouncedAll same value as .2.11 .24.2 InMsgsStatusBouncedInbound .24.3 InMsgsStatusBouncedOutbound .24.4 InMsgsStatusBouncedInternal .24.5 InMsgsStatusBouncedOriginating .24.6 InMsgsStatusBouncedOpenRelay
.25.1 InMsgsStatusRejectedAll same value as .2.12 .25.2 InMsgsStatusRejectedInbound .25.3 InMsgsStatusRejectedOutbound .25.4 InMsgsStatusRejectedInternal .25.5 InMsgsStatusRejectedOriginating .25.6 InMsgsStatusRejectedOpenRelay
.26.1 InMsgsStatusTempFailedAll same value as .2.13 .26.2 InMsgsStatusTempFailedInbound .26.3 InMsgsStatusTempFailedOutbound .26.4 InMsgsStatusTempFailedInternal .26.5 InMsgsStatusTempFailedOriginating .26.6 InMsgsStatusTempFailedOpenRelay
For compatibility, the following counters appear in the MIB at two locations (both locations present the same value): .19.1 or .2.7 InMsgsStatusAcceptedAll (.20.1 - InMsgsStatusRelayedUntaggedAll) (.21.1 - InMsgsStatusRelayedTaggedAll) .22.1 or .2.9 InMsgsStatusDiscardedAll .23.1 or .2.10 InMsgsStatusNoBounceAll .24.1 or .2.11 InMsgsStatusBouncedAll .25.1 or .2.12 InMsgsStatusRejectedAll .26.1 or .2.13 InMsgsStatusTempFailedAll
The value of a counter .2.8 InMsgsStatusRelayed is a sum of: .20.1 InMsgsStatusRelayedUntaggedAll .21.1 InMsgsStatusRelayedTaggedAll
Their semantics is documented in AMAVIS-MIB.txt . Suggested by Patrick Ben Koetter.
Example use: $policy_bank{'TRUSTED_BOOKSHOPS'} = { bypass_spam_checks_maps => [1], spam_lovers_maps => [1], ACTION => sub { Amavis::Util::do_log(2,'Buying a book?'); Amavis::Util::snmp_count64('UserCounter2'); }, }; @author_to_policy_bank_maps = ({ 'amazon.com' => 'TRUSTED_BOOKSHOPS', 'amazon.co.uk' => 'TRUSTED_BOOKSHOPS', 'amazon.de' => 'TRUSTED_BOOKSHOPS', });
The @virus_name_to_policy_bank_maps is a list of lookup tables. A lookup key is each virus name as reported by any virus scanner. A result of a lookup is expected to be a string containing a comma-separated list of policy bank names. Nonexistent policy banks are ignored. Duplicate names are merged into a single name. The most suitable lookup mechanisms are a regexp lookup and a hash lookup, as these are able to provide an arbitrary user-specifiable result (unlike a list-based (ACL) lookup, which can only provide a boolean value). Suggested by Patrick Ben Koetter.
Example use:
@virus_name_to_policy_bank_maps = (
new_RE( # a regexp lookup
[ qr'^(W32/MyDoom|W32/Netsky|Mal/BredoZp)' => 'VIRUS,MASS_VIRUS' ],
[ qr'?EICAR?'i => 'EICAR_TEST' ],
),
'VIRUS', # constant (pseudo)lookup, catchall for any other virus name
);
$policy_bank{'VIRUS'} = {
ACTION => sub { Amavis::Util::snmp_count('UserCounter1') },
};
$policy_bank{'EICAR_TEST'} = {
log_templ => $log_short_templ . ', EICAR test message, not to worry',
final_destiny_by_ccat => { CC_VIRUS() => D_BOUNCE },
};
$policy_bank{'MASS_VIRUS'} = { # mute everything using a big hammer
final_destiny_by_ccat => { CC_VIRUS() => D_DISCARD },
warnsender_by_ccat => { REPLACE => 1 },
warnrecip_maps_by_ccat => { REPLACE => 1 },
quarantine_method_by_ccat => { REPLACE => 1 },
admin_maps_by_ccat => { REPLACE => 1 },
newvirus_admin_maps => [],
log_templ => 'MASS VIRUS DROPPED, ' . $log_templ,
ACTION => sub { Amavis::Util::snmp_count('UserCounter2') },
};
Example use:
@listen_sockets = (
"$helpers_home/amavisd.sock1",
"$helpers_home/amavisd.sock2",
"$helpers_home/amavisd.sock3",
);
$interface_policy{"$helpers_home/amavisd.sock1"} = 'UX-S1';
$interface_policy{"$helpers_home/amavisd.sock2"} = 'UX-S2';
$interface_policy{"$helpers_home/amavisd.sock3"} = 'UX-S3';
$policy_bank{'UX-S1'} = { ... };
$policy_bank{'UX-S2'} = { ... };
$policy_bank{'UX-S3'} = { ... };
Transparent archiving is used when $archive_quarantine_to (actually the @archive_quarantine_to_maps) results in a reserved string '%a' for all recipients.
Think of the '%a' as a placeholder in a replacement string, being substituted by a full original recipient address. There may be other substitution placeholders available in the future, equivalent to placeholders %l, %d, etc. in SQL query templates.
Example: $archive_quarantine_method = 'smtp:127.0.0.1:7777'; $archive_quarantine_to = '%a'; @archive_quarantine_to_maps = (\$archive_quarantine_to);
or: $archive_quarantine_method = ['smtp:[::1]:7777', 'smtp:127.0.0.1:7777']; @archive_quarantine_to_maps = ( { '.example.com' => '%a', '.example.net' => '%a', '.example.org' => 'quarantine@example.org', '.' => undef, } );
The envelope sender address of messages sent to an archival quarantine is still controlled by the $mailfrom_to_quarantine setting as before. When this value is undef (which is a default) the envelope sender address remains unchanged - is the same as in a received message. Any other value replaces the original sender address, so an empty string implies a null return path.
When delivering quarantine messages to a dedicated SMTP server it must be ensured that the receiving server will not bounce or reject quarantine messages or deliver them to recipients specified in the SMTP envelope!
If a receiving SMTP server announces a DSN capability in its response to EHLO, amavisd will add option NOTIFY=NEVER with each recipient to prevent potential backscatter. This is an additional safeguard to prevent potential backscatter, therefore it is recommended that the receiving quarantine server implements and announces the DSN capability. Specifying an empty string for the $mailfrom_to_quarantine achieves the same effect (a null return path implies NOTIFY=NEVER) thus preventing backscatter, but loses original sender address in the envelope.
Suggested by Patrick Ben Koetter.
as a convenience, two pre-defined logging templates are provided: $log_short_templ and $log_verbose_templ. The former is the same as was a default $log_templ in previous versions, the later is quite verbose and provides most of the interesting information about a message. An initial value of $log_templ is taken from the $log_short_templ. To change $log_templ, either assign a new template directly as before, or, as a shorthand if $log_verbose_templ is appropriate, just assign it, e.g.: $log_templ = $log_verbose_templ;
added a configuration variable @debug_recipient_maps. Similarly to @debug_sender_maps, a debug level logging is temporarily turned on for the duration of processing of this message when a recipient address matches a list of lookup tables @debug_recipient_maps; suggested by Patrick Ben Koetter;
internal: a delivery_method() is now a property of a recipient instead of being a property of a message as a whole. This makes per-recipient forwarding method selection possible. When recipients of a multi-recipient message specify different forwarding methods, a message is forwarded in multiple transactions, one for each unique delivery_method() setting; i.e. recipients are clustered into sets with the same delivery_method setting and a message for each subset of recipients if forwarded as one transaction;
finer custom control over the forwarding method is available through a before_send() custom hook which may override the $r->delivery_method(...) for all or just some of the recipients with whatever forwarding method specification is suitable - for example a next-hop server's IP address or its port number can be chosen based on spam score or based on a sender domain or some other characteristics of a message or of a recipient;
a new configuration variable @forward_method_maps (along with making a delivery_method() a property of a recipient instead of being a property of a message) makes per-recipient forwarding method selection possible; suggested by Ralf Hildebrandt;
Example: @forward_method_maps = ({ # use lowercase keys with hash-type lookups! 'user@example.com' => 'smtp:[::1]:10025', '.sub1.example.com' => 'smtp:[::1]:10026', '.sub2.example.com' => 'smtp:[::1]:10027', '.example.net' => 'smtp:[127.0.0.1]:10025', '.example.org' => [ 'smtp:[192.0.2.9]:125', 'smtp:[2001:db8::f]:125' ], '.' => $forward_method, });
Example use:
$forward_method = 'smtp:[127.0.0.1]:10025';
%forward_method_maps_by_ccat = (
CC_BADH.',3', [ 'smtp:*:10027' ],
CC_BADH.',4', [ 'smtp:*:10027' ],
CC_BADH.',5', [ 'smtp:*:10027' ],
CC_BADH.',6', [ 'smtp:*:10027' ],
CC_BADH.',8', [ 'smtp:*:10027' ],
CC_SPAMMY, [ 'smtp:[192.0.2.22]:10025' ],
CC_CATCHALL, sub { ca('forward_method_maps') },
);
added a global configuration setting $allow_preserving_evidence, defaults to true. Turning it off disables preserving temporary files (as evidence) in case of trouble, which is potentially useful for unattended and unmonitored operation. The setting has no influence on preserving evidence in case of @debug_sender_maps or @debug_recipient_maps triggering, which always preserves evidence;
an entry for CC_UNCHECKED was added to %admin_maps_by_ccat, defaulting to @virus_admin_maps. Hence administrator notifications are also sent for messages which cannot be decoded (e.g. are encrypted or contain a mangled archive) if virus administrator notifications are enabled. To turn off sending administrator notifications for unchecked contents: delete $admin_maps_by_ccat{&CC_UNCHECKED};
to avoid a need to directly manipulate the *_by_ccat settings, a set of individual configuration variables associated with CC_UNCHECKED contents category was added, with their default values compatible with earlier versions of amavisd: $final_unchecked_destiny = D_PASS; $unchecked_quarantine_method = undef; $unchecked_quarantine_to = 'unchecked-quarantine'; @unchecked_quarantine_to_maps = (\$unchecked_quarantine_to); LDAP attribute: amavisUncheckedQuarantineTo SQL field: unchecked_quarantine_to
a failure of all virus scanners no longer automatically tempfails the operation, but flags a message with a CC_UNCHECKED contents category (just like a failure of decoders/dearchivers), and allows the usual controls (_destiny, quarantine*) to be used to choose behaviour; for example: $final_unchecked_destiny = D_TEMPFAIL; $unchecked_quarantine_method = 'local:unchecked/%m.gz';
To revert to a previous behaviour where a failure of all virus scanners resulted in a temporary failure, set the $virus_scanners_failure_is_fatal to true, e.g.: $virus_scanners_failure_is_fatal = 1; The setting $virus_scanners_failure_is_fatal is a member of policy banks.
support decompression of a .xz file format and legacy .lzma file formats through XZ Utils ( http://tukaani.org/xz/ ) if an entry with a decoding program is found in the @decoders list; it defaults to finding a program 'xz' or 'xzdec' in the $path ;
added two new functions: iso8601_year_and_week() and iso8601_yearweek() to accompany the existing function iso8601_week(); they all provide a week-of-the-year number (ISO 8601 / EN 28601, 1..53, in local time zone) given a Unix timestamp (seconds since 1970-01-01T00:00Z) as an argument, optionally together with the corresponding year number. The result is an integer or a pair of integers as follows:
$w = iso8601_week($unix_time); # e.g. 49 $yw = iso8601_yearweek($unix_time); # e.g. 201049 ($y, $w) = iso8601_year_and_week($unix_time); # e.g. (2010,49)
Semantics is equivalent to PostgreSQL extract(week from ...), and to MySQL week(date,3). These functions can be useful for assigning to a $partition_tag (in amavisd.conf), e.g.: $partition_tag = sub { my($msginfo)=@; iso8601_week($msginfo->rx_time) }; or: $partition_tag = sub { my($msginfo)=@; iso8601_yearweek($msginfo->rx_time) };
or based on a day of a week for short-term cycling (Mo=1, Tu=2,... Su=7): $partition_tag = sub { my($msginfo)=@; ((localtime($msginfo->rx_time))[6]+6)%7+1 }; (a note from a future: starting with 2.8.0 the following is equivalent: $partition_tag = sub { my($msginfo)=@; iso8601_weekday($msginfo->rx_time) }; )
Suggested by Michael Scheidell.
The following replacements are made: %a -> exact/unmodified e-mail address (same as the first entry in %k) %l -> full unmodified localpart (all up to, but not including the '@') %u -> lowercased username (a localpart without extension) %e -> lowercased address extension (including a delimiter), if any %d -> lowercased domain (without '@') %k -> a list of lookup strings, as before (see below)
For example, given an e-mail address: User+Foo@Sub.Example.COM the placeholders would be substituted by:
%a User+Foo@Sub.Example.COM
%l User+Foo
%u user
%e +foo
%d sub.example.com
%k User+Foo@sub.exAMPLE.COM
user+foo@sub.example.com
user@sub.example.com
user+foo
user
@sub.example.com
@.sub.example.com
@.example.com
@.com
@.
A lookup on a list of lookup tables @sa_userconf_maps may return undef or an empty string implying no user preferences file, or may provide a file name (absolute path, or relative to $MYHOME) of a SpamAssassin's 'user preferences' configuration file, or may start with a string 'sql:' which implies loading user preferences from a user_scores_dsn SQL database (as declared in a SpamAssassin's configuration file) for a username provided by a lookup on @sa_username_maps (see further down).
SpamAssassin will be requested to load a user preferences configuration through its read_scoreonly_config() or load_scoreonly_sql() method, which otherwise (in spamd) serves to load user's .spamassassin/user_prefs file or SQL preferences when switching users. See SpamAssassin documentation file sql/README for SQL details.
SpamAssassin's SQL database is only consulted if user_scores_dsn is declared in a SpamAssassin configuration file, and the @sa_userconf_maps returns a string starting with 'sql:' (case insensitive, the rest of the string is currently ignored). If a username as provided by a lookup on @sa_username_maps equals the username under which amavisd was started, SpamAssassin's SQL preferences for that username will not be loaded - it is assumed that preferences for a default username are empty, i.e. that it uses a default SpamAssassin configuration.
Each time that currently loaded configuration needs to be replaced by another or restored to a systemwide default, an initial SpamAssassin configuration is restored through SpamAssassin's copy_config() method.
Note that saving an original SpamAssassin configuration, loading a user configuration, and restoring to the original configuration does not come cheap: it can take 200 ms for a load and restore, and 370 ms for the initial saving of the configuration (saving is only done once per child process, and only if needed). Saved configuration can occupy additional 2 MB of virtual memory, so use the feature sparingly. No penalty occurs until a child process does its first loading of a user configuration, so rarely activated or inactive policy banks or per-recipient setting using this feature do not cause any additional processing or occupy additional memory.
According to SpamAssassin documentation, a user preferences file or SQL preferences can include scoring options, scores, whitelists and blacklists, etc. If 'allow_user_rules' is enabled (local.cf), then user preferences file can also include rule definitions and privileged settings - but not administrator settings.
The feature is only available since SpamAssassin 3.3.0.
Example: @sa_userconf_maps = ( { 'user1@example.com' => '/etc/mail/spamassassin/special_user_config', '.example.org' => 'sql:', } );
Based on a suggestion by Alexander Wirt and initially based on his patch;
The $sa_num_instances controls the number of Mail::SpamAssassin objects (instances) created by a parent amavisd process during a startup. Each SpamAssassin instance does its own initialization (loading of rules and configuration settings) during a program startup and occupies a sizable portion of virtual memory (like 7 MB on a 64-bit platform with SA 3.4 rules).
When switching SpamAssassin configurations (@sa_userconf_maps), and given more than one instance of the Mail::SpamAssassin object, amavisd has a choice of picking an instance which may already have loaded a selected user configuration file previously, and thus save some time by not having to store and reload SpamAssassin state again. This may be beneficial for example when a sizable portion of users use a default SpamAssassin configuration, while other users need a per-user or per-domain preferences settings;
Note that as of SpamAssassin 3.3.2 some features (like compiled rules) are global and not a property of a SpamAssassin instance object. The problem is tracked in the SpamAssassin project as Bug 6236. Until this is resolved please consider the feature experimental.
Switching between Bayes usernames is cheap compared to switching between SpamAssassin configuration files. A multi-recipient message whose recipients map to different usernames will be checked by SpamAssassin multiple times, once for each unique username;
Example: @sa_username_maps = ( { 'user1@example.com' => 'user1', 'user2@example.com' => 'user2', '.example.com' => 'user_ex', } );
passes a value of $originating flag to SpamAssassin through its suppl_attrib argument in a $spamassassin->parse call; it is expected that this information would be treated by SpamAssassin 3.4.0 similarly to msa_networks;
a new configuration variable $mail_id_size_bits allows setting the size of randomly generated mail_id and secret_id codes which are used to identify a message on releasing it from a quarantine, and are used as a key when logging to SQL (penpals) or storing to quarantine. The variable specifies a length of mail_id in bits, and must be an integral multiple of 24 (i.e. must be divisible by 6 and by 8). The mail_id is represented externally as a base64url-encoded string of $mail_id_size_bits / 6 characters, and internally as a string of $mail_id_size_bits / 8 octets. The default value is 72 bits, as in previous versions. Sensible values are 48, 72 and 96 bits.
See entry "introduce a concept of 'mail_id'" in amavisd-new-2.3.0 release notes for probability analysis of collisions. The default size should fit all practical current needs.
The size of SQL fields msgs.mail_id, msgs.secret_id, msgrcpt.mail_id and quarantine.mail_id may need increasing to accommodate $mail_id_size_bits/6 characters if using a non-default value of $mail_id_size_bits. See also the next entry regarding a type for these fields.
SQL fields msgs.mail_id, msgs.secret_id, msgrcpt.mail_id and quarantine.mail_id must be treated case-insensitively. A suitable data type for these fields in PostgreSQL is bytea, and varbinary in MySQL (of size 12 or 16 characters). In order not to lose entropy in mail_id, and not to increase a probability of collisions, please check existing database schema and adjust as necessary, either a data type, or chose a case-sensitive collation setting. See README.sql-pg and README.sql-mysql for an ALTER command to change data type of these fields.
added optional SQL and LDAP lookups for @spam_tag3_level_maps (sql field: policy.spam_tag3_level, ldap attribute: amavisSpamTag3Level), and for @spam_subject_tag3_maps (sql field: policy.spam_subject_tag3, ldap attribute: amavisSpamSubjectTag3); suggested by Thomas Johnson;
added the following LDAP settings to $default_ldap: sslversion, clientcert, clientkey, cafile, capath, verify, sasl, sasl_mech, sasl_auth_id, localaddr, scheme, inet6; allows bind authentication with a certificate or SASL and allows connections to an LDAP server over IPv6; based on a patch by Christian Roessner;
added macros ADDEDHEADERHAM and ADDEDHEADERSPAM, which expand to newly generated header fields which SpamAssassin prepared for insertion into a header section, in case the message is eventually declared to be nonspam or spam respectively; this information is available from SpamAssassin since version 3.3.0;
macro 'dkim' now recognizes two additional keywords: 'selector', 'sig_sd', and 'newsig_sd', see README.customize for details; in particular, the 'newsig_sd' allows to add information on newly applied signature to a main log entry, and is now included in a default log template;
added a macro 'mime2utf8' which takes a string as its first argument, and an optional truncation length as the second. The string is decoded as a MIME-Header string (understands Q or B character set encodings like =?iso-8859-2?Q?...?=, =?koi8-r?B?...?=) and is converted to UTF-8, optionally truncated to the specified size at clean UTF-8 boundaries, and returned as a result. Suggested by Bastian. The macro is useful in a logging template or in notification templates to decode Subject or From header fields, e.g.:
[? [:header_field|Subject]||,\ Subject: [:dquote|[:mime2utf8|[:header_field|Subject]|100]]]#
added a macro 'client_helo', which provides a client-supplied EHLO/HELO domain name of the original SMTP session. The information is obtained through an XFORWARD extension to an SMTP protocol as provided by Postfix, or through a 'helo_name' attribute in an AM.PDP request. Add something like: , helo=[:client_helo]# to the log template if it needs to be logged. Suggested by Xueron Nee;
added a macro 'client_addr' which is a synonym for macro 'a'; the information is obtained through an XFORWARD extension to an SMTP protocol as provided by Postfix, or through a 'client_address' attribute in an AM.PDP request; if neither of these are available, the client's IP address is parsed from a topmost Received header field;
added a macro 'client_port', yields a TCP source port number of an original SMTP session; the information is obtained through an XFORWARD extension to an SMTP protocol as provided by Postfix, or through a 'client_port' attribute in an AM.PDP request. See RFC 6302;
added a macro 'client_addr_port' which combines a client's IP address and a TCP source port number (if available) of an original SMTP session; it is similar to: [[:client_addr]]:[:client_port] or when a port number is not available: [[:client_addr]]
This macro is now included in a default main log template, so the TCP source port number is logged along its IP address. This information is useful in reporting abuse (e.g. client behind a NAT), troubleshooting, forensics and law enforcement. If this information is not desired, one may assign a customized template to the $log_templ configuration variable. See RFC 6302: Logging Recommendations for Internet-Facing Servers. Suggested by Rok Potočnik.
The following example illustrates the difference. A result is a single string in both cases, wrapped here for clarity, and shows a path in a message tree of a banned leaf node:
a macro 'banned_parts' can yield:
multipart/mixed |
application/octet-stream,.rar,Setup1.1.rar |
.exe,.exe-ms,setup.exe
while a macro 'banned_parts_as_attr' yields:
P=p003,L=1,M=multipart/mixed |
P=p002,L=1/2,M=application/octet-stream,T=rar,N=Setup1.1.rar |
P=p007,L=1/2/4,T=exe,T=exe-ms,N=setup.exe
for the same banned part in a message.
The single-character attribute names are unchanged from previous versions. For documentation, here is a legend: P: part's base name, i.e. a file name in a ./parts/ temporary directory L: part's location (path) in a mail tree (branch enumeration, top-down) M: MIME type as declared in MIME header fields of a message T: short part's content type according to a file(1) utility and mapped through @map_full_type_to_short_type_maps N: declared part names (none, one or more), as declared in MIME header fields or in an archive (tar, zip, ...) A: part's attributes as follows: U=undecodable, C=crypted, D=directory, S=special(device), L=link
macro 'header_field' and its alias 'HEADER' now has an optional third parameter (index), which chooses the header field in case of multiple header fields of the same name; the default (-1) is to return the last (bottommost) occurrence, as before; see README.customize for details;
added a macro 'actions_performed', which expands into a comma-separated list of words: Accepted, Relayed(Untagged), RelayedTagged, Discarded, Rejected, Bounced, NoBounce or TempFailed, followed by a mail flow direction word: Inbound, Internal, Outbound or OpenRelay. For brevity the 'RelayedUntagged' status appears in this list as 'Relayed'. Additionally, the list may include words Quarantined and Archived. For multirecipient messages it is possible that the list includes more than one combination.
The purpose of this macro is to augment the bare-bones 'Passed CLEAN' or 'Blocked SPAM' in the main log entry. For this purpose the default log template now includes this macro call. If the additional information is not desired in the log, please assign a customized template to the $log_templ configuration variable.
Some examples of the new log entries:
Passed CLEAN {RelayedOutbound}, ...
Passed CLEAN {RelayedInbound}, ...
Passed CLEAN {RelayedInternal,RelayedOutbound}, ...
Passed SPAMMY {RelayedTaggedInbound}, ...
Blocked SPAM {RejectedInbound,Quarantined}, ...
Blocked INFECTED (Mal/BredoZp-B) {DiscardedInbound,Quarantined}, ...
Semantics of entries in the 'actions_performed' list corresponds to the newly added SNMP variables 1.3.6.1.4.1.15312.2.1.1.19 - .26 (with the exception that 'RelayedUntagged' counter is abbreviated in this macro as 'Relayed'). Please see their detailed description in a file AMAVIS-MIB.txt .
added a macro 'rusage', which expands to a resource usage entry as provided by a system service getrusage(2); the argument to the macro should be one of the field names in the structure rusage (see getrusage(2) man page), e.g. ru_utime, ru_stime, ru_maxrss, ru_ixrss, ru_idrss, ru_isrss, ru_minflt, ru_majflt, ru_nswap, ru_inblock, ru_oublock, ru_msgsnd, ru_msgrcv, ru_nsignals, ru_nvcsw, ru_nivcsw; the information is only provided if an optional perl module Unix::Getrusage is available;
SpamAssassin-compatible macros 'YESNO' and 'YESNOCAPS' now optionally accept two string arguments, replacing the default strings 'Yes' and 'No' in the result;
settings $enable_dkim_verification and $enable_dkim_signing are now dynamic, i.e. became members of policy banks, thus facilitating selectively enabling or disabling these features on a policy bank basis;
internal: added a message property object dkim_signwith_sd() which allows custom hooks to provide the DKIM signing code with a selector and a domain name preferred for choosing a signing key. If this information is not available the signing code will consult an external signing service if provided ($dkim_signing_service), or else use the built-in algorithm for choosing a signing key. A custom hook may provide the information as follows: $msginfo->dkim_signwith_sd( ['some_selector', 'some_domain'] ); After a successful signing, the dkim_signwith_sd will contain a pair [selector,domain] which was actually chosen for signing;
recognize and insert header fields as prepared by SpamAssassin 3.3.0 or later through its 'add_header' configuration option; some of the standard X-Spam-* header fields are still overruled by equivalent ones generated by amavisd itself, primarily to provide true per-recipient handling; header field names must still be listed in the associative array %allowed_added_header_fields in order to be inserted; overrides are configurable through %prefer_our_added_header_fields, for example: $prefer_our_added_header_fields{lc('X-Spam-Status')} = 0;
added an attribute 'log_id' to server responses in an AM.PDP protocol, allowing the client to match its request with the amavisd daemon logging;
added LDAP attributes: amavisAddrExtensionVirus, amavisAddrExtensionSpam, amavisAddrExtensionBanned, and amavisAddrExtensionBadHeader for consistency with SQL; suggested by Stefan Palme;
added LDAP attributes: amavisSpamTag3Level, amavisSpamSubjectTag3, amavisUncheckedQuarantineTo, amavisCleanQuarantineTo, amavisUncheckedLover, amavisForwardMethod, amavisSaUserConf and amavisSaUserName for consistency with SQL;
added LDAP attribute amavisDisclaimerOptions, along with its corresponding SQL field 'disclaimer_options'. It finds its way to the list of lookup tables @disclaimer_options_bysender_maps, so the replacement of the OPTIONS placeholder in @altermime_args_disclaimer could be made dynamic; suggested by Quanah Gibson-Mount;
for consistencly, added LDAP attribute amavisUncheckedLover, along with its corresponding SQL field 'unchecked_lover' and a statical list of lookup tables @unchecked_lovers_maps, which appears in the %lovers_maps_by_ccat. Previously the CC_UNCHECKED entry of the %lovers_maps_by_ccat was (ab)used, and shared the @virus_lovers_maps value. Suggested by Patrick Ben Koetter;
added a variable $myprogram_name, which defaults to a program name (perl variable $0), but may be modified in a configuration file typically depending on a value of $instance_name. It is used to dynamically change a process name in $0, which shows up in a ps(1) and top(1) output on most Unix systems. Along with $syslog_ident, it offers a handy way to distinguish amavisd instances.
if running other spam scanners besides SpamAssassin through a @spam_scanners mechanism (such as DSPAM or CRM114), make header fields produced by them visible to SpamAssassin too, so that its rules can benefit from additional information. Note that in order for SpamAssassin to be able to see such header fields from other scanners, such scanners must be listed in the @spam_scanners list before the 'SpamAssassin' entry. Suggested by Marco.
OPTIMIZATIONS
Measured data rates of an SMTP DATA transfer between Postfix and amavisd on a loopback interface:
No TLS (no session encryption): . amavisd receiving, old code: 8.3 MiB/s . amavisd receiving, new code: 32.3 MiB/s . amavisd sending: 18 MiB/s
With TLS (encrypted session, AES256-SHA): . amavisd receiving, old code: 1.0 MiB/s . amavisd receiving, new code: 11.2 MiB/s . amavisd sending: 4.3 MiB/s
save about 6 MB of virtual memory per amavisd child process by properly deleting some larger data items from variables, known not to be reused; thanks to the insight of Perl Monks (ikegami) in: http://www.perlmonks.org/?node_id=803515
speed up lookup_ip_acl lookups on larger lists of CIDR network addresses (like @mynetworks and @inet_acl list) by using a radix trie (Patricia Trie) representation; the patricia trie is used when a module Net::Patricia is available and a list contains more than 20 elements; minimal required version of Net::Patricia is 1.015;
avoid entering and exiting a block in most map() and grep() calls saves on opcodes, achieving a small reduction of code size and a tiny speedup;
'use constant' for CC_ and D_ constants allows perl to inline them;
OTHER
provide a workaround for a Perl 5.8.9 bug #62502, where O_WRONLY, O_APPEND and other Fcntl constants can become tainted; this is an application of the same workaround as already applied in 2.6.3, but covers two additional code sections; the bug could manifest itself as a taint problem during opening a pipe to an external mail submission problem. This only affects perl 5.8.9; the 5.8.8 and 5.10.0 are fine. Tracked down and a patch provided by Petr Rehor;
RFC 5617 now defines an "Author Domain Signature" as a valid signature in which the domain name of the DKIM signing entity, i.e., the d= tag in the DKIM-Signature header field, is the same as the domain name in the Author Address. The change came with draft-ietf-dkim-ssp-10; previously the "Author Domain Signature" was based on an 'i' tag (identity).
This change is now followed by amavisd-new 2.7.0 in macro 'dkim', in evaluation of the @author_to_policy_bank_maps list, in internal attribute dkim_author_sig(), and reflected in logging. Similarly, the evaluation of @signer_reputation_maps is now based on a signing domain ('d' tag), instead of the 'i' tag as previously. The change only affects signatures where the domain name of a signing identity is a subdomain of a signing domain (not identical to the signing domain), which is rare in current practices.
updated generating and parsing of Authentication-Results headear field according to RFC 5451 and RFC 6008 - previously it followed a draft-kucherawy-sender-auth-header. This header field is now also inserted for new DKIM signatures as just-generated and inserted to a passed internal-to-internal message when it is eligible for signing; suggested by Florian Effenberger.
added a setting $myauthservid, also a member of policy banks, which controls the "authserv-id" token in the Authentication-Results header field, according to RFC 5451. Its default value is $myhostname as before. Its value must comply with the RFC 5451 syntax ("dot-atom"). Having a separate control may facilitate setups where a message is processed by amavisd more than once, e.g. for DKIM signing of a mailing list fanout messages, where the second pass should not remove an Authentication-Results header field from a first pass.
updated ARF notifications to RFC 5965 (An Extensible Format for Email Feedback Reports); the $report_format = 'arf' implementation was based on ARF draft, now it complies with RFC 5965;
tightened some sanity limits on DKIM verification to better handle mail messages with a huge number of signatures; problem reported by Tuomo Soini;
amavisd.conf: added file types ini, lib, ocx, sys, vxd to the commented-out long list of file types in $banned_filename_re, along with a commented-out list of type names for consideration: asd, asf, asx, url, vcs, wmd, wmz;
updated default @virus_name_to_spam_score_maps with new or changed entries: [ qr'^(Heuristics.)?Phishing.' => 0.1 ], [ qr'^Doppelstern.(Scam4|Phishing)' => 0.1 ], [ qr'^ScamNailer.Phish.' => 0.1 ], [ qr'^HTML/Bankish' => 0.1 ], thank to Giampaolo Tomassoni for Heuristics.Phishing and HTML/Bankish;
when inserting a subject tag into a Subject header field, remove existing copies of the same string first to avoid subjects like "UNCHECKED Fwd: UNCHECKED Re: foo bar" ; based on a patch by Thomas Arendsen Hein;
p0f-analyzer.pl: convert an 'IPv4-mapped IPv6 addresses in alternative form' to an IPv4 address, otherwise the p0f-analyzer.pl would ignore such queries, as the p0f daemon did not handle IPv6 until version 3. The 'IPv4-mapped IPv6 addresses' is returned for an IPv4 connection when TCP/IP stack is configured to allow inet6 sockets to accept inet sessions; problem reported by Vytautas Kasparavicius;
suppress generating a non-delivery notification if a SpamAssassin test DKIM_ADSP_DISCARD is hit, honouring RFC 5617;
amavisd.conf: commented-out calls to do_ascii to match defaults in the amavisd program; the uulib code (as invoked by Convert::UUlib) has a history of stability problems, seems it is causing more grief compared to the benefits it brings;
new AV entry for 'Avira for UNIX 3.x', thanks to g0rbi, Thomas Mueller, Steffen Ille, Klaus Fuerstberger and Andreas Schulze;
add three more exception cases to mercifully ignore an EBADF I/O error due to a Perl bug on line-by-line reading;
entries 'SpamAssassin' and 'SpamdClient' in the @spam_scanners list now recognize options 'mail_body_size_limit' and 'score_factor', to match their behaviour with 'DSPAM' and 'CRM114' entries;
dropped a logging level (from -1 to 2) on a warning message: INFO: dot-stuffing error (only one leading dot): ... as Postfix in a pre-queue proxy filtering setup does not do any dot-stuffing sanitation, so garbage in the DATA section as received by a Postfix smtpd service comes unchanged to a proxy filter; reported by Ralf Heidenreich, confirmed by Victor Duchovni and Wietse Venema;
use module File::Temp to create a temporary working directory, instead of using a home-brewed code; as a result, these directory names are now a bit longer;
avoid slurping the whole directory contents into memory when recursively tidying, removing, or checking a temporary directory, when purging old database files on a restart, and when preparing a list of files to be scanned;
collect a couple of random bytes from /dev/urandom (if available) at a start of the main process and at each child process birth (when our entropy pool is rather depleted), then stir our entropy pool and perl's srand() to prevent File::Temp from working with the same pseudorandom sequence in each child process;
reworked fetching random bits from entropy pool and deriving mail id from secret id (after re-reading RFC 4086); much less of the private entropy accumulator is now exposed to observers; added a new function fetch_entropy_bytes(), dropped a function fetch_entropy();
a macro %S no longer corresponds to sender_contact, which was a relict from times of early viruses; for compatibility with existing templates it is now equivalent to %s, but should no longer be used and might be retired or re-purposed with the next version; default notification templates were adjusted accordingly - please adjust your customized templates if using them;
drop dependency on Digest::SHA1;
README.chroot: document that sa-update needs to update rules in the jail and refresh the text somewhat; thanks to Francois Rolland;
CLEANING
retired often misused settings $warnvirussender and $warnspamsender (but kept marginally useful $warnbannedsender, $warnbadhsender, and their parent %warnsender_by_ccat). To bounce or reject viruses and spam use D_REJECT and D_BOUNCE settings for corresponding $final_*_destiny. It is no longer supported to both deliver (D_PASS) a virus or spam message while also sending a notification to sender. Both retired variables are still declared for compatibility with old config files, but their value is ignored. An attempt to set their value to a non-default value produces a warning.
retired a setting $syslog_priority, it was not particularly useful since the introduction of dynamic syslog priorities with amavisd-new-2.0 . The new behaviour is equivalent to a previous $syslog_priority='debug'; The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning.
retired a setting $SYSLOG_LEVEL, it was obsoleted by amavisd-new-2.4.0; please use the setting $syslog_facility instead, defaulting to $syslog_facility='mail' . The variable $SYSLOG_LEVEL is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning.
renamed $DO_SYSLOG to $do_syslog, and $LOGFILE to $logfile; old names are kept as aliases for compatibility;
retired a setting $relayhost_is_client, it became obsolete with amavisd-new-2.0. Please use a '' in place of a host IP address and port number when amavisd should pass a checked mail message back to the same host from which the request came, e.g.: $forward_method = 'smtp::*'; The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning.
retired a setting $sa_timeout, the variable is still declared for backward compatibility, but has no effect. Instead, the time available for spam scanning is automatically determined from $child_timeout, taking into consideration the actual time left till the deadline; An attempt to set its value to a non-default value produces a warning.
retired a setting $sa_spam_report_header, it was obsoleted in amavisd-new-2.4.3 with the introduction of %allowed_added_header_fields. To enable insertion of X-Spam-Report header field, please use instead: $allowed_added_header_fields{lc('X-Spam-Report')} = 1; The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning.
retired settings $sa_spam_modifies_subj and @spam_modifies_subj_maps. Disabling insertion of spam tag into a Subject header field can be achieved by turning off the corresponding entries in %subject_tag_maps_by_ccat: undef $subject_tag_maps_by_ccat{CC_SPAM()}; undef $subject_tag_maps_by_ccat{CC_SPAMMY.',1'}; undef $subject_tag_maps_by_ccat{CC_SPAMMY()}; undef $subject_tag_maps_by_ccat{CC_CLEAN.',1'}; or by emptying corresponding lists of lookup tables, e.g.: @spam_subject_tag_maps = (); @spam_subject_tag2_maps = (); @spam_subject_tag3_maps = (); or individually (by-recipient) by specifying suitable lookup tables in @spam_subject_tag_maps / @spam_subject_tag2_maps / @spam_subject_tag3_maps, either statically, or through SQL or LDAP lookups; Both settings are still declared for compatibility with old config files, but their value is ignored. An attempt to set the value of a variable $sa_spam_modifies_subj to a non-default value produces a warning.
retired a setting $insert_received_line, it was obsoleted in amavisd-new-2.4.3 with the introduction of %allowed_added_header_fields. To disable insertion of a Received header field, please use instead: $allowed_added_header_fields{lc('Received')} = 0; The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning.
retired a setting $notify_xmailer_header, the X-Mailer header field is not inserted into notifications, as was a default. The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning.
retired a setting $sa_auto_whitelist, it became obsolete with amavisd-new-2.1.0 and SpamAssassin 3.0.0 (released in 2004) by a 'use_auto_whitelist 1' option in local.cf . The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning.
retired a deprecated macro 'x-mailer', use macros 'header_field' or 'useragent' instead;
removed a constant CC_TEMPFAIL, it was retired with amavisd-new-2.5.0;
renamed a configuration variable $sql_partition_tag to $partition_tag in order to reflect its more general usage outside of SQL; the old name $sql_partition_tag is retained for compatibility and is an alias for $partition_tag;
dropped compatibility of $final_*_destiny settings with old numerical values (used by versions of amavisd older than amavisd-new-20030314);
internal: retire Amavis::In::Message::client_addr_mynets, no longer in use;
internal: retire Amavis::In::Message::PerRecip::infected, no longer in use;
internal: drop a redundant argument $conn from the following subroutines:
make_received_header_field, check_header_validity, defanged_mime_entity, msg_from_quarantine, check_amcl_policy, postfix_policy, add_forwarding_header_edits_common, add_forwarding_header_edits_per_recip, prepare_modified_mail, do_notify_and_quarantine, do_quarantine, save_info_preliminary, save_info_final, mail_dispatch, dispatch_from_quarantine, virus_scan, spam_scan, white_black_list, Amavis::SpamControl::{ExtProg,SpamdClient,SpamAssassin}::check
As a compatibility measure, the do_quarantine() may still be called with or without the first argument $conn, its value is now ignored if present.
Although the $conn argument is also redundant in calls to custom hooks (as this information is available through $msginfo->conn_obj), these calls are left unchanged for compatibility with existing custom hooks.
May 18, 2011
amavisd-new-2.6.6 release notes
This version is strictly a maintenance release, it incorporates bug fixes backported from 2.7.0-pre* series and/or posted as patches to the mailing list.
BUG FIXES
amavisd-release was not sending a 'mail_file' attribute when a quarantined message was a non-compressed file in a single-level directory quarantine (not SQL-based), causing a release failure; reported by Jarno Huuskonen;
quarantining to SQL was sporadically failing, reporting some unrelated random error (like 'not available' or 'OpenSSL error: header too long'); reported by Tonio;
avoid a warning "_WARN: Use of uninitialized value in string eq at ... line 275." when an SQL-based white/black-listing is used; reported by Tonio;
wrap the sql clause SET NAMES 'utf8' so that only a warning at a log level 2 is issued if an SQL server does not understand the command (SQLite, old versions of MySQL) instead of aborting; reported by Roland Holzner;
April 7, 2011
amavisd-new-2.6.5 release notes
This version is strictly a maintenance release, it incorporates bug fixes backported from 2.7.0-pre* series and/or posted as patches to the mailing list.
BUG FIXES
when a back-end MTA rejected a message, amavisd would send a non-delivery status notification, but also propagate the reject status back, which is wrong, only one or the other response would be appropriate. A fix also allows choosing either a D_REJECT, D_BOUNCE or D_DISCARD response for such a case, configurable through %final_destiny_by_ccat at a CC_MTA entry, defaulting to D_REJECT; reported by Peer Heinlein;
checking header section syntax could take excessive amounts of time in some degenerate cases of a very long header section, now fixed;
do not bypass spam checking of a bounce message when its referenced domain in Message-ID is non-local but pen pals are disabled; reported by Stefan;
removed some of the guesswork in bounce killer to prevent false positives in certain cases of forwarding a mail message as an attachment, at the expense of passing through some undesired but nonstandard bounces; (also, deal with non-delivery notifications from yahoogroups.com, and fixed one particular case of a false-positive in bounce killer (mixed/multipart with an attached full message, sent through a mailing list);
fixed a 'Zoo archive' entry in the $map_full_type_to_short_type_re list;
fixed a test for $myhostname being a FQDN to allow IDN domains (with a dash);
fixed a REPLACE hack (feature introduced in 2.6.2) on loading a policy bank;
fixed choosing the module IO::Socket::INET in ask_daemon_internal() to avoid versions of IO::Socket::INET6 older than 2.55 (2.56?) failing with "Address family not supported by protocol family" when an IPv4 address with a port number is specified for connections to a virus scanner; based on a patch by Phil Pearl (Lobbes);
do_unzip: avoid testing a version of Compress::Raw::Zlib, the module may not be loaded at all and the test would fail, resulting in inoperative zip unpacking; reported by Tuomo Soini;
when logging or quarantining to SQL, execute a clause: SET NAMES 'utf8' after connecting to a database to ensure the decoded Subject and From header fields are correctly interpreted by an SQL server as UTF-8 encoded strings. It seems the module DBD::mysql does not observe a MySQL setting for 'character_set_client' and needs an explicit SET NAMES. The problem did not affect PostgreSQL. Reported by Zhang Huangbin;
avoid LDAP lookups aborting the scan when a %d placeholder is used in a $default_ldap{base} setting and the resulting base does not exist in an LDAP schema; reported by Zhang Huangbin;
the amavisd-new 2.6.3 relaxed semantics of a number of hard links on a directory in TempDir::prepare(_dir), but left out an equivalent change necessary in TempDir::check, which is now fixed; the change only affects certain file system (like the one used on Mac OS X);
treat an empty PID file or a junk one-liner file the same as a nonexistent PID file; previously an empty PID file (e.g. after an unclean shutdown) would prevent amavisd from starting; problem reported by Michael Scheidell;
changed amavisd-release to only provide a 'quar_type' attribute in its request when it is reasonably sure of its appropriate value, otherwise leave the decision to the amavisd daemon; this solves releasing from a file-based quarantine when compression is not used and all files are at the top directory; reported by Voytek Eymont;
provide a workaround for a [perl #62048] bug affecting versions of perl older than (approx) 5.12.3, when a banning check if using rules in $banned_namepath_re and a lookup_re() could abort with an: Unwarranted "Malformed UTF-8 character" on certain tainted mail part names (with a valid UTF-8 representation); reported by Jakob Curdes;
provide a workaround for logging to syslog using an old version of Unix::Syslog which didn't prepare and keep its own copy of the 'ident' argument on a call to openlog(3); thanks to Bill Landry;
OTHER
ensure compatibility with a new version 5.500 of MIME-Tools, which changed the way mime attributes content-disposition.filename and content-type.name are decoded, now properly respecting their declared encodings (character set). As a result, the declared (recommended) file names of MIME parts are now represented as native Perl character strings (Unicode), and as such may also end up in reported names of banned parts. Regular expressions in @banned_filename_maps, $banned_filename_re and $banned_namepath_re may also see these strings as native Perl characters, along with their MIME-encoded form. The change also affects interpretation of names with earlier versions of MIME-Tools, making them behave more like the 5.500.
amavisd.conf: exclude names starting with 'cid:' from matching the double extensions banning rule, avoiding false positives;
a small update to a default @virus_name_to_spam_score_maps;
the 'originating' flag is now passed on to SpamAssassin through its %suppl_attrib argument - potentially useful with current trunk version of SpamAssassin (treats originating mail submission as a MSA submission), and ignored by older versions;
some documentation updates; the RELEASE_NOTES file is now encoded as UTF-8, instead of ISO-8859-1;
Please log in to add feedback.
This update has been submitted for testing by robert.
This update is currently being pushed to the Fedora EPEL 6 testing updates repository.
This update has been pushed to testing
This update has reached 14 days in testing and can be pushed to stable now if the maintainer wishes
This update has been submitted for stable by robert.
This update is currently being pushed to the Fedora EPEL 6 stable updates repository.
This update has been pushed to stable