Note for Users Upgrading to SpamAssassin 4.0.3 ---------------------------------------------- - SpamAssassin now requires HTML::Parser 3.72 or later. - Mail::SpamAssassin::Plugin::DecodeShortURLs has been merged into Mail::SpamAssassin::Plugin::Redirectors and is now a deprecated compatibility shim; all url_shortener*/short_url* settings and rules still work as aliases of url_redirector*/redir_url*. Redirect chains mixing shorteners and redirectors in any order are now followed correctly, and query strings are no longer stripped before fetching. - All Mail::SpamAssassin::Plugin::Redirectors settings, including the url_shortener* aliases, are now administrator settings, which spamd does not accept in users' preferences (user_prefs files, SQL or LDAP). In 4.0.2, users could set url_redirector, url_redirector_get, clear_url_redirector, url_redirector_params, url_shortener, url_shortener_get, clear_url_shortener and url_shortener_custom_user_agent, and the url_redirector_cache_* and url_shortener_cache_* settings with allow_user_rules. Move any of these from users' preferences to the site configuration. - The Mail::SpamAssassin::Plugin::OLEVBMacro limits olemacro_max_file, olemacro_num_mime, olemacro_num_zip and olemacro_zip_depth are now administrator settings, which spamd does not accept in users' preferences. Move any of these from users' preferences to the site configuration. - Mail::SpamAssassin::Plugin::OLEVBMacro no longer uses IO::String, and needs Archive::Zip 1.34 or later to check zip files and Office Open XML documents. - dns_available, dns_server, dns_options, dns_query_restriction, lock_method, bayes_sql_override_username, user_awl_sql_override_username, spamcop_relayhost, asn_lookup and asn_lookup_ipv6 (which needed allow_user_rules), are now administrator settings, which spamd does not accept in users' preferences (user_prefs files, SQL or LDAP, @GLOBAL rows included). Nor does it accept freemail_domains, freemail_welcomelist, hashbl_acl_* or bayes_stopword_* there. Move any of these from users' preferences to the site configuration. Setting bayes_sql_override_username or user_awl_sql_override_username in users' preferences to group users' SQL data no longer works. - spamd --ssl now needs IO::Socket::SSL 1.994 or later. - Mail::SpamAssassin::Plugin::ExtractText now has the possibility to cache results in order to speedup attachment processing. - Hashbl can now query FuzzyHash rbls using check_hashbl_bodyfuzzy() in order to detect emails similar to known spam messages. - TxRep now supports the txrep_skip_domain option in order to completely skip TxRep processing for some domains. The domain name is checked on From and EnvelopeFrom fields as well as on EHLO value. - SQL user preferences now read the @GLOBAL rows before the user's own whatever the database's collation. With PostgreSQL and a language collation such as en_US.UTF-8, users whose names sorted before "GLOBAL" got the @GLOBAL value of a setting instead of their own. A custom user_scores_sql_custom_query ending in ORDER BY username has the same problem; see its documentation for examples. - ARC (Authenticated Received Chain) verification has been separated from the DKIM plugin into its own plugin: Mail::SpamAssassin::Plugin::ARC. It is loaded by default from v403.pre. To use ARC authentication results as a DMARC fallback, you must configure at least one trusted ARC sealer domain: arc_trusted_sealers google.com microsoft.com yahoo.com Without this setting, ARC signatures are verified but ARC-Authentication-Results headers are not parsed. - Mail::SpamAssassin::Plugin::ARC re-uses ARC results from the AuthRes plugin if available, avoiding redundant cryptographic verification with Mail::DKIM. If no AuthRes results are available, the plugin falls back to its own Mail::DKIM::ARC::Verifier-based verification as before. - Mail::SpamAssassin::Plugin::SPF no longer parses Authentication-Results headers directly. It now uses results from Mail::SpamAssassin::Plugin::AuthRes if available, falling back to Received-SPF headers, then DNS lookups as a last resort. Users that previously relied on the SPF plugin to parse Authentication-Results headers will need to enable the AuthRes plugin (v401.pre). - Mail::SpamAssassin::Plugin::DKIM now re-uses DKIM results from the AuthRes plugin if available, avoiding redundant signature verification. Caller-supplied signature objects (via suppl_attrib) still take precedence, and the plugin falls back to its own Mail::DKIM-based verification when no AuthRes results are available. - Mail::SpamAssassin::Plugin::DMARC now uses DMARC results from the AuthRes plugin if available, avoiding redundant validation with Mail::DMARC::PurePerl. Both the result and the published domain policy must be present in the Authentication-Results header for this to apply. If no AuthRes results are available or the policy is missing, the plugin falls back to its own Mail::DMARC::PurePerl-based validation as before. - Mail::SpamAssassin::Plugin::DMARC now makes its DNS lookups through SpamAssassin's resolver, so they go to the dns_server servers like SpamAssassin's other lookups. Before, Mail::DMARC used the system's name servers even when dns_server was set. The new dmarc_timeout setting (default 10 seconds) limits how long the DMARC check can take, as spf_timeout and dkim_timeout do for SPF and DKIM. - New configuration option: multipart_alternative_preferred_part When set to "text/html" or "text/plain", only the preferred part type from multipart/alternative sections will be used for rendered body text, skipping the other alternative. By default this is unset and all parts are included as before. - New configuration option: scan_text_attachments By default, text/plain and text/html parts marked with "Content-Disposition: attachment" are excluded from the rendered body text that "body" rules run against. When this option is enabled, such attachments are rendered and included like any other part so body rules can inspect them. Default is 0 (off) to match existing behaviour. - New Mail::SpamAssassin::Plugin::NeuralNetwork This plugin checks messages using Fast Artificial Neural Network library. - Mail::SpamAssassin::Plugin::AskDNS now supports a "noskip" tflag. By default, any host or domain matching uridnsbl_skip_domains is ignored by askdns rules. The "noskip" tflag bypasses this check for specific rules. - Mail::SpamAssassin::Plugin::Redirectors can connect to a Selenium server in order to catch more redirectors that uses Javascript or other tricks. See the plugin's SECURITY CONSIDERATIONS section before enabling it: a real JavaScript-executing browser is pointed at URLs taken from untrusted mail, so the Selenium host should be isolated and its outbound network access restricted. - New MIME-part handler framework. A handler registers one of its methods as the handler for a content-type pattern -- an exact type such as "image/jpeg" or a major-type glob such as "image/*". During message processing each matching MIME part is dispatched to its handler, which can inject extracted text into the part and return synthetic child parts that are dispatched recursively (for example a nested archive or an embedded image). Dispatch is bounded by the new handler_max_depth, handler_max_parts, handler_max_bytes, and handler_time_limit options to guard against deeply nested or oversized content. Handlers are loaded with the new loadhandler and tryhandler directives, and a block of configuration can be made conditional on a handler with ifhandler. A handler is a subclass of the new Mail::SpamAssassin::Handler class (itself a subclass of Mail::SpamAssassin::Plugin); see its documentation for the API. Seven handlers ship with this release. HTML, Script, SVG and ICS are loaded by default from v403.pre; Image, PDF and Archive are included commented-out because they depend on external tools or non-core Perl modules (see below). * HTML (text/html) renders each HTML part and surfaces content that has no MIME part of its own -- inline data: images and embedded script -- as child parts for the image and Script handlers. Body script is inert and only surfaced when the HTML part is itself an attachment. IMPORTANT: this handler is now the only thing that renders HTML. If it is not loaded, text/html parts are not parsed at all: body rules and Bayes see no text from them, the html_* eval rules have no results to test, and URIs in HTML links (href, src, action, etc.) are never extracted -- so a mostly-HTML mail stream will score far lower than it did before and spam will be missed. It is loaded by default from v403.pre, so a stock install is unaffected. Check for it if you upgrade over a v403.pre you have edited, or keep an older copy of the file: loadhandler Mail::SpamAssassin::Handler::HTML SpamAssassin warns once per process when a message contains a text/html part and no handler is registered for it. * Script (text/javascript, text/vbscript) collects JavaScript and VBScript text for the new "script" rule type (like rawbody rules) and adds navigation/redirect URLs (e.g. window.location = '...') to the URI detail list with type "script". Provides the check_script_contains_recip_addr eval. * SVG (image/svg+xml) parses SVG attachments, exposing their text to the new "svgtext" rule type (kept separate from HTML body text), adding links to the URI detail list (type "svg"), and surfacing embedded script for the Script handler. Provides the check_svg_text_ratio eval, which flags a mostly-text "image" by the ratio of text words to graphics elements. * ICS (text/calendar) parses iCalendar invites and renders each event's SUMMARY and DESCRIPTION into the body so ordinary body rules can match it (SpamAssassin otherwise skips text/calendar); the same text is also exposed to the new "icstext" rule type, kept separate from the body. Links in the URL, ATTACH and LOCATION properties are added to the URI detail list (type "ics"). Provides the check_ics_attendee_count, check_ics_random_start_time and check_ics_event_prop evals, the last of which matches a regex against any event property (parameters included) for hunting arbitrary or extension properties. * Image (image/*) runs the tesseract OCR engine on image parts and injects the recognised text into the body so ordinary body rules can match it. Provides the new "imagetext" rule type and the check_image_text_ratio eval, which flags a message whose body text is mostly OCR'd from images. Requires the tesseract executable (see image_tesseract_path). OCR of HEIF/HEIC images additionally requires the heif-dec executable from libheif -- named heif-convert before libheif 1.18, and both names are accepted (see image_heif_dec_path); without it, HEIF images are skipped and all other image types are still OCR'd. Fedora's libheif decodes HEIC only with libheif-freeworld from RPM Fusion installed. Both tesseract and libheif pull in substantial recursive dependencies, which is why this handler is not enabled by default. * PDF (application/pdf) parses each PDF with the new pure-Perl Mail::SpamAssassin::PDF::Parser and exposes metadata (page/image/ link counts, ratios, encryption, JavaScript/OpenAction flags, etc.) via pdf2_* eval rules and _PDF2*_ tags; URLs are added to the URI detail list (type "pdf"), and embedded images can be extracted and passed to the image handler for OCR. The pdf2_ prefix avoids conflicting with the PDFInfo plugin. Decrypting encrypted PDFs needs Crypt::RC4 and Crypt::Mode::CBC. * Archive (application/zip, application/vnd.rar) opens zip and rar attachments and returns the files inside as child parts, which the framework re-dispatches to the other handlers by type. Provides the check_archive_file_count eval. Zip uses the Archive::Zip module (1.34 or later); rar shells out to RARLAB's unrar or unrar-free 0.2.0 or later (see archive_unrar_path). Note for Users Upgrading to SpamAssassin 4.0.2 ---------------------------------------------- - TxRep data when using a SQL backend might be wrong because of a bug in SpamAssassin 4.0.1, TxRep score can be limited with new txrep_min_score and txrep_max_score configuration options or you can remove email entries from TxRep database. - Redis replication is now supported, Redis connection parameters are splitted between read_only and read_write. Old syntax is still supported. - Internal Mail::SpamAssassin::GeoDB module now also supports IP::Geolocation::MMDB Perl module. - New Mail::SpamAssassin::Plugin::Redirectors plugin This plugin looks for URLs redirected by a list of URL redirector services. Upon finding a matching URL, plugin will send a HTTP request to the redirector service and retrieve the Location-header which points to the actual redirected URL. It then adds this URL to the list of URIs extracted by SpamAssassin which can then be accessed by uri rules and plugins such as URIDNSBL. - HashBL and Freemail plugins no longer ignores support@, sales@, info@, helpdesk@, contact@ and kontakt@ email addresses by default. - Phishstats.info domain is back online; "phishing_phishstats_feed" and "phishing_phishstats_minscore" options have been restored on Mail::SpamAssassin::Plugin::Phishing plugin. Note for Users Upgrading to SpamAssassin 4.0.1 ---------------------------------------------- - Phishstats.info domain has expired; "phishing_phishstats_feed" and "phishing_phishstats_minscore" options have been removed from Mail::SpamAssassin::Plugin::Phishing plugin. Note for Users Upgrading to SpamAssassin 4.0.0 ---------------------------------------------- Apache SpamAssassin 4.0.0 represents years of work by the project with numerous improvements, new rule types, and internal native handling of messages in international languages. We highly recommend looking through this file and all of the .pre files to evaluate your configuration thoroughly. Plugins have been added, removed, and improved throughout. - All rules, functions, command line options and modules that contain "whitelist" or "blacklist" have been renamed to contain more racially neutral "welcomelist" and "blocklist" terms. This allows acronyms like WL and BL to remain the same. Previous options will continue work at least until version 4.1.0 is released. If you have local settings including scores or meta rules referring to old rule names, these should be changed and "enable_compat welcomelist_blocklist" added in init.pre. See: https://wiki.apache.org/spamassassin/WelcomelistBlocklist (Bug 7826) - Meta rules no longer use priority values, they are evaluated dynamically when the rules they depend on are finished. (Bug 7735) - API: New $pms->rule_ready() function. Any asynchronous eval-function must now return undef (instead of 0 or 1), if rule result is not ready when exiting the function. $pms->rule_ready($rulename) or $pms->got_hit(...) must be called when the result has arrived. If these are not used, it can break depending meta rule evaluation. - Setting normalize_charset is now enabled by default. Note that rules should not expect specific non-UTF8 or UTF8 encoding in body. Matching is done against the raw data which may vary depending on normalize_charset setting and whether decoding to UTF8 was successful. See: https://wiki.apache.org/spamassassin/WritingRulesAdvanced - DKIM plugin has added support for ARC signature verification - The DecodeShortURL plugin has been added and decodes URIs from URL shorteners that may be used to evade scanning - Strings can now be captured from rules and later reused using the special %{TAGNAME} syntax - The Bayes stopwords, or noise words, are now configurable in order to optimize Bayes usage for non-English languages. Stopwords for 16 foreign languages have been included. See 60_bayes_stopwords.cf in the rules files. See Mail::SpamAssassin::Plugin::Bayes and the bayes_stopword_languages option if you wish to use a different stopword list. This is highly recommended if you are using Bayes and you are processing messages in languages other than English. - The OLEVBMacro plugin has been improved to identify more macros while also extracting uris from the attachments for automatic inclusion in RBL lookups - Internationalized domain name (IDN) support has been added and requires Net::LibIDN2 or Net::LibIDN module with a new Util::idn_to_ascii() function. (Bug 7215) - Improved internal header address (From/To/Cc) parser, now also handles multiple addresses and includes optional support for external Email::Address::XS parser, which can handle nested comments and other oddities. - Header :addr :name modifiers now return all addresses. Options of :first :last select only first (topmost) or last header to process when there are multiple headers with the same name. :addr and :name may still return multiple values from a single header. - API: $pms->get() can and should now be called in list context. Scalar context continues to return multiple values newline separated, but this should be considered deprecated. - New ExtractText plugin that extracts text from documents or images to feed the data into SpamAssassin for standard processing with existing rules, URIs extracted from documents will fall into normal RBL lookups. - New "nolog" tflag added to hide info coming from rules in SpamAssassin reports - All log output (stderr, file, syslog) is now escaped properly for \r \n \t \\, control chars, DEL, and UTF-8 sequences presented as \x{XX}. Whitespace is not normalized anymore like in versions prior to 4.0.0. - API: Logger::add() has new optional 'escape' parameter. New Logger::escape_str() function. - API: New $pms->add_uri_detail_list() function. Also new uri_detail_list types: unlinked, schemeless - Util::split_domain, trim_domain, and is_domain_valid functions have a new optional argument ($is_ascii) - Header names support new :host :domain :ip :revip modifiers - AskDNS: tag HEADER(hdrname) supported to query any header content similarly to header rules - The HashCash module and support has been removed completely, as it has been long since deprecated - URILocalBL: uri_block_cc/uri_block_cont now support negation (Bug 7528) - URILocalBL: IPv6 lookups for hosts is now support, if provided by your database - DNS and other asynchronous lookups such as Pyzor and DCC are now only launched when priority -100 is reached. This allows short circuiting at a lower priority without sending unneeded DNS queries and starting process forms. (Bug 5930) - API: New plugin method callback method check_dnsbl added to launch network lookups at priority -100 and check_post_dnsbl to harvest own network lookups - API: New plugin callback method check_cleanup for cleaning up things... - FreeMail: new options freemail_import_welcomelist_auth and freemail_import_def_welcomelist_auth added (Bug 6451) - New internal Mail::SpamAssassin::GeoDB module that provides a unified interface to modules MaxMind::DB::Reader (GeoIP2), Geo::IP, IP::Country::DB_File, and IP::Country::Fast. This is utilized by RelayCountry and URILocalBL with settings geodb_module, geodb_options, and geodb_search_path. Deprecated settings still work such as country_db_type, country_db_path, uri_country_db_path, and uri_country_db_isp_path but will print a warning to migrate to geodb_module/options. - Razor2 razor_fork option added to create separate Razor2 processes and read in the results later asynchronously, increasing throughput, and automatically adjusting rule priorities to -100. - DCC checks are now done asynchronously if using dccifd, improving throughput. With dccifd, rule priorities are automatically adjusted to -100. Commercial reputation rules can be ignored with the option "use_dcc_rep 0" to save a few CPU cycles. - Pyzor pyzor_fork option added to create separate Pyzor processes and read in the results later asynchronously, increasing throughput, and automatically adjusting rule priorities to -100. Renamed pyzor_max setting to pyzor_count_min. Added pyzor_welcomelist_min and pyzor_welcomelist_factor setting. Also try to improve false positives by ignoring "empty body" messages. - API: deprecated $pms->register_async_rule_start() and $pms->register_async_rule_finish() calls though left in for backwards compatibility. Plugins should only use $pms->bgsend_and_start_lookup(), which handles required things Automatically. Direct calls to bgsend or start_lookup should not be used. $pms->bgsend_and_start_lookup() should always contain $ent->{rulename} for correct meta dependency handling. Deprecated start_lookup, get_lookup, lookup_ns, harvest_until_rule_completes, and is_rule_complete. - SPF: Mail::SPF is now the only supported perl module and Mail::SPF::Query is deprecated along with the settings do_not_use_mail_spf, and do_not_use_mail_spf_query. SPF lookups are not done asynchronously so using an MTA filter such as pypolicyd-spf or spf-engine can generate Received-SPF for SpamAssassin to parse. - "ALL" pseudo-header now returns decoded headers, so it's usage is consistent with single header matching. Using the :raw option mimics the previous behavior of with undecoded and folded headers. - New dns_block_rule option handles blocked DNSBLs (Bug 6728) - ASN: Support GeoDB for ASN lookups (asn_use_geodb, asn_prefer_geodb, asn_use_dns). - ASN: Default sa-update ruleset doesn't make ASN lookups or add headers anymore. Configure desired methods, asn_use_geodb or asn_use_dns, and add_header clauses manually as described in the plugin documentation. Usage of asn_use_geodb without DNS is recommended unless ASNCIDR is needed. Do not use rules that check metadata X-ASN header! Only the new eval function check_asn() described in plugin manual works reliably. - sa-update: New --score-multiplier, --score-limit, and --forcemirror options added. #1 forcemirror: forces sa-update to use a specific mirror server, #2 score-multiplier: adjust all scores from update channel by a given multiplier to quickly level set scores to match your preferred threshold #3 score-limit adjusts all scores from update channel over a specified limit to a new limit - New dns_options "nov4" and "nov6" added. IMPORTANT:; You must set nov6 if your DNS resolver is filtering IPv6 AAAA replies. - API: Added Message::get_pristine_body_digest(), Message::get_msgid(), and Message::generate_msgid() functions. Removed deprecated private Plugin::Bayes::get_msgid() function. - Bayes and TxRep seen Message-ID tracking hashing method changed. No actions are required. If re-learning some old messages, they might be learned twice but old IDs should expire automatically. - report_charset defaults now to UTF-8. - Meta rules inherit net tflag setting from dependencies (Bug 7735) - BodyEval: Added plaintext_body_sig_ratio eval rules for the first text/plain MIME part's body and signature length ratio. - API: Now supports multiple calls of $pms->test_log() for rules. Added $pms->check_cleanup() to finalize tags, reports, etc. Deprecated internal $pms->{test_log_msgs}, renamed to $pms->{test_logs}. Deprecated $pms->clear_test_state() as it is not needed anymore. $pms->test_log() now accepts $rulename as second argument. - URIDNSBL: urirhsbl/urirhssub rules support "notrim" tflag to force querying the full hostname instead of just the domain. This works best if the specific uribl supports this mode. (Bug 7835) - Removed deprecated --auth-ident and --ident-timeout options from spamd - MIMEHeader: support matching ALL header, tflags range, and tflags concat - Autolearn: add new tflags autolearn_header/autolearn_body. These can force a rule to count as header or body points accordingly. (Bug 7907) - SSL client certificate support for spamc/spamd is now easier. New spamc options --ssl-cert, --ssl-key, --ssl-ca-file, and --ssl-ca-path. New spamd options --ssl-verify, --ssl-ca-file, and --ssl-ca-path (Bug 7267) - ArchiveIterator now automatically uncompressed all gzip, bzip2, xz, lz4, lzip, and lzo-compressed files (Bug 7598). These apply to spamassassin and sa-learn commands also. - New DMARC policy check plugin. - New project maintained DecodeShortURLs plugin which may not be directly compatible with rules from other third party plugins. See The plugin documentation for configuration and rule format. - Installing module Net::CIDR::Lite allows the use of dash-separated IP range format (e.g. 192.168.1.1-192.168.255.255) for NetSet tables including internal_networks, trusted_networks, msa_networks, and uri_local_cidr. - The HashBL plugin in v342.pre is now enabled by default. - HeaderEval check_for_unique_subject_id() function is deprecated. (end of UPGRADE)