Mailcorral Documentation
0. Preface
Eric Wilde,
ewilde@bsmdevelopment.com
V2.2.0, 2020 Mar 20
This document shows how to install the sendmail filter, MailCorral, into
sendmail as a milter. It describes how to configure sendmail, compile the
filter and set it up to run attended. Information about the operation of
the filter is also provided.
0.1 Copyright
Copyright (c) 2002, 2003, 2004, 2009, 2012, 2013, 2020 Eric Wilde
Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version 1.1
or any later version published by the Free Software Foundation; with no
Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts.
A copy of the license is included in Appendix A, entitled
"GNU Free Documentation License".
0.2 Distribution
This document is also available in
PDF format at:
MailCorral Documentation
0.3 Contributions
BSM Development is a small company of software
developers (perhaps, like yourself) and we would appreciate, if you so
choose, a small contribution to pay for this software. We hope you perceive
it to be of value and find it useful to you. As a suggestion, we would like
to think that it is worth at least US $100.00. However, you may send
whatever amount that you feel this product is worth. If you'd like to make
a contribution, you can do so directly via the
contributions page
of the BSM Development site (www.bsmdevelopment.com/Products/Contribute.html).
Alternately, you can send a check via snail mail to:
BSM Development
44 Whitewood Circle
Norwood, MA 02062
Please make your check payable to "BSM Development", thanks.
0.4 Change Log
Here are the version numbers, dates and a brief description of
each program change, in reverse chronological order:
2.2.0
- 2020 Mar 19 - Add options to distrust users (with respect to certain
local options) and reject entities with warnings.
- 2020 Mar 18 - Do a better job of classifying inline MIME types and
produce more informative errors and warnings.
- 2020 Mar 17 - Add additional well-known file and MIME types.
2.1.2
- 2013 Jul 21 - Fix possible "sed: command not found" error in configure
script on some systems.
- 2013 Jan 27 - Add the ability to process the SpamAssassin updates tree,
to find versioned whitelists.
- 2013 Jan 25 - Fix compile warnings for the free function in configuration
processing. Fix reentrancy problem with user configuration files.
2.1.1
- 2012 Jun 8 - Fix compile warnings for string/stdio functions in
expression evaluator and other modules. Add image/png MIME type and .png
extension to the list of acceptable mail attachements. Fix bogus definition
of REMAILHDRBYPASS. Fix longstanding problem with mail from local addresses
being erroneously filtered when the CheckExternal flag is not set.
2.1.0
- 2009 Nov 21 - Add .wmv extension to the file types list as an OK file.
- 2009 Nov 15 - Fix problem in with WeakRequire when module not found in
MailRelease and SpamNotify.
2.0.1
- 2009 Aug 7 - Fix quote handling by NormalizeName.
- 2009 Aug 3 - Fix problem writing empty message body to spam file when
spam messages also contain viruses and the virus scanner deletes the entire
message.
2.0.0
- 2009 Feb 4 - Add encoding and decoding of uuencoded entities.
- 2009 Jan 19 - Add support for ClamAV via clamd.
- 2009 Jan 16 - Add support for the spamd 1.4 protocol and the HEADERS
command.
- 2009 Jan 8 - Add archiving of messages to disk or to a third party
archiver via email forwarding. Add bounceback suppression for bounced messages
from non-functioning third party archivers. Use header from name for
blacklist and whitelist checks, along with envelope from name. Have name
normalization strip "<>" and "()" from addresses.
- 2004 Apr 19 - Add support for evaluating results from user supplied
arbitrons, plus internal arbitrons, via logical expressions.
- 2004 Jan 15 - Allow users to programatically supply arbitrons.
1.3.0
- 2007 Aug 13 - Fix problem with multiply defined bool under sendmail
8.14.1.
1.2.0
- 2003 Aug 21 - Fix problem with HTML tag parameters that are a single
quote crashing filter (filter goes defunct).
- 2003 Jun 13 - Try "htdocs" as well as "html" for the template file
location in ConfigEdit.cgi.
- 2003 Jun 8 - Add exit message if errors found in command line or options.
- 2003 May 29 - Treat "text" as "text/plain". Fix crash when internal
spam arbitron is used.
- 2003 May 15 - Allow all_spam_to for non-local users.
- 2003 May 12 - Fix problem with messages containing empty text entities.
- 2003 May 5 - Remove HTML tags from names to release in MailRelease.
- 2003 May 2 - Tweak MailRelease error responses and add a couple more
messages. Fix apparent Perl bug in MailRelease whereby push of lc($1)
pushes garbage. This affected domain lookup.
1.1.3
- 2003 May 1 - Conditionally load Zlib and DB_File, for real this time,
in MailRelease and SpamNotify.
- 2003 Apr 26 - Have SpamNotify unzip corralled messages, if
necessary (just in case someone zips all the spam in the corral, before we
have a chance to send out the notifications). Fix problem with '>' and
'<' in the HTML component of MailRelease notifications.
1.1.2
- 2003 Apr 24 - Add an option to keep all messages in the corral,
regardless of whether they were altered or not. Add the ability to zip
older corralled messages, if desired.
- 2003 Apr 17 - Include the latest SpamNotify in the distro.
- 2003 Apr 14 - Add global ignore options.
- 2003 Apr 13 - Allow source RPM to build on any i386 version of RedHat.
Template redhat.mc is missing on RH 8.x. The RPM now picks something else.
- 2003 Apr 11 - Update filterclean to get rid of empty partition
directories.
- 2003 Apr 10 - Fix compiler windge about ctime pointer.
1.1.1
- 2003 Apr 8 - Don't reencode second HTML entity if not changed. Allow
buffer space for badly formatted Quoted-printable entities.
- 2003 Apr 3 - Use tag delimiters to end mismatched quotes.
- 2003 Apr 2 - Handle mangling of short MIME types properly. Ignore
"Content-type: text" in headers.
- 2003 Mar 31 - Fix problem with addresses containing domain names before
the '@'.
- 2003 Mar 27 - Check messages from everybody, if paranoid.
1.1.0
- 2003 Mar 22 - Fixed a problem in HTML tag scanning when unmatched quotes
are encountered.
- 2003 Mar 19 - Rebuild messages with attachments only to allow virus and
spam warning text to be inserted, permit deletion of the attachment, etc.
- 2003 Mar 17 - Handle Apple resource forks better.
- 2003 Mar 16 - Allow attachments containing viruses to be deleted.
- 2003 Mar 12 - Partition spam as well.
- 2003 Mar 11 - Properly handle MIME type messages that have empty bodies.
- 2003 Mar 10 - Fix improper use of HAVE_DBM ifdef.
1.0.18
- 2003 Mar 9 - Implement ommitted cases in non-delivery test. Test for
Base64 encoded text/HTML MIME entities in internal, statistical arbitron.
Add text/plain and text/html as special cases for "UnknownDisposition MIME".
Remove junk after plussed user names.
- 2003 Mar 8 - Allow user to specify timeout for arbitron connection.
- 2003 Mar 7 - Use VerifyUser instead of getpwnam to look up home
directories, thereby avoiding a crash due to a threads bug.
1.0.17
- 2003 Mar 4 - Lowercase external addresses used to create corral files.
- 2003 Mar 3 - Fix up the "mailto" link in the remail message so that it
is really a link that works.
- 2003 Mar 1 - Allow IP addresses to be supplied in the local domain list.
Accept full reports from SpamAssassin versions greater than or equal to 2.50.
Make user name check optional in MailRelease. Allow MailRelease to release
corralled mail without path and partition name.
- 2003 Feb 28 - Add options to include SMTP AUTH and other special classes
of users in the local domain (for purposes of internal ->
internal/external determination). Remove the background parameter from the
HTML body tag.
- 2003 Feb 27 - Add options to filter mail transiting the system. Change
the handling of unknown file types so they are disposed of according to the
user's preference.
- 2003 Feb 26 - Allow statistical filter to be tuned through config file
options.
1.0.16
- 2003 Feb 19 - Remove certain HTML escape sequences.
- 2003 Feb 17 - Fix base 64 re-encoding problem for templates with no
line feeds (was causing occasional crash on badly-formed encoded messages).
Fix problem with HTML tag quoting algorithm.
- 2003 Feb 16 - Allow enough buffer space for badly formatted Base 64
encoded messages to be modified.
- 2003 Feb 13 - Create a man page for sendmailfilter.
- 2003 Feb 12 - Split message and deliver altered/unaltered copies to
those who request virus filtering and those who don't.
- 2003 Feb 9 - Allow filtered viruses to be partitioned in the corral.
Delete recipients from the envelope, if they request non-delivery of spam.
- 2003 Feb 7 - Cache /etc/passwd entries to avoid crash due to threads
bug.
- 2003 Feb 6 - Add statistical spam fast path checks.
- 2003 Feb 4 - Remove HTML comments embedded in the middle of words.
- 2003 Jan 30 - Use proxy userid for DBM lookup if no options found under
regular lookup.
- 2003 Jan 20 - Include sample sendmail.cf file in distro.
- 2003 Jan 19 - Check for extra options on command line. Make spam
arbitron error reportage more comprehensive.
- 2003 Jan 16 - Save PID and initial command string in PID file at
startup.
1.0.15
- 2003 Jan 12 - Allow virus/spam tags to be defined in smfopts.h.
Add options for ignoring spam and/or virus filtering.
- 2003 Jan 11 - Don't corral spam for nonexistant users.
- 2003 Jan 10 - Use the virtual user table to determine local users.
Only process spam if the virtual users are local users.
- 2003 Jan 7 - Convert the spam arbitron IP address ahead of time to
avoid a crash in gethostbyname on BSD.
- 2002 Dec 24 - Bail out if invalid options are detected on the command
line or in the global config file.
1.0.14
- 2002 Dec 15 - Fix crash introduced by language support. Fixed loop
caused by messages with bogus initial MIME separators.
- 2002 Dec 13 - Disable possible exploit through the "name" parameter of
the "Content-Type" header. Fixed bug that caused a memory overrun when
mangling certain file names.
1.0.13
- 2002 Dec 7 - Use flock simulator on systems with no flock.
- 2002 Dec 6 - Fixed problem with overly-greedy regular expression not
updating repeating fields greater than nine in ConfigEdit.cgi.
- 2002 Nov 26 - Sum user options if requested to do so.
- 2002 Nov 24 - Accept the ".csv" attachment type.
- 2002 Nov 23 - Use the "X-Accept-Language" tag to set the language used
for message inserts.
- 2002 Nov 22 - Handle blanks, etc. in to/from names.
- 2002 Nov 18 - Find attachments in malformed MIME entities.
- 2002 Nov 14 - Improved message insertion for encoded mail.
- 2002 Nov 13 - Added support for HTML tag filtering by parameter. Allow
innocuous parameters for the <body> and <meta> tags.
- 2002 Nov 12 - Changes to support Solaris properly during configuration.
Fixed problem with percent signs in addresses causing syslog to crash the
filter.
- 2002 Nov 8 - Fixed possible exploit through attachments defined directly
in message headers.
- 2002 Nov 6 - Allow the <html> tag to have parameters (for
Outlook's sake).
- 2002 Nov 5 - Strip pesky <Ctrl-M> from user names in MailRelease.
Handle long file names wrapped by mail readers. Clear BASH_ENV in case
running suidperl.
- 2002 Oct 21 - Fixed problem with comments being replaced in poorly
formed messages.
1.0.12
- 2002 Oct 18 - Fix local AutoRemail option having no effect.
- 2002 Oct 18 - Include all SpamAssassin response headers in reports, not
just X-Spam-Status and X-Spam-Checker-Version.
- 2002 Oct 16 - Add missing move of X-Tag options to local.
- 2002 Oct 13 - Use fully qualified names for config database lookup if
"-q" or "QualifyNames" is used. Add the Language, LanguageDirectory and "-L"
options.
1.0.11
- 2002 Oct 9 - Accept ISO encodings in MIME file name parameters.
- 2002 Oct 7 - Added generic config file editor, supporting flat files and
DBM files for storing user parameters.
- 2002 Oct 5 - Allow config lookup via DBM database.
1.0.10
- 2002 Sep 27 - Remove leading space from domain names, since people
appear not to be able to read the documentation. Fix problem with fputs
returning different return code under BSD. Updated list of HTML tags to
allow most innocuous ones. Download is now available in RPM form for
RedHat Linux.
- 2002 Sep 26 - Fix errno being reported incorrectly as zero.
- 2002 Sep 24 - Change the corral to /var/spool/MailCorral. Create the
corral directory if it doesn't exist.
- 2002 Sep 23 - Use the first local recipient for local options
processing.
1.0.9
- 2002 Sep 21 - Force add of "localhost" to domain list, if not already
present.
- 2002 Sep 18 - Fix problem with "-r" and "-rm" introduced by the adding
of options as keywords.
- 2002 Sep 17 - Removed a possible exploit that relies on HTML tags being
split across two pieces of an attachment.
- 2002 Sep 14 - Switch to wholly percentage based spam determination
because Spam Assassin returns the wrong flag value when SQL preferences are
used.
1.0.8
- 2002 Sep 10 - Added option to turn off spam fast path. Fixed problem
with HTML reports, sent to postmaster, not having MIME headers. Fix
handling of headers in responses from spamd 1.3 and up.
- 2002 Sep 9 - Allow all local options to be supplied on a per-user basis
from a local config file. Allow local options to turn off global options.
Add a protocol type for the internal fast path spam arbitron.
- 2002 Sep 8 - Add message options to allow tailoring of text inserted
into filtered mail. Allow options to be continued on multiple lines.
- 2002 Sep 7 - Allow all command line options to be specified as keywords
in config files. Allow all options to be supplied in a global config file.
- 2002 Sep 6 - Add flockfile/funlockfile around writes to debug file to
reduce overhead caused by locking in low-level I/O routines. Fix random
crashes when writing to logfile under heavy threading.
- 2002 Sep 4 - Add queue ID to log entries to assist in debugging when
multi-tasking.
1.0.7
- 2002 Sep 2 - Added MailRelease to allow filtered messages to be
remailed.
- 2002 Sep 1 - Fix crash when processing straight message/rfc822 types
(i.e. not MIME). Allow filtered messages to be automatically remailed.
1.0.6
- 2002 Aug 27 - Added the domain names option and domain names file
support.
- 2002 Aug 26 - Add support for spamd 1.3 and the INFO command.
1.0.5
- 2002 Aug 22 - Update list of filtered file extensions. Add flag for
checking external deliveries too. Add flag for spam reporting always. Add
flags for envelope and version info in the headers.
- 2002 Aug 21 - Remove parameters from <body> tag when laundering
HTML. All good/bad HTML tags are now configurable. Allow more innocuous
HTML tags to pass through unscathed. Only open save and work files if
really necessary to process message, instead of on spec.
- 2002 Aug 19 - Added bad file type ".mhtml". Added good file types
".p7c", ".p7m" and ".p7s". Added good MIME types "application/pkcs7-mime"
and "application/pkcs7-signature". Fixed problem with named text being
marked as an attachment when certain mail readers treat it as part of the
message. Fixed a problem with cleanup during abort, crashing if no
recipient seen.
- 2002 Aug 17 - Use received header date for spam tag instead of date
header (which can be omitted). Format sendmail socket address for debugging.
- 2002 Aug 15 - Fixed problem with unterminated HTML tags not being
laundered properly.
1.0.4
- 2002 Aug 14 - Eliminated erroneous spam check when plain old mail
messages are remailed. Added fast path spam lookup for white and black
lists.
- 2002 Aug 12 - Fix missing "[SPAM]" tag, if message delivery is bypassed
for reasons other than spam. Fix problems with malformed MIME entities that
have no trailing separator.
- 2002 Aug 10 - Added support for Spam Assassin option files.
- 2002 Aug 9 - Added debug levels.
1.0.3
- 2002 Aug 6 - Added options for spam delivery modes ("-sc", "-sd",
"-st") and reporting types ("-s1", "-s2", "-s3"). Added support for spam
reporting levels two and three for Spam Assassin. Fixed problem with
inserted text when all text/plain and text/html components of a message are
encoded base64.
- 2002 Aug 5 - Added <iframe> (the favorite tag of virus
professionals everywhere) to the list of HTML tags that are always removed,
regardless of the "-h" flag setting. Remove <cr><lf>s from
message subjects, per bug in Outlook that allows virus launcers in subjects.
- 2002 Aug 4 - Move spam processing to spamfilter.c. Fixed memory leak
when spam checking plain old mail messages. Improved spam checking
performance for multipart MIME types.
1.0.2
- 2002 Aug 3 - Fix problem with corralled spam only being held for the
last recipient in a list of recipients, when multiple recipients passed to
SMTP for a single messge. Problem occurred when envelope was addressed to
more than one user. Spam would be detected, processed and corralled but
only one notification was sent to a single recipient (last one on the
envelope), not all of the recipients.
- 2002 Aug 2 - Allow multiple domains in local domain list for "-i"
option, etc.
- 2002 Aug 1 - Add "-d" (debugging option) for dynamic debugging.
1.0.1
- 2002 Jul 29 - Add the "-q" (qualify recipient names in the spam corral)
and "-u" (supply a proxy userid for filter criteria lookup) options.
1.0
- 2002 Jul 12 - Add spam filtering.
- 2002 Jun 20 - Add decode of MIME components for HTML check.
- 2002 Feb 25 - Initial coding.