The sendmail program views the text that makes up rules and addresses as being composed of individual tokens. Rules are tokenized - divided up into individual parts - while the configuration file is being read and while they are being normalized. Addresses are tokenized at another time (as we'll show later), but the process is the same for both.
The text our.domain, for example,
is composed of three tokens: our, a dot, and domain.
These 10 characters are divided into tokens
by the list of separation characters defined by the
$o) option (see Section 34.8.45, OperatorChars or $o):
Do.:%@!^=/ prior to V8.7 O OperatorChars=.:%@!^/ V8.7 and above
When any of these separation characters are recognized in text, they are considered individual tokens. Any leftover text is then combined into the remaining tokens.
xxx@yyy;zzz becomes xxx @ yyy;zzz
@ is defined to be a token, but
; is not.
Therefore, the text is divided into three tokens.
However, in addition to the characters in the
$o) option, sendmail
defines 10 tokenizing characters internally (in parseaddr.c):
These two lists are combined into one master list that is used for all tokenizing. The above example, when divided by using this master list, becomes five tokens instead of just three:
xxx@yyy;zzz becomes xxx @ yyy ; zzz
In rules, quotation marks can be used to override the meaning of tokenizing characters defined in the master list. For example,
"xxx@yyy";zzz becomes "xxx@yyy" ; zzz
Here, three tokens are produced, because the
inside quotation marks. Note that the quotation
marks are retained.
Because the configuration file is read sequentially from start to finish,
should be defined before any rules are declared.
But note that beginning with V8.7 sendmail, omission of this option cause the
separation characters to default to
. : % @ ! ^ / [ ]
As we progress into the details of rules, you will see that certain
characters become operators when prefixed with a
Operators cause sendmail to perform actions, such as looking
for a match (
$* is a wildcard operator) or replacing tokens with others by position
$1 is a replacement operator).
For tokenizing purposes, operators always divide one token from another, just as the characters in the master list did. For example
xxx$*zzz becomes xxx $* zzz
xxx zzz becomes xxx zzz
Second, although the space character separates tokens, it is not itself a token. That is, in the above example the seven characters on the left (the seventh is the space in the middle) become two tokens of three letters each, not three tokens. Therefore the space character can be used inside the LHS or RHS of rules for improved clarity but does not itself become a token or change the meaning of the rule.
After an address has passed through all the rules (and has been modified by rewriting), the tokens that form it are pasted back together to form a single string. The pasting process is very straightforward in that it mirrors the tokenizing process:
xxx @ yyy becomes xxx@yyy
The only exception to this straightforward pasting process occurs when
two adjoining tokens are both simple text. Simple text is anything
other than the separation characters (defined by the
$o) option, see Section 34.8.45,
and internally by sendmail) or the operators (characters prefixed
$ character). The
yyy above are
both simple text.
When two tokens of simple text are pasted together, the character
defined by the
B) option (see Section 34.8.5, BlankSub (B)) is inserted between them.
option is defined as a dot, so two tokens of simple text would
have a dot inserted between them when they are joined:
 In the old days (RFC733), usernames to the left of the
@could contain spaces. But UNIX also uses spaces as command-line argument separators, so option
xxx yyy becomes xxx.yyy