Junk Mail Controls: Difference between revisions

From MozillaZine Knowledge Base
Jump to navigationJump to search
(added "Use custom headers added by your email provider" section)
 
(64 intermediate revisions by 7 users not shown)
Line 2: Line 2:




Both [[Thunderbird]] and the [[Mozilla Suite]] can detect and filter unwanted e-mail messages. In order for the junk-mail filtering to be effective, however, you first need to "train" it because it uses [http://en.wikipedia.org/wiki/Bayesian_filtering Bayesian filtering].
Both [[Thunderbird]] and the [[Mozilla Suite]] can detect unwanted e-mail messages. For junk-mail detection to be effective, however, you must "train" it because it uses [http://en.wikipedia.org/wiki/Bayesian_filtering Bayesian filtering]. Anti-phishing and other tools are also discussed below.


==Activating the Junk Mail Controls==
==Activating the Junk Mail Controls==
To start using the Junk Mail Controls in Thunderbird:
To start using the Junk Mail Controls in Thunderbird'''*''':
# In versions 1.5.0.x and earlier, go to "Tools -> Junk Mail Controls...". In 2.0.0.x, the controls are divided between two locations: (a) Tools -> Options -> Privacy -> Junk and (b) Tools -> Account Settings -> *each account* -> Junk Settings.
# Note: Controls are divided between two locations: (a) [[Menu differences in Windows, Linux, and Mac|Tools -> Options]] -> Security -> Junk, and (b) Tools -> Account Settings -> *each account* -> Junk Settings.
# From the dropdown list, choose the email account for which you want to activate the Junk Mail Controls. "Email account" means the folder structure under which the emails are saved, so if you use the [[Global Inbox]] in Thunderbird, you need to select "Local Folders" as the account. This is different from the [[filters (Thunderbird) | message filters]], where you select the account that retrieves the email.
# Click [[Menu differences in Windows, Linux, and Mac|Tools -> Options]] -> Security -> Junk
# Click the "Adaptive Filter" tab and make sure that the checkbox for "Enable adaptive junkmail detection" is checked. Even though the phrase "Configure Junk Settings for: <account name>" shows when you are on this tab, the option to reset training data applies to ALL of your accounts. There is only one set of training data, so don't reset it here thinking you will only affect the account shown.
# Change settings as desired
# Click the "Settings" tab and choose your settings for "White lists" and "Handling" as desired.
#* Caution: "Reset Training Data" applies to ALL of your accounts. There is only one set of training data, so don't reset it thinking it will only affect one account.
#* You can choose to have junk messages sent to the Junk folder, but the Junk folder normally will not appear alongside your other mail folders until it is first used.
# When finished, click the "OK" button
#* If you choose the option for "sanitizing " junk e-mails, this means that messages marked as junk will be displayed without images or other HTML formatting.
# In Account Settings, choose the email account for which you want to activate the Junk Mail Controls. "Email account" means the folder structure under which the emails are saved. So if you use the [[Global Inbox]] for a pop account, you must select "Local Folders" as the account. (This is different from the [[Message_Filters | message filters]], where you select the account that retrieves the email.)
# Click the "Junk Settings" tab
# Make sure that the checkbox for "Enable adaptive junk mail" is checked.  
# Choose your settings for [http://en.wikipedia.org/wiki/Whitelist Whitelists] at "Do not mark mail as junk...".
#* It only supports whitelisting specific email addresses, and only those in your address books. However, [[Message_Filters | message filters]] may be used to test whether the sender "contains" a domain, and set the junk status accordingly, because filters are run before the junk mail controls.  
# Make sure that the checkbox for "Move junk messages to" is checked, unless you want junk mail kept in your Inbox. (Junk folder normally will not appear alongside your other mail folders until you mark your first message as Junk.)
## Enable retention settings if desired.  But if you do not enable it, then you must periodically Empty Junk.
# If you have more than one email account, repeat the above steps for each account with which you want to use the Junk Mail Controls.
# If you have more than one email account, repeat the above steps for each account with which you want to use the Junk Mail Controls.
# When finished, click the "OK" button to exit the Junk Mail Controls dialog.
# When finished with account settings, click the "OK" button.
# Proceed next to training ...
 
* See this link for the menu paths for operating systems other than Windows -- http://kb.mozillazine.org/Menu_differences_in_Windows,_Linux,_and_Mac


==Training the Junk Mail Controls==
==Training the Junk Mail Controls==
To train the junk-mail filtering, you need to mark messages that you've received as either "junk" or "not junk", and it's important that you mark both types of messages rather than simply the ones that are junk. There are various ways that you can mark messages:
Junk processing must be well '''trained''' for it to work correctly. Training involves marking many messages as "junk" and many messages as "not junk".  It is important to mark '''both types of messages''', both good and bad, not just the ones that are junk.
 
'''Initially''', incoming messages might not be accurately junked because you have not trained it enough.
# Check your Junk folder to see if any non-junk messages have been detected as junk, and mark non-junk ones as not junk.
# Bayesian filtering requires at least 100 bad messages be marked as spam and 100 good messages marked as not junk to function. To work best, it needs a few hundred of each marked.
#* If you mark a thousand spam messages but do not mark legitimate messages, or very few, it won't work well.
#* It's best to mark '''different types''' of messages good and bad - marking 500 messages from the same source is not as good as marking 500 messages from different people.
 
'''After an initial training period'''
# You should find that Junk Mail Controls are effectively detecting unwanted junk emails and keeping them from your Inbox. If it is not, see the tweaking information below.
# You should occasionally train non-junk messages - doing it when you mark bad messages as junk is a good habit.  
 
There are various ways that you can mark messages:
* Select a message and type "Shift+J" (for not Junk) or "J" (for Junk).
* Right-click on a message and choose "Mark -> As Junk" (or "As Not Junk").
* Right-click on a message and choose "Mark -> As Junk" (or "As Not Junk").
* Select a message and from the "Message" menu, choose "Mark -> As Junk" (or "As Not Junk").
* Select a message and from the "Message" menu, choose "Mark -> As Junk" (or "As Not Junk").
* Select a message and click on the "Junk" icon on the toolbar.
* Select a message and click on the "Junk" icon on the toolbar.
* Select a message and click on the "Junk Status" column in the message-list pane (which will show a small "Junk" icon if the message is marked as junk).
* Select a message and click on the "Junk Status" column in the message-list pane (which will show a small "Junk" icon if the message is marked as junk).
* Select a message and type "J" (for Junk) or "Shift+J" (for not Junk).


Initially, the automatic junk mail detection for incoming messages might not be very accurate because it hasn't yet been trained very much, and you should thus be careful to check your Junk folder to see if any non-junk messages have been mistakenly detected as junk. After an initial training period, however, you should find that the Junk Mail Controls are very effectively detecting unwanted junk emails and keeping them from your Inbox.
You don't need to keep the messages you have marked - marking stores the information it needs as tokens in the training.dat file. Training data for the Junk Mail Controls is stored in "training.dat" in your [[Profile_folder | profile folder]]. You can view the Junk Mail Log by selecting "Junk Mail Log"  in "Tools -> Junk Mail Controls -> Logging", and then use the mouse to 'grab' any corner and stretch it until it expands enough for you to see the log data.
 
==Tweaking the Junk Mail Controls==
'''mail.adaptivefilters.junk_threshold''' is the preference that determines at what "level" messages are classified as junk. It defaults to 90. '''Lowering this value will make it easier to recognize messages as spam, though it increases the risk that it will classify a legitimate message as spam'''. This might help get more messages marked; for example, messages that look like text but are actually clickable images.  (Note however, adaptive processing does NOT examine images nor care about whether a message has an image - it is not part of the junk algorithm)
 
You can change the preference using [[Menu differences in Windows, Linux, and Mac|Tools -> Options]] -> Advanced -> General -> Config Editor. Enter ''junk'' in the Filter field to show only the preferences that contain junk in their name, and then double-click on ''mail.adaptivefilters.junk_threshold'', enter a value lower than the default ''90'' in the edit field and press the OK button.  Many users report good results with values of 30 or lower.
 
[https://addons.mozilla.org/en-US/thunderbird/addon/9886 JunQuilla] is an add-on which can help you tweak junk processing and help you better understand how it is working.  It provides a column that shows the odds that a message is spam, enhances the junk status column, and adds a Uncertain folder for messages requiring a users decision. It requires Thunderbird version 3 and newer.
 
[http://bayesjunktool.mozdev.org/ Bayes Junk Tool] can also be used to examine and modify the raw training data. Sometimes it helps to get rid of tokens that are just as likely to occur in spam and legitimate messages, especially if the training data file gets very large. The web site also has several sets of training data that you can import or merge with your existing training data.
 
Bayesian filters are useful, but they are not always the best tool. Sometimes checking whether the message was sent by somebody on a DNSBL list is more effective. See [[Download_only_certain_POP_messages | this article]] for how to integrate SpamPal and the junk mail controls, and control which messages are downloaded.
 
The [https://addons.mozilla.org/en-US/thunderbird/addon/1898 FolderFlags] extension can set various internal flags that Thunderbird uses to classify folders. If you set the "Junk" flag on a folder (other than the one spam is moved to) it won't scan that folder for spam.


==Any mention of Spam==
The [http://www.chuonthis.com/extensions/deletejunk.php Delete Junk Context Menu] extension adds "Delete Mail Marked as Junk" to a folders context menu. It can be configured to delete mail without moving it to using the Trash folder.
Thunderbird doesn't use the word Spam anywhere, it uses Junk. If you see the word Spam in a Subject or have a Spam folder it is either due to your email provider or a add-on (such as SpamPal). If its due to your email provider log into webmail using a browser and browse its help to get more information. They may support webmail commands to let you manage or disable whatever they're doing.
 
==Any mention of "spam"==
Thunderbird doesn't use or react to the word "spam" in its identification of junk mail, nor does Thunderbird put it in the subject.  
 
If you see the "spam" in a subject or have a folder called "Spam", this is either due to your email provider or an add-on product (such as SpamPal). If it's due to your email provider, log into your webmail using a browser and browse its help to get more information. Your provider may support webmail commands to let you manage or disable whatever they're doing.


==Use custom headers added by your email provider==
==Use custom headers added by your email provider==
Your email provider may run a spam filtering program on the mail server such as SpamAssassin, MailScanner, CRM114, SpamProbe, QSF or Bogofilter that analyzes each message and adds custom headers with information about its content. If they use SpamAssassin see the next section for how to integrate it with Thunderbirds junk mail controls. Otherwise use "View -> Message Source" and look for headers whose name begins with a 'X' and contain phrases such as Spam. They typically provide a spam score and/or keywords that you can test using [[Message_Filters | message filters]].  
Your email provider may run a spam filtering program on their mail server such as SpamAssassin, MailScanner, CRM114, SpamProbe, QSF or Bogofilter that analyzes each message and adds custom headers with information about its content. If they use SpamAssassin, see the next section for how to integrate it with Thunderbird's junk mail controls. Otherwise, use "View -> Message Source" and look for headers whose name begins with a 'X' and contain phrases such as Spam. They typically provide a spam score and/or keywords that you can test using [[Message_Filters | message filters]].  


For example:
For example:
Line 41: Line 79:


You could test whether X-Spam-score "is greater than" a certain value, whether X-Spam-source "doesn't contain" Country='US' (the example was from Sweden) or test whether X-Spam-hits "contains" certain keywords (they're the name of a test that increased the spam score) that you notice that the junk mail controls has problems recognizing spam with those attributes. Not every email provider will provide as much customization as the example, but it should at least have some sort of spam score you can test.
You could test whether X-Spam-score "is greater than" a certain value, whether X-Spam-source "doesn't contain" Country='US' (the example was from Sweden) or test whether X-Spam-hits "contains" certain keywords (they're the name of a test that increased the spam score) that you notice that the junk mail controls has problems recognizing spam with those attributes. Not every email provider will provide as much customization as the example, but it should at least have some sort of spam score you can test.
If your email provider supports Sender Policy Framework (SPF) and/or DomainKeys (DK) you could use the [https://addons.mozilla.org/en-US/thunderbird/addon/345 Sender Verification Extension]. SPF and DK are frameworks used to help figure out whether a From: address was spoofed. The add-on uses that information plus DNS black and white lists such as SURBL, Spamhaus, DNSWL, and Sender Score Certified to check the senders reputation. When you read a message the add-on adds a line at the top of message with the verification status of the sender. For example, "Reputable Sender", "This sender is a known malicious spammer or phisher. Discard this email.", or "Sending domain does not support verification (address could be forged).". Since you have to actually read the message to get the warning, its probably most useful as a alternative to Thunderbirds phishing protection (which most users disable due to its inability to learn, and many false positives).


==Trusting SpamAssassin and SpamPal==
==Trusting SpamAssassin and SpamPal==
Spammers sometimes use [http://en.wikipedia.org/wiki/Bayesian_poisoning Bayesian poisoning] to degrade spam filters that use Bayesian filtering. SpamPal uses [http://en.wikipedia.org/wiki/DNSBL DNS Blacklists] and SpamAssassin uses several methods (its most well known for its extensive testing of message headers) to filter spam so that type of attack has little or no effect on them. Both of them add special headers to a message to indicate whether its spam.  
Spammers sometimes use [http://en.wikipedia.org/wiki/Bayesian_poisoning Bayesian poisoning] to degrade spam filters that use Bayesian filtering. [http://spampal.sourceforge.net/index2.html SpamPal] uses [http://en.wikipedia.org/wiki/DNSBL DNS Blacklists] and SpamAssassin uses several methods (it's best known for its extensive testing of message headers) to filter spam so that type of attack has little or no effect on them. Both of them add special headers to a message to indicate whether it's spam. SpamPal is only available for Windows, but the [http://appdb.winehq.org/index.php Wine Application Database] rates it as platinum (i.e. install and run flawlessly on an out-of-the-box Wine installation) 


Tools -> Junk Mail Controls has a setting to tell Thunderbird to trust junk mail headers set by either [http://www.spampal.org/ SpamPal] or [http://spamassassin.apache.org/ SpamAssassin]. The order of processing is:
Tools -> Junk Mail Controls has a setting to tell Thunderbird to trust junk mail headers set by either [http://spampal.sourceforge.net/index2.html SpamPal] or [http://spamassassin.apache.org/ SpamAssassin]. The order of processing is:
# Message filters
# Message filters
# "Trust header"
# "Trust header"
Line 54: Line 90:
Some email providers customize the headers added by SpamAssassin, or modify the subject prefix. This can cause the junk mail controls to ignore the information. "Trust header" is actually a [http://mxr.mozilla.org/mozilla/source/mailnews/base/search/src/SpamAssassin.sfd standard message filter], stored in a isp subdirectory in the Thunderbird program directory. Thunderbird checks whether either X-Spam-Status: or X-Spam-Flag: begins with Yes, or the subject begins with ***SPAM***. If you run into this problem backup the SpamAssassin.sfd file and then change what it tests for using a text editor (not a word processor). There is also a SpamPal.sfd file.  
Some email providers customize the headers added by SpamAssassin, or modify the subject prefix. This can cause the junk mail controls to ignore the information. "Trust header" is actually a [http://mxr.mozilla.org/mozilla/source/mailnews/base/search/src/SpamAssassin.sfd standard message filter], stored in a isp subdirectory in the Thunderbird program directory. Thunderbird checks whether either X-Spam-Status: or X-Spam-Flag: begins with Yes, or the subject begins with ***SPAM***. If you run into this problem backup the SpamAssassin.sfd file and then change what it tests for using a text editor (not a word processor). There is also a SpamPal.sfd file.  


Some users have [https://bugzilla.mozilla.org/show_bug.cgi?id=381589 reported] that the trust SpamAssassin option sometimes ignores the junk mail headers in Thunderbird 2.x. Its not clear whether you can workaround this bug by disabling the option and adding the appropriate message filter.
Some users have [https://bugzilla.mozilla.org/show_bug.cgi?id=381589 reported] that the trust SpamAssassin option sometimes ignores the junk mail headers in Thunderbird 2.x. It's not clear whether you can work  around this bug by disabling the option and adding the appropriate message filter.


==Tweaking==
If you use SpamAssassin you might want to uncheck "enable adaptive junk mail controls for this account" in [[Menu_differences_in_Windows,_Linux,_and_Mac | Tools -> Account Settings]] -> Junk Settings. That controls the Bayesian filter in the junk mail controls, which is what it uses to learn how to recognize junk. SpamAssassin is normally configured with a Bayesian filter that is more sophisticated, has more data to learn from (it learns from all of your email providers messages, not just your messages), and is probably tweaked by the same admin who tweaks the SpamAssassin filters. Having two Bayesian filters for the same account makes things unnecessarily complex. For example, it's not defined what's supposed to happen if you mark a message as not junk, and later on the adaptive junk mail controls notices (again) that SpamAssassin says it's spam.
The '''mail.adaptivefilters.junk_threshold''' preference is a threshold used to determine when messages are classified as junk. It defaults to 90 in version 1.5.0.4. '''Lowering this value will make it easier to recognize messages as spam, though it increases the risk that it will classify a legitimate message as spam'''. This might be useful if you get spam messages that it seems to have a tough time learning about. For example, messages that look like text but are actually clickable images.  


You can change the preference using Tools -> Options -> Advanced -> General -> Config Editor. Enter ''junk'' in the Filter field to show only the preferences that contain junk in their name, and then double-click on ''mail.adaptivefilters.junk_threshold'', enter a value lower than the default ''90'' in the edit field and press the OK button.  Many users report good results with values of 30 or lower.
You could probably edit the SpamAssassin.sfd file in the isp subdirectory in the Thunderbird program directory to make the junk mail controls think another spam filter such as SpamBayes is actually SpamAssassin. The first step would be to figure out what the spam filter modifies in the message in order to identify whether it's spam by looking at a messages source.


The bayesian filter typically requires several hundred spam and several hundred legitimate messages in order to train itself to recognize spam. Its needs both, if you have a thousand spam messages but only a dozen legitimate messages it won't learn much. This doesn't mean its initially useless, just that how well it works will depend a lot more upon what spam messages you get. You don't need to keep the messages afterwards, it stores all of the information it needs as tokens in the training.dat file.  
====SpamBayes====
Thunderbird Junk mail controls do not support trusting [http://spambayes.sourceforge.net/windows.html SpamBayes]. But the [https://addons.mozilla.org/en-US/thunderbird/addon/4025 ThunderBayes] add-on integrates SpamBayes with Thunderbird with a toolbar button to classify a message as "Spam" or "Ham", and a preferences page in the account settings.


The [http://bayesjunktool.mozdev.org/ Bayes Junk Tool] can be used to examine and modify the training data. Sometimes it helps to get rid of tokens that are just as likely to occur in spam and legitimate messages, especially if the training data file gets very large. The web site also has several sets of training data that you can import or merge with your existing training data.
====Spamato====
The junk mail controls don't support trusting [http://www.spamato.net/ Spamato], possibly because the Spamato instructions recommend disabling the junk mail controls. There is a [http://sourceforge.net/forum/forum.php?forum_id=459856 Spamato forum], some documentation on the [http://spamato.wiki.sourceforge.net/filters different types of spam filters they support] and a [http://spamato.wiki.sourceforge.net/FAQ FAQ] on sourceforge.net. Spamato does not store its information in the [[Profile_folder_-_Thunderbird | Thunderbird profile]], it creates its own [http://spamato.wiki.sourceforge.net/faq#general_profile profile]


Bayesian filters are useful, but they're not always the best tool. Sometimes checking whether the message was sent by somebody on a DNSBL list is more effective. See [[Download_only_certain_POP_messages | this article]] for how to integrate SpamPal and the junk mail controls, and control which messages are downloaded.
==Image spam==
One way to weed out image-based spam is to create a message filter and set it to match all of the following:


The [https://addons.mozilla.org/en-US/thunderbird/addon/1898 FolderFlags] extension can set various internal flags that Thunderbird uses to classify folders. If you set the "Junk" flag on a folder (other than the one spam is moved to) it won't scan that folder for spam.
*Content-Type contains multipart/related
*From isn't in my address book (repeat this for each address book)
 
In "Perform these actions" add


The [http://www.chuonthis.com/extensions/deletejunk.php Delete Junk Context Menu] extension adds "Delete Mail Marked as Junk" to a folders context menu. It can be configured to delete mail without moving it to using the Trash folder.
*"Set Junk Status" to "Junk"
* "Move Message to" your junk mail folder.  


==Image Spam==
This will mark the message as junk and move it to the Junk folder if the Content-Type header contains multipart/related and the sender wasn't in your address book. The message filters don't know how to recognize Content-Type headers, you will need to add it using the "customize..." option at the bottom of the leftmost list box. This method is rather heavy-handed. If your email provider runs a spam filter program such as SpamAssassin, it will typically do a much better job recognizing image spam.
One way to weed out image based spam is to create a new filter and set it to match all of the following:


*Content-Type contains multipart/related
==Problems with junk processing==
*From isn't in my address book (repeat this for each address book)
* [[Junk_messages_not_moved|How to fix or work around junk mail processing problems]]
Then if all match, the message is sent to the Junk folder and marked junk.


==Other information==
==After classification filters==
* The file that stores your custom training data for the Junk Mail Controls is called "training.dat". It is stored in your [[profile folder]].


* To view your Junk Mail Log, you follow "Tools -> Junk Mail Controls -> Logging", select "Junk Mail Log" and then use the mouse to 'grab' any corner and stretch it open till it expands and the area where any log data is will show.
See [http://mesquilla.com/2009/08/28/managing-spam-with-after-classification-filters/ Managing spam with “after classification” filters] for how to use spam classification in a message filter.


==Regular Expressions - Advanced==
==Regular expressions - advanced==
Neither the junk mail controls or the message filters support wild cards or [http://en.wikipedia.org/wiki/Regular_expression regular expressions]. There don't appear to be any extensions that add support for that. However, [http://www.spampal.org/ SpamPal] (a mail classification program normally used for filtering spam) supports a [http://www.slabihoud.de/spampal/ RegExFilter Plugin] that adds regular expression support based on Perl Regular Expressions. If you configure the junk mail controls to trust SpamPal you could use regular expressions to filter spam. [http://forums.mozillazine.org/viewtopic.php?t=617487]
Neither the junk mail controls or the message filters support wild cards or [http://en.wikipedia.org/wiki/Regular_expression regular expressions]. There don't appear to be any extensions that add support for that. However, [http://spampal.sourceforge.net/index2.html SpamPal] (a mail classification program normally used for filtering spam) supports a [http://www.slabihoud.de/spampal/ RegExFilter Plugin] that adds regular expression support based on Perl Regular Expressions. If you configure the junk mail controls to trust SpamPal you could use regular expressions to filter spam. [http://forums.mozillazine.org/viewtopic.php?t=617487]


There are many other SpamPal plugins available [http://www.spampal.org/plugins.html here]. For example, you could extend white lists and black lists to apply to email addresses from any header, white list any message that contain words from a list of good words, filter on what web sites are mentioned, launch other programs (passing them information about the message as command line arguments) or run Ruby scripts. The main drawback is none of this is integrated into the junk mail controls - it just knows when SpamPal marks a message as spam.
There are many other SpamPal plugins available [http://www.spampal.org/plugins.html here]. For example, you could extend white lists and black lists to apply to email addresses from any header, white list any message that contain words from a list of good words, filter on what web sites are mentioned, launch other programs (passing them information about the message as command line arguments) or run Ruby scripts. The main drawback is none of this is integrated into the junk mail controls - it just knows when SpamPal marks a message as spam.
==Anti-phishing tools==
Sender Policy Framework (SPF) is a method for verifying an email sender's domain name, to figure out whether an address is spoofed. The [https://addons.mozilla.org/en-US/thunderbird/addon/345 Sender Verification Extension] makes use of SPF to verify the From address on emails, plus DNS black and white lists such as SURBL, Spamhaus, DNSWL, and Sender Score Certified to check the sender's reputation. When you open a message, the add-on adds a line at the top of message with the verification status of the sender. For example, "Reputable Sender", "This sender is a known malicious spammer or phisher. Discard this email", or "Sending domain does not support verification (address could be forged).". You have to actually read the message to get the warning (it is not meant to be used as a mail filter), and it's probably most useful as an alternative to Thunderbird's phishing protection (which most users disable due to its inability to learn and its many false positives).
==Check the spam score of messages you sent==
[http://www.mailingcheck.com/check-spam-score/ MailingCheck] is a free Windows program that calculates the spam score of a message based on SpamAssassin rules. Its useful if you suspect that some of your recipients aren't getting your message due to it being treated as spam and you can't verify that by having them check in their junk mail folder. [https://www.port25.com/support/authentication-center/email-verification/ Port25] (copy and paste a message) , [https://www.mail-tester.com/ mail-tester] (email them) and [http://www.emailspamtest.com/ emailspamtest] (copy and paste a message) are web sites that perform similar checks for a message.
You might want to also check if your mail server is black listed by checking it with a web site such as [http://www.mxtoolbox.com/blacklists.aspx MXToolbox].
==Junk filtering in newsgroups==
Experimental support for junk filtering of newsgroups was added to Thunderbird 3.0. The [https://addons.mozilla.org/en-US/thunderbird/addon/9886/ JunQuilla] add-on is needed to manage some features. See [http://mesquilla.com/2010/07/21/junk-management-for-newsgroups-in-thunderbird-3/ this blog post] for more information.
==See the webmail spam folder==
A POP account is only aware of one folder stored on the POP server, the inbox. This is a limitation of the POP3 protocol. However, its possible for some POP3 servers to be configured to lie to the email client, and provide the contents of both the inbox and the spam folder used by webmail when the email client checks for new mail. This is totally transparent to the email client. Email providers that support this have a webmail settings to configure this.
IMAP accounts don't have this limitation. That can use the same folders as webmail.
==Report Spam==
Most add-ons to report spam seem to report it to little known organizations. The best seems to be the [https://addons.mozilla.org/en-US/thunderbird/addon/habul/ HabuL] add-on that "Reports spam to the SpamCop, the FTC, FDA, SEC, ACMA and/or Knujon.com. It also allows you to put in your own custom addresses to report spam to such as your ISP or corporate abuse address."
If you are using a IMAP account some email providers let you specify (using webmail commands) that any messages in your junk folder should be used to help train a personal bayes database to recognize spam. This improves the spam filtering done by your email provider. [http://blog.fastmail.fm/2007/08/31/improved-spam-filtering-with-per-user-bayes-databases/ Improved spam filtering with per-user bayes databases] is one example.


==See also==
==See also==
* [[Message aging | Automatically delete messages based on age or number of messages per folder]]
* [[Message aging | Automatically delete messages based on age or number of messages per folder]]
* [[Download_only_certain_POP_messages | Download only certain POP messages]]
* [[Download_only_certain_POP_messages | Download only certain POP messages]]
* [[Foreign language spam]]
* [[Message Filters]]
* [[Modify_Thunderbird_settings | Modify Thunderbird settings]]
* [[Modify_Thunderbird_settings | Modify Thunderbird settings]]
* [[Use another email provider as a spam filter]]


==External links==
==External links==
* [http://www.nidelven-it.no/articles/introduction_to_thunderbird_5 Introduction to Thunderbird: Junk Mail Controls]
 
* [http://mesquilla.com/category/extensions/junquilla/ An excellent set of articles on Junk Processing]
* [http://opensourcearticles.com/thunderbird_15/english/part_05 Introduction to Thunderbird: Junk Mail Controls]
* [http://www.ischool.utexas.edu/technology/tutorials/email/mozillamail/9junk.html Mozilla Suite:  Junk Mail Controls]
* [http://www.ischool.utexas.edu/technology/tutorials/email/mozillamail/9junk.html Mozilla Suite:  Junk Mail Controls]
* [http://wiki.apache.org/spamassassin/AvoidingFpsForSenders Some tips for avoiding your mail being considered spam]
* [http://wiki.apache.org/spamassassin/AvoidingFpsForSenders Some tips for avoiding your mail being considered spam]
* [http://www.mozilla.org/start/1.5/extra/using-junk-control.html Using the Junk Mail Controls in Mozilla Suite] (from mozilla.org)
* [http://wiki.apache.org/spamassassin/MailProxy Proxies] to let an email client make a connection using SpamAssassin. This is useful if you want to install SpamAssassin but don't want to use fetchmail and/or procmail.
*[http://www.cdt.org/speech/spam/030319spamreport.shtml Why Am I Getting All This Spam?]
* [http://mesquilla.com/2009/08/28/managing-spam-with-after-classification-filters/ Managing spam with "after classification" filters]
*[http://www.mozilla.org/mailnews/arch/spam/purge.html Junk mail purging]
* [http://technet.microsoft.com/en-us/library/aa996878.aspx X-MS-Exchange-Organization-Antispam-Report header] added by Microsoft Exchange.
===Add-on to improve Thunderbird's junk processing===
 
* The [https://addons.mozilla.org/en-US/thunderbird/addon/9886 JunQuilla] add-on adds support for a column that shows the odds that a message is spam, enhances the junk status column, and adds a Uncertain folder for messages requiring a users decision. It works with version 3 and newer.
* The [https://addons.mozilla.org/en-US/thunderbird/addon/345 Sender Verification Anti-Phishing] add-on verifies the senders domain to help prevent phishing.
* The [https://addons.mozilla.org/en-US/thunderbird/addon/4121 Show Address] extension adds columns to show the senders and recipient's email address (rather than the display name). This can be useful when trying to identify spam.
* The [https://addons.mozilla.org/en-US/thunderbird/addon/4798 Spamness] add-on adds a color and size based display of how likely a message is spam based on its SpamAssassin score.
 
===Add-ons interfacing to external services===
 
* The [http://www.mailwasher.net/ MailWasher Free] proxy lets you view and bounce messages before downloading them. If you want to do that read [http://www.dontbouncespam.org/ this explanation] first on why bouncing by end users will just send them to an innocent person who may report the bouncer for sending them spam.
* [http://www.spamfighter.com/Product_Info.asp SPAMfighter Standard] is a free version of the SpamFighter spam filter that supports Thunderbird. It adds a SPAMfighter footer to your messages and has a limited number of blacklist/whitelist entries.
* The [https://addons.mozilla.org/en-US/thunderbird/addon/2738 Spamato4Thunderbird] add-on adds support for the Spamato spam filters. It requires Java. Its an alternative to installing Spamato as a local proxy. [http://camel-s-eye.findmysoft.com/ Camels's Eye] is another Java based spam filter It also provides message filtering. Unfortunately both of these applications haven't been updated for years and there isn't any support available.
* The [https://addons.mozilla.org/en-US/thunderbird/addon/4025 ThunderBayes] add-on adds support for the SpamBayes spam filter. Its no longer supported by the author. [https://addons.mozilla.org/en-US/thunderbird/addon/thunderbayespp/ ThunderbirdBayesPP] is a more recent alternative implementation. Neither supports IMAP.


[[Category:Organizing and finding messages (Thunderbird)]]
[[Category:Organizing and finding messages (Thunderbird)]]
[[Category:Mail (Mozilla Suite)]]
[[Category:Mail (Mozilla Suite)]]

Latest revision as of 10:35, 22 June 2017

This article applies to both Thunderbird and the Mozilla Suite. The Junk Mail Controls interface in the Mozilla Suite is slightly different from the Thunderbird interface, described below.


Both Thunderbird and the Mozilla Suite can detect unwanted e-mail messages. For junk-mail detection to be effective, however, you must "train" it because it uses Bayesian filtering. Anti-phishing and other tools are also discussed below.

Activating the Junk Mail Controls

To start using the Junk Mail Controls in Thunderbird*:

  1. Note: Controls are divided between two locations: (a) Tools -> Options -> Security -> Junk, and (b) Tools -> Account Settings -> *each account* -> Junk Settings.
  2. Click Tools -> Options -> Security -> Junk
  3. Change settings as desired
    • Caution: "Reset Training Data" applies to ALL of your accounts. There is only one set of training data, so don't reset it thinking it will only affect one account.
  4. When finished, click the "OK" button
  5. In Account Settings, choose the email account for which you want to activate the Junk Mail Controls. "Email account" means the folder structure under which the emails are saved. So if you use the Global Inbox for a pop account, you must select "Local Folders" as the account. (This is different from the message filters, where you select the account that retrieves the email.)
  6. Click the "Junk Settings" tab
  7. Make sure that the checkbox for "Enable adaptive junk mail" is checked.
  8. Choose your settings for Whitelists at "Do not mark mail as junk...".
    • It only supports whitelisting specific email addresses, and only those in your address books. However, message filters may be used to test whether the sender "contains" a domain, and set the junk status accordingly, because filters are run before the junk mail controls.
  9. Make sure that the checkbox for "Move junk messages to" is checked, unless you want junk mail kept in your Inbox. (Junk folder normally will not appear alongside your other mail folders until you mark your first message as Junk.)
    1. Enable retention settings if desired. But if you do not enable it, then you must periodically Empty Junk.
  10. If you have more than one email account, repeat the above steps for each account with which you want to use the Junk Mail Controls.
  11. When finished with account settings, click the "OK" button.
  12. Proceed next to training ...

Training the Junk Mail Controls

Junk processing must be well trained for it to work correctly. Training involves marking many messages as "junk" and many messages as "not junk". It is important to mark both types of messages, both good and bad, not just the ones that are junk.

Initially, incoming messages might not be accurately junked because you have not trained it enough.

  1. Check your Junk folder to see if any non-junk messages have been detected as junk, and mark non-junk ones as not junk.
  2. Bayesian filtering requires at least 100 bad messages be marked as spam and 100 good messages marked as not junk to function. To work best, it needs a few hundred of each marked.
    • If you mark a thousand spam messages but do not mark legitimate messages, or very few, it won't work well.
    • It's best to mark different types of messages good and bad - marking 500 messages from the same source is not as good as marking 500 messages from different people.

After an initial training period

  1. You should find that Junk Mail Controls are effectively detecting unwanted junk emails and keeping them from your Inbox. If it is not, see the tweaking information below.
  2. You should occasionally train non-junk messages - doing it when you mark bad messages as junk is a good habit.

There are various ways that you can mark messages:

  • Select a message and type "Shift+J" (for not Junk) or "J" (for Junk).
  • Right-click on a message and choose "Mark -> As Junk" (or "As Not Junk").
  • Select a message and from the "Message" menu, choose "Mark -> As Junk" (or "As Not Junk").
  • Select a message and click on the "Junk" icon on the toolbar.
  • Select a message and click on the "Junk Status" column in the message-list pane (which will show a small "Junk" icon if the message is marked as junk).

You don't need to keep the messages you have marked - marking stores the information it needs as tokens in the training.dat file. Training data for the Junk Mail Controls is stored in "training.dat" in your profile folder. You can view the Junk Mail Log by selecting "Junk Mail Log" in "Tools -> Junk Mail Controls -> Logging", and then use the mouse to 'grab' any corner and stretch it until it expands enough for you to see the log data.

Tweaking the Junk Mail Controls

mail.adaptivefilters.junk_threshold is the preference that determines at what "level" messages are classified as junk. It defaults to 90. Lowering this value will make it easier to recognize messages as spam, though it increases the risk that it will classify a legitimate message as spam. This might help get more messages marked; for example, messages that look like text but are actually clickable images. (Note however, adaptive processing does NOT examine images nor care about whether a message has an image - it is not part of the junk algorithm)

You can change the preference using Tools -> Options -> Advanced -> General -> Config Editor. Enter junk in the Filter field to show only the preferences that contain junk in their name, and then double-click on mail.adaptivefilters.junk_threshold, enter a value lower than the default 90 in the edit field and press the OK button. Many users report good results with values of 30 or lower.

JunQuilla is an add-on which can help you tweak junk processing and help you better understand how it is working. It provides a column that shows the odds that a message is spam, enhances the junk status column, and adds a Uncertain folder for messages requiring a users decision. It requires Thunderbird version 3 and newer.

Bayes Junk Tool can also be used to examine and modify the raw training data. Sometimes it helps to get rid of tokens that are just as likely to occur in spam and legitimate messages, especially if the training data file gets very large. The web site also has several sets of training data that you can import or merge with your existing training data.

Bayesian filters are useful, but they are not always the best tool. Sometimes checking whether the message was sent by somebody on a DNSBL list is more effective. See this article for how to integrate SpamPal and the junk mail controls, and control which messages are downloaded.

The FolderFlags extension can set various internal flags that Thunderbird uses to classify folders. If you set the "Junk" flag on a folder (other than the one spam is moved to) it won't scan that folder for spam.

The Delete Junk Context Menu extension adds "Delete Mail Marked as Junk" to a folders context menu. It can be configured to delete mail without moving it to using the Trash folder.

Any mention of "spam"

Thunderbird doesn't use or react to the word "spam" in its identification of junk mail, nor does Thunderbird put it in the subject.

If you see the "spam" in a subject or have a folder called "Spam", this is either due to your email provider or an add-on product (such as SpamPal). If it's due to your email provider, log into your webmail using a browser and browse its help to get more information. Your provider may support webmail commands to let you manage or disable whatever they're doing.

Use custom headers added by your email provider

Your email provider may run a spam filtering program on their mail server such as SpamAssassin, MailScanner, CRM114, SpamProbe, QSF or Bogofilter that analyzes each message and adds custom headers with information about its content. If they use SpamAssassin, see the next section for how to integrate it with Thunderbird's junk mail controls. Otherwise, use "View -> Message Source" and look for headers whose name begins with a 'X' and contain phrases such as Spam. They typically provide a spam score and/or keywords that you can test using message filters.

For example:

X-Spam-score: 1.5
X-Spam-hits: BAYES_60 1, HTML_IMAGE_ONLY_16 1.526, HTML_IMAGE_RATIO_08 0.001,
  HTML_MESSAGE 0.001, RCVD_IN_DNSWL_LOW -1, SPF_HELO_PASS -0.001,
  SPF_PASS -0.001, BAYES_USED global
X-Spam-source: IP='88.131.62.198', Host='mail.anp.se', Country='SE', FromHeader='com',
  MailFrom='se'

You could test whether X-Spam-score "is greater than" a certain value, whether X-Spam-source "doesn't contain" Country='US' (the example was from Sweden) or test whether X-Spam-hits "contains" certain keywords (they're the name of a test that increased the spam score) that you notice that the junk mail controls has problems recognizing spam with those attributes. Not every email provider will provide as much customization as the example, but it should at least have some sort of spam score you can test.

Trusting SpamAssassin and SpamPal

Spammers sometimes use Bayesian poisoning to degrade spam filters that use Bayesian filtering. SpamPal uses DNS Blacklists and SpamAssassin uses several methods (it's best known for its extensive testing of message headers) to filter spam so that type of attack has little or no effect on them. Both of them add special headers to a message to indicate whether it's spam. SpamPal is only available for Windows, but the Wine Application Database rates it as platinum (i.e. install and run flawlessly on an out-of-the-box Wine installation)

Tools -> Junk Mail Controls has a setting to tell Thunderbird to trust junk mail headers set by either SpamPal or SpamAssassin. The order of processing is:

  1. Message filters
  2. "Trust header"
  3. "Adaptive junk" (junk mail controls)

Some email providers customize the headers added by SpamAssassin, or modify the subject prefix. This can cause the junk mail controls to ignore the information. "Trust header" is actually a standard message filter, stored in a isp subdirectory in the Thunderbird program directory. Thunderbird checks whether either X-Spam-Status: or X-Spam-Flag: begins with Yes, or the subject begins with ***SPAM***. If you run into this problem backup the SpamAssassin.sfd file and then change what it tests for using a text editor (not a word processor). There is also a SpamPal.sfd file.

Some users have reported that the trust SpamAssassin option sometimes ignores the junk mail headers in Thunderbird 2.x. It's not clear whether you can work around this bug by disabling the option and adding the appropriate message filter.

If you use SpamAssassin you might want to uncheck "enable adaptive junk mail controls for this account" in Tools -> Account Settings -> Junk Settings. That controls the Bayesian filter in the junk mail controls, which is what it uses to learn how to recognize junk. SpamAssassin is normally configured with a Bayesian filter that is more sophisticated, has more data to learn from (it learns from all of your email providers messages, not just your messages), and is probably tweaked by the same admin who tweaks the SpamAssassin filters. Having two Bayesian filters for the same account makes things unnecessarily complex. For example, it's not defined what's supposed to happen if you mark a message as not junk, and later on the adaptive junk mail controls notices (again) that SpamAssassin says it's spam.

You could probably edit the SpamAssassin.sfd file in the isp subdirectory in the Thunderbird program directory to make the junk mail controls think another spam filter such as SpamBayes is actually SpamAssassin. The first step would be to figure out what the spam filter modifies in the message in order to identify whether it's spam by looking at a messages source.

SpamBayes

Thunderbird Junk mail controls do not support trusting SpamBayes. But the ThunderBayes add-on integrates SpamBayes with Thunderbird with a toolbar button to classify a message as "Spam" or "Ham", and a preferences page in the account settings.

Spamato

The junk mail controls don't support trusting Spamato, possibly because the Spamato instructions recommend disabling the junk mail controls. There is a Spamato forum, some documentation on the different types of spam filters they support and a FAQ on sourceforge.net. Spamato does not store its information in the Thunderbird profile, it creates its own profile

Image spam

One way to weed out image-based spam is to create a message filter and set it to match all of the following:

  • Content-Type contains multipart/related
  • From isn't in my address book (repeat this for each address book)

In "Perform these actions" add

  • "Set Junk Status" to "Junk"
  • "Move Message to" your junk mail folder.

This will mark the message as junk and move it to the Junk folder if the Content-Type header contains multipart/related and the sender wasn't in your address book. The message filters don't know how to recognize Content-Type headers, you will need to add it using the "customize..." option at the bottom of the leftmost list box. This method is rather heavy-handed. If your email provider runs a spam filter program such as SpamAssassin, it will typically do a much better job recognizing image spam.

Problems with junk processing

After classification filters

See Managing spam with “after classification” filters for how to use spam classification in a message filter.

Regular expressions - advanced

Neither the junk mail controls or the message filters support wild cards or regular expressions. There don't appear to be any extensions that add support for that. However, SpamPal (a mail classification program normally used for filtering spam) supports a RegExFilter Plugin that adds regular expression support based on Perl Regular Expressions. If you configure the junk mail controls to trust SpamPal you could use regular expressions to filter spam. [1]

There are many other SpamPal plugins available here. For example, you could extend white lists and black lists to apply to email addresses from any header, white list any message that contain words from a list of good words, filter on what web sites are mentioned, launch other programs (passing them information about the message as command line arguments) or run Ruby scripts. The main drawback is none of this is integrated into the junk mail controls - it just knows when SpamPal marks a message as spam.

Anti-phishing tools

Sender Policy Framework (SPF) is a method for verifying an email sender's domain name, to figure out whether an address is spoofed. The Sender Verification Extension makes use of SPF to verify the From address on emails, plus DNS black and white lists such as SURBL, Spamhaus, DNSWL, and Sender Score Certified to check the sender's reputation. When you open a message, the add-on adds a line at the top of message with the verification status of the sender. For example, "Reputable Sender", "This sender is a known malicious spammer or phisher. Discard this email", or "Sending domain does not support verification (address could be forged).". You have to actually read the message to get the warning (it is not meant to be used as a mail filter), and it's probably most useful as an alternative to Thunderbird's phishing protection (which most users disable due to its inability to learn and its many false positives).

Check the spam score of messages you sent

MailingCheck is a free Windows program that calculates the spam score of a message based on SpamAssassin rules. Its useful if you suspect that some of your recipients aren't getting your message due to it being treated as spam and you can't verify that by having them check in their junk mail folder. Port25 (copy and paste a message) , mail-tester (email them) and emailspamtest (copy and paste a message) are web sites that perform similar checks for a message.

You might want to also check if your mail server is black listed by checking it with a web site such as MXToolbox.

Junk filtering in newsgroups

Experimental support for junk filtering of newsgroups was added to Thunderbird 3.0. The JunQuilla add-on is needed to manage some features. See this blog post for more information.

See the webmail spam folder

A POP account is only aware of one folder stored on the POP server, the inbox. This is a limitation of the POP3 protocol. However, its possible for some POP3 servers to be configured to lie to the email client, and provide the contents of both the inbox and the spam folder used by webmail when the email client checks for new mail. This is totally transparent to the email client. Email providers that support this have a webmail settings to configure this.

IMAP accounts don't have this limitation. That can use the same folders as webmail.

Report Spam

Most add-ons to report spam seem to report it to little known organizations. The best seems to be the HabuL add-on that "Reports spam to the SpamCop, the FTC, FDA, SEC, ACMA and/or Knujon.com. It also allows you to put in your own custom addresses to report spam to such as your ISP or corporate abuse address."

If you are using a IMAP account some email providers let you specify (using webmail commands) that any messages in your junk folder should be used to help train a personal bayes database to recognize spam. This improves the spam filtering done by your email provider. Improved spam filtering with per-user bayes databases is one example.


See also

External links

Add-on to improve Thunderbird's junk processing

  • The JunQuilla add-on adds support for a column that shows the odds that a message is spam, enhances the junk status column, and adds a Uncertain folder for messages requiring a users decision. It works with version 3 and newer.
  • The Sender Verification Anti-Phishing add-on verifies the senders domain to help prevent phishing.
  • The Show Address extension adds columns to show the senders and recipient's email address (rather than the display name). This can be useful when trying to identify spam.
  • The Spamness add-on adds a color and size based display of how likely a message is spam based on its SpamAssassin score.

Add-ons interfacing to external services

  • The MailWasher Free proxy lets you view and bounce messages before downloading them. If you want to do that read this explanation first on why bouncing by end users will just send them to an innocent person who may report the bouncer for sending them spam.
  • SPAMfighter Standard is a free version of the SpamFighter spam filter that supports Thunderbird. It adds a SPAMfighter footer to your messages and has a limited number of blacklist/whitelist entries.
  • The Spamato4Thunderbird add-on adds support for the Spamato spam filters. It requires Java. Its an alternative to installing Spamato as a local proxy. Camels's Eye is another Java based spam filter It also provides message filtering. Unfortunately both of these applications haven't been updated for years and there isn't any support available.
  • The ThunderBayes add-on adds support for the SpamBayes spam filter. Its no longer supported by the author. ThunderbirdBayesPP is a more recent alternative implementation. Neither supports IMAP.