Compacting folders

From MozillaZine Knowledge Base
Revision as of 08:06, 12 March 2010 by American Finn (talk | contribs) (repeat for each account)
Jump to navigationJump to search
This article was written for Thunderbird but also applies to Mozilla Suite / SeaMonkey (though some menu sequences may differ).

When you delete messages in an email program ("email client") or move them to another folder in the program, they are not yet physically removed - even emptying the Trash does not remove them. Instead, Thunderbird and other email programs simply hide the "deleted" messages and mark them as ready for physical removal. The process of physically removing such no longer visible messages is called "compacting". This means that messages that you think you have deleted are not actually physically removed until you manually or automatically compact the folder they were (are) in.

If you don't compact your mail folders periodically, they can grow very large and make Thunderbird slow to a crawl. If you wait too long before compacting, you can also lose the entire contents of one or more folders. The reason many users have never heard of compacting is that most email programs default to automatically compacting the folder whenever a certain amount of space is wasted, whereas the user has to enable this feature in Thunderbird.

Compacting shouldn't be confused with compressing or zipping a file. Compressing/zipping only changes how data is stored; compacting instead changes the data itself.

If you suspect a folder is badly corrupted, don't compact it. Instead, try to fix it by following the instructions in compacting does not seem to work.

How to compact folders

It is best not to have any defragmentation or backup program running automatically or manually on your computer while Thunderbird is compacting because this can cause incorrect backups, folder corruption, and data loss.

Automatically

The easiest way to compact is to let Thunderbird do it automatically: "Tools -> Options -> Advanced -> Network & Disk Space -> Disk Space -> Compact folder when it will save over 100 kB -> OK."

However, it is best to not to do anything in Thunderbird except read messages if you notice that compacting has started. If you tag, mark, or move messages during compacting, this can cause folder corruption and data loss. In addition, if you are in the process of writing an email while compacting starts, you may get an error when you try to save or send it. (This can be especially annoying if you're responding to an email with interleaved replies because copy-pasting to a new message will often move quotation marks into the middle of lines in quoted sections).

If you often delete many messages in very large folders and feel that Thunderbird becomes unresponsive while trying to delete more messages, you may want to try using a larger figure. This will reduce the chance that compacting starts and locks up the folder while you are trying to delete more messages. If you use a figure much larger than 1000 kB, compacting will take much longer and there is a greater risk of folder corruption.

If you're using an IMAP account and you believe that the performance gains due to compacting are a poor trade off you can configure Thunderbird 3 to auto-expunge messages instead. This will automatically expunge (physically delete) the deleted messages when you have 20 deleted messages waiting to be expunged. See Deleting messages in IMAP accounts if you want to do that. If you're using offline folders and configured to AutoSync (Thunderbird 3 does this by default) it will expunge the message in both the remote folder and the offline copy of the folder (a mbox file).

Manually

If you are annoyed by Thunderbird suddenly becoming unresponsive while using it, you may want to turn off automatic compacting and instead compact manually when you don't need to use Thunderbird. If you want to combine the advantages of both methods by making Thunderbird ask for permission before it starts compacting automatically, see customizing compacting below.

To compact all folders in an account manually, click the account on the left, and then click "File -> Compact Folders". This has to be repeated for each account. Compacting an account may take from a few seconds to 10 minutes or more, depending on how much mail you have and how recently you last compacted the folders. If you have trouble doing this and the process stalls, try compacting one folder at a time by right-clicking on the folder and choosing "Compact This Folder". If you do not let Thunderbird compact automatically, you should do this regularly, at least once a week. If you do it daily, it will take only a fraction of the time.

Offline

Most people never have a problem compacting a folder when Thunderbird is online. However, if you get Nstmp folders, that's due to the compacting being interrupted while Thunderbird is downloading new messages. You can avoid this by going offline before compacting (go to "File -> Offline -> Work Offline", or simply click on the icon in the bottom left corner.) If you have an IMAP account, you can only compact it while you are online. However, this normally doesn't cause a problem since IMAP accounts only download the headers to the hard disk. (If you don't know what kind of account you have, it's probably a POP3 account, which can be compacted offline.)

Customizing compacting

If you are an experienced computer user, you can use the Config Editor to combine the advantages of automatic and manual compacting by making Thunderbird ask for permission before it starts compacting automatically. In Config Editor, type "mail.purge.ask" (without quotes) and double-click to make its value "true". If there is no mail.purge.ask entry, right-click on any entry in the list of preferences, select New and Boolean, type "mail.purge.ask" (without quotes), and then choose "true".

Other ways to compact folders

The Xpunge extension lets you add a button to empty the trash and compact the folders in multiple accounts (in one step). If you use the beta version you can set a timer to automate that.

The Mozilla Add-ons web site has several other extensions that add buttons to compact folders.

If you have an IMAP account there are two ways to automate compacting its folders.

  • If you check Tools -> Account Settings -> Server Settings -> Clean up ("Expunge") Inbox on Exit it will compact the Inbox folder when you exit Thunderbird. If you check "Empty Trash on Exit" it will also empty and compact the Trash folder when you exit.
  • If Tools -> Account Settings -> Server Settings -> "When I delete a message" is set to "Move it to the Trash folder" and you set mail.imap.expunge_after_delete to true using the Config editor Thunderbird will compact the folder immediately after you delete or move a message. Note: Thunderbird ignores the modified mail.imap.expunge_after_delete setting until after you have exited and restarted Thunderbird.

Undoing compacting

You cannot undo compacting, in other words there is no such thing as "uncompacting". Compacting physically removes messages marked for deletion and hidden from view. Once they're gone, they're gone. Don't confuse compacting a folder with compressing or zipping it.

Usually somebody asks how to "uncompact" a folder when they have lost messages due to compacting. You shouldn't lose messages when compacting unless a folder was badly corrupted, in which case you'd probably have already noticed problems such as message fragments, several messages with a date of 1969 (due to the Date: header in a message being deleted), "resurrected" deleted messages, or long startup times because Thunderbird gets caught in a loop parsing (analysing) the inbox.

If you suspect a folder is badly corrupted, don't compact it. It might make things worse because compacting depends upon Thunderbird being able to parse the mbox file (the text file named after the folder that contains all of the messages for that folder) and figure out where every message begins and ends. Thunderbird can't use the so-called index file (folder_name.msf) to do this. Once Thunderbird starts to lose track of where each message begins and ends, compacting can cause the removal of other messages besides those marked ready for physical removal. It's unusual but possible to lose every message in a folder by compacting it if the folder was badly corrupted.

If you suspect a folder is badly corrupted, try to fix it by following the instructions in compacting does not seem to work instead.

Problems when compacting

Compacting seems to take forever

Usually it only takes a couple of minutes to compact your folders unless your profile is over a gigabyte in size or your computer is slow. The worst that normally happens if you exit Thunderbird before it finishes compacting is that it fails to delete some nstmp temporary files that it created and you have to manually delete them.

If compacting takes a long time, the status bar at the bottom of the window may not remove the text about it compacting a folder when it finishes. It may also fail to update that text to mention the last folder in that account if you use "File -> Compact Folders". If you do something else that normally displays some text in the status bar, it should replace the message about compacting.

Compacting does not seem to work

If compacting folders doesn't seem to work even for a single folder (where you right-click on the folder and choose "Compact This Folder"), it is possible that the folders and/or the .msf mail summary files are corrupted.

Quick and dirty fix

If the corruption is mild you frequently can fix it by deleting the .msf files for the corrupt folders. There are two ways you can do this.

1. Delete the .msf file(s)

  • Exit Thunderbird.
  • Find your profile. If you can not see it read this article .
  • Make a temporary backup of it.
  • Delete all of the files ending in .msf in your profile. Thunderbird will recreate them when it starts.

2. Rebuild the index file within Thunderbird

Right click on each corrupt folder in Thunderbird, select Properties, and then press the Rebuild Index button. Its quicker/simpler if you only have a problem with a few folders, such as the Inbox.

Real fix

If the corruption is severe, deleting the .msf files won't help much, and compacting the folder may just make it worse. You'll typically run into this only with the Inbox folder. It's much more vulnerable to corruption because many users tend to store lots of messages in it, and they also frequently delete messages in it. That's why it's recommended that you don't permanently store messages in your Inbox folder and instead move them to other folders. You can fix the problem by replacing the corrupt folder with a new known good folder into which you copy the messages from the corrupt folder.

  1. Exit Thunderbird and make a temporary backup copy of your profile folder so that you can revert everything easily if necessary.
  2. Start Thunderbird, create a new folder and name it "TestFolder" (or similar).
  3. In the folder that is giving you problems (e.g. Inbox), select all the messages (highlight one and then press Ctrl+a) and copy them to TestFolder ("Message -> Copy" -> [account name] -> TestFolder"). In extreme cases (if Thunderbird and/or the computer become extremely sluggish), you may have to choose only a few emails at a time (choose one and then press Shift+Page Down or Page Up once or twice).
  4. Verify that all of the messages have been copied to TestFolder.
  5. Right-click on TestFolder and choose "Compact This Folder".
  6. Go to your profile folder and see if TestFolder looks like it has been compacted. It should be significantly smaller in size than the folder from which you copied the messages.
  7. If it looks like TestFolder can in fact be compacted successfully, exit Thunderbird, and go to your profile folder. Then:
  • Rename the file that was giving you problems (e.g. rename "Inbox" to "InboxOLD") and delete its .msf file (e.g. "Inbox.msf").
  • Rename the "TestFolder" folder to "Inbox" (or whatever the problematic folder was called) and delete "TestFolder.msf".
  • Restart Thunderbird, and use it as usual. If compacting folders seems to work correctly during a test period (such as one week) then you can delete the folder "InboxOLD".

What is compacting?

Email programs do not immediately physically remove messages deleted by users and instead only hide them from view because this improves performance when the users delete messages in large folders. This is because email messages are not separate files; instead, they are consecutive parts of one large file (one file per folder in Thunderbird, one file per entire message store in some other programs). If email programs immediately physically removed messages deleted by the user, they would have to rewrite the entire contents of a folder every time the user deletes a single message.

When a user deletes 10 messages one at a time in a 100MB folder, Thunderbird can quickly hide them by simply modifying a status header in each message, and Thunderbird doesn't need to rewrite the entire folder until later, when the user compacts it (or lets Thunderbird do it automatically). If the user had to wait while Thunderbird rewrote the entire contents of the folder every single time the user deleted a message, the user would have to wait for a total of 10 times as long as it takes to rewrite 100MB. Compacting is a good performance tradeoff for large folders, but for simplicity you can't choose which folders have to get compacted.

If a folder has a lot of messages, and if messages are frequently deleted in it, and if you don't compact it frequently, this greatly increases the chance of the folder becoming corrupted. This creates message fragments, and may cause erratic behavior as Thunderbird tries to parse (analyse) the file containing the messages for that folder. Usually this is only a problem for the inbox folder. That is why the keep it working article recommends that you keep the inbox folder as empty as possible (by moving to another folder any messages you want to keep) and configure Thunderbird to automatically compact folders.

What happens if you do not compact folders?

If you rarely or never compact folders, your mail files will accumulate more and more of the "hidden" messages that have been marked for permanent deletion but have not yet been removed. This can cause a lot of disk space to be used, and it can have a negative effect on Thunderbird’s performance.

  • Even if a mail folder seems to be empty or nearly empty, the mail file can become very large [1][2][3]. This wastes disk space, and when you back up your mail files for safekeeping, you will waste time backing up all these "hidden" messages as well.
  • When downloading messages, you might occasionally get duplicates of messages you’ve already received [4].
  • Messages that you have deleted or moved to other folders may unexpectedly reappear in their original folder [5][6].
  • Your anti-virus software might detect infected messages that you long ago deleted, even if you emptied the Trash [7].
  • It could cause problems when you try to defragment your hard disk [8].
  • Your Inbox might stay blank for minutes.
  • The new message count could become much larger than the actual number of new messages. A quick fix is to delete the .msf mail summary file for that folder—Thunderbird will create a new one the next time you run it—but this will not work if the folder is badly corrupted.
  • Deleted messages might not get moved to the Trash folder. [9]

Deleting attachments

Deleting an attachment doesn't immediately decrease the size of the mbox file used to store the messages for that folder. In fact it increases it.

When you delete an attachment it isn't physically deleted. Thunderbird marks the original message as deleted and creates a copy without the attachment. The copy doesn't have the attachment but has headers recording the fact that there used to be an attachment. This extra copy is what temporarily increases the size. You only see the copy because messages marked for deletion are automatically hidden from view.

When you compact the folder the original message is physically deleted, and only the copy without the attachment is left. This reduces the size of the mbox file.

See also

External links