From MozillaZine Knowledge Base

If you're a Windows user and launching Firefox, Thunderbird, Mozilla Suite (or any application at all) changes your screen resolution (making everything big), the likely cause is the settings on the shortcut you're using to launch that application.

1. Close the application
2. Right click on the shortcut you use to start the application
3. Properties
4. Compatibility
5. Uncheck "Run in 640 x 480 screen resolution" (note that other checked options in this tab can also cause problems - "Compatibility mode" can cause hangs, "Run in 256 colors" can affect the colors of websites)
6. OK
7. Start the application

Your resolution should now not be changed.