NOTE: I do not want this to become an OS fanboy flame war. I'm only interested in explaining the behavior of win7 and understanding a bit more about swap files, and what causes something to get paged out.


This has bugged me for years. I have a laptop which is pretty fast except for one thing - the hard disk which is 5400rpm. Over the last few days I've been getting more and more annoyed at how VS is constantly paging out. It seemed that every operation I did meant waiting 5 seconds for the hard disk to do something. For example, if I switch to chrome to post in a forum and then post back, VS has been paged out, and when I start typing I have to wait 5 seconds for something to page back in, meanwhile the UI thread is blocked.

According to resource monitor, 1.5gb out of my 2gb of physical memory was used, which means that there was more than enough memory than was required. I was curious. What was actually in virtual memory? So I cut down the cache file size to 512mb. My swap file had set itself to 3gb, which astonishes me - this means that at high load, 60% of all memory that is in use is being stored on the HDD. Might this explain the problem, and if so, why is it so huge?

The paging out has stopped. I can have an 8 project solution open in VS, the inline compiler is now almost instant, I can also have about 10 tabs in chrome, and a few other things, and VS rarely pages out. There is absolutely no doubt in my mind that everything has become faster - most telling, I do not hear the hard disk anywhere near as often as I did before. The start menu opens instantly even if I didn't use it for 20 minutes, applications launch fast, and chrome does not cause any hard disk access as I switch tabs.

What is going on? This behavior makes no sense. I am pretty certain that the problem is with windows and not with the set of all applications I use, because the change to the swap file size has improved everything at the same time, and previously, everything was in some way or other hanging due to waiting for hdd access.

This sums things up pretty well and explains why by default your cache is so large.
http://serverfault.com/questions/23621/any-benefit-or-detriment-from-removing-a-pagefile-on-an-8gb-ram-machine/23684#23684

That is pretty perplexing. Off hand the only thing I can think of that might cause something like that would be bad disk sectors within the swap file, and by changing the file size Windows is no longer trying to read / write to those bad sectors. It's a bit of a stretch though, I imagine you'd notice this in other ways. Maybe the swap file was fragmented and shrinking it reduced fragmentation? I have serious doubts that would impact performance notably though (for one thing, the swap file shouldn't be fragmented, and secondly, I'm pretty sure NTFS does a better job at dealing with this than people give it credit).

I'm also a bit skeptical that Windows 7, Chrome and Visual Studio can coexist happily in only 1.5GB of memory. Perhaps Windows 7 is better at memory management than I thought (the whole prefetching and caching thing introduced in Vista), but I'm using 2.1GB right now with nothing but IE9 open. Perhaps Windows was paging out most / all of Visual Studio's memory pages when you tabbed out. That might explain using so relatively little memory, but it certainly doesn't explain why reducing the swap file to 512MB improved performance.

For what it's worth, I think Windows actually sets 1.5 or 2 x physical memory as the default swap file size, which would explain why the page file was 3GB initially.

Ram is incredibly cheap these days, and it's one of the best upgrades you can do for your system if you have two or less GB or memory in your computer. I consider 4GB to be a minimum for development machines. That said, windows 7 runs surprisingly well with limited amounts of ram.

According to resource monitor, 1.5gb out of my 2gb of physical memory was used...

I also think Win is a bit too conservative sometimes. After all, if disk access goes on so frequently, what's the point of leaving the memory here? While I understand the interactions between pre-emptiveness and the need to eventually deal with allocations, I am somewhat uncertain this has to sum to a whopping 25% of the pool.

I'm also a bit skeptical that Windows 7, Chrome and Visual Studio can coexist happily in only 1.5GB of memory.

Seconded. I'm fairly surprised to be honest.

[quote name='speciesUnknown' timestamp='1314412867' post='4854292']According to resource monitor, 1.5gb out of my 2gb of physical memory was used...

I also think Win is a bit too conservative sometimes. After all, if disk access goes on so frequently, what's the point of leaving the memory here? While I understand the interactions between pre-emptiveness and the need to eventually deal with allocations, I am somewhat uncertain this has to sum to a whopping 25% of the pool.

I'm also a bit skeptical that Windows 7, Chrome and Visual Studio can coexist happily in only 1.5GB of memory.

Seconded. I'm fairly surprised to be honest.
[/quote]

Right now, I have the following things open:


4 chrome tabs: one grooveshark tab (playing), two gdnet tabs, and a page describing page file usage. (Maybe windows knows this and is playing nice... ITS A TRAP)
2 VS instances, each with a .sln loaded, the first with 10 .csproj and the other with 4 .csproj.
Steam, which seems to be happy with "only" 72mb
a calculator instance
windows classic interface (no aero glass, I don't need that shiny bullshit)
Google talk, synergy (no connections)

The total usage according to resmon.exe is 25mb reserved, 1193 in use, 94 modified, 497 standby, and 239 free. It reports that this is 58% of physical memory used overall.

What does occur to me is that I appear to have crossed somethreshold here - no longer can windows keep a copy of everything in physical memory also in a swap file - this is impossible, as it only has 512 swap space and yet ive used 1193mb of physical memory.

BTW, I think I should give a better description of what I mean by "faster".

Since I manually set my swap size to 512, things have stopped paging out, it happens almost never now - i can switch between applications, keep one open for 10 minutes, and so on, and VS in particular has stopped paging out. However, I can see in resource monitor that it is regularly throwing away cached items in favor of in use items. I can also see that some operations are actually now slower, as if they are regenerating data rather than getting it out of cache.

The overall effect is that I wait 1 second for some operations when I used to wait 0 seconds, but I do not wait 5 seconds (it was sometimes longer) for anything.

I also noticed something disturbing - I copied two folders, totalling 26mb, and explorer.exe allocated for itself 26mb of cache (at least I think so, as "standby" bar in resmon increased by approx 26 and "free" decreased by approx 26)


[addendum]
Another thing has started to happen as well - windows is now complaining that it is running out of memory who'd have thought THAT would happen

BTW, I think I should give a better description of what I mean by "faster".

Since I manually set my swap size to 512, things have stopped paging out, it happens almost never now - i can switch between applications, keep one open for 10 minutes, and so on, and VS in particular has stopped paging out. However, I can see in resource monitor that it is regularly throwing away cached items in favor of in use items. I can also see that some operations are actually now slower, as if they are regenerating data rather than getting it out of cache.

The overall effect is that I wait 1 second for some operations when I used to wait 0 seconds, but I do not wait 5 seconds (it was sometimes longer) for anything.

I also noticed something disturbing - I copied two folders, totalling 26mb, and explorer.exe allocated for itself 26mb of cache (at least I think so, as "standby" bar in resmon increased by approx 26 and "free" decreased by approx 26)


[addendum]
Another thing has started to happen as well - windows is now complaining that it is running out of memory who'd have thought THAT would happen

I've been experimenting with this over the last few days and I think I've got to the root of the problem. It concerns the way my hard disk performs - that is, incredibly badly. In the past I've complained that hybrid sleep takes far too long to wake from, in fact it takes longer than a full reboot.

I have been messing with the swap size, and have found that I get the performance boost i have noted provided the swap size is less than the amount of physical ram that I have, so now I am running a 1gb swap size. I occasionally get I seem to be triggering something in the logic windows uses to decide what to cache and what to throw away - if it has more swap than physical ram, it will seriously overuse the swap, and this is normally OK, based on some inaccurate heuristic about hard disk performance. It is at present unable to keep a copy of every single page thats in physical memory also in swap, so it is prioritising what gets kept in swap. Something important is of a LOW priority in terms of swap space, but is of a HIGH priority in physical ram, and so it is not being written to disk, and therefore, never thrown away in physical ram.

Your best bet is to buy more RAM then turn off the page file. It's a worthless ancient feature in operating systems that people falsely think windows handles correctly. It doesn't and virtual memory for all intents and purposes isn't needed nowadays. I have 16 GB in my laptop and with From Dust (random game on Steam) and tons of other programs open I'm at 3.57 GB. Windows 7 likes to shift HD stuff into RAM if you have a lot of it making things faster.

You can find some guides online for instance with FireFox to turn off HD usage such that it never uses the HD. This is useful if you have a SSD. Once in a while I'll let my computer idle and check which programs are using the HD. Normally it's just steam which has a flaw or something in it where it likes to write to the HD randomly.

Also if no one knows what species is talking about take out some stick from your computer and run with like 1 or 2 GB then open a lot of programs. VS and Adobe products page themselves out within a few minutes and there's this horrible lag when you try to open them. Happened on my old laptop with Windows 7 all the time.

This sums things up pretty well and explains why by default your cache is so large.
http://serverfault.com/questions/23621/any-benefit-or-detriment-from-removing-a-pagefile-on-an-8gb-ram-machine/23684#23684

The top rated comment was voted up by people don't understand how virtual memory functions and why in the real world the Windows 7 paging system isn't very good at figuring out what to page out and what to keep in memory. It lacks the foresight to know it's paged to much out then freaks out when it needs it all paged back in. This just ends up lagging for most people especially those with laptop mechanical drives.

Your best bet is to buy more RAM then turn off the page file. It's a worthless ancient feature in operating systems that people falsely think windows handles correctly. It doesn't and virtual memory for all intents and purposes isn't needed nowadays. I have 16 GB in my laptop and with From Dust (random game on Steam) and tons of other programs open I'm at 3.57 GB. Windows 7 likes to shift HD stuff into RAM if you have a lot of it making things faster.

My own experience doesn't really back this up. I often run lots of memory intensive programs such as Sibelius (with additional samplers all with huge things stored in RAM), Eclipse etc. all while rendering some huge thing in Blender. I have 4 GB of RAM, and things do page in and out but I rarely notice any lag. If on the other hand I disable virtual memory I'll start to get warning messages very quickly followed shortly by programs dying in horrible ways. Windows virtual memory is certainly a bit idiosyncratic and I think with some configurations it does more harm than good, but I'm not convinced that saying it's worthless or not correct is really accurate.