Thread: postgres.exe mem usage in task manager
I noticed a curious thing: the task manager does not seem to account for shared memory (memory mapped files) in the memory usage, making it appear that that the backends are allocating more memory than they really are. I confirmed that the total allocation for all the backends was reasonable by checking the system commit but if you add up allocations for each backend it appears much higher. checkpointing on win32 seems to run very fast...much faster than I'm used to, but this may be do to improvements in the bgwriter. It cleans the clock of the Interix port (MS's own unix runtime) by at least 15% on insert heavy situations. Overall, I am just blown away on how well the whole thing works compared to just a few weeks ago. Merlin
"Merlin Moncure" <merlin.moncure@rcsonline.com> writes:
> checkpointing on win32 seems to run very fast...much faster than I'm
> used to, but this may be do to improvements in the bgwriter.
Or it may be because no checkpoint actually happens :-(. How much
confidence have you got that Win32 and/or your disk drive isn't lying
about write-complete?
regards, tom lane
> I noticed a curious thing: the task manager does not seem to > account for shared memory (memory mapped files) in the memory > usage, making it appear that that the backends are allocating > more memory than they really are. Have you checked the "Virtual Memory Size" column? (Not enabled by default) > I confirmed that the total allocation for all the backends > was reasonable by checking the system commit but if you add > up allocations for each backend it appears much higher. If the above does not work, check with the performance monitor... Microsoft has changed the way task manager reports memory usage between pretty much every windows version (particularly the summary fields)... Which version are you on? > checkpointing on win32 seems to run very fast...much faster > than I'm used to, but this may be do to improvements in the > bgwriter. It cleans the clock of the Interix port (MS's own > unix runtime) by at least 15% on insert heavy situations. > Overall, I am just blown away on how well the whole thing > works compared to just a few weeks ago. Checkpointing is a no-op on win32. sync() does nothing. (OK, not quite no-op, it writes a checkpoint record.. But without checkpointing, that is actually incorrect). This is the item at the very top of the TODO list on Bruces win32 status page (http://momjian.postgresql.org/main/writings/pgsql/win32.html). Tom has offered to change how sync() is handled in general, which is pretty much needed before we can fix this on Win32. //Magnus
Magnus Hagander wrote: > Have you checked the "Virtual Memory Size" column? (Not enabled by > default) yes. Virtual memory is 0 or near 0, which is correct. Otherwise we really would be double dipping into memory (even if not necessarily physical memory). Memory is doled out to each process from the MMF, which does not belong to any process. The virtual memory allocation is in fact somewhere; I have no idea how to view it from the task manager: I deliberately hacked it down to 128M, after which the computer alerted me and bumped it up about when you'd expect. (aside: why do they let you set it if the computer always overrides you?) > Microsoft has changed the way task manager reports memory usage between > pretty much every windows version (particularly the summary fields)... > Which version are you on? XP SP1. > Checkpointing is a no-op on win32. sync() does nothing. (OK, not quite > no-op, it writes a checkpoint record.. But without checkpointing, that > is actually incorrect). Well, my observation was based on an apples to apples comparison with the Interix version which has no sync() either...but WAL, etc. are not all fsynced(), correct? My understanding is that with recent versions of pg that sync plays a less important role. Checkpointing does do something; the writer process grinds away for a few seconds, whereas on Interix it would take 5-10 seconds. Internal improvements since 7.4.1 may be skewing the results, however. The upshot is everything appears reliable and quick, even though current syncing can not be relied on. I just can't seem to figure out how to measure server memory consumption in a useful way :( Interix was much better in this regard...there you would allocate x megabytes to the server and you could watch them compete for memory resources. Also worth noting that the raw brute insert performance with fsync = off is about 30% faster than Interix and about 15% faster than cygwin...this pleases me greatly! Merlin
>> Have you checked the "Virtual Memory Size" column? (Not enabled by >> default) >yes. Virtual memory is 0 or near 0, which is correct. Otherwise we >really would be double dipping into memory (even if not necessarily >physical memory). Memory is doled out to each process from the MMF, >which does not belong to any process. > >The virtual memory allocation is in fact somewhere; I have no idea how >to view it from the task manager: I deliberately hacked it >down to 128M, >after which the computer alerted me and bumped it up about when you'd >expect. (aside: why do they let you set it if the computer always >overrides you?) The column "VM Size" is *zero*? (Note - this is totally unrelated to the Windows settings of Virtual Memory. Once again, same name, different things. This is the on you get by View->Select Columns in task manager). The only process supposed to have that one zero is the "System Idle Process", which has no memory allocagted at all. In my installation, I certainly have growing numbers there. Also, when increasing the shared memory a lot, the Mem Usage of postmaster goes up, but not of the other processes. If you use the performance monitor, you will see a large value for "Virtual Bytes" that corresponds to the shared buffers + everything else. A couple of notes about what counters are what: "Mem Usage" = Physical memory. "VM Size" = Private committed (non-shared) memory. "Virtual Bytes" = committed + reserved virtual space, including shared "Working Set" = physical working set size "Private Bytes" = private committed memory (again) >> Microsoft has changed the way task manager reports memory usage >between >> pretty much every windows version (particularly the summary >fields)... >> Which version are you on? > >XP SP1. Ok. Same as me then >> Checkpointing is a no-op on win32. sync() does nothing. (OK, >not quite >> no-op, it writes a checkpoint record.. But without >checkpointing, that >> is actually incorrect). > >Well, my observation was based on an apples to apples comparison with >the Interix version which has no sync() either...but WAL, etc. are not >all fsynced(), correct? Oh, ok. Yes, it should be. >The upshot is everything appears reliable and quick, even >though current >syncing can not be relied on. I just can't seem to figure out how to >measure server memory consumption in a useful way :( Interix was much >better in this regard...there you would allocate x megabytes to the >server and you could watch them compete for memory resources. Well, see above. The system wide counter to look at is the Total Commit Charge. And of course the usual suspects of hard page faults, but that's just to determine if you are already swapping. >Also worth noting that the raw brute insert performance with >fsync = off >is about 30% faster than Interix and about 15% faster than >cygwin...this >pleases me greatly! That's great to hear. I expected it to be faster, but not quite so much. //Magnus