Thread: Memory Context Info dump
In our customer environment as well as during development, we have observed that many time we need to get details of memory used by each contexts in order to analyze the memory consumption/leak.
So I would like to propose one contrib function interface, which will dump the whole memory context into a file.
One of the major infrastructure i.e. the function MemoryContextStats (which dump all memory context details on the console) is already there in PG but it is being automatically called for scenarios
such as “system is out of memory”.
So Now:
1. Create one contrib function dump_memctxt_info, which is when executed, it will call the existing function MemoryContextStats.
2. The file to dump will be created in the following format:
context_<process id>_<timestamp>.dump
3. The file handler will be passed to MemoryContextStats.
4. If the parameter to MemoryContextStats is NULL (All existing calls will pass NULL), then it will behave as earlier i.e. dump on console.
5. The output format of the context will be same as earlier.
6. One sample file is attached.
Use-case:
In order to check the memory leak, this contrib function can be called before and after execution of query. Then comparison of these two dump will be helpful to further analyze.
Attached is the patch. Please provide your opinion. If it is OK, I will add it to next commitFest.
Thanks and Regards,
Kumar Rajeev Rastogi
Attachment
Rajeev rastogi <rajeev.rastogi@huawei.com> writes:
> In our customer environment as well as during development, we have observed that many time we need to get details of
memoryused by each contexts in order to analyze the memory consumption/leak.
> So I would like to propose one contrib function interface, which will dump the whole memory context into a file.
Under what circumstances would you invoke this? Not when you were already
out of memory.
> Use-case:
> In order to check the memory leak, this contrib function can be called before and after execution of query. Then
comparisonof these two dump will be helpful to further analyze.
That's pretty uncompelling, considering that intra-query memory leaks are
one of the major concerns. The evidence would be gone before you could
capture it.
> 1. Create one contrib function dump_memctxt_info, which is when executed, it will call the existing function
MemoryContextStats.
> 2. The file to dump will be created in the following format:
> context_<process id>_<timestamp>.dump
Dumping to a file seems like probably the second least friendly API you
could devise ... although I'm not sure how to do better offhand. The
existing code is made to not need to allocate any more memory, and
anything that could return data to SQL would have to do that.
I don't disagree that it would be useful to have better info available in
this area, but this patch seems like a quick hack rather than a useful
tool.
As an example of potentially-more-useful aids, I'm wondering about
tracking the high-water memory consumption of each memory context.
(This probably wouldn't be terribly expensive if it were done at the
granularity of malloc requests rather than individual pallocs.)
Then perhaps something to log a context's peak usage at context
destruction time, if it exceeds some threshold or other.
Our past discussions about tracking total usage in a context tree
would be relevant here too.
regards, tom lane
On Tue, Sep 8, 2015 at 4:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > As an example of potentially-more-useful aids, I'm wondering about > tracking the high-water memory consumption of each memory context. > (This probably wouldn't be terribly expensive if it were done at the > granularity of malloc requests rather than individual pallocs.) > Then perhaps something to log a context's peak usage at context > destruction time, if it exceeds some threshold or other. What I've been itching for in my testing is a way to save the memory context stats, and then later print the difference between them. That would let me see where memory was actually going to. There are some cases where that could be tricky. If a context is destroyed and a new one by the same name is created is it the same context or a different one? If there are a bunch with the same name and later there are a bunch but some are the same context and some are new do we try to match them up, maybe sort by size? But it would be nice to be able to see right away what context the extra memory was allocated in between two points. -- greg