Re: memory usage of pg_upgrade - Mailing list pgsql-hackers

From Andrew Dunstan
Subject Re: memory usage of pg_upgrade
Date
Msg-id 522E4E2B.2030209@dunslane.net
Whole thread Raw
In response to memory usage of pg_upgrade  (Jeff Janes <jeff.janes@gmail.com>)
Responses Re: memory usage of pg_upgrade
List pgsql-hackers
On 09/09/2013 06:20 PM, Jeff Janes wrote:
> pg_upgrade reserves 5 times MAXPGPATH, or 5120 characters, for the
> tablespace name of every object (table, toast table, index) in the
> database being upgraded.  This adds up pretty quickly when there is a
> very large number of objects.  It could be changed to char* to a
> separately allocated name that takes only as much space it needs.  But
> maybe it would be better to point into os_info.old_tablespaces or
> something like that, as surely there are not going to be one
> independent file space per object.
>
>
> typedef struct
> {
>       ...
>      char        tablespace[MAXPGPATH];
> } RelInfo;
>
> The struct FileNameMap has 4 more .
>
> Since there seems to be some interest in improving the scalability of
> pg_upgrade, this is one of the things to consider fixing.  What is the
> best way to do it?


Send in a patch :-)

We recently ripped out some uses of statically sized strings in the 
parallel code and replaced them with pointers to palloc'ed strings. So 
there is good precedent for this. See 
<https://github.com/postgres/postgres/commit/910d3a458c15c1b4cc518ba480be2f712f42f179>

In the case of tablespaces, I should have thought you could keep a hash 
table of the names and just store an entry id in the table structure. 
But that's just my speculation without actually looking at the code, so 
don't take my word for it :-)

cheers

andrew




pgsql-hackers by date:

Previous
From: "MauMau"
Date:
Subject: Re: [bug fix] strerror() returns ??? in a UTF-8/C database with LC_MESSAGES=non-ASCII
Next
From: Tom Lane
Date:
Subject: Re: lcr v5 - introduction of InvalidCommandId