Re: Help with extracting large volumes of records across related tables - Mailing list pgsql-performance

From Pierre-Frédéric Caillaud
Subject Re: Help with extracting large volumes of records across related tables
Date
Msg-id opsd9qi9udcq72hf@musicbox
Whole thread Raw
In response to Re: Help with extracting large volumes of records across related tables  ("Damien Dougan" <damien.dougan@mobilecohesion.com>)
List pgsql-performance
    Thanks for the thanks !

    Generally, when grouping stuff together, it is a good idea to have two
sorted lists, and to scan them simultaneously. I have already used this
solution several times outside of Postgres, and it worked very well (it
was with Berkeley DB and there were 3 lists to scan in order). The fact
that Python can very easily virtualize these lists using generators makes
it possible to do it without consuming too much memory.

> Pierre-Frederic, Paul,
>
> Thanks for your fast response (especially for the python code and
> performance figure) - I'll chase this up as a solution - looks most
> promising!
>
> Cheers,
>
> Damien
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
>                http://archives.postgresql.org
>



pgsql-performance by date:

Previous
From: "Damien Dougan"
Date:
Subject: Re: Help with extracting large volumes of records across related tables
Next
From: Bruce Momjian
Date:
Subject: Re: Determine optimal fdatasync/fsync, O_SYNC/O_DSYNC options