Thread: Extending the volume size of the data directory volume
Hi, as I am importing gigabytes of data and the space on the volume where the data dictionary resides just became to small during that process, I resized it dynamically (it is a LVM volume) according to this procedure: http://www.techrepublic.com/blog/opensource/how-to-use-logical-volume-manager-lvm-to-grow-etx4-file-systems-online/3016 Everything went without any problems and the import continued. Now, it is suddenly stuck (pgAdmin shows it as idle (piped connection)) and there is a good chance (as estimated from the space used) it just started using one of the added LE-Blocks (HDD space that was added to the volume). The db imported so far can be accessed just fine. So from the postmaster architecture, is there something that would explain this behaviour based on the hypothesis that newly added space was used? Any chance to revive the import somehow? Thanks -- View this message in context: http://postgresql.1045698.n5.nabble.com/Extending-the-volume-size-of-the-data-directory-volume-tp5030663p5030663.html Sent from the PostgreSQL - general mailing list archive at Nabble.com.
here's what I would do to analyze this first: - vmstat 1 - iostat -k -x 3 - look into system logs, maybe something actually happened there... - look at the process list. find if some of Pg processes are in D state - strace -f -v <PID of the "hanging" writer process> 2011/11/29 panam <panam@gmx.net>: > Hi, > > as I am importing gigabytes of data and the space on the volume where the > data dictionary resides just became to small during that process, I resized > it dynamically (it is a LVM volume) according to this procedure: > http://www.techrepublic.com/blog/opensource/how-to-use-logical-volume-manager-lvm-to-grow-etx4-file-systems-online/3016 > Everything went without any problems and the import continued. Now, it is > suddenly stuck (pgAdmin shows it as idle (piped connection)) and there is a > good chance (as estimated from the space used) it just started using one of > the added LE-Blocks (HDD space that was added to the volume). The db > imported so far can be accessed just fine. > So from the postmaster architecture, is there something that would explain > this behaviour based on the hypothesis that newly added space was used? Any > chance to revive the import somehow? > > Thanks > > -- > View this message in context: http://postgresql.1045698.n5.nabble.com/Extending-the-volume-size-of-the-data-directory-volume-tp5030663p5030663.html > Sent from the PostgreSQL - general mailing list archive at Nabble.com. > > -- > Sent via pgsql-general mailing list (pgsql-general@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-general
Hi Filip, Thanks for the suggestions. There is no indication of disk activity triggered by postgres. iostat reports: Cannot find disk data (maybe because the system is hosted in a OpenVZ environment?) Systemlogs (syslog and postgresql-9.1-main.log) do not indicate something unusual. All pg threads are sleeping (S state). I will try to reproduce this, this time with a smaller initial disk size... Regards panam -- View this message in context: http://postgresql.1045698.n5.nabble.com/Extending-the-volume-size-of-the-data-directory-volume-tp5030663p5034257.html Sent from the PostgreSQL - general mailing list archive at Nabble.com.
On Tue, Nov 29, 2011 at 6:32 PM, panam <panam@gmx.net> wrote: > Hi Filip, > > Thanks for the suggestions. > There is no indication of disk activity triggered by postgres. > iostat reports: Cannot find disk data (maybe because the system is hosted in > a OpenVZ environment?) > Systemlogs (syslog and postgresql-9.1-main.log) do not indicate something > unusual. > All pg threads are sleeping (S state). > > I will try to reproduce this, this time with a smaller initial disk size... Have you tried doing something like stopping postgres and rebooting the server in case there's some volume info that didn't get updated when you grew the partition bigger?
On 11/30/2011 09:32 AM, panam wrote: > Hi Filip, > > Thanks for the suggestions. > There is no indication of disk activity triggered by postgres. > iostat reports: Cannot find disk data (maybe because the system is hosted in > a OpenVZ environment?) > Systemlogs (syslog and postgresql-9.1-main.log) do not indicate something > unusual. > All pg threads are sleeping (S state). > > I will try to reproduce this, this time with a smaller initial disk size... > Try checking where the postgres processes are waiting, too: ps -C postgres -o wchan= -- Craig Ringer
Hi, output is -- View this message in context: http://postgresql.1045698.n5.nabble.com/Extending-the-volume-size-of-the-data-directory-volume-tp5030663p5034494.html Sent from the PostgreSQL - general mailing list archive at Nabble.com.
No, but will try this first, thanks for the suggestion. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Extending-the-volume-size-of-the-data-directory-volume-tp5030663p5034495.html Sent from the PostgreSQL - general mailing list archive at Nabble.com.
Had to restart the import. This time, I tried with a smaller initial disk size (1GB) and extended it dynamically. It did not cause any problems. A different reason might be, that I remounted the volume in between during the last update to deactivate buffer flushing. Maybe a bad combination. Let's see how far it gets this time... -- View this message in context: http://postgresql.1045698.n5.nabble.com/Extending-the-volume-size-of-the-data-directory-volume-tp5030663p5036767.html Sent from the PostgreSQL - general mailing list archive at Nabble.com.
On Wed, Nov 30, 2011 at 4:19 PM, panam <panam@gmx.net> wrote:
Had to restart the import. This time, I tried with a smaller initial disk
size (1GB) and extended it dynamically. It did not cause any problems.
A different reason might be, that I remounted the volume in between during
the last update to deactivate buffer flushing. Maybe a bad combination.
Let's see how far it gets this time...
If you remounted without restarting postgres, I would say "that's your problem"
--JT
--
View this message in context: http://postgresql.1045698.n5.nabble.com/Extending-the-volume-size-of-the-data-directory-volume-tp5030663p5036767.htmlSent from the PostgreSQL - general mailing list archive at Nabble.com.
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Hey, Got the problem again, this time, it is definitely not caused by changes to the filesystem. I think it is more likely because of postgres being run in a virtualized environment. This time, it hung some time after starting a manual analyze on the biggest table so far (55Gb, still growing). Now: Something unusual with this? Regards, panam -- View this message in context: http://postgresql.1045698.n5.nabble.com/Extending-the-volume-size-of-the-data-directory-volume-tp5030663p5135704.html Sent from the PostgreSQL - general mailing list archive at Nabble.com.