Postgres Pro
Downloads
Home   >   mailing lists

Thread: Loading data from one table to another

Loading data from one table to another

From
Sabyasachi Mukherjee
Date:
Dear All,

We have a Postgres DB with 99 GB of data. The query performances are slow as expected. I am trying to introduce monthly partitions for this table.
To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
The DB is hosted in Windows server having 8 GB of RAM.

Note - we do not have any surrogate key in this table. And the event time recorded is also duplicated.

The command fails every time.
Can you please advise some strategy to get this done?

Once the pgadmin lost connection.
Another error that I have got is, 

ERROR:  Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
ERROR:  out of memory
SQL state: 53200
Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".

Thanks & regards

Sabyasachi Mukherjee


RE: Loading data from one table to another

From
Jean-Paul POZZI
Date:

Hello,

 

It seems that the command is too lonbg to execute from pgadmin, try to execute the same Sql from a terminal ... or split your work with "from date1 to date2", you can split the work in 2,3 or more tasks and it will succed.

 

Regards

 

JP P

 



De: "Sabyasachi Mukherjee" <mukherjee.sabyasachi@outlook.com>
À: "undefined" <pgsql-admin@lists.postgresql.org>
Envoyé: dimanche 13 octobre 2024 08:26
Objet: Loading data from one table to another

Dear All,
 
We have a Postgres DB with 99 GB of data. The query performances are slow as expected. I am trying to introduce monthly partitions for this table.
To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
The DB is hosted in Windows server having 8 GB of RAM.
 
Note - we do not have any surrogate key in this table. And the event time recorded is also duplicated.
 
The command fails every time.
Can you please advise some strategy to get this done?
 
Once the pgadmin lost connection.
Another error that I have got is, 
 
ERROR:  Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
ERROR:  out of memory
SQL state: 53200
Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".
 
Thanks & regards
 
Sabyasachi Mukherjee
 
 

RE: Loading data from one table to another

From
Holger Jakobs
Date:
Don't forget to look up the SQL state. There are some entries regarding it on stackoverflow

--
Holger Jakobs, Bergisch Gladbach
Tel. +49 178 9759012


Am 13. Oktober 2024 11:36:48 MESZ schrieb Jean-Paul POZZI <jp.pozzi@izzop.net>:

Hello,

 

It seems that the command is too lonbg to execute from pgadmin, try to execute the same Sql from a terminal ... or split your work with "from date1 to date2", you can split the work in 2,3 or more tasks and it will succed.

 

Regards

 

JP P

 



De: "Sabyasachi Mukherjee" <mukherjee.sabyasachi@outlook.com>
À: "undefined" <pgsql-admin@lists.postgresql.org>
Envoyé: dimanche 13 octobre 2024 08:26
Objet: Loading data from one table to another

Dear All,
 
We have a Postgres DB with 99 GB of data. The query performances are slow as expected. I am trying to introduce monthly partitions for this table.
To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
The DB is hosted in Windows server having 8 GB of RAM.
 
Note - we do not have any surrogate key in this table. And the event time recorded is also duplicated.
 
The command fails every time.
Can you please advise some strategy to get this done?
 
Once the pgadmin lost connection.
Another error that I have got is, 
 
ERROR:  Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
ERROR:  out of memory
SQL state: 53200
Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".
 
Thanks & regards
 
Sabyasachi Mukherjee
 
 

Re: Loading data from one table to another

From
Scott Ribe
Date:
Failing to allocate a memory segment of only 1MB seems suspicious. What are your hardware specs?

You might want to look at your memory & parallel query settings in postgresql.conf--perhaps some of those are set too
highrelative to your server's memory. 

I wouldn't really expect queries to be slow on 99GB, by the way. That is not a large database. Partitioning is
reasonableif you want to prune old data periodically, as then you can just drop old partitions. But unless your queries
areregularly have to perform table scans, I wouldn't expect them to be slow. And if they are performing table scans,
you'dneed to investigate how to fix that--what indexes to add. Partitioning will get you incremental performance gains
insuch a case, while the right index will get you exponential gains.

Re: Loading data from one table to another

From
Laurenz Albe
Date:
On Sun, 2024-10-13 at 06:24 +0000, Sabyasachi Mukherjee wrote:
> To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
>
> The command fails every time.
>
> ERROR:  Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
> ERROR:  out of memory
> SQL state: 53200
> Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".

You have a row level AFTER INSERT trigger on the destination table.
Try without the trigger.

Yours,
Laurenz Albe

Re: Loading data from one table to another

From
vignesh kumar
Date:
Check if you have foreign table references and they should hold on delete cascade that should resolve data being dupliated

From: Laurenz Albe <laurenz.albe@cybertec.at>
Sent: Monday, October 14, 2024 12:06:08 PM
To: Sabyasachi Mukherjee <mukherjee.sabyasachi@outlook.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
Subject: Re: Loading data from one table to another
 
On Sun, 2024-10-13 at 06:24 +0000, Sabyasachi Mukherjee wrote:
> To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
>
> The command fails every time.
>
> ERROR:  Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
> ERROR:  out of memory
> SQL state: 53200
> Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".

You have a row level AFTER INSERT trigger on the destination table.
Try without the trigger.

Yours,
Laurenz Albe