Thread: What is the tuplestore?
Hi, v9.6.6 2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary file: No space left on device I see this in the pg_log file, but #1 can't figure out what "tuplestore" is (Google doesn't help), and #2 there's lots of space on all my file systems. data/base, where pgsql_tmp lives, has 96GB free.) Thanks -- Angular momentum makes the world go 'round.
What if this error message pertains to something happening on the application side?
Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:
Hi,
v9.6.6
2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device
I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)
Thanks
--
Angular momentum makes the world go 'round.
There's certainly a problem with the application, but the error is in the pg_log, not the application log.
On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:
What if this error message pertains to something happening on the application side?Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:Hi,
v9.6.6
2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device
I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)
Thanks
--
Angular momentum makes the world go 'round.
Angular momentum makes the world go 'round.
Maybe the temp space got released right after the failure?
do you have space usage charts for that partition? doesn't it show a spike during that time?
Am Mo., 10. Dez. 2018 um 15:54 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:
There's certainly a problem with the application, but the error is in the pg_log, not the application log.On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:What if this error message pertains to something happening on the application side?Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:Hi,
v9.6.6
2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device
I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)
Thanks--
Angular momentum makes the world go 'round.
Which file system (specifically, which directory)? Is it data/base/pgsql_tmp? There's 96GB free, which is 74% of the volume.
On 12/10/2018 04:50 PM, Rene Romero Benavides wrote:
Maybe the temp space got released right after the failure?do you have space usage charts for that partition? doesn't it show a spike during that time?Am Mo., 10. Dez. 2018 um 15:54 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:There's certainly a problem with the application, but the error is in the pg_log, not the application log.On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:What if this error message pertains to something happening on the application side?Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:Hi,
v9.6.6
2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device
I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)
Thanks
--
Angular momentum makes the world go 'round.
Angular momentum makes the world go 'round.
Yes, pgsql_tmp, you ought to be looking for a sudden and drastic jump in space utilization around the time of the error message. You're not concerned with the current space utilization, but with the one around that time, because, it probably got freed right after the error was raised.
How many times has this happened ? what kind of queries were running at that time? can you identify something that could have required lots of temporary space?
Am Mo., 10. Dez. 2018 um 17:33 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:
Which file system (specifically, which directory)? Is it data/base/pgsql_tmp? There's 96GB free, which is 74% of the volume.On 12/10/2018 04:50 PM, Rene Romero Benavides wrote:Maybe the temp space got released right after the failure?do you have space usage charts for that partition? doesn't it show a spike during that time?Am Mo., 10. Dez. 2018 um 15:54 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:There's certainly a problem with the application, but the error is in the pg_log, not the application log.On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:What if this error message pertains to something happening on the application side?Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:Hi,
v9.6.6
2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device
I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)
Thanks--
Angular momentum makes the world go 'round.
On Tue, Dec 11, 2018 at 2:56 AM Ron <ronljohnsonjr@gmail.com> wrote: > 2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT > PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary > file: No space left on device > > I see this in the pg_log file, but #1 can't figure out what "tuplestore" is > (Google doesn't help), and #2 there's lots of space on all my file systems. > data/base, where pgsql_tmp lives, has 96GB free.) Maybe the setting log_temp_files could help you identify the query that is doing it? https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOG-TEMP-FILES -- Thomas Munro http://www.enterprisedb.com
I extracted the queries from pg_log and sent them to the customer team. To fill up 96GB of disk space seems like the customer selected a huge date range.
On 12/10/2018 06:07 PM, Rene Romero Benavides wrote:
Yes, pgsql_tmp, you ought to be looking for a sudden and drastic jump in space utilization around the time of the error message. You're not concerned with the current space utilization, but with the one around that time, because, it probably got freed right after the error was raised.How many times has this happened ? what kind of queries were running at that time? can you identify something that could have required lots of temporary space?Am Mo., 10. Dez. 2018 um 17:33 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:Which file system (specifically, which directory)? Is it data/base/pgsql_tmp? There's 96GB free, which is 74% of the volume.On 12/10/2018 04:50 PM, Rene Romero Benavides wrote:Maybe the temp space got released right after the failure?do you have space usage charts for that partition? doesn't it show a spike during that time?Am Mo., 10. Dez. 2018 um 15:54 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:There's certainly a problem with the application, but the error is in the pg_log, not the application log.On 12/10/2018 03:21 PM, Rene Romero Benavides wrote:What if this error message pertains to something happening on the application side?Am Mo., 10. Dez. 2018 um 09:56 Uhr schrieb Ron <ronljohnsonjr@gmail.com>:Hi,
v9.6.6
2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device
I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)
Thanks--
Angular momentum makes the world go 'round.--
--
Angular momentum makes the world go 'round.
Angular momentum makes the world go 'round.
On 12/10/2018 06:24 PM, Thomas Munro wrote: > On Tue, Dec 11, 2018 at 2:56 AM Ron <ronljohnsonjr@gmail.com> wrote: >> 2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT >> PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary >> file: No space left on device >> >> I see this in the pg_log file, but #1 can't figure out what "tuplestore" is >> (Google doesn't help), and #2 there's lots of space on all my file systems. >> data/base, where pgsql_tmp lives, has 96GB free.) > Maybe the setting log_temp_files could help you identify the query > that is doing it? > > https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOG-TEMP-FILES Great idea; I've made the change. What do I grep for in the pg_log file? -- Angular momentum makes the world go 'round.
On Tue, Dec 11, 2018 at 3:58 PM Ron <ronljohnsonjr@gmail.com> wrote: > On 12/10/2018 06:24 PM, Thomas Munro wrote: > > On Tue, Dec 11, 2018 at 2:56 AM Ron <ronljohnsonjr@gmail.com> wrote: > >> 2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT > >> PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary > >> file: No space left on device > >> > >> I see this in the pg_log file, but #1 can't figure out what "tuplestore" is > >> (Google doesn't help), and #2 there's lots of space on all my file systems. > >> data/base, where pgsql_tmp lives, has 96GB free.) > > Maybe the setting log_temp_files could help you identify the query > > that is doing it? > > > > https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOG-TEMP-FILES > > Great idea; I've made the change. What do I grep for in the pg_log file? It should say something like: LOG: temporary file: path "base/pgsql_tmp/...", size ... STATEMENT: ... -- Thomas Munro http://www.enterprisedb.com
When one get a "No space left on device" and there is a lot of space it is sometimes caused by lack of inodes.
Try run the command:
df --inodes
On Mon, Dec 10, 2018 at 4:56 PM Ron <ronljohnsonjr@gmail.com> wrote:
Hi,
v9.6.6
2018-12-07 06:21:55.504 EST 10.140.181.89(35868) CDS CDSLBXW 13748 SELECT
PostgreSQL JDBC Driver 53100 ERROR: could not write to tuplestore temporary
file: No space left on device
I see this in the pg_log file, but #1 can't figure out what "tuplestore" is
(Google doesn't help), and #2 there's lots of space on all my file systems.
data/base, where pgsql_tmp lives, has 96GB free.)
Thanks
--
Angular momentum makes the world go 'round.