PANIC: could not write to file "pg_wal/xlogtemp.11399": No spaceleft on device - Mailing list pgsql-general

Hello list,
[10:47:13] [postgres@host] $  psql --version
psql (PostgreSQL) 11.5
[11:06:36] [postgres@host] $ cat /etc/redhat-release
CentOS release 6.10 (Final)
[11:06:33] [postgres@host] $ uname -a
Linux host 2.6.32-754.18.2.el6.x86_64 #1 SMP Wed Aug 14 16:26:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

I have been in the process of upgrading my pg9.3 systems to pg11.5, all went well with my upgrade using the pg_upgrade
method. Once the upgrade was complete I ran the following: 
/usr/pgsql-11/bin/vacuumdb -v -j 6 --all --analyze

I have pg_wal on a separate partition that ran out of space, when I saw this, I issued what I think was a premature
(servicepostgresql-11 stop), then moved the pg_wal log to a larger partition and started postgres, the database did
recoverafter some time of the recovering process running which is great! 

My question is should I be concerned with data loss when a vacuumdb with analyze crashed due to space, and I restarted
thedatabase possibly when it was trying to recover?  The hinted message at 2019-09-12 23:26:15.978  in the log scares
mea bit. 

2019-09-12 23:15:23.519 EDT [10101] ERROR:  canceling autovacuum task
2019-09-12 23:15:23.519 EDT [10101] CONTEXT:  automatic vacuum of table "famnet5.public.private_tablename"
2019-09-12 23:24:51.640 EDT [11399] PANIC:  could not write to file "pg_wal/xlogtemp.11399": No space left on device
2019-09-12 23:24:51.640 EDT [11399] CONTEXT:  writing block 288 of relation base/16402/4190122
2019-09-12 23:24:51.640 EDT [11399] STATEMENT:  VACUUM (VERBOSE, ANALYZE) public.private_table_name;
2019-09-12 23:24:51.686 EDT [9792] LOG:  server process (PID 11399) was terminated by signal 6: Aborted
2019-09-12 23:24:51.686 EDT [9792] DETAIL:  Failed process was running: VACUUM (VERBOSE, ANALYZE)
public.private_tablename;
2019-09-12 23:24:51.687 EDT [9792] LOG:  terminating any other active server processes
2019-09-12 23:24:51.689 EDT [11396] WARNING:  terminating connection because of crash of another server process
2019-09-12 23:24:51.689 EDT [11396] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupte 
d shared memory.
2019-09-12 23:24:51.689 EDT [11396] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
2019-09-12 23:24:51.689 EDT [9799] WARNING:  terminating connection because of crash of another server process
2019-09-12 23:24:51.689 EDT [9799] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupted 
 shared memory.
2019-09-12 23:24:51.689 EDT [9799] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
2019-09-12 23:24:51.689 EDT [11398] WARNING:  terminating connection because of crash of another server process
2019-09-12 23:24:51.689 EDT [11398] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupte 
d shared memory.
2019-09-12 23:24:51.689 EDT [11398] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
2019-09-12 23:24:51.689 EDT [11397] WARNING:  terminating connection because of crash of another server process
2019-09-12 23:24:51.689 EDT [11397] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupte 
d shared memory.
2019-09-12 23:24:51.689 EDT [11397] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
2019-09-12 23:24:51.691 EDT [12616] LOG:  PID 11398 in cancel request did not match any process
2019-09-12 23:24:51.692 EDT [9792] LOG:  all server processes terminated; reinitializing
2019-09-12 23:24:51.758 EDT [12618] LOG:  PID 11399 in cancel request did not match any process
2019-09-12 23:24:51.758 EDT [12617] LOG:  database system was interrupted; last known up at 2019-09-12 23:21:01 EDT
2019-09-12 23:24:54.110 EDT [12617] LOG:  database system was not properly shut down; automatic recovery in progress
2019-09-12 23:24:54.121 EDT [12617] LOG:  redo starts at 119/3A8EAF50
2019-09-12 23:25:34.712 EDT [9792] LOG:  received fast shutdown request
2019-09-12 23:25:34.735 EDT [9792] LOG:  abnormal database system shutdown
2019-09-12 23:25:34.779 EDT [9792] LOG:  database system is shut down
2019-09-12 23:26:15.978 EDT [12972] LOG:  database system was interrupted while in recovery at 2019-09-12 23:24:54 EDT
2019-09-12 23:26:15.978 EDT [12972] HINT:  This probably means that some data is corrupted and you will have to use the
lastbackup for recovery. 
2019-09-12 23:26:19.213 EDT [12972] LOG:  database system was not properly shut down; automatic recovery in progress
2019-09-12 23:26:19.218 EDT [12972] LOG:  redo starts at 119/3A8EAF50
2019-09-12 23:32:06.654 EDT [12972] LOG:  redo done at 119/73FFFC30
2019-09-12 23:32:06.665 EDT [12972] LOG:  last completed transaction was at log time 2019-09-12 23:24:49.792961-04
2019-09-12 23:32:10.087 EDT [12969] LOG:  database system is ready to accept connections
2019-09-12 23:53:12.221 EDT [12969] LOG:  received fast shutdown request
2019-09-12 23:53:12.226 EDT [12969] LOG:  aborting any active transactions
2019-09-12 23:53:12.228 EDT [12969] LOG:  background worker "logical replication launcher" (PID 14020) exited with exit
code1 
2019-09-12 23:53:12.309 EDT [14015] LOG:  shutting down
2019-09-12 23:53:12.407 EDT [12969] LOG:  database system is shut down
2019-09-12 23:53:19.029 EDT [16953] LOG:  database system was shut down at 2019-09-12 23:53:12 EDT
2019-09-12 23:53:19.041 [16950] LOG:  database system is ready to accept connections

Jason Ralph

This message contains confidential information and is intended only for the individual named. If you are not the named
addresseeyou should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if
youhave received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be
guaranteedto be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or
incomplete,or contain viruses. The sender therefore does not accept liability for any errors or omissions in the
contentsof this message, which arise as a result of e-mail transmission. If verification is required please request a
hard-copyversion.