Thread: BUG #16208: background worker "logical replication worker" was terminated by signal 11: Segmentation

The following bug has been logged on the website:

Bug reference:      16208
Logged by:          Firdaus Firdaus
Email address:      mr.kaleek@gmail.com
PostgreSQL version: 12.1
Operating system:   Ubuntu 16.04.6 LTS
Description:

I got error when replicate with native logical method from PostgreSQL 11.6
to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already replicated
around more than 2 hours, server can't access and recovery mode. 

Actually we are facing problem when replicating partition table from 11.6 to
12.1, but if there is no partition table the replication works so well.
As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version:

core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Can you help us to find out the solution and configuration?

 please find below :

sudo gdb -p 15803
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 15803
Reading symbols from /usr/lib/postgresql/12/bin/postgres...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols
from
/usr/lib/debug/.build-id/b1/7c21299099640a6d863e423d99265824e7bb16.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no
debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/librt-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libm-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicui18n.so.55...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicuuc.so.55...(no
debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libsystemd.so.0...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libc-2.23.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/ld-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols
found)...done.
Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no
debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no
debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading symbols
from /usr/lib/debug//lib/x86_64-linux-gnu/libresolv-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi.so.3...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls.so.30...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no
debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicudata.so.55...(no
debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libselinux.so.1...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libheimntlm.so.0...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.26...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libasn1.so.8...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhcrypto.so.4...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libroken.so.18...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libidn.so.11...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.6...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.6...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.4...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3...(no debugging
symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libwind.so.0...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libheimbase.so.1...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhx509.so.5...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libsqlite3.so.0...(no
debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols
from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging
symbols found)...done.
Reading symbols from /usr/lib/postgresql/12/lib/pg_stat_statements.so...(no
debugging symbols found)...done.
Reading symbols from /usr/lib/postgresql/12/lib/pg_partman_bgw.so...(no
debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading
symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.23.so...done.
done.
0x00007f5ce4bf85b3 in __select_nocancel () at
../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) cont
Continuing.

Program received signal SIGUSR1, User defined signal 1.
0x00007f5ce4b30730 in __sigprocmask (how=2, set=0x56135acfba40 <UnBlockSig>,
oset=0x0)
    at ../sysdeps/unix/sysv/linux/x86_64/sigprocmask.c:36
36    ../sysdeps/unix/sysv/linux/x86_64/sigprocmask.c: No such file or
directory.
(gdb) detach
Detaching from program: /usr/lib/postgresql/12/bin/postgres, process 15803
(gdb) b errfinish
Breakpoint 1 at 0x56135a814a94

$ for f in `pgrep postgres`; do ls -l /usr/lib/postgresql/12/bin/postgres;
done
-rwxr-xr-x 1 root root 7978032 Nov 12 19:34
/usr/lib/postgresql/12/bin/postgres
-rwxr-xr-x 1 root root 7978032 Nov 12 19:34
/usr/lib/postgresql/12/bin/postgres
-rwxr-xr-x 1 root root 7978032 Nov 12 19:34
/usr/lib/postgresql/12/bin/postgres

 $./pg_ctl -D /data/postgres/data -l logfile -c start
-su: ./pg_ctl: No such file or directory
$ pg_ctl -D /data/postgres/data -l logfile -c start
pg_ctl: command not found

on datadir I did not see core.PID, let me know where the dir.
$ ls -lrth
total 144K
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_twophase
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_tblspc
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_snapshots
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_serial
drwx------ 4 postgres postgres 4.0K Jan  2 16:34 pg_multixact
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_dynshmem
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_commit_ts
-rw------- 1 postgres postgres    3 Jan  2 16:34 PG_VERSION
-rw------- 1 postgres postgres   88 Jan  2 16:34 postgresql.auto.conf
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_xact
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_subtrans
drwx------ 6 postgres postgres 4.0K Jan  2 16:44 base
drwx------ 5 postgres postgres 4.0K Jan  3 18:29 pg_replslot
-rw------- 1 postgres postgres  127 Jan 15 12:38 postmaster.opts
drwx------ 2 postgres postgres 4.0K Jan 16 01:00 pg_log
-rw------- 1 postgres postgres   47 Jan 16 01:00 current_logfiles
drwx------ 3 postgres postgres  52K Jan 16 11:00 pg_wal
drwx------ 4 postgres postgres 4.0K Jan 16 11:40 pg_logical
-rw------- 1 postgres postgres   93 Jan 16 11:40 postmaster.pid
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat_tmp
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_notify
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 global
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat
$ cd postmaster.pid
-su: cd: postmaster.pid: Not a directory
$ more postmaster.pid
15803
/data/postgres/data
1579063112
5432
/var/run/postgresql
*
  5432001 169312256
ready

On Thu, Jan 16, 2020 at 12:26:08PM +0000, PG Bug reporting form wrote:
> on datadir I did not see core.PID, let me know where the dir.

This depends on the setting you have for kernel.core_pattern.
--
Michael

On Sat, Jan 18, 2020 at 03:17:31PM +0700, Mr Kaleek wrote:
> I don't know where the path of *CORE.PID*, because on datadir and base dir
> I didn't see. can you help me to find it.

As the PostgreSQL processes run from the root of the data folder, any
core files would be saved there if your kernel settings don't redirect
things to another location (this changes across distributions as far
as I know).
--
Michael