Re: AWS and postgres issues - Mailing list pgsql-general

On Mon, Apr 08, 2013 at 04:24:45PM -0700, David Kerr wrote:
- On Mon, Apr 08, 2013 at 02:59:56PM -0700, David Kerr wrote:
- - On Mon, Apr 08, 2013 at 02:09:42PM -0700, David Kerr wrote:
- - - On Mon, Apr 08, 2013 at 02:14:14PM -0600, Quentin Hartman wrote:
- - - - What version of pgpool are you using?
- - - -
- - - - Are there other commands you have a problem with? I would suspect that the
- - - - restart is causing the postgres server to go away, pgpool decides to
- - - - disconnect, and then it has to be manually added back to the cluster.
- - - - Unless of course you've got automatic failback setup, but even then I would
- - - - expect that command to do weird things when issued through middleware like
- - - - pgpool, regardless of what sort of infrastructure you are running on.
- - - -
- - - - QH
- - -
- - - This is actually from the command line, PgPool isn't involved at all. I just
- - - mentioned it to give some context.
- -
- - I had a brief conversation with Quentin offline which indicated that I wasn't
- - being nearly clear nor direct enough.
- -
- - I believe that this probelm is specific to AWS+Postgres. (and possibly specific to
- - VPC+Amazon Linux). Non postgres commands run fine, and even psql works fine.
- - so far just pg_ctl fails.
- -
- - I'm running the command directly from an interactive shell, so this is as basic as it gets.
- -
- - The command runs correctly on the remote server, it just never exits the ssh connection.
- - specifically I never get: "debug2: channel 0: rcvd close  " as if the packet gets
- - dropped every time.
- -
- - I don't believe it's a network hiccup because I can reproduce it every time.
- -
- - It's likely something with Amazon's infrastructure that's eating it, but whateever
- - it is, it seems to specifically not like pg_ctl.
- -
- - Here is what I see happen:
- - [pgpool@ccpgp05 ~]$ ssh -vvv postgres@10.0.1.30 '/usr/pgsql-9.2/bin/pg_ctl -D /db/pg -m fast restart'
- - OpenSSH_6.1p1, OpenSSL 1.0.1e-fips 11 Feb 2013
- - [..snip..]
- - debug1: Sending command: /usr/pgsql-9.2/bin/pg_ctl -D /db/pg -m fast restart
- - [..snip..]
- - waiting for server to shut down.... done
- - server stopped
- - server starting
- - [..snip..]
- - debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
- - debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
- - debug2: channel 0: rcvd eow
- - debug2: channel 0: close_read
- - debug2: channel 0: input open -> closed
- - ^Cdebug1: channel 0: free: client-session, nchannels 1                  # <---- This is where I ^C it
- - debug3: channel 0: status: The following connections are open:
- -   #0 client-session (t4 r0 i3/0 o0/0 fd -1/5 cc -1)
- -
- -
- - Notice the input open -> closed is where it basically hangs
- -
- - Now look at:
- - [pgpool@ccpgp05 ~]$ ssh -vvv postgres@10.0.1.30 'ls -ltr'
- - OpenSSH_6.1p1, OpenSSL 1.0.1e-fips 11 Feb 2013
- - [..snip..]
- - debug1: Sending command: ls -ltr
- - debug2: channel 0: output open -> drain
- - debug1: channel 0: forcing write
- - total 8
- - drwx------ 4 postgres postgres 4096 Apr  4 22:50 9.2
- - drwx------ 2 postgres postgres 4096 Apr  5 22:58 bin
- - [..snip..]
- - debug2: channel 0: input open -> closed
- - debug2: channel 0: rcvd close
- - debug3: channel 0: will not send data after close
- - debug2: channel 0: almost dead
- - debug2: channel 0: gc: notify user
- - debug2: channel 0: gc: user detached
- - debug2: channel 0: send close
- - debug2: channel 0: is dead
- - debug2: channel 0: garbage collecting
- - debug1: channel 0: free: client-session, nchannels 1
- - debug3: channel 0: status: The following connections are open:
- -   #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
- -
- - Transferred: sent 2456, received 2448 bytes, in 0.0 seconds
- - Bytes per second: sent 105666.4, received 105322.3
- - debug1: Exit status 0
-
-
- I've verified that it's not related to the linux flavor. I've tried it with a Server on both Amazon Linux
- and Ubuntu. And with a client on Amazon Linux and my own desktop.
-
- I can't be the only person using PG in AWS+VPC, can someone else with a similar test bed give it a shot
- and tell me if it works for them? (at least then I'd know if it's likely something I'm doing...)
-
- Thanks

While debugging this with a coworker we figured out that pg_ctl was attaching to the tty and then it clicked
that we needed to be using '-t' where I was using -T or (neither).

So mystery solved!