Thread: The backend has broken the connection...
I am running several databases on a PostgresQL server and webapps to go with them on another server. For the past couple of days, several times a day I've been getting a "The backend has broken the connection. Possibly the action you have attempted has caused it to close." error. When this happens, all connections close and anyone in the middle of a session is screwed. They have to shut down their browsers and start over. I'm keeping logs of almost all SQL queries executed, and there seems to be no pattern to when this happens. I am running postmaster with a buffer size of 1024 and a max backends of 256, but I am not reaching the 256 limit. I doubt that I'm even getting to 100 concurrent connections at a time. my specs: postgresql 7.1.3 java sdk 1.4 beta 1 apache 1.3.19 tomcat 4.0.1 I have a second web server on which I tried java sdk 1.4 beta 3 and the most recent JDBC driver (which doesn't compile under Java 1.4, btw), and I don't think it helped (although it's hard to tell because the two web servers are accessing the same databases concurrently). I can't use Java 1.3 because certain JSP pages crash the VM when compiled. Can anyone tell me why I keep getting this message and what I can do about it? Thanks Ben bcarterette@liberty.k12.mo.us
bcarterette@mail.liberty.k12.mo.us writes:
> I am running several databases on a PostgresQL server and webapps to go
> with them on another server. For the past couple of days, several times
> a day I've been getting a "The backend has broken the connection.
> Possibly the action you have attempted has caused it to close." error.
> When this happens, all connections close and anyone in the middle of a
> session is screwed.
Sounds like you are seeing backend crashes. Does the postmaster log
show any complaints about unexpected child process exits? Are there
any core files laying about in the database subdirectories? Ideally
I'd like to see a debugger backtrace from a core file ...
In any case, this is not a Java or JDBC problem, it's a backend problem.
regards, tom lane
I experienced a similar problem some time ago that actually was related to the Java end- Unfortunately I can't remember the exact message, but it might be worth checking out- Tomcat attempts to reload all of the affected classes if you recompile a class that is part of a currently loaded context. In V3.2 they still didn't have this process working very well, & so compiling a seemingly unrelated class would cause the class that our Connection lives in to get reloaded, thus invalidating all of the Connection instances. This would of course only occur in a situation where you were doing development work on the server in question. -Nick > -----Original Message----- > From: pgsql-jdbc-owner@postgresql.org > [mailto:pgsql-jdbc-owner@postgresql.org]On Behalf Of Tom Lane > Sent: Tuesday, January 08, 2002 5:38 PM > To: bcarterette@mail.liberty.k12.mo.us > Cc: pgsql-jdbc@postgresql.org > Subject: Re: [JDBC] The backend has broken the connection... > > > bcarterette@mail.liberty.k12.mo.us writes: > > I am running several databases on a PostgresQL server and webapps to go > > with them on another server. For the past couple of days, > several times > > a day I've been getting a "The backend has broken the connection. > > Possibly the action you have attempted has caused it to close." error. > > When this happens, all connections close and anyone in the middle of a > > session is screwed. > > Sounds like you are seeing backend crashes. Does the postmaster log > show any complaints about unexpected child process exits? Are there > any core files laying about in the database subdirectories? Ideally > I'd like to see a debugger backtrace from a core file ... > > In any case, this is not a Java or JDBC problem, it's a backend problem. > > regards, tom lane > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster >
I solved it. The problem was that when I got up to around 70 or 80 concurrent connections, the database server had too many files open. I added lines in rc.local to increase /proc/sys/fs/file-max and /proc/sys/fs/inode-max fourfold, and since then I've been up to almost 120 concurrent connections with no problem. I had started running a large new webapp about a week ago and I drastically underestimated the effect it would have. The number of pages I'm serving per hour sextupled from one week to the next. The number of concurrent database connections seems to have nearly tripled. Not to mention the number of calls I get per day... thanks, ben On Tuesday, January 8, 2002, at 04:37 PM, Tom Lane wrote: > bcarterette@mail.liberty.k12.mo.us writes: >> I am running several databases on a PostgresQL server and webapps to go >> with them on another server. For the past couple of days, several >> times >> a day I've been getting a "The backend has broken the connection. >> Possibly the action you have attempted has caused it to close." error. >> When this happens, all connections close and anyone in the middle of a >> session is screwed. > > Sounds like you are seeing backend crashes. Does the postmaster log > show any complaints about unexpected child process exits? Are there > any core files laying about in the database subdirectories? Ideally > I'd like to see a debugger backtrace from a core file ... > > In any case, this is not a Java or JDBC problem, it's a backend problem. > > regards, tom lane >
bcarterette@mail.liberty.k12.mo.us writes:
> I solved it. The problem was that when I got up to around 70 or 80
> concurrent connections, the database server had too many files open. I
> added lines in rc.local to increase /proc/sys/fs/file-max and
> /proc/sys/fs/inode-max fourfold, and since then I've been up to almost
> 120 concurrent connections with no problem.
Hmm. Usually when Postgres runs the system out of file descriptors,
it's everything else that starts falling over, not Postgres itself.
(The backend has code to close other open file descriptor(s) if it
needs another one. But most Unix programs just die if they can't
open a file the first try.)
What PG version are you running, exactly? Could you look in the
postmaster log and see what log messages were being generated when
these things happened? I think this deserves a more careful look
than just "I fixed it".
regards, tom lane