Thread: Dangling backends on win32 7.2.1 port (peerdirect).

I have been testing the postgresql Peer Direct port.  I've used both the
released binary and my own compiled version and get the same behavior.
Please not that this is for testing purposes and I am not asking for
support.

I'm running an import routine which moves records from a cobol database
to postgres.  This is strictly an import routine, where all queries are
serialized.  The postgres.conf file is set to defaults, except for
fsync=false and the change in the WAL setting (see below).  All sql
statements are made using the libpq interface, and are simply create
table followed by create index... followed by insert (repeat).  Table
sizes range from 0 to about 300k records, and there about 100 tables.
Most of the tables have at least 10 fields, and all of the tables have
at least one index, usually several.  The client and the server are
running on the same machine.  There is no other activity on the
database, and the import does not use transactions of any kind.

About once in every 3000-5000 inserts, a new backend gets created
without the previous one getting shut down.  About 75% of the way though
the process, too many backends get created and the import fails.  The
first time through, postmaster is reporting things like:
DEBUG: Removing transaction log file 000000000000000000
DEBUG: XLogWrite: new log file created - consider increasing WAL_FILES

Other times, I get the more ominous
DEBUG: rename from C:\postgres\peer_direct\data/pg_xlog/0000000000000003
to C:\postgres\peer_direct\data/pg_xlog/000000000000000A (initialization
of log file 0, segment 10) failed: Permission denied.

If this happens about 10 times I will have about 7 backends up with 6
doing nothing and only 44k memory allocated for each.  Killing the
client app kills all the backends.

The second time through I tried increasing the WAL files from 0 to
eight.  This did remove the XLogWrite: messages but did not fix the
problem.  There is not much point in turning fsync on because this would
take too long.  I could get the data in very easily with a preformatted
copy, but I thought I'd pass this information along!

Merlin

I wrote:
>
> Other times, I get the more ominous
> DEBUG: rename from
C:\postgres\peer_direct\data/pg_xlog/0000000000000003
> to C:\postgres\peer_direct\data/pg_xlog/000000000000000A
(initialization
> of log file 0, segment 10) failed: Permission denied.
>
> If this happens about 10 times I will have about 7 backends up with 6
> doing nothing and only 44k memory allocated for each.  Killing the
> client app kills all the backends.

OK, I read the readme file and saw the note about the permission denied
error, so I'm not crazy.  However, there was no mention of the extra
processes which seems to me to be a catastrophic side affect.  The
processes appear to be waiting on some sort of lock on the transaction
files, and seem to be in some sort of limbo until the original
connection is closed.   I can create very reasonable conditions which
will take a database down within a few hours.

Has this been fixed?  If not, I'm prepared to start slogging it out.
The way I see it, a production database is 100% likely to shut down
within a very short period of time (hours) unless special care is taken
to reset all the database connections or at least TerminateProcess()
dormant processes (yuck!).  I know the peerdirect patches are being
applied to the cvs version.

Aside from this problem and the very silly divide by zero error, the
win32 port has been very well behaved, with decent, if not great,
performance.

Merlin

TRY TEST WIN32 PORT.  DATABASE GO BOOM!  TRY FIX NOBODY CARE.  WIN32
PORT COME OUT MANY DATABASE GO BOOM!  TRY HELP GET IGNORED.  JUST WANT
HELP.  BUG FIX?

Regards,
Merlin

Merlin Moncure wrote:
> 
> TRY TEST WIN32 PORT.  DATABASE GO BOOM!  TRY FIX NOBODY CARE.  WIN32
> PORT COME OUT MANY DATABASE GO BOOM!  TRY HELP GET IGNORED.  JUST WANT
> HELP.  BUG FIX?

Pardonne moi?

What exactly did you test? If it is the PeerDirect Beta version of
PostgreSQL for Windows named UltraSQL, please know that thus far there
has to my knowledge not been an offical, supported release of UltraSQL
yet - and since the company eliminated the group working on that port, I
doubt there ever will.

PeerDirect tried to contribute the Win32 port portion of their work to
the open source project. The PostgreSQL global development team has not
yet made any final decision if or what parts of that code will or will
not become part of the regular PostgreSQL distribution.

So ... YOU DO NOT USE ANYTHING OFFICIALLY SUPPORTED BY US!!!

Could you therefore calm down, stop shouting, comb your hair, get the
sources of the current work in progress and send in your
complains/suggestions in context diff format?


Have a nice day,
Jan

> 
> Regards,
> Merlin
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org


-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #

Jan Wieck wrote:
> Merlin Moncure wrote:
> > 
> > TRY TEST WIN32 PORT.  DATABASE GO BOOM!  TRY FIX NOBODY CARE.  WIN32
> > PORT COME OUT MANY DATABASE GO BOOM!  TRY HELP GET IGNORED.  JUST WANT
> > HELP.  BUG FIX?
> 
> Pardonne moi?
> 
> What exactly did you test? If it is the PeerDirect Beta version of
> PostgreSQL for Windows named UltraSQL, please know that thus far there
> has to my knowledge not been an offical, supported release of UltraSQL
> yet - and since the company eliminated the group working on that port, I
> doubt there ever will.
> 
> PeerDirect tried to contribute the Win32 port portion of their work to
> the open source project. The PostgreSQL global development team has not
> yet made any final decision if or what parts of that code will or will
> not become part of the regular PostgreSQL distribution.

Actually, I believe we have decided to use the PeerDirect code to do a
native Win32 port for 7.4.  I just posted my first patch, and will spend
most of the next two months completing the job.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 

Jan Wieck wrote:
> What exactly did you test? If it is the PeerDirect Beta version of
> PostgreSQL for Windows named UltraSQL, please know that thus far there
> has to my knowledge not been an offical, supported release of UltraSQL
> yet - and since the company eliminated the group working on that port,
I
> doubt there ever will.

I tested the binary released version and my own compiled version.  I
have been trying to compile the version that is patched to the latest
official release but so far I've been having trouble getting that
compiled.

>
> PeerDirect tried to contribute the Win32 port portion of their work to
> the open source project. The PostgreSQL global development team has
not
> yet made any final decision if or what parts of that code will or will
> not become part of the regular PostgreSQL distribution.
>
> So ... YOU DO NOT USE ANYTHING OFFICIALLY SUPPORTED BY US!!!
>
> Could you therefore calm down, stop shouting, comb your hair, get the
> sources of the current work in progress and send in your
> complains/suggestions in context diff format?

Sorry, I didn't mean to offend. My understanding is that the UltraSQL
patches are likely to go into general release when the source goes
7.4/8.0.
There is at least one show-stopping bug which I believe is related to
the shared memory emulation.  I am almost 100% sure that the bug is in
the win32 side and if still present will carry over to the latest
release.
I offered to help find and catch it: but before I spend the time to do
that I'd like to at least know if the problem has already been
addressed.  I was responding to the call for capable developers to help
test upcoming patch, posted on this very list about two weeks ago.  I'm
not asking for support but trying to help.

Sorry to bother,
Merlin