Thread: Storage Manager crash at mdwrite()

On Thu, 2012-03-15 at 13:49 -0400, Tareq Aljabban wrote:
> I'm implementing an extention to mdwrite() at
> backend/storage/smgr/md.c
> When a block is written to the local storage using mdwrite(), I'm
> sending this block to an HDFS storage. 
> So far I don't need to read back the values I'm writing to HDFS. This
> approach is working fine in the initDB phase.
> However, when I'm running postgres (bin/pg_ctl start), the first few
> write operations run successfully, and then suddenly (after writing
> exactly 3 files to HDFS), I get a 130 exit code with the following
> message showing the JVM thread dump of HDFS:
> 
> 
> LOG:  background writer process (PID 29347) exited with exit code 130
> LOG:  terminating any other active server processes

...

> This seems like an HDFS issue, but the same code worked properly in
> initDB(). I replaced this HDFS write code with a code that writes
> always the same block (empty one) to HDFS regardless from the value
> received by mdwrite().. Kept getting the same issue after writing 3
> files.
> I copied this exact code to a separate C application and ran it there
> and it executed without any problems (I wrote/deleted 100 files).
> That's why I'm doubting that it's something related to postgreSQL.
> 
> 
> Any ideas on what's going wrong?

What code, exactly, did you change in md.c, and anywhere else in
postgres? Are you linking in new libraries/code from somewhere into the
postgres backend?

Regards,Jeff Davis

On Thu, 2012-03-15 at 19:36 -0400, Tareq Aljabban wrote:
> When configuring postgreSQL, I'm adding the libraries needed to run
> HDFS C API (libhdfs).
> 

>From the information below, it looks like C++.

> 
>  ./configure --prefix=/diskless/myUser/Workspace/EclipseWS1/pgsql
> --enable-depend --enable-cassert --enable-debug CFLAGS="$CFLAGS
> -I/diskless/myUser/Workspace/HDFS_Append/hdfs/src/c++/libhdfs
> -I/usr/lib/jvm/default-java/include" LDFLAGS="$LDFLAGS
> -L/diskless/myUser/Workspace/HDFS_Append/hdfs/src/c++/libhdfs
> -L/diskless/myUser/Workspace/HDFS_Append/build/c++/Linux-i386-32/lib
> -L/usr/lib/jvm/default-java/jre/lib/i386/server -ljvm -lhdfs"
> 
> 
> 
> 
> 
> I have done lots of changes so far on how the storage manager works.
> In fact, I changed smgr.c so instead of calling regular md.c
> functions, that it would call my functions .
> For simplicity, you can say that whenever mdwrite() was supposed to be
> called, another function is also called beside it. so now what is
> being called is:
> mdwrite(param1, param2, buffer, ....);
> hdfswrite(param1, param2, buffer, ....);
> 
> 
> where hdfswrite() is the function where I write the buffer to HDFS.
> I changed hdfswrite() so it will always write only the same (dummy)
> buffer down to HDFS storage. Let's say "dummyBufferFile". After
> writing this file 3 times to HDFS, I'm getting the message that I've
> shown in my first email.
> The same hdfswrite() code works without any issues when I run it in a
> separate application.
> 
> 
> Hope it's clear now.

Well, it's clear that there's a lot going on ;)

In other words, is there a reason you think that these would all work
together nicely?

I don't know the specifics, but I understand there are numerous perils
to linking complex C++ code into complex C code, particularly around
exception handling. Look in the archives for previous discussions around
C++.

Regards,Jeff Davis

On Fri, 2012-03-16 at 18:02 -0400, Tareq Aljabban wrote:
> Thanks for your response Jeff.. 
> It's true that libhdfs code resides under the c++ folder, but in all
> of the documentation, libhdfs is referred to as the C interface of
> HDFS.
> Now what you're saying makes sense.. that nothing guarantees this will
> work well.. but in the phase of initDB, the function calls are done
> nicely without any exceptions.. when starting the postmaster,
> something wrong happens after 3 calls to libhdfs.. that's what I'm
> confused about..
> What it's the difference between the two processes (initDB, start
> postmaster), that might cause this crash to happen?

There is a lot of difference between those two. In particular, it looks
like the problem you are seeing is coming from the background writer,
which is not running during initdb.

I think the next step would be for you to compile in debug mode with
-O0, and attach to the bgwriter process with gdb, and step through it.
Then, you can see the exact path which causes the bgwriter to exit, and
that will give you a better idea where the problem is.

Regards,Jeff Davis

On Fri, Mar 16, 2012 at 11:29 PM, Jeff Davis <pgsql@j-davis.com> wrote:
> There is a lot of difference between those two. In particular, it looks
> like the problem you are seeing is coming from the background writer,
> which is not running during initdb.

The difference that comes to mind is that the postmaster forks. If the
library opens any connections prior to forking and then uses them
after forking that would work at first but it would get confused
quickly once more than one backend tries to use the same connection.
The data being sent would all be mixed together and they would see
responses to requests other processes sent.

You need to ensure that any network connections are opened up *after*
the new processes are forked.

There are other things going on that could cause problems. initdb
probably doesn't deal with many errors so it might not be casing any
longjmps that happen when Postgres handles errors. I suspect it
doesn't create any temporary files, etc.

There's also a long history of threads not playing well with Postgres.
I think that's overblown and I believe they should work fine. Most of
it was caused by a bug in an old version of libc. But you do have to
ensure that the other threads don't call any Postgres functions
because the Postgres code is very much not thread-aware.

-- 
greg