Thread: running pg_dump from python

I ahve the following python file that I am running as an external method
in Zope.

def backup():
    import  os
    os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  >
c:/scholarpack/ancillary/scholarpack.sql")
    data=open('c:/scholarpack/ancillary/scholarpack.sql','r')
    r=data.read()
    data.close
    return r

However, when I run this script it creates the file scholarpack.sql but
the file is empty.
Any help much appreciated.
Regards
Garry

Scott Mead wrote:
> On Sun, Jun 14, 2009 at 4:06 PM, Garry Saddington
> <garry@schoolteachers.co.uk <mailto:garry@schoolteachers.co.uk>> wrote:
>
>     I ahve the following python file that I am running as an external
>     method in Zope.
>
>     def backup():
>       import  os
>       os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  >
>     c:/scholarpack/ancillary/scholarpack.sql")
>
>
>     Have you tried running that command on the command line by itself
> (as the same user that runs the phython)?  If that gives you the same
> result, then you know for sure that it's a function of the pg_dump
> options and not the python script.
>
>     Are you looking for the full SQL of the scholarpack database?

Yes
> What user is this running as?
scholarpack but no OS user scholarpack, but why should it create a file
then not fill the contents?
> Remember, in your case, pg_dump is going to try to connect as the OS
> username running your script.  you may want to include the username
> option to pg_dump:
>
>      pg_dump -U <username> scholarpack
Zope is running as user scholarpack and I have tried the above as -U
scholarpack and it works

>
>      Try running that on the commandline first, by itself, as the same
> user that runs the python script.

Looks like the script is at fault
regards
Garry

Garry Saddington <garry@schoolteachers.co.uk> writes:
>> import  os
>> os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  >
>> c:/scholarpack/ancillary/scholarpack.sql")

> scholarpack but no OS user scholarpack, but why should it create a file
> then not fill the contents?

Because that's exactly what will happen if pg_dump fails.  The shell
creates the empty output file and then tries to run the program.
If program fails before sending anything to the output file, that's
what you're left with.

The real problem with this script is it's not even considering the
possibility of program failure --- it's not checking for an error exit
code, much less ensuring that error messages go someplace where you
could look at them to find out what the problem is.

            regards, tom lane

On 2009-06-14, Garry Saddington <garry@schoolteachers.co.uk> wrote:
> I ahve the following python file that I am running as an external method
> in Zope.
>
> def backup():
>     import  os
>     os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  >
> c:/scholarpack/ancillary/scholarpack.sql")

are you sure you're using os.popen correctly?
you don't appear to be waiting for the pg_dump process to finish.


>     data=open('c:/scholarpack/ancillary/scholarpack.sql','r')
>     r=data.read()
>     data.close
>     return r
>

On Jun 15, 2009, at 5:17 AM, Jasen Betts wrote:

> On 2009-06-14, Garry Saddington <garry@schoolteachers.co.uk> wrote:
>> I ahve the following python file that I am running as an external
>> method
>> in Zope.
>>
>> def backup():
>>    import  os
>>    os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  >
>> c:/scholarpack/ancillary/scholarpack.sql")
>
> are you sure you're using os.popen correctly?
> you don't appear to be waiting for the pg_dump process to finish.



Right, the popen stuff should be something like:

p = os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  > c:/
scholarpack/ancillary/scholarpack.sql 2> c:/scholarpack/ancillary/
dump.err")
status = p.close()

Then check status to see if the command was successful or not.

Erik Jones, Database Administrator
Engine Yard
Support, Scalability, Reliability
866.518.9273 x 260
Location: US/Pacific
IRC: mage2k

On 2009-06-18, Erik Jones <ejones@engineyard.com> wrote:
>
> On Jun 15, 2009, at 5:17 AM, Jasen Betts wrote:
>
>> On 2009-06-14, Garry Saddington <garry@schoolteachers.co.uk> wrote:
>>> I ahve the following python file that I am running as an external
>>> method
>>> in Zope.
>>>
>>> def backup():
>>>    import  os
>>>    os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  >
>>> c:/scholarpack/ancillary/scholarpack.sql")
>>
>> are you sure you're using os.popen correctly?
>> you don't appear to be waiting for the pg_dump process to finish.
>
> Right, the popen stuff should be something like:
>
> p = os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  > c:/
> scholarpack/ancillary/scholarpack.sql 2> c:/scholarpack/ancillary/
> dump.err")
>
> status = p.close()

I suspect you should also read from the handle until it returns a blocking EOF
or take some other action to ensure the process has finished before
closing the pipe.

> Then check status to see if the command was successful or not.

I suspect Garry really wanted to use os.system instead as the command
he's running is not likely to produce any data on stdout.

or perhaps he wanted to runs the command and grab stdout in the pipe...

Hi,

Erik Jones <ejones@engineyard.com> writes:

> On Jun 15, 2009, at 5:17 AM, Jasen Betts wrote:
>
>> On 2009-06-14, Garry Saddington <garry@schoolteachers.co.uk> wrote:
>>> def backup():
>>>    import  os
>>>    os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  >
>>> c:/scholarpack/ancillary/scholarpack.sql")
>>
>> are you sure you're using os.popen correctly?
>> you don't appear to be waiting for the pg_dump process to finish.
>
> Right, the popen stuff should be something like:
>
> p = os.popen("c:/scholarpack/postgres/bin/pg_dump scholarpack  > c:/
> scholarpack/ancillary/scholarpack.sql 2> c:/scholarpack/ancillary/
> dump.err")
> status = p.close()
>
> Then check status to see if the command was successful or not.

Well, use subprocess:

def run_command(command, expected_retcodes = 0, stdin = None):
    """run a command and raise an exception if retcode not in expected_retcode"""

    # we want expected_retcode to be a tuple but will manage integers
    if type(expected_retcodes) == type(0):
        expected_retcodes = (expected_retcodes,)

    # we want the command to be a list, but accomodate when given a string
    cmd = command
    if type(cmd) == type('string'):
        cmd = shlex.split(command)

    proc = subprocess.Popen(cmd,
                            stdin  = stdin,
                            stdout = subprocess.PIPE,
                            stderr = subprocess.PIPE)

    out, err = proc.communicate()

    if proc.returncode not in expected_retcodes:
        # when nothing gets to stderr, add stdout to Detail
        if err.strip() == '':
            err = out

        mesg  = 'Error [%d]: %s' % (proc.returncode, command)
        mesg += '\nDetail: %s' % err
        raise Exception, mesg

    return proc.returncode, out, err


Regards,
--
dim