Thread: New user: Windows, Postgresql, Python
Hi, I'm just starting to look at Postgresql. My platform (for better or worse) is Windows, and I'm quite interested in the pl/python support. However, when I run the binary installer, it is not offered to me as an option (it's there, but greyed out). The plpython.dll file is installed, however. When I check, it looks like plpython.dll is linked against Python 2.3. I have Python 2.4 installed on my PC, and I don't really want to downgrade. I suppose my first (lazy) question is, is there a Python 2.4 compatible plpython.dll available anywhere? Alternatively, is there a way I can build one for myself? I'm happy enough doing my own build (I have mingw and msys available), but I'd rather not build the whole of postgresql if possible, just for the sake of one DLL.... Thanks in advance, Paul. -- "Bother," said the Borg, "We've assimilated Pooh."
> Hi, > I'm just starting to look at Postgresql. My platform (for better or > worse) is Windows, and I'm quite interested in the pl/python support. > However, when I run the binary installer, it is not offered > to me as an option (it's there, but greyed out). The > plpython.dll file is installed, however. > > When I check, it looks like plpython.dll is linked against > Python 2.3. I have Python 2.4 installed on my PC, and I don't > really want to downgrade. > > I suppose my first (lazy) question is, is there a Python 2.4 > compatible plpython.dll available anywhere? Alternatively, is > there a way I can build one for myself? I'm happy enough > doing my own build (I have mingw and msys available), but I'd > rather not build the whole of postgresql if possible, just > for the sake of one DLL.... Not that I know of. IFF the libraries export the same entrypoints without changing things, you could try just copying "python24.dll" to "python23.dll". I don't know how the Python guys are with binary compatibility, though. Might be worth a shot. On a different note, can't you have both python 2.3 *and* 2.4 on the asme system? Considering they put the version number in the filename, it seems this should be possible? //Magnus
mha@sollentuna.net ("Magnus Hagander") writes:
>> I suppose my first (lazy) question is, is there a Python 2.4
>> compatible plpython.dll available anywhere? Alternatively, is
>> there a way I can build one for myself? I'm happy enough
>> doing my own build (I have mingw and msys available), but I'd
>> rather not build the whole of postgresql if possible, just
>> for the sake of one DLL....
>
> Not that I know of. IFF the libraries export the same entrypoints
> without changing things, you could try just copying "python24.dll" to
> "python23.dll". I don't know how the Python guys are with binary
> compatibility, though. Might be worth a shot.
As per my earlier posting, I actually found that building postgresql
wasn't at all hard. Once I'd built with Python 2.4 support, I had a
compatible plpython.dll I could just copy in.
I'm not sure renaming the Python DLL would have worked - it's
definitely frowned on...
> On a different note, can't you have both python 2.3 *and* 2.4 on the
> asme system? Considering they put the version number in the filename, it
> seems this should be possible?
I could, but I try to avoid this, as it involves double installs of
any extensions I want to use, or incompatible environments. More
laziness on my part, really :-)
Thanks for the suggestions,
Paul.
PS Thanks to the developers who made building postgresql on Windows
such an easy job! I was very impressed - I genuinely didn't think
that building such a large piece of software would be so hassle-free.
--
Never keep up with the Joneses. Drag them down to your level. --
Quentin Crisp
On Tue, Mar 15, 2005 at 07:05:22PM +0000, Paul Moore wrote: > As per my earlier posting, I actually found that building postgresql > wasn't at all hard. Once I'd built with Python 2.4 support, I had a > compatible plpython.dll I could just copy in. Pardon the interruption, but do you have a PostgreSQL server with PL/Python running on Windows? Have you been following the "plpython function problem workaround" thread? http://archives.postgresql.org/pgsql-general/2005-03/msg00599.php We (the thread participants) could use somebody with a Windows server to do some testing. Specifically, we're wondering if Python on Windows requires embedded Python code to have CRLF (\r\n) as a line ending, or if it requires (or at least permits) LF (\n) only. If you're able to help, could you could post the results of the following? CREATE FUNCTION pytest_lf() RETURNS integer AS 'x = 1\nreturn x\n' LANGUAGE plpythonu; CREATE FUNCTION pytest_crlf() RETURNS integer AS 'x = 1\r\nreturn x\r\n' LANGUAGE plpythonu; SELECT pytest_lf(); SELECT pytest_crlf(); With PostgreSQL 8.0.1, Python 2.4.1c1, and Solaris 9, I get this: test=# SELECT pytest_lf(); pytest_lf ----------- 1 (1 row) test=# SELECT pytest_crlf(); ERROR: plpython: could not compile function "pytest_crlf" DETAIL: exceptions.SyntaxError: invalid syntax (line 2) If you have the ability to compile standalone C programs with embedded Python, we'd also be interested in seeing what happens if you run the programs in the following messages: http://archives.postgresql.org/pgsql-general/2005-01/msg00876.php http://archives.postgresql.org/pgsql-general/2005-03/msg00630.php Any test results or comments you can provide would be appreciated. Thanks. -- Michael Fuhr http://www.fuhr.org/~mfuhr/
mike@fuhr.org (Michael Fuhr) writes: > We (the thread participants) could use somebody with a Windows > server to do some testing. Glad to help... This is with postgresql 8.0.1, Python 2.4. > Specifically, we're wondering if Python on Windows requires embedded > Python code to have CRLF (\r\n) as a line ending, or if it requires > (or at least permits) LF (\n) only. If you're able to help, could > you could post the results of the following? > > CREATE FUNCTION pytest_lf() RETURNS integer AS > 'x = 1\nreturn x\n' > LANGUAGE plpythonu; > > CREATE FUNCTION pytest_crlf() RETURNS integer AS > 'x = 1\r\nreturn x\r\n' > LANGUAGE plpythonu; > > SELECT pytest_lf(); > SELECT pytest_crlf(); > > With PostgreSQL 8.0.1, Python 2.4.1c1, and Solaris 9, I get this: > > test=# SELECT pytest_lf(); > pytest_lf > ----------- > 1 > (1 row) > > test=# SELECT pytest_crlf(); > ERROR: plpython: could not compile function "pytest_crlf" > DETAIL: exceptions.SyntaxError: invalid syntax (line 2) I get exactly the same results. > If you have the ability to compile standalone C programs with > embedded Python, we'd also be interested in seeing what happens if > you run the programs in the following messages: > > http://archives.postgresql.org/pgsql-general/2005-01/msg00876.php I get: >test1 What hath Guido wrought? > http://archives.postgresql.org/pgsql-general/2005-03/msg00630.php I get: >test2 > Initialized. > Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] > running: print 1 print 2 1 2 > end > running: print 1 print 2 File "<string>", line 1 print 1 ^ SyntaxError: invalid syntax > end > Finalized. I don't know if this helps? It seems reasonable to me - as far as Python C code is concerned, code strings should be \n-separated, just like in Unix. The only place CRLF is applicable is in code read from files, where the C runtime converts it to \n-delimited before the Python APIs see it (as far as I understand it, which isn't very far...) The long and short of it is that I believe you just use \n to delimit lines on Windows, just like anywhere else. Regards, Paul. -- SCSI is not magic. There are fundamental technical reasons why it is necessary to sacrifice a young goat to your SCSI chain now and then. -- John Woods
On Tue, Mar 15, 2005 at 10:46:09PM +0000, Paul Moore wrote: > The long and short of it is that I believe you just use \n to delimit > lines on Windows, just like anywhere else. Many thanks -- your test results contain the info we've been seeking. -- Michael Fuhr http://www.fuhr.org/~mfuhr/
On Wed, Mar 16, 2005 at 01:46:23PM +0100, Marco Colombo wrote: > > It seems python documentation is plain wrong, or I'm not able to > read it at all: > > http://docs.python.org/ref/physical.html > > "A physical line ends in whatever the current platform's convention is for > terminating lines. On Unix, this is the ASCII LF (linefeed) character. On > Windows, it is the ASCII sequence CR LF (return followed by linefeed). On > Macintosh, it is the ASCII CR (return) character." Perhaps the Python documentation could use some clarification about when the platform's convention is required and when it isn't. The "Embedding Python" documentation shows embedded code with lines ending in \n and without saying anything about requiring the platform's convention: http://docs.python.org/ext/high-level-embedding.html > This is the language _reference_ manual, btw. I'm very surprised to hear > python on windows is so broken. > > Anyway, that makes life simpler for us. plpython programs are \n separated, > no matter what platform the server runs on. That the behavior makes life simpler is an argument against it being broken (although it would be even less broken if it were more flexible about what line endings it allows). A detailed response would be getting off-topic for PostgreSQL, but I'll stand by what I said earlier: I would find it bizarre if embedded Python code had to use different line endings on different platforms. That would mean the programmer couldn't simply do this: PyRun_SimpleString("x = 1\n" "print x\n"); Instead, the programmer would have to do a compile-time or run-time check and build the string in a platform-dependent manner. What problem would the language be solving by requiring that? -- Michael Fuhr http://www.fuhr.org/~mfuhr/
On Wed, 16 Mar 2005, Michael Fuhr wrote: > On Wed, Mar 16, 2005 at 01:46:23PM +0100, Marco Colombo wrote: >> >> It seems python documentation is plain wrong, or I'm not able to >> read it at all: >> >> http://docs.python.org/ref/physical.html >> >> "A physical line ends in whatever the current platform's convention is for >> terminating lines. On Unix, this is the ASCII LF (linefeed) character. On >> Windows, it is the ASCII sequence CR LF (return followed by linefeed). On >> Macintosh, it is the ASCII CR (return) character." > > Perhaps the Python documentation could use some clarification about > when the platform's convention is required and when it isn't. > > The "Embedding Python" documentation shows embedded code with lines > ending in \n and without saying anything about requiring the > platform's convention: > > http://docs.python.org/ext/high-level-embedding.html > >> This is the language _reference_ manual, btw. I'm very surprised to hear >> python on windows is so broken. >> >> Anyway, that makes life simpler for us. plpython programs are \n separated, >> no matter what platform the server runs on. > > That the behavior makes life simpler is an argument against it being > broken (although it would be even less broken if it were more > flexible about what line endings it allows). broken == 'not conforming to the specifications or the documentation' The fact it helps us is just a side effect. > A detailed response > would be getting off-topic for PostgreSQL, but I'll stand by what > I said earlier: I would find it bizarre if embedded Python code had > to use different line endings on different platforms. That would > mean the programmer couldn't simply do this: > > PyRun_SimpleString("x = 1\n" > "print x\n"); > > Instead, the programmer would have to do a compile-time or run-time > check and build the string in a platform-dependent manner. What > problem would the language be solving by requiring that? This one: aprogram = "x = 1\nprint x\n"; printf(aprogram); PyRun_SimpleString(aprogram); See? THIS program requires compile-time or run-time checks. You can't run it on Windows, or Mac: it'll write garbage to the screen (something that looks like garbage, that is). Make it more general: aprogram = get_program_from_somewhere(); PyRun_SimpleString(aprogram); write_program_to_somefile_possibly_stdout(aprogram); What if get_program_from_somewhere() reads user input? On Windows lines will be \r\n separated. Now, should this program make platform checks? Why not simply read a file (or stdin) in text mode, and pass the result to PyRun_SimpleString()? The same applies to output, of course. Now something strikes me... in his tests, Paul tried my program and the output looks identical to Linux. Now... I was expecting program1 (the one with just \n) do display badly under Windows. Am I missing something? Does C runtime support in Windows convert \n into \r\n automatically in printf()? If so, I'm on the wrong track. It may do the same with scanf() and other stdio functions. I must say I wasn't expecting my program to run just fine, with all those \n I used in it. Staring from printf("> Initialized.\n"); Paul can you please tell me which compiler you used under Windows to complile my program and if you used some weird compiling options? TIA. .TM. -- ____/ ____/ / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _____/ _____/ _/ Colombo@ESI.it
On Wed, Mar 16, 2005 at 04:17:51PM +0100, Marco Colombo wrote: > > aprogram = "x = 1\nprint x\n"; > printf(aprogram); > PyRun_SimpleString(aprogram); > > See? THIS program requires compile-time or run-time checks. You > can't run it on Windows, or Mac: it'll write garbage to the screen > (something that looks like garbage, that is). Are you sure about that? It's been forever since I programmed in a Microsoft environment, but as I recall, I/O streams opened in "text mode" do automatic translations between \n and \r\n. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/HTML/_crt_fopen.2c_._wfopen.asp "Also, in text mode, carriage return-linefeed combinations are translated into single linefeeds on input, and linefeed characters are translated to carriage return-linefeed combinations on output." I didn't look up Mac behavior but I'd be surprised if it didn't offer the same "text mode" and "binary mode" behaviors. It's annoying that these platforms use different line endings, but at least their implementations of standard C libraries offer a way to hide that difference from the programmer. > Now something strikes me... in his tests, Paul tried my program and > the output looks identical to Linux. Now... I was expecting > program1 (the one with just \n) do display badly under Windows. > Am I missing something? Does C runtime support in Windows convert > \n into \r\n automatically in printf()? If so, I'm on the wrong track. > It may do the same with scanf() and other stdio functions. I think that's exactly what happens with I/O streams in "text mode." -- Michael Fuhr http://www.fuhr.org/~mfuhr/
On Wed, 16 Mar 2005, Michael Fuhr wrote:
> On Wed, Mar 16, 2005 at 04:17:51PM +0100, Marco Colombo wrote:
>>
>> aprogram = "x = 1\nprint x\n";
>> printf(aprogram);
>> PyRun_SimpleString(aprogram);
>>
>> See? THIS program requires compile-time or run-time checks. You
>> can't run it on Windows, or Mac: it'll write garbage to the screen
>> (something that looks like garbage, that is).
>
> Are you sure about that? It's been forever since I programmed in
> a Microsoft environment, but as I recall, I/O streams opened in
> "text mode" do automatic translations between \n and \r\n.
No I wasn't sure and I actually was wrong. I've never programmed under
Windows. I've just learned something.
Apparently, as far as Python is concerned, the platform presents \n
at C level, so it makes sense for PyRun_SimpleString() to expect \n
as line terminator. Still I don't understand when the lexxer would
use \r\n as pysical line ending on Windows, but I can live with it. :-)
It seems that any client application under Windows is likely to use
only \n-delimited text, as long as it uses stdio functions and text
mode. Problems arise when it gets text from some other source. But since
at C level text is expected to be \n-delimited, the application should
take care of the conversion as soon as it receives the data.
I think that if we want to be conservative, any input that is supposed
to be treated (actively) as text by the server, should be \n-delimited.
That includes any function source.
I'm against to any on-the-fly conversion, now.
I don't like the idea of PostgreSQL accepting input in one form
(\r\n) and providing output in a different form (\n). Also think of
a function definition with mixed \r\n and \n lines: we'd have no way
to reconstruct the original input. I think we should just state that
text used for function definitions is \n-delimited. Some languages may
accept \r\n as well, but that's undocumented side effect, and bad practice.
Now that I learned that C programs on Windows are expected to handle
\n-delimited text, I can't think of any reason why an application should
send \r\n-delimited text via libpq as a function definition, unless
the programmer forgot to perform the "standard" \r\n to \n conversion
somewhere.
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
[I've changed the Subject back to the thread that started this
discussion.]
On Wed, Mar 16, 2005 at 05:52:02PM +0100, Marco Colombo wrote:
> I'm against to any on-the-fly conversion, now.
> I don't like the idea of PostgreSQL accepting input in one form
> (\r\n) and providing output in a different form (\n). Also think of
> a function definition with mixed \r\n and \n lines: we'd have no way
> to reconstruct the original input.
Yeah, that's a reasonable argument against modifying the function
source code before storing it in pg_proc. But I expect this problem
will come up again, and some people might not care about being able
to reconstruct the original input if it's just a matter of stripped
carriage returns, especially if the function logic doesn't use
literal carriage return characters that would be missed. For those
people, the validator hack might be an acceptable way to deal with
a client interface that inserts carriage returns that the programmer
didn't intend anyway. Not necessarily as part of the core PostgreSQL
code or even distributed with PostgreSQL, but as something they
could install if they wanted to.
> I think we should just state that text used for function definitions
> is \n-delimited. Some languages may accept \r\n as well, but that's
> undocumented side effect, and bad practice.
Whether it's an "undocumented side effect" depends on the language,
and whether it's bad practice is a matter of opinion. In any case,
that's the language's concern and not something PostgreSQL should
judge or enforce. PostgreSQL shouldn't have to know or care about a
procedural language's syntax -- a function's source code should be an
opaque object that PostgreSQL stores and passes to the language's
handler without caring about its contents. Syntax enforcement should
be in the language's validator or handler according to the language's
own rules.
Speaking of code munging and syntax enforcement, have a look at this:
CREATE FUNCTION foo() RETURNS text AS $$
return """line 1
line 2
line 3
"""
$$ LANGUAGE plpythonu;
SELECT foo();
foo
--------------------------
line 1
line 2
line 3
(1 row)
Eh? Where'd those leading tabs come from? Why, they came from
PLy_procedure_munge_source() in src/pl/plpython/plpython.c:
mrc = PLy_malloc(mlen);
plen = snprintf(mrc, mlen, "def %s():\n\t", name);
Assert(plen >= 0 && plen < mlen);
sp = src;
mp = mrc + plen;
while (*sp != '\0')
{
if (*sp == '\n')
{
*mp++ = *sp++;
*mp++ = '\t';
}
else
*mp++ = *sp++;
}
*mp++ = '\n';
*mp++ = '\n';
*mp = '\0';
How about them apples? The PL/Python handler is already doing some
fixup behind the scenes (and potentially causing problems, as the
example illustrates).
--
Michael Fuhr
http://www.fuhr.org/~mfuhr/
On Wed, 16 Mar 2005, Michael Fuhr wrote:
> [I've changed the Subject back to the thread that started this
> discussion.]
>
> On Wed, Mar 16, 2005 at 05:52:02PM +0100, Marco Colombo wrote:
>
>> I'm against to any on-the-fly conversion, now.
>> I don't like the idea of PostgreSQL accepting input in one form
>> (\r\n) and providing output in a different form (\n). Also think of
>> a function definition with mixed \r\n and \n lines: we'd have no way
>> to reconstruct the original input.
>
> Yeah, that's a reasonable argument against modifying the function
> source code before storing it in pg_proc. But I expect this problem
> will come up again, and some people might not care about being able
> to reconstruct the original input if it's just a matter of stripped
> carriage returns, especially if the function logic doesn't use
> literal carriage return characters that would be missed. For those
> people, the validator hack might be an acceptable way to deal with
> a client interface that inserts carriage returns that the programmer
> didn't intend anyway. Not necessarily as part of the core PostgreSQL
> code or even distributed with PostgreSQL, but as something they
> could install if they wanted to.
Agreed.
>> I think we should just state that text used for function definitions
>> is \n-delimited. Some languages may accept \r\n as well, but that's
>> undocumented side effect, and bad practice.
>
> Whether it's an "undocumented side effect" depends on the language,
> and whether it's bad practice is a matter of opinion.
Sure. I mean, we may just state that, per spec. Program data
should be \n-delimeted, full stop. It sounds sensible to me.
Just put it somewhere in the docs, problem solved. We're loosing
nothing. I'm just proposing to add that to the docs/specs.
> In any case,
> that's the language's concern and not something PostgreSQL should
> judge or enforce. PostgreSQL shouldn't have to know or care about a
> procedural language's syntax -- a function's source code should be an
> opaque object that PostgreSQL stores and passes to the language's
> handler without caring about its contents. Syntax enforcement should
> be in the language's validator or handler according to the language's
> own rules.
That's what we do now. My point being it's not our job to "fix" data
coming from the client. If a client app creates a plpython function
the wrong way, fix it. Why should we place a paperbag on a client bug?
> Speaking of code munging and syntax enforcement, have a look at this:
>
> CREATE FUNCTION foo() RETURNS text AS $$
> return """line 1
> line 2
> line 3
> """
> $$ LANGUAGE plpythonu;
>
> SELECT foo();
> foo
> --------------------------
> line 1
> line 2
> line 3
>
> (1 row)
>
> Eh? Where'd those leading tabs come from? Why, they came from
> PLy_procedure_munge_source() in src/pl/plpython/plpython.c:
>
> mrc = PLy_malloc(mlen);
> plen = snprintf(mrc, mlen, "def %s():\n\t", name);
> Assert(plen >= 0 && plen < mlen);
>
> sp = src;
> mp = mrc + plen;
>
> while (*sp != '\0')
> {
> if (*sp == '\n')
> {
> *mp++ = *sp++;
> *mp++ = '\t';
> }
> else
> *mp++ = *sp++;
> }
> *mp++ = '\n';
> *mp++ = '\n';
> *mp = '\0';
>
> How about them apples? The PL/Python handler is already doing some
> fixup behind the scenes (and potentially causing problems, as the
> example illustrates).
OMG! It's indenting the funtion body. I think you can't do that
w/o being syntax-aware. I'm not familiar with the code, why is it
adding a 'def' in front of it at all? I undestand that once you do
it you'll have to shift the code by an indentation level.
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
On Thu, Mar 17, 2005 at 01:03:36PM +0100, Marco Colombo wrote: > OMG! It's indenting the funtion body. I think you can't do that > w/o being syntax-aware. I'm not familiar with the code, why is it > adding a 'def' in front of it at all? I undestand that once you do > it you'll have to shift the code by an indentation level. Presumbly because it wants to create a function, which can later be called. Since python is sensetive to whitespace it has to indent the code to make it work. There was an example on the web somewhere (the link has been posted to this list) of a peice of python which you can load into the interpreter which will allow it to accept \r\n terminated lines. I don't recall if anyone actually tried it out or not... Won't fix the indenting problem though... -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a > tool for doing 5% of the work and then sitting around waiting for someone > else to do the other 95% so you can sue them.
Attachment
Martijn van Oosterhout <kleptog@svana.org> writes:
> On Thu, Mar 17, 2005 at 01:03:36PM +0100, Marco Colombo wrote:
>> OMG! It's indenting the funtion body. I think you can't do that
>> w/o being syntax-aware. I'm not familiar with the code, why is it
>> adding a 'def' in front of it at all? I undestand that once you do
>> it you'll have to shift the code by an indentation level.
> Presumbly because it wants to create a function, which can later be
> called. Since python is sensetive to whitespace it has to indent the
> code to make it work.
Seems like we have to upgrade that thing to have a complete
understanding of Python lexical rules --- at least enough to know where
the line boundaries are. Which is pretty much exactly the same as
knowing which CRs to strip out. So I guess we have a candidate place
for a solution.
Anyone want to code it up? I don't know enough Python to do it ...
regards, tom lane
pgsql@esiway.net (Marco Colombo) writes: > No I wasn't sure and I actually was wrong. I've never programmed under > Windows. I've just learned something. Indeed, the Windows C runtime translates CRLF to \n on input, and \n to CRLF on output, for files in "text" mode. Unix programmers tend not to be aware of the distinction between text and binary modes (it's actually in standard C) as it makes no difference on Unix. But it does on Windows (and possibly other platforms). <offtopic> Ironically, at the lowest level, Windows behaves just like Unix (files are pure byte streams) - it's only in the C runtime and application code that CRLF issues arise, and that's a backward-compatibility hack dating back to the days of MS-DOS. </offtopic> > Apparently, as far as Python is concerned, the platform presents \n > at C level, so it makes sense for PyRun_SimpleString() to expect \n > as line terminator. Still I don't understand when the lexxer would > use \r\n as pysical line ending on Windows, but I can live with it. :-) Internally, Python uses C string semantics, where \n represents a newline. Recent versions of Python have "universal newline" support, which in the broadest sense attempts to be forgiving over line endings, and treat LF, CRLF, and even bare CR, as line endings. I don't know exactly where it applies, though, so I believe the most sensible approach is to always use \n (LF) in strings passed to Python APIs. This is essentially the "be conservative in what you send" philosophy. Paul. -- A little inaccuracy sometimes saves tons of explanation -- Saki
On Thursday 17 March 2005 23:17, Paul Moore wrote: > <offtopic> > Ironically, at the lowest level, Windows behaves just like Unix > (files are pure byte streams) - it's only in the C runtime and > application code that CRLF issues arise, and that's a > backward-compatibility hack dating back to the days of MS-DOS. > </offtopic> Even more offtopic: Actually, the CR/LF pair dates back to the ancient teletype writers, which needed one character for the right-to-left movement of the paper carriage (hence the literal meaning of "Carriage Return"), and one for the vertical movement. I believe it was Tom Swan who, in his "Programming Turbo Pascal" from the eighties, said something to the effect that "this is not only a case of the tail wagging the dog, but a tail that keeps on wagging twenty years after the dog has rolled over and died." Sorry-for-spinning-of-on-a-tangent-ly yours - -- Leif Biberg Kristensen http://solumslekt.org/
On Thu, Mar 17, 2005 at 10:49:24AM -0500, Tom Lane wrote:
> Seems like we have to upgrade that thing to have a complete
> understanding of Python lexical rules --- at least enough to know where
> the line boundaries are. Which is pretty much exactly the same as
> knowing which CRs to strip out. So I guess we have a candidate place
> for a solution.
>
> Anyone want to code it up? I don't know enough Python to do it ...
[Sound of crickets]
More pabulum for pondering:
% cat -v foo.py
print '''line 1^M
line^M2^M
line 3^M
'''^M
% python foo.py | cat -v
line 1
line
2
line 3
% cat -v bar.py
print 'line 1^M'
% python bar.py
File "bar.py", line 1
print 'line 1
^
SyntaxError: EOL while scanning single-quoted string
Line-ending CRs stripped, even inside quotes; mid-line CRs converted
to LF. Tests done with Python 2.4 on FreeBSD 4.11-STABLE; I wonder
what Python on Windows would do. If it behaves the same way, then
a munging algorithm might be CRLF => LF, otherwise CR => LF. Or
we could take Marco's suggestion and do nothing, putting the burden
on the client to send the right thing.
That doesn't address the indentation munging, though. That appears
to be a matter of knowing whether you're inside a quote or not when
a LF appears.
--
Michael Fuhr
http://www.fuhr.org/~mfuhr/
Michael Fuhr <mike@fuhr.org> writes:
> Line-ending CRs stripped, even inside quotes; mid-line CRs converted
> to LF. Tests done with Python 2.4 on FreeBSD 4.11-STABLE; I wonder
> what Python on Windows would do.
Unfortunately, I don't think that proves anything, because according
to earlier discussion Python will do newline-munging when it reads
a file (including a script file). The question that we have to deal
with is what are the rules for a string fed to PyRun_String ... and it
seems those rules are not the same.
regards, tom lane
On Thu, Mar 17, 2005 at 09:48:51PM -0500, Tom Lane wrote: > Michael Fuhr <mike@fuhr.org> writes: > > Line-ending CRs stripped, even inside quotes; mid-line CRs converted > > to LF. Tests done with Python 2.4 on FreeBSD 4.11-STABLE; I wonder > > what Python on Windows would do. > > Unfortunately, I don't think that proves anything, because according > to earlier discussion Python will do newline-munging when it reads > a file (including a script file). The question that we have to deal > with is what are the rules for a string fed to PyRun_String ... and it > seems those rules are not the same. I was curious about how Python's munging works with quotes that span lines, i.e., when the CRs and LFs might be considered part of a quoted string. Apparently any CR or LF is considered a line ending in an ordinary Python script, with CR and CRLF normalized to LF before being passed to the interpreter, so I'm thinking that a Python programmer wouldn't expect to be able to embed CRs in a string literal and have them remain unchanged. If that's the case, then concerns about CR conversions potentially messing up a user's strings might be unfounded. PL/Python currently treats the function source as a string that's passed unchanged (except for the added "def" and indentation) to PyRun_String. But that's an implementation detail that the user shouldn't have to care about: I'm wondering if, instead, PL/Python should treat the function source as Python would treat a file and do the same conversions that Python would, namely CRLF => LF and lone CR => LF. That should solve the complaints, and it should be justifiable as more than just a hack: PL/Python would simply be doing the same thing that Python would do if it had read the source from a file. That might even be less surprising than the current behavior. Marco, you've stated that you're against munging the code because "it's not our job to 'fix' data coming from the client." But I'm suggesting that we think about the code in a different way than the current implementation does: not as a literal that we pass untouched to the Python interpreter, but rather as code that Python would munge anyway if it had read that code from a file. We could still store the code exactly as received and have the language handler munge it on the fly, as we've discovered it's already doing. Comments? Have I overlooked anything? Could munging CRs have effects that a Python programmer wouldn't expect if the same code had been read from a file? Since it mimics Python's own behavior with code read from a file, can anybody justify not doing it? -- Michael Fuhr http://www.fuhr.org/~mfuhr/
Michael Fuhr <mike@fuhr.org> writes: > Apparently any CR or LF is considered a line > ending in an ordinary Python script, with CR and CRLF normalized > to LF before being passed to the interpreter, so I'm thinking that > a Python programmer wouldn't expect to be able to embed CRs in a > string literal and have them remain unchanged. If that's the case, > then concerns about CR conversions potentially messing up a user's > strings might be unfounded. Yeah, it looks like you are right: http://cvs.sourceforge.net/viewcvs.py/python/python/nondist/peps/pep-0278.txt?rev=1.2 So that part of it can be solved fairly easily. Unfortunately we are still left with the issue of having to lex Python programs enough to know how to indent them ... regards, tom lane
On Fri, Mar 18, 2005 at 12:35:07AM -0500, Tom Lane wrote: > Michael Fuhr <mike@fuhr.org> writes: > > then concerns about CR conversions potentially messing up a user's > > strings might be unfounded. > > Yeah, it looks like you are right: > > http://cvs.sourceforge.net/viewcvs.py/python/python/nondist/peps/pep-0278.txt?rev=1.2 > > So that part of it can be solved fairly easily. Should I submit a patch? It should be only a few additional lines in PLy_procedure_munge_source(). Would you apply it only to HEAD, or would it be considered a bug fix that REL8_0_STABLE could get as well? It might be nice to have it in 8.0.2, whenever that comes out. > Unfortunately we are still left with the issue of having to lex > Python programs enough to know how to indent them ... Yeah, that one's a little extra work, but I don't think anybody's complained about it yet. So far I'm thinking it only affects multi-line strings as in the example I posted, and even then it might not matter unless you use that string as a column value or return it to the caller. If you're doing something else like building a query statement, then the bogus indentation is probably irrelevant. It ought to be fixed, but it doesn't seem to be squeaking very loudly. -- Michael Fuhr http://www.fuhr.org/~mfuhr/
On Thu, 17 Mar 2005, Tom Lane wrote:
> Martijn van Oosterhout <kleptog@svana.org> writes:
>> On Thu, Mar 17, 2005 at 01:03:36PM +0100, Marco Colombo wrote:
>>> OMG! It's indenting the funtion body. I think you can't do that
>>> w/o being syntax-aware. I'm not familiar with the code, why is it
>>> adding a 'def' in front of it at all? I undestand that once you do
>>> it you'll have to shift the code by an indentation level.
>
>> Presumbly because it wants to create a function, which can later be
>> called. Since python is sensetive to whitespace it has to indent the
>> code to make it work.
>
> Seems like we have to upgrade that thing to have a complete
> understanding of Python lexical rules --- at least enough to know where
> the line boundaries are. Which is pretty much exactly the same as
> knowing which CRs to strip out. So I guess we have a candidate place
> for a solution.
>
> Anyone want to code it up? I don't know enough Python to do it ...
I'm no expert but I'll look into it. Unless someone else already
tried it, I want to investigate first if it's possible to create
a callable object w/o using 'def', which alters the name space and
(the thing we're interested to) needs an extra identation level.
At first sight, what we do now (at function creation time) is:
1) execute a function definition;
2) compile a function call, and save the resulting code object for later use.
I'm wondering if we can save one step, and use a python callable object.
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
On Thu, 17 Mar 2005, Michael Fuhr wrote:
> On Thu, Mar 17, 2005 at 09:48:51PM -0500, Tom Lane wrote:
>> Michael Fuhr <mike@fuhr.org> writes:
>>> Line-ending CRs stripped, even inside quotes; mid-line CRs converted
>>> to LF. Tests done with Python 2.4 on FreeBSD 4.11-STABLE; I wonder
>>> what Python on Windows would do.
>>
>> Unfortunately, I don't think that proves anything, because according
>> to earlier discussion Python will do newline-munging when it reads
>> a file (including a script file). The question that we have to deal
>> with is what are the rules for a string fed to PyRun_String ... and it
>> seems those rules are not the same.
>
> Marco, you've stated that you're against munging the code because
> "it's not our job to 'fix' data coming from the client." But I'm
> suggesting that we think about the code in a different way than the
> current implementation does: not as a literal that we pass untouched
> to the Python interpreter, but rather as code that Python would
> munge anyway if it had read that code from a file. We could still
> store the code exactly as received and have the language handler
> munge it on the fly, as we've discovered it's already doing.
>
> Comments? Have I overlooked anything? Could munging CRs have
> effects that a Python programmer wouldn't expect if the same code
> had been read from a file? Since it mimics Python's own behavior
> with code read from a file, can anybody justify not doing it?
If you put it that way, I'm 100% with you.
Just let me suggest not to mimic its behaviour, but to use the
Real Thing if we manage to. That is, directly use the Universal Line
Support code provided by python itself, so that we don't even have
to think about being compatible.
Unfortunately, I'm new to python embedding. I think I'll ask on
python lists about the function definition code. Actually, we are
kind of a corner case: we ask the user to provide the function
body w/o the initial def declaration. We're treating partial, incomplete
python code and not a well-formed program, so we have to munge it
anyway. I have no idea if and how the python C API lets you
control such low level details. I think what we really want is to
create a callable (anonymous) object from the source of its "body".
I'm experimenting a bit, trying to write a PLy_procedure_compile()
that does not require source munging. I'm aiming at removing the
need for extra indentation. The \r\n thing is another beast, and I'm
not sure it belongs to the same place in our code.
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
On Fri, Mar 18, 2005 at 11:34:46AM +0100, Marco Colombo wrote: > Just let me suggest not to mimic its behaviour, but to use the > Real Thing if we manage to. That is, directly use the Universal Line > Support code provided by python itself, so that we don't even have > to think about being compatible. Sounds good if the Python API provides the hooks for doing so. I had started looking into that but didn't spent much time on it. > I'm experimenting a bit, trying to write a PLy_procedure_compile() > that does not require source munging. I'm aiming at removing the > need for extra indentation. Sounds good too, if that'll work. Looking forward to seeing what you find out. -- Michael Fuhr http://www.fuhr.org/~mfuhr/
Michael Fuhr <mike@fuhr.org> writes:
> On Fri, Mar 18, 2005 at 12:35:07AM -0500, Tom Lane wrote:
>> So that part of it can be solved fairly easily.
> Should I submit a patch? It should be only a few additional lines
> in PLy_procedure_munge_source(). Would you apply it only to HEAD,
> or would it be considered a bug fix that REL8_0_STABLE could get
> as well? It might be nice to have it in 8.0.2, whenever that comes
> out.
I think it would be reasonable to back-patch a small fix to convert CRLF.
The sort of rewrite Marco is considering, I wouldn't back-patch.
regards, tom lane
On Fri, 18 Mar 2005, Tom Lane wrote:
> Michael Fuhr <mike@fuhr.org> writes:
>> On Fri, Mar 18, 2005 at 12:35:07AM -0500, Tom Lane wrote:
>>> So that part of it can be solved fairly easily.
>
>> Should I submit a patch? It should be only a few additional lines
>> in PLy_procedure_munge_source(). Would you apply it only to HEAD,
>> or would it be considered a bug fix that REL8_0_STABLE could get
>> as well? It might be nice to have it in 8.0.2, whenever that comes
>> out.
>
> I think it would be reasonable to back-patch a small fix to convert CRLF.
> The sort of rewrite Marco is considering, I wouldn't back-patch.
>
> regards, tom lane
Short update: it isn't possible to get a callable object directly from
the source provided by the user, since 'return' is not valid outside
function definitions in Python.
Right now I'm parsing the string first, changing the resulting
parse tree adding missing nodes (def, INDENT, DEINDENT) and
then compiling it. Grammar definitions for a python function is:
funcdef: [decorators] 'def' NAME parameters ':' suite
suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT
What we get from the users is stmt+ (a sequence of stmt). The INDENT
and DEDENT tokens are generated by the parser only when indentation
level _changes_.
My plan is to generate two parse trees, one from this code
(with the right fname of course):
def fname():
pass
and one from the function definition. Then, we "attach" the root
of the second tree where the "pass" node is in the first tree.
We should get a parse tree ready for compilation.
I wish I could "push" the right tokens in the right places,
but it seems it's not possible.
Stay tuned.
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
Marco Colombo <pgsql@esiway.net> writes:
> Right now I'm parsing the string first, changing the resulting
> parse tree adding missing nodes (def, INDENT, DEINDENT) and
> then compiling it.
Hmmm ... is this really going to be simpler or more robust than lexing
the string carefully enough to insert tabs at the right places? The
impression I had so far was that you'd only need to understand about
Python's string-literal conventions to get that right ... and that's
something that's not likely to change. I'm not so sure that parse
trees can be regarded as an immutable API.
regards, tom lane
On Fri, Mar 18, 2005 at 10:33:01AM -0500, Tom Lane wrote: > > I think it would be reasonable to back-patch a small fix to convert CRLF. > The sort of rewrite Marco is considering, I wouldn't back-patch. I just submitted a small patch to convert CRLF => LF, CR => LF. -- Michael Fuhr http://www.fuhr.org/~mfuhr/
On Fri, 18 Mar 2005, Tom Lane wrote: > Marco Colombo <pgsql@esiway.net> writes: >> Right now I'm parsing the string first, changing the resulting >> parse tree adding missing nodes (def, INDENT, DEINDENT) and >> then compiling it. > > Hmmm ... is this really going to be simpler or more robust than lexing > the string carefully enough to insert tabs at the right places? The > impression I had so far was that you'd only need to understand about > Python's string-literal conventions to get that right ... and that's > something that's not likely to change. I'm not so sure that parse > trees can be regarded as an immutable API. > > regards, tom lane I've completed a proof of concept, I think I can answer: - simpler? not at all. It requires understanding of how the parser works. The whole thing is about 50 lines long, but quite a bit of parser magic is going on. And I'm far from the point I can be confident about it doing always the right thing. I still have to handle (de)allocations correctly. - more robust - yes. The only way to make sure we're lexing the string the same way python does is to use its lexer. Every single difference however subtle would call for a bug. And it's re-invening the wheel. But there's no way you can work at lexer level that I'm aware of. That is, to add tokens before sending them to the parser. So you have to work on the parser output. - I have no idea if the "node" API is immutable at all. For sure, the interface I'm using is one or two levels below the current one, and yes, it's more likely to change. I share your concerns here. If our problem is only string literals, maybe we can handle them with a dedicated lexer. Python string literals are quite complex (compared to other languages): http://docs.python.org/ref/strings.html but not that hard. Still, my first concern is that one day we find another corner case in python syntax that makes our "blind" tab adding fail. And we're back to square one. BTW, I'm not preparing a patch for now, I'm working with a test program. As soon as I finish it, either I'll post it or prepare a patch against plpython.c, for consideration. I won't say it is ready for inclusion until someone else more knowledgeable than me on both PostgreSQL and python embedding looks at it, anyway. .TM. -- ____/ ____/ / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _____/ _____/ _/ Colombo@ESI.it
Michael Fuhr wrote: > On Tue, Mar 15, 2005 at 10:46:09PM +0000, Paul Moore wrote: > > >>The long and short of it is that I believe you just use \n to delimit >>lines on Windows, just like anywhere else. > > > Many thanks -- your test results contain the info we've been seeking. > Thanks a lot Paul. Micheal, you were right. It seems python documentation is plain wrong, or I'm not able to read it at all: http://docs.python.org/ref/physical.html "A physical line ends in whatever the current platform's convention is for terminating lines. On Unix, this is the ASCII LF (linefeed) character. On Windows, it is the ASCII sequence CR LF (return followed by linefeed). On Macintosh, it is the ASCII CR (return) character." This is the language _reference_ manual, btw. I'm very surprised to hear python on windows is so broken. Anyway, that makes life simpler for us. plpython programs are \n separated, no matter what platform the server runs on. Client applications just need to conply, which is what I suggested some time ago. I'm glad to hear there's nothing to change on the server side. .TM. -- ____/ ____/ / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _____/ _____/ _/ Colombo@ESI.it
marco@esi.it (Marco Colombo) writes: > It seems python documentation is plain wrong, or I'm not able to > read it at all: > > http://docs.python.org/ref/physical.html > > "A physical line ends in whatever the current platform's convention is for > terminating lines. On Unix, this is the ASCII LF (linefeed) character. On > Windows, it is the ASCII sequence CR LF (return followed by linefeed). On > Macintosh, it is the ASCII CR (return) character." > > This is the language _reference_ manual, btw. I'm very surprised to hear > python on windows is so broken. I believe this is wrong in two ways - first, it hasn't been updated to cater for the recent "Universal Newline" support, and second, it applies only to Python source code files (embedded code using the C APIs should pass code using C newline conventions, ie \n characters, as we have confirmed). I've submitted a Python bug report (SF ref 1167922) against the documentation. I've suggested updated wording for this section as follows: """ A physical line is a sequence of characters terminated by an end-of-line sequence. In source files, any of the standard platform line termination sequences can be used - the \UNIX form using \ASCII{} LF (linefeed), the Windows form using the \ASCII{} sequence CR LF (return followed by linefeed), or the Macintosh form using the \ASCII{} CR (return) character. All of these forms can be used equally, regardless of platform. When embedding Python, source code strings should be passed to Python APIs using the standard C conventions for newline characters (the \code{\e n} character, representing \ASCII{} LF, is the line terminator). """ Is that clearer? Paul. -- Once the game is over, the King and the pawn go back in the same box. -- Italian Proverb
I'm OWNER of the db but I get `permission denied` when doing updating table pg_class ???? Any help appreciated
From
David Gagnon
Date:
Hi all, I just created a new db wich userX is owner. I log via pgAdminIII with the same user but I can't update the pg_class. UPDATE pg_class SET reltriggers = 0 WHERE relname = 'ic' I get:ERROR: permission denied for relation pg_class I do that on my dev env. The only difference I saw beetween users is that my DEV user as priviledge to create database(But it shouln't matter...?!) We create the db with : createdb -O userX -E UNICODE webCatalogTest Thanks for your help! /David
David Gagnon wrote: > Hi all, > > I just created a new db wich userX is owner. I log via pgAdminIII with > the same user but I can't update the pg_class. > > UPDATE pg_class SET reltriggers = 0 WHERE relname = 'ic' > > I get:ERROR: permission denied for relation pg_class > > I do that on my dev env. The only difference I saw beetween users is > that my DEV user as priviledge to create database(But it shouln't > matter...?!) I'm guessing your dev user is a superuser, and your other user isn't. Don't forget pg_class isn't in your database, it's shared by all. -- Richard Huxton Archonet Ltd
David Gagnon wrote: > Hi all, > > I just created a new db wich userX is owner. I log via pgAdminIII > with the same user but I can't update the pg_class. You are a datdba but not a superuser :). You have to be a super user to update pg_class. Sincerely, Joshua D. Drake > > UPDATE pg_class SET reltriggers = 0 WHERE relname = 'ic' > > I get:ERROR: permission denied for relation pg_class > > I do that on my dev env. The only difference I saw beetween users is > that my DEV user as priviledge to create database(But it shouln't > matter...?!) > > We create the db with : createdb -O userX -E UNICODE webCatalogTest > > Thanks for your help! > > /David > > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) -- Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC Postgresql support, programming shared hosting and dedicated hosting. +1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com PostgreSQL Replicator -- production quality replication for PostgreSQL
Attachment
Thanks It's the problem. /David Richard Huxton wrote: > David Gagnon wrote: > >> Hi all, >> >> I just created a new db wich userX is owner. I log via pgAdminIII >> with the same user but I can't update the pg_class. >> >> UPDATE pg_class SET reltriggers = 0 WHERE relname = 'ic' >> >> I get:ERROR: permission denied for relation pg_class >> >> I do that on my dev env. The only difference I saw beetween users is >> that my DEV user as priviledge to create database(But it shouln't >> matter...?!) > > > I'm guessing your dev user is a superuser, and your other user isn't. > Don't forget pg_class isn't in your database, it's shared by all. >
On Wed, Mar 23, 2005 at 04:07:59PM +0000, Richard Huxton wrote:
> Don't forget pg_class isn't in your database, it's shared by all.
Each database has its own pg_class:
SELECT relname, relkind, relisshared
FROM pg_class
WHERE relname = 'pg_class';
relname | relkind | relisshared
----------+---------+-------------
pg_class | r | f
(1 row)
You can query pg_class to see that its contents are different in
different databases, and you can use "ls -li" on the on-disk files
to see that they have different inode numbers and (usually) different
sizes and modified times.
Here are the shared objects in an 8.0.1 database (excluding indexes):
SELECT relname, relkind
FROM pg_class
WHERE relkind <> 'i' AND relisshared IS TRUE
ORDER BY relname;
relname | relkind
---------------+---------
pg_database | r
pg_group | r
pg_shadow | r
pg_tablespace | r
pg_toast_1260 | t
pg_toast_1261 | t
pg_toast_1262 | t
pg_xactlock | s
(8 rows)
--
Michael Fuhr
http://www.fuhr.org/~mfuhr/
Michael Fuhr wrote: > On Wed, Mar 23, 2005 at 04:07:59PM +0000, Richard Huxton wrote: > > >>Don't forget pg_class isn't in your database, it's shared by all. > > > Each database has its own pg_class: You're quite right Michael, I'm talking rubbish. Why is it always when I don't bother to read what I'm writing that I spout nonsense? Ho hum. -- Richard Huxton Archonet Ltd
On Fri, Mar 18, 2005 at 10:12:05PM -0700, Michael Fuhr wrote: > > I just submitted a small patch to convert CRLF => LF, CR => LF. This patch is in 8.0.2beta1, so PL/Python users might want to test it before 8.0.2 is released. See the recent "8.0.2 Beta Available" announcement: http://archives.postgresql.org/pgsql-general/2005-03/msg01311.php -- Michael Fuhr http://www.fuhr.org/~mfuhr/
The only ?issue? that I have found with it is similar to an issue I
posted about multiline in general, which does not seem to be
considered a bug.
I would say if it is documented that any newlines in a python
function, including embedded newlines, are *NIX newlines no matter
what operating system the function is created on, that would suffice.
As an example - Windows PGAdmin client. Linux Server:
create table test1(f1 varchar(50));
insert into test1 values('this is a multi line string
line2
line3
')
select * from test1 where f1='this is a multi line string
line2
line3
'
--returns 1 row
create or replace function testnewlines() returns int as
$$
x=plpy.execute("""select f1 from test1 where f1='this is a multi line string\r\nline2\r\nline3\r\n'""")
return x.nrows()
$$ language 'plpythonu'
--returns 1
create or replace function testnewlines() returns int as
$$
x=plpy.execute("""select f1 from test1 where f1='this is a multi line string
line2
line3
'""")
return x.nrows()
$$ language 'plpythonu'
--returns 0
Thank You
Sim Zacks
________________________________________________________________________________
On Fri, Mar 18, 2005 at 10:12:05PM -0700, Michael Fuhr wrote:
>
> I just submitted a small patch to convert CRLF => LF, CR => LF.
This patch is in 8.0.2beta1, so PL/Python users might want to test
it before 8.0.2 is released. See the recent "8.0.2 Beta Available"
announcement:
http://archives.postgresql.org/pgsql-general/2005-03/msg01311.php
--
Michael Fuhr
http://www.fuhr.org/~mfuhr/
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
On Tue, 29 Mar 2005, Sim Zacks wrote:
> The only ?issue? that I have found with it is similar to an issue I
> posted about multiline in general, which does not seem to be
> considered a bug.
I've posted similar concerns in the past. The whole point is that
there are two possible approaches:
1) treat text as binary - as we do now;
2) do on the wire conversion - like FTP ASCII mode.
Both have disadvantages, and both lead to unexpected results.
As I wrote before, 2) is more problematic. You'll have to reject
any file with a bare \n from a Windows, otherwise you won't be able
to process it correclty.
I think if you do:
insert into test (sometext) values ('Line one\nLine two\r\n');
-- with the literal chars, not the escape sequences
you're expecting exaclty the same on output. If the server
converts it in the Unix form:
'Line one\nLine two\n'
for storing and the converts back to the Windows form, when you do:
select sometext from test; -- from a Windows client
you get:
Line one\r\nLine two\r\n
which is not the same you entered.
I doubt FTP ASCII mode handles this correctly.
As for the examples you made (the python functions), it's a problem
with python string literals (just don't use them). Let's try this:
---------- CUT HERE 8< ------------
#!/usr/bin/env python
import pgdb
db = pgdb.connect()
curs = db.cursor()
# this is only to emulate PGAdmin under Windows (I don't have it)
# (I ran the script with these uncommented on Linux)
#q = "create temp table test1(f1 varchar(50));"
#curs.execute(q)
#q = "insert into test1 values('this is a multi line string\r\nline2\r\nline3\r\n');"
#curs.execute(q)
#### real test
# embedded in string literal
q1 = """select count(f1) from test1 where f1 = 'this is a multi line string
line2
line3
'"""
# escapes (expanded by python)
q2 = "select count(f1) from test1 where f1 = 'this is a multi line string\r\nline2\r\nline3\r\n'"
# escapes (expanded by PostgreSQL)
q3 = r"select count(f1) from test1 where f1 = 'this is a multi line string\r\nline2\r\nline3\r\n'"
curs.execute(q3)
# stating the obvious
print "Comparisons:"
print "%-10s%-10s%-10s" % ("q1 == q2", "q1 == q3", "q2 == q3")
print "%-10s%-10s%-10s" % (q1 == q2, q1 == q3, q2 == q3)
print "\nRunning tests..."
curs.execute(q1)
print "Test 1 (string literal):", curs.fetchone()[0]
curs.execute(q2)
print "Test 2 (Python escapes):", curs.fetchone()[0]
curs.execute(q3)
print "Test 3 (PG escapes):", curs.fetchone()[0]
# in case someone wonders, let's try using query parameters
astring = """this is a multi line string
line2
line3
"""
q = "select count(f1) from test1 where f1 = %(mystr)s"
curs.execute(q, { "mystr": astring })
print "Test 4 (parameters):", curs.fetchone()[0]
---------- >8 CUT HERE ------------
This is the output (on Linux):
Comparisons:
q1 == q2 q1 == q3 q2 == q3
False False False
Running tests...
Test 1 (string literal): 0
Test 2 (Python escapes): 1
Test 3 (PG escapes): 1
Test 4 (parameters): 0
Which is consistent with your examples, that is, it works _only_
with explicit escapes (either at Python level or at PostgreSQL level).
If client-side python works this way, why are you expecting server-side
python to work differently?
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
On Tue, 29 Mar 2005, Marco Colombo wrote:
> # escapes (expanded by PostgreSQL)
> q3 = r"select count(f1) from test1 where f1 = 'this is a multi line
> string\r\nline2\r\nline3\r\n'"
> curs.execute(q3)
^^^^^^^^^^^^^^^^^^
This line (no. 28) is useless (but harmless), please ignore it
(just a cut&paste error).
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
On 3/23/2005 11:22 AM, Joshua D. Drake wrote: > David Gagnon wrote: > >> Hi all, >> >> I just created a new db wich userX is owner. I log via pgAdminIII >> with the same user but I can't update the pg_class. > > > You are a datdba but not a superuser :). You have to be a super user > to update pg_class. and not only a superuser, but one with usecatupd set to true as well. Jan > > Sincerely, > > Joshua D. Drake > >> >> UPDATE pg_class SET reltriggers = 0 WHERE relname = 'ic' >> >> I get:ERROR: permission denied for relation pg_class >> >> I do that on my dev env. The only difference I saw beetween users is >> that my DEV user as priviledge to create database(But it shouln't >> matter...?!) >> >> We create the db with : createdb -O userX -E UNICODE webCatalogTest >> >> Thanks for your help! >> >> /David >> >> >> >> >> >> ---------------------------(end of broadcast)--------------------------- >> TIP 2: you can get off all lists at once with the unregister command >> (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) > > > > > > ------------------------------------------------------------------------ > > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #