Thread: Return of "can't paste into psql" issue
Hackers, We've seen this before: http://www.postgresql.org/message-id/b42b73150906271019k6212ba71u6263bc43a8ec8e94@mail.gmail.com I've run across this again, and am posting it here just in case someone should be struck by some amazing inspiration on where the problem might be. Here's the issue: Pasting text blocks over 100 lines into psql running in a terminal causes lines to be arbitrarily truncated, mangling thequery or function. The exact truncation appears random, but is 100% consistent for any given block of text. For example, one test I ran all lines 101-200 where truncated at character #32, but lines 1-100 were untouched. In another test, lines #82 and #163 were truncated after the 14th caracter, but other lines were fine. In the Ubuntu 12.10 tests below, the truncation occured in the same places regardless of whether I was using kterm, gterm or xterm. Here's where I've seen this issue occur: - Ubuntu 12.10: - 9.2 installed from source with readline - 9.4 installed from source with readline, and with libedit - usingGnome Terminal, Kterm, or xterm - Ubuntu 12.04: - 9.2 installed using packages from apt.postgresql.org - over SSH from a Gnome Terminal - Mac OSX 10.6 - 9.2 installed from source with readline Interestingly, here's where it does NOT occur: - Scientific Linux 6.1 - 9.2 installed from yum.postgresql.org packages - over SSH from a Gnome Terminal on Ubuntu - Ubuntu 12.10 - 9.4 from source, compiled --without-readline Also, the same mangling does not happen with any of the following applications on Ubuntu 12.10: - nano - emacs - joe - gedit - cat, including cat | psql I could not find another readline-equipped terminal application to test against. Ubuntu 12.10 has libreadline 6.2.9, and SciLinux 6.1 has version 6.0. However, since this happens with libedit as well, I don't think it's purely a readline issue. It seems like we're having some kind of bizarre interaction of psql's readline/libedit hooks and specific terminals or library versions, but darned if I can even figure out where to look. If anybody has inspiration strike, please post! -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On 07/17/2013 02:50 PM, Josh Berkus wrote: > Hackers, > > We've seen this before: > > http://www.postgresql.org/message-id/b42b73150906271019k6212ba71u6263bc43a8ec8e94@mail.gmail.com > > I've run across this again, and am posting it here just in case someone > should be struck by some amazing inspiration on where the problem might > be. Here's the issue: > > Pasting text blocks over 100 lines into psql running in > a terminal causes lines to be arbitrarily truncated, mangling > the query or function. [snip] > > It seems like we're having some kind of bizarre interaction of psql's > readline/libedit hooks and specific terminals or library versions, but > darned if I can even figure out where to look. If anybody has > inspiration strike, please post! > What happens when you do this with "psql -n" ? My suspicion is it's the tab completion code, which I occasionally find it useful to disable this way - sadly one can't do that on the fly AFAIK. cheers andrew
On Wed, Jul 17, 2013 at 2:20 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > On 07/17/2013 02:50 PM, Josh Berkus wrote: >> >> Hackers, >> >> We've seen this before: >> >> >> http://www.postgresql.org/message-id/b42b73150906271019k6212ba71u6263bc43a8ec8e94@mail.gmail.com >> >> I've run across this again, and am posting it here just in case someone >> should be struck by some amazing inspiration on where the problem might >> be. Here's the issue: >> >> Pasting text blocks over 100 lines into psql running in >> a terminal causes lines to be arbitrarily truncated, mangling >> the query or function. > > > [snip] > > >> >> It seems like we're having some kind of bizarre interaction of psql's >> readline/libedit hooks and specific terminals or library versions, but >> darned if I can even figure out where to look. If anybody has >> inspiration strike, please post! >> > > > > What happens when you do this with "psql -n" ? My suspicion is it's the tab > completion code, which I occasionally find it useful to disable this way - > sadly one can't do that on the fly AFAIK. huh -- you're right. merlin
On 07/17/2013 12:20 PM, Andrew Dunstan wrote: > What happens when you do this with "psql -n" ? My suspicion is it's the > tab completion code, which I occasionally find it useful to disable this > way - sadly one can't do that on the fly AFAIK. Well, that works. But it disables readline, not just tab completion. So it's nice that I don't have to *compile* without readline, but we still have the issue that you can't paste large files and also have readline. So, an even more practical workaround (I've been using cat | psql), but still a mysterious issue. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Josh Berkus wrote: > So, an even more practical workaround (I've been using cat | psql), but > still a mysterious issue. How often do your files contain tabs? I have seen cases where tab completion messes things up by asking you after a tab whether you really want to complete due to the number of possible completions, and such. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On 07/17/2013 03:37 PM, Alvaro Herrera wrote: > Josh Berkus wrote: > >> So, an even more practical workaround (I've been using cat | psql), but >> still a mysterious issue. > > How often do your files contain tabs? I have seen cases where tab > completion messes things up by asking you after a tab whether you really > want to complete due to the number of possible completions, and such. First thing I thought of, so I made sure my text editor was replacing tabs with spaces. Also, my test cases had no tabs at all (nor spaces neither). -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Josh Berkus <josh@agliodbs.com> writes: > So, an even more practical workaround (I've been using cat | psql), but > still a mysterious issue. As a workaround you might try \e with EDITOR=emacs or some of the other solutions you've been pasting, maybe even cat, so that you can switch that readline-completion-bug-free environment for just that paste? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On 07/17/2013 04:30 PM, Dimitri Fontaine wrote: > Josh Berkus <josh@agliodbs.com> writes: >> So, an even more practical workaround (I've been using cat | psql), but >> still a mysterious issue. > > As a workaround you might try \e with EDITOR=emacs or some of the other > solutions you've been pasting, maybe even cat, so that you can switch > that readline-completion-bug-free environment for just that paste? Oh, there's lots of good workarounds. I just wanna know why the bug is happening in the first place. (FWIW, I hit this because I'm using a graphical editor for PL/Python which pipes stuff to the terminal to execute it, and its piping hits the same issues as paste-from-clipboard) -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On Wed, Jul 17, 2013 at 6:30 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote: > Josh Berkus <josh@agliodbs.com> writes: >> So, an even more practical workaround (I've been using cat | psql), but >> still a mysterious issue. > > As a workaround you might try \e with EDITOR=emacs or some of the other > solutions you've been pasting, maybe even cat, so that you can switch > that readline-completion-bug-free environment for just that paste? Well, I use \e=vim and it works fine. But, sometimes you forget and the results can be destructive to the database. I messed around for a little bit with the tab completion callback routine yesterday and I now believe that's not the issue. It only fires when you tab AFAICT -- so the problem is in the readline function itself which reduces the set of solutions somewhat -- either you run it or you don't. One thing that could solve a lot of issues would be to disable readline when inside a dollar quote etc. merlin
On Thu, Jul 18, 2013 at 7:57 AM, Merlin Moncure <mmoncure@gmail.com> wrote: > One thing that could solve a lot of issues would be to disable > readline when inside a dollar quote etc. actually, that's dumb (pre-coffee). merlin
On 07/18/2013 08:59 AM, Merlin Moncure wrote: > On Thu, Jul 18, 2013 at 7:57 AM, Merlin Moncure <mmoncure@gmail.com> wrote: >> One thing that could solve a lot of issues would be to disable >> readline when inside a dollar quote etc. > actually, that's dumb (pre-coffee). > Yeah, but what would be useful would be a way to disable and re-enable readline on the fly. I looked a while back and didn't find an obvious way to do that, but I might well have missed something. cheers andrew