Thread: Problem in GIT mirror (was Re: pgsql: Fix all the server-side SIGQUIT handlers ...)

Tom Lane wrote:
> Log Message:
> -----------
> Fix all the server-side SIGQUIT handlers (grumble ... why so many identical
> copies?) to ensure they really don't run proc_exit/shmem_exit callbacks,
> as was intended.  I broke this behavior recently by installing atexit
> callbacks without thinking about the one case where we truly don't want
> to run those callback functions.  Noted in an example from Dave Page.

Something seems wrong on the GIT mirror.  This patch appears as
http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e5a8ea96adf14b9d822aff82d8f4ebf41d3a8e95
however, it only shows the diff for xlog.c, not the other files that
were touched by this commit.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera wrote:
> Tom Lane wrote:
> > Log Message:
> > -----------
> > Fix all the server-side SIGQUIT handlers (grumble ... why so many identical
> > copies?) to ensure they really don't run proc_exit/shmem_exit callbacks,
> > as was intended.  I broke this behavior recently by installing atexit
> > callbacks without thinking about the one case where we truly don't want
> > to run those callback functions.  Noted in an example from Dave Page.
> 
> Something seems wrong on the GIT mirror.  This patch appears as
> http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e5a8ea96adf14b9d822aff82d8f4ebf41d3a8e95
> however, it only shows the diff for xlog.c, not the other files that
> were touched by this commit.

So I foolishly assumed that it would fix itself when the next sync
occured, but several hours have passed and the problem still exists.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera wrote:
> Alvaro Herrera wrote:
>> Tom Lane wrote:
>>> Log Message:
>>> -----------
>>> Fix all the server-side SIGQUIT handlers (grumble ... why so many identical
>>> copies?) to ensure they really don't run proc_exit/shmem_exit callbacks,
>>> as was intended.  I broke this behavior recently by installing atexit
>>> callbacks without thinking about the one case where we truly don't want
>>> to run those callback functions.  Noted in an example from Dave Page.
>> Something seems wrong on the GIT mirror.  This patch appears as
>> http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e5a8ea96adf14b9d822aff82d8f4ebf41d3a8e95
>> however, it only shows the diff for xlog.c, not the other files that
>> were touched by this commit.
> 
> So I foolishly assumed that it would fix itself when the next sync
> occured, but several hours have passed and the problem still exists.
> 

I tried to figure something out about this, but I have no clue.

The rsync:ed copy of the cvs repository that the conversion runs at has
all the changes.

The git repository has only the xlog.c one (checked in the server repo
as well just to be sure it wasn't anything in the gitweb install or
anything)

The ruby magic ("fromcvs") that does the importing says nothing - it
runs without any complaints and says there is nothing to do.


Any ideas at all where to look?

I wonder if the many identical copies somehow broke the import script or
something...

//Magnus

Magnus Hagander wrote:
> Alvaro Herrera wrote:
>> Alvaro Herrera wrote:
>>> Tom Lane wrote:
>>>> Log Message:
>>>> -----------
>>>> Fix all the server-side SIGQUIT handlers (grumble ... why so many identical
>>>> copies?) to ensure they really don't run proc_exit/shmem_exit callbacks,
>>>> as was intended.  I broke this behavior recently by installing atexit
>>>> callbacks without thinking about the one case where we truly don't want
>>>> to run those callback functions.  Noted in an example from Dave Page.
>>> Something seems wrong on the GIT mirror.  This patch appears as
>>> http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e5a8ea96adf14b9d822aff82d8f4ebf41d3a8e95
>>> however, it only shows the diff for xlog.c, not the other files that
>>> were touched by this commit.
>> So I foolishly assumed that it would fix itself when the next sync
>> occured, but several hours have passed and the problem still exists.
>>
> 
> I tried to figure something out about this, but I have no clue.
> 
> The rsync:ed copy of the cvs repository that the conversion runs at has
> all the changes.
> 
> The git repository has only the xlog.c one (checked in the server repo
> as well just to be sure it wasn't anything in the gitweb install or
> anything)
> 
> The ruby magic ("fromcvs") that does the importing says nothing - it
> runs without any complaints and says there is nothing to do.
> 
> 
> Any ideas at all where to look?
> 
> I wonder if the many identical copies somehow broke the import script or
> something...


Ok, a bit further.

I tried restoring a backup and re-running the conversion script. Then it
works.

The only reason I can see for this failure is that the rsync happened
just as the files were committed, and for that reason we got a partial
commit (since cvs doesn't to atomic operations). Then when we got the
rest of the modifications, the migration script had already flagged that
one was imported.

We can work off tihs that came off the backup, but that invalides the
last couple of commits, which will AFAIK break everybody who has cloned
the repository?

Does anybody know of a better way to fix it, without this happening?

//Magnus

On Sat, May 16, 2009 at 01:51:59PM +0200, Magnus Hagander wrote:
> Magnus Hagander wrote:
> > Alvaro Herrera wrote:
> >> Alvaro Herrera wrote:
> >>> Tom Lane wrote:
> >>>> Log Message:
> >>>> -----------
> >>>> Fix all the server-side SIGQUIT handlers (grumble ... why so many identical
> >>>> copies?) to ensure they really don't run proc_exit/shmem_exit callbacks,
> >>>> as was intended.  I broke this behavior recently by installing atexit
> >>>> callbacks without thinking about the one case where we truly don't want
> >>>> to run those callback functions.  Noted in an example from Dave Page.
> >>> Something seems wrong on the GIT mirror.  This patch appears as
> >>> http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e5a8ea96adf14b9d822aff82d8f4ebf41d3a8e95
> >>> however, it only shows the diff for xlog.c, not the other files that
> >>> were touched by this commit.
> >> So I foolishly assumed that it would fix itself when the next sync
> >> occured, but several hours have passed and the problem still exists.
> >>
> > 
> > I tried to figure something out about this, but I have no clue.
> > 
> > The rsync:ed copy of the cvs repository that the conversion runs at has
> > all the changes.
> > 
> > The git repository has only the xlog.c one (checked in the server repo
> > as well just to be sure it wasn't anything in the gitweb install or
> > anything)
> > 
> > The ruby magic ("fromcvs") that does the importing says nothing - it
> > runs without any complaints and says there is nothing to do.
> > 
> > 
> > Any ideas at all where to look?
> > 
> > I wonder if the many identical copies somehow broke the import script or
> > something...
> 
> 
> Ok, a bit further.
> 
> I tried restoring a backup and re-running the conversion script. Then it
> works.
> 
> The only reason I can see for this failure is that the rsync happened
> just as the files were committed, and for that reason we got a partial
> commit (since cvs doesn't to atomic operations). Then when we got the
> rest of the modifications, the migration script had already flagged that
> one was imported.
> 
> We can work off tihs that came off the backup, but that invalides the
> last couple of commits, which will AFAIK break everybody who has cloned
> the repository?
> 
> Does anybody know of a better way to fix it, without this happening?

Not sure exactly, but the nice people at
<http://repo.or.cz/w/PostgreSQL.git> appear to be managing not to have
this problem.  What say we look into how they're doing theirs? :)

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david@fetter.org> writes:
> On Sat, May 16, 2009 at 01:51:59PM +0200, Magnus Hagander wrote:
>> Does anybody know of a better way to fix it, without this happening?

> Not sure exactly, but the nice people at
> <http://repo.or.cz/w/PostgreSQL.git> appear to be managing not to have
> this problem.  What say we look into how they're doing theirs? :)

Or they've just been lucky (so far).

But if this was a case of the git mirror seeing the commit as non
atomic, I don't understand why it wouldn't have later picked up the
rest of the changes as a separate commit with the same message.
Seems like you'd have to work at it to prevent that from happening
... maybe there is some heuristic for identifying "a single commit"
that goes horribly wrong here?
        regards, tom lane

On Saturday 16 May 2009 20:31:39 David Fetter wrote:
> Not sure exactly, but the nice people at
> <http://repo.or.cz/w/PostgreSQL.git> appear to be managing not to have
> this problem.  What say we look into how they're doing theirs?

They are doing it exactly the same way.  I even copied their scripts, 
initially.

On Sun, May 17, 2009 at 05:44:30PM +0300, Peter Eisentraut wrote:
> On Saturday 16 May 2009 20:31:39 David Fetter wrote:
> > Not sure exactly, but the nice people at
> > <http://repo.or.cz/w/PostgreSQL.git> appear to be managing not to
> > have this problem.  What say we look into how they're doing
> > theirs?
> 
> They are doing it exactly the same way.  I even copied their
> scripts, initially.

Sounds like this method just got bitten by the documented timing
problem.

What features, if any, that git's CVS emulator doesn't provide do
committers use?

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter wrote:

> What features, if any, that git's CVS emulator doesn't provide do
> committers use?

What you're obviously thinking is tabled (?) to be discussed during
PGCon developer's meeting, so let's wait for the outcome of that
discussion.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre@commandprompt.com> writes:
> David Fetter wrote:
>> What features, if any, that git's CVS emulator doesn't provide do
>> committers use?

> What you're obviously thinking is tabled (?) to be discussed during
> PGCon developer's meeting, so let's wait for the outcome of that
> discussion.

Hmmm ... 

Problem:cvs-to-git mirror is buggy

Proposed solution:install git and git-to-cvs emulator

We should assume this will be less buggy?  Oh, and the first case is
not project-critical infrastructure but the second will be.

I think that is going to be an extremely short discussion.
        regards, tom lane

Tom Lane wrote:

> Problem:
>     cvs-to-git mirror is buggy
> 
> Proposed solution:
>     install git and git-to-cvs emulator
> 
> We should assume this will be less buggy?

Well, actually yes -- the git-to-cvs mechanism is integrated in Git,
whereas the cvs-to-git mirror is a rather unsupported hack.

http://www.kernel.org/pub/software/scm/git/docs/git-cvsserver.html

Something you will definitively not like about Git is that we will lose
the $PostgreSQL$ expansion stuff -- and there's no equivalent
functionality AFAIK.

> Oh, and the first case is not project-critical infrastructure but the
> second will be.

Git has been used by the Linux developers long enough that it's easy to
have confidence that it will serve us fine.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera <alvherre@commandprompt.com> writes:
> Tom Lane wrote:
>> We should assume this will be less buggy?

> Well, actually yes -- the git-to-cvs mechanism is integrated in Git,
> whereas the cvs-to-git mirror is a rather unsupported hack.
> http://www.kernel.org/pub/software/scm/git/docs/git-cvsserver.html

I quote from that page:

>>> It is highly functional. However, not all methods are implemented,
>>> and for those methods that are implemented, not all switches are
>>> implemented. 

>>> Testing has been done using both the CLI CVS client, and the Eclipse
>>> CVS plugin. Most functionality works fine with both of these clients. 

This is the project's *own* description of its level of
completeness/reliability.  Warm fuzzy feelings it does not leave me
with.

I'm prepared to believe that core git is reasonably well debugged by
now, but I have little confidence in its ability to interact with cvs
in either direction.  It doesn't look like upstream has put much work
into getting the bugs out of that.  (There's no reason to think they
would care much about it...)  If we make that transition I think it
will have to be all-or-nothing.

> Something you will definitively not like about Git is that we will lose
> the $PostgreSQL$ expansion stuff -- and there's no equivalent
> functionality AFAIK.

Well, that particular thing always struck me as something we did because
we could, not because there was any important requirement for it.
        regards, tom lane

On Mon, May 18, 2009 at 11:29:00AM -0400, Tom Lane wrote:
> Alvaro Herrera <alvherre@commandprompt.com> writes:
> > Tom Lane wrote:
> >> We should assume this will be less buggy?
> 
> > Well, actually yes -- the git-to-cvs mechanism is integrated in Git,
> > whereas the cvs-to-git mirror is a rather unsupported hack.
> > http://www.kernel.org/pub/software/scm/git/docs/git-cvsserver.html
> 
> I quote from that page:
> 
> >>> It is highly functional. However, not all methods are
> >>> implemented, and for those methods that are implemented, not all
> >>> switches are implemented. 
> 
> >>> Testing has been done using both the CLI CVS client, and the
> >>> Eclipse CVS plugin. Most functionality works fine with both of
> >>> these clients. 
> 
> This is the project's *own* description of its level of
> completeness/reliability.  Warm fuzzy feelings it does not leave me
> with.
> 
> I'm prepared to believe that core git is reasonably well debugged by
> now, but I have little confidence in its ability to interact with
> cvs in either direction.  It doesn't look like upstream has put much
> work into getting the bugs out of that.  (There's no reason to think
> they would care much about it...)  If we make that transition I
> think it will have to be all-or-nothing.

+1 for all.  Right when we branch 8.5 would be great :)

> > Something you will definitively not like about Git is that we will
> > lose the $PostgreSQL$ expansion stuff -- and there's no equivalent
> > functionality AFAIK.
> 
> Well, that particular thing always struck me as something we did
> because we could, not because there was any important requirement
> for it.

What's it doing exactly?  Is there some way to get similar
functionality in git?

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter wrote:
> On Mon, May 18, 2009 at 11:29:00AM -0400, Tom Lane wrote:
> > Alvaro Herrera <alvherre@commandprompt.com> writes:

> > > Something you will definitively not like about Git is that we will
> > > lose the $PostgreSQL$ expansion stuff -- and there's no equivalent
> > > functionality AFAIK.
> > 
> > Well, that particular thing always struck me as something we did
> > because we could, not because there was any important requirement
> > for it.
> 
> What's it doing exactly?

What it does is expand the $PostgreSQL$ tag found in the files to
a string containing the current revision number, last author and change
date, etc.

> Is there some way to get similar functionality in git?

No AFAIK.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Magnus Hagander wrote:
> Magnus Hagander wrote:
>> Alvaro Herrera wrote:
>>> Alvaro Herrera wrote:
>>>> Tom Lane wrote:
>>>>> Log Message:
>>>>> -----------
>>>>> Fix all the server-side SIGQUIT handlers (grumble ... why so many identical
>>>>> copies?) to ensure they really don't run proc_exit/shmem_exit callbacks,
>>>>> as was intended.  I broke this behavior recently by installing atexit
>>>>> callbacks without thinking about the one case where we truly don't want
>>>>> to run those callback functions.  Noted in an example from Dave Page.
>>>> Something seems wrong on the GIT mirror.  This patch appears as
>>>> http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e5a8ea96adf14b9d822aff82d8f4ebf41d3a8e95
>>>> however, it only shows the diff for xlog.c, not the other files that
>>>> were touched by this commit.
>>> So I foolishly assumed that it would fix itself when the next sync
>>> occured, but several hours have passed and the problem still exists.
>>>
>> I tried to figure something out about this, but I have no clue.
>>
>> The rsync:ed copy of the cvs repository that the conversion runs at has
>> all the changes.
>>
>> The git repository has only the xlog.c one (checked in the server repo
>> as well just to be sure it wasn't anything in the gitweb install or
>> anything)
>>
>> The ruby magic ("fromcvs") that does the importing says nothing - it
>> runs without any complaints and says there is nothing to do.
>>
>>
>> Any ideas at all where to look?
>>
>> I wonder if the many identical copies somehow broke the import script or
>> something...
> 
> 
> Ok, a bit further.
> 
> I tried restoring a backup and re-running the conversion script. Then it
> works.

I've done this now for the production repositories. I tried on my git
clone, and it actually works to do "git pull". I get two commits for the
broken one, but the code that is in the files in my master branch is the
correct one.

It may be that this fails copmletely if there is a merge failure or
something. In that case, I'm told you can do a git reset --hard to get
back to life. It's painful, but it's something.

We figured it was just as well to bite the bullet now, because the
longer we wait the more issues people would have with it.

I'd appreciated if people can run some tests and make sure there aren't
any issues in it now - for the moment, I still have the old corrupt
repository around if we need it :-)

//Magnus