Thread: Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...
Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...
From
Kang Yuzhe
Date:
Thanks Amit for taking your time and pointing to some useful stuff on the Internals of PostgreSQL.
One thing I have learned is that PG community is not as hostile/harsh as I imagined to newbies. Rather, its the reverse.postgresql-9.6.2$ patch --dry-run -p1 < corresponding_clause_v12.patch
patching file doc/src/sgml/queries.sgml
Hunk #1 succeeded at 1603 (offset 2 lines).
Hunk #2 succeeded at 1622 (offset 2 lines).
Hunk #3 succeeded at 1664 (offset 2 lines).
patching file doc/src/sgml/sql.sgml
patching file src/backend/nodes/copyfuncs.c
Hunk #1 succeeded at 2807 (offset -188 lines).
Hunk #2 succeeded at 2823 (offset -188 lines).
Hunk #3 succeeded at 4251 (offset -340 lines).
patching file src/backend/nodes/equalfuncs.c
Hunk #1 succeeded at 995 (offset -55 lines).
Hunk #2 succeeded at 1009 (offset -55 lines).
Hunk #3 succeeded at 2708 (offset -230 lines).
patching file src/backend/nodes/nodeFuncs.c
Hunk #1 succeeded at 3384 (offset -60 lines).
patching file src/backend/nodes/outfuncs.c
Hunk #1 succeeded at 2500 (offset -164 lines).
Hunk #2 succeeded at 2793 (offset -179 lines).
Hunk #3 succeeded at 2967 (offset -184 lines).
patching file src/backend/nodes/readfuncs.c
Hunk #1 succeeded at 414 (offset -2 lines).
patching file src/backend/nodes/value.c
patching file src/backend/optimizer/prep/prepunion.c
Hunk #1 succeeded at 92 (offset 1 line).
Hunk #2 succeeded at 112 (offset 1 line).
Hunk #3 succeeded at 190 (offset 1 line).
Hunk #4 succeeded at 273 (offset 1 line).
Hunk #5 succeeded at 339 (offset 1 line).
Hunk #6 succeeded at 445 (offset 1 line).
Hunk #7 succeeded at 1057 (offset 1 line).
Hunk #8 succeeded at 1080 (offset 1 line).
Hunk #9 succeeded at 2190 (offset -13 lines).
patching file src/backend/parser/analyze.c
Hunk #1 succeeded at 75 (offset -1 lines).
Hunk #2 succeeded at 1600 (offset -61 lines).
Hunk #3 succeeded at 1882 (offset -69 lines).
Hunk #4 succeeded at 1892 (offset -69 lines).
Hunk #5 succeeded at 1994 (offset -69 lines).
patching file src/backend/parser/gram.y
Hunk #1 succeeded at 158 (offset -8 lines).
Hunk #2 FAILED at 394.
Hunk #3 succeeded at 573 with fuzz 2 (offset -41 lines).
Hunk #4 succeeded at 3328 (offset -251 lines).
Hunk #5 succeeded at 10182 (offset -699 lines).
Hunk #6 succeeded at 13470 (offset -771 lines).
Hunk #7 succeeded at 13784 (offset -773 lines).
Hunk #8 succeeded at 14581 (offset -811 lines).
Hunk #9 succeeded at 14589 (offset -811 lines).
1 out of 9 hunks FAILED -- saving rejects to file src/backend/parser/gram.y.rej
patching file src/backend/parser/parse_type.c
Hunk #1 succeeded at 736 (offset 1 line).
patching file src/backend/utils/adt/ruleutils.c
Hunk #1 succeeded at 5166 (offset -276 lines).
patching file src/include/nodes/parsenodes.h
Hunk #1 succeeded at 1285 (offset -175 lines).
Hunk #2 succeeded at 1321 (offset -175 lines).
Hunk #3 succeeded at 1350 (offset -175 lines).
patching file src/include/nodes/value.h
patching file src/include/parser/kwlist.h
Hunk #1 succeeded at 95 (offset -2 lines).
patching file src/test/regress/expected/create_view.out
Hunk #1 succeeded at 1571 (offset -55 lines).
patching file src/test/regress/expected/rules.out
Hunk #1 succeeded at 2260 (offset -85 lines).
patching file src/test/regress/expected/union.out
Hunk #1 succeeded at 59 with fuzz 2.
Hunk #3 succeeded at 479 (offset -1 lines).
Hunk #4 succeeded at 609 (offset -1 lines).
Hunk #5 succeeded at 684 (offset -1 lines).
Hunk #6 succeeded at 785 with fuzz 1 (offset -1 lines).
Hunk #7 succeeded at 838 (offset -1 lines).
patching file src/test/regress/sql/create_view.sql
Hunk #1 succeeded at 524 (offset -27 lines).
patching file src/test/regress/sql/union.sql
Hunk #1 succeeded at 20 with fuzz 2.
Hunk #2 succeeded at 69 with fuzz 2.
Hunk #3 succeeded at 149 (offset -1 lines).
Hunk #4 succeeded at 194 (offset -1 lines).
Hunk #5 succeeded at 218 (offset -1 lines).
Hunk #6 succeeded at 252 with fuzz 2 (offset -1 lines).
Hunk #7 succeeded at 281 (offset -1 lines).
..../postgresql-9.6.2$
On Wed, Mar 29, 2017 at 5:53 AM, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote:
Hi,
On 2017/03/28 15:40, Kang Yuzhe wrote:
> Thanks Tsunakawa for such an informative reply.
>
> Almost all of the docs related to the internals of PG are of introductory
> concepts only.
> There is even more useful PG internals site entitled "The Internals of
> PostgreSQL" in http://www.interdb.jp/pg/ translation of the Japanese PG
> Internals.
>
> The query processing framework that is described in the manual as you
> mentioned is of informative and introductory nature.
> In theory, the query processing framework described in the manual is
> understandable.
>
> Unfortunate, it is another story to understand how query processing
> framework in PG codebase really works.
> It has become a difficult task for me to walk through the PG source code
> for example how SELECT/INSERT/TRUNCATE in the the different modules under
> "src/..". really works.
>
> I wish there were Hands-On with PostgreSQL Internals like
> https://bkmjournal.wordpress.com/2017/01/22/hands-on-with- postgresql-internals/
> for more complex PG features.
>
> For example, MERGE SQL standard is not supported yet by PG. I wish there
> were Hands-On with PostgreSQL Internals for MERGE/UPSERT. How it is
> implemented in parser/executor/storage etc. modules with detailed
> explanation for each code and debugging and other important concepts
> related to system programming.
I am not sure if I can show you that one place where you could learn all
of that, but many people who started with PostgreSQL development at some
point started by exploring the source code itself (either for learning or
to write a feature patch), articles on PostgreSQL wiki, and many related
presentations accessible using the Internet. I liked the following among
many others:
Introduction to Hacking PostgreSQL:
http://www.neilconway.org/talks/hacking/
Inside the PostgreSQL Query Optimizer:
http://www.neilconway.org/talks/optimizer/optimizer.pdf
Postgres Internals Presentations:
http://momjian.us/main/presentations/internals.html
Thanks,
Amit
Attachment
Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PGHacking...
From
Amit Langote
Date:
Hi, On 2017/04/18 16:54, Kang Yuzhe wrote: > Thanks Amit for taking your time and pointing to some useful stuff on the > Internals of PostgreSQL. > > > One thing I have learned is that PG community is not as hostile/harsh as I > imagined to newbies. Rather, its the reverse. > I am feeling at home here. > > Amit, would you please help out on how to apply some patches in PG source > code. For example, there are two patches attached here: one on > CORRESPONDING CLAUSE and one on MERGE SQL Standard. > > There are some errors saying Hunk failed(src/backend/parser/gram.y.rej). > > postgresql-9.6.2$ patch --dry-run -p1 < corresponding_clause_v12.patch > patching file doc/src/sgml/queries.sgml > Hunk #1 succeeded at 1603 (offset 2 lines). > Hunk #2 succeeded at 1622 (offset 2 lines). > Hunk #3 succeeded at 1664 (offset 2 lines). [ ... ] > ..../postgresql-9.6.2$ Firstly, it looks like you're trying to apply the patch to the 9.6 source tree (are you working with the PostgreSQL source git repository?). But, since all the new feature patches are created against the master development branch of the git repository, the patch most likely won't apply cleanly against a source tree from the older branch. If you're not using the git repository currently, you may have better luck trying the development branch snapshot tarballs (see the link below): https://www.postgresql.org/ftp/snapshot/dev/ Also, it's a good idea to reply on the email thread from where you downloaded the patch to ask them to update the patch, so that they can send a fresh patch that applies cleanly. The MERGE patch looks very old (from 2010 probably), so properly applying it to the source tree of today is going to be hard. Actually, it most likely won't be in a working condition anymore. You can try recently proposed patches, for example, those in the next commitfest: https://commitfest.postgresql.org/14/ Patches listed on the above page are more likely to apply cleanly and be in working condition. But of course, you will need to be interested in the topics those patches are related to. There are some new SQL feature patches, for example: https://commitfest.postgresql.org/14/839/ Thanks, Amit
Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...
From
Kang Yuzhe
Date:
Hello Amit,
Thanks gain for being patient with me. YES, I am working with the PostgreSQL source git repository but I don't think I updated my local forked/cloned branch. I am also working on standalone PG 9.6.2 source code as well.
On Tue, Apr 18, 2017 at 11:25 AM, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote:
Hi,
On 2017/04/18 16:54, Kang Yuzhe wrote:
> Thanks Amit for taking your time and pointing to some useful stuff on the
> Internals of PostgreSQL.
>
>
> One thing I have learned is that PG community is not as hostile/harsh as I
> imagined to newbies. Rather, its the reverse.
> I am feeling at home here.
>
> Amit, would you please help out on how to apply some patches in PG source
> code. For example, there are two patches attached here: one on
> CORRESPONDING CLAUSE and one on MERGE SQL Standard.
>
> There are some errors saying Hunk failed(src/backend/parser/gram.y.rej).
>
> postgresql-9.6.2$ patch --dry-run -p1 < corresponding_clause_v12.patch
> patching file doc/src/sgml/queries.sgml
> Hunk #1 succeeded at 1603 (offset 2 lines).
> Hunk #2 succeeded at 1622 (offset 2 lines).
> Hunk #3 succeeded at 1664 (offset 2 lines).
[ ... ]
> ..../postgresql-9.6.2$
Firstly, it looks like you're trying to apply the patch to the 9.6 source
tree (are you working with the PostgreSQL source git repository?). But,
since all the new feature patches are created against the master
development branch of the git repository, the patch most likely won't
apply cleanly against a source tree from the older branch.
If you're not using the git repository currently, you may have better luck
trying the development branch snapshot tarballs (see the link below):
https://www.postgresql.org/ftp/snapshot/dev/
Also, it's a good idea to reply on the email thread from where you
downloaded the patch to ask them to update the patch, so that they can
send a fresh patch that applies cleanly.
The MERGE patch looks very old (from 2010 probably), so properly applying
it to the source tree of today is going to be hard. Actually, it most
likely won't be in a working condition anymore. You can try recently
proposed patches, for example, those in the next commitfest:
https://commitfest.postgresql.org/14/
Patches listed on the above page are more likely to apply cleanly and be
in working condition. But of course, you will need to be interested in
the topics those patches are related to. There are some new SQL feature
patches, for example:
https://commitfest.postgresql.org/14/839/
Thanks,
Amit