Thread: libpq single-row mode performance

libpq single-row mode performance

From
James Duong
Date:
<div class="WordSection1"><p class="MsoNormal">Hi,<p class="MsoNormal"> <p class="MsoNormal">I’m writing an app on top
oflibpq. To avoid running out of memory I’m using the single-row mode, but am finding the overhead in using this quite
significant.<pclass="MsoNormal"> <p class="MsoNormal">What I see is that ~60% of my application’s run time when
retrievingdata is spent in calls to either PQgetResult() and PQclear(). I’ve added a switch in my app to turn off
single-rowmode and the performance roughly doubles.<p class="MsoNormal"> <p class="MsoNormal">Would it be possible to
optimizethe single-row mode? For example, add an API call PQnextRow(PGConn*), which is only usable in single-row mode
andwill just update the existing result with the contents of the next row? This would let us avoid to overhead of:<p
class="MsoListParagraph"style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><span style="mso-list:Ignore">1.<span
style="font:7.0pt"Times New Roman"">       </span></span>Malloc’ing a PGResult and initializing its defaults.<p
class="MsoListParagraph"style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><span style="mso-list:Ignore">2.<span
style="font:7.0pt"Times New Roman"">       </span></span>Copying the column metadata to the new results.<p
class="MsoListParagraph"style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><span style="mso-list:Ignore">3.<span
style="font:7.0pt"Times New Roman"">       </span></span>Possibly we can avoid malloc’ing cell data if the next row has
cellsthe same size or smaller than a previous row, though I’m not sure of the internals here.<p
class="MsoListParagraph"style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><span style="mso-list:Ignore">4.<span
style="font:7.0pt"Times New Roman"">       </span></span>Release memory with PQclear().<p class="MsoNormal"> <p
class="MsoNormal"><b><spanlang="EN-US"
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black;mso-fareast-language:EN-CA">James
Duong</span></b><spanlang="EN-US" style="font-size:8.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-CA">
</span><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA">|</span></b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA"></span><span
lang="EN-US"style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black;mso-fareast-language:EN-CA">Senior
ComputerScientist</span><span lang="EN-US"
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-CA"></span><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA">|</span></b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA">Simba Technologies
Inc.</span><pclass="MsoNormal"><span lang="EN-US"
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black;mso-fareast-language:EN-CA">Tel+1.604.633.0008
ext.120</span><span lang="EN-US" style="font-size:8.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-CA">
</span><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA">|</span></b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA"></span><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA">Fax+1.604.633.0004
</span><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA">|</span></b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA"></span><span
lang="EN-US"style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:blue;mso-fareast-language:EN-CA"><a
href="mailto:jamesd@simba.com"><spanstyle="color:blue">jamesd@simba.com</span></a></span><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA"></span><p
class="MsoNormal"><span
style="font-size:5.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA"> </span><p
class="MsoNormal"><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA">938West
8<sup>th</sup>Avenue </span><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA">|</span></b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA">Vancouver, BC
</span><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA">|</span></b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA">Canada
</span><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA">|</span></b><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA"></span></b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA">V5Z1E5</span><p
class="MsoNormal"><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA">TheData Access and
AnalyticsExperts</span></b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA"> </span><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#595959;mso-fareast-language:EN-CA">|</span></b><b><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA"> </span></b><span
style="mso-fareast-language:EN-CA"><ahref="http://www.simba.com/"><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:blue;letter-spacing:1.6pt">www.simba.co</span><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:blue">m</span></a></span><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-CA"></span><p class="MsoNormal"><span
style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#262626;mso-fareast-language:EN-CA"> </span><p
class="MsoNormal"><ahref="http://twitter.com/#!/simbatech"><span
style="color:blue;mso-fareast-language:EN-CA;text-decoration:none"><imgalt="Description: Description: Description:
Description:twitter" border="0" height="16" id="Picture_x0020_13" src="cid:image001.png@01CF9620.34DCF710" width="16"
/></span></a><spanstyle="color:#262626;mso-fareast-language:EN-CA">  </span><a
href="http://www.youtube.com/user/SimbaTechnologies"><span
style="color:blue;mso-fareast-language:EN-CA;text-decoration:none"><imgalt="Description: Description: Description:
Description:youtube" border="0" height="16" id="Picture_x0020_12" src="cid:image002.png@01CF9620.34DCF710" width="16"
/></span></a><spanstyle="color:#262626;mso-fareast-language:EN-CA">  </span><a
href="http://ca.linkedin.com/in/simbatechnologies"><span
style="color:blue;mso-fareast-language:EN-CA;text-decoration:none"><imgalt="Description: Description: Description:
Description:linkedin" border="0" height="16" id="Picture_x0020_11" src="cid:image003.png@01CF9620.34DCF710" width="16"
/></span></a><spanstyle="color:#262626;mso-fareast-language:EN-CA">  </span><a
href="http://www.facebook.com/pages/Simba-Technologies-Inc/163133087033256"><span
style="color:blue;mso-fareast-language:EN-CA;text-decoration:none"><imgalt="Description: Description: Description:
Description:facebook" border="0" height="16" id="Picture_x0020_10" src="cid:image004.png@01CF9620.34DCF710" width="16"
/></span></a><spanstyle="color:#262626;mso-fareast-language:EN-CA">  </span><a
href="https://plus.google.com/116908198145591559055/posts"><span
style="color:blue;mso-fareast-language:EN-CA;text-decoration:none"><imgalt="Description: Description: Description:
Description:google-plus" border="0" height="16" id="Picture_x0020_9" src="cid:image005.png@01CF9620.34DCF710"
width="16"/></span></a><span style="color:#262626;mso-fareast-language:EN-CA">  </span><a
href="http://blogs.simba.com/"><spanstyle="color:blue;mso-fareast-language:EN-CA;text-decoration:none"><img
alt="Description:Description: Description: Description: typepad" border="0" height="16" id="Picture_x0020_8"
src="cid:image006.png@01CF9620.34DCF710"width="16" /></span></a><span
style="color:#262626;mso-fareast-language:EN-CA"></span><pclass="MsoNormal"><span
style="font-size:7.5pt;font-family:"Tahoma","sans-serif";color:gray;mso-fareast-language:EN-CA"> </span><p
class="MsoNormal"><span
style="font-size:7.5pt;font-family:"Tahoma","sans-serif";color:gray;mso-fareast-language:EN-CA">Thisemail message is
forthe sole use of the intended recipient(s) and may contain confidential and privileged information.  Any unauthorized
review,use, disclosure, or distribution is prohibited.  If you are not the intended recipient, please contact the
senderby reply email and destroy all copies of the original message.  Thank you.</span><span
style="font-size:7.5pt;font-family:"Tahoma","sans-serif";color:#1F497D;mso-fareast-language:EN-CA"></span><p
class="MsoNormal"> </div>

Re: libpq single-row mode performance

From
Pavel Golub
Date:
Hello, James.

You wrote:

JD> Hi,
JD>  
JD> I’m writing an app on top of libpq. To avoid running out of
JD> memory I’m using the single-row mode, but am finding the overhead in using this quite significant.
JD>  
JD> What I see is that ~60% of my application’s run time when
JD> retrieving data is spent in calls to either PQgetResult() and
JD> PQclear(). I’ve added a switch in my app to turn off single-row
JD> mode and the performance roughly doubles.
JD>  
JD> Would it be possible to optimize the single-row mode? For
JD> example, add an API call PQnextRow(PGConn*), which is only usable
JD> in single-row mode and will just update the existing result with
JD> the contents of the next row? This would let us avoid to overhead of:
JD> 1.       Malloc’ing a PGResult and initializing its defaults.
JD> 2.       Copying the column metadata to the new results.
JD> 3.       Possibly we can avoid malloc’ing cell data if the next
JD> row has cells the same size or smaller than a previous row, though
JD> I’m not sure of the internals here.
JD> 4.       Release memory with PQclear().
JD>  
JD> James Duong | Senior Computer Scientist | Simba Technologies Inc.
JD> Tel +1.604.633.0008 ext. 120 | Fax +1.604.633.0004 | jamesd@simba.com
JD>  
JD> 938 West 8th Avenue | Vancouver, BC | Canada | V5Z 1E5
JD> The Data Access and Analytics Experts  |  www.simba.com 
JD>  
JD>                 
JD>  
JD> This email message is for the sole use of the intended
JD> recipient(s) and may contain confidential and privileged
JD> information.  Any unauthorized review, use, disclosure, or
JD> distribution is prohibited.  If you are not the intended
JD> recipient, please contact the sender by reply email and destroy
JD> all copies of the original message.  Thank you.
JD>  

Good idea. +1 from me. Should we send this message to pg-hackers as
well?

-- 
With best wishes,Pavel                          mailto:pavel@gf.microolap.com