Thread: Speeds using a transaction vrs not
I wrote a .net program to move my data to postgres (works great on SQ_ASCII). In fiddling around I tried it using the odbc driver and a transaction originally, but converted it to using the .net connectivity but no transaction. What I found was it moved my database (4 gig in MSSQL) in 2 hours using the .net, but 12 hours with the odbc and transaction. Have any of you played around to see if using a transaction should be that much slower or is it the odbc versus .net? I paid a consultant to look at what would work best and his speed tests indicated the odbc and the .net were pretty close, so I am assuming it is because I am using a transaction on my odbc test. I can run again without it or with it on .net driver, but thought I would ask. We only use transaction on important multiple table updates in our system now, so should not be a huge thing, but was curious. Joel Fradkin
Joel Fradkin wrote: > I wrote a .net program to move my data to postgres (works great on > SQ_ASCII). > > In fiddling around I tried it using the odbc driver and a transaction > originally, but converted it to using the .net connectivity but no > transaction. > > What I found was it moved my database (4 gig in MSSQL) in 2 hours using the > .net, but 12 hours with the odbc and transaction. You *are* using transactions, you don't have a choice. Did you do the transfer of all 4GB in ONE transaction with the ODBC? Please describe the process in more detail. -- Richard Huxton Archonet Ltd
No I did not do it in on transaction (although in .net I never started or commited a transaction. ODBC :myTrans = myConnection.BeginTransaction(IsolationLevel.ReadCommitted) ' Assign transaction object for a pending local transactionmyCommand.Transaction = myTrans 'example of insert 'myCommand.CommandText = "Insert into Region (RegionID, RegionDescription) VALUES (101, 'Description')" myCommand.CommandText = insertsqltext myCommand.ExecuteNonQuery() myTrans.Commit() .net driver: Dim cmd As New NpgsqlCommand(insertsqltext, cnn) cmd.ExecuteNonQuery() cmd.Dispose() Joel Fradkin Wazagua, Inc. 2520 Trailmate Dr Sarasota, Florida 34243 Tel. 941-753-7111 ext 305 jfradkin@wazagua.com www.wazagua.com Powered by Wazagua Providing you with the latest Web-based technology & advanced tools. C 2004. WAZAGUA, Inc. All rights reserved. WAZAGUA, IncThis email message is for the use of the intended recipient(s) andmay contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and delete and destroy all copies of the original message, including attachments. -----Original Message----- From: Richard Huxton [mailto:dev@archonet.com] Sent: Wednesday, February 23, 2005 3:03 AM To: Joel Fradkin Cc: pgsql-sql@postgresql.org Subject: Re: [SQL] Speeds using a transaction vrs not Joel Fradkin wrote: > I wrote a .net program to move my data to postgres (works great on > SQ_ASCII). > > In fiddling around I tried it using the odbc driver and a transaction > originally, but converted it to using the .net connectivity but no > transaction. > > What I found was it moved my database (4 gig in MSSQL) in 2 hours using the > .net, but 12 hours with the odbc and transaction. You *are* using transactions, you don't have a choice. Did you do the transfer of all 4GB in ONE transaction with the ODBC? Please describe the process in more detail. -- Richard Huxton Archonet Ltd
Joel Fradkin wrote: > No I did not do it in on transaction (although in .net I never started or > commited a transaction. All inserts/updates/etc take place within a transaction with PostgreSQL. Some client libraries autocommit for you - you'll need to read the documentation. > ODBC : > myCommand.CommandText = insertsqltext > myCommand.ExecuteNonQuery() > myTrans.Commit() > > .net driver: > Dim cmd As New NpgsqlCommand(insertsqltext, cnn) > cmd.ExecuteNonQuery() > cmd.Dispose() Both look to me like they are producing one transaction per insert (the slowest possible way to bulk-copy data). That's assuming each block of commands is within one loop. Precisely what is happening will be easiest to see by turning statement logging on in your postgresql.conf and comparing two runs. The delay might be in overheads of setting up the transaction etc. with the ODBC driver. In any case, if bulk-copying data you'll find a huge improvement grouping rows together in batches of 100 - 10,000. -- Richard Huxton Archonet Ltd
Thanks. I guess I could add that logic, but this is a one time process going to run the night we go live on postgres. With .net it took like 2 hours to do the whole shebang, so I am happy. But I was curious as we will be using odbc for our asp (aprx 90% of our app). I could test wrapping the .net in a transaction the way I do in odbc, but you indicate it is doing that behind the scenes. So far progress is going really great on our conversion. I cant wait to see us up on the new platform. I have not researched the details on doing autovacuum and backup. I have tested manually backing up and restoring. We are only like 10% done converting the code (I had hoped it going faster, but I have not been hands on helping, so maybe it go faster when I get time to start helping). I did convert a fairly large app and it seemed to run faster on the postgres box then the MSSQL box, so my first project after actually getting the data and views has gone very smooth. Everyone has been so helpful on the lists and that is the only reason I am where I am so far so thanks again to everyone who has bothered to answer my noob questions. Joel Fradkin Wazagua, Inc. 2520 Trailmate Dr Sarasota, Florida 34243 Tel. 941-753-7111 ext 305 jfradkin@wazagua.com www.wazagua.com Powered by Wazagua Providing you with the latest Web-based technology & advanced tools. C 2004. WAZAGUA, Inc. All rights reserved. WAZAGUA, IncThis email message is for the use of the intended recipient(s) andmay contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and delete and destroy all copies of the original message, including attachments. -----Original Message----- From: Richard Huxton [mailto:dev@archonet.com] Sent: Thursday, February 24, 2005 9:21 AM To: Joel Fradkin Cc: pgsql-sql@postgresql.org Subject: Re: [SQL] Speeds using a transaction vrs not Joel Fradkin wrote: > No I did not do it in on transaction (although in .net I never started or > commited a transaction. All inserts/updates/etc take place within a transaction with PostgreSQL. Some client libraries autocommit for you - you'll need to read the documentation. > ODBC : > myCommand.CommandText = insertsqltext > myCommand.ExecuteNonQuery() > myTrans.Commit() > > .net driver: > Dim cmd As New NpgsqlCommand(insertsqltext, cnn) > cmd.ExecuteNonQuery() > cmd.Dispose() Both look to me like they are producing one transaction per insert (the slowest possible way to bulk-copy data). That's assuming each block of commands is within one loop. Precisely what is happening will be easiest to see by turning statement logging on in your postgresql.conf and comparing two runs. The delay might be in overheads of setting up the transaction etc. with the ODBC driver. In any case, if bulk-copying data you'll find a huge improvement grouping rows together in batches of 100 - 10,000. -- Richard Huxton Archonet Ltd
Joel Fradkin wrote: > Thanks. > > I guess I could add that logic, but this is a one time process going to run > the night we go live on postgres. > With .net it took like 2 hours to do the whole shebang, so I am happy. Fair enough. I tend to produce a text-file and use COPY for bulk transfers, but that's just me. > I did convert a fairly large app and it seemed to run faster on the postgres > box then the MSSQL box, so my first project after actually getting the data > and views has gone very smooth. Don't forget to set aside some time for tweaking configuration settings so you can tune the system. -- Richard Huxton Archonet Ltd