Postgres Pro
Downloads
Home   >   mailing lists

Thread: Performance

Performance

From
Anex Hul
Date:
Hello everyone,

Testing 100 million records data import from Azure blob storage to Azure postgresql. I did run the test 5 times and the time it took keep increasing for each run. 
Is there know justification for this linear increment of the time it took for same size of data?

Re: Performance

From
Ron Johnson
Date:
On Sun, Dec 15, 2024 at 8:22 PM Anex Hul <anexsql2014@gmail.com> wrote:
Hello everyone,

Testing 100 million records data import from Azure blob storage to Azure postgresql. I did run the test 5 times and the time it took keep increasing for each run. 
Is there know justification for this linear increment of the time it took for same size of data?

1. What version of PG is it?  ("SELECT VERSION();" should tell you.)
2. Are you truncating the table after each test run, or deleting all records, or appending?
3. Is the blob data stored in BYTEA column data, or are you using the (discouraged) "Large Objects"?
4. How are you loading the blob data?

--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

Re: Performance

From
Rui DeSousa
Date:

> On Dec 15, 2024, at 8:22 PM, Anex Hul <anexsql2014@gmail.com> wrote:
>
> Hello everyone,
>
> Testing 100 million records data import from Azure blob storage to Azure postgresql. I did run the test 5 times and
thetime it took keep increasing for each run.  
> Is there know justification for this linear increment of the time it took for same size of data?

Check you I/O quotas; you might have hit quota limits and being throttled.

Re: Performance

From
Anex Hul
Date:

Thank you all for your response.

Show quoted text
1. What version of PG is it?  ("SELECT VERSION();" should tell you.)

PG Version 16

2. Are you truncating the table after each test run, or deleting all records, or appending?

created new schema for each run.

3. Is the blob data stored in BYTEA column data, or are you using the (discouraged) "Large Objects"?

Blob storage

4. How are you loading the blob data?

used the Import data using a COPY statement, followed this doc

https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-use-pg-azure-storage?tabs=portal-01%2Cportal-02%2Cportal-03%2Cportal-04%2Cportal-05

On Sun, Dec 15, 2024, 10:22 PM Rui DeSousa <rui.desousa@icloud.com> wrote:


> On Dec 15, 2024, at 8:22 PM, Anex Hul <anexsql2014@gmail.com> wrote:
>
> Hello everyone,
>
> Testing 100 million records data import from Azure blob storage to Azure postgresql. I did run the test 5 times and the time it took keep increasing for each run.
> Is there know justification for this linear increment of the time it took for same size of data?

Check you I/O quotas; you might have hit quota limits and being throttled.

Re: Performance

From
Ron Johnson
Date:
On Mon, Dec 16, 2024 at 9:05 AM Anex Hul <anexsql2014@gmail.com> wrote:
[snip] 

2. Are you truncating the table after each test run, or deleting all records, or appending?

created new schema for each run.

3. Is the blob data stored in BYTEA column data, or are you using the (discouraged) "Large Objects"?

Blob storage

Postgresql does not know what "Blob storage" means.
 

4. How are you loading the blob data?

used the Import data using a COPY statement, followed this doc

https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-use-pg-azure-storage?tabs=portal-01%2Cportal-02%2Cportal-03%2Cportal-04%2Cportal-05

If you're using a Microsoft extension, then you'd better ask Microsoft.

--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!