Thread: Performance degradation on concurrent COPY into a single relation in PG16.

Hi all,

While testing PG16, I observed that in PG16 there is a big performance
degradation in concurrent COPY into a single relation with 2 - 16
clients in my environment. I've attached a test script that measures
the execution time of COPYing 5GB data in total to the single relation
while changing the number of concurrent insertions, in PG16 and PG15.
Here are the results on my environment (EC2 instance, RHEL 8.6, 128
vCPUs, 512GB RAM):

* PG15 (4b15868b69)
PG15: nclients = 1, execution time = 14.181
PG15: nclients = 2, execution time = 9.319
PG15: nclients = 4, execution time = 5.872
PG15: nclients = 8, execution time = 3.773
PG15: nclients = 16, execution time = 3.202
PG15: nclients = 32, execution time = 3.023
PG15: nclients = 64, execution time = 3.829
PG15: nclients = 128, execution time = 4.111
PG15: nclients = 256, execution time = 4.158

* PG16 (c24e9ef330)
PG16: nclients = 1, execution time = 17.112
PG16: nclients = 2, execution time = 14.084
PG16: nclients = 4, execution time = 27.997
PG16: nclients = 8, execution time = 10.554
PG16: nclients = 16, execution time = 7.074
PG16: nclients = 32, execution time = 4.607
PG16: nclients = 64, execution time = 2.093
PG16: nclients = 128, execution time = 2.141
PG16: nclients = 256, execution time = 2.202

PG16 has better scalability (more than 64 clients) but it took much
more time than PG15, especially at 1 - 16 clients.

The relevant commit is 00d1e02be2 "hio: Use ExtendBufferedRelBy() to
extend tables more efficiently". With commit 1cbbee0338 (the previous
commit of 00d1e02be2), I got a better numbers, it didn't have a better
scalability, though:

PG16: nclients = 1, execution time = 17.444
PG16: nclients = 2, execution time = 10.690
PG16: nclients = 4, execution time = 7.010
PG16: nclients = 8, execution time = 4.282
PG16: nclients = 16, execution time = 3.373
PG16: nclients = 32, execution time = 3.205
PG16: nclients = 64, execution time = 3.705
PG16: nclients = 128, execution time = 4.196
PG16: nclients = 256, execution time = 4.201

While investigating the cause, I found an interesting fact that in
mdzeroextend if I use only either FileFallocate() or FileZero, we can
get better numbers. For example, If I always use FileZero with the
following change:

@@ -574,7 +574,7 @@ mdzeroextend(SMgrRelation reln, ForkNumber forknum,
         * that decision should be made though? For now just use a cutoff of
         * 8, anything between 4 and 8 worked OK in some local testing.
         */
-       if (numblocks > 8)
+       if (false)
        {
            int         ret;

I got:

PG16: nclients = 1, execution time = 16.898
PG16: nclients = 2, execution time = 8.740
PG16: nclients = 4, execution time = 4.656
PG16: nclients = 8, execution time = 2.733
PG16: nclients = 16, execution time = 2.021
PG16: nclients = 32, execution time = 1.693
PG16: nclients = 64, execution time = 1.742
PG16: nclients = 128, execution time = 2.180
PG16: nclients = 256, execution time = 2.296

After further investigation, the performance degradation comes from
calling posix_fallocate() (called via FileFallocate()) and pwritev()
(called via FileZero) alternatively depending on how many blocks we
extend by. And it happens only on the xfs filesystem. Does anyone
observe a similar performance issue with the attached benchmark
script?

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Mon, Jul 3, 2023 at 11:55 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> After further investigation, the performance degradation comes from
> calling posix_fallocate() (called via FileFallocate()) and pwritev()
> (called via FileZero) alternatively depending on how many blocks we
> extend by. And it happens only on the xfs filesystem.

FYI, the attached simple C program proves the fact that calling
alternatively posix_fallocate() and pwrite() causes slow performance
on posix_fallocate():

$ gcc -o test test.c
$ time ./test test.1 1
total   200000
fallocate       200000
filewrite       0

real    0m1.305s
user    0m0.050s
sys     0m1.255s

$ time ./test test.2 2
total   200000
fallocate       100000
filewrite       100000

real    1m29.222s
user    0m0.139s
sys     0m3.139s

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On 03/07/2023 05:59, Masahiko Sawada wrote:
> On Mon, Jul 3, 2023 at 11:55 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>
>> After further investigation, the performance degradation comes from
>> calling posix_fallocate() (called via FileFallocate()) and pwritev()
>> (called via FileZero) alternatively depending on how many blocks we
>> extend by. And it happens only on the xfs filesystem.
> 
> FYI, the attached simple C program proves the fact that calling
> alternatively posix_fallocate() and pwrite() causes slow performance
> on posix_fallocate():
> 
> $ gcc -o test test.c
> $ time ./test test.1 1
> total   200000
> fallocate       200000
> filewrite       0
> 
> real    0m1.305s
> user    0m0.050s
> sys     0m1.255s
> 
> $ time ./test test.2 2
> total   200000
> fallocate       100000
> filewrite       100000
> 
> real    1m29.222s
> user    0m0.139s
> sys     0m3.139s

This must be highly dependent on the underlying OS and filesystem. I'm 
not seeing that effect on my laptop:

/data$ time /tmp/test test.0 0
total    200000
fallocate    0
filewrite    200000

real    0m1.856s
user    0m0.140s
sys    0m1.688s
/data$ time /tmp/test test.1 1
total    200000
fallocate    200000
filewrite    0

real    0m1.335s
user    0m0.156s
sys    0m1.179s
/data$ time /tmp/test test.2 2
total    200000
fallocate    100000
filewrite    100000

real    0m2.159s
user    0m0.165s
sys    0m1.880s

/data$ uname -a
Linux heikkilaptop 6.0.0-6-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.0.12-1 
(2022-12-09) x86_64 GNU/Linux

/data is an nvme drive with ext4 filesystem.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

On Mon, Jul 3, 2023 at 4:36 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 03/07/2023 05:59, Masahiko Sawada wrote:
> > On Mon, Jul 3, 2023 at 11:55 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >>
> >> After further investigation, the performance degradation comes from
> >> calling posix_fallocate() (called via FileFallocate()) and pwritev()
> >> (called via FileZero) alternatively depending on how many blocks we
> >> extend by. And it happens only on the xfs filesystem.
> >
> > FYI, the attached simple C program proves the fact that calling
> > alternatively posix_fallocate() and pwrite() causes slow performance
> > on posix_fallocate():
> >
> > $ gcc -o test test.c
> > $ time ./test test.1 1
> > total   200000
> > fallocate       200000
> > filewrite       0
> >
> > real    0m1.305s
> > user    0m0.050s
> > sys     0m1.255s
> >
> > $ time ./test test.2 2
> > total   200000
> > fallocate       100000
> > filewrite       100000
> >
> > real    1m29.222s
> > user    0m0.139s
> > sys     0m3.139s
>
> This must be highly dependent on the underlying OS and filesystem.

Right. The above were the result where I created the file on the xfs
filesystem. The kernel version and the xfs filesystem version are:

% uname -rms
Linux 4.18.0-372.9.1.el8.x86_64 x86_64

% sudo xfs_db -r /dev/nvme4n1p2
xfs_db> version
versionnum [0xb4b5+0x18a] =
V5,NLINK,DIRV2,ATTR,ALIGN,LOGV2,EXTFLG,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT,CRC,FTYPE,FINOBT,SPARSE_INODES,REFLINK

As far as I tested, it happens only on the xfs filesystem (at least
the above version) and doesn't happen on ext4 and ext3 filesystems.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hello,

On 2023-Jul-03, Masahiko Sawada wrote:

> While testing PG16, I observed that in PG16 there is a big performance
> degradation in concurrent COPY into a single relation with 2 - 16
> clients in my environment. I've attached a test script that measures
> the execution time of COPYing 5GB data in total to the single relation
> while changing the number of concurrent insertions, in PG16 and PG15.

This item came up in the RMT meeting.  Andres, I think this item belongs
to you, because of commit 00d1e02be2.

The regression seems serious enough at low client counts:

> * PG15 (4b15868b69)
> PG15: nclients = 1, execution time = 14.181
> PG15: nclients = 2, execution time = 9.319
> PG15: nclients = 4, execution time = 5.872
> PG15: nclients = 8, execution time = 3.773
> PG15: nclients = 16, execution time = 3.202

> * PG16 (c24e9ef330)
> PG16: nclients = 1, execution time = 17.112
> PG16: nclients = 2, execution time = 14.084
> PG16: nclients = 4, execution time = 27.997
> PG16: nclients = 8, execution time = 10.554
> PG16: nclients = 16, execution time = 7.074

So the fact that the speed has clearly gone up at larger client counts
is not an excuse for not getting it fixed, XFS-specificity
notwithstanding.

> The relevant commit is 00d1e02be2 "hio: Use ExtendBufferedRelBy() to
> extend tables more efficiently". With commit 1cbbee0338 (the previous
> commit of 00d1e02be2), I got a better numbers, it didn't have a better
> scalability, though:
> 
> PG16: nclients = 1, execution time = 17.444
> PG16: nclients = 2, execution time = 10.690
> PG16: nclients = 4, execution time = 7.010
> PG16: nclients = 8, execution time = 4.282
> PG16: nclients = 16, execution time = 3.373

Well, these numbers are better, but they still look worse than PG15.
I suppose there are other commits that share blame.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"La virtud es el justo medio entre dos defectos" (Aristóteles)

Hi,

On 2023-07-10 15:25:41 +0200, Alvaro Herrera wrote:
> On 2023-Jul-03, Masahiko Sawada wrote:
> 
> > While testing PG16, I observed that in PG16 there is a big performance
> > degradation in concurrent COPY into a single relation with 2 - 16
> > clients in my environment. I've attached a test script that measures
> > the execution time of COPYing 5GB data in total to the single relation
> > while changing the number of concurrent insertions, in PG16 and PG15.
> 
> This item came up in the RMT meeting.  Andres, I think this item belongs
> to you, because of commit 00d1e02be2.

I'll take a look - I wasn't even aware of this thread until now.

Greetings,

Andres Freund

Hi,

On 2023-07-03 11:55:13 +0900, Masahiko Sawada wrote:
> While testing PG16, I observed that in PG16 there is a big performance
> degradation in concurrent COPY into a single relation with 2 - 16
> clients in my environment. I've attached a test script that measures
> the execution time of COPYing 5GB data in total to the single relation
> while changing the number of concurrent insertions, in PG16 and PG15.
> Here are the results on my environment (EC2 instance, RHEL 8.6, 128
> vCPUs, 512GB RAM):

Gah, RHEL with its frankenkernels, the bane of my existance.

FWIW, I had extensively tested this with XFS, just with a newer kernel. Have
you tested this on RHEL9 as well by any chance?

Greetings,

Andres Freund

Hi,

On 2023-07-03 11:59:38 +0900, Masahiko Sawada wrote:
> On Mon, Jul 3, 2023 at 11:55 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > After further investigation, the performance degradation comes from
> > calling posix_fallocate() (called via FileFallocate()) and pwritev()
> > (called via FileZero) alternatively depending on how many blocks we
> > extend by. And it happens only on the xfs filesystem.
>
> FYI, the attached simple C program proves the fact that calling
> alternatively posix_fallocate() and pwrite() causes slow performance
> on posix_fallocate():
>
> $ gcc -o test test.c
> $ time ./test test.1 1
> total   200000
> fallocate       200000
> filewrite       0
>
> real    0m1.305s
> user    0m0.050s
> sys     0m1.255s
>
> $ time ./test test.2 2
> total   200000
> fallocate       100000
> filewrite       100000
>
> real    1m29.222s
> user    0m0.139s
> sys     0m3.139s

On an xfs filesystem, with a very recent kernel:

time /tmp/msw_test /srv/dev/fio/msw 0
total    200000
fallocate    0
filewrite    200000

real    0m0.456s
user    0m0.017s
sys    0m0.439s


time /tmp/msw_test /srv/dev/fio/msw 1
total    200000
fallocate    200000
filewrite    0

real    0m0.141s
user    0m0.010s
sys    0m0.131s


time /tmp/msw_test /srv/dev/fio/msw 2
total    200000
fallocate    100000
filewrite    100000

real    0m0.297s
user    0m0.017s
sys    0m0.280s


So I don't think I can reproduce your problem on that system...

I also tried adding a fdatasync() into the loop, but that just made things
uniformly slow.


I guess I'll try to dig up whether this is a problem in older upstream
kernels, or whether it's been introduced in RHEL.

Greetings,

Andres Freund

Hi,

On 2023-07-03 11:53:56 +0200, Jakub Wartak wrote:
> Out of curiosity I've tried and it is reproducible as you have stated : XFS
> @ 4.18.0-425.10.1.el8_7.x86_64:
>...
> According to iostat and blktrace -d /dev/sda -o - | blkparse -i - output ,
> the XFS issues sync writes while ext4 does not, xfs looks like constant
> loop of sync writes (D) by kworker/2:1H-kblockd:

That clearly won't go well.  It's not reproducible on newer systems,
unfortunately :(. Or well, fortunately maybe.


I wonder if a trick to avoid this could be to memorialize the fact that we
bulk extended before and extend by that much going forward? That'd avoid the
swapping back and forth.


One thing that confuses me is that Sawada-san observed a regression at a
single client - yet from what I can tell, there's actually not a single
fallocate() being generated in that case, because the table is so narrow that
we never end up extending by a sufficient number of blocks in
RelationAddBlocks() to reach that path. Yet there's a regression at 1 client.

I don't yet have a RHEL8 system at hand, could either of you send the result
of a
  strace -c -p $pid_of_backend_doing_copy

Greetings,

Andres Freund

On Tue, Jul 11, 2023 at 12:34 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2023-07-03 11:55:13 +0900, Masahiko Sawada wrote:
> > While testing PG16, I observed that in PG16 there is a big performance
> > degradation in concurrent COPY into a single relation with 2 - 16
> > clients in my environment. I've attached a test script that measures
> > the execution time of COPYing 5GB data in total to the single relation
> > while changing the number of concurrent insertions, in PG16 and PG15.
> > Here are the results on my environment (EC2 instance, RHEL 8.6, 128
> > vCPUs, 512GB RAM):
>
> Gah, RHEL with its frankenkernels, the bane of my existance.
>
> FWIW, I had extensively tested this with XFS, just with a newer kernel. Have
> you tested this on RHEL9 as well by any chance?

I've tested this on RHEL9 (m5.24xlarge; 96vCPUs, 384GB RAM), and it
seems to be reproducible on RHEL9 too.

$ uname -rms
Linux 5.14.0-284.11.1.el9_2.x86_64 x86_64
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 9.2 (Plow)

PG15: nclients = 1, execution time = 22.193
PG15: nclients = 2, execution time = 12.430
PG15: nclients = 4, execution time = 8.677
PG15: nclients = 8, execution time = 6.144
PG15: nclients = 16, execution time = 5.938
PG15: nclients = 32, execution time = 5.775
PG15: nclients = 64, execution time = 5.928
PG15: nclients = 128, execution time = 6.346
PG15: nclients = 256, execution time = 6.641

PG16: nclients = 1, execution time = 24.601
PG16: nclients = 2, execution time = 27.845
PG16: nclients = 4, execution time = 40.575
PG16: nclients = 8, execution time = 24.379
PG16: nclients = 16, execution time = 15.835
PG16: nclients = 32, execution time = 8.370
PG16: nclients = 64, execution time = 4.042
PG16: nclients = 128, execution time = 2.956
PG16: nclients = 256, execution time = 2.591

Tests with test.c program:

$ rm -f test.data; time ./test test.data 0
total   200000
fallocate       0
filewrite       200000

real    0m0.709s
user    0m0.057s
sys     0m0.649s

$ rm -f test.data; time ./test test.data 1
total   200000
fallocate       200000
filewrite       0

real    0m0.853s
user    0m0.058s
sys     0m0.791s

$ rm -f test.data; time ./test test.data 2
total   200000
fallocate       100000
filewrite       100000

real    2m10.002s
user    0m0.366s
sys     0m6.649s

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Tue, Jul 11, 2023 at 1:24 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2023-07-03 11:53:56 +0200, Jakub Wartak wrote:
> > Out of curiosity I've tried and it is reproducible as you have stated : XFS
> > @ 4.18.0-425.10.1.el8_7.x86_64:
> >...
> > According to iostat and blktrace -d /dev/sda -o - | blkparse -i - output ,
> > the XFS issues sync writes while ext4 does not, xfs looks like constant
> > loop of sync writes (D) by kworker/2:1H-kblockd:
>
> That clearly won't go well.  It's not reproducible on newer systems,
> unfortunately :(. Or well, fortunately maybe.
>
>
> I wonder if a trick to avoid this could be to memorialize the fact that we
> bulk extended before and extend by that much going forward? That'd avoid the
> swapping back and forth.
>
>
> One thing that confuses me is that Sawada-san observed a regression at a
> single client - yet from what I can tell, there's actually not a single
> fallocate() being generated in that case, because the table is so narrow that
> we never end up extending by a sufficient number of blocks in
> RelationAddBlocks() to reach that path. Yet there's a regression at 1 client.
>
> I don't yet have a RHEL8 system at hand, could either of you send the result
> of a
>   strace -c -p $pid_of_backend_doing_copy
>

Here are the results:

* PG16: nclients = 1, execution time = 23.758
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 53.18    0.308675           0    358898           pwrite64
 33.92    0.196900           2     81202           pwritev
  7.78    0.045148           0     81378           lseek
  5.06    0.029371           2     11141           read
  0.04    0.000250           2        91           openat
  0.02    0.000094           1        89           close
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0        84           brk
  0.00    0.000000           0         1           sendto
  0.00    0.000000           0         2         1 recvfrom
  0.00    0.000000           0         2           kill
  0.00    0.000000           0         1           futex
  0.00    0.000000           0         1           epoll_wait
------ ----------- ----------- --------- --------- ----------------
100.00    0.580438           1    532891         1 total

* PG16: nclients = 2, execution time = 18.045
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 54.19    0.218721           1    187803           pwrite64
 29.24    0.118002           2     40242           pwritev
  6.23    0.025128           0     41239           lseek
  5.03    0.020285           2      9133           read
  2.04    0.008229           9       855           fallocate
  1.28    0.005151           0      5598      1000 futex
  1.12    0.004516           1      3965           kill
  0.78    0.003141           1      3058         1 epoll_wait
  0.06    0.000224           2       100           openat
  0.03    0.000136           1        98           close
  0.01    0.000050           0        84           brk
  0.00    0.000013           0        22           setitimer
  0.00    0.000006           0        15         1 rt_sigreturn
  0.00    0.000002           2         1           munmap
  0.00    0.000002           2         1           sendto
  0.00    0.000002           0         3         2 recvfrom
------ ----------- ----------- --------- --------- ----------------
100.00    0.403608           1    292217      1004 total

* PG16: nclients = 4, execution time = 18.807
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 64.61    0.240171           2     93868           pwrite64
 15.11    0.056173           4     11337           pwritev
  7.29    0.027100           7      3465           fallocate
  4.05    0.015048           2      5179           read
  3.55    0.013188           0     14941           lseek
  2.65    0.009858           1      8675      2527 futex
  1.76    0.006536           1      4190           kill
  0.88    0.003268           1      2199           epoll_wait
  0.06    0.000213           2       101           openat
  0.03    0.000130           1        99           close
  0.01    0.000031           1        18           rt_sigreturn
  0.01    0.000027           1        17           setitimer
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0        84           brk
  0.00    0.000000           0         1           sendto
  0.00    0.000000           0         1           recvfrom
------ ----------- ----------- --------- --------- ----------------
100.00    0.371743           2    144176      2527 total

* PG16: nclients = 8, execution time = 11.982
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 73.16    0.180095           3     47895           pwrite64
  8.61    0.021194           5      4199           pwritev
  5.93    0.014598           6      2199           fallocate
  3.42    0.008425           1      6723      2206 futex
  3.18    0.007824           2      3068           read
  2.44    0.005995           0      6510           lseek
  1.82    0.004475           1      2665           kill
  1.27    0.003118           1      1758         2 epoll_wait
  0.10    0.000239           2        95           openat
  0.06    0.000141           1        93           close
  0.01    0.000034           2        16           setitimer
  0.01    0.000020           2        10         2 rt_sigreturn
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0        84           brk
  0.00    0.000000           0         1           sendto
  0.00    0.000000           0         2         1 recvfrom
------ ----------- ----------- --------- --------- ----------------
100.00    0.246158           3     75319      2211 total

* PG16: nclients = 16, execution time = 7.507
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 79.45    0.078310           5     14870           pwrite64
  5.52    0.005440           5       973           pwritev
  4.51    0.004443           6       640           fallocate
  3.69    0.003640           1      2884      1065 futex
  2.23    0.002200           2       866           read
  1.80    0.001775           1      1685           lseek
  1.44    0.001421           1       782           kill
  1.08    0.001064           2       477         1 epoll_wait
  0.13    0.000129           2        57           openat
  0.08    0.000078           1        56           close
  0.06    0.000055           0        84           brk
  0.00    0.000003           3         1           munmap
  0.00    0.000003           3         1           sendto
  0.00    0.000003           1         2         1 recvfrom
  0.00    0.000002           0         5           setitimer
  0.00    0.000001           0         3         1 rt_sigreturn
------ ----------- ----------- --------- --------- ----------------
100.00    0.098567           4     23386      1068 total

* PG16: nclients = 32, execution time = 4.644
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 88.90    0.147208          12     11571           pwrite64
  3.11    0.005151           1      2643       943 futex
  2.61    0.004314           4      1039           pwritev
  1.74    0.002879           8       327           fallocate
  1.21    0.001998           3       624           read
  0.90    0.001498           1      1439           lseek
  0.66    0.001090           3       358         1 epoll_wait
  0.63    0.001049           2       426           kill
  0.12    0.000206           3        65           openat
  0.07    0.000118           1        64           close
  0.03    0.000045           0        84           brk
  0.01    0.000011          11         1           munmap
  0.00    0.000008           8         1           sendto
  0.00    0.000007           3         2         1 recvfrom
  0.00    0.000002           0         3         1 rt_sigreturn
  0.00    0.000001           0         3           setitimer
------ ----------- ----------- --------- --------- ----------------
100.00    0.165585           8     18650       946 total

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Mon, Jul 10, 2023 at 6:24 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2023-07-03 11:53:56 +0200, Jakub Wartak wrote:
> > Out of curiosity I've tried and it is reproducible as you have stated : XFS
> > @ 4.18.0-425.10.1.el8_7.x86_64:
> >...
> > According to iostat and blktrace -d /dev/sda -o - | blkparse -i - output ,
> > the XFS issues sync writes while ext4 does not, xfs looks like constant
> > loop of sync writes (D) by kworker/2:1H-kblockd:
>
> That clearly won't go well.  It's not reproducible on newer systems,
> unfortunately :(. Or well, fortunately maybe.
>
>
> I wonder if a trick to avoid this could be to memorialize the fact that we
> bulk extended before and extend by that much going forward? That'd avoid the
> swapping back and forth.

I haven't seen this thread [1] "Question on slow fallocate", from XFS
mailing list being mentioned here (it was started by Masahiko), but I
do feel it contains very important hints even challenging the whole
idea of zeroing out files (or posix_fallocate()). Please especially
see Dave's reply. He also argues that posix_fallocate() !=
fallocate().  What's interesting is that it's by design and newer
kernel versions should not prevent such behaviour, see my testing
result below.

All I can add is that this those kernel versions (4.18.0) seem to very
popular across customers (RHEL, Rocky) right now and that I've tested
on most recent available one (4.18.0-477.15.1.el8_8.x86_64) using
Masahiko test.c and still got 6-7x slower time when using XFS on that
kernel. After installing kernel-ml (6.4.2) the test.c result seems to
be the same (note it it occurs only when 1st allocating space, but of
course it doesnt if the same file is rewritten/"reallocated"):

[root@rockyora ~]# uname -r
6.4.2-1.el8.elrepo.x86_64
[root@rockyora ~]# time ./test test.0 0
total   200000
fallocate       0
filewrite       200000

real    0m0.405s
user    0m0.006s
sys     0m0.391s
[root@rockyora ~]# time ./test test.0 1
total   200000
fallocate       200000
filewrite       0

real    0m0.137s
user    0m0.005s
sys     0m0.132s
[root@rockyora ~]# time ./test test.1 1
total   200000
fallocate       200000
filewrite       0

real    0m0.968s
user    0m0.020s
sys     0m0.928s
[root@rockyora ~]# time ./test test.2 2
total   200000
fallocate       100000
filewrite       100000

real    0m6.059s
user    0m0.000s
sys     0m0.788s
[root@rockyora ~]# time ./test test.2 2
total   200000
fallocate       100000
filewrite       100000

real    0m0.598s
user    0m0.003s
sys     0m0.225s
[root@rockyora ~]#

iostat -x reports during first "time ./test test.2 2" (as you can see
w_awiat is not that high but it accumulates):
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s
%rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              0.00 15394.00      0.00    122.02     0.00    13.00
0.00   0.08    0.00    0.05   0.75     0.00     8.12   0.06 100.00
dm-0             0.00 15407.00      0.00    122.02     0.00     0.00
0.00   0.00    0.00    0.06   0.98     0.00     8.11   0.06 100.00

So maybe that's just a hint that you should try on slower storage
instead? (I think that on NVMe this issue would be hardly noticeable
due to low IO latency, not like here)

-J.

[1] - https://www.spinics.net/lists/linux-xfs/msg73035.html

Hi,

On 2023-07-11 09:09:43 +0200, Jakub Wartak wrote:
> On Mon, Jul 10, 2023 at 6:24 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2023-07-03 11:53:56 +0200, Jakub Wartak wrote:
> > > Out of curiosity I've tried and it is reproducible as you have stated : XFS
> > > @ 4.18.0-425.10.1.el8_7.x86_64:
> > >...
> > > According to iostat and blktrace -d /dev/sda -o - | blkparse -i - output ,
> > > the XFS issues sync writes while ext4 does not, xfs looks like constant
> > > loop of sync writes (D) by kworker/2:1H-kblockd:
> >
> > That clearly won't go well.  It's not reproducible on newer systems,
> > unfortunately :(. Or well, fortunately maybe.
> >
> >
> > I wonder if a trick to avoid this could be to memorialize the fact that we
> > bulk extended before and extend by that much going forward? That'd avoid the
> > swapping back and forth.
>
> I haven't seen this thread [1] "Question on slow fallocate", from XFS
> mailing list being mentioned here (it was started by Masahiko), but I
> do feel it contains very important hints even challenging the whole
> idea of zeroing out files (or posix_fallocate()). Please especially
> see Dave's reply.

I think that's just due to the reproducer being a bit too minimal and the
actual problem being addressed not being explained.


> He also argues that posix_fallocate() != fallocate().  What's interesting is
> that it's by design and newer kernel versions should not prevent such
> behaviour, see my testing result below.

I think the problem there was that I was not targetting a different file
between the different runs, somehow assuming the test program would be taking
care of that.

I don't think the test program actually tests things in a particularly useful
way - it does fallocate()s in 8k chunks - which postgres never does.



> All I can add is that this those kernel versions (4.18.0) seem to very
> popular across customers (RHEL, Rocky) right now and that I've tested
> on most recent available one (4.18.0-477.15.1.el8_8.x86_64) using
> Masahiko test.c and still got 6-7x slower time when using XFS on that
> kernel. After installing kernel-ml (6.4.2) the test.c result seems to
> be the same (note it it occurs only when 1st allocating space, but of
> course it doesnt if the same file is rewritten/"reallocated"):

test.c really doesn't reproduce postgres behaviour in any meaningful way,
using it as a benchmark is a bad idea.

Greetings,

Andres Freund

Hi,

On 2023-07-03 11:55:13 +0900, Masahiko Sawada wrote:
> While testing PG16, I observed that in PG16 there is a big performance
> degradation in concurrent COPY into a single relation with 2 - 16
> clients in my environment. I've attached a test script that measures
> the execution time of COPYing 5GB data in total to the single relation
> while changing the number of concurrent insertions, in PG16 and PG15.
> Here are the results on my environment (EC2 instance, RHEL 8.6, 128
> vCPUs, 512GB RAM):
>
> * PG15 (4b15868b69)
> PG15: nclients = 1, execution time = 14.181
>
> * PG16 (c24e9ef330)
> PG16: nclients = 1, execution time = 17.112

> The relevant commit is 00d1e02be2 "hio: Use ExtendBufferedRelBy() to
> extend tables more efficiently". With commit 1cbbee0338 (the previous
> commit of 00d1e02be2), I got a better numbers, it didn't have a better
> scalability, though:
>
> PG16: nclients = 1, execution time = 17.444

I think the single client case is indicative of an independent regression, or
rather regressions - it can't have anything to do with the fallocate() issue
and reproduces before that too in your numbers.

1) COPY got slower, due to:
9f8377f7a27 Add a DEFAULT option to COPY  FROM

This added a new palloc()/free() to every call to NextCopyFrom(). It's not at
all clear to me why this needs to happen in NextCopyFrom(), particularly
because it's already stored in CopyFromState?


2) pg_strtoint32_safe() got substantially slower, mainly due
   to
faff8f8e47f Allow underscores in integer and numeric constants.
6fcda9aba83 Non-decimal integer literals

pinned to one cpu, turbo mode disabled, I get the following best-of-three times for
  copy test from '/tmp/tmp_4.data'
(too impatient to use the larger file every time)

15:
6281.107 ms

HEAD:
7000.469 ms

backing out 9f8377f7a27:
6433.516 ms

also backing out faff8f8e47f, 6fcda9aba83:
6235.453 ms


I suspect 1) can relatively easily be fixed properly. But 2) seems much
harder. The changes increased the number of branches substantially, that's
gonna cost in something as (previously) tight as pg_strtoint32().



For higher concurrency numbers, I now was able to reproduce the regression, to
a smaller degree. Much smaller after fixing the above. The reason we run into
the issue here is basically that the rows in the test are very narrow and reach

#define MAX_BUFFERED_TUPLES        1000

at a small number of pages, so we go back and forth between extending with
fallocate() and not.

I'm *not* saying that that is the solution, but after changing that to 5000,
the numbers look a lot better (with the other regressions "worked around"):

(this is again with turboboost disabled, to get more reproducible numbers)

clients                1       2       4       8       16      32

15,buffered=1000                25725   13211   9232    5639    4862    4700
15,buffered=5000                26107   14550   8644    6050    4943    4766
HEAD+fixes,buffered=1000        25875   14505   8200    4900    3565    3433
HEAD+fixes,buffered=5000        25830   12975   6527    3594    2739    2642

Greetings,

Andres Freund

[1] https://postgr.es/m/CAD21AoAEwHTLYhuQ6PaBRPXKWN-CgW9iw%2B4hm%3D2EOFXbJQ3tOg%40mail.gmail.com

On Wed, Jul 12, 2023 at 3:52 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2023-07-03 11:55:13 +0900, Masahiko Sawada wrote:
> > While testing PG16, I observed that in PG16 there is a big performance
> > degradation in concurrent COPY into a single relation with 2 - 16
> > clients in my environment. I've attached a test script that measures
> > the execution time of COPYing 5GB data in total to the single relation
> > while changing the number of concurrent insertions, in PG16 and PG15.
> > Here are the results on my environment (EC2 instance, RHEL 8.6, 128
> > vCPUs, 512GB RAM):
> >
> > * PG15 (4b15868b69)
> > PG15: nclients = 1, execution time = 14.181
> >
> > * PG16 (c24e9ef330)
> > PG16: nclients = 1, execution time = 17.112
>
> > The relevant commit is 00d1e02be2 "hio: Use ExtendBufferedRelBy() to
> > extend tables more efficiently". With commit 1cbbee0338 (the previous
> > commit of 00d1e02be2), I got a better numbers, it didn't have a better
> > scalability, though:
> >
> > PG16: nclients = 1, execution time = 17.444
>
> I think the single client case is indicative of an independent regression, or
> rather regressions - it can't have anything to do with the fallocate() issue
> and reproduces before that too in your numbers.

Right.

>
> 1) COPY got slower, due to:
> 9f8377f7a27 Add a DEFAULT option to COPY  FROM
>
> This added a new palloc()/free() to every call to NextCopyFrom(). It's not at
> all clear to me why this needs to happen in NextCopyFrom(), particularly
> because it's already stored in CopyFromState?

Yeah, it seems to me that we can palloc the bool array once and use it
for the entire COPY FROM. With the attached small patch, the
performance becomes much better:

15:
14.70500 sec

16:
17.42900 sec

16 w/ patch:
14.85600 sec

>
>
> 2) pg_strtoint32_safe() got substantially slower, mainly due
>    to
> faff8f8e47f Allow underscores in integer and numeric constants.
> 6fcda9aba83 Non-decimal integer literals

Agreed.

>
> pinned to one cpu, turbo mode disabled, I get the following best-of-three times for
>   copy test from '/tmp/tmp_4.data'
> (too impatient to use the larger file every time)
>
> 15:
> 6281.107 ms
>
> HEAD:
> 7000.469 ms
>
> backing out 9f8377f7a27:
> 6433.516 ms
>
> also backing out faff8f8e47f, 6fcda9aba83:
> 6235.453 ms
>
>
> I suspect 1) can relatively easily be fixed properly. But 2) seems much
> harder. The changes increased the number of branches substantially, that's
> gonna cost in something as (previously) tight as pg_strtoint32().

I'll look at how to fix 2).

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Wed, Jul 12, 2023 at 5:40 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Jul 12, 2023 at 3:52 AM Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2023-07-03 11:55:13 +0900, Masahiko Sawada wrote:
> > > While testing PG16, I observed that in PG16 there is a big performance
> > > degradation in concurrent COPY into a single relation with 2 - 16
> > > clients in my environment. I've attached a test script that measures
> > > the execution time of COPYing 5GB data in total to the single relation
> > > while changing the number of concurrent insertions, in PG16 and PG15.
> > > Here are the results on my environment (EC2 instance, RHEL 8.6, 128
> > > vCPUs, 512GB RAM):
> > >
> > > * PG15 (4b15868b69)
> > > PG15: nclients = 1, execution time = 14.181
> > >
> > > * PG16 (c24e9ef330)
> > > PG16: nclients = 1, execution time = 17.112
> >
> > > The relevant commit is 00d1e02be2 "hio: Use ExtendBufferedRelBy() to
> > > extend tables more efficiently". With commit 1cbbee0338 (the previous
> > > commit of 00d1e02be2), I got a better numbers, it didn't have a better
> > > scalability, though:
> > >
> > > PG16: nclients = 1, execution time = 17.444
> >
> > I think the single client case is indicative of an independent regression, or
> > rather regressions - it can't have anything to do with the fallocate() issue
> > and reproduces before that too in your numbers.
>
> Right.
>
> >
> > 1) COPY got slower, due to:
> > 9f8377f7a27 Add a DEFAULT option to COPY  FROM
> >
> > This added a new palloc()/free() to every call to NextCopyFrom(). It's not at
> > all clear to me why this needs to happen in NextCopyFrom(), particularly
> > because it's already stored in CopyFromState?
>
> Yeah, it seems to me that we can palloc the bool array once and use it
> for the entire COPY FROM. With the attached small patch, the
> performance becomes much better:
>
> 15:
> 14.70500 sec
>
> 16:
> 17.42900 sec
>
> 16 w/ patch:
> 14.85600 sec
>
> >
> >
> > 2) pg_strtoint32_safe() got substantially slower, mainly due
> >    to
> > faff8f8e47f Allow underscores in integer and numeric constants.
> > 6fcda9aba83 Non-decimal integer literals
>
> Agreed.
>
> >
> > pinned to one cpu, turbo mode disabled, I get the following best-of-three times for
> >   copy test from '/tmp/tmp_4.data'
> > (too impatient to use the larger file every time)
> >
> > 15:
> > 6281.107 ms
> >
> > HEAD:
> > 7000.469 ms
> >
> > backing out 9f8377f7a27:
> > 6433.516 ms
> >
> > also backing out faff8f8e47f, 6fcda9aba83:
> > 6235.453 ms
> >
> >
> > I suspect 1) can relatively easily be fixed properly. But 2) seems much
> > harder. The changes increased the number of branches substantially, that's
> > gonna cost in something as (previously) tight as pg_strtoint32().
>
> I'll look at how to fix 2).

I have made some progress on dealing with performance regression on
single client COPY. I've attached a patch to fix 2). With the patch I
shared[1] to deal with 1), single client COPY performance seems to be
now as good as (or slightly better than) PG15 . Here are the results
(averages of 5 times) of loading 50M rows via COPY:

15:
7.609 sec

16:
8.637 sec

16 w/ two patches:
7.179 sec

Regards,

[1] https://www.postgresql.org/message-id/CAD21AoBb9Sbddh%2BnQk1BxUFaRHaa%2BfL8fCzO-Lvxqb6ZcpAHqw%40mail.gmail.com

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Wed, 19 Jul 2023 at 09:24, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > > 2) pg_strtoint32_safe() got substantially slower, mainly due
> > >    to
> > > faff8f8e47f Allow underscores in integer and numeric constants.
> > > 6fcda9aba83 Non-decimal integer literals
> >
> > Agreed.
> >
> I have made some progress on dealing with performance regression on
> single client COPY. I've attached a patch to fix 2). With the patch I
> shared[1] to deal with 1), single client COPY performance seems to be
> now as good as (or slightly better than) PG15 . Here are the results
> (averages of 5 times) of loading 50M rows via COPY:
>

Hmm, I'm somewhat sceptical about this second patch. It's not obvious
why adding such tests would speed it up, and indeed, testing on my
machine with 50M rows, I see a noticeable speed-up from patch 1, and a
slow-down from patch 2:


PG15
====

7390.461 ms
7497.655 ms
7485.850 ms
7406.336 ms

HEAD
====

8388.707 ms
8283.484 ms
8391.638 ms
8363.306 ms

HEAD + P1
=========

7255.128 ms
7185.319 ms
7197.822 ms
7191.176 ms

HEAD + P2
=========

8687.164 ms
8654.907 ms
8641.493 ms
8668.865 ms

HEAD + P1 + P2
==============

7780.126 ms
7786.427 ms
7775.047 ms
7785.938 ms


So for me at least, just applying patch 1 gives the best results, and
makes it slightly faster than PG15 (possibly due to 6b423ec677).

Regards,
Dean

On Wed, 19 Jul 2023 at 23:14, Dean Rasheed <dean.a.rasheed@gmail.com> wrote:
> Hmm, I'm somewhat sceptical about this second patch. It's not obvious
> why adding such tests would speed it up, and indeed, testing on my
> machine with 50M rows, I see a noticeable speed-up from patch 1, and a
> slow-down from patch 2:

I noticed that 6fcda9aba added quite a lot of conditions that need to
be checked before we process a normal decimal integer string. I think
we could likely do better and code it to assume that most strings will
be decimal and put that case first rather than last.

In the attached, I've changed that for the 32-bit version only.  A
more complete patch would need to do the 16 and 64-bit versions too.

-- setup
create table a (a int);
insert into a select x from generate_series(1,10000000)x;
copy a to '~/a.dump';

-- test
truncate a; copy a from '/tmp/a.dump';

master @ ab29a7a9c
Time: 2152.633 ms (00:02.153)
Time: 2121.091 ms (00:02.121)
Time: 2100.995 ms (00:02.101)
Time: 2101.724 ms (00:02.102)
Time: 2103.949 ms (00:02.104)

master + pg_strtoint32_base_10_first.patch
Time: 2061.464 ms (00:02.061)
Time: 2035.513 ms (00:02.036)
Time: 2028.356 ms (00:02.028)
Time: 2043.218 ms (00:02.043)
Time: 2037.035 ms (00:02.037) (~3.6% faster)

Without that, we need to check if the first digit is '0' a total of 3
times and also check if the 2nd digit is any of 'x', 'X', 'o', 'O',
'b' or 'B'. It seems to be coded with the assumption that hex strings
are the most likely. I think decimals are the most likely by far.

David

On Thu, 20 Jul 2023 at 00:56, David Rowley <dgrowleyml@gmail.com> wrote:
>
> I noticed that 6fcda9aba added quite a lot of conditions that need to
> be checked before we process a normal decimal integer string. I think
> we could likely do better and code it to assume that most strings will
> be decimal and put that case first rather than last.

That sounds sensible, but ...

> In the attached, I've changed that for the 32-bit version only.  A
> more complete patch would need to do the 16 and 64-bit versions too.
>
> Without that, we need to check if the first digit is '0' a total of 3
> times and also check if the 2nd digit is any of 'x', 'X', 'o', 'O',
> 'b' or 'B'.

That's not what I see. For me, the compiler (gcc 11, using either -O2
or -O3) is smart enough to spot that the first part of each of the 3
checks is the same, and it only tests whether the first digit is '0'
once, before immediately jumping to the decimal parsing code. I didn't
test other compilers though, so I can believe that different compilers
might not be so smart.

OTOH, this test in your patch:

+    /* process decimal digits */
+    if (isdigit((unsigned char) ptr[0]) &&
+        (isdigit((unsigned char) ptr[1]) || ptr[1] == '_' || ptr[1]
== '\0' || isspace(ptr[1])))

is doing more work than it needs to, and actually makes things
noticeably worse for me. It needs to do a minimum of 2 isdigit()
checks before it will parse the input as a decimal, whereas before
(for me at least) it just did one simple comparison of ptr[0] against
'0'.

I agree with the principal though. In the attached updated patch, I
replaced that test with a simpler one:

+    /*
+     * Process the number's digits. We optimize for decimal input (expected to
+     * be the most common case) first. Anything that doesn't start with a base
+     * prefix indicator must be decimal.
+     */
+
+   /* process decimal digits */
+   if (likely(ptr[0] != '0' || !isalpha((unsigned char) ptr[1])))

So hopefully any compiler should only need to do the one comparison
against '0' for any non-zero decimal input.

Doing that didn't give any measurable performance improvement for me,
but it did at least not make it noticeably worse, and seems more
likely to generate better code with not-so-smart compilers. I'd be
interested to know how that performs for you (and if your compiler
really does generate 3 "first digit is '0'" tests for the unpatched
code).

Regards,
Dean

---

Here were my test results (where P1 is the "fix_COPY_DEFAULT.patch"),
and I tested COPY loading 50M rows:

HEAD + P1
=========

7137.966 ms
7193.190 ms
7094.491 ms
7123.520 ms

HEAD + P1 + pg_strtoint32_base_10_first.patch
=============================================

7561.658 ms
7548.282 ms
7551.360 ms
7560.671 ms

HEAD + P1 + pg_strtoint32_base_10_first.v2.patch
================================================

7238.775 ms
7184.937 ms
7123.257 ms
7159.618 ms

On Thu, 20 Jul 2023 at 20:37, Dean Rasheed <dean.a.rasheed@gmail.com> wrote:
>
> On Thu, 20 Jul 2023 at 00:56, David Rowley <dgrowleyml@gmail.com> wrote:
> I agree with the principal though. In the attached updated patch, I
> replaced that test with a simpler one:
>
> +    /*
> +     * Process the number's digits. We optimize for decimal input (expected to
> +     * be the most common case) first. Anything that doesn't start with a base
> +     * prefix indicator must be decimal.
> +     */
> +
> +   /* process decimal digits */
> +   if (likely(ptr[0] != '0' || !isalpha((unsigned char) ptr[1])))
>
> So hopefully any compiler should only need to do the one comparison
> against '0' for any non-zero decimal input.
>
> Doing that didn't give any measurable performance improvement for me,
> but it did at least not make it noticeably worse, and seems more
> likely to generate better code with not-so-smart compilers. I'd be
> interested to know how that performs for you (and if your compiler
> really does generate 3 "first digit is '0'" tests for the unpatched
> code).

That seems better.  I benchmarked it on two machines:

gcc12.2/linux/amd3990x
create table a (a int);
insert into a select x from generate_series(1,10000000)x;
copy a to '/tmp/a.dump';

master @ ab29a7a9c
postgres=# truncate a; copy a from '/tmp/a.dump';
Time: 2226.912 ms (00:02.227)
Time: 2214.168 ms (00:02.214)
Time: 2206.481 ms (00:02.206)
Time: 2219.640 ms (00:02.220)
Time: 2205.004 ms (00:02.205)

master + pg_strtoint32_base_10_first.v2.patch
postgres=# truncate a; copy a from '/tmp/a.dump';
Time: 2067.108 ms (00:02.067)
Time: 2070.401 ms (00:02.070)
Time: 2073.423 ms (00:02.073)
Time: 2065.407 ms (00:02.065)
Time: 2066.536 ms (00:02.067) (~7% faster)

apple m2 pro/clang

master @ 9089287a
postgres=# truncate a; copy a from '/tmp/a.dump';
Time: 1286.369 ms (00:01.286)
Time: 1300.534 ms (00:01.301)
Time: 1295.661 ms (00:01.296)
Time: 1296.404 ms (00:01.296)
Time: 1268.361 ms (00:01.268)
Time: 1259.321 ms (00:01.259)

master + pg_strtoint32_base_10_first.v2.patch
postgres=# truncate a; copy a from '/tmp/a.dump';
Time: 1253.519 ms (00:01.254)
Time: 1235.256 ms (00:01.235)
Time: 1269.501 ms (00:01.270)
Time: 1267.801 ms (00:01.268)
Time: 1275.758 ms (00:01.276)
Time: 1261.478 ms (00:01.261) (a bit noisy but avg of ~1.8% faster)

David

Hi,

Hm, in some cases your patch is better, but in others both the old code
(8692f6644e7) and HEAD beat yours on my machine. TBH, not entirely sure why.

prep:
COPY (SELECT generate_series(1, 2000000) a, (random() * 100000 - 50000)::int b, 3243423 c) TO '/tmp/lotsaints.copy';
DROP TABLE lotsaints; CREATE UNLOGGED TABLE lotsaints(a int, b int, c int);

benchmark:
psql -qX -c 'truncate lotsaints' && pgbench -n -P1 -f <( echo "COPY lotsaints FROM '/tmp/lotsaints.copy';") -t 15

I disabled turbo mode, pinned the server to a single core of my Xeon Gold 5215:

HEAD:                           812.690

your patch:                     821.354

strtoint from 8692f6644e7:      824.543

strtoint from 6b423ec677d^:     806.678

(when I say strtoint from, I did not replace the goto labels, so that part is
unchanged and unrelated)


IOW, for me the code from 15 is the fastest by a good bit... There's an imul,
sure, but the fact that it sets a flag makes it faster than having to add more
tests and branches.


Looking at a profile reminded me of how silly it is that we are wasting a good
chunk of the time in these isdigit() checks, even though we already rely on on
the ascii values via (via *ptr++ - '0'). I think that's done in the headers
for some platforms, but not others (glibc). And we've even done already for
octal and binary!

Open coding isdigit() gives us:


HEAD:                           797.434

your patch:                     803.570

strtoint from 8692f6644e7:      778.943

strtoint from 6b423ec677d^:     777.741


It's somewhat odd that HEAD and your patch switch position here...


> -    else if (ptr[0] == '0' && (ptr[1] == 'o' || ptr[1] == 'O'))
> +    /* process hex digits */
> +    else if (ptr[1] == 'x' || ptr[1] == 'X')
>      {
>
>          firstdigit = ptr += 2;

I find this unnecessarily hard to read. I realize it's been added in
6fcda9aba83, but I don't see a reason to use such a construct here.


I find it somewhat grating how much duplication there now is in this
code due to being repeated for all the bases...


Greetings,

Andres Freund

On Tue, 25 Jul 2023 at 17:34, Andres Freund <andres@anarazel.de> wrote:
> prep:
> COPY (SELECT generate_series(1, 2000000) a, (random() * 100000 - 50000)::int b, 3243423 c) TO '/tmp/lotsaints.copy';
> DROP TABLE lotsaints; CREATE UNLOGGED TABLE lotsaints(a int, b int, c int);
>
> benchmark:
> psql -qX -c 'truncate lotsaints' && pgbench -n -P1 -f <( echo "COPY lotsaints FROM '/tmp/lotsaints.copy';") -t 15
>
> I disabled turbo mode, pinned the server to a single core of my Xeon Gold 5215:
>
> HEAD:                           812.690
>
> your patch:                     821.354
>
> strtoint from 8692f6644e7:      824.543
>
> strtoint from 6b423ec677d^:     806.678

I'm surprised to see the imul version is faster. It's certainly not
what we found when working on 6b423ec67.

I've been fooling around a bit to try to learn what might be going on.
I wrote 2 patches:

1) pg_strtoint_imul.patch:  Effectively reverts 6b423ec67 and puts the
code how it likely would have looked had I not done that work at all.
(I've assumed that the hex, octal, binary parsing would have been
added using the overflow multiplication functions just as the base 10
parsing).

2) pg_strtoint_optimize.patch: I've made another pass over the
functions with the current overflow checks and optimized the digit
checking code so that it can be done in a single check like: if (digit
< 10).  This can be done by assigning the result of *ptr - '0' to an
unsigned char.  Anything less than '0' will wrap around and anything
above '9' will remain so.  I've also removed a few inefficiencies with
the isspace checking. We didn't need to do "while (*ptr &&
isspace(*ptr)).  It's fine just to do while (isspace(*ptr)) since '\0'
isn't a space char.  I also got rid of the isxdigit call.  The
hexlookup array contains -1 for non-hex chars. We may as well check
the digit value is >= 0.

Here are the results I get using your test as quoted above:

master @ 62e9af4c63f + fix_COPY_DEFAULT.patch

latency average = 568.102 ms

master @ 62e9af4c63f + fix_COPY_DEFAULT.patch + pg_strtoint_optimize.patch

latency average = 531.238 ms

master @ 62e9af4c63f + fix_COPY_DEFAULT.patch + pg_strtoint_imul.patch

latency average = 559.333 ms (surprisingly also faster on my machine)

The optimized version of the pg_strtoint functions wins over the imul
patch.  Could you test to see if this is also the case in your Xeon
machine?

> (when I say strtoint from, I did not replace the goto labels, so that part is
> unchanged and unrelated)

I didn't quite follow this.

I've not really studied the fix_COPY_DEFAULT.patch patch.  Is there a
reason to delay committing that?  It would be good to eliminate that
as a variable for the current performance regression.

David

Hi,

On 2023-07-25 23:37:08 +1200, David Rowley wrote:
> On Tue, 25 Jul 2023 at 17:34, Andres Freund <andres@anarazel.de> wrote:
> I've not really studied the fix_COPY_DEFAULT.patch patch.  Is there a
> reason to delay committing that?  It would be good to eliminate that
> as a variable for the current performance regression.

Yea, I don't think there's a reason to hold off on that. Sawada-san, do you
want to commit it? Or Andrew?


> > prep:
> > COPY (SELECT generate_series(1, 2000000) a, (random() * 100000 - 50000)::int b, 3243423 c) TO
'/tmp/lotsaints.copy';
> > DROP TABLE lotsaints; CREATE UNLOGGED TABLE lotsaints(a int, b int, c int);
> >
> > benchmark:
> > psql -qX -c 'truncate lotsaints' && pgbench -n -P1 -f <( echo "COPY lotsaints FROM '/tmp/lotsaints.copy';") -t 15
> >
> > I disabled turbo mode, pinned the server to a single core of my Xeon Gold 5215:
> >
> > HEAD:                           812.690
> >
> > your patch:                     821.354
> >
> > strtoint from 8692f6644e7:      824.543
> >
> > strtoint from 6b423ec677d^:     806.678
> 
> I'm surprised to see the imul version is faster. It's certainly not
> what we found when working on 6b423ec67.

What CPUs did you test it on? I'd not be surprised if this were heavily
dependent on the microarchitecture.


> I've been fooling around a bit to try to learn what might be going on.
> I wrote 2 patches:
> 
> 1) pg_strtoint_imul.patch:  Effectively reverts 6b423ec67 and puts the
> code how it likely would have looked had I not done that work at all.
> (I've assumed that the hex, octal, binary parsing would have been
> added using the overflow multiplication functions just as the base 10
> parsing).


> 2) pg_strtoint_optimize.patch: I've made another pass over the
> functions with the current overflow checks and optimized the digit
> checking code so that it can be done in a single check like: if (digit
> < 10).  This can be done by assigning the result of *ptr - '0' to an
> unsigned char.  Anything less than '0' will wrap around and anything
> above '9' will remain so.  I've also removed a few inefficiencies with
> the isspace checking. We didn't need to do "while (*ptr &&
> isspace(*ptr)).  It's fine just to do while (isspace(*ptr)) since '\0'
> isn't a space char.  I also got rid of the isxdigit call.  The
> hexlookup array contains -1 for non-hex chars. We may as well check
> the digit value is >= 0.
> 
> Here are the results I get using your test as quoted above:
> 
> master @ 62e9af4c63f + fix_COPY_DEFAULT.patch
> 
> latency average = 568.102 ms
> 
> master @ 62e9af4c63f + fix_COPY_DEFAULT.patch + pg_strtoint_optimize.patch
> 
> latency average = 531.238 ms
> 
> master @ 62e9af4c63f + fix_COPY_DEFAULT.patch + pg_strtoint_imul.patch
> 
> latency average = 559.333 ms (surprisingly also faster on my machine)
> 
> The optimized version of the pg_strtoint functions wins over the imul
> patch.  Could you test to see if this is also the case in your Xeon
> machine?

(these are the numbers with turbo disabled, if I enable it they're all in the
6xx ms range, but the variance is higher)


fix_COPY_DEFAULT.patch
774.344

fix_COPY_DEFAULT.patch + pg_strtoint32_base_10_first.v2.patch
777.673

fix_COPY_DEFAULT.patch + pg_strtoint_optimize.patch
777.545

fix_COPY_DEFAULT.patch + pg_strtoint_imul.patch
795.298

fix_COPY_DEFAULT.patch + pg_strtoint_imul.patch + likely(isdigit())
773.477

fix_COPY_DEFAULT.patch + pg_strtoint32_base_10_first.v2.patch + pg_strtoint_imul.patch
774.443

fix_COPY_DEFAULT.patch + pg_strtoint32_base_10_first.v2.patch + pg_strtoint_imul.patch + likely(isdigit())
774.513

fix_COPY_DEFAULT.patch + pg_strtoint32_base_10_first.v2.patch + pg_strtoint_imul.patch + likely(isdigit()) +
unlikely(*ptr== '_')
 
763.879


One idea I had was to add a fastpath that won't parse all strings, but will
parse the strings that we would generate, and fall back to the more general
variant if it fails. See the attached, rough, prototype:

fix_COPY_DEFAULT.patch + fastpath.patch:
746.971

fix_COPY_DEFAULT.patch + fastpath.patch + isdigit.patch:
715.570

Now, the precise contents of this fastpath are not yet clear (wrt imul or
not), but I think the idea has promise.



> > (when I say strtoint from, I did not replace the goto labels, so that part is
> > unchanged and unrelated)
> 
> I didn't quite follow this.

I meant that I didn't undo ereport()->ereturn().

Greetings,

Andres Freund

Hi,

On 2023-07-25 08:50:19 -0700, Andres Freund wrote:
> One idea I had was to add a fastpath that won't parse all strings, but will
> parse the strings that we would generate, and fall back to the more general
> variant if it fails. See the attached, rough, prototype:
> 
> fix_COPY_DEFAULT.patch + fastpath.patch:
> 746.971
> 
> fix_COPY_DEFAULT.patch + fastpath.patch + isdigit.patch:
> 715.570
> 
> Now, the precise contents of this fastpath are not yet clear (wrt imul or
> not), but I think the idea has promise.

Btw, I strongly suspect that fastpath wants to be branchless SSE when it grows
up.

Greetings,

Andres Freund

> On 2023-07-25 23:37:08 +1200, David Rowley wrote:
> > On Tue, 25 Jul 2023 at 17:34, Andres Freund <andres@anarazel.de> wrote:
> > > HEAD:                           812.690
> > >
> > > your patch:                     821.354
> > >
> > > strtoint from 8692f6644e7:      824.543
> > >
> > > strtoint from 6b423ec677d^:     806.678
> >
> > I'm surprised to see the imul version is faster. It's certainly not
> > what we found when working on 6b423ec67.
>
> What CPUs did you test it on? I'd not be surprised if this were heavily
> dependent on the microarchitecture.

This was on AMD 3990x.

> One idea I had was to add a fastpath that won't parse all strings, but will
> parse the strings that we would generate, and fall back to the more general
> variant if it fails. See the attached, rough, prototype:

There were a couple of problems with fastpath.patch.  You need to
reset the position of ptr at the start of the slow path and also you
were using tmp in the if (neg) part instead of tmp_s in the fast path
section.

I fixed that up and made two versions of the patch, one using the
overflow functions (pg_strtoint_fastpath1.patch) and one testing if
the number is going to overflow (same as current master)
(pg_strtoint_fastpath2.patch)

AMD 3990x:

master + fix_COPY_DEFAULT.patch:
latency average = 525.226 ms

master + fix_COPY_DEFAULT.patch + pg_strtoint_fastpath1.patch:
latency average = 488.171 ms

master + fix_COPY_DEFAULT.patch + pg_strtoint_fastpath2.patch:
latency average = 481.827 ms


Apple M2 Pro:

master + fix_COPY_DEFAULT.patch:
latency average = 348.433 ms

master + fix_COPY_DEFAULT.patch + pg_strtoint_fastpath1.patch:
latency average = 336.778 ms

master + fix_COPY_DEFAULT.patch + pg_strtoint_fastpath2.patch:
latency average = 335.992 ms

Zen 4 7945HX CPU:

master + fix_COPY_DEFAULT.patch:
latency average = 296.881 ms

master + fix_COPY_DEFAULT.patch + pg_strtoint_fastpath1.patch:
latency average = 287.052 ms

master + fix_COPY_DEFAULT.patch + pg_strtoint_fastpath2.patch:
latency average = 280.742 ms

The M2 chip does not seem to be clearly faster with the fastpath2
method of overflow checking, but both AMD CPUs seem pretty set on
fastpath2 being faster.

It would be really good if someone with another a newish intel CPU
could test this too.

David

On Wed, 26 Jul 2023 at 03:50, Andres Freund <andres@anarazel.de> wrote:
> On 2023-07-25 23:37:08 +1200, David Rowley wrote:
> > On Tue, 25 Jul 2023 at 17:34, Andres Freund <andres@anarazel.de> wrote:
> > I've not really studied the fix_COPY_DEFAULT.patch patch.  Is there a
> > reason to delay committing that?  It would be good to eliminate that
> > as a variable for the current performance regression.
>
> Yea, I don't think there's a reason to hold off on that. Sawada-san, do you
> want to commit it? Or Andrew?

Just to keep this moving and to make it easier for people to test the
pg_strtoint patches, I've pushed the fix_COPY_DEFAULT.patch patch.
The only thing I changed was to move the line that was allocating the
array to a location more aligned with the order that the fields are
defined in the struct.

David

On Thu, 27 Jul 2023 at 14:51, David Rowley <dgrowleyml@gmail.com> wrote:
> Just to keep this moving and to make it easier for people to test the
> pg_strtoint patches, I've pushed the fix_COPY_DEFAULT.patch patch.
> The only thing I changed was to move the line that was allocating the
> array to a location more aligned with the order that the fields are
> defined in the struct.

I just did another round of benchmarking to see where we're at after
fox_COPY_DEFAULT.patch has been committed.

Below I've benchmarked REL_15_STABLE up to REL_16_STABLE with some
hand-selected key commits, many of which have been mentioned on this
thread already because they seem to affect the performance of COPY.

To summarise, REL_15_STABLE can run this benchmark in 526.014 ms on my
AMD 3990x machine.  Today's REL_16_STABLE takes 530.344 ms. We're
talking about another patch to speed up the pg_strtoint functions
which gets this down to 483.790 ms. Do we need to do this for v16 or
can we just leave this as it is already?  The slowdown does not seem
to be much above what we'd ordinarily classify as noise using this
test on my machine.

Benchmark setup:

COPY (SELECT generate_series(1, 2000000) a, (random() * 100000 -
50000)::int b, 3243423 c) TO '/tmp/lotsaints.copy';
DROP TABLE lotsaints; CREATE UNLOGGED TABLE lotsaints(a int, b int, c int);

Benchmark:
psql -qX -c 'truncate lotsaints' && pgbench -n -P1 -f <( echo "COPY
lotsaints FROM '/tmp/lotsaints.copy';") -t 15

2864eb977 REL_15_STABLE
latency average = 526.014 ms

8.84%  postgres          [.] pg_strtoint32

29452de73 Sat Dec  3 10:50:39 2022 -0500 The commit before "Improve
performance of pg_strtointNN functions"
latency average = 508.453 ms

10.21%  postgres          [.] pg_strtoint32

6b423ec67 Sun Dec  4 16:18:18 2022 +1300 Improve performance of
pg_strtointNN functions
latency average = 492.943 ms

7.73%  postgres          [.] pg_strtoint32

1939d2628 Fri Dec  9 10:08:44 2022 -0500 The commit before "Convert a
few datatype input functions to use "soft" error reporting."
latency average = 485.016 ms

8.43%  postgres          [.] pg_strtoint32

ccff2d20e Fri Dec  9 10:14:53 2022 -0500 Convert a few datatype input
functions to use "soft" error reporting.
latency average = 501.325 ms

6.90%  postgres          [.] pg_strtoint32_safe

60684dd83 Tue Dec 13 17:33:28 2022 -0800 The commit before
"Non-decimal integer literals"
latency average = 500.889 ms

8.27%  postgres          [.] pg_strtoint32_safe

6fcda9aba Wed Dec 14 05:40:38 2022 +0100 Non-decimal integer literals
latency average = 521.904 ms

9.02%  postgres          [.] pg_strtoint32_safe

1b6f632a3 Sat Feb 4 07:56:09 2023 +0100 The commit before "Allow
underscores in integer and numeric constants."
latency average = 523.195 ms

9.21%  postgres          [.] pg_strtoint32_safe

faff8f8e4 Sat Feb  4 09:48:51 2023 +0000 Allow underscores in integer
and numeric constants.
latency average = 493.064 ms

10.25%  postgres          [.] pg_strtoint32_safe

9f8377f7a Mon Mar 13 10:01:56 2023 -0400 Add a DEFAULT option to COPY  FROM
latency average = 597.617 ms

  12.91%  postgres          [.] CopyReadLine
  10.62%  postgres          [.] CopyReadAttributesText
  10.51%  postgres          [.] pg_strtoint32_safe
   7.97%  postgres          [.] NextCopyFrom

REL_16_STABLE @ c1308ce2d Thu Jul 27 14:48:44 2023 +1200 Fix
performance problem with new COPY DEFAULT code
latency average = 530.344 ms

  13.51%  postgres          [.] CopyReadLine
   9.62%  postgres          [.] pg_strtoint32_safe
   8.97%  postgres          [.] CopyReadAttributesText
   8.43%  postgres          [.] NextCopyFrom

REL_16_STABLE + pg_strtoint_fastpath1.patch
latency average = 493.136 ms

  13.79%  postgres          [.] CopyReadLine
  11.82%  postgres          [.] CopyReadAttributesText
   7.07%  postgres          [.] NextCopyFrom
   6.81%  postgres          [.] pg_strtoint32_safe

REL_16_STABLE + pg_strtoint_fastpath2.patch
latency average = 483.790 ms

  13.87%  postgres          [.] CopyReadLine
  10.40%  postgres          [.] CopyReadAttributesText
   8.22%  postgres          [.] NextCopyFrom
   5.52%  postgres          [.] pg_strtoint32_safe

David

Hi,

On 2023-07-27 20:53:16 +1200, David Rowley wrote:
> To summarise, REL_15_STABLE can run this benchmark in 526.014 ms on my
> AMD 3990x machine.  Today's REL_16_STABLE takes 530.344 ms. We're
> talking about another patch to speed up the pg_strtoint functions
> which gets this down to 483.790 ms. Do we need to do this for v16 or
> can we just leave this as it is already?  The slowdown does not seem
> to be much above what we'd ordinarily classify as noise using this
> test on my machine.

I think we need to do something for 16 - it appears on recent-ish AMD the
regression is quite a bit smaller than on intel.  You see something < 1%, I
see more like 4%.  I think there's also other cases where the slowdown is more
substantial.

Besides intel vs amd, it also looks like the gcc version might make a
difference. The code generated by 13 is noticeably slower than 12 for me...

> Benchmark setup:
> 
> COPY (SELECT generate_series(1, 2000000) a, (random() * 100000 -
> 50000)::int b, 3243423 c) TO '/tmp/lotsaints.copy';
> DROP TABLE lotsaints; CREATE UNLOGGED TABLE lotsaints(a int, b int, c int);

There's a lot of larger numbers in the file, which likely reduces the impact
some. And there's the overhead of actually inserting the rows into the table,
making the difference appear smaller than it is.

If I avoid the actual insert into the table and use more columns, I see an about
10% regression here.

COPY (SELECT generate_series(1, 1000) a, 10 b, 20 c, 30 d, 40 e, 50 f FROM generate_series(1, 10000)) TO
'/tmp/lotsaints_wide.copy';

psql -c 'DROP TABLE IF EXISTS lotsaints_wide; CREATE UNLOGGED TABLE lotsaints_wide(a int, b int, c int, d int, e int, f
int);'&& \
 
  pgbench -n -P1 -f <( echo "COPY lotsaints_wide FROM '/tmp/lotsaints_wide.copy' WHERE false") -t 5

15:             2992.605
HEAD:           3325.201
fastpath1.patch 2932.606
fastpath2.patch 2783.915


Interestingly fastpath1 is slower now, even though it wasn't with earlier
patches (which still is repeatable). I do not have the foggiest as to why.

Greetings,

Andres Freund

On Mon, 31 Jul 2023 at 21:39, John Naylor <john.naylor@enterprisedb.com> wrote:
> master + pg_strtoint_fastpath1.patch
> latency average = 938.146 ms
> latency stddev = 9.354 ms
>
> master + pg_strtoint_fastpath2.patch
> latency average = 902.808 ms
> latency stddev = 3.957 ms

Thanks for checking those two on your machine. I'm glad to see
fastpath2 faster on yours too.

David

On Tue, 1 Aug 2023 at 13:25, Andres Freund <andres@anarazel.de> wrote:
> There's a lot of larger numbers in the file, which likely reduces the impact
> some. And there's the overhead of actually inserting the rows into the table,
> making the difference appear smaller than it is.

It might be worth special casing the first digit as we can save doing
the multiplication by 10 and the overflow checks on the first digit.
That should make it slightly faster to parse smaller numbers.

> COPY (SELECT generate_series(1, 1000) a, 10 b, 20 c, 30 d, 40 e, 50 f FROM generate_series(1, 10000)) TO
'/tmp/lotsaints_wide.copy';
>
> psql -c 'DROP TABLE IF EXISTS lotsaints_wide; CREATE UNLOGGED TABLE lotsaints_wide(a int, b int, c int, d int, e int,
fint);' && \
 
>   pgbench -n -P1 -f <( echo "COPY lotsaints_wide FROM '/tmp/lotsaints_wide.copy' WHERE false") -t 5
>
> 15:             2992.605
> HEAD:           3325.201
> fastpath1.patch 2932.606
> fastpath2.patch 2783.915
>
> Interestingly fastpath1 is slower now, even though it wasn't with earlier
> patches (which still is repeatable). I do not have the foggiest as to why.

I'm glad to see that.

I've adjusted the patch to add the fast path for the 16 and 64-bit
versions of the function.  I also added the special case for
processing the first digit, which looks like:

/* process the first digit */
digit = (*ptr - '0');

if (likely(digit < 10))
{
    ptr++;
    tmp = digit;
}

/* process remaining digits */
for (;;)

I tried adding the "at least 1 digit check" by adding an else { goto
slow; } in the above code, but it seems to generate slower code than
just checking if (unlikely(ptr == s)) { goto slow; } after the loop.

I also noticed that I wasn't getting the same performance after
adjusting the 16 and 64 bit versions. I assume that's down to code
alignment, but unsure of that.  I ended up adjusting all the "while
(*ptr)" loops into "for (;;)" loops
since the NUL char check is handled by the "else break;".  I also
removed the needless NUL char check in the isspace loops. It can't be
isspace and '\0'. I also replaced the isdigit() function call and
replaced it for manually checking the digit range.  I see my compiler
(gcc12.2) effectively generates the same code as the unsigned char
fast path version checking if (digit < 10). Once I did that, I got the
performance back again.

With your new test with the small-sized ints, I get:

REL_15_STABLE:
latency average = 1696.390 ms

master @ d3a38318a
latency average = 1928.803 ms

master + fastpath1.patch:
latency average = 1634.736 ms

master + fastpath2.patch:
latency average = 1628.704 ms

master + fastpath3.patch
latency average = 1590.417 ms

I see no particular reason not to go ahead with the attached patch and
get this thread closed off. Any objections?

David

On Tue, 1 Aug 2023 at 13:55, David Rowley <dgrowleyml@gmail.com> wrote:
>
> I tried adding the "at least 1 digit check" by adding an else { goto
> slow; } in the above code, but it seems to generate slower code than
> just checking if (unlikely(ptr == s)) { goto slow; } after the loop.
>

That check isn't quite right, because "ptr" will not equal "s" if
there is a sign character, so it won't detect an input with no digits
in that case.

Regards,
Dean

On Wed, 2 Aug 2023 at 01:26, Dean Rasheed <dean.a.rasheed@gmail.com> wrote:
>
> On Tue, 1 Aug 2023 at 13:55, David Rowley <dgrowleyml@gmail.com> wrote:
> >
> > I tried adding the "at least 1 digit check" by adding an else { goto
> > slow; } in the above code, but it seems to generate slower code than
> > just checking if (unlikely(ptr == s)) { goto slow; } after the loop.
> >
>
> That check isn't quite right, because "ptr" will not equal "s" if
> there is a sign character, so it won't detect an input with no digits
> in that case.

Ah, yeah. Thanks.

Here's a patch with an else condition when the first digit check fails.

master + fastpath4.patch:
latency average = 1579.576 ms
latency average = 1572.716 ms
latency average = 1563.398 ms

(appears slightly faster than fastpath3.patch)

David

On Tue, 1 Aug 2023 at 15:01, David Rowley <dgrowleyml@gmail.com> wrote:
>
> Here's a patch with an else condition when the first digit check fails.
>
> master + fastpath4.patch:
> latency average = 1579.576 ms
> latency average = 1572.716 ms
> latency average = 1563.398 ms
>
> (appears slightly faster than fastpath3.patch)
>

Running the new test on slightly older Intel hardware (i9-9900K, gcc
11.3), I get the following:

REL_15_STABLE
latency average = 1687.695 ms
latency stddev = 3.068 ms

REL_16_STABLE
latency average = 1931.756 ms
latency stddev = 2.065 ms

REL_16_STABLE + pg_strtoint_fastpath1.patch
latency average = 1635.731 ms
latency stddev = 3.074 ms

REL_16_STABLE + pg_strtoint_fastpath2.patch
latency average = 1687.303 ms
latency stddev = 4.243 ms

REL_16_STABLE + pg_strtoint_fastpath3.patch
latency average = 1610.307 ms
latency stddev = 2.193 ms

REL_16_STABLE + pg_strtoint_fastpath4.patch
latency average = 1577.604 ms
latency stddev = 4.060 ms

HEAD
latency average = 1868.737 ms
latency stddev = 6.114 ms

HEAD + pg_strtoint_fastpath1.patch
latency average = 1683.215 ms
latency stddev = 2.322 ms

HEAD + pg_strtoint_fastpath2.patch
latency average = 1650.014 ms
latency stddev = 3.920 ms

HEAD + pg_strtoint_fastpath3.patch
latency average = 1670.337 ms
latency stddev = 5.074 ms

HEAD + pg_strtoint_fastpath4.patch
latency average = 1653.568 ms
latency stddev = 8.224 ms

I don't know why HEAD and v16 aren't consistent, but it's seems to be
quite reproducible, even though the numutils source is the same in
both branches, and using gdb to dump the disassembly for
pg_strtoint32_safe() shows that it's also the same.

Anyway, insofar as these results can be trusted, fastpath4.patch looks good.

Regards,
Dean

On Wed, 2 Aug 2023 at 07:38, Dean Rasheed <dean.a.rasheed@gmail.com> wrote:
> Running the new test on slightly older Intel hardware (i9-9900K, gcc
> 11.3), I get the following:

Thanks for running those tests. I've now pushed the fastpath4.patch
after making a few adjustments to the header comments to mention the
new stuff that was added in v16.

> I don't know why HEAD and v16 aren't consistent, but it's seems to be
> quite reproducible, even though the numutils source is the same in
> both branches, and using gdb to dump the disassembly for
> pg_strtoint32_safe() shows that it's also the same.

I also see it's inconsistent, but the other way around. Here are some
fresh tests with master and REL_16_STABLE with the committed code and
the version directly prior to the commit:

master @ 3845577cb
latency average = 1575.879 ms

   6.79%  postgres          [.] pg_strtoint32_safe

master~1
latency average = 1968.004 ms

  14.28%  postgres          [.] pg_strtoint32_safe

REL_16_STABLE
latency average = 1735.163 ms

   6.04%  postgres          [.] pg_strtoint32_safe

REL_16_STABLE~1
latency average = 2188.186 ms

  13.83%  postgres          [.] pg_strtoint32_safe

David

On Wed, 2 Aug 2023 at 12:25, David Rowley <dgrowleyml@gmail.com> wrote:
> master @ 3845577cb
> latency average = 1575.879 ms
>
>    6.79%  postgres          [.] pg_strtoint32_safe
>
> master~1
> latency average = 1968.004 ms
>
>   14.28%  postgres          [.] pg_strtoint32_safe
>
> REL_16_STABLE
> latency average = 1735.163 ms
>
>    6.04%  postgres          [.] pg_strtoint32_safe
>
> REL_16_STABLE~1
> latency average = 2188.186 ms
>
>   13.83%  postgres          [.] pg_strtoint32_safe

And just to complete that set, here's the REL_15_STABLE performance
using the same test:

latency average = 1829.108 ms

15.46%  postgres          [.] pg_strtoint32

So, it looks like this item can be closed off.  I'll hold off from
doing that for a few days just in case anyone else wants to give
feedback or test themselves.

David

On Wed, 2 Aug 2023 at 13:35, David Rowley <dgrowleyml@gmail.com> wrote:
> So, it looks like this item can be closed off.  I'll hold off from
> doing that for a few days just in case anyone else wants to give
> feedback or test themselves.

Alright, closed.

David

On Mon, Aug 7, 2023 at 3:16 PM David Rowley <dgrowleyml@gmail.com> wrote:
>
> On Wed, 2 Aug 2023 at 13:35, David Rowley <dgrowleyml@gmail.com> wrote:
> > So, it looks like this item can be closed off.  I'll hold off from
> > doing that for a few days just in case anyone else wants to give
> > feedback or test themselves.
>
> Alright, closed.

IIUC the problem with multiple concurrent COPY is not resolved yet.
I've run the same benchmark that I used for the first report:

* PG15 (cb2ae5741f)
 nclients = 1, execution time = 15.213
 nclients = 2, execution time = 9.470
 nclients = 4, execution time = 6.508
 nclients = 8, execution time = 4.526
 nclients = 16, execution time = 3.788
 nclients = 32, execution time = 3.837
 nclients = 64, execution time = 4.286
 nclients = 128, execution time = 4.388
 nclients = 256, execution time = 4.333

* PG16 (67a007dc0c)
 nclients = 1, execution time = 14.494
 nclients = 2, execution time = 12.962
 nclients = 4, execution time = 17.757
 nclients = 8, execution time = 10.865
 nclients = 16, execution time = 7.371
 nclients = 32, execution time = 4.929
 nclients = 64, execution time = 2.212
 nclients = 128, execution time = 2.020
 nclients = 256, execution time = 2.196

The result of nclients = 1 became better thanks to recent fixes, but
there still seems to be the performance regression at nclient = 2~16
(on RHEL 8 and 9). Andres reported[1] that after changing
MAX_BUFFERED_TUPLES to 5000 the numbers became a lot better but it
would not be the solution, as he mentioned.

Regards,

[1] https://www.postgresql.org/message-id/20230711185159.v2j5vnyrtodnwhgz%40awork3.anarazel.de

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On 2023-08-07 23:05:39 +0900, Masahiko Sawada wrote:
> On Mon, Aug 7, 2023 at 3:16 PM David Rowley <dgrowleyml@gmail.com> wrote:
> >
> > On Wed, 2 Aug 2023 at 13:35, David Rowley <dgrowleyml@gmail.com> wrote:
> > > So, it looks like this item can be closed off.  I'll hold off from
> > > doing that for a few days just in case anyone else wants to give
> > > feedback or test themselves.
> >
> > Alright, closed.
>
> IIUC the problem with multiple concurrent COPY is not resolved yet.

Yea - it was just hard to analyze until the other regressions were fixed.


> The result of nclients = 1 became better thanks to recent fixes, but
> there still seems to be the performance regression at nclient = 2~16
> (on RHEL 8 and 9). Andres reported[1] that after changing
> MAX_BUFFERED_TUPLES to 5000 the numbers became a lot better but it
> would not be the solution, as he mentioned.

I think there could be a quite simple fix: Track by how much we've extended
the relation previously in the same bistate. If we already extended by many
blocks, it's very likey that we'll do so further.

A simple prototype patch attached. The results for me are promising. I copied
a smaller file [1], to have more accurate throughput results at shorter runs
(15s).

HEAD before:
clients         tps
1          41
2          76
4         136
8         248
16         360
32         375
64         317


HEAD after:
clients         tps
1          43
2          80
4         155
8         280
16         369
32         405
64         344

Any chance you could your benchmark? I don't see as much of a regression vs 16
as you...

Greetings,

Andres Freund

[1] COPY (SELECT generate_series(1, 100000)) TO '/tmp/data.copy';

On Tue, Aug 8, 2023 at 3:10 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2023-08-07 23:05:39 +0900, Masahiko Sawada wrote:
> > On Mon, Aug 7, 2023 at 3:16 PM David Rowley <dgrowleyml@gmail.com> wrote:
> > >
> > > On Wed, 2 Aug 2023 at 13:35, David Rowley <dgrowleyml@gmail.com> wrote:
> > > > So, it looks like this item can be closed off.  I'll hold off from
> > > > doing that for a few days just in case anyone else wants to give
> > > > feedback or test themselves.
> > >
> > > Alright, closed.
> >
> > IIUC the problem with multiple concurrent COPY is not resolved yet.
>
> Yea - it was just hard to analyze until the other regressions were fixed.
>
>
> > The result of nclients = 1 became better thanks to recent fixes, but
> > there still seems to be the performance regression at nclient = 2~16
> > (on RHEL 8 and 9). Andres reported[1] that after changing
> > MAX_BUFFERED_TUPLES to 5000 the numbers became a lot better but it
> > would not be the solution, as he mentioned.
>
> I think there could be a quite simple fix: Track by how much we've extended
> the relation previously in the same bistate. If we already extended by many
> blocks, it's very likey that we'll do so further.
>
> A simple prototype patch attached. The results for me are promising. I copied
> a smaller file [1], to have more accurate throughput results at shorter runs
> (15s).

Thank you for the patch!

>
> HEAD before:
> clients      tps
> 1             41
> 2             76
> 4            136
> 8            248
> 16           360
> 32           375
> 64           317
>
>
> HEAD after:
> clients      tps
> 1             43
> 2             80
> 4            155
> 8            280
> 16           369
> 32           405
> 64           344
>
> Any chance you could your benchmark? I don't see as much of a regression vs 16
> as you...

Sure. The results are promising for me too:

 nclients = 1, execution time = 13.743
 nclients = 2, execution time = 7.552
 nclients = 4, execution time = 4.758
 nclients = 8, execution time = 3.035
 nclients = 16, execution time = 2.172
 nclients = 32, execution time = 1.959
nclients = 64, execution time = 1.819
nclients = 128, execution time = 1.583
nclients = 256, execution time = 1.631

Here are results of the same benchmark test you used:

w/o patch:
clients    tps
1       66.702
2       59.696
4       73.783
8       168.115
16      400.134
32      574.098
64      565.373
128     526.303
256     591.751

w/ patch:
clients   tps
1       67.735
2       122.534
4       240.707
8       398.944
16      541.097
32      643.083
64      614.775
128     616.007
256     577.885

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On 2023-08-08 12:45:05 +0900, Masahiko Sawada wrote:
> > I think there could be a quite simple fix: Track by how much we've extended
> > the relation previously in the same bistate. If we already extended by many
> > blocks, it's very likey that we'll do so further.
> >
> > A simple prototype patch attached. The results for me are promising. I copied
> > a smaller file [1], to have more accurate throughput results at shorter runs
> > (15s).
> 
> Thank you for the patch!

Attached is a mildly updated version of that patch. No functional changes,
just polished comments and added a commit message.


> > Any chance you could your benchmark? I don't see as much of a regression vs 16
> > as you...
> 
> Sure. The results are promising for me too:
>
>  nclients = 1, execution time = 13.743
>  nclients = 2, execution time = 7.552
>  nclients = 4, execution time = 4.758
>  nclients = 8, execution time = 3.035
>  nclients = 16, execution time = 2.172
>  nclients = 32, execution time = 1.959
> nclients = 64, execution time = 1.819
> nclients = 128, execution time = 1.583
> nclients = 256, execution time = 1.631

Nice. We are consistently better than both 15 and "post integer parsing 16".


I'm really a bit baffled at myself for not using this approach from the get
go.

This also would make it much more beneficial to use a BulkInsertState in
nodeModifyTable.c, even without converting to table_multi_insert().


I was tempted to optimize RelationAddBlocks() a bit, by not calling
RelationExtensionLockWaiterCount() if we are already extending by
MAX_BUFFERS_TO_EXTEND_BY. Before this patch, it was pretty much impossible to
reach that case, because of the MAX_BUFFERED_* limits in copyfrom.c, but now
it's more common. But that probably ought to be done only HEAD, not 16.

A related oddity: RelationExtensionLockWaiterCount()->LockWaiterCount() uses
an exclusive lock on the lock partition - which seems not at all necessary?


Unless somebody sees a reason not to, I'm planning to push this?

Greetings,

Andres Freund

Hello,

On 2023-Aug-12, Andres Freund wrote:

> On 2023-08-08 12:45:05 +0900, Masahiko Sawada wrote:

> > > Any chance you could your benchmark? I don't see as much of a regression vs 16
> > > as you...
> > 
> > Sure. The results are promising for me too:
> >
> >  nclients = 1, execution time = 13.743
> >  nclients = 2, execution time = 7.552
> >  nclients = 4, execution time = 4.758
> >  nclients = 8, execution time = 3.035
> >  nclients = 16, execution time = 2.172
> >  nclients = 32, execution time = 1.959
> > nclients = 64, execution time = 1.819
> > nclients = 128, execution time = 1.583
> > nclients = 256, execution time = 1.631
> 
> Nice. We are consistently better than both 15 and "post integer parsing 16".

Since the wins from this patch were replicated and it has been pushed, I
understand that this open item can be marked as closed, so I've done
that.

Thanks,

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"Hay dos momentos en la vida de un hombre en los que no debería
especular: cuando puede permitírselo y cuando no puede" (Mark Twain)

On 2023-08-16 13:15:46 +0200, Alvaro Herrera wrote:
> Since the wins from this patch were replicated and it has been pushed, I
> understand that this open item can be marked as closed, so I've done
> that.

Thanks!

Andres Freund <andres@anarazel.de> writes:
> On 2023-08-16 13:15:46 +0200, Alvaro Herrera wrote:
>> Since the wins from this patch were replicated and it has been pushed, I
>> understand that this open item can be marked as closed, so I've done
>> that.

> Thanks!

It turns out that this patch is what's making buildfarm member
chipmunk fail in contrib/pg_visibility [1].  That's easily reproduced
by running the test with shared_buffers = 10MB.  I didn't dig further
than the "git bisect" result:

$ git bisect bad
82a4edabd272f70d044faec8cf7fd1eab92d9991 is the first bad commit
commit 82a4edabd272f70d044faec8cf7fd1eab92d9991
Author: Andres Freund <andres@anarazel.de>
Date:   Mon Aug 14 09:54:03 2023 -0700

    hio: Take number of prior relation extensions into account

but I imagine the problem is that the patch's more aggressive
relation-extension heuristic is causing the table to have more
pages than the test case expects.  Is it really a good idea
for such a heuristic to depend on shared_buffers?  If you don't
want to change the heuristic then we'll have to find a way to
tweak this test to avoid it.

            regards, tom lane

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=chipmunk&dt=2023-09-06%2014%3A14%3A51

Hi,

On 2023-09-06 18:01:53 -0400, Tom Lane wrote:
> It turns out that this patch is what's making buildfarm member
> chipmunk fail in contrib/pg_visibility [1].  That's easily reproduced
> by running the test with shared_buffers = 10MB.  I didn't dig further
> than the "git bisect" result:

At first I was a bit confounded by not being able to reproduce this. My test
environment had max_connections=110 for some other reason - and the problem
doesn't reproduce with that setting...


> $ git bisect bad
> 82a4edabd272f70d044faec8cf7fd1eab92d9991 is the first bad commit
> commit 82a4edabd272f70d044faec8cf7fd1eab92d9991
> Author: Andres Freund <andres@anarazel.de>
> Date:   Mon Aug 14 09:54:03 2023 -0700
>
>     hio: Take number of prior relation extensions into account
>
> but I imagine the problem is that the patch's more aggressive
> relation-extension heuristic is causing the table to have more
> pages than the test case expects.  Is it really a good idea
> for such a heuristic to depend on shared_buffers?

The heuristic doesn't directly depend on shared buffers. However, the amount
we extend by is limited by needing to pin shared buffers covering all the
newly extended buffers.

That's what ends up limiting things here - shared_buffers = 10MB and
max_connections = 10 doesn't allow for a lot of buffers to be pinned
concurrently in each backend.  Although perhaps LimitAdditionalPins() is a bit
too conservative, due to not checking the private refcount array and just
assuming REFCOUNT_ARRAY_ENTRIES.


> If you don't want to change the heuristic then we'll have to find a way to
> tweak this test to avoid it.

We could tweak LimitAdditionalPins() by checking PrivateRefCountArray instead
of assuming the worst-case REFCOUNT_ARRAY_ENTRIES.

However, it seems that the logic in the test is pretty fragile independent of
this issue? Different alignment, block size or an optimization of the page
layout could also break the test?

Unfortunately a query that doesn't falsely alert in this case is a bit ugly,
due to needing to deal with the corner case of an empty page at the end:

select *
from pg_visibility_map('copyfreeze')
where
  (not all_visible or not all_frozen)
  -- deal with trailing empty pages due to potentially bulk-extending too aggressively
  and exists(SELECT * FROM copyfreeze WHERE ctid >= ('('||blkno||', 0)')::tid)
;

Before 82a4edabd27 this situation was rare - you'd have needed contended
extensions. But after it has become more common. I worry that that might cause
other issues :(. OTOH, I think we'll need to extend way more aggressively at
some point...

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2023-09-06 18:01:53 -0400, Tom Lane wrote:
>> It turns out that this patch is what's making buildfarm member
>> chipmunk fail in contrib/pg_visibility [1].  That's easily reproduced
>> by running the test with shared_buffers = 10MB.  I didn't dig further
>> than the "git bisect" result:

> At first I was a bit confounded by not being able to reproduce this. My test
> environment had max_connections=110 for some other reason - and the problem
> doesn't reproduce with that setting...

I just did a git bisect run to discover when the failure documented
in bug #18130 [1] started.  And the answer is commit 82a4edabd.
Now, it's pretty obvious that that commit didn't in itself cause
problems like this:

postgres=# \copy test from 'bug18130.csv' csv
ERROR:  could not read block 5 in file "base/5/17005": read only 0 of 8192 bytes
CONTEXT:  COPY test, line 472: "0,185647715,222655,489637,2,2023-07-31,9100.0000000,302110385,2023-07-30
14:16:36.750981+00,14026347..."

IMO there must be some very nasty bug lurking in the new
multiple-block extension logic, that happens to be exposed by this
test case with 82a4edabd's adjustments to the when-to-extend choices
but wasn't before that.

To save other people the trouble of extracting the in-line data
in the bug submission, I've attached the test files I was using.
The DDL is simplified slightly from what was submitted.  I'm not
entirely sure why a no-op trigger is needed to provoke the bug...

            regards, tom lane

[1] https://www.postgresql.org/message-id/18130-7a86a7356a75209d%40postgresql.org

-- run this, then \copy test from 'bug18130.csv' csv

create table if not exists test (
    a smallint,
    b bigint,
    c bigint,
    d bigint,
    e smallint,
    plan_date date,
    g numeric(18,7),
    h bigint,
    i timestamptz,
    j bigint,
    k integer,
    l smallint,
    primary key (a, b, c, d, e, plan_date)
) partition by list(a);

/* this is just to generate partition structure automatically */
create or replace function test_add_partitions(a integer, year integer)
    returns void
    volatile strict
as
$$
declare
    parent text;
    root text;
begin
    root := 'test_' || a::text;
    execute 'create table if not exists ' || root || ' partition of test for
values in (' || a::text || ') partition by hash (b);';
    for b in 0..7 loop
            parent := root || '_' || b::text;

            execute 'create table if not exists ' || parent ||
                    ' partition of ' || root || ' for values with (modulus
8, remainder ' || b::text || ')' ||
                    ' partition by range (plan_date);';

            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm1 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-01-01'') to
(''' || year::text ||
                    '-02-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm2 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-02-01'') to
(''' || year::text ||
                    '-03-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm3 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-03-01'') to
(''' || year::text ||
                    '-04-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm4 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-04-01'') to
(''' || year::text ||
                    '-05-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm5 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-05-01'') to
(''' || year::text ||
                    '-06-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm6 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-06-01'') to
(''' || year::text ||
                    '-07-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm7 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-07-01'') to
(''' || year::text ||
                    '-08-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm8 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-08-01'') to
(''' || year::text ||
                    '-09-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm9 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-09-01'') to
(''' || year::text ||
                    '-10-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm10 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-10-01'') to
(''' || year::text ||
                    '-11-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm11 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-11-01'') to
(''' || year::text ||
                    '-12-01'');';
            execute 'create table if not exists ' ||
                    parent || '_y' || year::text || 'm12 partition of ' ||
parent ||
                    ' for values from (''' || year::text || '-12-01'') to
(''' || (year + 1)::text ||
                    '-01-01'');';
        end loop;
end;
$$ language plpgsql;

select test_add_partitions(0, 2022);
select test_add_partitions(0, 2023);
select test_add_partitions(0, 2024);

create or replace function t_test_trigger_func()
    returns trigger as
$$
BEGIN
--    NEW.h := nextval(TG_ARGV[0]::regclass);
--    NEW.i := NOW();
    RETURN NEW;
END;
$$ language plpgsql volatile;

-- create sequence if not exists test_seq;

create or replace trigger t_test_trigger before insert or update on test for each row execute procedure
t_test_trigger_func();-- ('test_seq'); 
0,634055868,275174,702098,1,2023-07-30,3000.0000000,204560046,2023-07-30 14:16:36.750981+00,1729747992,720946,0
0,634055868,275174,702098,1,2023-07-31,3000.0000000,204560047,2023-07-30 14:16:36.750981+00,1729747993,720946,0
0,634055868,275174,702098,2,2023-07-30,3000.0000000,204560210,2023-07-30 14:16:36.750981+00,1729747749,720947,0
0,634055868,275174,702098,2,2023-07-31,3000.0000000,204560211,2023-07-30 14:16:36.750981+00,1729747750,720947,0
0,634055868,275183,702123,1,2023-07-30,3000.0000000,204560374,2023-07-30 14:16:36.750981+00,1729748984,720971,0
0,634055868,275183,702123,1,2023-07-31,3000.0000000,204560375,2023-07-30 14:16:36.750981+00,1729748985,720971,0
0,634055868,275183,702123,2,2023-07-30,3000.0000000,204560538,2023-07-30 14:16:36.750981+00,1729748741,720972,0
0,634055868,275183,702123,2,2023-07-31,3000.0000000,204560539,2023-07-30 14:16:36.750981+00,1729748742,720972,0
0,634055868,275282,702619,1,2023-07-30,3000.0000000,204560702,2023-07-30 14:16:36.750981+00,1438187999,721216,0
0,634055868,275282,702619,1,2023-07-31,3000.0000000,204560703,2023-07-30 14:16:36.750981+00,1438188000,721216,0
0,634055868,275282,702619,2,2023-07-30,3000.0000000,204560866,2023-07-30 14:16:36.750981+00,1438187850,721217,0
0,634055868,275282,702619,2,2023-07-31,3000.0000000,204560867,2023-07-30 14:16:36.750981+00,1438187851,721217,0
0,634055868,275283,702635,1,2023-07-30,3500.0000000,204561030,2023-07-30 14:16:36.750981+00,1729749718,721218,0
0,634055868,275283,702635,1,2023-07-31,3500.0000000,204561031,2023-07-30 14:16:36.750981+00,1729749719,721218,0
0,634055868,275283,702635,2,2023-07-30,3500.0000000,204561194,2023-07-30 14:16:36.750981+00,1729749475,721219,0
0,634055868,275283,702635,2,2023-07-31,3500.0000000,204561195,2023-07-30 14:16:36.750981+00,1729749476,721219,0
0,634055868,275284,702637,1,2023-07-30,3500.0000000,204561358,2023-07-30 14:16:36.750981+00,1729748498,721220,0
0,634055868,275284,702637,1,2023-07-31,3500.0000000,204561359,2023-07-30 14:16:36.750981+00,1729748499,721220,0
0,634055868,275284,702637,2,2023-07-30,3500.0000000,204561522,2023-07-30 14:16:36.750981+00,1729748255,721221,0
0,634055868,275284,702637,2,2023-07-31,3500.0000000,204561523,2023-07-30 14:16:36.750981+00,1729748256,721221,0
0,120894640,306460,830190,1,2023-07-30,1900.0000000,230145128,2023-07-30 14:16:36.750981+00,1586279154,813424,0
0,120894640,306460,830190,1,2023-07-31,1900.0000000,230145129,2023-07-30 14:16:36.750981+00,1586279621,813424,0
0,120894640,306460,830190,2,2023-07-30,1900.0000000,230145283,2023-07-30 14:16:36.750981+00,1586279125,813425,0
0,120894640,306460,830190,2,2023-07-31,1900.0000000,230145284,2023-07-30 14:16:36.750981+00,1586279591,813425,0
0,120894640,306481,830248,1,2023-07-30,1100.0000000,230145438,2023-07-30 14:16:36.750981+00,1586279212,813497,0
0,120894640,306481,830248,1,2023-07-31,1100.0000000,230145439,2023-07-30 14:16:36.750981+00,1586279681,813497,0
0,120894640,306481,830248,2,2023-07-30,1600.0000000,230145593,2023-07-30 14:16:36.750981+00,1586279183,813498,0
0,120894640,306481,830248,2,2023-07-31,1600.0000000,230145594,2023-07-30 14:16:36.750981+00,1586279651,813498,0
0,120894640,306482,830249,1,2023-07-30,2600.0000000,230145748,2023-07-30 14:16:36.750981+00,1586279328,813499,0
0,120894640,306482,830249,1,2023-07-31,2600.0000000,230145749,2023-07-30 14:16:36.750981+00,1586279801,813499,0
0,120894640,306482,830249,2,2023-07-30,2600.0000000,230145903,2023-07-30 14:16:36.750981+00,1586279299,813500,0
0,120894640,306482,830249,2,2023-07-31,2600.0000000,230145904,2023-07-30 14:16:36.750981+00,1586279771,813500,0
0,120894640,306482,830249,3,2023-07-30,3000.0000000,230146058,2023-07-30 14:16:36.750981+00,1586279270,813501,0
0,120894640,306482,830249,3,2023-07-31,3000.0000000,230146059,2023-07-30 14:16:36.750981+00,1586279741,813501,0
0,120894640,306482,830249,4,2023-07-30,3000.0000000,230146213,2023-07-30 14:16:36.750981+00,1586279241,813502,0
0,120894640,306482,830249,4,2023-07-31,3000.0000000,230146214,2023-07-30 14:16:36.750981+00,1586279711,813502,0
0,120894640,306487,830254,1,2023-07-30,1100.0000000,230146368,2023-07-30 14:16:36.750981+00,1586279415,813509,0
0,120894640,306487,830254,1,2023-07-31,1100.0000000,230146369,2023-07-30 14:16:36.750981+00,1586279891,813509,0
0,120894640,306487,830254,2,2023-07-30,1600.0000000,230146523,2023-07-30 14:16:36.750981+00,1586279386,813510,0
0,120894640,306487,830254,2,2023-07-31,1600.0000000,230146524,2023-07-30 14:16:36.750981+00,1586279861,813510,0
0,120894640,306487,830254,3,2023-07-30,1950.0000000,230146678,2023-07-30 14:16:36.750981+00,1586279357,813511,0
0,120894640,306487,830254,3,2023-07-31,1950.0000000,230146679,2023-07-30 14:16:36.750981+00,1586279831,813511,0
0,120894640,306491,830265,1,2023-07-30,1900.0000000,230146833,2023-07-30 14:16:36.750981+00,1586279473,813520,0
0,120894640,306491,830265,1,2023-07-31,1900.0000000,230146834,2023-07-30 14:16:36.750981+00,1586279951,813520,0
0,120894640,306491,830265,2,2023-07-30,1900.0000000,230146988,2023-07-30 14:16:36.750981+00,1586279444,813521,0
0,120894640,306491,830265,2,2023-07-31,1900.0000000,230146989,2023-07-30 14:16:36.750981+00,1586279921,813521,0
0,120894640,306536,830463,1,2023-07-30,2000.0000000,230147143,2023-07-30 14:16:36.750981+00,1586279589,813659,0
0,120894640,306536,830463,1,2023-07-31,2000.0000000,230147144,2023-07-30 14:16:36.750981+00,1586280071,813659,0
0,120894640,306536,830463,2,2023-07-30,2000.0000000,230147298,2023-07-30 14:16:36.750981+00,1586279560,813660,0
0,120894640,306536,830463,2,2023-07-31,2000.0000000,230147299,2023-07-30 14:16:36.750981+00,1586280041,813660,0
0,120894640,306536,830463,3,2023-07-30,2000.0000000,230147453,2023-07-30 14:16:36.750981+00,1586279531,813661,0
0,120894640,306536,830463,3,2023-07-31,2000.0000000,230147454,2023-07-30 14:16:36.750981+00,1586280011,813661,0
0,120894640,306536,830463,4,2023-07-30,2000.0000000,230147608,2023-07-30 14:16:36.750981+00,1586279502,813662,0
0,120894640,306536,830463,4,2023-07-31,2000.0000000,230147609,2023-07-30 14:16:36.750981+00,1586279981,813662,0
0,954493183,216176,461514,1,2023-07-30,5500.0000000,204562378,2023-07-30 14:16:36.750981+00,1183374851,548487,0
0,954493183,216176,461514,1,2023-07-31,5500.0000000,204562379,2023-07-30 14:16:36.750981+00,1183374855,548487,0
0,954493183,216176,461514,2,2023-07-30,5500.0000000,204562899,2023-07-30 14:16:36.750981+00,1183375018,548488,0
0,954493183,216176,461514,2,2023-07-31,5500.0000000,204562900,2023-07-30 14:16:36.750981+00,1183375021,548488,0
0,954493183,216312,462120,1,2023-07-30,4500.0000000,204563420,2023-07-30 14:16:36.750981+00,1183375244,548882,0
0,954493183,216312,462120,1,2023-07-31,4500.0000000,204563421,2023-07-30 14:16:36.750981+00,1183375245,548882,0
0,954493183,216312,462120,2,2023-07-30,4500.0000000,204563941,2023-07-30 14:16:36.750981+00,1183375142,548883,0
0,954493183,216312,462120,2,2023-07-31,4500.0000000,204563942,2023-07-30 14:16:36.750981+00,1183375145,548883,0
0,954493183,216439,462715,1,2023-07-30,3600.0000000,204564462,2023-07-30 14:16:36.750981+00,1183375449,549233,0
0,954493183,216439,462715,1,2023-07-31,3600.0000000,204564463,2023-07-30 14:16:36.750981+00,1183375451,549233,0
0,954493183,216439,462715,2,2023-07-30,3600.0000000,204564983,2023-07-30 14:16:36.750981+00,1183375337,549234,0
0,954493183,216439,462715,2,2023-07-31,3600.0000000,204564984,2023-07-30 14:16:36.750981+00,1183375341,549234,0
0,954493183,216441,462727,1,2023-07-30,3600.0000000,204565504,2023-07-30 14:16:36.750981+00,1183375594,549237,0
0,954493183,216441,462727,1,2023-07-31,3600.0000000,204565505,2023-07-30 14:16:36.750981+00,1183375598,549237,0
0,954493183,216441,462727,2,2023-07-30,3600.0000000,204566025,2023-07-30 14:16:36.750981+00,1183375729,549238,0
0,954493183,216441,462727,2,2023-07-31,3600.0000000,204566026,2023-07-30 14:16:36.750981+00,1183375732,549238,0
0,954493183,216442,462742,1,2023-07-30,3000.0000000,204566546,2023-07-30 14:16:36.750981+00,1183373103,549239,0
0,954493183,216442,462742,1,2023-07-31,3000.0000000,204566547,2023-07-30 14:16:36.750981+00,1183373107,549239,0
0,954493183,216442,462742,2,2023-07-30,3000.0000000,204567067,2023-07-30 14:16:36.750981+00,1183372939,549240,0
0,954493183,216442,462742,2,2023-07-31,3000.0000000,204567068,2023-07-30 14:16:36.750981+00,1183372942,549240,0
0,954493183,216462,462836,1,2023-07-30,810.0000000,204567588,2023-07-30 14:16:36.750981+00,1183373274,549302,0
0,954493183,216462,462836,1,2023-07-31,810.0000000,204567589,2023-07-30 14:16:36.750981+00,1183373277,549302,0
0,954493183,289707,762166,1,2023-07-30,2000.0000000,204568109,2023-07-30 14:16:36.750981+00,1183374685,763816,0
0,954493183,289707,762166,1,2023-07-31,2000.0000000,204568110,2023-07-30 14:16:36.750981+00,1183374690,763816,0
0,954493183,289707,762166,2,2023-07-30,2500.0000000,204568630,2023-07-30 14:16:36.750981+00,1183373464,763817,0
0,954493183,289707,762166,2,2023-07-31,2500.0000000,204568631,2023-07-30 14:16:36.750981+00,1183373467,763817,0
0,185647715,222578,489241,1,2023-07-30,3300.0000000,302101023,2023-07-30 14:16:36.750981+00,1401895960,567261,0
0,640725189,101742,203163,1,2023-08-01,2700.0000000,286017479,2023-07-30 14:16:36.750981+00,1403251616,257550,0
0,640725189,101742,203163,1,2023-08-02,2700.0000000,286017480,2023-07-30 14:16:36.750981+00,1404302246,257550,0
0,640725189,101742,203163,1,2023-08-03,2700.0000000,286017481,2023-07-30 14:16:36.750981+00,1405471662,257550,0
0,640725189,101742,203163,1,2023-08-04,2700.0000000,286017482,2023-07-30 14:16:36.750981+00,1406677101,257550,0
0,640725189,101742,203163,1,2023-08-05,2700.0000000,286017483,2023-07-30 14:16:36.750981+00,1407621363,257550,0
0,640725189,101742,203163,1,2023-08-06,2700.0000000,286017484,2023-07-30 14:16:36.750981+00,1408532966,257550,0
0,640725189,101742,203163,1,2023-08-07,2700.0000000,286017485,2023-07-30 14:16:36.750981+00,1409248611,257550,0
0,640725189,101742,203163,1,2023-08-08,2700.0000000,286017486,2023-07-30 14:16:36.750981+00,1409861546,257550,0
0,640725189,101742,203163,1,2023-08-09,2700.0000000,286017487,2023-07-30 14:16:36.750981+00,1410801637,257550,0
0,640725189,101742,203163,1,2023-08-10,2700.0000000,286017488,2023-07-30 14:16:36.750981+00,1411840671,257550,0
0,640725189,101742,203163,1,2023-08-11,2700.0000000,286017489,2023-07-30 14:16:36.750981+00,1412852354,257550,0
0,640725189,101742,203163,1,2023-08-12,2700.0000000,286017490,2023-07-30 14:16:36.750981+00,1413988755,257550,0
0,640725189,101742,203163,1,2023-08-13,2700.0000000,286017491,2023-07-30 14:16:36.750981+00,1414738233,257550,0
0,640725189,101742,203163,1,2023-08-14,2700.0000000,286017492,2023-07-30 14:16:36.750981+00,1415333532,257550,0
0,640725189,101742,203163,1,2023-08-15,2700.0000000,286017493,2023-07-30 14:16:36.750981+00,1415991454,257550,0
0,640725189,101742,203163,1,2023-08-16,2700.0000000,286017494,2023-07-30 14:16:36.750981+00,1417133229,257550,0
0,640725189,101742,203163,1,2023-08-17,2700.0000000,286017495,2023-07-30 14:16:36.750981+00,1417827139,257550,0
0,640725189,101742,203163,1,2023-08-18,2700.0000000,286017496,2023-07-30 14:16:36.750981+00,1418815182,257550,0
0,640725189,101742,203163,1,2023-08-19,2700.0000000,286017497,2023-07-30 14:16:36.750981+00,1420121236,257550,0
0,640725189,101742,203163,1,2023-08-20,2700.0000000,286017498,2023-07-30 14:16:36.750981+00,1421302951,257550,0
0,640725189,101742,203163,1,2023-08-21,2700.0000000,286017499,2023-07-30 14:16:36.750981+00,1421893783,257550,0
0,640725189,101742,203163,1,2023-08-22,2700.0000000,286017500,2023-07-30 14:16:36.750981+00,1422574736,257550,0
0,640725189,101742,203163,1,2023-08-23,2700.0000000,286017501,2023-07-30 14:16:36.750981+00,1423641864,257550,0
0,640725189,101742,203163,1,2023-08-24,2700.0000000,286017502,2023-07-30 14:16:36.750981+00,1425001752,257550,0
0,640725189,101742,203163,1,2023-08-25,2700.0000000,286017503,2023-07-30 14:16:36.750981+00,1426189082,257550,0
0,640725189,101742,203163,1,2023-08-26,2700.0000000,286017504,2023-07-30 14:16:36.750981+00,1426909542,257550,0
0,640725189,101742,203163,1,2023-08-27,2700.0000000,286017505,2023-07-30 14:16:36.750981+00,1427745853,257550,0
0,640725189,101742,203163,1,2023-08-28,2700.0000000,286017506,2023-07-30 14:16:36.750981+00,1428251161,257550,0
0,640725189,101742,203163,1,2023-08-29,2700.0000000,286017507,2023-07-30 14:16:36.750981+00,1429371327,257550,0
0,640725189,101742,203163,1,2023-08-30,2700.0000000,286017508,2023-07-30 14:16:36.750981+00,1430351705,257550,0
0,640725189,101742,203163,1,2023-08-31,2700.0000000,286017509,2023-07-30 14:16:36.750981+00,1431233924,257550,0
0,640725189,101742,549736,1,2023-08-01,2350.0000000,286017634,2023-07-30 14:16:36.750981+00,1403251631,257550,0
0,640725189,101742,549736,1,2023-08-02,2350.0000000,286017635,2023-07-30 14:16:36.750981+00,1404302253,257550,0
0,640725189,101742,549736,1,2023-08-03,2350.0000000,286017636,2023-07-30 14:16:36.750981+00,1405471669,257550,0
0,640725189,101742,549736,1,2023-08-04,2350.0000000,286017637,2023-07-30 14:16:36.750981+00,1406677108,257550,0
0,640725189,101742,549736,1,2023-08-05,2350.0000000,286017638,2023-07-30 14:16:36.750981+00,1407621370,257550,0
0,640725189,101742,549736,1,2023-08-06,2350.0000000,286017639,2023-07-30 14:16:36.750981+00,1408532973,257550,0
0,640725189,101742,549736,1,2023-08-07,2350.0000000,286017640,2023-07-30 14:16:36.750981+00,1409248618,257550,0
0,640725189,101742,549736,1,2023-08-08,2350.0000000,286017641,2023-07-30 14:16:36.750981+00,1409861553,257550,0
0,640725189,101742,549736,1,2023-08-09,2350.0000000,286017642,2023-07-30 14:16:36.750981+00,1410801644,257550,0
0,640725189,101742,549736,1,2023-08-10,2350.0000000,286017643,2023-07-30 14:16:36.750981+00,1411840678,257550,0
0,640725189,101742,549736,1,2023-08-11,2350.0000000,286017644,2023-07-30 14:16:36.750981+00,1412852362,257550,0
0,640725189,101742,549736,1,2023-08-12,2350.0000000,286017645,2023-07-30 14:16:36.750981+00,1413988762,257550,0
0,640725189,101742,549736,1,2023-08-13,2350.0000000,286017646,2023-07-30 14:16:36.750981+00,1414738251,257550,0
0,640725189,101742,549736,1,2023-08-14,2350.0000000,286017647,2023-07-30 14:16:36.750981+00,1415333539,257550,0
0,640725189,101742,549736,1,2023-08-15,2350.0000000,286017648,2023-07-30 14:16:36.750981+00,1415991461,257550,0
0,640725189,101742,549736,1,2023-08-16,2350.0000000,286017649,2023-07-30 14:16:36.750981+00,1417133255,257550,0
0,640725189,101742,549736,1,2023-08-17,2350.0000000,286017650,2023-07-30 14:16:36.750981+00,1417827146,257550,0
0,640725189,101742,549736,1,2023-08-18,2350.0000000,286017651,2023-07-30 14:16:36.750981+00,1418815189,257550,0
0,640725189,101742,549736,1,2023-08-19,2350.0000000,286017652,2023-07-30 14:16:36.750981+00,1420121243,257550,0
0,640725189,101742,549736,1,2023-08-20,2350.0000000,286017653,2023-07-30 14:16:36.750981+00,1421302958,257550,0
0,640725189,101742,549736,1,2023-08-21,2350.0000000,286017654,2023-07-30 14:16:36.750981+00,1421893790,257550,0
0,640725189,101742,549736,1,2023-08-22,2350.0000000,286017655,2023-07-30 14:16:36.750981+00,1422574743,257550,0
0,640725189,101742,549736,1,2023-08-23,2350.0000000,286017656,2023-07-30 14:16:36.750981+00,1423641871,257550,0
0,640725189,101742,549736,1,2023-08-24,2350.0000000,286017657,2023-07-30 14:16:36.750981+00,1425001759,257550,0
0,640725189,101742,549736,1,2023-08-25,2350.0000000,286017658,2023-07-30 14:16:36.750981+00,1426189089,257550,0
0,640725189,101742,549736,1,2023-08-26,2350.0000000,286017659,2023-07-30 14:16:36.750981+00,1426909549,257550,0
0,640725189,101742,549736,1,2023-08-27,2350.0000000,286017660,2023-07-30 14:16:36.750981+00,1427745860,257550,0
0,640725189,101742,549736,1,2023-08-28,2350.0000000,286017661,2023-07-30 14:16:36.750981+00,1428251168,257550,0
0,640725189,101742,549736,1,2023-08-29,2350.0000000,286017662,2023-07-30 14:16:36.750981+00,1429371334,257550,0
0,640725189,101742,549736,1,2023-08-30,2350.0000000,286017663,2023-07-30 14:16:36.750981+00,1430351712,257550,0
0,640725189,101742,549736,1,2023-08-31,2350.0000000,286017664,2023-07-30 14:16:36.750981+00,1431233931,257550,0
0,640725189,101743,203164,1,2023-08-01,3350.0000000,286017789,2023-07-30 14:16:36.750981+00,1403251618,257551,0
0,640725189,101743,203164,1,2023-08-02,3350.0000000,286017790,2023-07-30 14:16:36.750981+00,1404302247,257551,0
0,640725189,101743,203164,1,2023-08-03,3350.0000000,286017791,2023-07-30 14:16:36.750981+00,1405471663,257551,0
0,640725189,101743,203164,1,2023-08-04,3350.0000000,286017792,2023-07-30 14:16:36.750981+00,1406677102,257551,0
0,640725189,101743,203164,1,2023-08-05,3350.0000000,286017793,2023-07-30 14:16:36.750981+00,1407621364,257551,0
0,640725189,101743,203164,1,2023-08-06,3350.0000000,286017794,2023-07-30 14:16:36.750981+00,1408532967,257551,0
0,640725189,101743,203164,1,2023-08-07,3350.0000000,286017795,2023-07-30 14:16:36.750981+00,1409248612,257551,0
0,640725189,101743,203164,1,2023-08-08,3350.0000000,286017796,2023-07-30 14:16:36.750981+00,1409861547,257551,0
0,640725189,101743,203164,1,2023-08-09,3350.0000000,286017797,2023-07-30 14:16:36.750981+00,1410801638,257551,0
0,640725189,101743,203164,1,2023-08-10,3350.0000000,286017798,2023-07-30 14:16:36.750981+00,1411840672,257551,0
0,640725189,101743,203164,1,2023-08-11,3350.0000000,286017799,2023-07-30 14:16:36.750981+00,1412852355,257551,0
0,640725189,101743,203164,1,2023-08-12,3350.0000000,286017800,2023-07-30 14:16:36.750981+00,1413988756,257551,0
0,640725189,101743,203164,1,2023-08-13,3350.0000000,286017801,2023-07-30 14:16:36.750981+00,1414738235,257551,0
0,640725189,101743,203164,1,2023-08-14,3350.0000000,286017802,2023-07-30 14:16:36.750981+00,1415333533,257551,0
0,640725189,101743,203164,1,2023-08-15,3350.0000000,286017803,2023-07-30 14:16:36.750981+00,1415991455,257551,0
0,640725189,101743,203164,1,2023-08-16,3350.0000000,286017804,2023-07-30 14:16:36.750981+00,1417133235,257551,0
0,640725189,101743,203164,1,2023-08-17,3350.0000000,286017805,2023-07-30 14:16:36.750981+00,1417827140,257551,0
0,640725189,101743,203164,1,2023-08-18,3350.0000000,286017806,2023-07-30 14:16:36.750981+00,1418815183,257551,0
0,640725189,101743,203164,1,2023-08-19,3350.0000000,286017807,2023-07-30 14:16:36.750981+00,1420121237,257551,0
0,640725189,101743,203164,1,2023-08-20,3350.0000000,286017808,2023-07-30 14:16:36.750981+00,1421302952,257551,0
0,640725189,101743,203164,1,2023-08-21,3350.0000000,286017809,2023-07-30 14:16:36.750981+00,1421893784,257551,0
0,640725189,101743,203164,1,2023-08-22,3350.0000000,286017810,2023-07-30 14:16:36.750981+00,1422574737,257551,0
0,640725189,101743,203164,1,2023-08-23,3350.0000000,286017811,2023-07-30 14:16:36.750981+00,1423641865,257551,0
0,640725189,101743,203164,1,2023-08-24,3350.0000000,286017812,2023-07-30 14:16:36.750981+00,1425001753,257551,0
0,640725189,101743,203164,1,2023-08-25,3350.0000000,286017813,2023-07-30 14:16:36.750981+00,1426189083,257551,0
0,640725189,101743,203164,1,2023-08-26,3350.0000000,286017814,2023-07-30 14:16:36.750981+00,1426909543,257551,0
0,640725189,101743,203164,1,2023-08-27,3350.0000000,286017815,2023-07-30 14:16:36.750981+00,1427745854,257551,0
0,640725189,101743,203164,1,2023-08-28,3350.0000000,286017816,2023-07-30 14:16:36.750981+00,1428251162,257551,0
0,640725189,101743,203164,1,2023-08-29,3350.0000000,286017817,2023-07-30 14:16:36.750981+00,1429371328,257551,0
0,640725189,101743,203164,1,2023-08-30,3350.0000000,286017818,2023-07-30 14:16:36.750981+00,1430351706,257551,0
0,640725189,101743,203164,1,2023-08-31,3350.0000000,286017819,2023-07-30 14:16:36.750981+00,1431233925,257551,0
0,640725189,101743,549737,1,2023-08-01,3000.0000000,286017944,2023-07-30 14:16:36.750981+00,1403251632,257551,0
0,640725189,101743,549737,1,2023-08-02,3000.0000000,286017945,2023-07-30 14:16:36.750981+00,1404302254,257551,0
0,640725189,101743,549737,1,2023-08-03,3000.0000000,286017946,2023-07-30 14:16:36.750981+00,1405471670,257551,0
0,640725189,101743,549737,1,2023-08-04,3000.0000000,286017947,2023-07-30 14:16:36.750981+00,1406677109,257551,0
0,640725189,101743,549737,1,2023-08-05,3000.0000000,286017948,2023-07-30 14:16:36.750981+00,1407621371,257551,0
0,640725189,101743,549737,1,2023-08-06,3000.0000000,286017949,2023-07-30 14:16:36.750981+00,1408532974,257551,0
0,640725189,101743,549737,1,2023-08-07,3000.0000000,286017950,2023-07-30 14:16:36.750981+00,1409248619,257551,0
0,640725189,101743,549737,1,2023-08-08,3000.0000000,286017951,2023-07-30 14:16:36.750981+00,1409861554,257551,0
0,640725189,101743,549737,1,2023-08-09,3000.0000000,286017952,2023-07-30 14:16:36.750981+00,1410801645,257551,0
0,640725189,101743,549737,1,2023-08-10,3000.0000000,286017953,2023-07-30 14:16:36.750981+00,1411840679,257551,0
0,640725189,101743,549737,1,2023-08-11,3000.0000000,286017954,2023-07-30 14:16:36.750981+00,1412852363,257551,0
0,640725189,101743,549737,1,2023-08-12,3000.0000000,286017955,2023-07-30 14:16:36.750981+00,1413988763,257551,0
0,640725189,101743,549737,1,2023-08-13,3000.0000000,286017956,2023-07-30 14:16:36.750981+00,1414738253,257551,0
0,640725189,101743,549737,1,2023-08-14,3000.0000000,286017957,2023-07-30 14:16:36.750981+00,1415333540,257551,0
0,640725189,101743,549737,1,2023-08-15,3000.0000000,286017958,2023-07-30 14:16:36.750981+00,1415991462,257551,0
0,640725189,101743,549737,1,2023-08-16,3000.0000000,286017959,2023-07-30 14:16:36.750981+00,1417133258,257551,0
0,640725189,101743,549737,1,2023-08-17,3000.0000000,286017960,2023-07-30 14:16:36.750981+00,1417827147,257551,0
0,640725189,101743,549737,1,2023-08-18,3000.0000000,286017961,2023-07-30 14:16:36.750981+00,1418815190,257551,0
0,640725189,101743,549737,1,2023-08-19,3000.0000000,286017962,2023-07-30 14:16:36.750981+00,1420121244,257551,0
0,640725189,101743,549737,1,2023-08-20,3000.0000000,286017963,2023-07-30 14:16:36.750981+00,1421302959,257551,0
0,640725189,101743,549737,1,2023-08-21,3000.0000000,286017964,2023-07-30 14:16:36.750981+00,1421893791,257551,0
0,640725189,101743,549737,1,2023-08-22,3000.0000000,286017965,2023-07-30 14:16:36.750981+00,1422574744,257551,0
0,640725189,101743,549737,1,2023-08-23,3000.0000000,286017966,2023-07-30 14:16:36.750981+00,1423641872,257551,0
0,640725189,101743,549737,1,2023-08-24,3000.0000000,286017967,2023-07-30 14:16:36.750981+00,1425001760,257551,0
0,640725189,101743,549737,1,2023-08-25,3000.0000000,286017968,2023-07-30 14:16:36.750981+00,1426189090,257551,0
0,640725189,101743,549737,1,2023-08-26,3000.0000000,286017969,2023-07-30 14:16:36.750981+00,1426909550,257551,0
0,640725189,101743,549737,1,2023-08-27,3000.0000000,286017970,2023-07-30 14:16:36.750981+00,1427745861,257551,0
0,640725189,101743,549737,1,2023-08-28,3000.0000000,286017971,2023-07-30 14:16:36.750981+00,1428251169,257551,0
0,640725189,101743,549737,1,2023-08-29,3000.0000000,286017972,2023-07-30 14:16:36.750981+00,1429371335,257551,0
0,640725189,101743,549737,1,2023-08-30,3000.0000000,286017973,2023-07-30 14:16:36.750981+00,1430351713,257551,0
0,640725189,101743,549737,1,2023-08-31,3000.0000000,286017974,2023-07-30 14:16:36.750981+00,1431233932,257551,0
0,640725189,101744,203165,2,2023-08-01,3950.0000000,286018099,2023-07-30 14:16:36.750981+00,1403251621,257553,0
0,640725189,101744,203165,2,2023-08-02,3950.0000000,286018100,2023-07-30 14:16:36.750981+00,1404302248,257553,0
0,640725189,101744,203165,2,2023-08-03,3950.0000000,286018101,2023-07-30 14:16:36.750981+00,1405471664,257553,0
0,640725189,101744,203165,2,2023-08-04,3950.0000000,286018102,2023-07-30 14:16:36.750981+00,1406677103,257553,0
0,640725189,101744,203165,2,2023-08-05,3950.0000000,286018103,2023-07-30 14:16:36.750981+00,1407621365,257553,0
0,640725189,101744,203165,2,2023-08-06,3950.0000000,286018104,2023-07-30 14:16:36.750981+00,1408532968,257553,0
0,640725189,101744,203165,2,2023-08-07,3950.0000000,286018105,2023-07-30 14:16:36.750981+00,1409248613,257553,0
0,640725189,101744,203165,2,2023-08-08,3950.0000000,286018106,2023-07-30 14:16:36.750981+00,1409861548,257553,0
0,640725189,101744,203165,2,2023-08-09,3950.0000000,286018107,2023-07-30 14:16:36.750981+00,1410801639,257553,0
0,640725189,101744,203165,2,2023-08-10,3950.0000000,286018108,2023-07-30 14:16:36.750981+00,1411840673,257553,0
0,640725189,101744,203165,2,2023-08-11,3950.0000000,286018109,2023-07-30 14:16:36.750981+00,1412852356,257553,0
0,640725189,101744,203165,2,2023-08-12,3950.0000000,286018110,2023-07-30 14:16:36.750981+00,1413988757,257553,0
0,640725189,101744,203165,2,2023-08-13,3950.0000000,286018111,2023-07-30 14:16:36.750981+00,1414738238,257553,0
0,640725189,101744,203165,2,2023-08-14,3950.0000000,286018112,2023-07-30 14:16:36.750981+00,1415333534,257553,0
0,640725189,101744,203165,2,2023-08-15,3950.0000000,286018113,2023-07-30 14:16:36.750981+00,1415991456,257553,0
0,640725189,101744,203165,2,2023-08-16,3950.0000000,286018114,2023-07-30 14:16:36.750981+00,1417133238,257553,0
0,640725189,101744,203165,2,2023-08-17,3950.0000000,286018115,2023-07-30 14:16:36.750981+00,1417827141,257553,0
0,640725189,101744,203165,2,2023-08-18,3950.0000000,286018116,2023-07-30 14:16:36.750981+00,1418815184,257553,0
0,640725189,101744,203165,2,2023-08-19,3950.0000000,286018117,2023-07-30 14:16:36.750981+00,1420121238,257553,0
0,640725189,101744,203165,2,2023-08-20,3950.0000000,286018118,2023-07-30 14:16:36.750981+00,1421302953,257553,0
0,640725189,101744,203165,2,2023-08-21,3950.0000000,286018119,2023-07-30 14:16:36.750981+00,1421893785,257553,0
0,640725189,101744,203165,2,2023-08-22,3950.0000000,286018120,2023-07-30 14:16:36.750981+00,1422574738,257553,0
0,640725189,101744,203165,2,2023-08-23,3950.0000000,286018121,2023-07-30 14:16:36.750981+00,1423641866,257553,0
0,640725189,101744,203165,2,2023-08-24,3950.0000000,286018122,2023-07-30 14:16:36.750981+00,1425001754,257553,0
0,640725189,101744,203165,2,2023-08-25,3950.0000000,286018123,2023-07-30 14:16:36.750981+00,1426189084,257553,0
0,640725189,101744,203165,2,2023-08-26,3950.0000000,286018124,2023-07-30 14:16:36.750981+00,1426909544,257553,0
0,640725189,101744,203165,2,2023-08-27,3950.0000000,286018125,2023-07-30 14:16:36.750981+00,1427745855,257553,0
0,640725189,101744,203165,2,2023-08-28,3950.0000000,286018126,2023-07-30 14:16:36.750981+00,1428251163,257553,0
0,640725189,101744,203165,2,2023-08-29,3950.0000000,286018127,2023-07-30 14:16:36.750981+00,1429371329,257553,0
0,640725189,101744,203165,2,2023-08-30,3950.0000000,286018128,2023-07-30 14:16:36.750981+00,1430351707,257553,0
0,640725189,101744,203165,2,2023-08-31,3950.0000000,286018129,2023-07-30 14:16:36.750981+00,1431233926,257553,0
0,640725189,101744,549738,2,2023-08-01,3250.0000000,286018254,2023-07-30 14:16:36.750981+00,1403251633,257553,0
0,640725189,101744,549738,2,2023-08-02,3250.0000000,286018255,2023-07-30 14:16:36.750981+00,1404302255,257553,0
0,640725189,101744,549738,2,2023-08-03,3250.0000000,286018256,2023-07-30 14:16:36.750981+00,1405471671,257553,0
0,640725189,101744,549738,2,2023-08-04,3250.0000000,286018257,2023-07-30 14:16:36.750981+00,1406677110,257553,0
0,640725189,101744,549738,2,2023-08-05,3250.0000000,286018258,2023-07-30 14:16:36.750981+00,1407621372,257553,0
0,640725189,101744,549738,2,2023-08-06,3250.0000000,286018259,2023-07-30 14:16:36.750981+00,1408532975,257553,0
0,640725189,101744,549738,2,2023-08-07,3250.0000000,286018260,2023-07-30 14:16:36.750981+00,1409248620,257553,0
0,640725189,101744,549738,2,2023-08-08,3250.0000000,286018261,2023-07-30 14:16:36.750981+00,1409861555,257553,0
0,640725189,101744,549738,2,2023-08-09,3250.0000000,286018262,2023-07-30 14:16:36.750981+00,1410801646,257553,0
0,640725189,101744,549738,2,2023-08-10,3250.0000000,286018263,2023-07-30 14:16:36.750981+00,1411840680,257553,0
0,640725189,101744,549738,2,2023-08-11,3250.0000000,286018264,2023-07-30 14:16:36.750981+00,1412852365,257553,0
0,640725189,101744,549738,2,2023-08-12,3250.0000000,286018265,2023-07-30 14:16:36.750981+00,1413988764,257553,0
0,640725189,101744,549738,2,2023-08-13,3250.0000000,286018266,2023-07-30 14:16:36.750981+00,1414738255,257553,0
0,640725189,101744,549738,2,2023-08-14,3250.0000000,286018267,2023-07-30 14:16:36.750981+00,1415333541,257553,0
0,640725189,101744,549738,2,2023-08-15,3250.0000000,286018268,2023-07-30 14:16:36.750981+00,1415991463,257553,0
0,640725189,101744,549738,2,2023-08-16,3250.0000000,286018269,2023-07-30 14:16:36.750981+00,1417133261,257553,0
0,640725189,101744,549738,2,2023-08-17,3250.0000000,286018270,2023-07-30 14:16:36.750981+00,1417827148,257553,0
0,640725189,101744,549738,2,2023-08-18,3250.0000000,286018271,2023-07-30 14:16:36.750981+00,1418815191,257553,0
0,640725189,101744,549738,2,2023-08-19,3250.0000000,286018272,2023-07-30 14:16:36.750981+00,1420121245,257553,0
0,640725189,101744,549738,2,2023-08-20,3250.0000000,286018273,2023-07-30 14:16:36.750981+00,1421302960,257553,0
0,640725189,101744,549738,2,2023-08-21,3250.0000000,286018274,2023-07-30 14:16:36.750981+00,1421893792,257553,0
0,640725189,101744,549738,2,2023-08-22,3250.0000000,286018275,2023-07-30 14:16:36.750981+00,1422574745,257553,0
0,640725189,101744,549738,2,2023-08-23,3250.0000000,286018276,2023-07-30 14:16:36.750981+00,1423641873,257553,0
0,640725189,101744,549738,2,2023-08-24,3250.0000000,286018277,2023-07-30 14:16:36.750981+00,1425001761,257553,0
0,640725189,101744,549738,2,2023-08-25,3250.0000000,286018278,2023-07-30 14:16:36.750981+00,1426189091,257553,0
0,640725189,101744,549738,2,2023-08-26,3250.0000000,286018279,2023-07-30 14:16:36.750981+00,1426909551,257553,0
0,640725189,101744,549738,2,2023-08-27,3250.0000000,286018280,2023-07-30 14:16:36.750981+00,1427745862,257553,0
0,640725189,101744,549738,2,2023-08-28,3250.0000000,286018281,2023-07-30 14:16:36.750981+00,1428251170,257553,0
0,640725189,101744,549738,2,2023-08-29,3250.0000000,286018282,2023-07-30 14:16:36.750981+00,1429371336,257553,0
0,640725189,101744,549738,2,2023-08-30,3250.0000000,286018283,2023-07-30 14:16:36.750981+00,1430351714,257553,0
0,640725189,101744,549738,2,2023-08-31,3250.0000000,286018284,2023-07-30 14:16:36.750981+00,1431233933,257553,0
0,640725189,101745,203166,1,2023-08-01,3650.0000000,286018409,2023-07-30 14:16:36.750981+00,1403251623,257557,0
0,640725189,101745,203166,1,2023-08-02,3650.0000000,286018410,2023-07-30 14:16:36.750981+00,1404302249,257557,0
0,640725189,101745,203166,1,2023-08-03,3650.0000000,286018411,2023-07-30 14:16:36.750981+00,1405471665,257557,0
0,640725189,101745,203166,1,2023-08-04,3650.0000000,286018412,2023-07-30 14:16:36.750981+00,1406677104,257557,0
0,640725189,101745,203166,1,2023-08-05,3650.0000000,286018413,2023-07-30 14:16:36.750981+00,1407621366,257557,0
0,640725189,101745,203166,1,2023-08-06,3650.0000000,286018414,2023-07-30 14:16:36.750981+00,1408532969,257557,0
0,640725189,101745,203166,1,2023-08-07,3650.0000000,286018415,2023-07-30 14:16:36.750981+00,1409248614,257557,0
0,640725189,101745,203166,1,2023-08-08,3650.0000000,286018416,2023-07-30 14:16:36.750981+00,1409861549,257557,0
0,640725189,101745,203166,1,2023-08-09,3650.0000000,286018417,2023-07-30 14:16:36.750981+00,1410801640,257557,0
0,640725189,101745,203166,1,2023-08-10,3650.0000000,286018418,2023-07-30 14:16:36.750981+00,1411840674,257557,0
0,640725189,101745,203166,1,2023-08-11,3650.0000000,286018419,2023-07-30 14:16:36.750981+00,1412852357,257557,0
0,640725189,101745,203166,1,2023-08-12,3650.0000000,286018420,2023-07-30 14:16:36.750981+00,1413988758,257557,0
0,640725189,101745,203166,1,2023-08-13,3650.0000000,286018421,2023-07-30 14:16:36.750981+00,1414738241,257557,0
0,640725189,101745,203166,1,2023-08-14,3650.0000000,286018422,2023-07-30 14:16:36.750981+00,1415333535,257557,0
0,640725189,101745,203166,1,2023-08-15,3650.0000000,286018423,2023-07-30 14:16:36.750981+00,1415991457,257557,0
0,640725189,101745,203166,1,2023-08-16,3650.0000000,286018424,2023-07-30 14:16:36.750981+00,1417133242,257557,0
0,640725189,101745,203166,1,2023-08-17,3650.0000000,286018425,2023-07-30 14:16:36.750981+00,1417827142,257557,0
0,640725189,101745,203166,1,2023-08-18,3650.0000000,286018426,2023-07-30 14:16:36.750981+00,1418815185,257557,0
0,640725189,101745,203166,1,2023-08-19,3650.0000000,286018427,2023-07-30 14:16:36.750981+00,1420121239,257557,0
0,640725189,101745,203166,1,2023-08-20,3650.0000000,286018428,2023-07-30 14:16:36.750981+00,1421302954,257557,0
0,640725189,101745,203166,1,2023-08-21,3650.0000000,286018429,2023-07-30 14:16:36.750981+00,1421893786,257557,0
0,640725189,101745,203166,1,2023-08-22,3650.0000000,286018430,2023-07-30 14:16:36.750981+00,1422574739,257557,0
0,640725189,101745,203166,1,2023-08-23,3650.0000000,286018431,2023-07-30 14:16:36.750981+00,1423641867,257557,0
0,640725189,101745,203166,1,2023-08-24,3650.0000000,286018432,2023-07-30 14:16:36.750981+00,1425001755,257557,0
0,640725189,101745,203166,1,2023-08-25,3650.0000000,286018433,2023-07-30 14:16:36.750981+00,1426189085,257557,0
0,640725189,101745,203166,1,2023-08-26,3650.0000000,286018434,2023-07-30 14:16:36.750981+00,1426909545,257557,0
0,640725189,101745,203166,1,2023-08-27,3650.0000000,286018435,2023-07-30 14:16:36.750981+00,1427745856,257557,0
0,640725189,101745,203166,1,2023-08-28,3650.0000000,286018436,2023-07-30 14:16:36.750981+00,1428251164,257557,0
0,640725189,101745,203166,1,2023-08-29,3650.0000000,286018437,2023-07-30 14:16:36.750981+00,1429371330,257557,0
0,640725189,101745,203166,1,2023-08-30,3650.0000000,286018438,2023-07-30 14:16:36.750981+00,1430351708,257557,0
0,640725189,101745,203166,1,2023-08-31,3650.0000000,286018439,2023-07-30 14:16:36.750981+00,1431233927,257557,0
0,640725189,101745,203166,2,2023-08-01,4400.0000000,286018564,2023-07-30 14:16:36.750981+00,1403251627,257558,0
0,640725189,101745,203166,2,2023-08-02,4400.0000000,286018565,2023-07-30 14:16:36.750981+00,1404302250,257558,0
0,640725189,101745,203166,2,2023-08-03,4400.0000000,286018566,2023-07-30 14:16:36.750981+00,1405471666,257558,0
0,640725189,101745,203166,2,2023-08-04,4400.0000000,286018567,2023-07-30 14:16:36.750981+00,1406677105,257558,0
0,640725189,101745,203166,2,2023-08-05,4400.0000000,286018568,2023-07-30 14:16:36.750981+00,1407621367,257558,0
0,640725189,101745,203166,2,2023-08-06,4400.0000000,286018569,2023-07-30 14:16:36.750981+00,1408532970,257558,0
0,640725189,101745,203166,2,2023-08-07,4400.0000000,286018570,2023-07-30 14:16:36.750981+00,1409248615,257558,0
0,640725189,101745,203166,2,2023-08-08,4400.0000000,286018571,2023-07-30 14:16:36.750981+00,1409861550,257558,0
0,640725189,101745,203166,2,2023-08-09,4400.0000000,286018572,2023-07-30 14:16:36.750981+00,1410801641,257558,0
0,640725189,101745,203166,2,2023-08-10,4400.0000000,286018573,2023-07-30 14:16:36.750981+00,1411840675,257558,0
0,640725189,101745,203166,2,2023-08-11,4400.0000000,286018574,2023-07-30 14:16:36.750981+00,1412852358,257558,0
0,640725189,101745,203166,2,2023-08-12,4400.0000000,286018575,2023-07-30 14:16:36.750981+00,1413988759,257558,0
0,640725189,101745,203166,2,2023-08-13,4400.0000000,286018576,2023-07-30 14:16:36.750981+00,1414738244,257558,0
0,640725189,101745,203166,2,2023-08-14,4400.0000000,286018577,2023-07-30 14:16:36.750981+00,1415333536,257558,0
0,640725189,101745,203166,2,2023-08-15,4400.0000000,286018578,2023-07-30 14:16:36.750981+00,1415991458,257558,0
0,640725189,101745,203166,2,2023-08-16,4400.0000000,286018579,2023-07-30 14:16:36.750981+00,1417133246,257558,0
0,640725189,101745,203166,2,2023-08-17,4400.0000000,286018580,2023-07-30 14:16:36.750981+00,1417827143,257558,0
0,640725189,101745,203166,2,2023-08-18,4400.0000000,286018581,2023-07-30 14:16:36.750981+00,1418815186,257558,0
0,640725189,101745,203166,2,2023-08-19,4400.0000000,286018582,2023-07-30 14:16:36.750981+00,1420121240,257558,0
0,640725189,101745,203166,2,2023-08-20,4400.0000000,286018583,2023-07-30 14:16:36.750981+00,1421302955,257558,0
0,640725189,101745,203166,2,2023-08-21,4400.0000000,286018584,2023-07-30 14:16:36.750981+00,1421893787,257558,0
0,640725189,101745,203166,2,2023-08-22,4400.0000000,286018585,2023-07-30 14:16:36.750981+00,1422574740,257558,0
0,640725189,101745,203166,2,2023-08-23,4400.0000000,286018586,2023-07-30 14:16:36.750981+00,1423641868,257558,0
0,640725189,101745,203166,2,2023-08-24,4400.0000000,286018587,2023-07-30 14:16:36.750981+00,1425001756,257558,0
0,640725189,101745,203166,2,2023-08-25,4400.0000000,286018588,2023-07-30 14:16:36.750981+00,1426189086,257558,0
0,640725189,101745,203166,2,2023-08-26,4400.0000000,286018589,2023-07-30 14:16:36.750981+00,1426909546,257558,0
0,640725189,101745,203166,2,2023-08-27,4400.0000000,286018590,2023-07-30 14:16:36.750981+00,1427745857,257558,0
0,640725189,101745,203166,2,2023-08-28,4400.0000000,286018591,2023-07-30 14:16:36.750981+00,1428251165,257558,0
0,640725189,101745,203166,2,2023-08-29,4400.0000000,286018592,2023-07-30 14:16:36.750981+00,1429371331,257558,0
0,640725189,101745,203166,2,2023-08-30,4400.0000000,286018593,2023-07-30 14:16:36.750981+00,1430351709,257558,0
0,640725189,101745,203166,2,2023-08-31,4400.0000000,286018594,2023-07-30 14:16:36.750981+00,1431233928,257558,0
0,640725189,101745,549739,1,2023-08-01,3300.0000000,286018719,2023-07-30 14:16:36.750981+00,1403251635,257557,0
0,640725189,101745,549739,1,2023-08-02,3300.0000000,286018720,2023-07-30 14:16:36.750981+00,1404302257,257557,0
0,640725189,101745,549739,1,2023-08-03,3300.0000000,286018721,2023-07-30 14:16:36.750981+00,1405471673,257557,0
0,640725189,101745,549739,1,2023-08-04,3300.0000000,286018722,2023-07-30 14:16:36.750981+00,1406677112,257557,0
0,640725189,101745,549739,1,2023-08-05,3300.0000000,286018723,2023-07-30 14:16:36.750981+00,1407621374,257557,0
0,640725189,101745,549739,1,2023-08-06,3300.0000000,286018724,2023-07-30 14:16:36.750981+00,1408532977,257557,0
0,640725189,101745,549739,1,2023-08-07,3300.0000000,286018725,2023-07-30 14:16:36.750981+00,1409248622,257557,0
0,640725189,101745,549739,1,2023-08-08,3300.0000000,286018726,2023-07-30 14:16:36.750981+00,1409861557,257557,0
0,640725189,101745,549739,1,2023-08-09,3300.0000000,286018727,2023-07-30 14:16:36.750981+00,1410801648,257557,0
0,640725189,101745,549739,1,2023-08-10,3300.0000000,286018728,2023-07-30 14:16:36.750981+00,1411840682,257557,0
0,640725189,101745,549739,1,2023-08-11,3300.0000000,286018729,2023-07-30 14:16:36.750981+00,1412852368,257557,0
0,640725189,101745,549739,1,2023-08-12,3300.0000000,286018730,2023-07-30 14:16:36.750981+00,1413988766,257557,0
0,640725189,101745,549739,1,2023-08-13,3300.0000000,286018731,2023-07-30 14:16:36.750981+00,1414738261,257557,0
0,640725189,101745,549739,1,2023-08-14,3300.0000000,286018732,2023-07-30 14:16:36.750981+00,1415333543,257557,0
0,640725189,101745,549739,1,2023-08-15,3300.0000000,286018733,2023-07-30 14:16:36.750981+00,1415991465,257557,0
0,640725189,101745,549739,1,2023-08-16,3300.0000000,286018734,2023-07-30 14:16:36.750981+00,1417133267,257557,0
0,640725189,101745,549739,1,2023-08-17,3300.0000000,286018735,2023-07-30 14:16:36.750981+00,1417827150,257557,0
0,640725189,101745,549739,1,2023-08-18,3300.0000000,286018736,2023-07-30 14:16:36.750981+00,1418815193,257557,0
0,640725189,101745,549739,1,2023-08-19,3300.0000000,286018737,2023-07-30 14:16:36.750981+00,1420121247,257557,0
0,640725189,101745,549739,1,2023-08-20,3300.0000000,286018738,2023-07-30 14:16:36.750981+00,1421302962,257557,0
0,640725189,101745,549739,1,2023-08-21,3300.0000000,286018739,2023-07-30 14:16:36.750981+00,1421893794,257557,0
0,640725189,101745,549739,1,2023-08-22,3300.0000000,286018740,2023-07-30 14:16:36.750981+00,1422574747,257557,0
0,640725189,101745,549739,1,2023-08-23,3300.0000000,286018741,2023-07-30 14:16:36.750981+00,1423641875,257557,0
0,640725189,101745,549739,1,2023-08-24,3300.0000000,286018742,2023-07-30 14:16:36.750981+00,1425001763,257557,0
0,640725189,101745,549739,1,2023-08-25,3300.0000000,286018743,2023-07-30 14:16:36.750981+00,1426189093,257557,0
0,640725189,101745,549739,1,2023-08-26,3300.0000000,286018744,2023-07-30 14:16:36.750981+00,1426909553,257557,0
0,640725189,101745,549739,1,2023-08-27,3300.0000000,286018745,2023-07-30 14:16:36.750981+00,1427745864,257557,0
0,640725189,101745,549739,1,2023-08-28,3300.0000000,286018746,2023-07-30 14:16:36.750981+00,1428251172,257557,0
0,640725189,101745,549739,1,2023-08-29,3300.0000000,286018747,2023-07-30 14:16:36.750981+00,1429371338,257557,0
0,640725189,101745,549739,1,2023-08-30,3300.0000000,286018748,2023-07-30 14:16:36.750981+00,1430351716,257557,0
0,640725189,101745,549739,1,2023-08-31,3300.0000000,286018749,2023-07-30 14:16:36.750981+00,1431233935,257557,0
0,640725189,101745,549739,2,2023-08-01,3700.0000000,286018874,2023-07-30 14:16:36.750981+00,1403251634,257558,0
0,640725189,101745,549739,2,2023-08-02,3700.0000000,286018875,2023-07-30 14:16:36.750981+00,1404302256,257558,0
0,640725189,101745,549739,2,2023-08-03,3700.0000000,286018876,2023-07-30 14:16:36.750981+00,1405471672,257558,0
0,640725189,101745,549739,2,2023-08-04,3700.0000000,286018877,2023-07-30 14:16:36.750981+00,1406677111,257558,0
0,640725189,101745,549739,2,2023-08-05,3700.0000000,286018878,2023-07-30 14:16:36.750981+00,1407621373,257558,0
0,640725189,101745,549739,2,2023-08-06,3700.0000000,286018879,2023-07-30 14:16:36.750981+00,1408532976,257558,0
0,640725189,101745,549739,2,2023-08-07,3700.0000000,286018880,2023-07-30 14:16:36.750981+00,1409248621,257558,0
0,640725189,101745,549739,2,2023-08-08,3700.0000000,286018881,2023-07-30 14:16:36.750981+00,1409861556,257558,0
0,640725189,101745,549739,2,2023-08-09,3700.0000000,286018882,2023-07-30 14:16:36.750981+00,1410801647,257558,0
0,640725189,101745,549739,2,2023-08-10,3700.0000000,286018883,2023-07-30 14:16:36.750981+00,1411840681,257558,0
0,640725189,101745,549739,2,2023-08-11,3700.0000000,286018884,2023-07-30 14:16:36.750981+00,1412852367,257558,0
0,640725189,101745,549739,2,2023-08-12,3700.0000000,286018885,2023-07-30 14:16:36.750981+00,1413988765,257558,0
0,640725189,101745,549739,2,2023-08-13,3700.0000000,286018886,2023-07-30 14:16:36.750981+00,1414738258,257558,0
0,640725189,101745,549739,2,2023-08-14,3700.0000000,286018887,2023-07-30 14:16:36.750981+00,1415333542,257558,0
0,640725189,101745,549739,2,2023-08-15,3700.0000000,286018888,2023-07-30 14:16:36.750981+00,1415991464,257558,0
0,640725189,101745,549739,2,2023-08-16,3700.0000000,286018889,2023-07-30 14:16:36.750981+00,1417133264,257558,0
0,640725189,101745,549739,2,2023-08-17,3700.0000000,286018890,2023-07-30 14:16:36.750981+00,1417827149,257558,0
0,640725189,101745,549739,2,2023-08-18,3700.0000000,286018891,2023-07-30 14:16:36.750981+00,1418815192,257558,0
0,640725189,101745,549739,2,2023-08-19,3700.0000000,286018892,2023-07-30 14:16:36.750981+00,1420121246,257558,0
0,640725189,101745,549739,2,2023-08-20,3700.0000000,286018893,2023-07-30 14:16:36.750981+00,1421302961,257558,0
0,640725189,101745,549739,2,2023-08-21,3700.0000000,286018894,2023-07-30 14:16:36.750981+00,1421893793,257558,0
0,640725189,101745,549739,2,2023-08-22,3700.0000000,286018895,2023-07-30 14:16:36.750981+00,1422574746,257558,0
0,640725189,101745,549739,2,2023-08-23,3700.0000000,286018896,2023-07-30 14:16:36.750981+00,1423641874,257558,0
0,640725189,101745,549739,2,2023-08-24,3700.0000000,286018897,2023-07-30 14:16:36.750981+00,1425001762,257558,0
0,640725189,101745,549739,2,2023-08-25,3700.0000000,286018898,2023-07-30 14:16:36.750981+00,1426189092,257558,0
0,640725189,101745,549739,2,2023-08-26,3700.0000000,286018899,2023-07-30 14:16:36.750981+00,1426909552,257558,0
0,640725189,101745,549739,2,2023-08-27,3700.0000000,286018900,2023-07-30 14:16:36.750981+00,1427745863,257558,0
0,640725189,101745,549739,2,2023-08-28,3700.0000000,286018901,2023-07-30 14:16:36.750981+00,1428251171,257558,0
0,640725189,101745,549739,2,2023-08-29,3700.0000000,286018902,2023-07-30 14:16:36.750981+00,1429371337,257558,0
0,640725189,101745,549739,2,2023-08-30,3700.0000000,286018903,2023-07-30 14:16:36.750981+00,1430351715,257558,0
0,640725189,101745,549739,2,2023-08-31,3700.0000000,286018904,2023-07-30 14:16:36.750981+00,1431233934,257558,0
0,640725189,101747,203168,1,2023-08-01,5300.0000000,286019029,2023-07-30 14:16:36.750981+00,1403251629,257561,0
0,640725189,101747,203168,1,2023-08-02,5300.0000000,286019030,2023-07-30 14:16:36.750981+00,1404302251,257561,0
0,640725189,101747,203168,1,2023-08-03,5300.0000000,286019031,2023-07-30 14:16:36.750981+00,1405471667,257561,0
0,640725189,101747,203168,1,2023-08-04,5300.0000000,286019032,2023-07-30 14:16:36.750981+00,1406677106,257561,0
0,640725189,101747,203168,1,2023-08-05,5300.0000000,286019033,2023-07-30 14:16:36.750981+00,1407621368,257561,0
0,640725189,101747,203168,1,2023-08-06,5300.0000000,286019034,2023-07-30 14:16:36.750981+00,1408532971,257561,0
0,185647715,222578,489241,1,2023-07-31,3300.0000000,302101024,2023-07-30 14:16:36.750981+00,1402634706,567261,0
0,185647715,222578,896171,1,2023-07-30,4000.0000000,302101276,2023-07-30 14:16:36.750981+00,1401895973,567261,0
0,185647715,222578,896171,1,2023-07-31,4000.0000000,302101277,2023-07-30 14:16:36.750981+00,1402634719,567261,0
0,185647715,222578,896172,1,2023-07-30,4700.0000000,302101529,2023-07-30 14:16:36.750981+00,1401895986,567261,0
0,185647715,222578,896172,1,2023-07-31,4700.0000000,302101530,2023-07-30 14:16:36.750981+00,1402634732,567261,0
0,185647715,222578,1016845,1,2023-07-30,2750.0000000,302101782,2023-07-30 14:16:36.750981+00,1401895999,567261,0
0,185647715,222578,1016845,1,2023-07-31,2750.0000000,302101783,2023-07-30 14:16:36.750981+00,1402634745,567261,0
0,185647715,222639,489556,1,2023-07-30,3700.0000000,302102035,2023-07-30 14:16:36.750981+00,1401895962,567421,0
0,185647715,222639,489556,1,2023-07-31,3700.0000000,302102036,2023-07-30 14:16:36.750981+00,1402634708,567421,0
0,185647715,222639,489556,2,2023-07-30,5200.0000000,302102288,2023-07-30 14:16:36.750981+00,1401895961,567422,0
0,185647715,222639,489556,2,2023-07-31,5200.0000000,302102289,2023-07-30 14:16:36.750981+00,1402634707,567422,0
0,185647715,222639,896171,1,2023-07-30,4400.0000000,302102541,2023-07-30 14:16:36.750981+00,1401895975,567421,0
0,185647715,222639,896171,1,2023-07-31,4400.0000000,302102542,2023-07-30 14:16:36.750981+00,1402634721,567421,0
0,185647715,222639,896171,2,2023-07-30,6600.0000000,302102794,2023-07-30 14:16:36.750981+00,1401895974,567422,0
0,185647715,222639,896171,2,2023-07-31,6600.0000000,302102795,2023-07-30 14:16:36.750981+00,1402634720,567422,0
0,185647715,222639,896172,1,2023-07-30,5100.0000000,302103047,2023-07-30 14:16:36.750981+00,1401895988,567421,0
0,185647715,222639,896172,1,2023-07-31,5100.0000000,302103048,2023-07-30 14:16:36.750981+00,1402634734,567421,0
0,185647715,222639,896172,2,2023-07-30,8000.0000000,302103300,2023-07-30 14:16:36.750981+00,1401895987,567422,0
0,185647715,222639,896172,2,2023-07-31,8000.0000000,302103301,2023-07-30 14:16:36.750981+00,1402634733,567422,0
0,185647715,222639,1016845,1,2023-07-30,3150.0000000,302103553,2023-07-30 14:16:36.750981+00,1401896001,567421,0
0,185647715,222639,1016845,1,2023-07-31,3150.0000000,302103554,2023-07-30 14:16:36.750981+00,1402634747,567421,0
0,185647715,222639,1016845,2,2023-07-30,4100.0000000,302103806,2023-07-30 14:16:36.750981+00,1401896000,567422,0
0,185647715,222639,1016845,2,2023-07-31,4100.0000000,302103807,2023-07-30 14:16:36.750981+00,1402634746,567422,0
0,185647715,222648,489600,1,2023-07-30,3700.0000000,302104059,2023-07-30 14:16:36.750981+00,1401895964,567436,0
0,185647715,222648,489600,1,2023-07-31,3700.0000000,302104060,2023-07-30 14:16:36.750981+00,1402634710,567436,0
0,185647715,222648,489600,2,2023-07-30,5200.0000000,302104312,2023-07-30 14:16:36.750981+00,1401895963,567437,0
0,185647715,222648,489600,2,2023-07-31,5200.0000000,302104313,2023-07-30 14:16:36.750981+00,1402634709,567437,0
0,185647715,222648,896171,1,2023-07-30,4400.0000000,302104565,2023-07-30 14:16:36.750981+00,1401895977,567436,0
0,185647715,222648,896171,1,2023-07-31,4400.0000000,302104566,2023-07-30 14:16:36.750981+00,1402634723,567436,0
0,185647715,222648,896171,2,2023-07-30,6600.0000000,302104818,2023-07-30 14:16:36.750981+00,1401895976,567437,0
0,185647715,222648,896171,2,2023-07-31,6600.0000000,302104819,2023-07-30 14:16:36.750981+00,1402634722,567437,0
0,185647715,222648,896172,1,2023-07-30,5100.0000000,302105071,2023-07-30 14:16:36.750981+00,1401895990,567436,0
0,185647715,222648,896172,1,2023-07-31,5100.0000000,302105072,2023-07-30 14:16:36.750981+00,1402634736,567436,0
0,185647715,222648,896172,2,2023-07-30,8000.0000000,302105324,2023-07-30 14:16:36.750981+00,1401895989,567437,0
0,185647715,222648,896172,2,2023-07-31,8000.0000000,302105325,2023-07-30 14:16:36.750981+00,1402634735,567437,0
0,185647715,222648,1016845,1,2023-07-30,3150.0000000,302105577,2023-07-30 14:16:36.750981+00,1401896003,567436,0
0,185647715,222648,1016845,1,2023-07-31,3150.0000000,302105578,2023-07-30 14:16:36.750981+00,1402634749,567436,0
0,185647715,222648,1016845,2,2023-07-30,4100.0000000,302105830,2023-07-30 14:16:36.750981+00,1401896002,567437,0
0,185647715,222648,1016845,2,2023-07-31,4100.0000000,302105831,2023-07-30 14:16:36.750981+00,1402634748,567437,0
0,185647715,222653,489627,1,2023-07-30,4800.0000000,302106083,2023-07-30 14:16:36.750981+00,1401895966,567446,0
0,185647715,222653,489627,1,2023-07-31,4800.0000000,302106084,2023-07-30 14:16:36.750981+00,1402634712,567446,0
0,185647715,222653,489627,2,2023-07-30,5800.0000000,302106336,2023-07-30 14:16:36.750981+00,1401895965,567447,0
0,185647715,222653,489627,2,2023-07-31,5800.0000000,302106337,2023-07-30 14:16:36.750981+00,1402634711,567447,0
0,185647715,222653,896171,1,2023-07-30,5500.0000000,302106589,2023-07-30 14:16:36.750981+00,1401895979,567446,0
0,185647715,222653,896171,1,2023-07-31,5500.0000000,302106590,2023-07-30 14:16:36.750981+00,1402634725,567446,0
0,185647715,222653,896171,2,2023-07-30,7200.0000000,302106842,2023-07-30 14:16:36.750981+00,1401895978,567447,0
0,185647715,222653,896171,2,2023-07-31,7200.0000000,302106843,2023-07-30 14:16:36.750981+00,1402634724,567447,0
0,185647715,222653,896172,1,2023-07-30,6200.0000000,302107095,2023-07-30 14:16:36.750981+00,1401895992,567446,0
0,185647715,222653,896172,1,2023-07-31,6200.0000000,302107096,2023-07-30 14:16:36.750981+00,1402634738,567446,0
0,185647715,222653,896172,2,2023-07-30,8600.0000000,302107348,2023-07-30 14:16:36.750981+00,1401895991,567447,0
0,185647715,222653,896172,2,2023-07-31,8600.0000000,302107349,2023-07-30 14:16:36.750981+00,1402634737,567447,0
0,185647715,222653,1016845,1,2023-07-30,4250.0000000,302107601,2023-07-30 14:16:36.750981+00,1401896005,567446,0
0,185647715,222653,1016845,1,2023-07-31,4250.0000000,302107602,2023-07-30 14:16:36.750981+00,1402634751,567446,0
0,185647715,222653,1016845,2,2023-07-30,4700.0000000,302107854,2023-07-30 14:16:36.750981+00,1401896004,567447,0
0,185647715,222653,1016845,2,2023-07-31,4700.0000000,302107855,2023-07-30 14:16:36.750981+00,1402634750,567447,0
0,185647715,222654,489632,1,2023-07-30,7200.0000000,302108107,2023-07-30 14:16:36.750981+00,1401895968,567449,0
0,185647715,222654,489632,1,2023-07-31,7200.0000000,302108108,2023-07-30 14:16:36.750981+00,1402634714,567449,0
0,185647715,222654,489632,2,2023-07-30,7200.0000000,302108360,2023-07-30 14:16:36.750981+00,1401895967,567450,0
0,185647715,222654,489632,2,2023-07-31,7200.0000000,302108361,2023-07-30 14:16:36.750981+00,1402634713,567450,0
0,185647715,222654,896171,1,2023-07-30,6900.0000000,302108613,2023-07-30 14:16:36.750981+00,1401895981,567449,0
0,185647715,222654,896171,1,2023-07-31,6900.0000000,302108614,2023-07-30 14:16:36.750981+00,1402634727,567449,0
0,185647715,222654,896171,2,2023-07-30,8600.0000000,302108866,2023-07-30 14:16:36.750981+00,1401895980,567450,0
0,185647715,222654,896171,2,2023-07-31,8600.0000000,302108867,2023-07-30 14:16:36.750981+00,1402634726,567450,0
0,185647715,222654,896172,1,2023-07-30,7600.0000000,302109119,2023-07-30 14:16:36.750981+00,1401895994,567449,0
0,185647715,222654,896172,1,2023-07-31,7600.0000000,302109120,2023-07-30 14:16:36.750981+00,1402634740,567449,0
0,185647715,222654,896172,2,2023-07-30,10000.0000000,302109372,2023-07-30 14:16:36.750981+00,1401895993,567450,0
0,185647715,222654,896172,2,2023-07-31,10000.0000000,302109373,2023-07-30 14:16:36.750981+00,1402634739,567450,0
0,185647715,222654,1016845,1,2023-07-30,5650.0000000,302109625,2023-07-30 14:16:36.750981+00,1401896007,567449,0
0,185647715,222654,1016845,1,2023-07-31,5650.0000000,302109626,2023-07-30 14:16:36.750981+00,1402634753,567449,0
0,185647715,222654,1016845,2,2023-07-30,6100.0000000,302109878,2023-07-30 14:16:36.750981+00,1401896006,567450,0
0,185647715,222654,1016845,2,2023-07-31,6100.0000000,302109879,2023-07-30 14:16:36.750981+00,1402634752,567450,0
0,185647715,222655,489637,1,2023-07-30,7800.0000000,302110131,2023-07-30 14:16:36.750981+00,1401895970,567452,0
0,185647715,222655,489637,1,2023-07-31,7800.0000000,302110132,2023-07-30 14:16:36.750981+00,1402634716,567452,0
0,185647715,222655,489637,2,2023-07-30,9100.0000000,302110384,2023-07-30 14:16:36.750981+00,1401895969,567453,0
0,185647715,222655,489637,2,2023-07-31,9100.0000000,302110385,2023-07-30 14:16:36.750981+00,1402634715,567453,0
0,185647715,222655,896171,1,2023-07-30,8500.0000000,302110637,2023-07-30 14:16:36.750981+00,1401895983,567452,0
0,185647715,222655,896171,1,2023-07-31,8500.0000000,302110638,2023-07-30 14:16:36.750981+00,1402634729,567452,0
0,185647715,222655,896171,2,2023-07-30,10500.0000000,302110890,2023-07-30 14:16:36.750981+00,1401895982,567453,0
0,185647715,222655,896171,2,2023-07-31,10500.0000000,302110891,2023-07-30 14:16:36.750981+00,1402634728,567453,0
0,185647715,222655,896172,1,2023-07-30,9200.0000000,302111143,2023-07-30 14:16:36.750981+00,1401895996,567452,0
0,185647715,222655,896172,1,2023-07-31,9200.0000000,302111144,2023-07-30 14:16:36.750981+00,1402634742,567452,0
0,185647715,222655,896172,2,2023-07-30,11900.0000000,302111396,2023-07-30 14:16:36.750981+00,1401895995,567453,0
0,185647715,222655,896172,2,2023-07-31,11900.0000000,302111397,2023-07-30 14:16:36.750981+00,1402634741,567453,0
0,185647715,222655,1016845,1,2023-07-30,7250.0000000,302111649,2023-07-30 14:16:36.750981+00,1401896009,567452,0
0,185647715,222655,1016845,1,2023-07-31,7250.0000000,302111650,2023-07-30 14:16:36.750981+00,1402634755,567452,0
0,185647715,222655,1016845,2,2023-07-30,8000.0000000,302111902,2023-07-30 14:16:36.750981+00,1401896008,567453,0
0,185647715,222655,1016845,2,2023-07-31,8000.0000000,302111903,2023-07-30 14:16:36.750981+00,1402634754,567453,0
0,185647715,222656,489642,1,2023-07-30,12000.0000000,302112155,2023-07-30 14:16:36.750981+00,1401895972,567455,0
0,185647715,222656,489642,1,2023-07-31,12000.0000000,302112156,2023-07-30 14:16:36.750981+00,1402634718,567455,0
0,185647715,222656,489642,2,2023-07-30,14500.0000000,302112408,2023-07-30 14:16:36.750981+00,1401895971,567456,0
0,185647715,222656,489642,2,2023-07-31,14500.0000000,302112409,2023-07-30 14:16:36.750981+00,1402634717,567456,0
0,185647715,222656,896171,1,2023-07-30,12700.0000000,302112661,2023-07-30 14:16:36.750981+00,1401895985,567455,0
0,185647715,222656,896171,1,2023-07-31,12700.0000000,302112662,2023-07-30 14:16:36.750981+00,1402634731,567455,0
0,185647715,222656,896171,2,2023-07-30,15900.0000000,302112914,2023-07-30 14:16:36.750981+00,1401895984,567456,0
0,185647715,222656,896171,2,2023-07-31,15900.0000000,302112915,2023-07-30 14:16:36.750981+00,1402634730,567456,0
0,185647715,222656,896172,1,2023-07-30,13400.0000000,302113167,2023-07-30 14:16:36.750981+00,1401895998,567455,0
0,185647715,222656,896172,1,2023-07-31,13400.0000000,302113168,2023-07-30 14:16:36.750981+00,1402634744,567455,0
0,185647715,222656,896172,2,2023-07-30,17300.0000000,302113420,2023-07-30 14:16:36.750981+00,1401895997,567456,0
0,185647715,222656,896172,2,2023-07-31,17300.0000000,302113421,2023-07-30 14:16:36.750981+00,1402634743,567456,0
0,185647715,222656,1016845,1,2023-07-30,11450.0000000,302113673,2023-07-30 14:16:36.750981+00,1401896011,567455,0
0,185647715,222656,1016845,1,2023-07-31,11450.0000000,302113674,2023-07-30 14:16:36.750981+00,1402634757,567455,0
0,185647715,222656,1016845,2,2023-07-30,13400.0000000,302113926,2023-07-30 14:16:36.750981+00,1401896010,567456,0
0,185647715,222656,1016845,2,2023-07-31,13400.0000000,302113927,2023-07-30 14:16:36.750981+00,1402634756,567456,0
0,640725189,101742,203163,1,2023-07-30,2700.0000000,286017477,2023-07-30 14:16:36.750981+00,1401882107,257550,0
0,640725189,101742,203163,1,2023-07-31,2700.0000000,286017478,2023-07-30 14:16:36.750981+00,1402623424,257550,0
0,640725189,101742,549736,1,2023-07-30,2350.0000000,286017632,2023-07-30 14:16:36.750981+00,1401882114,257550,0
0,640725189,101742,549736,1,2023-07-31,2350.0000000,286017633,2023-07-30 14:16:36.750981+00,1402623431,257550,0
0,640725189,101743,203164,1,2023-07-30,3350.0000000,286017787,2023-07-30 14:16:36.750981+00,1401882108,257551,0
0,640725189,101743,203164,1,2023-07-31,3350.0000000,286017788,2023-07-30 14:16:36.750981+00,1402623425,257551,0
0,640725189,101743,549737,1,2023-07-30,3000.0000000,286017942,2023-07-30 14:16:36.750981+00,1401882115,257551,0
0,640725189,101743,549737,1,2023-07-31,3000.0000000,286017943,2023-07-30 14:16:36.750981+00,1402623432,257551,0
0,640725189,101744,203165,2,2023-07-30,3950.0000000,286018097,2023-07-30 14:16:36.750981+00,1401882109,257553,0
0,640725189,101744,203165,2,2023-07-31,3950.0000000,286018098,2023-07-30 14:16:36.750981+00,1402623426,257553,0
0,640725189,101744,549738,2,2023-07-30,3250.0000000,286018252,2023-07-30 14:16:36.750981+00,1401882116,257553,0
0,640725189,101744,549738,2,2023-07-31,3250.0000000,286018253,2023-07-30 14:16:36.750981+00,1402623433,257553,0
0,640725189,101745,203166,1,2023-07-30,3650.0000000,286018407,2023-07-30 14:16:36.750981+00,1401882110,257557,0
0,640725189,101745,203166,1,2023-07-31,3650.0000000,286018408,2023-07-30 14:16:36.750981+00,1402623427,257557,0
0,640725189,101745,203166,2,2023-07-30,4400.0000000,286018562,2023-07-30 14:16:36.750981+00,1401882111,257558,0
0,640725189,101745,203166,2,2023-07-31,4400.0000000,286018563,2023-07-30 14:16:36.750981+00,1402623428,257558,0
0,640725189,101745,549739,1,2023-07-30,3300.0000000,286018717,2023-07-30 14:16:36.750981+00,1401882118,257557,0
0,640725189,101745,549739,1,2023-07-31,3300.0000000,286018718,2023-07-30 14:16:36.750981+00,1402623436,257557,0
0,640725189,101745,549739,2,2023-07-30,3700.0000000,286018872,2023-07-30 14:16:36.750981+00,1401882117,257558,0
0,640725189,101745,549739,2,2023-07-31,3700.0000000,286018873,2023-07-30 14:16:36.750981+00,1402623435,257558,0
0,640725189,101747,203168,1,2023-07-30,5300.0000000,286019027,2023-07-30 14:16:36.750981+00,1401882112,257561,0
0,640725189,101747,203168,1,2023-07-31,5300.0000000,286019028,2023-07-30 14:16:36.750981+00,1402623429,257561,0
0,640725189,101747,203168,2,2023-07-30,5800.0000000,286019182,2023-07-30 14:16:36.750981+00,1401882113,257562,0
0,640725189,101747,203168,2,2023-07-31,5800.0000000,286019183,2023-07-30 14:16:36.750981+00,1402623430,257562,0
0,640725189,101747,549740,1,2023-07-30,4950.0000000,286019337,2023-07-30 14:16:36.750981+00,1401882120,257561,0
0,640725189,101747,549740,1,2023-07-31,4950.0000000,286019338,2023-07-30 14:16:36.750981+00,1402623440,257561,0
0,640725189,101747,549740,2,2023-07-30,5100.0000000,286019492,2023-07-30 14:16:36.750981+00,1401882119,257562,0
0,640725189,101747,549740,2,2023-07-31,5100.0000000,286019493,2023-07-30 14:16:36.750981+00,1402623438,257562,0
0,961513435,345079,972890,1,2023-07-30,2500.0000000,230140479,2023-07-30 14:16:36.750981+00,1165235851,925030,0
0,961513435,345079,972890,1,2023-07-31,2500.0000000,230140480,2023-07-30 14:16:36.750981+00,1165235852,925030,0
0,961513435,355770,972890,1,2023-07-30,1300.0000000,230140944,2023-07-30 14:16:36.750981+00,1184969008,955856,0
0,961513435,355770,972890,1,2023-07-31,1300.0000000,230140945,2023-07-30 14:16:36.750981+00,1184969011,955856,0
0,961513435,355770,1012400,1,2023-07-30,1300.0000000,230141409,2023-07-30 14:16:36.750981+00,1165236070,955856,0
0,961513435,355770,1012400,1,2023-07-31,1300.0000000,230141410,2023-07-30 14:16:36.750981+00,1165236071,955856,0
0,961513435,355771,972890,1,2023-07-30,1300.0000000,230141431,2023-07-30 14:16:36.750981+00,1184970062,955857,0
0,961513435,355771,972890,1,2023-07-31,1300.0000000,230141432,2023-07-30 14:16:36.750981+00,1184970063,955857,0
0,961513435,355771,1012401,1,2023-07-30,1300.0000000,230141896,2023-07-30 14:16:36.750981+00,1165236293,955857,0
0,961513435,355771,1012401,1,2023-07-31,1300.0000000,230141897,2023-07-30 14:16:36.750981+00,1165236295,955857,0
0,954493183,216176,461514,1,2024-12-01,5500.0000000,204562868,2023-07-30 14:16:36.750981+00,1756312461,548487,0
0,640725189,101747,203168,1,2023-08-07,5300.0000000,286019035,2023-07-30 14:16:36.750981+00,1409248616,257561,0
0,640725189,101747,203168,1,2023-08-08,5300.0000000,286019036,2023-07-30 14:16:36.750981+00,1409861551,257561,0
0,640725189,101747,203168,1,2023-08-09,5300.0000000,286019037,2023-07-30 14:16:36.750981+00,1410801642,257561,0
0,640725189,101747,203168,1,2023-08-10,5300.0000000,286019038,2023-07-30 14:16:36.750981+00,1411840676,257561,0
0,640725189,101747,203168,1,2023-08-11,5300.0000000,286019039,2023-07-30 14:16:36.750981+00,1412852359,257561,0
0,640725189,101747,203168,1,2023-08-12,5300.0000000,286019040,2023-07-30 14:16:36.750981+00,1413988760,257561,0
0,640725189,101747,203168,1,2023-08-13,5300.0000000,286019041,2023-07-30 14:16:36.750981+00,1414738246,257561,0
0,640725189,101747,203168,1,2023-08-14,5300.0000000,286019042,2023-07-30 14:16:36.750981+00,1415333537,257561,0
0,640725189,101747,203168,1,2023-08-15,5300.0000000,286019043,2023-07-30 14:16:36.750981+00,1415991459,257561,0
0,640725189,101747,203168,1,2023-08-16,5300.0000000,286019044,2023-07-30 14:16:36.750981+00,1417133249,257561,0
0,640725189,101747,203168,1,2023-08-17,5300.0000000,286019045,2023-07-30 14:16:36.750981+00,1417827144,257561,0
0,640725189,101747,203168,1,2023-08-18,5300.0000000,286019046,2023-07-30 14:16:36.750981+00,1418815187,257561,0
0,640725189,101747,203168,1,2023-08-19,5300.0000000,286019047,2023-07-30 14:16:36.750981+00,1420121241,257561,0
0,640725189,101747,203168,1,2023-08-20,5300.0000000,286019048,2023-07-30 14:16:36.750981+00,1421302956,257561,0
0,640725189,101747,203168,1,2023-08-21,5300.0000000,286019049,2023-07-30 14:16:36.750981+00,1421893788,257561,0
0,640725189,101747,203168,1,2023-08-22,5300.0000000,286019050,2023-07-30 14:16:36.750981+00,1422574741,257561,0
0,640725189,101747,203168,1,2023-08-23,5300.0000000,286019051,2023-07-30 14:16:36.750981+00,1423641869,257561,0
0,640725189,101747,203168,1,2023-08-24,5300.0000000,286019052,2023-07-30 14:16:36.750981+00,1425001757,257561,0
0,640725189,101747,203168,1,2023-08-25,5300.0000000,286019053,2023-07-30 14:16:36.750981+00,1426189087,257561,0
0,640725189,101747,203168,1,2023-08-26,5300.0000000,286019054,2023-07-30 14:16:36.750981+00,1426909547,257561,0
0,640725189,101747,203168,1,2023-08-27,5300.0000000,286019055,2023-07-30 14:16:36.750981+00,1427745858,257561,0
0,640725189,101747,203168,1,2023-08-28,5300.0000000,286019056,2023-07-30 14:16:36.750981+00,1428251166,257561,0
0,640725189,101747,203168,1,2023-08-29,5300.0000000,286019057,2023-07-30 14:16:36.750981+00,1429371332,257561,0
0,640725189,101747,203168,1,2023-08-30,5300.0000000,286019058,2023-07-30 14:16:36.750981+00,1430351710,257561,0
0,640725189,101747,203168,1,2023-08-31,5300.0000000,286019059,2023-07-30 14:16:36.750981+00,1431233929,257561,0
0,640725189,101747,203168,2,2023-08-01,5800.0000000,286019184,2023-07-30 14:16:36.750981+00,1403251630,257562,0
0,640725189,101747,203168,2,2023-08-02,5800.0000000,286019185,2023-07-30 14:16:36.750981+00,1404302252,257562,0
0,640725189,101747,203168,2,2023-08-03,5800.0000000,286019186,2023-07-30 14:16:36.750981+00,1405471668,257562,0
0,640725189,101747,203168,2,2023-08-04,5800.0000000,286019187,2023-07-30 14:16:36.750981+00,1406677107,257562,0
0,640725189,101747,203168,2,2023-08-05,5800.0000000,286019188,2023-07-30 14:16:36.750981+00,1407621369,257562,0
0,640725189,101747,203168,2,2023-08-06,5800.0000000,286019189,2023-07-30 14:16:36.750981+00,1408532972,257562,0
0,640725189,101747,203168,2,2023-08-07,5800.0000000,286019190,2023-07-30 14:16:36.750981+00,1409248617,257562,0
0,640725189,101747,203168,2,2023-08-08,5800.0000000,286019191,2023-07-30 14:16:36.750981+00,1409861552,257562,0
0,640725189,101747,203168,2,2023-08-09,5800.0000000,286019192,2023-07-30 14:16:36.750981+00,1410801643,257562,0
0,640725189,101747,203168,2,2023-08-10,5800.0000000,286019193,2023-07-30 14:16:36.750981+00,1411840677,257562,0
0,640725189,101747,203168,2,2023-08-11,5800.0000000,286019194,2023-07-30 14:16:36.750981+00,1412852360,257562,0
0,640725189,101747,203168,2,2023-08-12,5800.0000000,286019195,2023-07-30 14:16:36.750981+00,1413988761,257562,0
0,640725189,101747,203168,2,2023-08-13,5800.0000000,286019196,2023-07-30 14:16:36.750981+00,1414738249,257562,0
0,640725189,101747,203168,2,2023-08-14,5800.0000000,286019197,2023-07-30 14:16:36.750981+00,1415333538,257562,0
0,640725189,101747,203168,2,2023-08-15,5800.0000000,286019198,2023-07-30 14:16:36.750981+00,1415991460,257562,0
0,640725189,101747,203168,2,2023-08-16,5800.0000000,286019199,2023-07-30 14:16:36.750981+00,1417133252,257562,0
0,640725189,101747,203168,2,2023-08-17,5800.0000000,286019200,2023-07-30 14:16:36.750981+00,1417827145,257562,0
0,640725189,101747,203168,2,2023-08-18,5800.0000000,286019201,2023-07-30 14:16:36.750981+00,1418815188,257562,0
0,640725189,101747,203168,2,2023-08-19,5800.0000000,286019202,2023-07-30 14:16:36.750981+00,1420121242,257562,0
0,640725189,101747,203168,2,2023-08-20,5800.0000000,286019203,2023-07-30 14:16:36.750981+00,1421302957,257562,0
0,640725189,101747,203168,2,2023-08-21,5800.0000000,286019204,2023-07-30 14:16:36.750981+00,1421893789,257562,0
0,640725189,101747,203168,2,2023-08-22,5800.0000000,286019205,2023-07-30 14:16:36.750981+00,1422574742,257562,0
0,640725189,101747,203168,2,2023-08-23,5800.0000000,286019206,2023-07-30 14:16:36.750981+00,1423641870,257562,0
0,640725189,101747,203168,2,2023-08-24,5800.0000000,286019207,2023-07-30 14:16:36.750981+00,1425001758,257562,0
0,640725189,101747,203168,2,2023-08-25,5800.0000000,286019208,2023-07-30 14:16:36.750981+00,1426189088,257562,0
0,640725189,101747,203168,2,2023-08-26,5800.0000000,286019209,2023-07-30 14:16:36.750981+00,1426909548,257562,0
0,640725189,101747,203168,2,2023-08-27,5800.0000000,286019210,2023-07-30 14:16:36.750981+00,1427745859,257562,0
0,640725189,101747,203168,2,2023-08-28,5800.0000000,286019211,2023-07-30 14:16:36.750981+00,1428251167,257562,0
0,640725189,101747,203168,2,2023-08-29,5800.0000000,286019212,2023-07-30 14:16:36.750981+00,1429371333,257562,0
0,640725189,101747,203168,2,2023-08-30,5800.0000000,286019213,2023-07-30 14:16:36.750981+00,1430351711,257562,0
0,640725189,101747,203168,2,2023-08-31,5800.0000000,286019214,2023-07-30 14:16:36.750981+00,1431233930,257562,0
0,640725189,101747,549740,1,2023-08-01,4950.0000000,286019339,2023-07-30 14:16:36.750981+00,1403251637,257561,0
0,640725189,101747,549740,1,2023-08-02,4950.0000000,286019340,2023-07-30 14:16:36.750981+00,1404302259,257561,0
0,640725189,101747,549740,1,2023-08-03,4950.0000000,286019341,2023-07-30 14:16:36.750981+00,1405471675,257561,0
0,640725189,101747,549740,1,2023-08-04,4950.0000000,286019342,2023-07-30 14:16:36.750981+00,1406677114,257561,0
0,640725189,101747,549740,1,2023-08-05,4950.0000000,286019343,2023-07-30 14:16:36.750981+00,1407621376,257561,0
0,640725189,101747,549740,1,2023-08-06,4950.0000000,286019344,2023-07-30 14:16:36.750981+00,1408532979,257561,0
0,640725189,101747,549740,1,2023-08-07,4950.0000000,286019345,2023-07-30 14:16:36.750981+00,1409248624,257561,0
0,640725189,101747,549740,1,2023-08-08,4950.0000000,286019346,2023-07-30 14:16:36.750981+00,1409861559,257561,0
0,640725189,101747,549740,1,2023-08-09,4950.0000000,286019347,2023-07-30 14:16:36.750981+00,1410801650,257561,0
0,640725189,101747,549740,1,2023-08-10,4950.0000000,286019348,2023-07-30 14:16:36.750981+00,1411840684,257561,0
0,640725189,101747,549740,1,2023-08-11,4950.0000000,286019349,2023-07-30 14:16:36.750981+00,1412852372,257561,0
0,640725189,101747,549740,1,2023-08-12,4950.0000000,286019350,2023-07-30 14:16:36.750981+00,1413988768,257561,0
0,640725189,101747,549740,1,2023-08-13,4950.0000000,286019351,2023-07-30 14:16:36.750981+00,1414738265,257561,0
0,640725189,101747,549740,1,2023-08-14,4950.0000000,286019352,2023-07-30 14:16:36.750981+00,1415333545,257561,0
0,640725189,101747,549740,1,2023-08-15,4950.0000000,286019353,2023-07-30 14:16:36.750981+00,1415991467,257561,0
0,640725189,101747,549740,1,2023-08-16,4950.0000000,286019354,2023-07-30 14:16:36.750981+00,1417133272,257561,0
0,640725189,101747,549740,1,2023-08-17,4950.0000000,286019355,2023-07-30 14:16:36.750981+00,1417827152,257561,0
0,640725189,101747,549740,1,2023-08-18,4950.0000000,286019356,2023-07-30 14:16:36.750981+00,1418815195,257561,0
0,640725189,101747,549740,1,2023-08-19,4950.0000000,286019357,2023-07-30 14:16:36.750981+00,1420121249,257561,0
0,640725189,101747,549740,1,2023-08-20,4950.0000000,286019358,2023-07-30 14:16:36.750981+00,1421302964,257561,0
0,640725189,101747,549740,1,2023-08-21,4950.0000000,286019359,2023-07-30 14:16:36.750981+00,1421893796,257561,0
0,640725189,101747,549740,1,2023-08-22,4950.0000000,286019360,2023-07-30 14:16:36.750981+00,1422574749,257561,0
0,640725189,101747,549740,1,2023-08-23,4950.0000000,286019361,2023-07-30 14:16:36.750981+00,1423641877,257561,0
0,640725189,101747,549740,1,2023-08-24,4950.0000000,286019362,2023-07-30 14:16:36.750981+00,1425001765,257561,0
0,640725189,101747,549740,1,2023-08-25,4950.0000000,286019363,2023-07-30 14:16:36.750981+00,1426189095,257561,0
0,640725189,101747,549740,1,2023-08-26,4950.0000000,286019364,2023-07-30 14:16:36.750981+00,1426909555,257561,0
0,640725189,101747,549740,1,2023-08-27,4950.0000000,286019365,2023-07-30 14:16:36.750981+00,1427745866,257561,0
0,640725189,101747,549740,1,2023-08-28,4950.0000000,286019366,2023-07-30 14:16:36.750981+00,1428251174,257561,0
0,640725189,101747,549740,1,2023-08-29,4950.0000000,286019367,2023-07-30 14:16:36.750981+00,1429371340,257561,0
0,640725189,101747,549740,1,2023-08-30,4950.0000000,286019368,2023-07-30 14:16:36.750981+00,1430351718,257561,0
0,640725189,101747,549740,1,2023-08-31,4950.0000000,286019369,2023-07-30 14:16:36.750981+00,1431233937,257561,0
0,640725189,101747,549740,2,2023-08-01,5100.0000000,286019494,2023-07-30 14:16:36.750981+00,1403251636,257562,0
0,640725189,101747,549740,2,2023-08-02,5100.0000000,286019495,2023-07-30 14:16:36.750981+00,1404302258,257562,0
0,640725189,101747,549740,2,2023-08-03,5100.0000000,286019496,2023-07-30 14:16:36.750981+00,1405471674,257562,0
0,640725189,101747,549740,2,2023-08-04,5100.0000000,286019497,2023-07-30 14:16:36.750981+00,1406677113,257562,0
0,640725189,101747,549740,2,2023-08-05,5100.0000000,286019498,2023-07-30 14:16:36.750981+00,1407621375,257562,0
0,640725189,101747,549740,2,2023-08-06,5100.0000000,286019499,2023-07-30 14:16:36.750981+00,1408532978,257562,0
0,640725189,101747,549740,2,2023-08-07,5100.0000000,286019500,2023-07-30 14:16:36.750981+00,1409248623,257562,0
0,640725189,101747,549740,2,2023-08-08,5100.0000000,286019501,2023-07-30 14:16:36.750981+00,1409861558,257562,0
0,640725189,101747,549740,2,2023-08-09,5100.0000000,286019502,2023-07-30 14:16:36.750981+00,1410801649,257562,0
0,640725189,101747,549740,2,2023-08-10,5100.0000000,286019503,2023-07-30 14:16:36.750981+00,1411840683,257562,0
0,640725189,101747,549740,2,2023-08-11,5100.0000000,286019504,2023-07-30 14:16:36.750981+00,1412852370,257562,0
0,640725189,101747,549740,2,2023-08-12,5100.0000000,286019505,2023-07-30 14:16:36.750981+00,1413988767,257562,0
0,640725189,101747,549740,2,2023-08-13,5100.0000000,286019506,2023-07-30 14:16:36.750981+00,1414738263,257562,0
0,640725189,101747,549740,2,2023-08-14,5100.0000000,286019507,2023-07-30 14:16:36.750981+00,1415333544,257562,0
0,640725189,101747,549740,2,2023-08-15,5100.0000000,286019508,2023-07-30 14:16:36.750981+00,1415991466,257562,0
0,640725189,101747,549740,2,2023-08-16,5100.0000000,286019509,2023-07-30 14:16:36.750981+00,1417133270,257562,0
0,640725189,101747,549740,2,2023-08-17,5100.0000000,286019510,2023-07-30 14:16:36.750981+00,1417827151,257562,0
0,640725189,101747,549740,2,2023-08-18,5100.0000000,286019511,2023-07-30 14:16:36.750981+00,1418815194,257562,0
0,640725189,101747,549740,2,2023-08-19,5100.0000000,286019512,2023-07-30 14:16:36.750981+00,1420121248,257562,0
0,640725189,101747,549740,2,2023-08-20,5100.0000000,286019513,2023-07-30 14:16:36.750981+00,1421302963,257562,0
0,640725189,101747,549740,2,2023-08-21,5100.0000000,286019514,2023-07-30 14:16:36.750981+00,1421893795,257562,0
0,640725189,101747,549740,2,2023-08-22,5100.0000000,286019515,2023-07-30 14:16:36.750981+00,1422574748,257562,0
0,640725189,101747,549740,2,2023-08-23,5100.0000000,286019516,2023-07-30 14:16:36.750981+00,1423641876,257562,0
0,640725189,101747,549740,2,2023-08-24,5100.0000000,286019517,2023-07-30 14:16:36.750981+00,1425001764,257562,0
0,640725189,101747,549740,2,2023-08-25,5100.0000000,286019518,2023-07-30 14:16:36.750981+00,1426189094,257562,0
0,640725189,101747,549740,2,2023-08-26,5100.0000000,286019519,2023-07-30 14:16:36.750981+00,1426909554,257562,0
0,640725189,101747,549740,2,2023-08-27,5100.0000000,286019520,2023-07-30 14:16:36.750981+00,1427745865,257562,0
0,640725189,101747,549740,2,2023-08-28,5100.0000000,286019521,2023-07-30 14:16:36.750981+00,1428251173,257562,0
0,640725189,101747,549740,2,2023-08-29,5100.0000000,286019522,2023-07-30 14:16:36.750981+00,1429371339,257562,0
0,640725189,101747,549740,2,2023-08-30,5100.0000000,286019523,2023-07-30 14:16:36.750981+00,1430351717,257562,0
0,640725189,101747,549740,2,2023-08-31,5100.0000000,286019524,2023-07-30 14:16:36.750981+00,1431233936,257562,0
0,954493183,216176,461514,1,2023-08-01,5500.0000000,204562380,2023-07-30 14:16:36.750981+00,1183374858,548487,0
0,954493183,216176,461514,1,2023-08-02,5500.0000000,204562381,2023-07-30 14:16:36.750981+00,1183374861,548487,0
0,954493183,216176,461514,1,2023-08-03,5500.0000000,204562382,2023-07-30 14:16:36.750981+00,1183374869,548487,0
0,954493183,216176,461514,1,2023-08-04,5500.0000000,204562383,2023-07-30 14:16:36.750981+00,1183374870,548487,0
0,954493183,216176,461514,1,2023-08-05,5500.0000000,204562384,2023-07-30 14:16:36.750981+00,1183374873,548487,0
0,954493183,216176,461514,1,2023-08-06,5500.0000000,204562385,2023-07-30 14:16:36.750981+00,1183374877,548487,0
0,954493183,216176,461514,1,2023-08-07,5500.0000000,204562386,2023-07-30 14:16:36.750981+00,1183374881,548487,0
0,954493183,216176,461514,1,2023-08-08,5500.0000000,204562387,2023-07-30 14:16:36.750981+00,1183374883,548487,0
0,954493183,216176,461514,1,2023-08-09,5500.0000000,204562388,2023-07-30 14:16:36.750981+00,1183374887,548487,0
0,954493183,216176,461514,1,2023-08-10,5500.0000000,204562389,2023-07-30 14:16:36.750981+00,1183374890,548487,0
0,954493183,216176,461514,1,2023-08-11,5500.0000000,204562390,2023-07-30 14:16:36.750981+00,1183374896,548487,0
0,954493183,216176,461514,1,2023-08-12,5500.0000000,204562391,2023-07-30 14:16:36.750981+00,1183374902,548487,0
0,954493183,216176,461514,1,2023-08-13,5500.0000000,204562392,2023-07-30 14:16:36.750981+00,1183374908,548487,0
0,954493183,216176,461514,1,2023-08-14,5500.0000000,204562393,2023-07-30 14:16:36.750981+00,1183374910,548487,0
0,954493183,216176,461514,1,2023-08-15,5500.0000000,204562394,2023-07-30 14:16:36.750981+00,1183374916,548487,0
0,954493183,216176,461514,1,2023-08-16,5500.0000000,204562395,2023-07-30 14:16:36.750981+00,1183374921,548487,0
0,954493183,216176,461514,1,2023-08-17,5500.0000000,204562396,2023-07-30 14:16:36.750981+00,1183374925,548487,0
0,954493183,216176,461514,1,2023-08-18,5500.0000000,204562397,2023-07-30 14:16:36.750981+00,1183374931,548487,0
0,954493183,216176,461514,1,2023-08-19,5500.0000000,204562398,2023-07-30 14:16:36.750981+00,1183374938,548487,0
0,954493183,216176,461514,1,2023-08-20,5500.0000000,204562399,2023-07-30 14:16:36.750981+00,1183374943,548487,0
0,954493183,216176,461514,1,2023-08-21,5500.0000000,204562400,2023-07-30 14:16:36.750981+00,1183374945,548487,0
0,954493183,216176,461514,1,2023-08-22,5500.0000000,204562401,2023-07-30 14:16:36.750981+00,1183374951,548487,0
0,954493183,216176,461514,1,2023-08-23,5500.0000000,204562402,2023-07-30 14:16:36.750981+00,1183374955,548487,0
0,954493183,216176,461514,1,2023-08-24,5500.0000000,204562403,2023-07-30 14:16:36.750981+00,1183374960,548487,0
0,954493183,216176,461514,1,2023-08-25,5500.0000000,204562404,2023-07-30 14:16:36.750981+00,1183374963,548487,0
0,954493183,216176,461514,1,2023-08-26,5500.0000000,204562405,2023-07-30 14:16:36.750981+00,1183374967,548487,0
0,954493183,216176,461514,1,2023-08-27,5500.0000000,204562406,2023-07-30 14:16:36.750981+00,1183374971,548487,0
0,954493183,216176,461514,1,2023-08-28,5500.0000000,204562407,2023-07-30 14:16:36.750981+00,1183374974,548487,0
0,954493183,216176,461514,1,2023-08-29,5500.0000000,204562408,2023-07-30 14:16:36.750981+00,1183374976,548487,0
0,954493183,216176,461514,1,2023-08-30,5500.0000000,204562409,2023-07-30 14:16:36.750981+00,1183374979,548487,0
0,954493183,216176,461514,1,2023-08-31,5500.0000000,204562410,2023-07-30 14:16:36.750981+00,1183374981,548487,0
0,954493183,216176,461514,2,2023-08-01,5500.0000000,204562901,2023-07-30 14:16:36.750981+00,1183375024,548488,0
0,954493183,216176,461514,2,2023-08-02,5500.0000000,204562902,2023-07-30 14:16:36.750981+00,1183375029,548488,0
0,954493183,216176,461514,2,2023-08-03,5500.0000000,204562903,2023-07-30 14:16:36.750981+00,1183375033,548488,0
0,954493183,216176,461514,2,2023-08-04,5500.0000000,204562904,2023-07-30 14:16:36.750981+00,1183375035,548488,0
0,954493183,216176,461514,2,2023-08-05,5500.0000000,204562905,2023-07-30 14:16:36.750981+00,1183375038,548488,0
0,954493183,216176,461514,2,2023-08-06,5500.0000000,204562906,2023-07-30 14:16:36.750981+00,1183375042,548488,0
0,954493183,216176,461514,2,2023-08-07,5500.0000000,204562907,2023-07-30 14:16:36.750981+00,1183375046,548488,0
0,954493183,216176,461514,2,2023-08-08,5500.0000000,204562908,2023-07-30 14:16:36.750981+00,1183375049,548488,0
0,954493183,216176,461514,2,2023-08-09,5500.0000000,204562909,2023-07-30 14:16:36.750981+00,1183375052,548488,0
0,954493183,216176,461514,2,2023-08-10,5500.0000000,204562910,2023-07-30 14:16:36.750981+00,1183375056,548488,0
0,954493183,216176,461514,2,2023-08-11,5500.0000000,204562911,2023-07-30 14:16:36.750981+00,1183375060,548488,0
0,954493183,216176,461514,2,2023-08-12,5500.0000000,204562912,2023-07-30 14:16:36.750981+00,1183375064,548488,0
0,954493183,216176,461514,2,2023-08-13,5500.0000000,204562913,2023-07-30 14:16:36.750981+00,1183375068,548488,0
0,954493183,216176,461514,2,2023-08-14,5500.0000000,204562914,2023-07-30 14:16:36.750981+00,1183375072,548488,0
0,954493183,216176,461514,2,2023-08-15,5500.0000000,204562915,2023-07-30 14:16:36.750981+00,1183375074,548488,0
0,954493183,216176,461514,2,2023-08-16,5500.0000000,204562916,2023-07-30 14:16:36.750981+00,1183375075,548488,0
0,954493183,216176,461514,2,2023-08-17,5500.0000000,204562917,2023-07-30 14:16:36.750981+00,1183375078,548488,0
0,954493183,216176,461514,2,2023-08-18,5500.0000000,204562918,2023-07-30 14:16:36.750981+00,1183375079,548488,0
0,954493183,216176,461514,2,2023-08-19,5500.0000000,204562919,2023-07-30 14:16:36.750981+00,1183375082,548488,0
0,954493183,216176,461514,2,2023-08-20,5500.0000000,204562920,2023-07-30 14:16:36.750981+00,1183375083,548488,0
0,954493183,216176,461514,2,2023-08-21,5500.0000000,204562921,2023-07-30 14:16:36.750981+00,1183375085,548488,0
0,954493183,216176,461514,2,2023-08-22,5500.0000000,204562922,2023-07-30 14:16:36.750981+00,1183375088,548488,0
0,954493183,216176,461514,2,2023-08-23,5500.0000000,204562923,2023-07-30 14:16:36.750981+00,1183375090,548488,0
0,954493183,216176,461514,2,2023-08-24,5500.0000000,204562924,2023-07-30 14:16:36.750981+00,1183375093,548488,0
0,954493183,216176,461514,2,2023-08-25,5500.0000000,204562925,2023-07-30 14:16:36.750981+00,1183375098,548488,0
0,954493183,216176,461514,2,2023-08-26,5500.0000000,204562926,2023-07-30 14:16:36.750981+00,1183375099,548488,0
0,954493183,216176,461514,2,2023-08-27,5500.0000000,204562927,2023-07-30 14:16:36.750981+00,1183375101,548488,0
0,954493183,216176,461514,2,2023-08-28,5500.0000000,204562928,2023-07-30 14:16:36.750981+00,1183375104,548488,0
0,954493183,216176,461514,2,2023-08-29,5500.0000000,204562929,2023-07-30 14:16:36.750981+00,1183375106,548488,0
0,954493183,216176,461514,2,2023-08-30,5500.0000000,204562930,2023-07-30 14:16:36.750981+00,1183375110,548488,0
0,954493183,216176,461514,2,2023-08-31,5500.0000000,204562931,2023-07-30 14:16:36.750981+00,1183375111,548488,0
0,954493183,216312,462120,1,2023-08-01,4500.0000000,204563422,2023-07-30 14:16:36.750981+00,1183375246,548882,0
0,954493183,216312,462120,1,2023-08-02,4500.0000000,204563423,2023-07-30 14:16:36.750981+00,1183375248,548882,0
0,954493183,216312,462120,1,2023-08-03,4500.0000000,204563424,2023-07-30 14:16:36.750981+00,1183375251,548882,0
0,954493183,216312,462120,1,2023-08-04,4500.0000000,204563425,2023-07-30 14:16:36.750981+00,1183375253,548882,0
0,954493183,216312,462120,1,2023-08-05,4500.0000000,204563426,2023-07-30 14:16:36.750981+00,1183375256,548882,0
0,954493183,216312,462120,1,2023-08-06,4500.0000000,204563427,2023-07-30 14:16:36.750981+00,1183375257,548882,0
0,954493183,216312,462120,1,2023-08-07,4500.0000000,204563428,2023-07-30 14:16:36.750981+00,1183375258,548882,0
0,954493183,216312,462120,1,2023-08-08,4500.0000000,204563429,2023-07-30 14:16:36.750981+00,1183375260,548882,0
0,954493183,216312,462120,1,2023-08-09,4500.0000000,204563430,2023-07-30 14:16:36.750981+00,1183375263,548882,0
0,954493183,216312,462120,1,2023-08-10,4500.0000000,204563431,2023-07-30 14:16:36.750981+00,1183375264,548882,0
0,954493183,216312,462120,1,2023-08-11,4500.0000000,204563432,2023-07-30 14:16:36.750981+00,1183375267,548882,0
0,954493183,216312,462120,1,2023-08-12,4500.0000000,204563433,2023-07-30 14:16:36.750981+00,1183375269,548882,0
0,954493183,216312,462120,1,2023-08-13,4500.0000000,204563434,2023-07-30 14:16:36.750981+00,1183375270,548882,0
0,954493183,216312,462120,1,2023-08-14,4500.0000000,204563435,2023-07-30 14:16:36.750981+00,1183375273,548882,0
0,954493183,216312,462120,1,2023-08-15,4500.0000000,204563436,2023-07-30 14:16:36.750981+00,1183375274,548882,0
0,954493183,216312,462120,1,2023-08-16,4500.0000000,204563437,2023-07-30 14:16:36.750981+00,1183375277,548882,0
0,954493183,216312,462120,1,2023-08-17,4500.0000000,204563438,2023-07-30 14:16:36.750981+00,1183375278,548882,0
0,954493183,216312,462120,1,2023-08-18,4500.0000000,204563439,2023-07-30 14:16:36.750981+00,1183375279,548882,0
0,954493183,216312,462120,1,2023-08-19,4500.0000000,204563440,2023-07-30 14:16:36.750981+00,1183375280,548882,0
0,954493183,216312,462120,1,2023-08-20,4500.0000000,204563441,2023-07-30 14:16:36.750981+00,1183375283,548882,0
0,954493183,216312,462120,1,2023-08-21,4500.0000000,204563442,2023-07-30 14:16:36.750981+00,1183375285,548882,0
0,954493183,216312,462120,1,2023-08-22,4500.0000000,204563443,2023-07-30 14:16:36.750981+00,1183375288,548882,0
0,954493183,216312,462120,1,2023-08-23,4500.0000000,204563444,2023-07-30 14:16:36.750981+00,1183375289,548882,0
0,954493183,216312,462120,1,2023-08-24,4500.0000000,204563445,2023-07-30 14:16:36.750981+00,1183375291,548882,0
0,954493183,216312,462120,1,2023-08-25,4500.0000000,204563446,2023-07-30 14:16:36.750981+00,1183375292,548882,0
0,954493183,216312,462120,1,2023-08-26,4500.0000000,204563447,2023-07-30 14:16:36.750981+00,1183375294,548882,0
0,954493183,216312,462120,1,2023-08-27,4500.0000000,204563448,2023-07-30 14:16:36.750981+00,1183375297,548882,0
0,954493183,216312,462120,1,2023-08-28,4500.0000000,204563449,2023-07-30 14:16:36.750981+00,1183375299,548882,0
0,954493183,216312,462120,1,2023-08-29,4500.0000000,204563450,2023-07-30 14:16:36.750981+00,1183375301,548882,0
0,954493183,216312,462120,1,2023-08-30,4500.0000000,204563451,2023-07-30 14:16:36.750981+00,1183375305,548882,0
0,954493183,216312,462120,1,2023-08-31,4500.0000000,204563452,2023-07-30 14:16:36.750981+00,1183375306,548882,0
0,954493183,216312,462120,2,2023-08-01,4500.0000000,204563943,2023-07-30 14:16:36.750981+00,1183375148,548883,0
0,954493183,216312,462120,2,2023-08-02,4500.0000000,204563944,2023-07-30 14:16:36.750981+00,1183375152,548883,0
0,954493183,216312,462120,2,2023-08-03,4500.0000000,204563945,2023-07-30 14:16:36.750981+00,1183375154,548883,0
0,954493183,216312,462120,2,2023-08-04,4500.0000000,204563946,2023-07-30 14:16:36.750981+00,1183375157,548883,0
0,954493183,216312,462120,2,2023-08-05,4500.0000000,204563947,2023-07-30 14:16:36.750981+00,1183375162,548883,0
0,954493183,216312,462120,2,2023-08-06,4500.0000000,204563948,2023-07-30 14:16:36.750981+00,1183375164,548883,0
0,954493183,216312,462120,2,2023-08-07,4500.0000000,204563949,2023-07-30 14:16:36.750981+00,1183375170,548883,0
0,954493183,216312,462120,2,2023-08-08,4500.0000000,204563950,2023-07-30 14:16:36.750981+00,1183375172,548883,0
0,954493183,216312,462120,2,2023-08-09,4500.0000000,204563951,2023-07-30 14:16:36.750981+00,1183375174,548883,0
0,954493183,216312,462120,2,2023-08-10,4500.0000000,204563952,2023-07-30 14:16:36.750981+00,1183375178,548883,0
0,954493183,216312,462120,2,2023-08-11,4500.0000000,204563953,2023-07-30 14:16:36.750981+00,1183375180,548883,0
0,954493183,216312,462120,2,2023-08-12,4500.0000000,204563954,2023-07-30 14:16:36.750981+00,1183375184,548883,0
0,954493183,216312,462120,2,2023-08-13,4500.0000000,204563955,2023-07-30 14:16:36.750981+00,1183375186,548883,0
0,954493183,216312,462120,2,2023-08-14,4500.0000000,204563956,2023-07-30 14:16:36.750981+00,1183375189,548883,0
0,954493183,216312,462120,2,2023-08-15,4500.0000000,204563957,2023-07-30 14:16:36.750981+00,1183375192,548883,0
0,954493183,216312,462120,2,2023-08-16,4500.0000000,204563958,2023-07-30 14:16:36.750981+00,1183375194,548883,0
0,954493183,216312,462120,2,2023-08-17,4500.0000000,204563959,2023-07-30 14:16:36.750981+00,1183375195,548883,0
0,954493183,216312,462120,2,2023-08-18,4500.0000000,204563960,2023-07-30 14:16:36.750981+00,1183375197,548883,0
0,954493183,216312,462120,2,2023-08-19,4500.0000000,204563961,2023-07-30 14:16:36.750981+00,1183375199,548883,0
0,954493183,216312,462120,2,2023-08-20,4500.0000000,204563962,2023-07-30 14:16:36.750981+00,1183375201,548883,0
0,954493183,216312,462120,2,2023-08-21,4500.0000000,204563963,2023-07-30 14:16:36.750981+00,1183375203,548883,0
0,954493183,216312,462120,2,2023-08-22,4500.0000000,204563964,2023-07-30 14:16:36.750981+00,1183375205,548883,0
0,954493183,216312,462120,2,2023-08-23,4500.0000000,204563965,2023-07-30 14:16:36.750981+00,1183375208,548883,0
0,954493183,216312,462120,2,2023-08-24,4500.0000000,204563966,2023-07-30 14:16:36.750981+00,1183375209,548883,0
0,954493183,216312,462120,2,2023-08-25,4500.0000000,204563967,2023-07-30 14:16:36.750981+00,1183375213,548883,0
0,954493183,216312,462120,2,2023-08-26,4500.0000000,204563968,2023-07-30 14:16:36.750981+00,1183375216,548883,0
0,954493183,216312,462120,2,2023-08-27,4500.0000000,204563969,2023-07-30 14:16:36.750981+00,1183375218,548883,0
0,954493183,216312,462120,2,2023-08-28,4500.0000000,204563970,2023-07-30 14:16:36.750981+00,1183375219,548883,0
0,954493183,216312,462120,2,2023-08-29,4500.0000000,204563971,2023-07-30 14:16:36.750981+00,1183375220,548883,0
0,954493183,216312,462120,2,2023-08-30,4500.0000000,204563972,2023-07-30 14:16:36.750981+00,1183375221,548883,0
0,954493183,216312,462120,2,2023-08-31,4500.0000000,204563973,2023-07-30 14:16:36.750981+00,1183375223,548883,0
0,954493183,216439,462715,1,2023-08-01,3600.0000000,204564464,2023-07-30 14:16:36.750981+00,1183375453,549233,0
0,954493183,216439,462715,1,2023-08-02,3600.0000000,204564465,2023-07-30 14:16:36.750981+00,1183375456,549233,0
0,954493183,216439,462715,1,2023-08-03,3600.0000000,204564466,2023-07-30 14:16:36.750981+00,1183375458,549233,0
0,954493183,216439,462715,1,2023-08-04,3600.0000000,204564467,2023-07-30 14:16:36.750981+00,1183375460,549233,0
0,954493183,216439,462715,1,2023-08-05,3600.0000000,204564468,2023-07-30 14:16:36.750981+00,1183375462,549233,0
0,954493183,216439,462715,1,2023-08-06,3600.0000000,204564469,2023-07-30 14:16:36.750981+00,1183375465,549233,0
0,954493183,216439,462715,1,2023-08-07,3600.0000000,204564470,2023-07-30 14:16:36.750981+00,1183375469,549233,0
0,954493183,216439,462715,1,2023-08-08,3600.0000000,204564471,2023-07-30 14:16:36.750981+00,1183375472,549233,0
0,954493183,216439,462715,1,2023-08-09,3600.0000000,204564472,2023-07-30 14:16:36.750981+00,1183375474,549233,0
0,954493183,216439,462715,1,2023-08-10,3600.0000000,204564473,2023-07-30 14:16:36.750981+00,1183375477,549233,0
0,954493183,216439,462715,1,2023-08-11,3600.0000000,204564474,2023-07-30 14:16:36.750981+00,1183375481,549233,0
0,954493183,216439,462715,1,2023-08-12,3600.0000000,204564475,2023-07-30 14:16:36.750981+00,1183375485,549233,0
0,954493183,216439,462715,1,2023-08-13,3600.0000000,204564476,2023-07-30 14:16:36.750981+00,1183375489,549233,0
0,954493183,216439,462715,1,2023-08-14,3600.0000000,204564477,2023-07-30 14:16:36.750981+00,1183375493,549233,0
0,954493183,216439,462715,1,2023-08-15,3600.0000000,204564478,2023-07-30 14:16:36.750981+00,1183375497,549233,0
0,954493183,216439,462715,1,2023-08-16,3600.0000000,204564479,2023-07-30 14:16:36.750981+00,1183375501,549233,0
0,954493183,216439,462715,1,2023-08-17,3600.0000000,204564480,2023-07-30 14:16:36.750981+00,1183375504,549233,0
0,954493183,216439,462715,1,2023-08-18,3600.0000000,204564481,2023-07-30 14:16:36.750981+00,1183375506,549233,0
0,954493183,216439,462715,1,2023-08-19,3600.0000000,204564482,2023-07-30 14:16:36.750981+00,1183375508,549233,0
0,954493183,216439,462715,1,2023-08-20,3600.0000000,204564483,2023-07-30 14:16:36.750981+00,1183375511,549233,0
0,954493183,216439,462715,1,2023-08-21,3600.0000000,204564484,2023-07-30 14:16:36.750981+00,1183375514,549233,0
0,954493183,216439,462715,1,2023-08-22,3600.0000000,204564485,2023-07-30 14:16:36.750981+00,1183375518,549233,0
0,954493183,216439,462715,1,2023-08-23,3600.0000000,204564486,2023-07-30 14:16:36.750981+00,1183375520,549233,0
0,954493183,216439,462715,1,2023-08-24,3600.0000000,204564487,2023-07-30 14:16:36.750981+00,1183375525,549233,0
0,954493183,216439,462715,1,2023-08-25,3600.0000000,204564488,2023-07-30 14:16:36.750981+00,1183375527,549233,0
0,954493183,216439,462715,1,2023-08-26,3600.0000000,204564489,2023-07-30 14:16:36.750981+00,1183375531,549233,0
0,954493183,216439,462715,1,2023-08-27,3600.0000000,204564490,2023-07-30 14:16:36.750981+00,1183375535,549233,0
0,954493183,216439,462715,1,2023-08-28,3600.0000000,204564491,2023-07-30 14:16:36.750981+00,1183375540,549233,0
0,954493183,216439,462715,1,2023-08-29,3600.0000000,204564492,2023-07-30 14:16:36.750981+00,1183375546,549233,0
0,954493183,216439,462715,1,2023-08-30,3600.0000000,204564493,2023-07-30 14:16:36.750981+00,1183375552,549233,0
0,954493183,216439,462715,1,2023-08-31,3600.0000000,204564494,2023-07-30 14:16:36.750981+00,1183375558,549233,0
0,954493183,216439,462715,2,2023-08-01,3600.0000000,204564985,2023-07-30 14:16:36.750981+00,1183375344,549234,0
0,954493183,216439,462715,2,2023-08-02,3600.0000000,204564986,2023-07-30 14:16:36.750981+00,1183375346,549234,0
0,954493183,216439,462715,2,2023-08-03,3600.0000000,204564987,2023-07-30 14:16:36.750981+00,1183375347,549234,0
0,954493183,216439,462715,2,2023-08-04,3600.0000000,204564988,2023-07-30 14:16:36.750981+00,1183375350,549234,0
0,954493183,216439,462715,2,2023-08-05,3600.0000000,204564989,2023-07-30 14:16:36.750981+00,1183375353,549234,0
0,954493183,216439,462715,2,2023-08-06,3600.0000000,204564990,2023-07-30 14:16:36.750981+00,1183375357,549234,0
0,954493183,216439,462715,2,2023-08-07,3600.0000000,204564991,2023-07-30 14:16:36.750981+00,1183375360,549234,0
0,954493183,216439,462715,2,2023-08-08,3600.0000000,204564992,2023-07-30 14:16:36.750981+00,1183375364,549234,0
0,954493183,216439,462715,2,2023-08-09,3600.0000000,204564993,2023-07-30 14:16:36.750981+00,1183375367,549234,0
0,954493183,216439,462715,2,2023-08-10,3600.0000000,204564994,2023-07-30 14:16:36.750981+00,1183375371,549234,0
0,954493183,216439,462715,2,2023-08-11,3600.0000000,204564995,2023-07-30 14:16:36.750981+00,1183375374,549234,0
0,954493183,216439,462715,2,2023-08-12,3600.0000000,204564996,2023-07-30 14:16:36.750981+00,1183375376,549234,0
0,954493183,216439,462715,2,2023-08-13,3600.0000000,204564997,2023-07-30 14:16:36.750981+00,1183375379,549234,0
0,954493183,216439,462715,2,2023-08-14,3600.0000000,204564998,2023-07-30 14:16:36.750981+00,1183375382,549234,0
0,954493183,216439,462715,2,2023-08-15,3600.0000000,204564999,2023-07-30 14:16:36.750981+00,1183375385,549234,0
0,954493183,216439,462715,2,2023-08-16,3600.0000000,204565000,2023-07-30 14:16:36.750981+00,1183375386,549234,0
0,954493183,216439,462715,2,2023-08-17,3600.0000000,204565001,2023-07-30 14:16:36.750981+00,1183375387,549234,0
0,954493183,216439,462715,2,2023-08-18,3600.0000000,204565002,2023-07-30 14:16:36.750981+00,1183375390,549234,0
0,954493183,216439,462715,2,2023-08-19,3600.0000000,204565003,2023-07-30 14:16:36.750981+00,1183375391,549234,0
0,954493183,216439,462715,2,2023-08-20,3600.0000000,204565004,2023-07-30 14:16:36.750981+00,1183375393,549234,0
0,954493183,216439,462715,2,2023-08-21,3600.0000000,204565005,2023-07-30 14:16:36.750981+00,1183375395,549234,0
0,954493183,216439,462715,2,2023-08-22,3600.0000000,204565006,2023-07-30 14:16:36.750981+00,1183375398,549234,0
0,954493183,216439,462715,2,2023-08-23,3600.0000000,204565007,2023-07-30 14:16:36.750981+00,1183375399,549234,0
0,954493183,216439,462715,2,2023-08-24,3600.0000000,204565008,2023-07-30 14:16:36.750981+00,1183375401,549234,0
0,954493183,216439,462715,2,2023-08-25,3600.0000000,204565009,2023-07-30 14:16:36.750981+00,1183375403,549234,0
0,954493183,216439,462715,2,2023-08-26,3600.0000000,204565010,2023-07-30 14:16:36.750981+00,1183375405,549234,0
0,954493183,216439,462715,2,2023-08-27,3600.0000000,204565011,2023-07-30 14:16:36.750981+00,1183375407,549234,0
0,954493183,216439,462715,2,2023-08-28,3600.0000000,204565012,2023-07-30 14:16:36.750981+00,1183375408,549234,0
0,954493183,216439,462715,2,2023-08-29,3600.0000000,204565013,2023-07-30 14:16:36.750981+00,1183375410,549234,0
0,954493183,216439,462715,2,2023-08-30,3600.0000000,204565014,2023-07-30 14:16:36.750981+00,1183375412,549234,0
0,954493183,216439,462715,2,2023-08-31,3600.0000000,204565015,2023-07-30 14:16:36.750981+00,1183375413,549234,0
0,954493183,216441,462727,1,2023-08-01,3600.0000000,204565506,2023-07-30 14:16:36.750981+00,1183375601,549237,0
0,954493183,216441,462727,1,2023-08-02,3600.0000000,204565507,2023-07-30 14:16:36.750981+00,1183375605,549237,0
0,954493183,216441,462727,1,2023-08-03,3600.0000000,204565508,2023-07-30 14:16:36.750981+00,1183375607,549237,0
0,954493183,216441,462727,1,2023-08-04,3600.0000000,204565509,2023-07-30 14:16:36.750981+00,1183375610,549237,0
0,954493183,216441,462727,1,2023-08-05,3600.0000000,204565510,2023-07-30 14:16:36.750981+00,1183375612,549237,0
0,954493183,216441,462727,1,2023-08-06,3600.0000000,204565511,2023-07-30 14:16:36.750981+00,1183375614,549237,0
0,954493183,216441,462727,1,2023-08-07,3600.0000000,204565512,2023-07-30 14:16:36.750981+00,1183375617,549237,0
0,954493183,216441,462727,1,2023-08-08,3600.0000000,204565513,2023-07-30 14:16:36.750981+00,1183375620,549237,0
0,954493183,216441,462727,1,2023-08-09,3600.0000000,204565514,2023-07-30 14:16:36.750981+00,1183375621,549237,0
0,954493183,216441,462727,1,2023-08-10,3600.0000000,204565515,2023-07-30 14:16:36.750981+00,1183375624,549237,0
0,954493183,216441,462727,1,2023-08-11,3600.0000000,204565516,2023-07-30 14:16:36.750981+00,1183375626,549237,0
0,954493183,216441,462727,1,2023-08-12,3600.0000000,204565517,2023-07-30 14:16:36.750981+00,1183375630,549237,0
0,954493183,216441,462727,1,2023-08-13,3600.0000000,204565518,2023-07-30 14:16:36.750981+00,1183375633,549237,0
0,954493183,216441,462727,1,2023-08-14,3600.0000000,204565519,2023-07-30 14:16:36.750981+00,1183375637,549237,0
0,954493183,216441,462727,1,2023-08-15,3600.0000000,204565520,2023-07-30 14:16:36.750981+00,1183375640,549237,0
0,954493183,216441,462727,1,2023-08-16,3600.0000000,204565521,2023-07-30 14:16:36.750981+00,1183375643,549237,0
0,954493183,216441,462727,1,2023-08-17,3600.0000000,204565522,2023-07-30 14:16:36.750981+00,1183375646,549237,0
0,954493183,216441,462727,1,2023-08-18,3600.0000000,204565523,2023-07-30 14:16:36.750981+00,1183375650,549237,0
0,954493183,216441,462727,1,2023-08-19,3600.0000000,204565524,2023-07-30 14:16:36.750981+00,1183375653,549237,0
0,954493183,216441,462727,1,2023-08-20,3600.0000000,204565525,2023-07-30 14:16:36.750981+00,1183375657,549237,0
0,954493183,216441,462727,1,2023-08-21,3600.0000000,204565526,2023-07-30 14:16:36.750981+00,1183375661,549237,0
0,954493183,216441,462727,1,2023-08-22,3600.0000000,204565527,2023-07-30 14:16:36.750981+00,1183375664,549237,0
0,954493183,216441,462727,1,2023-08-23,3600.0000000,204565528,2023-07-30 14:16:36.750981+00,1183375666,549237,0
0,954493183,216441,462727,1,2023-08-24,3600.0000000,204565529,2023-07-30 14:16:36.750981+00,1183375671,549237,0
0,954493183,216441,462727,1,2023-08-25,3600.0000000,204565530,2023-07-30 14:16:36.750981+00,1183375678,549237,0
0,954493183,216441,462727,1,2023-08-26,3600.0000000,204565531,2023-07-30 14:16:36.750981+00,1183375680,549237,0
0,954493183,216441,462727,1,2023-08-27,3600.0000000,204565532,2023-07-30 14:16:36.750981+00,1183375684,549237,0
0,954493183,216441,462727,1,2023-08-28,3600.0000000,204565533,2023-07-30 14:16:36.750981+00,1183375687,549237,0
0,954493183,216441,462727,1,2023-08-29,3600.0000000,204565534,2023-07-30 14:16:36.750981+00,1183375692,549237,0
0,954493183,216441,462727,1,2023-08-30,3600.0000000,204565535,2023-07-30 14:16:36.750981+00,1183375693,549237,0
0,954493183,216441,462727,1,2023-08-31,3600.0000000,204565536,2023-07-30 14:16:36.750981+00,1183375697,549237,0
0,954493183,216441,462727,2,2023-08-01,3600.0000000,204566027,2023-07-30 14:16:36.750981+00,1183375734,549238,0
0,954493183,216441,462727,2,2023-08-02,3600.0000000,204566028,2023-07-30 14:16:36.750981+00,1183375737,549238,0
0,954493183,216441,462727,2,2023-08-03,3600.0000000,204566029,2023-07-30 14:16:36.750981+00,1183375741,549238,0
0,954493183,216441,462727,2,2023-08-04,3600.0000000,204566030,2023-07-30 14:16:36.750981+00,1183375744,549238,0
0,954493183,216441,462727,2,2023-08-05,3600.0000000,204566031,2023-07-30 14:16:36.750981+00,1183375746,549238,0
0,954493183,216441,462727,2,2023-08-06,3600.0000000,204566032,2023-07-30 14:16:36.750981+00,1183375749,549238,0
0,954493183,216441,462727,2,2023-08-07,3600.0000000,204566033,2023-07-30 14:16:36.750981+00,1183375752,549238,0

Hi,

On 2023-09-25 15:42:26 -0400, Tom Lane wrote:
> I just did a git bisect run to discover when the failure documented
> in bug #18130 [1] started.  And the answer is commit 82a4edabd.
> Now, it's pretty obvious that that commit didn't in itself cause
> problems like this:
> 
> postgres=# \copy test from 'bug18130.csv' csv
> ERROR:  could not read block 5 in file "base/5/17005": read only 0 of 8192 bytes
> CONTEXT:  COPY test, line 472: "0,185647715,222655,489637,2,2023-07-31,9100.0000000,302110385,2023-07-30
14:16:36.750981+00,14026347..."

Ugh.


> IMO there must be some very nasty bug lurking in the new
> multiple-block extension logic, that happens to be exposed by this
> test case with 82a4edabd's adjustments to the when-to-extend choices
> but wasn't before that.

> To save other people the trouble of extracting the in-line data
> in the bug submission, I've attached the test files I was using.

Thanks, looking at this now.


> The DDL is simplified slightly from what was submitted.  I'm not
> entirely sure why a no-op trigger is needed to provoke the bug...

A trigger might prevent using the multi-insert API, which would lead to
different execution paths...


Greetings,

Andres Freund

Hi,

On 2023-09-25 12:48:30 -0700, Andres Freund wrote:
> On 2023-09-25 15:42:26 -0400, Tom Lane wrote:
> > I just did a git bisect run to discover when the failure documented
> > in bug #18130 [1] started.  And the answer is commit 82a4edabd.
> > Now, it's pretty obvious that that commit didn't in itself cause
> > problems like this:
> > 
> > postgres=# \copy test from 'bug18130.csv' csv
> > ERROR:  could not read block 5 in file "base/5/17005": read only 0 of 8192 bytes
> > CONTEXT:  COPY test, line 472: "0,185647715,222655,489637,2,2023-07-31,9100.0000000,302110385,2023-07-30
14:16:36.750981+00,14026347..."
> 
> Ugh.
> 
> 
> > IMO there must be some very nasty bug lurking in the new
> > multiple-block extension logic, that happens to be exposed by this
> > test case with 82a4edabd's adjustments to the when-to-extend choices
> > but wasn't before that.
> 
> > To save other people the trouble of extracting the in-line data
> > in the bug submission, I've attached the test files I was using.
> 
> Thanks, looking at this now.

(had to switch locations in between)

Uh, huh.  The problem is that COPY uses a single BulkInsertState for multiple
partitions. Which to me seems to run counter to the following comment:
 *    The caller can also provide a BulkInsertState object to optimize many
 *    insertions into the same relation.  This keeps a pin on the current
 *    insertion target page (to save pin/unpin cycles) and also passes a
 *    BULKWRITE buffer selection strategy object to the buffer manager.
 *    Passing NULL for bistate selects the default behavior.

The reason this doesn't cause straight up corruption due to reusing a pin from
another relation is that b1ecb9b3fcfb added ReleaseBulkInsertStatePin() and a
call to it. But I didn't make ReleaseBulkInsertStatePin() reset the bulk
insertion state, which is what leads to the errors from the bug report.

Resetting the relevant BulkInsertState fields fixes the problem. But I'm not
sure that's the right fix. ISTM that independent of whether we fix this via
ReleaseBulkInsertStatePin() resetting the fields or via not reusing
BulkInsertState, we should add assertions defending against future issues like
this (e.g. by adding a relation field to BulkInsertState in cassert builds,
and asserting that the relation is the same as in prior calls unless
ReleaseBulkInsertStatePin() has been called).

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
>> On 2023-09-25 15:42:26 -0400, Tom Lane wrote:
>>> I just did a git bisect run to discover when the failure documented
>>> in bug #18130 [1] started.  And the answer is commit 82a4edabd.

> Uh, huh.  The problem is that COPY uses a single BulkInsertState for multiple
> partitions. Which to me seems to run counter to the following comment:
>  *    The caller can also provide a BulkInsertState object to optimize many
>  *    insertions into the same relation.  This keeps a pin on the current
>  *    insertion target page (to save pin/unpin cycles) and also passes a
>  *    BULKWRITE buffer selection strategy object to the buffer manager.
>  *    Passing NULL for bistate selects the default behavior.

> The reason this doesn't cause straight up corruption due to reusing a pin from
> another relation is that b1ecb9b3fcfb added ReleaseBulkInsertStatePin() and a
> call to it. But I didn't make ReleaseBulkInsertStatePin() reset the bulk
> insertion state, which is what leads to the errors from the bug report.

> Resetting the relevant BulkInsertState fields fixes the problem. But I'm not
> sure that's the right fix. ISTM that independent of whether we fix this via
> ReleaseBulkInsertStatePin() resetting the fields or via not reusing
> BulkInsertState, we should add assertions defending against future issues like
> this (e.g. by adding a relation field to BulkInsertState in cassert builds,
> and asserting that the relation is the same as in prior calls unless
> ReleaseBulkInsertStatePin() has been called).

Ping?  We really ought to have a fix for this committed in time for
16.1.

            regards, tom lane

Hi,

On 2023-10-12 11:44:09 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> >> On 2023-09-25 15:42:26 -0400, Tom Lane wrote:
> >>> I just did a git bisect run to discover when the failure documented
> >>> in bug #18130 [1] started.  And the answer is commit 82a4edabd.
> 
> > Uh, huh.  The problem is that COPY uses a single BulkInsertState for multiple
> > partitions. Which to me seems to run counter to the following comment:
> >  *    The caller can also provide a BulkInsertState object to optimize many
> >  *    insertions into the same relation.  This keeps a pin on the current
> >  *    insertion target page (to save pin/unpin cycles) and also passes a
> >  *    BULKWRITE buffer selection strategy object to the buffer manager.
> >  *    Passing NULL for bistate selects the default behavior.
> 
> > The reason this doesn't cause straight up corruption due to reusing a pin from
> > another relation is that b1ecb9b3fcfb added ReleaseBulkInsertStatePin() and a
> > call to it. But I didn't make ReleaseBulkInsertStatePin() reset the bulk
> > insertion state, which is what leads to the errors from the bug report.
> 
> > Resetting the relevant BulkInsertState fields fixes the problem. But I'm not
> > sure that's the right fix. ISTM that independent of whether we fix this via
> > ReleaseBulkInsertStatePin() resetting the fields or via not reusing
> > BulkInsertState, we should add assertions defending against future issues like
> > this (e.g. by adding a relation field to BulkInsertState in cassert builds,
> > and asserting that the relation is the same as in prior calls unless
> > ReleaseBulkInsertStatePin() has been called).
> 
> Ping?  We really ought to have a fix for this committed in time for
> 16.1.

I kind of had hoped somebody would comment on the approach.  Given that nobody
has, I'll push the minimal fix of resetting the state in
ReleaseBulkInsertStatePin(), even though I think architecturally that's not
great.

Greetings,

Andres Freund

Hi,

On 2023-10-12 09:24:19 -0700, Andres Freund wrote:
> On 2023-10-12 11:44:09 -0400, Tom Lane wrote:
> > Andres Freund <andres@anarazel.de> writes:
> > >> On 2023-09-25 15:42:26 -0400, Tom Lane wrote:
> > >>> I just did a git bisect run to discover when the failure documented
> > >>> in bug #18130 [1] started.  And the answer is commit 82a4edabd.
> > 
> > > Uh, huh.  The problem is that COPY uses a single BulkInsertState for multiple
> > > partitions. Which to me seems to run counter to the following comment:
> > >  *    The caller can also provide a BulkInsertState object to optimize many
> > >  *    insertions into the same relation.  This keeps a pin on the current
> > >  *    insertion target page (to save pin/unpin cycles) and also passes a
> > >  *    BULKWRITE buffer selection strategy object to the buffer manager.
> > >  *    Passing NULL for bistate selects the default behavior.
> > 
> > > The reason this doesn't cause straight up corruption due to reusing a pin from
> > > another relation is that b1ecb9b3fcfb added ReleaseBulkInsertStatePin() and a
> > > call to it. But I didn't make ReleaseBulkInsertStatePin() reset the bulk
> > > insertion state, which is what leads to the errors from the bug report.
> > 
> > > Resetting the relevant BulkInsertState fields fixes the problem. But I'm not
> > > sure that's the right fix. ISTM that independent of whether we fix this via
> > > ReleaseBulkInsertStatePin() resetting the fields or via not reusing
> > > BulkInsertState, we should add assertions defending against future issues like
> > > this (e.g. by adding a relation field to BulkInsertState in cassert builds,
> > > and asserting that the relation is the same as in prior calls unless
> > > ReleaseBulkInsertStatePin() has been called).
> > 
> > Ping?  We really ought to have a fix for this committed in time for
> > 16.1.
> 
> I kind of had hoped somebody would comment on the approach.  Given that nobody
> has, I'll push the minimal fix of resetting the state in
> ReleaseBulkInsertStatePin(), even though I think architecturally that's not
> great.

I spent some time working on a test that shows the problem more cheaply than
the case upthread. I think it'd be desirable to have a test that's likely to
catch an issue like this fairly quickly. We've had other problems in this
realm before - there's only a single test that fails if I remove the
ReleaseBulkInsertStatePin() call, and I don't think that's guaranteed at all.

I'm a bit on the fence on how large to make the relation. For me the bug
triggers when filling both relations up to the 3rd block, but how many rows
that takes is somewhat dependent on space utilization on the page and stuff.

Right now the test uses data/desc.data and ends up with 328kB and 312kB in two
partitions. Alternatively I could make the test create a new file to load with
copy that has fewer rows than data/desc.data - I didn't see another data file
that works conveniently and has fewer rows.  The copy is reasonably fast, even
under something as expensive as rr (~60ms). So I'm inclined to just go with
that?

Greetings,

Andres Freund

Hi,

On 2023-10-13 10:39:10 -0700, Andres Freund wrote:
> On 2023-10-12 09:24:19 -0700, Andres Freund wrote:
> > I kind of had hoped somebody would comment on the approach.  Given that nobody
> > has, I'll push the minimal fix of resetting the state in
> > ReleaseBulkInsertStatePin(), even though I think architecturally that's not
> > great.
> 
> I spent some time working on a test that shows the problem more cheaply than
> the case upthread. I think it'd be desirable to have a test that's likely to
> catch an issue like this fairly quickly. We've had other problems in this
> realm before - there's only a single test that fails if I remove the
> ReleaseBulkInsertStatePin() call, and I don't think that's guaranteed at all.
> 
> I'm a bit on the fence on how large to make the relation. For me the bug
> triggers when filling both relations up to the 3rd block, but how many rows
> that takes is somewhat dependent on space utilization on the page and stuff.
> 
> Right now the test uses data/desc.data and ends up with 328kB and 312kB in two
> partitions. Alternatively I could make the test create a new file to load with
> copy that has fewer rows than data/desc.data - I didn't see another data file
> that works conveniently and has fewer rows.  The copy is reasonably fast, even
> under something as expensive as rr (~60ms). So I'm inclined to just go with
> that?

Patch with fix and test attached (0001).

Given how easy a missing ReleaseBulkInsertStatePin() can cause corruption (not
due to this bug, but the issue fixed in b1ecb9b3fcf), I think we should
consider adding an assertion along the lines of 0002 to HEAD. Perhaps adding a
new bufmgr.c function to avoid having to get the fields in the buffer tag we
don't care about. Or perhaps we should just promote the check to an elog, we
already call BufferGetBlockNumber(), using BufferGetTag() instead doesn't cost
much more.

Greetings,

Andres Freund

Hi,

On 2023-10-13 11:30:35 -0700, Andres Freund wrote:
> On 2023-10-13 10:39:10 -0700, Andres Freund wrote:
> > On 2023-10-12 09:24:19 -0700, Andres Freund wrote:
> > > I kind of had hoped somebody would comment on the approach.  Given that nobody
> > > has, I'll push the minimal fix of resetting the state in
> > > ReleaseBulkInsertStatePin(), even though I think architecturally that's not
> > > great.
> > 
> > I spent some time working on a test that shows the problem more cheaply than
> > the case upthread. I think it'd be desirable to have a test that's likely to
> > catch an issue like this fairly quickly. We've had other problems in this
> > realm before - there's only a single test that fails if I remove the
> > ReleaseBulkInsertStatePin() call, and I don't think that's guaranteed at all.
> > 
> > I'm a bit on the fence on how large to make the relation. For me the bug
> > triggers when filling both relations up to the 3rd block, but how many rows
> > that takes is somewhat dependent on space utilization on the page and stuff.
> > 
> > Right now the test uses data/desc.data and ends up with 328kB and 312kB in two
> > partitions. Alternatively I could make the test create a new file to load with
> > copy that has fewer rows than data/desc.data - I didn't see another data file
> > that works conveniently and has fewer rows.  The copy is reasonably fast, even
> > under something as expensive as rr (~60ms). So I'm inclined to just go with
> > that?
> 
> Patch with fix and test attached (0001).

Pushed that.

Greetings,

Andres