Re: Is it possible to have a "fast-write" Index? - Mailing list pgsql-hackers

From deavid
Subject Re: Is it possible to have a "fast-write" Index?
Date
Msg-id CAFR-75s9EWP9DOiZ_pE6=kaMwDQAN+k-p1StR+Qn2XB9z4N7Xg@mail.gmail.com
Whole thread Raw
In response to Re: Is it possible to have a "fast-write" Index?  (Gavin Flower <GavinFlower@archidevsys.co.nz>)
Responses Re: Is it possible to have a "fast-write" Index?  ("ktm@rice.edu" <ktm@rice.edu>)
Re: Is it possible to have a "fast-write" Index?  (Jim Nasby <Jim.Nasby@BlueTreble.com>)
List pgsql-hackers
Thanks to everybody for answering. I wasn't expecting this attention; this is a great community :-)

Jim asked me about something real. Well, the problem is this showed up more than five years ago, and keeps popping from time to time since in different circumstances. I solved them in different ways each time, depending the exact use-case. I wanted to generalize, because seems a good feature for several situations; and I don't expect a solution for me as each time I hit with this I found some way to sort it out.
As Jim said, we need here are figures for real examples, and i don't have yet. I'll do my "homework" and email back with exact problems with exact timing. Give me a week or two. 

Also, some of you are talking about IO. Well, it's hard to say without the figures here, but I'm pretty sure I'm hitting CPU time only. We use SSD on those big databases, and also in my tests i tried setting fsync=off.

So the problem is: i see a low iowait, and CPU time for one core is at 80-90% most of the time. I can buy more ram, better disks, or cpu's with more cores. But one cpu core would have more-or-less the same speed no matter how much money you invest.

When someone wants a delayed-write index is similar to setting      "synchronous_commit = off". We want to give an OK to the backend as soon as is possible and do this work in background. But we also want some reliability against crashes.

Also, if the task is done in background it may be done from other backend, so probably several indexes could be packed at once using different backend processes. We could use the entire cpu if our index writes aren't tied to the session who wrote the row.

PD: I'm very interested on existent approaches like GIN or BRIN (this one is new to me). Thanks a lot; i'll try them in my tests. 

pgsql-hackers by date:

Previous
From: Gavin Flower
Date:
Subject: Re: Is it possible to have a "fast-write" Index?
Next
From: Alvaro Herrera
Date:
Subject: Re: could not truncate directory "pg_subtrans": apparent wraparound