Thread: Infinite Cache
All:
--
Anthony Presley
Was curious if there was some sort of Open Source version of Infinite Cache, and/or a memcache layer that can be "dropped" in front of PostgreSQL without application changes (which seems to be the "key" piece of Infinite Cache), or is this something that EnterpriseDB owns and you have to buy their version of the software to use?
I'm fine with piecing together a few different OS projects, but would prefer to not modify the app too much.
Thanks!
--
Anthony Presley
On 07/01/2011 10:43 AM, Anthony Presley wrote: > Was curious if there was some sort of Open Source version of Infinite > Cache, and/or a memcache layer that can be "dropped" in front of > PostgreSQL without application changes (which seems to be the "key" > piece of Infinite Cache), or is this something that EnterpriseDB owns > and you have to buy their version of the software to use? The best solution available for this class of problem is pgmemcache: http://pgfoundry.org/projects/pgmemcache/ There's not too much documentation about it around, but you'll find an intro talk at http://projects.2ndquadrant.com/char10 I found helpful when Hannu presented it. It does take some work to utilize, including application code changes. The hardest part of which is usually making sure the key hashing scheme it uses to identify re-usable queries is useful to you. And that isn't always the case. This approach scales better than "Infinite Cache" because you can move the whole mess onto another server optimized to be a caching system. Those systems have a very different set of trade-offs and correspondingly economics than a database server must have. The cache system can be a cheap box with a bunch of RAM, that's it. And the read traffic it avoids passing to the server really doesn't touch the database at all, which is way better than going to the database but being serviced quickly. Everyone would prefer performance improvements that don't involve any modification of their application. The unfortunate reality of database design is that any server tuning can only provide a modest gain; if you make things twice as fast you've done a great job. Whereas when doing application redesign for better performance, I aim for a 10X speedup and often do much better than that. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD Comprehensive and Customized PostgreSQL Training Classes: http://www.2ndquadrant.us/postgresql-training/
On Jul 1, 2011, at 9:43 AM, Anthony Presley wrote: > Was curious if there was some sort of Open Source version of Infinite Cache, and/or a memcache layer that can be "dropped"in front of PostgreSQL without application changes (which seems to be the "key" piece of Infinite Cache), or isthis something that EnterpriseDB owns and you have to buy their version of the software to use? There had been some talk at one point about getting the backend-changes to support Infinite Cache into mainline Postgres.If that ever happened you could build your own version of it. BTW, thanks to the compression feature of IC I've heard it can actually be beneficial to run it on the same server. -- Jim C. Nasby, Database Architect jim@nasby.net 512.569.9461 (cell) http://jim.nasby.net
On Fri, Jul 1, 2011 at 11:37 PM, Jim Nasby <jim@nasby.net> wrote: > On Jul 1, 2011, at 9:43 AM, Anthony Presley wrote: >> Was curious if there was some sort of Open Source version of Infinite Cache, and/or a memcache layer that can be "dropped"in front of PostgreSQL without application changes (which seems to be the "key" piece of Infinite Cache), or isthis something that EnterpriseDB owns and you have to buy their version of the software to use? > > There had been some talk at one point about getting the backend-changes to support Infinite Cache into mainline Postgres.If that ever happened you could build your own version of it. > > BTW, thanks to the compression feature of IC I've heard it can actually be beneficial to run it on the same server. Correct. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Sat, Jul 2, 2011 at 00:37, Jim Nasby <jim@nasby.net> wrote: > On Jul 1, 2011, at 9:43 AM, Anthony Presley wrote: >> Was curious if there was some sort of Open Source version of Infinite Cache, and/or a memcache layer that can be "dropped"in front of PostgreSQL without application changes (which seems to be the "key" piece of Infinite Cache), or isthis something that EnterpriseDB owns and you have to buy their version of the software to use? > > There had been some talk at one point about getting the backend-changes to support Infinite Cache into mainline Postgres.If that ever happened you could build your own version of it. I was at one point told that *all* of infinite cache would be submitted to the community, but it was in need of some cleanup first. But by now I think that decision has been changed - I certainly hope it didn't take years to clean up ;) So I wouldn't hold my breath for that one. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
On 07/01/2011 06:37 PM, Jim Nasby wrote: > BTW, thanks to the compression feature of IC I've heard it can > actually be beneficial to run it on the same server. Sure, its implementation works in a way that helps improve performance on the database server. My point was that I'd be shocked if it were even possible to double performance if you use it. Whereas putting a pgmemcache server in front of the database can do much better than that, on a system that reads the same things many times per update. "Infinite Cache" is a useful technology and the fact that it work transparently the application is a nice benefit of EDB's commercial product. But it's usually the case that if you really want to do the best possible implementation of an approach, optimizing very specifically for your application is what's needed. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD Comprehensive and Customized PostgreSQL Training Classes: http://www.2ndquadrant.us/postgresql-training/
On 07/03/2011 06:21 AM, Magnus Hagander wrote: > I was at one point told that *all* of infinite cache would be > submitted to the community, but it was in need of some cleanup > first. I'm not sure what kind of cleanup would be involved, but we had some problems with index corruption that wasn't fixed until a February patch was applied. My guess is that earlier versions of Infinite Cache weren't all that careful with verifying certain edge cases during connection renegotiation or timeout scenarios. It only seemed to pop up once every few billion queries, but that's all it takes to bring down a heavy OLTP system. I'd say it's probably safe enough these days. But it's also one of those exclusive selling points they're using right now to garner EDB customers. So I doubt it'll be released any time *soon*, though may make it eventually. -- Shaun Thomas OptionsHouse | 141 W. Jackson Blvd. | Suite 800 | Chicago IL, 60604 312-676-8870 sthomas@peak6.com ______________________________________________ See http://www.peak6.com/email_disclaimer.php for terms and conditions related to this email
On 05.07.2011 16:35, Shaun Thomas wrote: > I'd say it's probably safe enough these days. But it's also one of those > exclusive selling points they're using right now to garner EDB > customers. So I doubt it'll be released any time *soon*, though may make > it eventually. I doubt the community would want it even if it was open sourced. As an open source project, what would probably make more sense is a similar caching mechanism built into the kernel, somewhere between the filesystem cache and user-space. That way any you could use it with any application that benefits from the kind of large cache that Infinite Cache provides. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On Fri, Jul 8, 2011 at 15:34, Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> wrote: > On 05.07.2011 16:35, Shaun Thomas wrote: >> >> I'd say it's probably safe enough these days. But it's also one of those >> exclusive selling points they're using right now to garner EDB >> customers. So I doubt it'll be released any time *soon*, though may make >> it eventually. > > I doubt the community would want it even if it was open sourced. As an open > source project, what would probably make more sense is a similar caching > mechanism built into the kernel, somewhere between the filesystem cache and > user-space. That way any you could use it with any application that benefits > from the kind of large cache that Infinite Cache provides. Don't underestimate how much easier it is to get something into an environment as part of the database than as an extra module or app that you have to patch your kernel with. Unless you can get it all the way into the baseline kernel of course, but that's not going to be easy... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/