Thread: Compression Library and Usages
Hi, The topic came on IRC and it might be that the later attempts at using another library missed one of the offering, namely FastLZ. It's made for being quick rather than minimize size, it's MIT licenced, 551 lines of portable ansi-C code, already tested on a host of systems and compilers. http://www.fastlz.org/ http://www.fastlz.org/compilers.htm http://www.fastlz.org/benchmarks.htm http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c Example call sites: http://code.google.com/p/fastlz/source/browse/trunk/6pack.c http://code.google.com/p/fastlz/source/browse/trunk/6unpack.c Maybe the only drawback is that it does not seem to be well maintained any more, so it could mean the code would have to be considered for inclusion into the PostgreSQL tree. Still I though I'd mention the lib. As far as the usage part is concerned, not only TOAST could benefit, but some IRC users are talking about protocol level compression. Should this one be fast enough and an option on the other grounds, how open to libpq compression are we? Regards, -- dim
Dimitri Fontaine <dfontaine@hi-media.com> wrote: > The topic came on IRC and it might be that the later attempts at > using another library missed one of the offering, namely FastLZ. > It's made for being quick rather than minimize size, it's MIT > licenced, 551 lines of portable ansi-C code, already tested on a > host of systems and compilers. Are the concerns from previous discussions off-base? http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php -Kevin
"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes: > Are the concerns from previous discussions off-base? > > http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php I knew I was forgetting about something, thanks for the reminder. Damn it, patents. -- dim
Dimitri Fontaine <dfontaine@hi-media.com> wrote: >> http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php > > I knew I was forgetting about something, thanks for the reminder. > Damn it, patents. I'm not sure that there was anything there which absolutely ruled out using LZ0; all of the patents still in effect seemed to be around picky little details of *how* you use it. Others seem to have picked their way through that particular mine field. -Kevin
On Mon, Jan 11, 2010 at 5:25 PM, Kevin Grittner <Kevin.Grittner@wicourts.gov> wrote: > Dimitri Fontaine <dfontaine@hi-media.com> wrote: > >>> > http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php >> >> I knew I was forgetting about something, thanks for the reminder. >> Damn it, patents. > > I'm not sure that there was anything there which absolutely ruled > out using LZ0; all of the patents still in effect seemed to be > around picky little details of *how* you use it. Others seem to > have picked their way through that particular mine field. plus, it looks like that most of the patents have either expired, or are about to expire. lzo is used all over the place, including the linux kernel...i think the burden of proof rests with anyone claiming there are patent problems, not the other way around. lzo is also gpl so we can't use it :D. regarding fastlz and patents, who knows? I'm curious...does anyone know of a case where a high profile open source project was found to be violating a patent? merlin
Merlin Moncure <mmoncure@gmail.com> writes:
> plus, it looks like that most of the patents have either expired, or
> are about to expire. lzo is used all over the place, including the
> linux kernel...i think the burden of proof rests with anyone claiming
> there are patent problems, not the other way around. lzo is also gpl
> so we can't use it :D. regarding fastlz and patents, who knows? I'm
> curious...does anyone know of a case where a high profile open source
> project was found to be violating a patent?
You have got that 100% backwards. We are not going to bet the survival
of the Postgres project on whether we can get away with violating
somebody's patent.
regards, tom lane
On Mon, 2010-01-11 at 19:54 -0500, Tom Lane wrote: > Merlin Moncure <mmoncure@gmail.com> writes: > > plus, it looks like that most of the patents have either expired, or > > are about to expire. lzo is used all over the place, including the > > linux kernel...i think the burden of proof rests with anyone claiming > > there are patent problems, not the other way around. lzo is also gpl > > so we can't use it :D. regarding fastlz and patents, who knows? I'm > > curious...does anyone know of a case where a high profile open source > > project was found to be violating a patent? > > You have got that 100% backwards. We are not going to bet the survival > of the Postgres project on whether we can get away with violating > somebody's patent. +1 Joshua D. Drake > > regards, tom lane > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Mon, Jan 11, 2010 at 7:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Merlin Moncure <mmoncure@gmail.com> writes: >> plus, it looks like that most of the patents have either expired, or >> are about to expire. lzo is used all over the place, including the >> linux kernel...i think the burden of proof rests with anyone claiming >> there are patent problems, not the other way around. lzo is also gpl >> so we can't use it :D. regarding fastlz and patents, who knows? I'm >> curious...does anyone know of a case where a high profile open source >> project was found to be violating a patent? > > You have got that 100% backwards. We are not going to bet the survival > of the Postgres project on whether we can get away with violating > somebody's patent. I was only talking about the specific case of lzo, which is used absolutely everywhere (not that this means anything but...). merlin
On Mon, 2010-01-11 at 19:54 -0500, Tom Lane wrote: > Merlin Moncure <mmoncure@gmail.com> writes: > > plus, it looks like that most of the patents have either expired, or > > are about to expire. lzo is used all over the place, including the > > linux kernel...i think the burden of proof rests with anyone claiming > > there are patent problems, not the other way around. lzo is also gpl > > so we can't use it :D. regarding fastlz and patents, who knows? I'm > > curious...does anyone know of a case where a high profile open source > > project was found to be violating a patent? > > You have got that 100% backwards. We are not going to bet the survival > of the Postgres project on whether we can get away with violating > somebody's patent. +1 Joshua D. Drake > > regards, tom lane > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On mån, 2010-01-11 at 22:39 -0500, Merlin Moncure wrote: > On Mon, Jan 11, 2010 at 7:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Merlin Moncure <mmoncure@gmail.com> writes: > >> plus, it looks like that most of the patents have either expired, or > >> are about to expire. lzo is used all over the place, including the > >> linux kernel...i think the burden of proof rests with anyone claiming > >> there are patent problems, not the other way around. lzo is also gpl > >> so we can't use it :D. regarding fastlz and patents, who knows? I'm > >> curious...does anyone know of a case where a high profile open source > >> project was found to be violating a patent? > > > > You have got that 100% backwards. We are not going to bet the survival > > of the Postgres project on whether we can get away with violating > > somebody's patent. > > I was only talking about the specific case of lzo, which is used > absolutely everywhere (not that this means anything but...). I think this compression business warrants a FAQ entry by now: N a) Why don't you use compression library X? Many compression libraries have an unclear patent situation, and the potential benefits using them do not appear to outweigh that risk. N b) But LZO cannot possibly have any relevant patents on it because a lot of open-source software uses it. The LZO library is licensed under the GPL, so we can't use it.