Thread: Postgresql, PSN hack and table limits
Somebody is making a very specific claim that Postgres can support a limited number of rows: "INPS (a data forensics team) said that there is 7 main Databases all hosted at different data centers but linked over a type of 'cloud' Each database uses PostGRESSQL which would mean the most amount of data each database could hold with no stability issues is aproximitely equal to that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh Does anybody have any idea where they've got hold of this figure? -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
On Sun, May 1, 2011 at 07:50, Mark Morgan Lloyd <markMLl.pgsql-general@telemetry.co.uk> wrote: > Somebody is making a very specific claim that Postgres can support a limited > number of rows: > > "INPS (a data forensics team) said that there is 7 main Databases all hosted > at different data centers but linked over a type of 'cloud' Each database > uses PostGRESSQL which would mean the most amount of data each database > could hold with no stability issues is aproximitely equal to that of > 10,348,439 Rows" http://pastebin.com/MtX1MDdh > > Does anybody have any idea where they've got hold of this figure? PostgreSQL, of course, has no such ridiculous limits. Whether a specific application running on top of PostgreSQL would have a limitation like that, is of course a different question - that might certainly be possible, even though the limit mentioned is a really weird number. I find it really hard to parse the text of that post to even understand what they mean, but it's rather obviously filled with other completely incorrect technical statements, so I wouldn't pay any attention to this one in particular. (e.g. since when did you need port 25 to download an email attachment? and suddenly they mention oracle metasploits, which obviously wouldn't work on postgres) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
On 04/30/11 10:50 PM, Mark Morgan Lloyd wrote: > Somebody is making a very specific claim that Postgres can support a > limited number of rows: > > "INPS (a data forensics team) said that there is 7 main Databases all > hosted at different data centers but linked over a type of 'cloud' > Each database uses PostGRESSQL which would mean the most amount of > data each database could hold with no stability issues is aproximitely > equal to that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh > > Does anybody have any idea where they've got hold of this figure? > a pastebin is the 'source' for this rant full of random WTF ?
Magnus Hagander wrote: > On Sun, May 1, 2011 at 07:50, Mark Morgan Lloyd > <markMLl.pgsql-general@telemetry.co.uk> wrote: >> Somebody is making a very specific claim that Postgres can support a limited >> number of rows: >> >> "INPS (a data forensics team) said that there is 7 main Databases all hosted >> at different data centers but linked over a type of 'cloud' Each database >> uses PostGRESSQL which would mean the most amount of data each database >> could hold with no stability issues is aproximitely equal to that of >> 10,348,439 Rows" http://pastebin.com/MtX1MDdh >> >> Does anybody have any idea where they've got hold of this figure? > > PostgreSQL, of course, has no such ridiculous limits. > > Whether a specific application running on top of PostgreSQL would have > a limitation like that, is of course a different question - that might > certainly be possible, even though the limit mentioned is a really > weird number. > > I find it really hard to parse the text of that post to even > understand what they mean, but it's rather obviously filled with other > completely incorrect technical statements, so I wouldn't pay any > attention to this one in particular. (e.g. since when did you need > port 25 to download an email attachment? and suddenly they mention > oracle metasploits, which obviously wouldn't work on postgres) I'm not paying attention because of the amount of rubbish in that text, but other people are likely to if Google makes the connection between the current unpleasantness and this project- which is why I'm avoiding mentioning certain names. I don't know for certain who these INPS people are or why they think they're qualified to pontificate. The Oracle angle could be related to older SQL injection attacks (2008?) or somebody could be assuming that a compatibility layer introduces vulnerabilities. -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
On 2011-05-01, Mark Morgan Lloyd <markMLl.pgsql-general@telemetry.co.uk> wrote: > Somebody is making a very specific claim that Postgres can support a > limited number of rows: > > "INPS (a data forensics team) said that there is 7 main Databases all > hosted at different data centers but linked over a type of 'cloud' Each > database uses PostGRESSQL which would mean the most amount of data each > database could hold with no stability issues is aproximitely equal to > that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh > > Does anybody have any idea where they've got hold of this figure? the figure is within 1% of the maximun size for data stored in text (or bytea) column. It doesn't seem close to any other PG limit that I know of. -- ⚂⚃ 100% natural
Jasen Betts <jasen@xnet.co.nz> writes: > On 2011-05-01, Mark Morgan Lloyd <markMLl.pgsql-general@telemetry.co.uk> wrote: >> Somebody is making a very specific claim that Postgres can support a >> limited number of rows: >> >> "INPS (a data forensics team) said that there is 7 main Databases all >> hosted at different data centers but linked over a type of 'cloud' Each >> database uses PostGRESSQL which would mean the most amount of data each >> database could hold with no stability issues is aproximitely equal to >> that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh >> >> Does anybody have any idea where they've got hold of this figure? > the figure is within 1% of the maximun size for data stored in text > (or bytea) column. No it isn't; the max size per field is 1GB. Although actually manipulating such field values will probably not work very well unless you have a 64-bit machine, else you'll hit address-space issues. I could believe that a specific application using specific fields in a specific way in a 32-bit machine might start to hit "out of memory" errors for field widths somewhere in the tens-of-MB range. But the stated claim is about number of rows, not row width, and the exactness and breadth of the claim is, well, ridiculous on its face. I think INPS's level of knowledge about PG must be about as good as their ability to spell it :-( BTW, there *is* a hard limit of 32TB per table, arising from the limited size of BlockNumber. But it's hard to believe that INPS's claim has anything to do with that. regards, tom lane
Tom Lane wrote: > Jasen Betts <jasen@xnet.co.nz> writes: >> On 2011-05-01, Mark Morgan Lloyd <markMLl.pgsql-general@telemetry.co.uk> wrote: >>> Somebody is making a very specific claim that Postgres can support a >>> limited number of rows: >>> >>> "INPS (a data forensics team) said that there is 7 main Databases all >>> hosted at different data centers but linked over a type of 'cloud' Each >>> database uses PostGRESSQL which would mean the most amount of data each >>> database could hold with no stability issues is aproximitely equal to >>> that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh >>> >>> Does anybody have any idea where they've got hold of this figure? > >> the figure is within 1% of the maximun size for data stored in text >> (or bytea) column. > > No it isn't; the max size per field is 1GB. Although actually > manipulating such field values will probably not work very well unless > you have a 64-bit machine, else you'll hit address-space issues. > > I could believe that a specific application using specific fields in > a specific way in a 32-bit machine might start to hit "out of memory" > errors for field widths somewhere in the tens-of-MB range. But the > stated claim is about number of rows, not row width, and the exactness > and breadth of the claim is, well, ridiculous on its face. > > I think INPS's level of knowledge about PG must be about as good as > their ability to spell it :-( > > BTW, there *is* a hard limit of 32TB per table, arising from the limited > size of BlockNumber. But it's hard to believe that INPS's claim has > anything to do with that. I suspect that INPS is something to do with the French police, so might have been briefed by somebody who purports to know what he's talking about. But I'm assuming that whoever put those ramblings on Pastebin is unreliable, the only reason I mentioned it was that specific claims were being made which might become widely repeated. -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
On 05/01/2011 01:50 AM, Mark Morgan Lloyd wrote: > Somebody is making a very specific claim that Postgres can support a > limited number of rows Did you find this via http://www.reversecurity.com/2011/04/new-details-from-psn-hack.html ? That was the only Google-indexed source leading to it I found. I just left a note there about the silliness of these claims. I could run more than a 10M row PostgreSQL instance on my phone. Unless there's a new 16-bit only Vic 20 port of PostgreSQL available or something, it's seems unlikely the data had to be partitioned due to any hard limit. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
Greg Smith wrote: > On 05/01/2011 01:50 AM, Mark Morgan Lloyd wrote: >> Somebody is making a very specific claim that Postgres can support a >> limited number of rows > > > Did you find this via > http://www.reversecurity.com/2011/04/new-details-from-psn-hack.html ? > That was the only Google-indexed source leading to it I found. I just > left a note there about the silliness of these claims. I could run more > than a 10M row PostgreSQL instance on my phone. Unless there's a new > 16-bit only Vic 20 port of PostgreSQL available or something, it's seems > unlikely the data had to be partitioned due to any hard limit. Yes, via Google. I was digging around for any information about what sort of access and APIs the network made available to end-user (or ersatz developer) systems. -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]