Thread: In the belly of the beast (MySQLCon)
Hello, I am currently chilling at MySQLCon. If any other Elephant riders who are doing a little Dolphin hunting are about... I am in Ballroom E about to give a talk on what Mysql can learn from PostgreSQL. Sincerely, Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit
Attachment
Dude, you have got to let us know how that goes... On Apr 17, 2008, at 1:29 PM, Joshua D. Drake wrote: > Hello, > > I am currently chilling at MySQLCon. If any other Elephant riders who > are doing a little Dolphin hunting are about... I am in Ballroom E > about to give a talk on what Mysql can learn from PostgreSQL. > > Sincerely, > > Joshua D. Drake > > -- > The PostgreSQL Company since 1997: http://www.commandprompt.com/ > PostgreSQL Community Conference: http://www.postgresqlconference.org/ > Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate > PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit > Erik Jones DBA | Emma® erik@myemma.com 800.595.4401 or 615.292.5888 615.292.0777 (fax) Emma helps organizations everywhere communicate & market in style. Visit us online at http://www.myemma.com
Joshua D. Drake wrote: > Hello, > > I am currently chilling at MySQLCon. If any other Elephant riders who > are doing a little Dolphin hunting are about... I am in Ballroom E > about to give a talk on what Mysql can learn from PostgreSQL. Check the room for barrels of tar and feathers before you start your presentation... -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
On Thu, Apr 17, 2008 at 11:29 AM, Joshua D. Drake <jd@commandprompt.com> wrote: > Hello, > > I am currently chilling at MySQLCon. If any other Elephant riders who > are doing a little Dolphin hunting are about... I am in Ballroom E > about to give a talk on what Mysql can learn from PostgreSQL. is there a live video feed those of us not attending can watch? I've got my popcorn in the microwave already...
On Thu, 17 Apr 2008 11:29:56 -0700 "Joshua D. Drake" <jd@commandprompt.com> wrote: > Hello, > > I am currently chilling at MySQLCon. If any other Elephant riders > who are doing a little Dolphin hunting are about... I am in > Ballroom E about to give a talk on what Mysql can learn from > PostgreSQL. Is your presentation available online at all? Hope it goes well, Josh
On Thu, 17 Apr 2008 22:19:23 -0500 Josh Trutwin <josh@trutwins.homeip.net> wrote: > Is your presentation available online at all? Blogging the bad boy up right now.... Will be available soon. Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit
Attachment
On Thu, 17 Apr 2008 16:43:04 -0400 Geoffrey <lists@serioustechnology.com> wrote: > Joshua D. Drake wrote: > > Hello, > > > > I am currently chilling at MySQLCon. If any other Elephant riders > > who are doing a little Dolphin hunting are about... I am in > > Ballroom E about to give a talk on what Mysql can learn from > > PostgreSQL. > > Check the room for barrels of tar and feathers before you start your > presentation... O.k. the presentation can not be found on planet... www.planetpostgresql.org . Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit
Attachment
jd@commandprompt.com ("Joshua D. Drake") writes:
> On Thu, 17 Apr 2008 22:19:23 -0500
> Josh Trutwin <josh@trutwins.homeip.net> wrote:
>
>> Is your presentation available online at all?
>
> Blogging the bad boy up right now.... Will be available soon.
The presentation seems pretty good...
... But what is more interesting is what was the reaction?
I note in the blog that the "in place upgrade" issue came up.
(Interesting to observe that it *also* came up pretty prominently in
the intro session at PG East... This is obviously a matter of Not
Inconsiderable Interest...) It would be nice to hear what other
things got a reaction. Of course, some of that may be yet to arrive,
as there are doubtless conversations yet to happen!
--
"cbbrowne","@","cbbrowne.com"
http://www3.sympatico.ca/cbbrowne/spiritual.html
FLORIDA: We've been Gored by the bull of politics and we're Bushed.
On Friday 18 April 2008, Chris Browne <cbbrowne@acm.org> wrote: > I note in the blog that the "in place upgrade" issue came up. > (Interesting to observe that it *also* came up pretty prominently in > the intro session at PG East... This is obviously a matter of Not > Inconsiderable Interest...) Upgrades are certainly the biggest PostgreSQL operational issue for me. Uhh, boss, I need to bring the database down for 3 days to do an upgrade. Yes, that means clients can't do anything during that time. Boss - !!@#@#$# Hence why I'm still running 8.1 .... -- Alan
Alan Hodgson wrote: > On Friday 18 April 2008, Chris Browne <cbbrowne@acm.org> wrote: >> I note in the blog that the "in place upgrade" issue came up. >> (Interesting to observe that it *also* came up pretty prominently in >> the intro session at PG East... This is obviously a matter of Not >> Inconsiderable Interest...) > > Upgrades are certainly the biggest PostgreSQL operational issue for me. > > Uhh, boss, I need to bring the database down for 3 days to do an upgrade. > Yes, that means clients can't do anything during that time. Boss - !!@#@#$# > > Hence why I'm still running 8.1 .... What about the: 8.1 -> slony -> 8.3 switch users to 8.3 databases solution. -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
On Fri, 18 Apr 2008 13:54:52 -0400 Geoffrey <lists@serioustechnology.com> wrote: > What about the: > > 8.1 -> slony -> 8.3 > switch users to 8.3 databases > > solution. Is it? What is your transactional velocity? How long will the initial sync transaction have to run? You know vacuum isn't working while that transaction is open right? Are you going to have to setup up a dozen different replicated sets in order to get it done? Slony works in "some" cases it does not work in all nor the most critical (in terms of large sets with minimal downtime requirement). Sincerely, Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ United States PostgreSQL Association: http://www.postgresql.us/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
Joshua D. Drake wrote: > On Fri, 18 Apr 2008 13:54:52 -0400 > Geoffrey <lists@serioustechnology.com> wrote: > >> What about the: >> >> 8.1 -> slony -> 8.3 >> switch users to 8.3 databases >> >> solution. > > Is it? What is your transactional velocity? How long will the initial > sync transaction have to run? You know vacuum isn't working while > that transaction is open right? Are you going to have to setup up a > dozen different replicated sets in order to get it done? > > Slony works in "some" cases it does not work in all nor the most > critical (in terms of large sets with minimal downtime requirement). I don't know that slony is the answer. It was more of a question then an answer. We are hoping to use that solution to migrate from 7.4.19 to 8.3.1. The primary reason is to reduce downtime. -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
On Fri, 18 Apr 2008 14:30:18 -0400 Geoffrey <lists@serioustechnology.com> wrote: > I don't know that slony is the answer. It was more of a question > then an answer. We are hoping to use that solution to migrate from > 7.4.19 to 8.3.1. The primary reason is to reduce downtime. Well :) That is why I asked the questions I did. It very well may be the answer. It might not. For example, what size is your database? How many transactions per hour are you doing? Sincerely, Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ United States PostgreSQL Association: http://www.postgresql.us/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
Joshua D. Drake wrote: > On Fri, 18 Apr 2008 14:30:18 -0400 > Geoffrey <lists@serioustechnology.com> wrote: > >> I don't know that slony is the answer. It was more of a question >> then an answer. We are hoping to use that solution to migrate from >> 7.4.19 to 8.3.1. The primary reason is to reduce downtime. > > Well :) That is why I asked the questions I did. It very well may be > the answer. It might not. For example, what size is your database? How > many transactions per hour are you doing? We have 13 separate databases, not terribly large, my WAG would be that our largest database might have 30 million rows. I couldn't honestly tell you how many trans/hour we are doing without doing a little research. -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
On Friday 18 April 2008, Geoffrey <lists@serioustechnology.com> wrote: > What about the: > > 8.1 -> slony -> 8.3 > switch users to 8.3 databases > > solution. 15+ million row inserts/updates a day across 1000+ tables. Oh, and an extensive existing Slony structure for some portions of the database. I could conceivably construct Slony sets for the currently non-replicated tables and iteratively subscribe them (so the initial subscribe doesn't take a week ...). I'm not at all sure Slony could keep up with our update load, though, especially not while maintaining current database operations and also handling those subscriptions. Slony doesn't really work well with high transaction volumes, in my experience. I am going to play with this and see where it breaks, but it's going to be an enormous time investment to babysit it. -- Alan
Geoffrey wrote: > Joshua D. Drake wrote: >> On Fri, 18 Apr 2008 14:30:18 -0400 >> Geoffrey <lists@serioustechnology.com> wrote: >> >>> I don't know that slony is the answer. It was more of a question >>> then an answer. We are hoping to use that solution to migrate from >>> 7.4.19 to 8.3.1. The primary reason is to reduce downtime. >> >> Well :) That is why I asked the questions I did. It very well may be >> the answer. It might not. For example, what size is your database? How >> many transactions per hour are you doing? > > We have 13 separate databases, not terribly large, my WAG would be that > our largest database might have 30 million rows. I couldn't honestly > tell you how many trans/hour we are doing without doing a little > research. I find it pretty unlikely that Slony would be unable to help you upgrade here. Obviously you can upgrade one database at a time. Also, mostly it's not the database size what's a concern, but rather the size of the largest table. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On Fri, 18 Apr 2008 14:59:34 -0400 Alvaro Herrera <alvherre@commandprompt.com> wrote: > I find it pretty unlikely that Slony would be unable to help you > upgrade here. Obviously you can upgrade one database at a time. > > Also, mostly it's not the database size what's a concern, but rather > the size of the largest table. As I recall (I could be wrong) Slony syncs the whole set as a single transaction. So if he has his entire database as a set he may have a problem regardless of the largest or smallest table. I would also agree that 30 million rows is likely not a problem but he should still check his velocity. Sincerely, Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ United States PostgreSQL Association: http://www.postgresql.us/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
Alvaro Herrera wrote: > Geoffrey wrote: >> Joshua D. Drake wrote: >>> On Fri, 18 Apr 2008 14:30:18 -0400 >>> Geoffrey <lists@serioustechnology.com> wrote: >>> >>>> I don't know that slony is the answer. It was more of a question >>>> then an answer. We are hoping to use that solution to migrate from >>>> 7.4.19 to 8.3.1. The primary reason is to reduce downtime. >>> Well :) That is why I asked the questions I did. It very well may be >>> the answer. It might not. For example, what size is your database? How >>> many transactions per hour are you doing? >> We have 13 separate databases, not terribly large, my WAG would be that >> our largest database might have 30 million rows. I couldn't honestly >> tell you how many trans/hour we are doing without doing a little >> research. > > I find it pretty unlikely that Slony would be unable to help you upgrade > here. Obviously you can upgrade one database at a time. > > Also, mostly it's not the database size what's a concern, but rather the > size of the largest table. Well, we have done some extensive testing and it seems feasible. We've got more testing to do, so we've not decided. We hope to use slony to replicate the data in order to have a hot spare solution as well. -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
Joshua D. Drake wrote: > On Fri, 18 Apr 2008 14:59:34 -0400 > Alvaro Herrera <alvherre@commandprompt.com> wrote: > >> I find it pretty unlikely that Slony would be unable to help you >> upgrade here. Obviously you can upgrade one database at a time. >> >> Also, mostly it's not the database size what's a concern, but rather >> the size of the largest table. > > As I recall (I could be wrong) Slony syncs the whole set as a single > transaction. So if he has his entire database as a set he may > have a problem regardless of the largest or smallest table. I would also > agree that 30 million rows is likely not a problem but he should still > check his velocity. We have actually replicated all 13 databases and there was no apparent impact on production. This was in the evening, but we are a 24/7 shop. I'm really not altogether sure what you mean by transaction velocity. I'm pretty sure the electrons are traveling pretty close to the speed of light. ;) -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
Joshua D. Drake wrote: > On Fri, 18 Apr 2008 14:59:34 -0400 > Alvaro Herrera <alvherre@commandprompt.com> wrote: > > > I find it pretty unlikely that Slony would be unable to help you > > upgrade here. Obviously you can upgrade one database at a time. > > > > Also, mostly it's not the database size what's a concern, but rather > > the size of the largest table. > > As I recall (I could be wrong) Slony syncs the whole set as a single > transaction. So if he has his entire database as a set he may > have a problem regardless of the largest or smallest table. Yes, if he does that then you are right, but it's unlikely to be the best idea. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Geoffrey wrote: > I'm really not altogether sure what you mean by transaction velocity. > I'm pretty sure the electrons are traveling pretty close to the speed of > light. ;) Actually, electrons themselves flow rather slowly -- millimeters per second according to Wikipedia. The signal propagation is a bit faster: "typically 75% of light speed", Wikipedia again. Anyway, he means "transaction rate". But the actual useful figure is further removed from that because we're mostly only interested in write transactions, not read-only (though those still have an effect in versions earlier than 8.3.) -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Alvaro Herrera wrote: > Geoffrey wrote: > >> I'm really not altogether sure what you mean by transaction velocity. >> I'm pretty sure the electrons are traveling pretty close to the speed of >> light. ;) > > Actually, electrons themselves flow rather slowly -- millimeters per > second according to Wikipedia. The signal propagation is a bit faster: > "typically 75% of light speed", Wikipedia again. Well, with respect to the velocity of my car, they're much closer to the speed of light. It's all relative. In that comparison, the difference is certainly imperceptible. Let's not take this to the quantum level okay? :) > Anyway, he means "transaction rate". But the actual useful figure is > further removed from that because we're mostly only interested in write > transactions, not read-only (though those still have an effect in > versions earlier than 8.3.) Understood. I don't think we'll have a problem, based on our testing thus far. We have to get to 8.3 first though, and that's on the front burner. -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
On Apr 18, 2008, at 2:04 PM, Joshua D. Drake wrote: > On Fri, 18 Apr 2008 14:59:34 -0400 > Alvaro Herrera <alvherre@commandprompt.com> wrote: > >> I find it pretty unlikely that Slony would be unable to help you >> upgrade here. Obviously you can upgrade one database at a time. >> >> Also, mostly it's not the database size what's a concern, but rather >> the size of the largest table. > > As I recall (I could be wrong) Slony syncs the whole set as a single > transaction. So if he has his entire database as a set he may > have a problem regardless of the largest or smallest table. I would > also > agree that 30 million rows is likely not a problem but he should still > check his velocity. FWIW, Londiste doesn't work with sets and, instead, maintains replication state, as well as giving you the option to add/remove tables from replication, on a table-by-table which can be observed from the command line. Erik Jones DBA | Emma® erik@myemma.com 800.595.4401 or 615.292.5888 615.292.0777 (fax) Emma helps organizations everywhere communicate & market in style. Visit us online at http://www.myemma.com
alvherre@commandprompt.com (Alvaro Herrera) writes:
> Geoffrey wrote:
>> Joshua D. Drake wrote:
>>> On Fri, 18 Apr 2008 14:30:18 -0400
>>> Geoffrey <lists@serioustechnology.com> wrote:
>>>
>>>> I don't know that slony is the answer. It was more of a question
>>>> then an answer. We are hoping to use that solution to migrate from
>>>> 7.4.19 to 8.3.1. The primary reason is to reduce downtime.
>>>
>>> Well :) That is why I asked the questions I did. It very well may be
>>> the answer. It might not. For example, what size is your database? How
>>> many transactions per hour are you doing?
>>
>> We have 13 separate databases, not terribly large, my WAG would be that
>> our largest database might have 30 million rows. I couldn't honestly
>> tell you how many trans/hour we are doing without doing a little
>> research.
>
> I find it pretty unlikely that Slony would be unable to help you upgrade
> here. Obviously you can upgrade one database at a time.
>
> Also, mostly it's not the database size what's a concern, but rather the
> size of the largest table.
Indeed.
There is the "big bang" approach where you subscribe to the Whole
Thang in One Swell Foop, and, if it takes 8h to dump data, 12h to load
it into the new DB (which will take place essentially in parallel with
the 8), and 16h to regenerate indices, then that means you've got a
transaction open against the source DB for something like 28h (e.g. -
12h + 16h), with the obvious attendant adverse effects on vacuuming
and such.
However, it is unusual for a database to consist of just one table of
that sort. If you have a case like this, it will make plenty of sense
to split this set of tables into pieces, and add them in at least
somewhat incrementally.
In a 28h subscription, there are likely to be 10 tables that are of
"significant size" that could be broken into independent
subscriptions, so that there's not the huge hit of one giant 28h
transaction.
--
output = ("cbbrowne" "@" "cbbrowne.com")
http://linuxdatabases.info/info/wp.html
Why isn't phonetic spelled the way it sounds?
On Apr 18, 2008, at 2:18 PM, Alvaro Herrera wrote: > Actually, electrons themselves flow rather slowly -- millimeters per > second according to Wikipedia. The signal propagation is a bit > faster: > "typically 75% of light speed", Wikipedia again. Yeah, electrons move *very* slowly in a solid. Presumably somewhat faster in liquid. Definitely faster in a gas, but you have to have a pretty good vacuum for them to actually approach the speed of light. -- Decibel!, aka Jim C. Nasby, Database Architect decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828
Attachment
On Apr 18, 2008, at 2:42 PM, Chris Browne wrote: > However, it is unusual for a database to consist of just one table of > that sort. If you have a case like this, it will make plenty of sense > to split this set of tables into pieces, and add them in at least > somewhat incrementally. Does anyone happen to have any scripts/code that will just trawl through a database, adding tables to a set one at a time? -- Decibel!, aka Jim C. Nasby, Database Architect decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828
Attachment
> I am going to play with this and see where it breaks, but it's going to be > an enormous time investment to babysit it. One thing to remember, since you've already got backup in place and this replica would just be for upgrading, not a hot spare, turn off fsync during the initial subscription. My data set is quite a bit smaller, and I've gotten into the habit of turning off fsync during the initial post-upgrade load, to shorten my downtime. -- Scott Ribe scott_ribe@killerbytes.com http://www.killerbytes.com/ (303) 722-0567 voice
On Sun, Apr 20, 2008 at 11:12 AM, Scott Ribe <scott_ribe@killerbytes.com> wrote: > > I am going to play with this and see where it breaks, but it's going to be > > an enormous time investment to babysit it. > > One thing to remember, since you've already got backup in place and this > replica would just be for upgrading, not a hot spare, turn off fsync during > the initial subscription. My data set is quite a bit smaller, and I've > gotten into the habit of turning off fsync during the initial post-upgrade > load, to shorten my downtime. Exactly. There are several things you can do on the replica that you wouldn't do on the master for better performance. Really big numbers of WAL segments, no background writer, fsync off as mentioned, WAL on a RAID-0 with 10 disks, no battery back on a caching RAID controller, and so on. You also might look into cheap but useful things like bonded gigabit networking between the two servers with a dedicated switch / rolled cable. I wonder if there's a comprehensive list somewhere... What I keep dreaming of is a process that lets slony use pg_bulkloader or something like it to do the initial load...
On Sun, 20 Apr 2008 11:32:58 -0600 "Scott Marlowe" <scott.marlowe@gmail.com> wrote: > Exactly. There are several things you can do on the replica that you > wouldn't do on the master for better performance. Really big numbers > of WAL segments, no background writer, fsync off as mentioned, WAL on > a RAID-0 with 10 disks, no battery back on a caching RAID controller, > and so on. You are still limited by how fast you can pull data from the master over the network to the slave. Which when dealing with 500GB+ is a significant limitation. Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ United States PostgreSQL Association: http://www.postgresql.us/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
On Sun, Apr 20, 2008 at 12:48 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > On Sun, 20 Apr 2008 11:32:58 -0600 > "Scott Marlowe" <scott.marlowe@gmail.com> wrote: > > > Exactly. There are several things you can do on the replica that you > > wouldn't do on the master for better performance. Really big numbers > > of WAL segments, no background writer, fsync off as mentioned, WAL on > > a RAID-0 with 10 disks, no battery back on a caching RAID controller, > > and so on. > > You are still limited by how fast you can pull data from the master > over the network to the slave. Which when dealing with 500GB+ is a > significant limitation. Agreed. Hence the part of my post about using bonded gigabit ethernet to boost that speed. Admitted, there's only so much you can do there.
On Fri, Apr 18, 2008 at 9:04 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> On Fri, 18 Apr 2008 14:59:34 -0400
> Alvaro Herrera <alvherre@commandprompt.com> wrote:
>
> > I find it pretty unlikely that Slony would be unable to help you
> > upgrade here. Obviously you can upgrade one database at a time.
> >
> > Also, mostly it's not the database size what's a concern, but rather
> > the size of the largest table.
>
> As I recall (I could be wrong) Slony syncs the whole set as a single
> transaction. So if he has his entire database as a set he may
> have a problem regardless of the largest or smallest table. I would also
> agree that 30 million rows is likely not a problem but he should still
> check his velocity.
Well, you can sync one table at a time (create a (temporary) set with
one able, subscribe it, SYNC, MERGE SET, rince, lather, repeat). :)
Regards,
Dawid
On Sun, 2008-04-20 at 11:32 -0600, Scott Marlowe wrote: > On Sun, Apr 20, 2008 at 11:12 AM, Scott Ribe <scott_ribe@killerbytes.com> wrote: > > > I am going to play with this and see where it breaks, but it's going to be > > > an enormous time investment to babysit it. One more suggestion: if you happen to use the alt_perl tools, be sure to uncomment/change the line: $SYNC_CHECK_INTERVAL = 1000; In my experience it made a big difference to set that to 60000 (meaning sync events created once per minute instead of once per second) for the synchronizing after copy phase. The number of events generated while the copy over is running can be so big that it will never get in sync again otherwise... Cheers, Csaba.
scott.marlowe@gmail.com ("Scott Marlowe") writes:
> What I keep dreaming of is a process that lets slony use pg_bulkloader
> or something like it to do the initial load...
Does there seem to be some likelihood of some portion of pg_bulkloader
getting added to core? It sounds like it's worth looking at
pg_bulkloader...
--
select 'cbbrowne' || '@' || 'linuxdatabases.info';
http://www3.sympatico.ca/cbbrowne/wp.html
"It has every known bug fix to everything." -- KLH (out of context)
nagy@ecircle-ag.com (Csaba Nagy) writes: > On Sun, 2008-04-20 at 11:32 -0600, Scott Marlowe wrote: >> On Sun, Apr 20, 2008 at 11:12 AM, Scott Ribe <scott_ribe@killerbytes.com> wrote: >> > > I am going to play with this and see where it breaks, but it's going to be >> > > an enormous time investment to babysit it. > > One more suggestion: if you happen to use the alt_perl tools, be sure to > uncomment/change the line: > > $SYNC_CHECK_INTERVAL = 1000; > > In my experience it made a big difference to set that to 60000 (meaning > sync events created once per minute instead of once per second) for the > synchronizing after copy phase. The number of events generated while the > copy over is running can be so big that it will never get in sync again > otherwise... +1 I'm adding a note about that to the "best practices" in the admin guide. -- select 'cbbrowne' || '@' || 'linuxfinances.info'; http://cbbrowne.com/info/lsf.html Disco is to music what Etch-A-Sketch is to art.
On Sun, 20 Apr 2008, Scott Marlowe wrote: > I wonder if there's a comprehensive list somewhere... There's a good list on the database side at http://www.postgresql.org/docs/current/interactive/populate.html but it doesn't dwelve into hardware changes. I put a reference to that as a placeholder at http://wiki.postgresql.org/wiki/Performance_Optimization with the intention of including broader suggestions in a new page there one day. You're welcome to take a stab at that yourself. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
On Fri, Apr 18, 2008 at 11:15:09AM -0700, Joshua D. Drake wrote: > sync transaction have to run? You know vacuum isn't working while > that transaction is open right? Are you going to have to setup up a > dozen different replicated sets in order to get it done? A dozen sets isn't really a big deal, though. The big problem is when you have really huge tables. A