Thread: pg_restore -j
Hi, I just noticed (!) that Make accepts an argument-less -j option, which it takes to mean "use as many parallel jobs as possible". As far as I see in our pg_restore code, we don't even accept an argumentless -j option; was this deviation from the Make precedent on purpose, or were we just not following Make at all on this? I have to admit that I'm not really sure whether this kind of usage would be a reasonable thing for pg_restore to support. (Even if this was a good idea, I'm not suggesting that it be implemented for 8.4. But if it is, then maybe it deserves a TODO entry.) Thoughts? -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Alvaro Herrera wrote: > Hi, > > I just noticed (!) that Make accepts an argument-less -j option, which > it takes to mean "use as many parallel jobs as possible". As far as I > see in our pg_restore code, we don't even accept an argumentless -j > option; was this deviation from the Make precedent on purpose, or were > we just not following Make at all on this? > > I have to admit that I'm not really sure whether this kind of usage > would be a reasonable thing for pg_restore to support. > > (Even if this was a good idea, I'm not suggesting that it be implemented > for 8.4. But if it is, then maybe it deserves a TODO entry.) An unlimited pg_restore -j seems pretty scary. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Alvaro Herrera wrote: > Hi, > > I just noticed (!) that Make accepts an argument-less -j option, which > it takes to mean "use as many parallel jobs as possible". As far as I > see in our pg_restore code, we don't even accept an argumentless -j > option; was this deviation from the Make precedent on purpose, or were > we just not following Make at all on this? > > I have to admit that I'm not really sure whether this kind of usage > would be a reasonable thing for pg_restore to support. > > (Even if this was a good idea, I'm not suggesting that it be implemented > for 8.4. But if it is, then maybe it deserves a TODO entry.) > > Thoughts? > > There was no intention to follow Make. And I think it's far far too early to be planning "improvements" of this kind. We need to see how it gets used in the field. cheers andrew
Bruce Momjian <bruce@momjian.us> writes:
> Alvaro Herrera wrote:
>> I just noticed (!) that Make accepts an argument-less -j option, which
>> it takes to mean "use as many parallel jobs as possible".
> An unlimited pg_restore -j seems pretty scary.
Yeah. Even if Make has a sane way to estimate how many jobs it should
use, I'm not sure that pg_restore does. (The most obvious heuristic
for Make is to try to find out how many CPUs there are --- but at
least it's running on the same machine it's going to be eating CPU
on. pg_restore can't assume that.)
regards, tom lane
On Thursday 23 April 2009 01:26:04 Alvaro Herrera wrote: > I just noticed (!) that Make accepts an argument-less -j option, which > it takes to mean "use as many parallel jobs as possible". As far as I > see in our pg_restore code, we don't even accept an argumentless -j > option; was this deviation from the Make precedent on purpose, or were > we just not following Make at all on this? There was likely no strong intention to follow make on this. A small problem would be that getopt doesn't portably support single-letter options with optional arguments. The main problem, however, is that make -j is pretty useless and dangerous. Using it on a large parallel-make-safe project can easily lock up (thrash) your machine. make -j together with -l (--load-average) is kind of useful, I guess, but exactly how "load average" translated to a PostgreSQL database system is to be determined.
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Yeah. Even if Make has a sane way to estimate how many jobs it should
> use, I'm not sure that pg_restore does. (The most obvious heuristic
> for Make is to try to find out how many CPUs there are --- but at
> least it's running on the same machine it's going to be eating CPU
> on. pg_restore can't assume that.)
I'm not sure if I'd consider it 'sane', but make basically uses the
dependency information, if a job can be run based on its dependency
requirements, then it's started. For small projects, this isn't
necessairly terrible, but it's not something I would generally
recommend.
I don't see any reasonable implementation, or justification, for
supporting something like that in pg_restore.
Thanks,
Stephen