== PostgreSQL Weekly News - January 28 2007 == - Mailing list pgsql-announce

== PostgreSQL Weekly News - January 28 2007 ==

Another Lively Discussion is going on on -hackers about improvements
to autovacuum.

Andy Astor, CEO of EnterpriseDB has vociferously denied that there was
a policy of gathering email addresses from the PostgreSQL lists and
then sending them marketing materials.

== PostgreSQL Product News ==

Libpqxx 2.6.9 pre-release is out.
http://thaiopensource.org/~jtv/tmp/pqxx/

SQL/PSM 0.1 released.
http://postgresql.interweb.cz/index.php/SQL/PSM

DBI-Link 2.0.0 released.
http://pgfoundry.org/projects/dbi-link/

Another PostgreSQL Diff Tool 1.0.0_beta16 released.
http://pgfoundry.org/projects/apgdiff/

PostgresDAC 2.3.7 released.
http://microolap.com/products/connectivity/postgresdac/download/

Mammoth Replicator 8.0.10-1.7 released.
https://www.commandprompt.com/products/mammothreplicator/

== PostgreSQL Jobs for January ==

http://archives.postgresql.org/pgsql-jobs/2007-01/threads.php

== PostgreSQL Local ==

Major contributor Gavin Sherry and Postgres-R's lead Markus will be
speaking at the SFPUG meeting in San Francisco at Hyperic LLC's
headquarters.
http://postgresql.meetup.com/1/calendar/5380885/

PostgreSQLFr will have a booth at SolutionsLinux 2007, in Paris
(France) from 30th january to 1st February, with special guests Devrim
GUNDUZ and Magnus Hagander.  Jean-Paul Argudo, treasurer of the
PostgreSQLFr non-profit assocation, will lead a session about FLOSS
databases.  More information in French here:
http://www.solutionslinux.fr/index.php

There will be a PostgreSQL booth at FOSDEM on February 24 and 25 in
Brussels, Belgium.  Many of the usual suspects from the EU PostgreSQL
communities will be there.  Contact de@postgresql.org to participate.
http://www.fosdem.org/2007/

The Italian PostgreSQL community will be holding a PostgreSQL day this
summer.  Bookmark the link below to participate.
http://www.pgday.it

== PostgreSQL in the News ==

Planet PostgreSQL: http://www.planetpostgresql.org/

General Bits, Archives and occasional new articles:
http://www.varlena.com/GeneralBits/

PostgreSQL Weekly News is brought to you this week by David Fetter
and Devrim GUNDUZ

To get your submission into the upcoming issue, get it to
david@fetter.org by Sunday at 3:00pm Pacific Time.

== Applied Patches ==

Andrew Dunstan committed:

- Fix plpython MSVC build in non-debug mode.

- Fix path problem in MSVC bison wrapper. per Joachim Wieland.

- In PL/Perl, remove unnecessary and now inaccurate cast which I
  should have removed with other old code.

- In PL/Perl, allow args to spi_prepare to be standard type aliaes as
  well as those known in pg_type.  Fixes bug #2917.  Add some
  regression tests for these cases.

Michael Meskes committed:

- In ecpg, fixed expected files, so they are in sync with tests again.

- In ecpg, removed compiler warning due to unneeded unsigned
  declaration and removed regression test that triggers those libc
  precision bugs on some archs.

Neil Conway committed:

- Rename the uuid_t type to pg_uuid_t, to avoid a conflict with any
  definitions of uuid_t that may be provided by the system headers.
  This should hopefully fix the Win32 build problems reported by
  Magnus.

- Gevik Babakhani's patch which add a new builtin type, "uuid". This
  implements a UUID type, similar to that defined in RFC 4122.  This
  patch includes the basic implementation, plus regression tests.
  Documentation and perhaps some additional functionality will come
  later. Catversion bumped.  Review from Peter Eisentraut, Tom Lane,
  and Neil Conway.

- Magnus Hagander's patch which squelches some VC++ compiler warnings.
  Mark float literals with the "f" suffix, to distinguish them from
  doubles. Make some function declarations and definitions use the
  "const" qualifier for arguments consistently.  Ignore warning 4102
  ("unreferenced label"), because such warnings are always emitted by
  bison-generated code.

- Magnus Hagander's patch which adds a setlocal command to the
  beginning of build.bat. This is required to deal with buildenv.bat
  properly, so that PATH (for example) doesn't expand infintly. Per
  report from Joachim Wieland.

- Magnus Hagander's patch which adds a setlocal command to the
  beginning of build.bat. This is required to deal with buildenv.bat
  properly, so that PATH (for example) doesn't expand infintly.  Per
  report from Joachim Wieland.

Peter Eisentraut committed:

- In pgsql/src/backend/utils/adt/xml.c, reactivate libxml memory
  management via palloc, now that I think I've classified the
  conditions under which this is safe to do (see source code comment).

- In pgsql/src/backend/utils/adt/xml.c, add trailing zero byte in
  Unicode codepoint conversion.

- Various fixes in the logic of XML functions:  Add new SQL command
  SET XML OPTION (also available via regular GUC) to control the
  DOCUMENT vs. CONTENT option in implicit parsing and serialization
  operations.  Subtle corrections in the handling of the standalone
  property in xmlroot().  Allow xmlroot() to work on content
  fragments.  Subtle corrections in the handling of the version
  property in xmlconcat().  Code refactoring for producing XML
  declarations.

- Simplify handling of XML error messages: Just use the string
  provided by libxml as the detail message.  As per
  <http://archives.postgresql.org/pgsql-hackers/2006-12/msg01087.php>.
  For converting error codes to messages, we only need to cover those
  codes that we raise ourselves now.

Alvaro Herrera committed:

- Magnus Hagander's patch which is required for vcbuild to work after
  the changes to pg_proc.h (it's the same changes that are in
  gen_fmgrtab.sh in the unix build).

Tom Lane committed:

- Improve hash join to discard input tuples immediately if they can't
  match because they contain a null join key (and the join operator is
  known strict).  Improves performance significantly when the inner
  relation contains a lot of nulls, as per bug #2930.

- Remove unnecessary checkpoint from PL regression tests.  This was
  once handy to prevent core dump files from disappearing, but it's
  useless now because (a) we don't drop core in individual DB
  subdirectories anymore, and (b) CREATE DATABASE forces an internal
  checkpoint anyway.

- Remove some unnecessary conversion work in build_regtype_array().

- Repair oversight in creation of "append relations": we should set up
  rel->tuples as well as rel->rows, since some estimation functions
  expect both to be valid in every baserel.  Per report from Dave
  Dutcher.

- Make some small improvements in the accuracy of plpgsql's error
  location reports; inspired by the misleading CONTEXT lines shown in
  recent bug report from Stefan Kaltenbrunner.  Also, allow
  statement-type names shown in these messages to be translated.

- Fix up plpgsql's "simple expression" evaluation mechanism so that it
  behaves safely in the presence of subtransactions.  To ensure that
  any ExprContext shutdown callbacks are called at the right times, we
  have to have a separate EState for each level of subtransaction.
  Per "TupleDesc reference leak" bug report from Stefan Kaltenbrunner.
  Although I'm convinced the code is wrong as far back as 8.0, it
  doesn't seem that there are any ways for the problem to really
  manifest before 8.2: AFAICS, 8.0 and 8.1 only use the
  ExprContextCallback mechanism to handle set-returning functions,
  which cannot usefully be executed in a "simple expression" anyway.
  Hence, no backpatch before 8.2 --- the risk of unforeseen breakage
  seems to outweigh the chance of fixing something.

- Drat, can't fit an additional argument into log_error.  Is it worth
  an sprintf pushup to be sure we can report something useful for
  out-of-range exitstatus?

- Clean up broken usage of HAVE_DECL_SYS_SIGLIST and
  inconsistent/poorly formatted error messages.

- Add a delay at the start of the stats test, to let any prior stats
  activity quiesce.  Possibly this will fix the large increase in
  non-reproducible stats test failures we've noted since turning on
  stats_row_level by default.

- Dept of second thoughts: the IQ of estimate_array_length() needs to
  be kept on par with that of scalararraysel(), else estimates that
  should track might not.  Hence teach it about binary-compatible
  cases, too.

- Fix scalararraysel() to cope with binary-compatible cases, such as
  text[] versus varchar[].  This oversight probably explains Ryan
  Holmes' recent complaint --- he was getting a generic selectivity
  estimate instead of anything intelligent.

- Correct an old logic error in btree page splitting: when considering
  a split exactly at the point where we need to insert a new item, the
  calculation used the wrong size for the "high key" of the new left
  page.  This could lead to choosing an unworkable split, resulting in
  "PANIC: failed to add item to the left sibling" (or "right sibling")
  failure.  Although this bug has been there a long time, it's very
  difficult to trigger a failure before 8.2, since there was generally
  a lot of free space on both sides of a chosen split.  In 8.2, where
  the user-selected fill factor determines how much free space the
  code tries to leave, an unworkable split is much more likely.
  Report by Joe Conway, diagnosis and fix by Heikki Linnakangas.

- Back-port changes of Jan 16 and 17 to "revoke" pending fsync
  requests during DROP TABLE and DROP DATABASE.  Should prevent
  unexpected "permission denied" failures on Windows, and is cleaner
  on other platforms too since we no longer have to take it on faith
  that ENOENT is okay during an fsync attempt.  Patched as far back as
  8.1; per recent discussion I think we are not going to worry about
  Windows-specific issues in 8.0 anymore.

- On Windows, use pgwin32_waitforsinglesocket() instead of select() to
  wait for input in the stats collector.  Our select() emulation is
  apparently buggy for UDP sockets :-(.  This should resolve problems
  with stats collection (and hence autovacuum) failing under more than
  minimal load.  Diagnosis and patch by Magnus Hagander.

- Get pg_utf_mblen(), pg_utf2wchar_with_len(), and utf2ucs() all on
  the same page about the maximum UTF8 sequence length we support (4
  bytes since 8.1, 3 before that).  pg_utf2wchar_with_len never got
  updated to support 4-byte characters at all, and in any case had a
  buffer-overrun risk in that it could produce multiple pg_wchars from
  what mblen claims to be just one UTF8 character.  The only reason we
  don't have a major security hole is that most callers allocate
  worst-case output buffers; the sole exception in released versions
  appears to be pre-8.2 iwchareq() (ie, ILIKE), which can be crashed
  due to zeroing out its return address --- but AFAICS that can't be
  exploited for anything more than a crash, due to inability to
  control what gets written there.  Per report from James Russell and
  Michael Fuhr.  Pre-8.1 the risk is much less, but I still think
  pg_utf2wchar_with_len's behavior given an incomplete final character
  risks buffer overrun, so back-patch that logic change anyway.  This
  patch also makes sure that UTF8 sequences exceeding the supported
  length (whichever it is) are consistently treated as error cases,
  rather than being treated like a valid shorter sequence in some
  places.

- Relax an Assert() that has been found to be too strict in some
  situations involving unions of types having typmods.  Variants of
  the failure are known to occur in 8.1 and up; not sure if it's
  possible in 8.0 and 7.4, but since the code exists that far back,
  I'll just patch 'em all.  Per report from Brian Hurt.

- Update xindex.sgml to discuss operator families.

- Update pg_dump to support dumping operator families.

- Add CREATE/ALTER/DROP OPERATOR FAMILY commands, also COMMENT ON
  OPERATOR FAMILY; and add FAMILY option to CREATE OPERATOR CLASS to
  allow adding a class to a pre-existing family.  Per previous
  discussion.  Man, what a tedious lot of cutting and pasting ...

- Put back planner's ability to cache the results of
  mergejoinscansel(), which I had removed in the first cut of the
  EquivalenceClass rewrite to simplify that patch a little.  But it's
  still important --- in a four-way join problem mergejoinscansel()
  was eating about 40% of the planning time according to gprof.  Also,
  improve the EquivalenceClass code to re-use join RestrictInfos
  rather than generating fresh ones for each join considered.  This
  saves some memory space but more importantly improves the
  effectiveness of caching planning info in RestrictInfos.

Bruce Momjian committed:

- Use autoconf build-in sys_siglist macro AC_DECL_SYS_SIGLIST, rather
  than create our own.

- Add signal.h for sys_siglist reference.

- Use sys_siglist[] to print out signal names for signal exits, rather
  than just numbers.

- Modify SGML makefile to allow 'gmake draft' to build draft html, so
  'draft' is now a target and no longer a modifier.

- Add to TODO: "Enforce typmod for function inputs, function results
  and parameters for spi_prepare'd statements called from PLs."

- Add to TODO: "Consider having the background writer update the
  transaction status hint bits before writing out the page."

- Add to TODO: "Consider increasing NUM_CLOG_BUFFERS."

- Update installation wording for an upgrade to state that
  dump/restore should not be done, per Peter.

- Control openjade draft-mode by variable DRAFT, rather than whether
  the version tag is 'devel'.

- Update wording of installation when upgrading, to distinguish major
  release upgrades more clearly from minor ones.

- Modify draft SGML instructions to use DRAFT=Y because recursion was
  causing html to be called twice --- no way to exit the makefile
  after the recursion returns.

- Update BSD/OS platform for 8.2.

- In TODO, add URL for shrinking tuple to just its headers.

- Update Win32 exception comment.

- Update pg_dumpall -f option description.

- In TODO, mark as done, "Add a GUC variable to control the tablespace
  for temporary objects pg_dumpall" per Albert Cervera Areny's patch.

- Albert Cervera Areny's patch with documentation by Jaime Casanova to
  add GUC temp_tablespaces to provide a default location for temporary
  objects.

- Properly detoast access to bytea field pg_trigger.tgargs.  Old code
  might cause server crash.  Backpatch to 8.2.X.

- Dave Page's patch which adds 'output file' option for pg_dumpall,
  especially useful for Win32, where output redirection of child
  processes (pg_dump) doesn't work.

- Remove developers list from TODO list now that we have URLs to
  reference discussions.

- In TODO, mark as done, "Add -f to pg_dumpall" per Dave Page's patch.

- Dave Page patch which allows pg_dumpall to specify a database name
  rather than the default 'template1'.

- Dave Page's patch which adds --tablespaces-only and --roles-only
  options to pg_dumpall.

- Simon Riggs's patch to prevent WAL logging when COPY is done in the
  same transaction that created it.

- Magnus Hagander's patch that disables the building of ecpg when
  using MSVC if pthreads is not specified.

- In pgsql/src/include/port/win32.h, add comment about exception lists
  in both winnt.h and ntstatus.h.

- Remove newline from error message because URL is gone.

- Back out use of FormatMessage(), does error values, not exception
  values.  Point to /include/ntstatus.h for an exception list, rather
  than a URL.

- Print meaningfull error text for abonormal process exit on Win32,
  rather than hex codes, using FormatMessage().

- In FAQ, update Bruce Momjian's email address.

- Use errhint() for WIN32 SIGTERM message, where possible.

- In pgsql/src/include/port/win32.h, put hex value in upper case.

- When system() fails in Win32, report it as an exception, print the
  exception value in hex, and give a URL where the value can be
  looked-up.

== Rejected Patches (for now) ==

Guido Goldstein's patch to fix PL/PythonU functions;  return
true/false for boolean, rather than 1/0.  This helps when creating
trigger functions that output SQL.  It also broke most machines on the
buildfarm because it only works with Python 2.3 or higher.

Magnus Hagander's first version of his patch to clean up warnings
generated by VC++.  It was too large and invasive, and had some
thinkos.  He is working on the next version.

== Pending Patches ==

Arul Shaji sent in another version of his implementation of
pg_get_domaindef.

Gevik Babakhani sent in another version of his uuid patch.  Peter
Eisentraut gave some feedback.  Gevik sent in another version per the
comments and added tests.

Jeremy Drake sent in a couple of versions of a patch which allows
unprivileged users to CREATE LANGUAGE for trusted languages already
present in pg_pltemplate.

Teodor Sigaev sent in a doc patch which updates the docbook
instructions for FreeBSD, and another which adds a table of
pairwise compatibilities between the different lock modes.

Andrew Dunstan sent in a patch which lets people use the aliases in
pg_type for parameters in PL/Tcl and PL/PythonU, e.g. integer
instead of int4.

Pavan Deolasee sent in a patch which should marginally improve the
ctid chain followin code path when current and the next tuple in the
chain are in the same block.

Alvaro Herrera sent in a patch which separates autovacuum in two
families of processes: one is the "launcher", in charge of examining
statistics and deciding when to start a worker.  The other is the
worker, which is started by the postmaster under command of the
launcher, and processes what the launcher tells it to process (by way
of setting info up in shared memory).

Pavel Stehule sent in a patch to add ANSI SQL scrollable cursor
support to SPI and PL/PgSQL.

Jeremy Drake sent in another version of his patch which lets you
install PLs without being superuser.