I did:
# /sbin/sysctl -w vm.overcommit_memory=2
following
http://www.postgresql.org/docs/7.4/static/kernel-resources.html#AEN17068
And got the same error:
ERROR: out of memory
DETAIL: Failed on request of size 44.
CONTEXT: PL/pgSQL function "group_dup" line 9 at SQL statement
The difference now is that the process was killed before overcommiting.
Regards, Clodoaldo Pinto