Whilst building the 10.0 PDF docs for the website, I found I needed to modify the fop configuration on the Debian Jessie build host. What worked for me was:
pgsql@xxxx:~$ cat ~/.foprc
JAVA_ARGS='-Xms1024M -Xmx2048M'
The suggested settings in the docs didn't seem to be enough (I suspect -Xms isn't needed).