We recently built a new server for our Production database. The machine is top of the line with 128GB of RAM, dual E5-2650. We also included NVME drives for ZIL and L2ARC. Currently we have 3 zpools. First one holds the FreeBSD install. Second holds the jails, and third holds all of the database data. Needless to say it's fast.
FWIW I did not find having a ZIL beneficial for my workload on a similarly huge servers, also running FreeBSD 10. I do have the L2ARC on the SSD, but the size of my data set usually leaves the ARC as sufficient to handle almost all requests. That is, the L2ARC is mostly empty most of the time (or at least never gets re-fetched from).
I'd start by testing the speed of the driver running the NVME drive. Does it show up as a normal drive in FreeBSD? I've only ever used regular Intel SSDs. I don't know if NVME devices connect differently. I hear if you ask very nicely to the right people you can get special drivers for some fancy PCI-e based "drives" which make them really fast.
