With my dataset I have been able to take advantage of the L2ARC. Currently using about 80GB on ARC and 260GB on L2ARC. With the ARC currently having the greater Hit ratio.
Did you tell postgres that the effective_cache_size = Shared Buffers + ARC? I don't include in postgres the L2ARC size as it is not really in memory. Also, I lie to postgres about the random_page_cost and seq_page_cost and set them both to 0.3 since there is so much cached.
In my current test config, I set the effective_cache_size to half the available ram, so roughly 50GBs. I have not tweaked the random_page_cost or seq_page_cost yet.
Set full_page_writes = off since ZFS already takes care of that for you, too.
I currently have the full_page_writes on. I'll disable it and test.
