Thread: taking a sparse slice of an array
Thank you for the useful responses to 'Big Wide Datasets'
I decided to have a go with the array option and initially it looked very promising
time psql test -c "SELECT sample_id, snp_data[40000] FROM gwas_test" > /dev/null
returned in 5 minutes from a 100000 row table.
The trouble is that it does not scale
time psql test -c "SELECT sample_id, snp_data[40000], snp_data[50000] FROM gwas_test" > /dev/null
takes 10 minutes :-(
Is there anyway to speed this up I can see snp_data[40000:50000] returns values from 40 to 50 thousand,
is there any way to so something like
snp_data[1,55,1000,40000,50000]
which would just return a 5 element array?
--
Michael
I decided to have a go with the array option and initially it looked very promising
time psql test -c "SELECT sample_id, snp_data[40000] FROM gwas_test" > /dev/null
returned in 5 minutes from a 100000 row table.
The trouble is that it does not scale
time psql test -c "SELECT sample_id, snp_data[40000], snp_data[50000] FROM gwas_test" > /dev/null
takes 10 minutes :-(
Is there anyway to speed this up I can see snp_data[40000:50000] returns values from 40 to 50 thousand,
is there any way to so something like
snp_data[1,55,1000,40000,50000]
which would just return a 5 element array?
--
Michael
On Fri, 9 Dec 2011 08:15:17 +0000 Michael Lush <mjlush@gmail.com> wrote: > time psql test -c "SELECT sample_id, snp_data[40000], snp_data[50000] FROM > gwas_test" > /dev/null > > takes 10 minutes :-( It seems in your case that not using arrays could lead to better access time. -- "I'd love to go out with you, but I'm taking punk totem pole carving."