Thread: storage of sensor data with Fourier transforms

Hello!<br /><br />I have a potential situation where I will have a lot of sensor data coming in very often. (every
secondor so) The sensor data is from physics type measurements, and will normally follow a slowly changing pattern with
sinusoidaldisturbances. The overall shape of the data is more important than catching high frequency disturbances. <br
/><br/>I had the idea of taking the Fourier transform of the waveform and storing the waveform internally that way to
reducestorage requirements. When I get some free cycles, I may try doing this. I would want it to be done in the
databasein such a way that the user can still join to a table using this internal storage scheme. <br /><br />Why am I
mailingthis list? I'd like to ask if anyone has heard of someone doing anything like this. I did a small search of the
listsand internet but didn't come up with anything. I just don't want to re-invent the wheel. <br /><br />Thanks for
yourtime,<br />Nathan<br /><br />-- <br />_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _<br />Got Mole
problems?Call Avogadro at 6.02 x 10^23.  

On May 4, 2007, at 10:13 PM, Nathan Buchanan wrote:

> Hello!
>
> I have a potential situation where I will have a lot of sensor data  
> coming in very often. (every second or so) The sensor data is from  
> physics type measurements, and will normally follow a slowly  
> changing pattern with sinusoidal disturbances. The overall shape of  
> the data is more important than catching high frequency disturbances.
>
> I had the idea of taking the Fourier transform of the waveform and  
> storing the waveform internally that way to reduce storage  
> requirements. When I get some free cycles, I may try doing this. I  
> would want it to be done in the database in such a way that the  
> user can still join to a table using this internal storage scheme.
>
> Why am I mailing this list? I'd like to ask if anyone has heard of  
> someone doing anything like this. I did a small search of the lists  
> and internet but didn't come up with anything. I just don't want to  
> re-invent the wheel.

It's an interesting idea in theory (sounds like a simple low-pass  
filter would do most of what you want) but it doesn't sound like that  
great an idea in practice.

Sampling at 1Hz isn't going to generate very much data at all, unless  
you're capturing ridiculous amounts of data at each sample point (I  
have boxes averaging hundreds of pieces of data a second totaling 10s  
of k, along with a bunch of index updates without breaking much of an  
I/O sweat).

Unless you have some very tight storage constraints (embedded  
controllers, flash storage, ...) you're not going to see a problem  
with storing the actual data, and you're likely to regret both  
discarding the original data and expending much effort developing  
software low-pass pre-processing for the input signal.

(Processing it for analysis is a whole other thing, but even that  
will be much easier to do if you can apply variants on your algorithm  
to the untouched original data).

Cheers,  Steve

"Nathan Buchanan" <nbinont@gmail.com> writes:
> I had the idea of taking the Fourier transform of the waveform and storing
> the waveform internally that way to reduce storage requirements.

Aside from what Steve said: The Fourier transform in itself doesn't
reduce data size --- it's N points in, N points out.  If you want to
reduce storage requirements you have to resort to lossy compression, ie,
deliberately throwing away some data.  The transformed data might be
more suitable for doing that (eg you can selectively discard
high-frequency components) but do you really want to?  Usually the point
of storing measurements is so you can do unspecified analysis on them
later.  Applying lossy compression will restrict what you can
(meaningfully) do later on.
        regards, tom lane

Thanks for the replies. It seems I need to examine my plan more closely.

Nathan