Re: Stock Market Price Data & postgreSQL? HELLPPP Please - Mailing list pgsql-general

On Thu, Aug 20, 2009 at 4:23 AM, Tom Lane<tgl@sss.pgh.pa.us> wrote:

> It does sound like you are doing pretty much exactly what we were
> doing.  One thing to think about is that the real-time case is actually
> much slower and easier to deal with than back-testing.  When you are
> back-testing, you'd like to test a trading strategy over say a year's
> worth of data, and you'd like that to require a bit less than a year
> to run, no?  So the path you have to optimize is the one feeding stored,
> pre-aggregated data into your test engine.  The people hollering about
> real-time performance are just betraying that they've never built one of
> these things.  I'm not familiar with this CEP software, but it sounds to
> me like you want that as part of the real-time data collection process
> and nowhere near your back-testing data path.
>
CEP (Continuous Event Processing...just check out the explanation para
at http://esper.codehaus.org/ ) is basically running queries on data
streams in memory (i got that as rdbms tables in memory)...on time
series data (whatever that is)..... no concept of storage etc.

BTW, would you please look back a bit in this thread, a post (by
Adrian Klaver see..
http://archives.postgresql.org/pgsql-general/2009-08/msg00745.php )
about a database built by a berkeley computer science deptt. project
on postgres 7.2, that stores & queries, time series data just like
this and returns data in a stream instead of a recordset. It is said
that this type of databases are made for analyzing this type of data.
And it has normal postgres mode for returning records and streams mode
for returning results in a stream....which I think is a nothing but a
TCP-IP socket which receives results on an ongoing basis

I have also posted a question in its reply (see ..
http://archives.postgresql.org/pgsql-general/2009-08/msg00748.php ),
about why postgres itself does not have this (whatever this time
series data is in mathematical terms) when that project itself extends
postgres 7.2. Will you please have a look at these posts and this
project which is using postgres itself.

And please explain in layman terms what these guys are doing different
that we cannot have a time series data types (we do have GIS & tons of
other data types)  & stream data returns. Maybe not immediately of use
to me but I understand ODBMS, RDBMS and I want to understand this at a
conceptual level, at least.
>
> Another little tip: if your trading strategies are like ours were,
> they need to ramp up for actual trading by pre-scanning some amount of
> historical data.  So you're going to need a mechanism that starts by
> reading the historical data at high speed and smoothly segues into
> reading the real-time feed (at which the passage of time in the model
> suddenly slows to one-to-one with real time).  Also consider what
> happens when your machine crashes (it happens) and you need to not only
> redo that startup process, but persuade the model that its actual
> trading position is whatever you had open.  Or you changed the model a
> bit and restart as above.  The model might now wish it was in a different
> position, but it has to cope with your actual position.
>
> The crash reliability of a DBMS is a strong reason why you track your
> live positions in one, btw ... your brokerage isn't going to forget you
> were short IBM, even if your model crashes.

Thats very good advise....we had already planned it that way. And yes,
the brokerages are not going to care that we ran out of money while
our server was restarting ;-)

>
>                        regards, tom lane
>