Thread: Tetra-bytes database / large indexes needs

Tetra-bytes database / large indexes needs

From
Jean-Michel POURE
Date:
Dear all,

A friend of mine needs to import and query a very large amount of data, coming
from real-time acquisition systems. The database is growing fast, several
Tetra-bytes a day.

What is the advancement of the community in the field of very large databases?
Could you point out to me some useful information, techdocs, etc...?

Are there working groups, private fundings in this precise field?

Also, in Brussels, at FOSDEM, I met a very brilliant Russian student
interested in large indexing technologies. Could he possibly contact me?

Best regards,
Jean-Michel POURE

Re: [HACKERS] Tetra-bytes database / large indexes needs

From
Hannu Krosing
Date:
Jean-Michel POURE kirjutas N, 22.05.2003 kell 11:38:
> Dear all,
>
> A friend of mine needs to import and query a very large amount of data, coming
> from real-time acquisition systems.

What kind of querying does he need ?

You could check out Telegraph for continuous queries

http://telegraph.cs.berkeley.edu/

>  The database is growing fast, several
> Tetra-bytes a day.

if you mean terabytes TB then 1 TB/day ~= 12.7 MB/sec, just about as
fast as you can write on an average ide drive if you do nothing else, or
what can come in over a 100 base T ethernet.

even in ide disks (the biggest available disks apiece) you have to add 5
disks a day just to store the incoming 1TB/day.

> What is the advancement of the community in the field of very large databases?
> Could you point out to me some useful information, techdocs, etc...?

no ;)

> Are there working groups, private fundings in this precise field?

dunno..


You are possibly on the verge of impossible ?

If you can hardly write down the data due to physical constraints
(memory and bus speeds) , it is very hard to also query it, unless you
deploy some terribly clever techniques which extract and compress your
data before it even gets to your DB ;)

Perhaps you (or your friend) are after filtering and not queries at all
?

--------------
Hannu


Re: [HACKERS] Tetra-bytes database / large indexes needs

From
Sailesh Krishnamurthy
Date:
>>>>> "Hannu" == Hannu Krosing <hannu@tm.ee> writes:


    Hannu> You are possibly on the verge of impossible ?

Eeep .. it's because of his data rates that I stayed quiet :-)

--
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh


Re: [HACKERS] Tetra-bytes database / large indexes needs

From
Antonis Antoniou
Date:

Jean-Michel POURE wrote:

>Dear all,
>
>A friend of mine needs to import and query a very large amount of data, coming
>from real-time acquisition systems. The database is growing fast, several
>Tetra-bytes a day.
>
>What is the advancement of the community in the field of very large databases?
>Could you point out to me some useful information, techdocs, etc...?
>
>Are there working groups, private fundings in this precise field?
>
>Also, in Brussels, at FOSDEM, I met a very brilliant Russian student
>interested in large indexing technologies. Could he possibly contact me?
>
>
>

I don't know why but as I was reading this I remembered Frank Sinatra's
song "Fly Me To The Moon".

Fly me to the moon
Let me play among the stars
Let me see what spring is like
On a-Jupiter and Mars
In other words, hold my Tera_hand
In other words, baby, Tera_kiss me

:-)