Thread: BIG Data and Perl

I've got a fairly good size database that has in one table around 50,000
records in it.

I'm using a perl script that is located on **another** machine in the same
network to access the DB with.

Once a week I have to do a "SELECT * ...." from one of the tables, get the
data, open another field from disk read in some of that data and finially
write it all together back to disk in small files that can be emailed out.

The query looks like: SELECT * from mytable order by member_id


-- cut --
    $result = $conn->exec("$query");

        $ntuples = $result->ntuples;
        print STDOUT "Total: $ntuples \n\n";

        while ( @row = $result->fetchrow ) {
        do some stuff here...ie, open file and read
    }
-- cut --

Here's the strange part and this could very well be anotherr part of this
script(I've inherited it).

It starts of and processes the first 300-400 rows fast and then gets
slower in time and eventually just quits. It'll run for about 4-6 hours
before it quits.

Any idea what may be going on here?

Thanks

Andy

On Fri, 15 Oct 1999, Andy Lewis wrote:

> -- cut --
>     $result = $conn->exec("$query");
>
>         $ntuples = $result->ntuples;
>         print STDOUT "Total: $ntuples \n\n";
>
>         while ( @row = $result->fetchrow ) {
>         do some stuff here...ie, open file and read
>     }

I've seen similar effects when working with arrays.  I'm not sure since
I'm still new to PostgreSQL but would a foreach loop work.

    foreach @row ( $result->fetchrow ) {
       @columns = @$row;
    }

I seem to remember reading somewhere that foreach's are faster and more
memory conservative. (Been reading too many perl books lately so I can't
say where I saw this.)

Rod
--
Roderick A. Anderson
raanders@altoplanos.net               Altoplanos Information Systems, Inc.
Voice: 208.765.6149                            212 S. 11th Street, Suite 5
FAX: 208.664.5299                                  Coeur d'Alene, ID 83814