Thread: Thanks for Rekall link!

A long time ago, in a galaxy far, far away, Typing80wpm@aol.com wrote:
> I must say one intereting thing. When I downloaded the trial version
> from TheKompany, and asked it to browse a test file in PGSql which I
> loaded with 250,000 rows, it started to read them, and read for a
> long long time (as MSAccess does), but the seemed to get stuck,
> whereas MSAccess is able to browse the entire file.  I must
> experiment more with the demo version from theKompany, and also with
> this free version from the site you gave me.

This sort of problem is characteristic of the use of "array" objects
in graphical toolkits.

Suppose you're populating something with 250K rows, perhaps with a
dozen fields per row.  In such a case, the toolkit is slinging around
3-4 million objects, and having to evaluate which of them are visible
on screen at any given time.

_Any_ kind of inefficiency in the library, or in the use of the
library, can easily lead to rendering turning out really, really
badly.

The X Window system has gotten heavily criticized for speed problems,
commonly with respect to how Mozilla used to work when rendering large
web pages.  Reality was that Mozilla was implemented (this is no
longer true, by the way) atop a platform-independent library called
Rogue Wave which then had a mapping to Motif (which is noted as Not
Everyone's Favorite Graphics Library ;-)) which then rendered things
using X.  The True Problem lay somewhere in that set of layers and,
since several of the layers were pretty inscrutable, it was
essentially impractical to address the performance problem.

Much the same thing took place with the Tcl/Tk application, "cbb"
(Check Book Balancer); the Tk 'array' object got to behave
increasingly badly with increasing thousands of rows.  And changing
one transaction near the top of an account would lead to cascading
balance updates, therefore walking (linear fashion, more than likely
leading to superlinear resource consumption :-() through the rest of
the transactions to update every single balance...

Gigahertz, Gigabytes, and upgrades may overcome that, to some degree,
but it wouldn't be overly surprising if you were hitting some such
unfortunate case.  It might represent something fixed in a later
release of Rekall; it could represent something thorny to resolve.

I would really hate the notion of depending on a GUI to manage
millions of objects in this manner; it is just so easy for it to go
badly.
--
"cbbrowne","@","gmail.com"
http://linuxdatabases.info/info/nonrdbms.html
Rules of the Evil Overlord #10.  "I will not interrogate my enemies in
the inner sanctum  -- a small hotel well outside  my borders will work
just as well." <http://www.eviloverlord.com/>

Christopher Browne <cbbrowne@acm.org> writes:
> A long time ago, in a galaxy far, far away, Typing80wpm@aol.com wrote:
>> I must say one intereting thing. When I downloaded the trial version
>> from TheKompany, and asked it to browse a test file in PGSql which I
>> loaded with 250,000 rows, it started to read them, and read for a
>> long long time (as MSAccess does), but the seemed to get stuck,
>> whereas MSAccess is able to browse the entire file.  I must
>> experiment more with the demo version from theKompany, and also with
>> this free version from the site you gave me.
>
> This sort of problem is characteristic of the use of "array" objects
> in graphical toolkits.
>
> Suppose you're populating something with 250K rows, perhaps with a
> dozen fields per row.  In such a case, the toolkit is slinging around
> 3-4 million objects, and having to evaluate which of them are visible
> on screen at any given time.
>
> _Any_ kind of inefficiency in the library, or in the use of the
> library, can easily lead to rendering turning out really, really
> badly.

I think I took this down something of the wrong road, in explaining
why this turns out badly.

The other "fork" in the road has a big sign marked "Yes, pulling all
the data into the GUI turns out badly.  That's why you shouldn't do
that!!!"

If there are 250K rows, that's probably way more than any user can
cope with working with at any given moment.

If the database grows to have 2.5 million entries, or 25 million
entries, or some billions of entries, it is absolutely the wrong idea
to try to pull it all out and stuff it into some GUI objects.

If the data won't all fit on the screen, you probably ought to wonder
if maybe you need to think about a better way to get at it...
--
(format nil "~S@~S" "cbbrowne" "acm.org")
http://www.ntlug.org/~cbbrowne/sap.html
Rules of the Evil Overlord #78.  "I will not tell my Legions of Terror
"And he must  be taken alive!" The command will be:  ``And try to take
him alive if it is reasonably practical.''"
<http://www.eviloverlord.com/>