Thread: Experience wanted: low-cost Photo-Archive in TB range
Hello We are planning and offering a low-cost Photo-Archive in TB range. Is there any experience available with that amount of Data (1-3 TB)? What kind of machines (CPU, RAM)? What kind of I/O-System? What kind of Network (100Mbit Ethernet, Fibre, etc.). What kind of backup-system? How to backup that amount of data, etc. Any comment is welcome! Regards Oli ------------------------------------------------------- Oli Sennhauser Database-Engineer (Oracle & PostgreSQL) Rebenweg 6 CH - 8610 Uster / Switzerland Phone (+41) 1 940 24 82 or Mobile (+41) 79 450 49 14 e-Mail oli.sennhauser@bluewin.ch Website http://mypage.bluewin.ch/shinguz/PostgreSQL/ Secure (signed/encrypted) e-Mail with a Free Personal SwissSign ID: http://www.swisssign.ch Import the SwissSign Root Certificate: http://swisssign.net/cgi-bin/trust/import
Attachment
W liście z pią, 28-11-2003, godz. 13:51, Oli Sennhauser pisze: > Hello > > We are planning and offering a low-cost Photo-Archive in TB range. Is > there any experience available with that amount of Data (1-3 TB)? > What kind of machines (CPU, RAM)? What kind of I/O-System? What kind of > Network (100Mbit Ethernet, Fibre, etc.). > What kind of backup-system? How to backup that amount of data, etc. Well, you want to keep pictures inside database, as a blobs. Maybe it would be better to hold pictures on disk, on hashed directories ( eg spring_picture.jpg as a s/p/r/i/spring_picture.jpg faster access when there is a lot of files ) and in the database hold only file_path to this images, and some other data. Then file_path would be taken from database, written to html page, and the image would be taken from disk. Would it be fast enough, and less demanding from database and machine ? greetings Marek > > Any comment is welcome! > > Regards Oli > > ------------------------------------------------------- > > Oli Sennhauser > Database-Engineer (Oracle & PostgreSQL) > Rebenweg 6 > CH - 8610 Uster / Switzerland > > Phone (+41) 1 940 24 82 or Mobile (+41) 79 450 49 14 > e-Mail oli.sennhauser@bluewin.ch > Website http://mypage.bluewin.ch/shinguz/PostgreSQL/ > > Secure (signed/encrypted) e-Mail with a Free Personal SwissSign ID: http://www.swisssign.ch > > Import the SwissSign Root Certificate: http://swisssign.net/cgi-bin/trust/import >
> Hello > > We are planning and offering a low-cost Photo-Archive in TB range. Is > there any experience available with that amount of Data (1-3 TB)? > What kind of machines (CPU, RAM)? What kind of I/O-System? What kind of > Network (100Mbit Ethernet, Fibre, etc.). > What kind of backup-system? How to backup that amount of data, etc. Well, one can generalize a bit. You don't talk about the economics at all, so I can't help you there. Assuming one is providing access to small chunks of data (generally only a meg or two) via metadata about it (account, name, flag, permissions, maybe other things) then you can generalize a bit. Best practice for large storage arrays is proprietary SAN storage critter or some sort. EMC, Netapp, etc. Since you're asking this on a Postgres list, I assume you don't want to go that route. Clustering several machines to provide NFS/Coda/AFS could work. Then you don't care so much about the machines themselves, and can buy cheap, and a lot of them. The network is trickier. What sort of access patterns do you expect? I get the feeling you expect lots of access. So, partition storage from delivery. Delivery should be staightforward - web content delivery isn't that hard. At that point you can talk about the network - what do you expect to handle? Where do you need caches? How do you handle failover? Backup is business specific. This is a recovery question - what is the damage if data is lost vs. what are you willing to spend? There really isn't a best practice here, because cost gets in the way. As far as structuring the data, well, you don't ask about that, and don't talk about the application at all. Blobs might be in order; you might want to manage metadata in the database and files on a filesystem; it depends on what you want to do with it. If you want to provide more info, I'll chat with you - I've done large storage networks before. You have to provide more info to get better answers. I gvague hint that you're not going to spend much money on it, and that's a mistake. Reliable terabyte-scale installations are expensive. -j -- Jamie Lawrence jal@jal.org It it ain't broke, let me have a shot at it.