Thread: PostgreSQL + PostGIS - "mining" FGDC metadata of GIS datasets
Hello, I am looking in to building a web site with PostgreSQL, PostGIS, MapServer, and OpenLayers so users can access an online catalogue of what GIS data is available within an organisation - spatially, temporally and/or with other criteria. Most the existing data in ESRI geodatabases and "so forth"... Any pointers on automating mining the FGDC metadata 'in the backend' from the existing spatial datasets and getting the data into PostgreSQL + PostGIS (I can build the templates I guess but was hoping someone else had (!), both for the text data and the spatial extent data to poke into the geographic component of the record.) The web front end will just be a learning curve I think rather than meeting anything insurmountable. I see as much as possible this system being 'automated' - with "it" 'reaching out' to the geodatabase/s on a regular basis, triggers (sic?) to add new metadata details, sync any changes, etc.... Note that at this stage the database would NOT be handling the GIS data itself, just its metadata (the hopefully the GIS managers and analysts will keep maintained using ESRI's tools in the geodatabase/s...) I am new to a lot (most!) of this, so was hoping for advice on: (i) does this seems like a 'reasonable approach'? (and I know I have only described it broadly) (ii) can it be made flexible enough? (esoteric question I know), and (iii) is this a "big job" for me, a relative newbie as mentioned? Thanks in advance... ---------- Regards, GREG COCKS Data Management Specialist / Scientist / GIS Analyst IV gcocks@stoller.com S. M. Stoller Corp 105 Technology Drive, Suite 190 Broomfield, CO 80021 www.stoller.com 303-546-4300 303-443-1408 fax 303-546-4422 direct 303-828-7576 cell
Greg Cocks wrote: > Hello, > > I am looking in to building a web site with PostgreSQL, PostGIS, > MapServer, and OpenLayers so users can access an online catalogue of > what GIS data is available within an organisation - spatially, > temporally and/or with other criteria. > > Most the existing data in ESRI geodatabases and "so forth"... > > I am new to a lot (most!) of this, so was hoping for advice on: > > (i) does this seems like a 'reasonable approach'? (and I know I have > only described it broadly) > > (ii) can it be made flexible enough? (esoteric question I know), and > > (iii) is this a "big job" for me, a relative newbie as mentioned? Hi Greg, I am not sure how big a job it will be. Much of it depends on your other background and experience. However, there are a couple of open source GIS projects already underway that could give you some ideas, or even some code you can start with. As long as you don't plan to sell it as your own invention, most of the OSS licenses would allow you free reign. Who knows, you might get good enough to contribute improvements back to those projects. There are search functions on Google, Sourceforge and Freshmeat that should point you in the right direction. There are also a number of APRS (Amateur Position Reporting Service) systems around. I am not sure what the difference is, but I believe they use a lot of the same information. Bob McConnell N2SPP
Hi Greg, You mentioned that your data is in the "ESRI geodatabase" format, if it is stored in the PostGIS datatype called geometrythen you will be able to use the data in the open source software stack that you are proposing. The data being recognized a geodatabase does not have any bearing on your ability to use it with open source stack, as longas the type in which it resides is understood by PostGIS. Sincerely, Kasia -----Original Message----- From: pgsql-novice-owner@postgresql.org [mailto:pgsql-novice-owner@postgresql.org] On Behalf Of Bob McConnell Sent: Monday, January 19, 2009 4:25 PM To: Greg Cocks Cc: PostgreSQL List - Novice Subject: Re: [NOVICE] PostgreSQL + PostGIS - "mining" FGDC metadata of GIS datasets Greg Cocks wrote: > Hello, > > I am looking in to building a web site with PostgreSQL, PostGIS, > MapServer, and OpenLayers so users can access an online catalogue of > what GIS data is available within an organisation - spatially, > temporally and/or with other criteria. > > Most the existing data in ESRI geodatabases and "so forth"... > > I am new to a lot (most!) of this, so was hoping for advice on: > > (i) does this seems like a 'reasonable approach'? (and I know I have > only described it broadly) > > (ii) can it be made flexible enough? (esoteric question I know), and > > (iii) is this a "big job" for me, a relative newbie as mentioned? Hi Greg, I am not sure how big a job it will be. Much of it depends on your other background and experience. However, there are a couple of open source GIS projects already underway that could give you some ideas, or even some code you can start with. As long as you don't plan to sell it as your own invention, most of the OSS licenses would allow you free reign. Who knows, you might get good enough to contribute improvements back to those projects. There are search functions on Google, Sourceforge and Freshmeat that should point you in the right direction. There are also a number of APRS (Amateur Position Reporting Service) systems around. I am not sure what the difference is, but I believe they use a lot of the same information. Bob McConnell N2SPP -- Sent via pgsql-novice mailing list (pgsql-novice@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-novice
Kasia, Thank you... Sorry for not explaining myself very well... The data is in the ESRI geodatabase format using the FGDC-compliant metadata tools that are part of 'your' software to generate the metadata - that is, already done and will be maintained by ESRI software users... I want to write some 'code' to loop through all of these metadata 'components' of the geodatabase, "mine" it, and put the textual data into a PostgreSQL data table (via a template) - including putting the matching spatial data *extents* from the metadata into a simple PostGIS 'rectangle' that is part each record... On the web page the user could then use a web map to see what spatial data is available in their area of interest and get info about it - say with a topo map, NAIP imagery, Google imagery, whatever to allow them to orientate themselves... I would like the process, once established, to be as automated as possible - that is, if the ESRI-software user changes, adds or deletes its metadata 'components' of the course of their work, then the PostgreSQL table/'app' would "see" these change/s and 'automagically' change its own contents to sync... (aka a nightly 'batch file'?) Later, *maybe*, the web server could show the *actual* data on a web map and/or allow download of the spatial data in some form - but to be honest, why reinvent the wheel, just get a copy of ArcGIS server and set it up! :-) (for now, "the wheel" is not in the budget!) Regards, GREG COCKS gcocks@stoller.com -----Original Message----- From: Kasia Tuszynska [mailto:ktuszynska@esri.com] Sent: Tuesday, January 20, 2009 11:59 AM To: Greg Cocks Cc: PostgreSQL List - Novice Subject: RE: [NOVICE] PostgreSQL + PostGIS - "mining" FGDC metadata of GIS datasets Hi Greg, You mentioned that your data is in the "ESRI geodatabase" format, if it is stored in the PostGIS datatype called geometry then you will be able to use the data in the open source software stack that you are proposing. The data being recognized a geodatabase does not have any bearing on your ability to use it with open source stack, as long as the type in which it resides is understood by PostGIS. Sincerely, Kasia -----Original Message----- From: pgsql-novice-owner@postgresql.org [mailto:pgsql-novice-owner@postgresql.org] On Behalf Of Bob McConnell Sent: Monday, January 19, 2009 4:25 PM To: Greg Cocks Cc: PostgreSQL List - Novice Subject: Re: [NOVICE] PostgreSQL + PostGIS - "mining" FGDC metadata of GIS datasets Greg Cocks wrote: > Hello, > > I am looking in to building a web site with PostgreSQL, PostGIS, > MapServer, and OpenLayers so users can access an online catalogue of > what GIS data is available within an organisation - spatially, > temporally and/or with other criteria. > > Most the existing data in ESRI geodatabases and "so forth"... > > I am new to a lot (most!) of this, so was hoping for advice on: > > (i) does this seems like a 'reasonable approach'? (and I know I have > only described it broadly) > > (ii) can it be made flexible enough? (esoteric question I know), and > > (iii) is this a "big job" for me, a relative newbie as mentioned? Hi Greg, I am not sure how big a job it will be. Much of it depends on your other background and experience. However, there are a couple of open source GIS projects already underway that could give you some ideas, or even some code you can start with. As long as you don't plan to sell it as your own invention, most of the OSS licenses would allow you free reign. Who knows, you might get good enough to contribute improvements back to those projects. There are search functions on Google, Sourceforge and Freshmeat that should point you in the right direction. There are also a number of APRS (Amateur Position Reporting Service) systems around. I am not sure what the difference is, but I believe they use a lot of the same information. Bob McConnell N2SPP -- Sent via pgsql-novice mailing list (pgsql-novice@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-novice
Hi Greg, I found some more information that can be useful to your cause. If I understand your project correctly, you want to extract the metadata from the geodatabase and make it available as informationit's self. ESRI has two products that do that: Metadata Server and Portal Took Kit, they each do it a little differentlyand use different underlying products. So, perhaps their implementations can help you in planning your project.The help for both of those products are freely available on line. As to the fundamental issue of using the metadata stored by the geodatabase, the metadata is ultimately stored as a blobin one of the geodatabase tables found in the database, to extract or read that blob you will need either ArcObject oruse the ArcSDE C api. Below, is a link to a short KB article and a link to a discussion forum on the subject. http://kbplus/kbdoc.asp?dbid=23024 http://forums.esri.com/thread.asp?c=2&f=59&t=110313#314199 Best Wishes, Kasia
VERY much appreciated - thank you! Regards, GREG COCKS gcocks@stoller.com -----Original Message----- From: Kasia Tuszynska [mailto:ktuszynska@esri.com] Sent: Wednesday, January 21, 2009 2:00 PM To: Greg Cocks Cc: PostgreSQL List - Novice Subject: RE: [NOVICE] PostgreSQL + PostGIS - "mining" FGDC metadata of GIS datasets Hi Greg, I found some more information that can be useful to your cause. If I understand your project correctly, you want to extract the metadata from the geodatabase and make it available as information it's self. ESRI has two products that do that: Metadata Server and Portal Took Kit, they each do it a little differently and use different underlying products. So, perhaps their implementations can help you in planning your project. The help for both of those products are freely available on line. As to the fundamental issue of using the metadata stored by the geodatabase, the metadata is ultimately stored as a blob in one of the geodatabase tables found in the database, to extract or read that blob you will need either ArcObject or use the ArcSDE C api. Below, is a link to a short KB article and a link to a discussion forum on the subject. http://kbplus/kbdoc.asp?dbid=23024 http://forums.esri.com/thread.asp?c=2&f=59&t=110313#314199 Best Wishes, Kasia
VERY much appreciated - thank you! Regards, GREG COCKS gcocks@stoller.com -----Original Message----- From: Kasia Tuszynska [mailto:ktuszynska@esri.com] Sent: Wednesday, January 21, 2009 2:00 PM To: Greg Cocks Cc: PostgreSQL List - Novice Subject: RE: [NOVICE] PostgreSQL + PostGIS - "mining" FGDC metadata of GIS datasets Hi Greg, I found some more information that can be useful to your cause. If I understand your project correctly, you want to extract the metadata from the geodatabase and make it available as information it's self. ESRI has two products that do that: Metadata Server and Portal Took Kit, they each do it a little differently and use different underlying products. So, perhaps their implementations can help you in planning your project. The help for both of those products are freely available on line. As to the fundamental issue of using the metadata stored by the geodatabase, the metadata is ultimately stored as a blob in one of the geodatabase tables found in the database, to extract or read that blob you will need either ArcObject or use the ArcSDE C api. Below, is a link to a short KB article and a link to a discussion forum on the subject. http://kbplus/kbdoc.asp?dbid=23024 http://forums.esri.com/thread.asp?c=2&f=59&t=110313#314199 Best Wishes, Kasia