Thread: Does derby have an embedded Mode like Derby ?
Hi Does postgres have an embedded mode to allow a database to be embedded with Java application without requiring seperate db manager, like the derby database does ? Paul
On Tue, Aug 04, 2009 at 01:37:34PM +0100, Paul Taylor wrote: > Does postgres have an embedded mode to allow a database to be embedded > with Java application without requiring seperate db manager, like the > derby database does ? No, and I think the consensus is that this would be bad. See: http://wiki.postgresql.org/wiki/FAQ#Can_PostgreSQL_be_embedded.3F -- Sam http://samason.me.uk/
Sam Mason wrote: > On Tue, Aug 04, 2009 at 01:37:34PM +0100, Paul Taylor wrote: > >> Does postgres have an embedded mode to allow a database to be embedded >> with Java application without requiring seperate db manager, like the >> derby database does ? >> > > No, and I think the consensus is that this would be bad. See: > > http://wiki.postgresql.org/wiki/FAQ#Can_PostgreSQL_be_embedded.3F > > Thats a shame, I wanted to write junit test for a java program that queried a database, requiring a full database to be available for unit tests is not really an environment I want to have. Paul
In response to Paul Taylor <ijabz@fastmail.fm>: > Sam Mason wrote: > > On Tue, Aug 04, 2009 at 01:37:34PM +0100, Paul Taylor wrote: > > > >> Does postgres have an embedded mode to allow a database to be embedded > >> with Java application without requiring seperate db manager, like the > >> derby database does ? > > > > No, and I think the consensus is that this would be bad. See: > > > > http://wiki.postgresql.org/wiki/FAQ#Can_PostgreSQL_be_embedded.3F > > Thats a shame, I wanted to write junit test for a java program that > queried a database, requiring a full database to be available for unit > tests is not really an environment I want to have. Has it occurred to you that testing a DB client when there's no DB isn't really a very accurate or realistic test? -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/
Bill Moran wrote: > In response to Paul Taylor <ijabz@fastmail.fm>: > > >> Sam Mason wrote: >> >>> On Tue, Aug 04, 2009 at 01:37:34PM +0100, Paul Taylor wrote: >>> >>> >>>> Does postgres have an embedded mode to allow a database to be embedded >>>> with Java application without requiring seperate db manager, like the >>>> derby database does ? >>>> >>> No, and I think the consensus is that this would be bad. See: >>> >>> http://wiki.postgresql.org/wiki/FAQ#Can_PostgreSQL_be_embedded.3F >>> >> Thats a shame, I wanted to write junit test for a java program that >> queried a database, requiring a full database to be available for unit >> tests is not really an environment I want to have. >> > > Has it occurred to you that testing a DB client when there's no > DB isn't really a very accurate or realistic test? > > I am testing the code that extracts information from a read only database. These are UNIT tests so only interested in getting the right results given a particular set of data, anything else is a distraction. Paul
Paul Taylor, 04.08.2009 15:48: > Thats a shame, I wanted to write junit test for a java program that > queried a database, requiring a full database to be available for unit > tests is not really an environment I want to have. Well if you want to test the database, you will need a full database. But you can easily setup and start a Postgres server through a batch file/shell script. I have a small batch file that simply uses the binary ZIP distribution to run initdb and create the initial databases. Weare using this to bundle Postgres e.g. with a Tomcat web application. You don't really need to run an installer and/or create registry entries (for windows). This would then resemble more theDerby network server setup. Thomas
In response to Paul Taylor <ijabz@fastmail.fm>: > Bill Moran wrote: > > In response to Paul Taylor <ijabz@fastmail.fm>: > > > >> Sam Mason wrote: > >> > >>> On Tue, Aug 04, 2009 at 01:37:34PM +0100, Paul Taylor wrote: > >>> > >>> > >>>> Does postgres have an embedded mode to allow a database to be embedded > >>>> with Java application without requiring seperate db manager, like the > >>>> derby database does ? > >>>> > >>> No, and I think the consensus is that this would be bad. See: > >>> > >>> http://wiki.postgresql.org/wiki/FAQ#Can_PostgreSQL_be_embedded.3F > >>> > >> Thats a shame, I wanted to write junit test for a java program that > >> queried a database, requiring a full database to be available for unit > >> tests is not really an environment I want to have. > > > > Has it occurred to you that testing a DB client when there's no > > DB isn't really a very accurate or realistic test? > > I am testing the code that extracts information from a read only > database. These are UNIT tests so only interested in getting the right > results given a particular set of data, anything else is a distraction. Then replace the DB client class with a class that returns fabricated data for the purpose of your test. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/
> Paul Taylor, 04.08.2009 15:48: >> Thats a shame, I wanted to write junit test for a java program that >> queried a database, requiring a full database to be available for unit >> tests is not really an environment I want to have. > > Well if you want to test the database, you will need a full database. > > But you can easily setup and start a Postgres server through a batch > file/shell script. I have a small batch file that simply uses the binary > ZIP distribution to run initdb and create the initial databases. We are > using this to bundle Postgres e.g. with a Tomcat web application. > You don't really need to run an installer and/or create registry entries > (for windows). This would then resemble more the Derby network server > setup. > > Thomas > > Of course you can always use Derby for testing the Postgres later. I have found HSQLDB to be closer to Postgres than Derby is. I do seem to recall the Netbeans group recommends using sever mode during development then switch to imbedded for distribution. I have done it both ways with equal success and similar efforts. Bayless
Bayless Kirtley wrote: > > > >> Paul Taylor, 04.08.2009 15:48: >>> Thats a shame, I wanted to write junit test for a java program that >>> queried a database, requiring a full database to be available for >>> unit tests is not really an environment I want to have. >> >> Well if you want to test the database, you will need a full database. >> >> But you can easily setup and start a Postgres server through a batch >> file/shell script. I have a small batch file that simply uses the >> binary ZIP distribution to run initdb and create the initial >> databases. We are using this to bundle Postgres e.g. with a Tomcat >> web application. >> You don't really need to run an installer and/or create registry >> entries (for windows). This would then resemble more the Derby >> network server setup. >> >> Thomas >> >> > Of course you can always use Derby for testing the Postgres later. I > have found HSQLDB to be closer to Postgres than Derby is. I do seem to > recall the Netbeans group recommends using sever mode during > development then switch to imbedded for distribution. I have done it > both ways with equal success and similar efforts. > > Bayless > > Thanks I am trying to use Derby but fallen foul of Postgres array datatype which Derby doesn't support, Ive just had a quick look at HSQLDB and it doesn't appear to support it either. Paul
Thomas Kellerer wrote: > Paul Taylor, 04.08.2009 15:48: >> Thats a shame, I wanted to write junit test for a java program that >> queried a database, requiring a full database to be available for >> unit tests is not really an environment I want to have. > > Well if you want to test the database, you will need a full database. It was a simple question, does Postgres have an embedded mode (which is still a full database) but unfortunately it does not. > > But you can easily setup and start a Postgres server through a batch > file/shell script. I have a small batch file that simply uses the > binary ZIP distribution to run initdb and create the initial > databases. We are using this to bundle Postgres e.g. with a Tomcat web > application. > You don't really need to run an installer and/or create registry > entries (for windows). This would then resemble more the Derby network > server setup. Yeah, but this is messy and low because you have to wait the database to be fully initilized before running the test. And for unit tests you should be start the database for each test, but wouldnt be possible with your method it would be to slow. Furthermore this is a Windows only solution, there would be additional work to do to setup for Mac or Linux thanks anyway Paul > >
Bill Moran wrote: > In response to Paul Taylor <ijabz@fastmail.fm>: > > >> Bill Moran wrote: >> >>> In response to Paul Taylor <ijabz@fastmail.fm>: >>> >>> >>>> Sam Mason wrote: >>>> >>>> >>>>> On Tue, Aug 04, 2009 at 01:37:34PM +0100, Paul Taylor wrote: >>>>> >>>>> >>>>> >>>>>> Does postgres have an embedded mode to allow a database to be embedded >>>>>> with Java application without requiring seperate db manager, like the >>>>>> derby database does ? >>>>>> >>>>>> >>>>> No, and I think the consensus is that this would be bad. See: >>>>> >>>>> http://wiki.postgresql.org/wiki/FAQ#Can_PostgreSQL_be_embedded.3F >>>>> >>>>> >>>> Thats a shame, I wanted to write junit test for a java program that >>>> queried a database, requiring a full database to be available for unit >>>> tests is not really an environment I want to have. >>>> >>> Has it occurred to you that testing a DB client when there's no >>> DB isn't really a very accurate or realistic test? >>> >> I am testing the code that extracts information from a read only >> database. These are UNIT tests so only interested in getting the right >> results given a particular set of data, anything else is a distraction. >> > > Then replace the DB client class with a class that returns fabricated > data for the purpose of your test. > Won't work because I am writing SQL and I want to test the SQL is correct
Paul Taylor wrote on 04.08.2009 17:04: >> Well if you want to test the database, you will need a full database. > It was a simple question, does Postgres have an embedded mode (which is > still a full database) but unfortunately it does not. I'm just trying to help get your test environment into proper shape. If you are not open for other solutions, then ignore the rest of this email > Yeah, but this is messy and low because you have to wait the database to > be fully initilized before running the test. Depends on what you mean with "initialize". You only need to run initdb once. After that you can e.g. use DbUnit to populate the tables needed for testing. If you do that, you can simply have an empty database cluster directory that you unzip before running the tests suite. > And for unit tests you > should be start the database for each test, but wouldnt be possible with > your method it would be to slow. Well if you use something like DbUnit, you can simply leave the postmaster running and re-initialize the tables for each test (something you would need to do with Derby as well). > Furthermore this is a Windows only > solution, there would be additional work to do to setup for Mac or Linux No I don't think it's windows only. The batch file simply calls the already existings commandline programs from the Postgres bin directory. They names are identical across all platforms. The only thing would be the zipped database cluster as that is platform specific. Btw: I find the H2 database a lot closer to Postgres than HSQLDB. The author is actively trying to add compatibility levels for various DBMS. It might be worth looking into that as well. But if you e.g. have triggers or stored procedures you'll have to use "the real thing". We have e.g. unit test for one of our application that we ran on MySQL just for the fun of it, and it failed miserable due to the different locking behaviour between Postgres and MySQL. I would expect similar problems with heavy transactional tests with any embedded engine as well. Thomas
On Tue, Aug 04, 2009 at 04:04:42PM +0100, Paul Taylor wrote: > >You don't really need to run an installer and/or create registry > >entries (for windows). This would then resemble more the Derby > >network server setup. > Yeah, but this is messy and low because you have to wait the > database to be fully initilized before running the test. And for > unit tests you should be start the database for each test, but > wouldnt be possible with your method it would be to slow. One could drop and re-create the databases as needed. No need to re-initdb the cluster or even stop the PostgreSQL process. That wouldn't be slow. Karsten -- GPG key ID E4071346 @ wwwkeys.pgp.net E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346
On Tue, Aug 4, 2009 at 8:37 AM, Paul Taylor<ijabz@fastmail.fm> wrote: > Bill Moran wrote: >> >> In response to Paul Taylor <ijabz@fastmail.fm>: >> >> >>> >>> Bill Moran wrote: >>> >>>> >>>> In response to Paul Taylor <ijabz@fastmail.fm>: >>>> >>>> >>>>> >>>>> Sam Mason wrote: >>>>> >>>>>> >>>>>> On Tue, Aug 04, 2009 at 01:37:34PM +0100, Paul Taylor wrote: >>>>>> >>>>>>> >>>>>>> Does postgres have an embedded mode to allow a database to be >>>>>>> embedded with Java application without requiring seperate db manager, like >>>>>>> the derby database does ? >>>>>>> >>>>>> >>>>>> No, and I think the consensus is that this would be bad. See: >>>>>> >>>>>> http://wiki.postgresql.org/wiki/FAQ#Can_PostgreSQL_be_embedded.3F >>>>>> >>>>> >>>>> Thats a shame, I wanted to write junit test for a java program that >>>>> queried a database, requiring a full database to be available for unit tests >>>>> is not really an environment I want to have. >>>>> >>>> >>>> Has it occurred to you that testing a DB client when there's no >>>> DB isn't really a very accurate or realistic test? >>>> >>> >>> I am testing the code that extracts information from a read only >>> database. These are UNIT tests so only interested in getting the right >>> results given a particular set of data, anything else is a distraction. >>> >> >> Then replace the DB client class with a class that returns fabricated >> data for the purpose of your test. >> > > Won't work because I am writing SQL and I want to test the SQL is correct > > Can you do what you need to in a virtual machine? I use several virtual machines in my development environment. One for cranking out code. One as a "clean" test environment. > -- > Sent via pgsql-general mailing list (pgsql-general@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-general >
>Thats a shame, I wanted to write junit test for a java program that queried a database, requiring a full database to beavailable for unit tests is not really an >environment I want to have. Why not? if you doesn't test the real thing you test is meaningless. you should test the real database your are going to use with a setup as similar to the deployment setup you are going to use in production.
In response to Paul Taylor <ijabz@fastmail.fm>: > Bill Moran wrote: > > > > Then replace the DB client class with a class that returns fabricated > > data for the purpose of your test. > > > Won't work because I am writing SQL and I want to test the SQL is correct Well, be warned that not all alternatives to PostgreSQL will have the same SQL compliance as Postgres ... so substituting another db backend is going to be less than reliable. I hope you don't take these next comments as an attack or anything, but I think your whole approach to testing is flawed. The fact that tests are complicated to set up and take a while to run is just life. I mean, who cares if they take a while to run? Computers don't need to sleep, so have them run overnight. And any test that's actually comprehensive is going to take effort to write anyway. Thing is, you'll only be setting up the PG startup/teardown stuff once, then any test that needs it can call those functions/scripts/whatever. If you need more computers or a different OS, then virtualize. All the tools are there for you. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/
Bill Moran wrote: > In response to Paul Taylor <ijabz@fastmail.fm>: > > >> Bill Moran wrote: >> >>> Then replace the DB client class with a class that returns fabricated >>> data for the purpose of your test. >>> >>> >> Won't work because I am writing SQL and I want to test the SQL is correct >> > > Well, be warned that not all alternatives to PostgreSQL will have the > same SQL compliance as Postgres ... so substituting another db backend > is going to be less than reliable. > > I hope you don't take these next comments as an attack or anything, but > I think your whole approach to testing is flawed. The fact that tests > are complicated to set up and take a while to run is just life. I mean, > who cares if they take a while to run? Computers don't need to sleep, so > have them run overnight. And any test that's actually comprehensive is > going to take effort to write anyway. Thing is, you'll only be setting > up the PG startup/teardown stuff once, then any test that needs it can > call those functions/scripts/whatever. If you need more computers or a > different OS, then virtualize. All the tools are there for you. > > No, I don't take it personally but I think you are missing the point these are UNIT test not INTEGRATION tests. As such they need to be run everytime a developer wants to commit some changes to the codebase and therefore should be quick and unobstrusive to run. Paul
In response to Paul Taylor <ijabz@fastmail.fm>: > Bill Moran wrote: > > In response to Paul Taylor <ijabz@fastmail.fm>: > > > >> Bill Moran wrote: > >> > >>> Then replace the DB client class with a class that returns fabricated > >>> data for the purpose of your test. > >>> > >>> > >> Won't work because I am writing SQL and I want to test the SQL is correct > >> > > > > Well, be warned that not all alternatives to PostgreSQL will have the > > same SQL compliance as Postgres ... so substituting another db backend > > is going to be less than reliable. > > > > I hope you don't take these next comments as an attack or anything, but > > I think your whole approach to testing is flawed. The fact that tests > > are complicated to set up and take a while to run is just life. I mean, > > who cares if they take a while to run? Computers don't need to sleep, so > > have them run overnight. And any test that's actually comprehensive is > > going to take effort to write anyway. Thing is, you'll only be setting > > up the PG startup/teardown stuff once, then any test that needs it can > > call those functions/scripts/whatever. If you need more computers or a > > different OS, then virtualize. All the tools are there for you. > > No, I don't take it personally but I think you are missing the point > these are UNIT test not INTEGRATION tests. I don't feel like I've missed the point, but I wonder if your definition of "unit" test includes something that I'm not familiar with. We do extensive unit tests that require a DB backend to work. I mean, if you're unit testing the DB client class, it needs a back end to talk to. > As such they need to be run > everytime a developer wants to commit some changes to the codebase and > therefore should be quick and unobstrusive to run. And I don't see how that precludes having a DB there that the developer can connect to for testing. We have a DB server specifically for this purpose so that devs can use it to test out their changes before checking code in. More specifically, we have one virtual machine that serves as a dedicated DB server for developer testing. I mean, if it won't work for you, OK. I'm just offering suggestions, and honestly am confused as to what your actual holdup is. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/
Paul Taylor wrote: > I am testing the code that extracts information from a read only > database. These are UNIT tests so only interested in getting the right > results given a particular set of data, anything else is a distraction. I don't understand your test environment. If all your code is doing is extracting info from a database, why would you want to have database management embedded in your test? Simply have a test database already configured and running, then run your JUnit tests against that existing database. That simple approach would obviously work across all platforms. The only reason I can see for doing DDL in a unit test would be if the end product will be doing such DDL. -- Guy Rouillier
----- Original Message ----- From: "Paul Taylor" <ijabz@fastmail.fm> To: <pgsql-general@postgresql.org> Sent: Tuesday, August 04, 2009 9:56 AM Subject: Re: [GENERAL] Does derby have an embedded Mode like Derby ? >> Of course you can always use Derby for testing the Postgres later. I have >> found HSQLDB to be closer to Postgres than Derby is. I do seem to recall >> the Netbeans group recommends using sever mode during development then >> switch to imbedded for distribution. I have done it both ways with equal >> success and similar efforts. >> >> Bayless >> >> > Thanks I am trying to use Derby but fallen foul of Postgres array > datatype which Derby doesn't support, Ive just had a quick look at HSQLDB > and it doesn't appear to support it either. > > Paul > > Yes, I'm afraid you're gonna be out of luck on finding the array type in any of the smaller embedded databases. Honestly, the beg project I've been on for a year or so has used Postgres right through full development and testing. It's not hard to start and stop the database whenever I need to although I usually just leave it running. Bayless
Bayless Kirtley wrote: > Yes, I'm afraid you're gonna be out of luck on finding the array type > in any > of > the smaller embedded databases. Honestly, the beg project I've been on > for a > year or so has used Postgres right through full development and testing. > It's > not hard to start and stop the database whenever I need to although I > usually > just leave it running. > Even if you find one supporting all the same SQL syntax that you are using, another database implementation could, in some rare circumstances, return slightly different data for the same query. In particular odd situations like this arise around nulls. Conceivably this could happen between versions of PostgreSQL and running such unit tests to test your code's expectations against a real database guards against this possibility. Eric
Guy Rouillier wrote: > Paul Taylor wrote: >> I am testing the code that extracts information from a read only >> database. These are UNIT tests so only interested in getting the >> right results given a particular set of data, anything else is a >> distraction. > > I don't understand your test environment. If all your code is doing > is extracting info from a database, why would you want to have > database management embedded in your test? Simply have a test > database already configured and running, then run your JUnit tests > against that existing database. That simple approach would obviously > work across all platforms. > > The only reason I can see for doing DDL in a unit test would be if the > end product will be doing such DDL. > Firstly the database side is quite straightforward, these issues with triggers and locking just don't apply for what I am doing. Secondly this is an opensource project and to enable others to contribute easily it is much easier if they can download the code and run mvn package to compile and test. Once you start introducing external database setups, and database configs things can easily start going wrong, and you can't share databases when doing automated testing Paul
In response to Paul Taylor <paul_t100@fastmail.fm>: > Guy Rouillier wrote: > > Paul Taylor wrote: > >> I am testing the code that extracts information from a read only > >> database. These are UNIT tests so only interested in getting the > >> right results given a particular set of data, anything else is a > >> distraction. > > > > I don't understand your test environment. If all your code is doing > > is extracting info from a database, why would you want to have > > database management embedded in your test? Simply have a test > > database already configured and running, then run your JUnit tests > > against that existing database. That simple approach would obviously > > work across all platforms. > > > > The only reason I can see for doing DDL in a unit test would be if the > > end product will be doing such DDL. > > > Firstly the database side is quite straightforward, these issues with > triggers and locking just don't apply for what I am doing. Secondly this > is an opensource project and to enable others to contribute easily it is > much easier if they can download the code and run mvn package to compile > and test. Once you start introducing external database setups, and > database configs things can easily start going wrong, and you can't > share databases when doing automated testing Gonna have to disagree yet again. As an example, the Bacula project requires a database to run, and has a full suite of testing stuff that multiple people run to help find bugs. The thing that makes it doable is the fact that the setup process is documented well enough that anyone who can follow instructions can set up a testing system. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/
On Wed, Aug 05, 2009 at 08:02:13AM -0400, Bill Moran wrote: > In response to Paul Taylor <paul_t100@fastmail.fm>: > > this > > is an opensource project and to enable others to contribute easily it is > > much easier if they can download the code and run mvn package to compile > > and test. Once you start introducing external database setups, and > > database configs things can easily start going wrong, and you can't > > share databases when doing automated testing > > Gonna have to disagree yet again. Yup, I'm wondering about it as well. Surely if it's an open source project, people are going to have a database setup to run the thing with anyway--if only to test it? For build/test farms it's going to make it a bit more complex yes, but wouldn't it be easier to configure each machine separately (or write a set of scripts to do the common setup if you have 20+ test boxes!) than to bake in a large set of assumptions into your test scripts? -- Sam http://samason.me.uk/
Sam Mason wrote: > On Wed, Aug 05, 2009 at 08:02:13AM -0400, Bill Moran wrote: > >> In response to Paul Taylor <paul_t100@fastmail.fm>: >> >>> this >>> is an opensource project and to enable others to contribute easily it is >>> much easier if they can download the code and run mvn package to compile >>> and test. Once you start introducing external database setups, and >>> database configs things can easily start going wrong, and you can't >>> share databases when doing automated testing >>> >> Gonna have to disagree yet again. >> > > Yup, I'm wondering about it as well. Surely if it's an open source > project, people are going to have a database setup to run the thing with > anyway--if only to test it? > > For build/test farms it's going to make it a bit more complex yes, but > wouldn't it be easier to configure each machine separately (or write a > set of scripts to do the common setup if you have 20+ test boxes!) than > to bake in a large set of assumptions into your test scripts? > > Ok, the original question was Postgres have an embedded mode. If it did then everything could be contained with the application with no scripts required AND no assumptions would be made about the database because the same database would be running, this is the ideal scenario for me - and I can't see any disadvantage in it. If instead you have to run a database standalone, then you do hit configurations problems, not only platform specific issues but also people bloody mindness about creating databases with different names and database users : whatever the documentation says which has to be accounted for in tests. To pretend there are no issues with setting up a database is unrealistic. Alternatively , use an embedded database of a different type but then the syntax would be different plus many other parts of the database which may/may not be relevant. So neither solution is great, for more complex applications would have to go with standalone database, but for simpler database interactions the second option is viable. Paul
On Wed, Aug 05, 2009 at 02:45:20PM +0100, Paul Taylor wrote: > Ok, the original question was Postgres have an embedded mode. If it did > then everything could be contained with the application with no scripts > required AND no assumptions would be made about the database because the > same database would be running, this is the ideal scenario for me - and > I can't see any disadvantage in it. Yes, it would make some common things easier. If this were true in general we wouldn't have operating systems on our computers, every program would run "on the metal" and do what it wanted to do. In practice having a multitasking operating system is actually quite nice, just like having a full database server is often nice. It makes some things more complicated, and others easier. > If instead you have to run a database standalone, then you do hit > configurations problems, not only platform specific issues but also > people bloody mindness about creating databases with different names and > database users : whatever the documentation says which has to be > accounted for in tests. To pretend there are no issues with setting up a > database is unrealistic. No, I'm not trying to say that. I'm just trying to explain the position that PG comes from. > Alternatively , use an embedded database of a different type but then > the syntax would be different plus many other parts of the database > which may/may not be relevant. > > So neither solution is great, for more complex applications would have > to go with standalone database, but for simpler database interactions > the second option is viable. Yup, but then you've got to support multiple database backends which can get tricky and both still really need to be tested. If both were PG then this would simplify your end, but the development of PG would be complicated a lot. I think I'm saying that, yes, it's going to be more of a fiddle that PG doesn't have an embedded mode, but unfortunately that's the way things are. We can try and suggest ways to make writing your tests easier, but I'm not sure what else we can do. -- Sam http://samason.me.uk/
paul_t100@fastmail.fm (Paul Taylor) writes: > If instead you have to run a database standalone, then you do hit > configurations problems, not only platform specific issues but also > people bloody mindness about creating databases with different names > and database users : whatever the documentation says which has to be > accounted for in tests. To pretend there are no issues with setting up > a database is unrealistic. Actually, it seems to me that these issues are there, regardless. I have been setting up regression tests lately :-), and what I've had to do is to set up scripts to manage setting up databases. I haven't gone so far as demanding to start out with "initdb"; I assume the presence of a functioning cluster. Even with that, since there are ~4 databases to set up, I've got scripts filled with stuff like: (Not literally excerpted; this is an excerpt from... http://main.slony.info/viewcvs/viewvc.cgi/slony1-engine/tests/settings.ik?view=log) DB1=${DB1:-"slonyregress1"} HOST1=${HOST1:-"localhost"} USER1=${USER1:-${PGUSER:-"postgres"}} WEAKUSER1=${WEAKUSER1:-${WEAKUSER:-${PGUSER:-"weakuser"}}} PORT1=${PORT1:-${PGPORT:-"5432"}} PGBINDIR1=${PGBINDIR1:-${PGBINDIR:-"/usr/local/pgsql/bin"}} DB2=${DB2:-"slonyregress2"} HOST2=${HOST2:-"localhost"} USER2=${USER2:-${PGUSER:-"postgres"}} WEAKUSER2=${WEAKUSER2:-${WEAKUSER:-${PGUSER:-"weakuser"}}} PORT2=${PORT2:-${PGPORT:-"5432"}} PGBINDIR2=${PGBINDIR2:-${PGBINDIR:-"/usr/local/pgsql/bin"}} DB3=${DB3:-"slonyregress3"} HOST3=${HOST3:-"localhost"} USER3=${USER3:-${PGUSER:-"postgres"}} WEAKUSER3=${WEAKUSER3:-${WEAKUSER:-${PGUSER:-"weakuser"}}} PORT3=${PORT3:-${PGPORT:-"5432"}} PGBINDIR3=${PGBINDIR3:-${PGBINDIR:-"/usr/local/pgsql/bin"}} DB4=${DB4:-"slonyregress4"} HOST4=${HOST4:-"localhost"} USER4=${USER4:-${PGUSER:-"postgres"}} WEAKUSER4=${WEAKUSER4:-${WEAKUSER:-${PGUSER:-"weakuser"}}} PORT4=${PORT4:-${PGPORT:-"5432"}} PGBINDIR4=${PGBINDIR4:-${PGBINDIR:-"/usr/local/pgsql/bin"}} I don't think things would be made notably worse if I included "initdb" in my process. That would add a few extra steps, and the mandate that I set up some configuration for: a) pg_hba.conf b) postgresql.conf c) where to find initdb This would all be somewhat simpler if I were using (say) SQLite. But the burdens aren't spectacularly heavy. And if you choose to run the database in an "embedded-ish" fashion, then you get to be somewhat "bloodyminded" about how your cluster gets set up. I've had the scenario where I had to tell QA (with somewhat more diplomacy than this ;-)): "If you're going to set up databases on the same backend, you can't use the same users I have coded with. If you *DO* get bloody-minded about it, and this causes some inconvenience, then it's right well YOUR problem that you're not following the stipulations documented for the system." I'm not going to try to force *stupid* constraints on people, but it doesn't seem to be "out there" to have some details stipulated. I'll note that in my favorite "common-preferences.sh" file, while practically everything can be configured, if one were so dumb as to ask to do so, enough defaults get inherited (via the shell :- construct) that *USUALLY*, the only values that people need to configure are: a) PGBINDIR, the directory where your favorite PostgreSQL psql client lives, and b) PGPORT, the port where your favorite cluster answers I'm pretty happy with this; *EVERYTHING* may be overridden, if need be, but it is typically sufficient to set those two environment variables and have things "just work." -- output = reverse("ofni.sesabatadxunil" "@" "enworbbc") http://linuxdatabases.info/info/rdbms.html str->str_pok |= SP_FBM; /* deep magic */ s = (unsigned char*)(str->str_ptr); /* deeper magic */ -- Larry Wall in util.c from the perl source code
On 2009-08-04, Paul Taylor <ijabz@fastmail.fm> wrote: > Bill Moran wrote: >> In response to Paul Taylor <ijabz@fastmail.fm>: >> >> >>> Bill Moran wrote: >>> >>>> In response to Paul Taylor <ijabz@fastmail.fm>: >>>> >>>> >>>>> Sam Mason wrote: >>>>> >>>>> >>>>>> On Tue, Aug 04, 2009 at 01:37:34PM +0100, Paul Taylor wrote: >>>>>> >>>>>>> Does postgres have an embedded mode to allow a database to be embedded >>>>>>> with Java application without requiring seperate db manager, like the >>>>>>> derby database does ? >>>>>>> you're looking for an embeddable SQL database written in java? >>>>> Thats a shame, I wanted to write junit test for a java program that >>>>> queried a database, requiring a full database to be available for unit >>>>> tests is not really an environment I want to have. fake it. use canned responses. >>>> Has it occurred to you that testing a DB client when there's no >>>> DB isn't really a very accurate or realistic test? >>>> >>> I am testing the code that extracts information from a read only >>> database. These are UNIT tests so only interested in getting the right >>> results given a particular set of data, anything else is a distraction. >>> >> >> Then replace the DB client class with a class that returns fabricated >> data for the purpose of your test. >> > Won't work because I am writing SQL and I want to test the SQL is correct if you know what SQL it should write compare what is written with what is expected.
On 2009-08-05, Paul Taylor <paul_t100@fastmail.fm> wrote: > Sam Mason wrote: >> On Wed, Aug 05, 2009 at 08:02:13AM -0400, Bill Moran wrote: >> >>> In response to Paul Taylor <paul_t100@fastmail.fm>: >>> >>>> this >>>> is an opensource project and to enable others to contribute easily it is >>>> much easier if they can download the code and run mvn package to compile >>>> and test. Once you start introducing external database setups, and >>>> database configs things can easily start going wrong, and you can't >>>> share databases when doing automated testing >>>> >>> Gonna have to disagree yet again. >>> >> >> Yup, I'm wondering about it as well. Surely if it's an open source >> project, people are going to have a database setup to run the thing with >> anyway--if only to test it? >> >> For build/test farms it's going to make it a bit more complex yes, but >> wouldn't it be easier to configure each machine separately (or write a >> set of scripts to do the common setup if you have 20+ test boxes!) than >> to bake in a large set of assumptions into your test scripts? >> >> > Ok, the original question was Postgres have an embedded mode. If it did > then everything could be contained with the application with no scripts > required AND no assumptions would be made about the database because the > same database would be running, this is the ideal scenario for me - and > I can't see any disadvantage in it. write a virtual machine in java and run postgres in that :^) > If instead you have to run a database standalone, then you do hit > configurations problems, not only platform specific issues but also > people bloody mindness about creating databases with different names and > database users : whatever the documentation says which has to be > accounted for in tests. To pretend there are no issues with setting up a > database is unrealistic. send them a script that sets up the database and users, have the unit test first probe the database, pick names that are likly to be unique. in the rare event that there is a collision or other confilict (the user may have to setp setup a separate cluster for the test database. A "cluster" here is a software entitiy, not a real or virtual host nor group thereof.