Thread: Insert records in the tavke only if they are not exist
Hi, All,
When my application starts up, ot creates some tables and insert records in them.
When the app starts for the second time it should check if the tables and the records in them are exist and skip the process.
Everything is good, except what if I have a connection from 2 different users?
I can run this inside transaction, but will this be enough? Will stating transaction lock the DB and the second user will wait for transaction to complete?
Thank you.
On Friday, December 6, 2024, Igor Korot <ikorot01@gmail.com> wrote:
When my application starts up, ot creates some tables and insert records in them.When the app starts for the second time it should check if the tables and the records in them are exist and skip the process.Everything is good, except what if I have a connection from 2 different users?I can run this inside transaction, but will this be enough? Will stating transaction lock the DB and the second user will wait for transaction to complete?
Seems like letting the create table fail would be a reliable way to determine what is happening. But this procedure overall just seems better avoided. Can’t you just run an installer/updater separate from running the application?
David J.
Hi, Dvid,
On Fri, Dec 6, 2024, 9:55 PM David G. Johnston <david.g.johnston@gmail.com> wrote:
On Friday, December 6, 2024, Igor Korot <ikorot01@gmail.com> wrote:When my application starts up, ot creates some tables and insert records in them.When the app starts for the second time it should check if the tables and the records in them are exist and skip the process.Everything is good, except what if I have a connection from 2 different users?I can run this inside transaction, but will this be enough? Will stating transaction lock the DB and the second user will wait for transaction to complete?Seems like letting the create table fail would be a reliable way to determine what is happening. But this procedure overall just seems better avoided. Can’t you just run an installer/updater separate from running the application?
Interesting idea bout the installer.
Except those tables will contain some additional info about the schema and the application.
So if I create/remove new table, either in my app or from psql the record will need to be inserted/deleted.
Thank you.
David J.
On 12/6/24 19:36, Igor Korot wrote: > Hi, All, > > When my application starts up, ot creates some tables and insert records > in them. > > When the app starts for the second time it should check if the tables > and the records in them are exist and skip the process. Does this also apply to starts after the second time? Is the data expected to change over time? > > Everything is good, except what if I have a connection from 2 different > users? Is it possible to have the app start up without external users connecting? > > I can run this inside transaction, but will this be enough? Will stating > transaction lock the DB and the second user will wait for transaction to > complete? > > Thank you. > -- Adrian Klaver adrian.klaver@aklaver.com
Hi, Adrian,
On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>
> On 12/6/24 19:36, Igor Korot wrote:
> > Hi, All,
> >
> > When my application starts up, ot creates some tables and insert records
> > in them.
> >
> > When the app starts for the second time it should check if the tables
> > and the records in them are exist and skip the process.
>
> Does this also apply to starts after the second time?
Yes, it does.
However, if the new table is created in the meantime - the new record
should be created, because it is not there yet.
>
> Is the data expected to change over time?
Data change is possible.
This is one of the tables:
"CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30)
NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\"
integer" ));";
"CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON
\"sys.abcatfmt\"(\"abf_name\" ASC);";
My understanding is that "INSERT OR IGNORE" will check the unique
index and will not do anything if the recrd
with such data on the index already exists.
>
> >
> > Everything is good, except what if I have a connection from 2 different
> > users?
> Is it possible to have the app start up without external users connecting?
Not sure what you mean here.
Are you asking if the computer can run the app?
Then the answer is NO. App is always started by the user.
Please clarify.
Thank you.
>
> >
> > I can run this inside transaction, but will this be enough? Will stating
> > transaction lock the DB and the second user will wait for transaction to
> > complete?
> >
> > Thank you.
> >
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>
On 12/7/24 09:59, Igor Korot wrote: > Hi, Adrian, > > On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote: >> >> On 12/6/24 19:36, Igor Korot wrote: >>> Hi, All, >>> >>> When my application starts up, ot creates some tables and insert records >>> in them. >>> >>> When the app starts for the second time it should check if the tables >>> and the records in them are exist and skip the process. >> >> Does this also apply to starts after the second time? > > Yes, it does. > > However, if the new table is created in the meantime - the new record > should be created, because it is not there yet. What determines that a new table needs to be created and populated? > >> >> Is the data expected to change over time? > > Data change is possible. > > This is one of the tables: > > "CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30) > NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\" > integer" ));"; > "CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON > \"sys.abcatfmt\"(\"abf_name\" ASC);"; > > My understanding is that "INSERT OR IGNORE" will check the unique It is INSERT ... ON CONFLICT DO NOTHING | UPDATE. > index and will not do anything if the recrd > with such data on the index already exists. So you have the option of either skipping the insert or updating selected fields in the row. > >> >>> >>> Everything is good, except what if I have a connection from 2 different >>> users? >> Is it possible to have the app start up without external users connecting? > > Not sure what you mean here. > > Are you asking if the computer can run the app? > Then the answer is NO. App is always started by the user. > Please clarify. This is going to need a more detailed explanation of what 'app' means. Are you talking about the front end that the user launches or the backend that runs the database or something else? > > Thank you. > >> >>> >>> I can run this inside transaction, but will this be enough? Will stating >>> transaction lock the DB and the second user will wait for transaction to >>> complete? >>> >>> Thank you. >>> >> >> -- >> Adrian Klaver >> adrian.klaver@aklaver.com >> -- Adrian Klaver adrian.klaver@aklaver.com
Hi, Adrian, On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote: > > On 12/7/24 09:59, Igor Korot wrote: > > Hi, Adrian, > > > > On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote: > >> > >> On 12/6/24 19:36, Igor Korot wrote: > >>> Hi, All, > >>> > >>> When my application starts up, ot creates some tables and insert records > >>> in them. > >>> > >>> When the app starts for the second time it should check if the tables > >>> and the records in them are exist and skip the process. > >> > >> Does this also apply to starts after the second time? > > > > Yes, it does. > > > > However, if the new table is created in the meantime - the new record > > should be created, because it is not there yet. > > What determines that a new table needs to be created and populated? Consider the following scenario: 1. Program is installed. 2. Program is started for the first time. 3. My tables are created and populated 4. Program is executed. 5. User closes the program. 6.. Later on the user decides that there is a need for another table (inside psql or any other client) 7. Then the program starts for the second time. At this time all my tables that were created will stay (courtesy of CREATE TABLE IF NOT EXIST). All records that were there are staying unchanged. However, for the table that was made in between the runs will be added Now if the program is installed on 2 different machines and started simultaneously on both - I want to ensure that only 1 set of tables is made and only 1 set of records in them is available Now, the creation/population is done inside a transaction. > > > > >> > >> Is the data expected to change over time? > > > > Data change is possible. > > > > This is one of the tables: > > > > "CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30) > > NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\" > > integer" ));"; > > "CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON > > \"sys.abcatfmt\"(\"abf_name\" ASC);"; > > > > My understanding is that "INSERT OR IGNORE" will check the unique > > It is INSERT ... ON CONFLICT DO NOTHING | UPDATE. > > > index and will not do anything if the recrd > > with such data on the index already exists. Good. I'm using INSERT ON CONFLICT DO NOTHING. > > So you have the option of either skipping the insert or updating > selected fields in the row. > > > > >> > >>> > >>> Everything is good, except what if I have a connection from 2 different > >>> users? > >> Is it possible to have the app start up without external users connecting? > > > > Not sure what you mean here. > > > > Are you asking if the computer can run the app? > > Then the answer is NO. App is always started by the user. > > Please clarify. > > This is going to need a more detailed explanation of what 'app' means. > > Are you talking about the front end that the user launches or the > backend that runs the database or something else? Front-end. This is a C++ app. > > > > > > Thank you. > > > >> > >>> > >>> I can run this inside transaction, but will this be enough? Will stating > >>> transaction lock the DB and the second user will wait for transaction to > >>> complete? > >>> > >>> Thank you. > >>> > >> > >> -- > >> Adrian Klaver > >> adrian.klaver@aklaver.com > >> > > -- > Adrian Klaver > adrian.klaver@aklaver.com >
On 12/7/24 12:17, Igor Korot wrote: > Hi, Adrian, > > On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote: >> >> On 12/7/24 09:59, Igor Korot wrote: >>> Hi, Adrian, >>> >>> On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote: >>>> >>>> On 12/6/24 19:36, Igor Korot wrote: >>>>> Hi, All, >>>>> >>>>> When my application starts up, ot creates some tables and insert records >>>>> in them. >>>>> >>>>> When the app starts for the second time it should check if the tables >>>>> and the records in them are exist and skip the process. >>>> >>>> Does this also apply to starts after the second time? >>> >>> Yes, it does. >>> >>> However, if the new table is created in the meantime - the new record >>> should be created, because it is not there yet. >> >> What determines that a new table needs to be created and populated? > > Consider the following scenario: I did and it made me start twitching. > > 1. Program is installed. > 2. Program is started for the first time. > 3. My tables are created and populated What distinguishes your tables from other users tables? > 4. Program is executed. > 5. User closes the program. > 6.. Later on the user decides that there is a need > for another table (inside psql or any other client) The above is where I started twitching. How do you keep them out of your tables? How do you get these changes to play nice with the existing structure? > 7. Then the program starts for the second time. > > At this time all my tables that were created will stay (courtesy > of CREATE TABLE IF NOT EXIST). You are depending on folks not knowing about DROP TABLE and/or you having thought out the permissions for access thoroughly. > All records that were there are staying unchanged. INSERT/UPDATE against your tables is not a possibility? > However, for the table that was made in between the runs > will be added > > Now if the program is installed on 2 different machines > and started simultaneously on both - I want to ensure that > only 1 set of tables is made and only 1 set of records in them > is available I assume this means they are both pointing at the same instance of a database? This is the part that confuses me. If you are going to allow ad hoc and at will changes how do you know what is actually the correct change? > > Now, the creation/population is done inside a transaction. I'm not sure that a transaction is going to solve the issue I raised above, it will just make one thing happen with no guarantee that it is the correct outcome. >> Are you talking about the front end that the user launches or the >> backend that runs the database or something else? > > Front-end. > This is a C++ app. To me this is the tail wagging the dog. The thought of allowing users to change the database structure and you dealing with it after the fact is just disturbing to me. > >> >> >>> >>> Thank you. -- Adrian Klaver adrian.klaver@aklaver.com
Hi, Adrian, On Sat, Dec 7, 2024 at 5:07 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote: > > On 12/7/24 12:17, Igor Korot wrote: > > Hi, Adrian, > > > > On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote: > >> > >> On 12/7/24 09:59, Igor Korot wrote: > >>> Hi, Adrian, > >>> > >>> On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote: > >>>> > >>>> On 12/6/24 19:36, Igor Korot wrote: > >>>>> Hi, All, > >>>>> > >>>>> When my application starts up, ot creates some tables and insert records > >>>>> in them. > >>>>> > >>>>> When the app starts for the second time it should check if the tables > >>>>> and the records in them are exist and skip the process. > >>>> > >>>> Does this also apply to starts after the second time? > >>> > >>> Yes, it does. > >>> > >>> However, if the new table is created in the meantime - the new record > >>> should be created, because it is not there yet. > >> > >> What determines that a new table needs to be created and populated? > > > > Consider the following scenario: > > I did and it made me start twitching. Good.. > > > > > 1. Program is installed. > > 2. Program is started for the first time. > > 3. My tables are created and populated > > What distinguishes your tables from other users' tables? I am making the tables and naming them with a specific pattern. Now I'm curious - if I start psql and will want to create a table named pg_am, what will happen? I presume psql will produce error saying that the system table with that name exists and the user can't create ome. Am I right? > > > 4. Program is executed. > > 5. User closes the program. > > 6.. Later on the user decides that there is a need > > for another table (inside psql or any other client) > > The above is where I started twitching. > > How do you keep them out of your tables? My tables are named with the specific pattern. Also - see above. > > How do you get these changes to play nice with the existing structure? Again - not sure what you mean here.... > > > 7. Then the program starts for the second time. > > > > At this time all my tables that were created will stay (courtesy > > of CREATE TABLE IF NOT EXIST). > > You are depending on folks not knowing about DROP TABLE and/or you > having thought out the permissions for access thoroughly. If one of my tables will be dropped - it will be re-created. I'm using CREATE TABLE IF NOT EXIST. > > > > All records that were there are staying unchanged. > > INSERT/UPDATE against your tables is not a possibility? It is.. All I;m saying that when the app starts-up, this is done automatically And if the user decides to insert some data - that's on him > > > However, for the table that was made in between the runs > > will be added > > > > Now if the program is installed on 2 different machines > > and started simultaneously on both - I want to ensure that > > only 1 set of tables is made and only 1 set of records in them > > is available > > I assume this means they are both pointing at the same instance of a > database? Correct. > > This is the part that confuses me. > > If you are going to allow ad hoc and at will changes how do you know > what is actually the correct change? I don't. Think about MS ACCESS-like applications. ACCESS creates the internal tables to keep track of some internals. Is there a possibility of a clash with the user table? Ofc there is. But MS is still doing it nevertheless, because chances of this occurring are really slim. And if a user acquire access to those internal tables and start modify them and in the process screw something up, well he uses MS product as a developer and so should know better as a developer And if MS can do it - why can't I? > > > > > Now, the creation/population is done inside a transaction. > > I'm not sure that a transaction is going to solve the issue I raised > above, it will just make one thing happen with no guarantee that it is > the correct outcome. What is considered correct outcome is strongly on user if we are talking about my app. > > > >> Are you talking about the front end that the user launches or the > >> backend that runs the database or something else? > > > > Front-end. > > This is a C++ app. > > To me this is the tail wagging the dog. The thought of allowing users to > change the database structure and you dealing with it after the fact is > just disturbing to me. But not to me. My application targets developers, not end users. Thank you. > > > > >> > >> > >>> > >>> Thank you. > > -- > Adrian Klaver > adrian.klaver@aklaver.com >
> But not to me. > My application targets developers, not end users. > > Thank you. What does your app enable developers to do?
On Sat, Dec 7, 2024 at 4:59 PM Igor Korot <ikorot01@gmail.com> wrote:
I am making the tables and naming them with a specific pattern.
Now I'm curious - if I start psql and will want to create a table
named pg_am, what will happen?
I presume psql will produce error saying that the system table with
that name exists and the user can't create ome.
Am I right?
You are wasting people's time asking a question like this when it takes but seconds to experiment.
Most of what you are doing should be considered something to test out and observe the behavior yourself. Formalize it as a test if possible if you rely heavily on it so you can be aware if a bug were to be introduced that broke such a behavior, or even just to get lots of exposure to concurrency if you are designing a system especially reliant on specific concurrent behavior.
As it stands the methodology you describe is quite non-traditional for a relational client-server database system.
David J.
On 12/7/24 15:58, Igor Korot wrote:
> Hi, Adrian,
>
>> What distinguishes your tables from other users' tables?
>
> I am making the tables and naming them with a specific pattern.
>
> Now I'm curious - if I start psql and will want to create a table
> named pg_am, what will happen?
> I presume psql will produce error saying that the system table with
> that name exists and the user can't create ome.
> Am I right?
When you refer to psql are you talking about the CLI program or the
Postgres server in general?
As to table names it depends. A schema is a namespace so if you put the
table in it's own schema then the server will not complain:
create table test_sch.pg_am(id integer);
CREATE TABLE
\d *.pg_am
Table "pg_catalog.pg_am"
Column | Type | Collation | Nullable | Default
-----------+---------+-----------+----------+---------
oid | oid | | not null |
amname | name | | not null |
amhandler | regproc | | not null |
amtype | "char" | | not null |
Indexes:
"pg_am_oid_index" PRIMARY KEY, btree (oid)
"pg_am_name_index" UNIQUE CONSTRAINT, btree (amname)
Table "test_sch.pg_am"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
id | integer | | |
I would strongly advise against this naming schema for the above reason.
>
>>
>>> 4. Program is executed.
>>> 5. User closes the program.
>>> 6.. Later on the user decides that there is a need
>>> for another table (inside psql or any other client)
>>
>> The above is where I started twitching.
>>
>> How do you keep them out of your tables?
>
> My tables are named with the specific pattern.
>
> Also - see above.
Yes, but if you allow users into the database:
a) They can figure out what that pattern is.
b) Mess with the tables.
>
>>
>> How do you get these changes to play nice with the existing structure?
>
> Again - not sure what you mean here....
Examples:
1) A user creates table that creates a FK relationship to one of 'your'
tables and you then change 'your' table and invalidate that relationship.
2) A user changes the structure of 'your' tables.
> If one of my tables will be dropped - it will be re-created.
> I'm using CREATE TABLE IF NOT EXIST.
This means every time the app is run it could potentially run one or
more schema/data migrations.
How are you going to track the state of the database in order to get it
back to what it was when the change or changes occurred?
>
>>
>>
>>> All records that were there are staying unchanged.
>>
>> INSERT/UPDATE against your tables is not a possibility?
>
> It is..
> All I;m saying that when the app starts-up, this is done automatically
Again, using what point in time state?
>
> And if the user decides to insert some data - that's on him
No the point of this thread is it's on you or you would not be asking
how to restore to a known state.
>>
>> This is the part that confuses me.
>>
>> If you are going to allow ad hoc and at will changes how do you know
>> what is actually the correct change?
>
> I don't.
> Think about MS ACCESS-like applications.
> ACCESS creates the internal tables to keep track of some
> internals.
> Is there a possibility of a clash with the user table? Ofc there is.
> But MS is still doing it nevertheless, because chances of this occurring
> are really slim.
> And if a user acquire access to those internal tables and start modify
> them and in the process screw something up, well he uses MS product
> as a developer and so should know better as a developer
> And if MS can do it - why can't I?
Because MS has the lawyers to keep at bay anyone complaining that Access
blew up their application.
>
>
>>
>>>
>>> Now, the creation/population is done inside a transaction.
>>
>> I'm not sure that a transaction is going to solve the issue I raised
>> above, it will just make one thing happen with no guarantee that it is
>> the correct outcome.
>
> What is considered correct outcome is strongly on user if we are talking
> about my app.
Except you are asking how to deal with changes and so you are taking
ownership of them. And in the case you describe, two different instances
of the app making changes, how do you decide which one wins?
>> To me this is the tail wagging the dog. The thought of allowing users to
>> change the database structure and you dealing with it after the fact is
>> just disturbing to me.
>
> But not to me.
> My application targets developers, not end users.
Who are users in your application.
>
> Thank you.
>
>>
>>>
>>>>
>>>>
>>>>>
>>>>> Thank you.
>>
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com
>>
--
Adrian Klaver
adrian.klaver@aklaver.com