Thread: RE: Berkeley DB license
> Rather than replacing just the storage manager, you'd be replacing > the access methods, buffer manager, transaction manager, and some > of the shared memory plumbing with our stuff. I wasn't sufficiently > clear in my earlier message, and talked about "no-overwrite" as if > it were the only component. > > Clearly, that's a lot of work. On the other hand, you'd have the > benefit of an extremely well-tested and widely deployed library to > provide those services. Lots of different groups have used the > software, so the abstractions that the API presents are well-thought > out and work well in most cases. True. But after replacement the system as whole would not be well-tested. "A lot of work" means "a lot of bugs, errors etc". > The group is interested in multi-version concurrency control, so that > readers never block on writers. If that's genuinely critical, we'd > be willing to see some work done to add it to Berkeley DB, so that it > can do either conventional 2PL without versioning, or MV. Naturally, This would be the first system with both types of CC -:) Well, so, before replacing anything we would have to add MVCC to BDB. I still didn't look at your sources, 'll do in a few days... Vadim
> > The group is interested in multi-version concurrency control, so that > > readers never block on writers. If that's genuinely critical, we'd > > be willing to see some work done to add it to Berkeley DB, so that it > > can do either conventional 2PL without versioning, or MV. Naturally, > > This would be the first system with both types of CC -:) > > Well, so, before replacing anything we would have to add MVCC to BDB. > I still didn't look at your sources, 'll do in a few days... Vadim, I thought you said you were going to be doing a new storage manager for 7.2, including an over-write storage manager that keeps MVCC tuples in a separate location. Could SDB work in that environment easier, without having MVCC integrated into SDB? -- Bruce Momjian | http://www.op.net/~candle pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
> > Well, so, before replacing anything we would have to add > > MVCC to BDB. I still didn't look at your sources, 'll do > > in a few days... > > Vadim, I thought you said you were going to be doing a new storage > manager for 7.2, including an over-write storage manager that > keeps MVCC tuples in a separate location. Could SDB work in that > environment easier, without having MVCC integrated into SDB? How can we integrate SDB code into PostgreSQL without MVCC support in SDB if we still want to have MVCC?! I missed something? Or you ask is replacement+changes_in_SDB_for_MVCC easier than WAL+new_our_smgr? I don't know. Vadim
> How can we integrate SDB code into PostgreSQL without MVCC support > in SDB if we still want to have MVCC?! I missed something? > Or you ask is replacement+changes_in_SDB_for_MVCC easier than > WAL+new_our_smgr? I don't know. I guess I was asking if MVCC could be implemented on top of SDB, rather than changes made to SDB itself. -- Bruce Momjian | http://www.op.net/~candle pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
[Charset iso-8859-1 unsupported, filtering to ASCII...] > > > Well, so, before replacing anything we would have to add > > > MVCC to BDB. I still didn't look at your sources, 'll do > > > in a few days... > > > > Vadim, I thought you said you were going to be doing a new storage > > manager for 7.2, including an over-write storage manager that > > keeps MVCC tuples in a separate location. Could SDB work in that > > environment easier, without having MVCC integrated into SDB? > > How can we integrate SDB code into PostgreSQL without MVCC support > in SDB if we still want to have MVCC?! I missed something? > Or you ask is replacement+changes_in_SDB_for_MVCC easier than > WAL+new_our_smgr? I don't know. You stated that the new storage manager will do over-writing, and that the MVCC-needed tuples will be kept somewhere else and removed when not needed. It is possible to use SDB, and keep the MVCC-needed tuples somewhere else, also in SDB, so we don't have to add MVCC into the SDB existing code, we just need to use SDB to implement MVCC. The issue was that SDB does two-phase locking, and I was asking if MVCC could be layered on top of SDB, rather than being added into SDB. -- Bruce Momjian | http://www.op.net/~candle pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
> You stated that the new storage manager will do over-writing, and that > the MVCC-needed tuples will be kept somewhere else and removed when not > needed. > > It is possible to use SDB, and keep the MVCC-needed tuples somewhere > else, also in SDB, so we don't have to add MVCC into the SDB existing > code, we just need to use SDB to implement MVCC. Possible, in theory. > The issue was that SDB does two-phase locking, and I was asking if MVCC ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Due to this fact seems we'll have to change SDB anyway. With MVCC per-tuple locking is not needed. Short-term per-buffer _latches_ are used to prevent concurrent changes in a buffer, no locks made via lock manager. I'm not sure does SDB API allow _any_ access to modified tuples or not. I would rather assume that it doesn't. > could be layered on top of SDB, rather than being added into SDB. As I said - possible, in theory, - and also not good thing to do, in theory. MVCC and 2PL are quite different approaches to problem of concurrency control. So, how good is layering one approach over another, who knows? Vadim
At 11:51 AM 5/16/00 -0700, Mikheev, Vadim wrote:
> > The issue was that SDB does two-phase locking, and I was asking if MVCC
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Due to this fact seems we'll have to change SDB anyway. With MVCC per-tuple
> locking is not needed. Short-term per-buffer _latches_ are used to prevent
> concurrent changes in a buffer, no locks made via lock manager.
The locking, logging, and buffer management interfaces are all available
independently. Certainly it would be possible to modify the current
implementation to do System R-style shadow paging, or some other technique,
but to continue to use the current log manager to roll back or reapply
changes as necessary.
In that case, you'd take out the current 2PL lock acquisition calls during
a transaction, and add code to create shadow pages for updates.
Certainly the interfaces are there. It'd be interesting to us if users
could select 2PL or MV for concurrency control on initialization, and
it appears on the face of it that that should be possible.
> I'm not sure does SDB API allow _any_ access to modified tuples or not.
> I would rather assume that it doesn't.
At present, Berkeley DB provides only 2PL. We don't permit dirty reads,
but that's an implementation detail that would be fairly simple to change.
Berkeley DB does page-level, not record-level, locking. There are short-
term mutex locks that protect shared structures.
My bet is that it would be better to use the lock manager, rather than
(say) spinlocks, to protect pages during updates; it's possible during
multi-page updates, like btree page splits. Since the updating thread
may be descheduled during relatively long operations, you'd rather not
be holding a spinlock. And you can acquire and release locks via lock
manager calls without trouble.
> MVCC and 2PL are quite different approaches to problem of concurrency
> control.
I'm joining the discussion quite late. Can you summarize for me the
thinking behind adopting MVCC in place of 2PL? Have you got performance
numbers for the two?
mike
> 1. MVCC semantics. If we can't keep MVCC then the deal's dead in the > water, no question. Vadim's by far the best man to look at > this issue; Vadim, do you have time to think about it soon? I'll comment just after weekends... But Mike Mascari raised up very interest issue about am I *allowed* to view the code at all. If we'll decide to continue with our own smgr+WAL I'll not be able to free my mind from what I've seen in SDB code. So..? Ok, I'll read API documentation first... Vadim
At 10:32 AM 5/17/00 -0700, Mikheev, Vadim wrote:
> But Mike Mascari raised up very interest issue about am I *allowed* to
> view the code at all. If we'll decide to continue with our own
> smgr+WAL I'll not be able to free my mind from what I've seen
> in SDB code. So..?
Berkeley DB implements a write-ahead log and two-phase locking using
well-known techniques from the literature (see, eg, Transaction
Processing by Gray et al). Hell, I've seen the server code for
Teradata, Illustra, and Informix, and have managed to make a living
in the industry for fifteen years anyway.
mike