Thread: atomics.h may not be included from frontend code
Hello hackers, My colleague Anastasia Lubennikova and I were discussing a weird piece of code in src/include/port/atomics.h: ``` #ifdef FRONTEND #error "atomics.h may not be included from frontend code" #endif ``` We tried to follow commit messages [1] and discussions [2]. However no matter how you try to look on this code it's weird. Basically it says that atomics are written the way that they can be used from one code and can't be used from another. So if you want to write a cross-platform parallel incremental backup tool (what Anastasia is currently working on) you have to use C++ and std::atomic (since MS doesn't develops C anymore) or write something like: ``` #undef FRONTEND #include <atomics.h> #define FRONTEND ``` We would like to know whether you share this concern and whether it would be a good idea to try to refactor the code so that atomics could be used not only from the backend. [1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=4eda0a64705 [2]: https://postgr.es/m/20150806070902.GE12214@awork2.anarazel.de -- Best regards, Aleksander Alekseev
Attachment
On Tue, Feb 27, 2018 at 8:49 AM, Aleksander Alekseev <a.alekseev@postgrespro.ru> wrote: > We would like to know whether you share this concern and whether it > would be a good idea to try to refactor the code so that atomics could > be used not only from the backend. I think the concern on the referenced threads was that atomics might be implemented using spinlocks if we don't have a real atomics implementation; and those in turn might be implemented as semaphores if we don't have a real spinlock implementation. When we emulate atomics using spinlocks, we use a fixed-size array of spinlocks stored in shared memory; and when we emulate spinlocks using semaphore, we use the semaphores in each PGPROC. So those emulation layers are deeply tied into the backend's shared-memory architecture, and untangling them might be a bit complicated. However, those are presumably rare configurations that many people (including many developers) don't care about. If #include "atomics.h" worked from all frontend code except where one of those emulation layers were in use, that would represent an improvement over the current status quo for people doing out-of-core development against PostgreSQL. On the other hand, it would also make it quite easy for the use of atomics to creep into frontend code within PG itself, and that's not OK so long as configurations that lack atomics and/or spinlocks are considered supported. So that's something to think about. I have to imagine that systems which lack atomics and/or spinlocks are almost extinct. It looks like we have at least one buildfarm member running with each of --disable-spinlocks and --disable-atomics, but systems that organically lack spinlocks and/or atomics must by now be extremely rare. At some point, I think we should just de-support such configurations, but I'm not sure if we're at that point yet. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Hi, On 2018-02-27 10:36:01 -0500, Robert Haas wrote: > On Tue, Feb 27, 2018 at 8:49 AM, Aleksander Alekseev > We tried to follow commit messages [1] and discussions [2]. However no matter > how you try to look on this code it's weird. I don't see how that makes the code weird. Not fit for your purpose, sure, but weird? > > We would like to know whether you share this concern and whether it > > would be a good idea to try to refactor the code so that atomics could > > be used not only from the backend. > > I think the concern on the referenced threads was that atomics might > be implemented using spinlocks if we don't have a real atomics > implementation; and those in turn might be implemented as semaphores > if we don't have a real spinlock implementation. When we emulate > atomics using spinlocks, we use a fixed-size array of spinlocks stored > in shared memory; and when we emulate spinlocks using semaphore, we > use the semaphores in each PGPROC. So those emulation layers are > deeply tied into the backend's shared-memory architecture, and > untangling them might be a bit complicated. Right. > However, those are presumably rare configurations that many people > (including many developers) don't care about. I don't think that's quite true anymore. We e.g. now rely on 64bit atomics being emulated on some machines, and they're unavailable on a bigger number of systems than atomics proper, particularly 32bit sytems. There's also hppa, of course ;) > If #include "atomics.h" worked from all frontend code except where one > of those emulation layers were in use, that would represent an > improvement over the current status quo for people doing out-of-core > development against PostgreSQL. I think if we wanted to enable this code to be used from frontend, we'd have to to implement the various fallbacks paths in a bit different way, so they can be supplied by frontend code. And then we can rely on it on frontend code for real, if we want to. Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> On 2018-02-27 10:36:01 -0500, Robert Haas wrote:
>> However, those are presumably rare configurations that many people
>> (including many developers) don't care about.
> I don't think that's quite true anymore. We e.g. now rely on 64bit
> atomics being emulated on some machines, and they're unavailable on a
> bigger number of systems than atomics proper, particularly 32bit
> sytems. There's also hppa, of course ;)
I had the idea that there were also still some MIPS machines out there
with no real atomics support. If there's not, I wouldn't complain
hard about deciding to desupport HPPA, whenever we want to rip out
the fallbacks.
I am not sure though that we want to promise that atomics.h would work
in arbitrary frontend environments in any case. Those functions are
very specifically intended to do what PG needs them to do; do we
really want them to try to serve multiple masters?
regards, tom lane
On 2018-02-27 13:43:23 -0500, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > On 2018-02-27 10:36:01 -0500, Robert Haas wrote: > >> However, those are presumably rare configurations that many people > >> (including many developers) don't care about. > > > I don't think that's quite true anymore. We e.g. now rely on 64bit > > atomics being emulated on some machines, and they're unavailable on a > > bigger number of systems than atomics proper, particularly 32bit > > sytems. There's also hppa, of course ;) > > I had the idea that there were also still some MIPS machines out there > with no real atomics support. If there's not, I wouldn't complain > hard about deciding to desupport HPPA, whenever we want to rip out > the fallbacks. There's definitely several machines that we currently support without good enough atomics support. I don't *think* mips is among them, the architectural manuals I had looked at when implementing it suggested ll/sc type support is available everywhere. https://wiki.postgresql.org/wiki/Atomics lacking 32 bit atomics: armv5, sparcv8, pa-risc, m32r, i386 lacking 64 bit atomics: some ppc, armv6, i486, !gcc !msvc x86 in 32bit mode None of these seem extremely interesting personally, but I'm way happier to drop platforms than some others ; > I am not sure though that we want to promise that atomics.h would work > in arbitrary frontend environments in any case. Those functions are > very specifically intended to do what PG needs them to do; do we > really want them to try to serve multiple masters? I think if we'd allow the client to implement a locking mechanism for the fallback implementation, there'd not be a big issue. Besides the fallback code there's really not that much PG specificity in there... Greetings, Andres Freund