Thread: So, can we stop supporting Windows native now?
http://www.zdnet.com/article/microsoft-and-canonical-partner-to-bring-ubuntu-to-windows-10/ ... could be good news for us ... -- -- Josh Berkus Red Hat OSAS (any opinions are my own)
On Thu, Mar 31, 2016 at 8:49 AM, Josh berkus <josh@agliodbs.com> wrote: > http://www.zdnet.com/article/microsoft-and-canonical-partner-to-bring-ubuntu-to-windows-10/ > > ... could be good news for us ... Possible. We are years ahead of that for sure. Also the outcome of the partnership, as well as the performance impact that there could be if we rely on an extra application layer, aka the layer in charge of transmitting POSIX-compliant calls to the Windows kernel, may be costly and needs to be evaluated carefully. 1st of April is close by btw. That's not the best timing for credibility. -- Michael
On 31 March 2016 at 07:49, Josh berkus <josh@agliodbs.com> wrote:
> So, can we stop supporting Windows native now?
Why would we want to?
About the only issue I think it causes is with the build system. Which isn't beautiful, to be sure, but is a burden carried mainly by those who care about Windows support. If we eventually get a CMake build system conversion that'll mostly go away too.
--
Craig Ringer <craig@2ndquadrant.com> writes:
> On 31 March 2016 at 07:49, Josh berkus <josh@agliodbs.com> wrote:
>> So, can we stop supporting Windows native now?
> Why would we want to?
> The cost is small.
Surely you jest. Windows is the single biggest PITA platform from a
portability perspective, and has been in every release cycle since we
first had a Windows port, and there is no close second.
If you have any doubts about this, please consult the commit logs over
the past week or three for a particularly painful sequence.
> ... is a burden carried mainly by those who
> care about Windows support.
Really? Good. I just committed my very last Windows-related fix, then.
Somebody else can deal with it.
Having thus vented, I do not seriously believe that we can drop
Windows-native. But please don't give me any jive about how it's
a small cost to support that platform.
regards, tom lane
PS: if you meant that as a 1-April troll, you're a day early.
On 2016-03-31 09:04:35 +0800, Craig Ringer wrote: > The cost is small. First off I agree we don't want to drop proper windows support. But I think "the cost is small" is a pretty bad mischaracterization. I don't do windows, and yet I've spent a lot of time figuring out windows only stuff, even writing windows only things (atomics, latches, recent bugfixes). There's a lot of architectural gunk in postgres just geared towards supporting windows (c.f. EXEC_BACKEND), and that makes new development harder in a number of cases. E.g. background workers, paralellism and such had quite some extra work cut out for them because of that. > but is a burden carried mainly by those who care about Windows > support. I don't think that's true. Tom e.g. seems to fight battles with it on a regular base. Andres Freund
On 2016-03-31 00:17:12 -0400, Tom Lane wrote: > Craig Ringer <craig@2ndquadrant.com> writes: > > On 31 March 2016 at 07:49, Josh berkus <josh@agliodbs.com> wrote: > >> So, can we stop supporting Windows native now? > > > Why would we want to? > > > The cost is small. > > Surely you jest. Windows is the single biggest PITA platform from a > portability perspective, and has been in every release cycle since we > first had a Windows port, and there is no close second. I think to a good degree this is caused by a lot of the windows specific code systematically having a lower code quality than the rest of PG. Partially that's because it's code you don't stumble upon: If you read recv() in code somewher, you're not immediately continuing to read pgwin32_recv(). But a look at it'll make pretty much anybody weep. Besides a lot of it being halfway hidden away in port/, another issue is that people just see it as portability hacks, deserving less attention. If you want to make win32 a reasonable postgres port, you'd have to clean up a lot of stuff. We've had fundamental issues with the windows event models for *years* without anybody even taking a stab at fixing it. Greetings, Andres Freund
Craig Ringer wrote: > If we eventually get a CMake build system conversion that'll > mostly go away too. I'm working on it. But the build system does not solve the problem of some hacks. We must nevertheless spend more time for Windows or at least goes to mingw64 (like ActivePerl). -- Yury Zhuravlev Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 31 March 2016 at 16:20, Andres Freund <andres@anarazel.de> wrote:
On 2016-03-31 09:04:35 +0800, Craig Ringer wrote:
> The cost is small.
First off I agree we don't want to drop proper windows support.
But I think "the cost is small" is a pretty bad mischaracterization. I
don't do windows, and yet I've spent a lot of time figuring out windows
only stuff, even writing windows only things (atomics, latches, recent
bugfixes). There's a lot of architectural gunk in postgres just geared
towards supporting windows (c.f. EXEC_BACKEND), and that makes new
development harder in a number of cases. E.g. background workers,
paralellism and such had quite some extra work cut out for them because
of that.
Fair point. It's not just about fixing windows or needing Windows-specific *features*, it's about making it harder to develop things because of Windows. I've seen that myself.
> but is a burden carried mainly by those who care about Windows
> support.
I don't think that's true. Tom e.g. seems to fight battles with it on a
regular base.
Yeah, you're right. He's not the only one either.
I was reacting to the original post, and TBH didn't think it through. The commit logs suggest there's a decent amount of work that goes in, and I'm sure a lot of it isn't visible when just looking for 'windows', 'win32', 'msvc', etc.
Even the build system affects people who don't use it, if they're adding features. I recently backported a bunch of 9.3 functionality to 9.1, and in the process simply stripped out all the Windows build system changes as "meh, too hard, don't care".
So yeah. I casually handwaved away a lot of work that's not highly visible, but still happens and is important, and was wrong to do so. I've done a bit on Windows myself but didn't fully recognise the burden support for it places on patches to core infrastructure and on committers.
Craig Ringer wrote: > Yeah, you're right. He's not the only one either. > > I was reacting to the original post, and TBH didn't think it > through. The commit logs suggest there's a decent amount of work > that goes in, and I'm sure a lot of it isn't visible when just > looking for 'windows', 'win32', 'msvc', etc. > > Even the build system affects people who don't use it, if > they're adding features. I recently backported a bunch of 9.3 > functionality to 9.1, and in the process simply stripped out all > the Windows build system changes as "meh, too hard, don't care". > > So yeah. I casually handwaved away a lot of work that's not > highly visible, but still happens and is important, and was > wrong to do so. I've done a bit on Windows myself but didn't > fully recognise the burden support for it places on patches to > core infrastructure and on committers. Maybe someone in the community to appoint a chief for Windows? -- Yury Zhuravlev Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 2016-03-31 13:30:58 +0300, Yury Zhuravlev wrote: > Craig Ringer wrote: > >Yeah, you're right. He's not the only one either. > > > >I was reacting to the original post, and TBH didn't think it through. The > >commit logs suggest there's a decent amount of work that goes in, and I'm > >sure a lot of it isn't visible when just looking for 'windows', 'win32', > >'msvc', etc. > > > >Even the build system affects people who don't use it, if they're adding > >features. I recently backported a bunch of 9.3 functionality to 9.1, and > >in the process simply stripped out all the Windows build system changes as > >"meh, too hard, don't care". > > > >So yeah. I casually handwaved away a lot of work that's not highly > >visible, but still happens and is important, and was wrong to do so. I've > >done a bit on Windows myself but didn't fully recognise the burden support > >for it places on patches to core infrastructure and on committers. > > Maybe someone in the community to appoint a chief for Windows? If somebody wanted to be that they'd just have to start doing it. It's not like windows problems are an infrequent occurrance.
On 03/31/2016 06:38 AM, Andres Freund wrote: > On 2016-03-31 13:30:58 +0300, Yury Zhuravlev wrote: >> Craig Ringer wrote: >>> Yeah, you're right. He's not the only one either. >>> >>> I was reacting to the original post, and TBH didn't think it through. The >>> commit logs suggest there's a decent amount of work that goes in, and I'm >>> sure a lot of it isn't visible when just looking for 'windows', 'win32', >>> 'msvc', etc. >>> >>> Even the build system affects people who don't use it, if they're adding >>> features. I recently backported a bunch of 9.3 functionality to 9.1, and >>> in the process simply stripped out all the Windows build system changes as >>> "meh, too hard, don't care". >>> >>> So yeah. I casually handwaved away a lot of work that's not highly >>> visible, but still happens and is important, and was wrong to do so. I've >>> done a bit on Windows myself but didn't fully recognise the burden support >>> for it places on patches to core infrastructure and on committers. >> Maybe someone in the community to appoint a chief for Windows? > If somebody wanted to be that they'd just have to start doing it. It's > not like windows problems are an infrequent occurrance. > > For the most part Magnus and I have been that de facto, as the committers most involved with Windows over the 11 or so years we've had the port. We also have a few other prominent developers who help out - e.g. with the recent VS2015 stuff. The worst thing about developing from my POV isn't something that actually affects core developers so much, namely the lack of a nice MSVC equivalent of PGXS. It would also be nice to find out why we can't usefully scale shared buffers higher like we can on *nix. cheers andrew
On 03/30/2016 09:17 PM, Tom Lane wrote: > Craig Ringer <craig@2ndquadrant.com> writes: >> On 31 March 2016 at 07:49, Josh berkus <josh@agliodbs.com> wrote: >>> So, can we stop supporting Windows native now? > >> Why would we want to? > >> The cost is small. > > Surely you jest. Windows is the single biggest PITA platform from a > portability perspective, and has been in every release cycle since we > first had a Windows port, and there is no close second. Add to that the cost on extension authors. I would surely *love* to dump Windows support in PL/R as it is a major league PITA. It is probably an understatement to say that over the last 10+ years, 95+% of the time I have spent maintaining and supporting PL/R has been directly attributable to the Windows port. Joe -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development
On 03/31/2016 06:53 AM, Andrew Dunstan wrote: > The worst thing about developing from my POV isn't something that > actually affects core developers so much, namely the lack of a nice MSVC > equivalent of PGXS. Bingo! +1 -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development
On Fri, Apr 1, 2016 at 2:53 AM, Andrew Dunstan <andrew@dunslane.net> wrote: > It would also be nice to find out why we can't usefully scale shared buffers > higher like we can on *nix. Has anyone ever looked into whether asking for SEC_LARGE_PAGES would help with that? I noticed that another popular RDBMS recommends enabling this to see a gain when its buffer pool is "several gigabytes". I don't do Windows myself, but from poking around in the docs, it looks like you need to grant SeLockMemoryPrivilege (Start > Control Panel > Administrative Tools > Local Security Policy > User Rights Assignment > Lock pages in memory > Action > Properties) because large pages can't be swapped out (just like on other OSs). So maybe it could work like huge_pages = try on Linux so that it works out of the box with 4K pages, but starts using 2MB (?) pages if you grant SeLockMemoryPrivilege. https://msdn.microsoft.com/en-us/library/windows/desktop/aa366543(v=vs.85).aspx Just a thought. -- Thomas Munro http://www.enterprisedb.com
Thomas Munro wrote: > On Fri, Apr 1, 2016 at 2:53 AM, Andrew Dunstan <andrew@dunslane.net> wrote: > > It would also be nice to find out why we can't usefully scale shared buffers > > higher like we can on *nix. > > Has anyone ever looked into whether asking for SEC_LARGE_PAGES would > help with that? The huge_pages feature is fairly new, and as I recall the intention is that other operating systems will be supported as we get patches for it. Feel free to submit something. If you can actually research the performance problem with large shared_buffers, that would also be welcome. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 31 March 2016 at 21:53, Andrew Dunstan <andrew@dunslane.net> wrote:
The worst thing about developing from my POV isn't something that actually affects core developers so much, namely the lack of a nice MSVC equivalent of PGXS.
I completely agree there. I wrote some documentation on how to make project files to build extensions, but it'd be nice to *generate* them instead, like we do for in-tree builds.
The problem is that we don't install all the Perl build stuff on Windows, it's only in the build tree. The time consuming part wx is
/ sll be cleaning that up so it can run outside src/tools/msvc, getting it to use pg_config, and making it installable. Then of course there's the issue that it won't be available before 9.7 at the soonest... unless some brave soul backports it as an installable add-on. Which means extension authors are still screwed for 5+ years. That's the main reason I haven't tackled it - so far I've got by pretty well with VS project files + property sheets, and any PGXS-like capability almost certainly couldn't be backported so it's nearly useless for about 5 years.
/ sll be cleaning that up so it can run outside src/tools/msvc, getting it to use pg_config, and making it installable. Then of course there's the issue that it won't be available before 9.7 at the soonest... unless some brave soul backports it as an installable add-on. Which means extension authors are still screwed for 5+ years. That's the main reason I haven't tackled it - so far I've got by pretty well with VS project files + property sheets, and any PGXS-like capability almost certainly couldn't be backported so it's nearly useless for about 5 years.
On Fri, Apr 1, 2016 at 9:05 AM, Craig Ringer <craig@2ndquadrant.com> wrote: > I completely agree there. I wrote some documentation on how to make project > files to build extensions, but it'd be nice to *generate* them instead, like > we do for in-tree builds. Yes, I maintain some code that builds extension on Windows, those are using the msvc scripts by linking to the tree directly, with some custom-made wrappers for the calls that cannot be exported. That's ugly, but anybody would have guessed that.. > The problem is that we don't install all the Perl build stuff on Windows, > it's only in the build tree. The time consuming part wx is > / sll be cleaning that up so it can run outside src/tools/msvc, getting it > to use pg_config, and making it installable. Then of course there's the > issue that it won't be available before 9.7 at the soonest... unless some > brave soul backports it as an installable add-on. Which means extension > authors are still screwed for 5+ years. That's the main reason I haven't > tackled it - so far I've got by pretty well with VS project files + property > sheets, and any PGXS-like capability almost certainly couldn't be backported > so it's nearly useless for about 5 years. Still, it would be interesting to see how cmake affects this area of the code before tackling anything, doing something now may be time wasted for nothing at the end. So I would refrain from even putting resources in this project. -- Michael
On Fri, Apr 1, 2016 at 11:44 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > Thomas Munro wrote: >> On Fri, Apr 1, 2016 at 2:53 AM, Andrew Dunstan <andrew@dunslane.net> wrote: >> > It would also be nice to find out why we can't usefully scale shared buffers >> > higher like we can on *nix. >> >> Has anyone ever looked into whether asking for SEC_LARGE_PAGES would >> help with that? > > The huge_pages feature is fairly new, and as I recall the intention is > that other operating systems will be supported as we get patches for it. > Feel free to submit something. I don't have a Windows development stack or any plans to get one but I did make some notes and write some blind code for this a while back when I was studying the shmem code. Here it is, in the hope that it might be a useful starting point for someone else... I suppose the first step is to find a benchmark that improves as you increase shared_buffers on Windows and Linux but plateaus at a much lower setting on Windows, and then get a patch like this working and see if huge_pages = on changes anything... -- Thomas Munro http://www.enterprisedb.com
Attachment
On Fri, Apr 1, 2016 at 11:44 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > The huge_pages feature is fairly new, and as I recall the intention is > that other operating systems will be supported as we get patches for it. BTW about other operating systems: It looks like FreeBSD is already clever enough to use super pages as it calls them[1][2]. This is something like Linux's 'transparent huge pages' except apparently somewhat more transparent. Though I wonder if there is some advantage in adding a MAP_ALIGNED_SUPER flag to the mmap request. It looks like MacOSX can do it with VM_FLAGS_SUPERPAGE_SIZE_ANY but such mappings are not inherited by child processes. [1] https://lists.freebsd.org/pipermail/freebsd-hackers/2012-June/039018.html [2] https://www.youtube.com/watch?v=0wIxny-n_Mg <-- a really good talk about this stuff! -- Thomas Munro http://www.enterprisedb.com
On Thu, Mar 31, 2016 at 09:04:35AM +0800, Craig Ringer wrote: > If we eventually get a CMake build system conversion that'll mostly go > away too. Well, maybe the good message about this new development is that autotools will start working much better on Windows and could be eventually used for the Windows port as well, making a CMake conversion moot. Michael
On Wed, Mar 30, 2016 at 7:49 PM, Josh berkus <josh@agliodbs.com> wrote: > http://www.zdnet.com/article/microsoft-and-canonical-partner-to-bring-ubuntu-to-windows-10/ > > ... could be good news for us ... I read a saying someplace that if you see a news headline that ends in a question mark, the answer is always "no". So here. :-) -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Wed, Mar 30, 2016 at 6:49 PM, Josh berkus <josh@agliodbs.com> wrote: > http://www.zdnet.com/article/microsoft-and-canonical-partner-to-bring-ubuntu-to-windows-10/ > > ... could be good news for us ... This is nothing new. Windows has had a unix subsystem (Interix AKA Windows Services for Unix) for quite some time now. See: http://www.postgresql.org/message-id/303E00EBDD07B943924382E153890E5434AA78@cuthbert.rcsinc.local ...it was missing sync() but not fork() -- I had a functional native non-cygwin postgres that beat the mingw port by a major version :-). I was astounded to see that Interix is still supported as of windows 8 but no longer will be available as of windows 10. ISTM the new ubuntu stuff is the same engine with some spit polish and a heavy dose of advertising. Thinking about this objectively, it would be nice to see: *) broad developer adoption *) performance matching native *) clean install process that could be integrated to the windows installer before considering deprecating the native port, the first of those being the most important. The cautionary note here is that Microsoft can and will kill the ubuntu subsystem if it's not broadly adopted (Interix is something of a cautionary tale here). merlin