Thread: [GENERAL] pg_upgrade --link on Windows
Hi The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use junction points on Windows. Shouldn't it rather user hard-links ? If I'm not mistaken, with junction points (i.e. soft-links to directories), the old data dir cannot be removed. With hard-links to file, we can get rid of the old data dir once we are sure that the upgrade is fine. Regards -- Arnaud
On Fri, Jun 9, 2017 at 12:00:56PM +0200, Arnaud L. wrote: > Hi > > The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use > junction points on Windows. > Shouldn't it rather user hard-links ? > If I'm not mistaken, with junction points (i.e. soft-links to directories), > the old data dir cannot be removed. > With hard-links to file, we can get rid of the old data dir once we are sure > that the upgrade is fine. I was told junction points on Windows were hard links and no one has ever complained about not being able to remove them. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On 06/09/2017 07:07 AM, Bruce Momjian wrote: > On Fri, Jun 9, 2017 at 12:00:56PM +0200, Arnaud L. wrote: >> Hi >> >> The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use >> junction points on Windows. >> Shouldn't it rather user hard-links ? >> If I'm not mistaken, with junction points (i.e. soft-links to directories), >> the old data dir cannot be removed. >> With hard-links to file, we can get rid of the old data dir once we are sure >> that the upgrade is fine. > > I was told junction points on Windows were hard links and no one has > ever complained about not being able to remove them. > https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx Seems to me the difference is hard links point to file, junctions to directories. So if I am following: https://en.wikipedia.org/wiki/NTFS_junction_point#Creating_or_deleting_a_junction_point You remove the junction and then the directory it points to. -- Adrian Klaver adrian.klaver@aklaver.com
Le 9/06/2017 à 16:07, Bruce Momjian a écrit : > I was told junction points on Windows were hard links and no one has > ever complained about not being able to remove them. Sorry, I think my explanation was not very clear. You can remove the link, but the point is to remove the target (i.e. the old-data-dir). You can do this with a hard link (there still exists a hardlink pointing to the inode so it remains), but with a soft link you end up with a link to nothing. Deleting a junction target in Windows will work, but you'll have an error trying to access the junction directory (directory not found). See this page for more details : http://cects.com/overview-to-understanding-hard-links-junction-points-and-symbolic-links-in-windows/ Under "Hard Link (Linking for individual files)" : "If the target is deleted, its content is still available through the hard link" Junction Point (Directory Hard Link): "If the target is moved, renamed or deleted, the Junction Point still exists, but points to a non-existing directory" BUT, when I try to "pg_upgrade --link --check" with old-data-dir and new-data-dir on different volumes, I get an error saying that both directories must be on the same volume if --link is used. So maybe pg_upgrade uses hard-links (i.e. to files), and only the documentation is wrong by calling them junctions (i.e. soft links to files) ? Regards -- Arnaud
On 06/09/2017 07:39 AM, Arnaud L. wrote: > Le 9/06/2017 à 16:07, Bruce Momjian a écrit : >> I was told junction points on Windows were hard links and no one has >> ever complained about not being able to remove them. > > Sorry, I think my explanation was not very clear. > You can remove the link, but the point is to remove the target (i.e. the > old-data-dir). > You can do this with a hard link (there still exists a hardlink pointing > to the inode so it remains), but with a soft link you end up with a link > to nothing. > Deleting a junction target in Windows will work, but you'll have an > error trying to access the junction directory (directory not found). > > See this page for more details : > http://cects.com/overview-to-understanding-hard-links-junction-points-and-symbolic-links-in-windows/ > > > Under "Hard Link (Linking for individual files)" : > "If the target is deleted, its content is still available through the > hard link" > > Junction Point (Directory Hard Link): > "If the target is moved, renamed or deleted, the Junction Point still > exists, but points to a non-existing directory" > > BUT, when I try to "pg_upgrade --link --check" with old-data-dir and > new-data-dir on different volumes, I get an error saying that both > directories must be on the same volume if --link is used. > So maybe pg_upgrade uses hard-links (i.e. to files), and only the > documentation is wrong by calling them junctions (i.e. soft links to > files) ? Looks that way. In file.c in ~/src/bin/pg_upgrade I see: #ifdef WIN32 300 /* implementation of pg_link_file() on Windows */ 301 static int 302 win32_pghardlink(const char *src, const char *dst) 303 { 304 /* 305 * CreateHardLinkA returns zero for failure 306 * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx 307 */ 308 if (CreateHardLinkA(dst, src, NULL) == 0) 309 { 310 _dosmaperr(GetLastError()); 311 return -1; 312 } 313 else 314 return 0; 315 } 316 #endif > > Regards > -- > Arnaud > > -- Adrian Klaver adrian.klaver@aklaver.com
Le 9/06/2017 à 16:55, Adrian Klaver a écrit :
> On 06/09/2017 07:39 AM, Arnaud L. wrote:
>> So maybe pg_upgrade uses hard-links (i.e. to files), and only the
>> documentation is wrong by calling them junctions (i.e. soft links to
>> files) ?
>
> Looks that way. In file.c in ~/src/bin/pg_upgrade I see:
>
> #ifdef WIN32
> 300 /* implementation of pg_link_file() on Windows */
> 301 static int
> 302 win32_pghardlink(const char *src, const char *dst)
> 303 {
> 304 /*
> 305 * CreateHardLinkA returns zero for failure
> 306 * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
> 307 */
> 308 if (CreateHardLinkA(dst, src, NULL) == 0)
> 309 {
> 310 _dosmaperr(GetLastError());
> 311 return -1;
> 312 }
> 313 else
> 314 return 0;
> 315 }
> 316 #endif
Great !
So I did a full upgrade for nothing (just for safety), but that's good
to know for next time !
Should this be submitted to postgresql-bugs, or is there something more
specific to the documentation ?
Regards
--
Arnaud
Le 9/06/2017 à 17:02, Arnaud L. a écrit :
> Le 9/06/2017 à 16:55, Adrian Klaver a écrit :
>> On 06/09/2017 07:39 AM, Arnaud L. wrote:
>>> So maybe pg_upgrade uses hard-links (i.e. to files), and only the
>>> documentation is wrong by calling them junctions (i.e. soft links to
>>> files) ?
>>
>> Looks that way. In file.c in ~/src/bin/pg_upgrade I see:
>>
>> #ifdef WIN32
>> 300 /* implementation of pg_link_file() on Windows */
>> 301 static int
>> 302 win32_pghardlink(const char *src, const char *dst)
>> 303 {
>> 304 /*
>> 305 * CreateHardLinkA returns zero for failure
>> 306 * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
>> 307 */
>> 308 if (CreateHardLinkA(dst, src, NULL) == 0)
>> 309 {
>> 310 _dosmaperr(GetLastError());
>> 311 return -1;
>> 312 }
>> 313 else
>> 314 return 0;
>> 315 }
>> 316 #endif
>
> Great !
> So I did a full upgrade for nothing (just for safety), but that's good
> to know for next time !
> Should this be submitted to postgresql-bugs, or is there something more
> specific to the documentation ?
I just found the pgsql-docs@postgresql.org list, for for the spam.
--
Arnaud
On Fri, 9 Jun 2017 10:07:24 -0400, Bruce Momjian <bruce@momjian.us> wrote: >On Fri, Jun 9, 2017 at 12:00:56PM +0200, Arnaud L. wrote: >> Hi >> >> The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use >> junction points on Windows. >> Shouldn't it rather user hard-links ? >> If I'm not mistaken, with junction points (i.e. soft-links to directories), >> the old data dir cannot be removed. >> With hard-links to file, we can get rid of the old data dir once we are sure >> that the upgrade is fine. > >I was told junction points on Windows were hard links and no one has >ever complained about not being able to remove them. NTFS junctions are a distinct type of symbolic link which is meant for filesystem mount points. In NTFS "normal" symlinks are restricted to targets within the same filesystem. You can use a junction anywhere you want a symlink, but not the reverse. The downside is that pathname parsing is slower with junctions than with symlinks because of the possibility that the path may cross into a different filesystem. The documentation is not very clear, IMO. https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa363878(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa365503(v=vs.85).aspx The mklink utility can create any of these types of links. Its documentation does not describe the differences, but is shows that hard links, symlinks, and junctions all are distinct concepts in Windows. https://technet.microsoft.com/en-us/library/cc753194(v=ws.11).aspx George
On Fri, 9 Jun 2017 07:24:03 -0700, Adrian Klaver <adrian.klaver@aklaver.com> wrote: >https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx > >Seems to me the difference is hard links point to file, junctions to >directories. You can make either hard links or symlinks to files. Junctions are distinct from normal symlinks in that junctions can cross filesystems. Microsoft's cmdline tools complain if you try to make a junction to a file, because Microsoft intended junctions for mount points ... but you can do it programmatically, or trick the tool by creating the link and then replacing the target, and in most cases it will work the same as a normal symlink. I have seen cases where a junction to a file didn't work, but they seemed to be application related rather than an OS issue. Prior to Vista, the mklink utility was not available, so people wanting to create symlinks were forced to use the sysinternals junction utility. https://technet.microsoft.com/en-us/sysinternals/bb545021.aspx George
On Fri, Jun 9, 2017 at 07:55:55AM -0700, Adrian Klaver wrote: > On 06/09/2017 07:39 AM, Arnaud L. wrote: > >See this page for more details : > >http://cects.com/overview-to-understanding-hard-links-junction-points-and-symbolic-links-in-windows/ > > > > > >Under "Hard Link (Linking for individual files)" : > >"If the target is deleted, its content is still available through the hard > >link" > > > >Junction Point (Directory Hard Link): > >"If the target is moved, renamed or deleted, the Junction Point still > >exists, but points to a non-existing directory" > > > >BUT, when I try to "pg_upgrade --link --check" with old-data-dir and > >new-data-dir on different volumes, I get an error saying that both > >directories must be on the same volume if --link is used. > >So maybe pg_upgrade uses hard-links (i.e. to files), and only the > >documentation is wrong by calling them junctions (i.e. soft links to > >files) ? > > Looks that way. In file.c in ~/src/bin/pg_upgrade I see: > > #ifdef WIN32 > 300 /* implementation of pg_link_file() on Windows */ > 301 static int > 302 win32_pghardlink(const char *src, const char *dst) > 303 { > 304 /* > 305 * CreateHardLinkA returns zero for failure > 306 * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx > 307 */ > 308 if (CreateHardLinkA(dst, src, NULL) == 0) > 309 { > 310 _dosmaperr(GetLastError()); > 311 return -1; > 312 } > 313 else > 314 return 0; > 315 } > 316 #endif [docs list added] I apologize for not being smarter on this thread. When I helped with the Windows port, I was told Windows didn't have hard links for use by tablespace directories, so I got it into my head that Windows didn't have hard links. Therefore, when I was writing the docs, I called them junction points. Looking back to Postgres 9.0 where pg_upgrade was added to the tree, I see that the code even at that time used hard links on Windows. I have created the attached patch which I will apply to all current Postgres versions to fix this error. Thanks for the report and the research. :-) -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Attachment
> -----Ursprüngliche Nachricht----- > > I apologize for not being smarter on this thread. When I helped with the > Windows port, I was told Windows didn't have hard links for use by tablespace > directories, so I got it into my head that Windows didn't have hard links. > Therefore, when I was writing the docs, I called them junction points. It's actually not "Windows" providing hard links, it is the file system NTFS. FAT and its modern cousins don't provide hard links - but this will rarely be used for databases these days. However, ReFS (introduced with server 2012 and providing some new features like automatic integrity checks, clustering etc.) does no longer provide hard links. Are hard links used anywhere else but in pg_upgrade? Klaus
On Wed, Jun 14, 2017 at 09:59:04AM +0200, Klaus P. Pieper wrote: > > -----Ursprüngliche Nachricht----- > > > > I apologize for not being smarter on this thread. When I helped with the > > Windows port, I was told Windows didn't have hard links for use by > tablespace > > directories, so I got it into my head that Windows didn't have hard links. > > Therefore, when I was writing the docs, I called them junction points. > > It's actually not "Windows" providing hard links, it is the file system > NTFS. FAT and its modern cousins don't provide hard links - but this will > rarely be used for databases these days. > However, ReFS (introduced with server 2012 and providing some new features > like automatic integrity checks, clustering etc.) does no longer provide > hard links. > Are hard links used anywhere else but in pg_upgrade? Nope, it is used only by transfer_relfile() calling linkFile(). -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On Tue, Jun 13, 2017 at 04:07:48PM -0400, Bruce Momjian wrote: > On Fri, Jun 9, 2017 at 07:55:55AM -0700, Adrian Klaver wrote: > I apologize for not being smarter on this thread. When I helped with > the Windows port, I was told Windows didn't have hard links for use by > tablespace directories, so I got it into my head that Windows didn't > have hard links. Therefore, when I was writing the docs, I called them > junction points. > > Looking back to Postgres 9.0 where pg_upgrade was added to the tree, I > see that the code even at that time used hard links on Windows. I have > created the attached patch which I will apply to all current Postgres > versions to fix this error. > > Thanks for the report and the research. :-) Patch applied all the way back to 9.3, where the junction point mention first appeared. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +