pg_upgrade --link on Windows

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

pg_upgrade --link on Windows

Arnaud L.
Hi

The pg_upgrade documentation for PostgreSQL 9.6 states that --link will
use junction points on Windows.
Shouldn't it rather user hard-links ?
If I'm not mistaken, with junction points (i.e. soft-links to
directories), the old data dir cannot be removed.
With hard-links to file, we can get rid of the old data dir once we are
sure that the upgrade is fine.

Regards

--
Arnaud


--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

Bruce Momjian
On Fri, Jun  9, 2017 at 12:00:56PM +0200, Arnaud L. wrote:
> Hi
>
> The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use
> junction points on Windows.
> Shouldn't it rather user hard-links ?
> If I'm not mistaken, with junction points (i.e. soft-links to directories),
> the old data dir cannot be removed.
> With hard-links to file, we can get rid of the old data dir once we are sure
> that the upgrade is fine.

I was told junction points on Windows were hard links and no one has
ever complained about not being able to remove them.

--
  Bruce Momjian  <[hidden email]>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +


--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

Adrian Klaver-4
On 06/09/2017 07:07 AM, Bruce Momjian wrote:

> On Fri, Jun  9, 2017 at 12:00:56PM +0200, Arnaud L. wrote:
>> Hi
>>
>> The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use
>> junction points on Windows.
>> Shouldn't it rather user hard-links ?
>> If I'm not mistaken, with junction points (i.e. soft-links to directories),
>> the old data dir cannot be removed.
>> With hard-links to file, we can get rid of the old data dir once we are sure
>> that the upgrade is fine.
>
> I was told junction points on Windows were hard links and no one has
> ever complained about not being able to remove them.
>

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx

Seems to me the difference is hard links point to file, junctions to
directories.

So if I am following:

https://en.wikipedia.org/wiki/NTFS_junction_point#Creating_or_deleting_a_junction_point

You remove the junction and then the directory it points to.

--
Adrian Klaver
[hidden email]


--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

Arnaud L.
In reply to this post by Bruce Momjian
Le 9/06/2017 à 16:07, Bruce Momjian a écrit :
> I was told junction points on Windows were hard links and no one has
> ever complained about not being able to remove them.

Sorry, I think my explanation was not very clear.
You can remove the link, but the point is to remove the target (i.e. the
old-data-dir).
You can do this with a hard link (there still exists a hardlink pointing
to the inode so it remains), but with a soft link you end up with a link
to nothing.
Deleting a junction target in Windows will work, but you'll have an
error trying to access the junction directory (directory not found).

See this page for more details :
http://cects.com/overview-to-understanding-hard-links-junction-points-and-symbolic-links-in-windows/

Under "Hard Link (Linking for individual files)" :
"If the target is deleted, its content is still available through the
hard link"

Junction Point (Directory Hard Link):
"If the target is moved, renamed or deleted, the Junction Point still
exists, but points to a non-existing directory"

BUT, when I try to "pg_upgrade --link --check" with old-data-dir and
new-data-dir on different volumes, I get an error saying that both
directories must be on the same volume if --link is used.
So maybe pg_upgrade uses hard-links (i.e. to files), and only the
documentation is wrong by calling them junctions (i.e. soft links to
files) ?

Regards
--
Arnaud


--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

Adrian Klaver-4
On 06/09/2017 07:39 AM, Arnaud L. wrote:

> Le 9/06/2017 à 16:07, Bruce Momjian a écrit :
>> I was told junction points on Windows were hard links and no one has
>> ever complained about not being able to remove them.
>
> Sorry, I think my explanation was not very clear.
> You can remove the link, but the point is to remove the target (i.e. the
> old-data-dir).
> You can do this with a hard link (there still exists a hardlink pointing
> to the inode so it remains), but with a soft link you end up with a link
> to nothing.
> Deleting a junction target in Windows will work, but you'll have an
> error trying to access the junction directory (directory not found).
>
> See this page for more details :
> http://cects.com/overview-to-understanding-hard-links-junction-points-and-symbolic-links-in-windows/ 
>
>
> Under "Hard Link (Linking for individual files)" :
> "If the target is deleted, its content is still available through the
> hard link"
>
> Junction Point (Directory Hard Link):
> "If the target is moved, renamed or deleted, the Junction Point still
> exists, but points to a non-existing directory"
>
> BUT, when I try to "pg_upgrade --link --check" with old-data-dir and
> new-data-dir on different volumes, I get an error saying that both
> directories must be on the same volume if --link is used.
> So maybe pg_upgrade uses hard-links (i.e. to files), and only the
> documentation is wrong by calling them junctions (i.e. soft links to
> files) ?

Looks that way. In file.c in ~/src/bin/pg_upgrade I see:

#ifdef WIN32
  300 /* implementation of pg_link_file() on Windows */
  301 static int
  302 win32_pghardlink(const char *src, const char *dst)
  303 {
  304     /*
  305      * CreateHardLinkA returns zero for failure
  306      * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
  307      */
  308     if (CreateHardLinkA(dst, src, NULL) == 0)
  309     {
  310         _dosmaperr(GetLastError());
  311         return -1;
  312     }
  313     else
  314         return 0;
  315 }
  316 #endif


>
> Regards
> --
> Arnaud
>
>


--
Adrian Klaver
[hidden email]


--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

Arnaud L.
Le 9/06/2017 à 16:55, Adrian Klaver a écrit :

> On 06/09/2017 07:39 AM, Arnaud L. wrote:
>> So maybe pg_upgrade uses hard-links (i.e. to files), and only the
>> documentation is wrong by calling them junctions (i.e. soft links to
>> files) ?
>
> Looks that way. In file.c in ~/src/bin/pg_upgrade I see:
>
> #ifdef WIN32
>    300 /* implementation of pg_link_file() on Windows */
>    301 static int
>    302 win32_pghardlink(const char *src, const char *dst)
>    303 {
>    304     /*
>    305      * CreateHardLinkA returns zero for failure
>    306      * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
>    307      */
>    308     if (CreateHardLinkA(dst, src, NULL) == 0)
>    309     {
>    310         _dosmaperr(GetLastError());
>    311         return -1;
>    312     }
>    313     else
>    314         return 0;
>    315 }
>    316 #endif

Great !
So I did a full upgrade for nothing (just for safety), but that's good
to know for next time !
Should this be submitted to postgresql-bugs, or is there something more
specific to the documentation ?

Regards
--
Arnaud




--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

Arnaud L.
Le 9/06/2017 à 17:02, Arnaud L. a écrit :

> Le 9/06/2017 à 16:55, Adrian Klaver a écrit :
>> On 06/09/2017 07:39 AM, Arnaud L. wrote:
>>> So maybe pg_upgrade uses hard-links (i.e. to files), and only the
>>> documentation is wrong by calling them junctions (i.e. soft links to
>>> files) ?
>>
>> Looks that way. In file.c in ~/src/bin/pg_upgrade I see:
>>
>> #ifdef WIN32
>>    300 /* implementation of pg_link_file() on Windows */
>>    301 static int
>>    302 win32_pghardlink(const char *src, const char *dst)
>>    303 {
>>    304     /*
>>    305      * CreateHardLinkA returns zero for failure
>>    306      * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
>>    307      */
>>    308     if (CreateHardLinkA(dst, src, NULL) == 0)
>>    309     {
>>    310         _dosmaperr(GetLastError());
>>    311         return -1;
>>    312     }
>>    313     else
>>    314         return 0;
>>    315 }
>>    316 #endif
>
> Great !
> So I did a full upgrade for nothing (just for safety), but that's good
> to know for next time !
> Should this be submitted to postgresql-bugs, or is there something more
> specific to the documentation ?

I just found the [hidden email] list, for for the spam.

--
Arnaud



--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

George Neuner
In reply to this post by Bruce Momjian
On Fri, 9 Jun 2017 10:07:24 -0400, Bruce Momjian <[hidden email]>
wrote:

>On Fri, Jun  9, 2017 at 12:00:56PM +0200, Arnaud L. wrote:
>> Hi
>>
>> The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use
>> junction points on Windows.
>> Shouldn't it rather user hard-links ?
>> If I'm not mistaken, with junction points (i.e. soft-links to directories),
>> the old data dir cannot be removed.
>> With hard-links to file, we can get rid of the old data dir once we are sure
>> that the upgrade is fine.
>
>I was told junction points on Windows were hard links and no one has
>ever complained about not being able to remove them.


NTFS junctions are a distinct type of symbolic link which is meant for
filesystem mount points.  In NTFS "normal" symlinks are restricted to
targets within the same filesystem.

You can use a junction anywhere you want a symlink, but not the
reverse.  The downside is that pathname parsing is slower with
junctions than with symlinks because of the possibility that the path
may cross into a different filesystem.


The documentation is not very clear, IMO.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363878(v=vs.85).aspx
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365503(v=vs.85).aspx


The mklink utility can create any of these types of links.  Its
documentation does not describe the differences, but is shows that
hard links, symlinks, and junctions all are distinct concepts in
Windows.

https://technet.microsoft.com/en-us/library/cc753194(v=ws.11).aspx


George



--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

George Neuner
In reply to this post by Adrian Klaver-4
On Fri, 9 Jun 2017 07:24:03 -0700, Adrian Klaver
<[hidden email]> wrote:


>https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx
>
>Seems to me the difference is hard links point to file, junctions to
>directories.

You can make either hard links or symlinks to files.  Junctions are
distinct from normal symlinks in that junctions can cross filesystems.
Microsoft's cmdline tools complain if you try to make a junction to a
file, because Microsoft intended junctions for mount points ... but
you can do it programmatically, or trick the tool by creating the link
and then replacing the target, and in most cases it will work the same
as a normal symlink.

I have seen cases where a junction to a file didn't work, but they
seemed to be application related rather than an OS issue.  Prior to
Vista, the mklink utility was not available, so people wanting to
create symlinks were forced to use the sysinternals junction utility.
https://technet.microsoft.com/en-us/sysinternals/bb545021.aspx

George



--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

kpi6288
In reply to this post by Adrian Klaver-4
> -----Ursprüngliche Nachricht-----
>
> I apologize for not being smarter on this thread.  When I helped with the
> Windows port, I was told Windows didn't have hard links for use by
tablespace
> directories, so I got it into my head that Windows didn't have hard links.
> Therefore, when I was writing the docs, I called them junction points.

It's actually not "Windows" providing hard links, it is the file system
NTFS. FAT and its modern cousins don't provide hard links - but this will
rarely be used for databases these days.
However, ReFS (introduced with server 2012 and providing some new features
like automatic integrity checks, clustering etc.) does no longer provide
hard links.
Are hard links used anywhere else but in pg_upgrade?

Klaus



--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Reply | Threaded
Open this post in threaded view
|

Re: pg_upgrade --link on Windows

Bruce Momjian
On Wed, Jun 14, 2017 at 09:59:04AM +0200, Klaus P. Pieper wrote:

> > -----Ursprüngliche Nachricht-----
> >
> > I apologize for not being smarter on this thread.  When I helped with the
> > Windows port, I was told Windows didn't have hard links for use by
> tablespace
> > directories, so I got it into my head that Windows didn't have hard links.
> > Therefore, when I was writing the docs, I called them junction points.
>
> It's actually not "Windows" providing hard links, it is the file system
> NTFS. FAT and its modern cousins don't provide hard links - but this will
> rarely be used for databases these days.
> However, ReFS (introduced with server 2012 and providing some new features
> like automatic integrity checks, clustering etc.) does no longer provide
> hard links.
> Are hard links used anywhere else but in pg_upgrade?

Nope, it is used only by transfer_relfile() calling linkFile().

--
  Bruce Momjian  <[hidden email]>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +


--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general