pg_preadv() and pg_pwritev()

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Thomas Munro-5
On Thu, Jan 14, 2021 at 9:26 AM Tom Lane <[hidden email]> wrote:
> * You need to remove pread.o and pwrite.o from the hard-wired
> part of the list in src/port/Makefile, else they get built
> whether needed or not.

Right, done.

> * I don't much like this in fd.h:
>
> @@ -46,6 +46,7 @@
>  #include <dirent.h>
>
>
> +struct iovec;
>  typedef int File;
>
> because it makes it look like iovec and File are of similar
> status, which they hardly are.  Perhaps more like
>
>  #include <dirent.h>
> +
> +struct iovec;                  /* avoid including sys/uio.h here */

Done, except I wrote port/pg_iovec.h.

> I confirm clean builds on Big Sur and Catalina with this.

Thanks for checking.  I also checked on Windows via CI.  Pushed.


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Tom Lane-2
In reply to this post by Tom Lane-2
I wrote:
> So I'm a little confused as to why this test is failing to fail
> with (I assume) newer Xcode.  Can we see the relevant part of
> config.log on your machine?

After further digging I believe I understand what's happening,
and it's a bit surprising we've not been bit by it before.
If the compiler believes (thanks to __API_AVAILABLE macros in
Apple's system headers) that a given library symbol might not
exist in the lowest macOS version it is compiling for, it will
still emit a normal call to that function ... but it also emits

       .weak_reference _preadv

marking the call as a weak reference.  If the linker then fails
to link that call, it doesn't throw an error, it just replaces
the call instruction with a NOP :-(.  This is why configure's
test appears to succeed, since it only checks whether you can
link not whether the call would work at runtime.  Apple's
assumption evidently is that you'll guard the call with a run-time
check to see if the function exists before you use it, and you
don't want your link to fail if it doesn't.

The solution to this, according to "man ld", is

     -no_weak_imports
                 Error if any symbols are weak imports (i.e. allowed to be
                 unresolved (NULL) at runtime). Useful for config based
                 projects that assume they are built and run on the same OS
                 version.

I don't particularly care that Apple is looking down their nose
at people who don't want to make their builds run on multiple OS
versions, so I think we should just use this and call it good.

Attached is an untested quick hack to make that happen --- Sergey,
can you verify that this fixes configure's results on your setup?

(This is not quite committable as-is, it needs something to avoid
adding -Wl,-no_weak_imports on ancient macOS versions.  But it
will do to see if the fix works on modern versions.)

                        regards, tom lane


diff --git a/src/template/darwin b/src/template/darwin
index 32414d21a9..22d04e7ba7 100644
--- a/src/template/darwin
+++ b/src/template/darwin
@@ -17,6 +17,9 @@ if test x"$PG_SYSROOT" != x"" ; then
   fi
 fi
 
+# Disable weak linking, else configure's checks will misbehave
+LDFLAGS="-Wl,-no_weak_imports $LDFLAGS"
+
 # Extra CFLAGS for code that will go into a shared library
 CFLAGS_SL=""
 
Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
Tom Lane wrote:
> The symptoms sound consistent with using bleeding-edge Xcode on a
> Catalina machine ... please report exact OS and Xcode versions.

macOS 10.15.7 (19H2)
Xcode 12.3 (12C33)
macOS SDK 11.1 (20C63)


> Attached is an untested quick hack to make that happen --- Sergey,
> can you verify that this fixes configure's results on your setup?

"-no_weak_imports" doesn't help.

configure:15161: checking for pwritev
configure:15161: gcc -o conftest -Wall -Wmissing-prototypes
-Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -Wno-unused-command-line-argument -O2 -isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
   -Wl,-no_weak_imports -isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
   conftest.c -lz  -lm  >&5
configure:15161: $? = 0
configure:15161: result: yes


Then I get the same compiler warnings about pwritev and an unrelated
link error:

ld: weak import of symbol '___darwin_check_fd_set_overflow' not
supported because of option: -no_weak_imports for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
make[2]: *** [postgres] Error 1
make[1]: *** [all-backend-recurse] Error 2
make: *** [all-src-recurse] Error 2

Please see the logs attached.

config.log (459K) Download Attachment
make.log (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
>> The symptoms sound consistent with using bleeding-edge Xcode on a
>> Catalina machine ... please report exact OS and Xcode versions.
>
> macOS 10.15.7 (19H2)
> Xcode 12.3 (12C33)
> macOS SDK 11.1 (20C63)
>

Everything is fine if I run "configure" with
PG_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

I noticed that "cc" invoked from command line uses:
-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

But "xcodebuild -version -sdk macosx Path" invoked by "configure" when
PG_SYSROOT is not provided gives:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk

Now I'm confused about different SDK versions and locations used by
Xcode and CommandLineTools :)


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Tom Lane-2
Sergey Shinderuk <[hidden email]> writes:
> I noticed that "cc" invoked from command line uses:
> -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

Hm, how did you determine that exactly?

                        regards, tom lane


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
On 14.01.2021 18:42, Tom Lane wrote:
>> I noticed that "cc" invoked from command line uses:
>> -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
>
> Hm, how did you determine that exactly?
>

% echo 'int main(void){}' >tmp.c
% cc -v tmp.c
Apple clang version 12.0.0 (clang-1200.0.32.28)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang"
-cc1 -triple x86_64-apple-macosx10.15.0 -Wdeprecated-objc-isa-usage
-Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration
-emit-obj -mrelax-all -disable-free -disable-llvm-verifier
-discard-value-names -main-file-name tmp.c -mrelocation-model pic
-pic-level 2 -mthread-model posix -mframe-pointer=all -fno-strict-return
-masm-verbose -munwind-tables -target-sdk-version=10.15.6
-fcompatibility-qualified-id-block-type-checking -target-cpu penryn
-dwarf-column-info -debugger-tuning=lldb -target-linker-version 609.8 -v
-resource-dir
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0
-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
-I/usr/local/include -internal-isystem
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include
-internal-isystem
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include
-internal-externc-isystem
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include
-internal-externc-isystem
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
-Wno-reorder-init-list -Wno-implicit-int-float-conversion
-Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt
-Wno-misleading-indentation -Wno-quoted-include-in-framework-header
-Wno-implicit-fallthrough -Wno-enum-enum-conversion
-Wno-enum-float-conversion -fdebug-compilation-dir /Users/shinderuk
-ferror-limit 19 -fmessage-length 238 -stack-protector 1 -fstack-check
-mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature
-fregister-global-dtors-with-atexit -fgnuc-version=4.2.1
-fobjc-runtime=macosx-10.15.0 -fmax-type-align=16
-fdiagnostics-show-option -fcolor-diagnostics -o
/var/folders/8x/jvqv7hyd5h98m7tz2zm9r0yc0000gn/T/tmp-91fb5e.o -x c tmp.c
clang -cc1 version 12.0.0 (clang-1200.0.32.28) default target
x86_64-apple-darwin19.6.0
ignoring nonexistent directory
"/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include"
ignoring nonexistent directory
"/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
  /usr/local/include
 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include
  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include
 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks
(framework directory)
End of search list.
 
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld"
-demangle -lto_library
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib
-no_deduplicate -dynamic -arch x86_64 -platform_version macos 10.15.0
10.15.6 -syslibroot
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -o a.out
-L/usr/local/lib
/var/folders/8x/jvqv7hyd5h98m7tz2zm9r0yc0000gn/T/tmp-91fb5e.o -lSystem
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.osx.a


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Tom Lane-2
Sergey Shinderuk <[hidden email]> writes:
> On 14.01.2021 18:42, Tom Lane wrote:
>>> I noticed that "cc" invoked from command line uses:
>>> -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

>> Hm, how did you determine that exactly?

> % cc -v tmp.c
> ...
> -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

Okay, interesting.  On my Catalina machine, I see

-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

which is also a 10.15 SDK, since I haven't upgraded Xcode past 12.0.
I wonder if that would change if I did upgrade (but I don't plan to
risk it, since this is my only remaining Catalina install).

After considerable playing around, I'm guessing that the reason
-no_weak_imports doesn't help is that it rejects calls that are
marked as weak references on the *calling* side.  Since AC_CHECK_FUNCS
doesn't bother to #include the relevant header file, the compiler
doesn't know that preadv() ought to be marked as a weak reference.
Then, when the test program gets linked against the stub libc that's
provided by the SDK, there is a version of preadv() there so no link
failure occurs.  (There are way more moving parts in this weak-reference
thing than I'd realized.)

It seems like the more productive approach would be to try to identify
the right sysroot to use.  I wonder if there is some less messy way
to find out the compiler's default sysroot than to scrape it out of
-v output.

Another thing I've been realizing while poking at this is that we
might not need to set -isysroot explicitly at all, which would then
lead to the compiler using its default sysroot automatically.
In some experimentation, it seems like what we need PG_SYSROOT for
is just for configure to be able to find tclConfig.sh and the Perl
header files.  So at this point I'm tempted to try ripping that
out altogether.  If you remove the lines in src/template/darwin
that inject PG_SYSROOT into CPPFLAGS and LDFLAGS, do things
work for you?

                        regards, tom lane


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Tom Lane-2
I wrote:
> It seems like the more productive approach would be to try to identify
> the right sysroot to use.  I wonder if there is some less messy way
> to find out the compiler's default sysroot than to scrape it out of
> -v output.

This is, of course, not terribly well documented by Apple.  But
Mr. Google suggests that "xcrun --show-sdk-path" might serve.
What does that print for you?

                        regards, tom lane


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Tom Lane-2
I borrowed my wife's Mac, which is still on Catalina and up to now
never had Xcode on it, and found some very interesting things.

Step 1: download/install Xcode 12.3, open it, agree to license,
wait for it to finish "installing components".

At this point, /Library/Developer/CommandLineTools doesn't exist,
and we have the following outputs from various probe commands:

% xcrun --show-sdk-path            
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
% xcrun --sdk macosx --show-sdk-path    
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
% xcodebuild -version -sdk macosx Path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk

Also, cc -v reports
  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

Unsurprisingly, Xcode 12.3 itself only contains

% ls -l /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs
total 0
drwxr-xr-x  5 root  wheel  160 Nov 30 07:27 DriverKit20.2.sdk
drwxr-xr-x  7 root  wheel  224 Nov 30 07:27 MacOSX.sdk
lrwxr-xr-x  1 root  wheel   10 Jan 14 15:57 MacOSX11.1.sdk -> MacOSX.sdk

Step 2: install command line tools (I used "xcode-select --install"
to fire this off, rather than the Xcode menu item).

Now I have

% ls -l /Library/Developer/CommandLineTools/SDKs
total 0
lrwxr-xr-x  1 root  wheel   14 Jan 14 16:42 MacOSX.sdk -> MacOSX11.1.sdk
drwxr-xr-x  8 root  wheel  256 Jul  9  2020 MacOSX10.15.sdk
drwxr-xr-x  7 root  wheel  224 Nov 30 07:33 MacOSX11.1.sdk

which is pretty interesting in itself, because the same directory on
my recently-updated-to-Big-Sur Macs does NOT have the 11.1 SDK.
I wonder what determines which versions get installed here.

More interesting yet:

% xcrun --show-sdk-path            
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
% xcrun --sdk macosx --show-sdk-path  
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
% xcodebuild -version -sdk macosx Path          
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk

and cc -v reports
  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

So apparently, "xcrun --show-sdk-path" (without any -sdk option)
is the most authoritative guide to the compiler's default sysroot.

However, googling turns up various people reporting that "xcrun
--show-sdk-path" returns an empty string for them, and our last
major investigation into this [1] found that there are some system
states where the compiler appears to have no default sysroot,
which I bet is the same thing.  I do not at this point have a recipe
to reproduce such a state, but we'd be fools to imagine it's no
longer possible.  My guess about it is that Apple's processes for
updating the default sysroot during system updates are just plain
buggy, with various corner cases that have ill-understood causes.

Also, after re-reading [1] I am not at all excited about trying to
remove the -isysroot switches from our *FLAGS.  What I propose to do
is keep that, but improve our mechanism for choosing a default value
for PG_SYSROOT.  It looks like first trying "xcrun --show-sdk-path",
and falling back to "xcodebuild -version -sdk macosx Path" if that
doesn't yield a valid path, is more likely to give a working build
than relying entirely on xcodebuild.  Maybe there's a case for trying
"xcrun --sdk macosx --show-sdk-path" in between; in my tests that
seemed noticeably faster than invoking xcodebuild, and I've not yet
seen a case where it gave a different answer.

Thoughts?

                        regards, tom lane

[1] https://www.postgresql.org/message-id/flat/20840.1537850987%40sss.pgh.pa.us


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
On 15.01.2021 01:13, Tom Lane wrote:

> I borrowed my wife's Mac, which is still on Catalina and up to now
> never had Xcode on it, and found some very interesting things.
>
> Step 1: download/install Xcode 12.3, open it, agree to license,
> wait for it to finish "installing components".
>
> At this point, /Library/Developer/CommandLineTools doesn't exist,
> and we have the following outputs from various probe commands:
>
> % xcrun --show-sdk-path
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
> % xcrun --sdk macosx --show-sdk-path
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
> % xcodebuild -version -sdk macosx Path
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
>
> Also, cc -v reports
>    -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
>
> Unsurprisingly, Xcode 12.3 itself only contains
>
> % ls -l /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs
> total 0
> drwxr-xr-x  5 root  wheel  160 Nov 30 07:27 DriverKit20.2.sdk
> drwxr-xr-x  7 root  wheel  224 Nov 30 07:27 MacOSX.sdk
> lrwxr-xr-x  1 root  wheel   10 Jan 14 15:57 MacOSX11.1.sdk -> MacOSX.sdk
>
> Step 2: install command line tools (I used "xcode-select --install"
> to fire this off, rather than the Xcode menu item).
>
> Now I have
>
> % ls -l /Library/Developer/CommandLineTools/SDKs
> total 0
> lrwxr-xr-x  1 root  wheel   14 Jan 14 16:42 MacOSX.sdk -> MacOSX11.1.sdk
> drwxr-xr-x  8 root  wheel  256 Jul  9  2020 MacOSX10.15.sdk
> drwxr-xr-x  7 root  wheel  224 Nov 30 07:33 MacOSX11.1.sdk
>
> which is pretty interesting in itself, because the same directory on
> my recently-updated-to-Big-Sur Macs does NOT have the 11.1 SDK.
> I wonder what determines which versions get installed here.
>
> More interesting yet:
>
> % xcrun --show-sdk-path
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
> % xcrun --sdk macosx --show-sdk-path
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
> % xcodebuild -version -sdk macosx Path
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
>
> and cc -v reports
>    -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
>
> So apparently, "xcrun --show-sdk-path" (without any -sdk option)
> is the most authoritative guide to the compiler's default sysroot.
>
> However, googling turns up various people reporting that "xcrun
> --show-sdk-path" returns an empty string for them, and our last
> major investigation into this [1] found that there are some system
> states where the compiler appears to have no default sysroot,
> which I bet is the same thing.  I do not at this point have a recipe
> to reproduce such a state, but we'd be fools to imagine it's no
> longer possible.  My guess about it is that Apple's processes for
> updating the default sysroot during system updates are just plain
> buggy, with various corner cases that have ill-understood causes.
>
> Also, after re-reading [1] I am not at all excited about trying to
> remove the -isysroot switches from our *FLAGS.  What I propose to do
> is keep that, but improve our mechanism for choosing a default value
> for PG_SYSROOT.  It looks like first trying "xcrun --show-sdk-path",
> and falling back to "xcodebuild -version -sdk macosx Path" if that
> doesn't yield a valid path, is more likely to give a working build
> than relying entirely on xcodebuild.  Maybe there's a case for trying
> "xcrun --sdk macosx --show-sdk-path" in between; in my tests that
> seemed noticeably faster than invoking xcodebuild, and I've not yet
> seen a case where it gave a different answer.
>
> Thoughts?
>
> regards, tom lane
>
> [1] https://www.postgresql.org/message-id/flat/20840.1537850987%40sss.pgh.pa.us
>

Thanks for thorough investigation and sorry for the late reply.

I spent quite some time trying to understand / reverse engineer the
logic behind xcrun's default SDK selection. Apparently, "man xcrun" is
not accurate saying:

        The  SDK  which  will  be searched defaults to the most recent
available...

I didn't find anything really useful or helpful.
"/Library/Developer/CommandLineTools" is hardcoded into
"libxcrun.dylib". On my machine xcrun scans

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs

and

/Library/Developer/CommandLineTools/SDKs

in that order, and loads "SDKSettings.plist" from each subdirectory. I
looked into plists, but couldn't find anything special about
"MacOSX10.15.sdk".


Okay, here is what I have:

% ls -l
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs
total 0
drwxr-xr-x  5 root  wheel  160 Nov 30 15:27 DriverKit20.2.sdk
drwxr-xr-x  7 root  wheel  224 Nov 30 15:27 MacOSX.sdk
lrwxr-xr-x  1 root  wheel   10 Dec 17 14:25 MacOSX11.1.sdk -> MacOSX.sdk

% ls -l /Library/Developer/CommandLineTools/SDKs
total 0
lrwxr-xr-x  1 root  wheel   14 Nov 17 02:21 MacOSX.sdk -> MacOSX11.0.sdk
drwxr-xr-x  8 root  wheel  256 Nov 17 02:22 MacOSX10.15.sdk
drwxr-xr-x  7 root  wheel  224 Oct 19 23:39 MacOSX11.0.sdk

% xcrun --show-sdk-path
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

% xcrun --sdk macosx --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk

Oh, that's weird! Nevertheless I like you suggestion to call "xcrun"
from "configure".

Adding "--verbose" doesn't really explain anything, but just in case.

% xcrun --verbose --no-cache --find cc
xcrun: note: PATH =
'/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin'
xcrun: note: SDKROOT =
'/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk'
xcrun: note: TOOLCHAINS = ''
xcrun: note: DEVELOPER_DIR = '/Applications/Xcode.app/Contents/Developer'
xcrun: note: XCODE_DEVELOPER_USR_PATH = ''
xcrun: note: xcrun_db =
'/var/folders/8x/jvqv7hyd5h98m7tz2zm9r0yc0000gn/T/xcrun_db'
xcrun: note: xcrun via cc (xcrun)
xcrun: note: database key is:
cc|/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk||/Applications/Xcode.app/Contents/Developer|
xcrun: note: looking up with
'/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
macosx -find cc 2> /dev/null'
xcrun: note: lookup resolved with 'xcodebuild -find' to
'/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc


% xcrun --verbose --no-cache --sdk macosx --find cc
xcrun: note: looking up SDK with
'/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
macosx -version Path'
xcrun: note: PATH =
'/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin'
xcrun: note: SDKROOT = 'macosx'
xcrun: note: TOOLCHAINS = ''
xcrun: note: DEVELOPER_DIR = '/Applications/Xcode.app/Contents/Developer'
xcrun: note: XCODE_DEVELOPER_USR_PATH = ''
xcrun: note: xcrun_db =
'/var/folders/8x/jvqv7hyd5h98m7tz2zm9r0yc0000gn/T/xcrun_db'
xcrun: note: lookup resolved to:
'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk'
xcrun: note: looking up SDK with
'/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
-version PlatformPath'
xcrun: note: PATH =
'/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin'
xcrun: note: SDKROOT =
'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk'
xcrun: note: TOOLCHAINS = ''
xcrun: note: DEVELOPER_DIR = '/Applications/Xcode.app/Contents/Developer'
xcrun: note: XCODE_DEVELOPER_USR_PATH = ''
xcrun: note: xcrun_db =
'/var/folders/8x/jvqv7hyd5h98m7tz2zm9r0yc0000gn/T/xcrun_db'
xcrun: note: lookup resolved to:
'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform'
xcrun: note: PATH =
'/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin'
xcrun: note: SDKROOT =
'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk'
xcrun: note: TOOLCHAINS = ''
xcrun: note: DEVELOPER_DIR = '/Applications/Xcode.app/Contents/Developer'
xcrun: note: XCODE_DEVELOPER_USR_PATH = ''
xcrun: note: xcrun_db =
'/var/folders/8x/jvqv7hyd5h98m7tz2zm9r0yc0000gn/T/xcrun_db'
xcrun: note: xcrun via cc (xcrun)
xcrun: note: database key is:
cc|/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk||/Applications/Xcode.app/Contents/Developer|
xcrun: note: looking up with
'/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
-find cc 2> /dev/null'
xcrun: note: lookup resolved with 'xcodebuild -find' to
'/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
In reply to this post by Tom Lane-2
On 14.01.2021 21:05, Tom Lane wrote:

> After considerable playing around, I'm guessing that the reason
> -no_weak_imports doesn't help is that it rejects calls that are
> marked as weak references on the *calling* side.  Since AC_CHECK_FUNCS
> doesn't bother to #include the relevant header file, the compiler
> doesn't know that preadv() ought to be marked as a weak reference.
> Then, when the test program gets linked against the stub libc that's
> provided by the SDK, there is a version of preadv() there so no link
> failure occurs.  (There are way more moving parts in this weak-reference
> thing than I'd realized.)
>

Oh, that's interesting. I've just played with it a bit and it looks
exactly as you say.

> Another thing I've been realizing while poking at this is that we
> might not need to set -isysroot explicitly at all, which would then
> lead to the compiler using its default sysroot automatically.
> In some experimentation, it seems like what we need PG_SYSROOT for
> is just for configure to be able to find tclConfig.sh and the Perl
> header files.  So at this point I'm tempted to try ripping that
> out altogether.  If you remove the lines in src/template/darwin
> that inject PG_SYSROOT into CPPFLAGS and LDFLAGS, do things
> work for you?

Yes, it works fine.


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Tom Lane-2
In reply to this post by Sergey Shinderuk
Sergey Shinderuk <[hidden email]> writes:

> On 15.01.2021 01:13, Tom Lane wrote:
>> Also, after re-reading [1] I am not at all excited about trying to
>> remove the -isysroot switches from our *FLAGS.  What I propose to do
>> is keep that, but improve our mechanism for choosing a default value
>> for PG_SYSROOT.  It looks like first trying "xcrun --show-sdk-path",
>> and falling back to "xcodebuild -version -sdk macosx Path" if that
>> doesn't yield a valid path, is more likely to give a working build
>> than relying entirely on xcodebuild.  Maybe there's a case for trying
>> "xcrun --sdk macosx --show-sdk-path" in between; in my tests that
>> seemed noticeably faster than invoking xcodebuild, and I've not yet
>> seen a case where it gave a different answer.

> I spent quite some time trying to understand / reverse engineer the
> logic behind xcrun's default SDK selection.

Yeah, I wasted a fair amount of time on that too, going so far as
to ktrace xcrun (as I gather you did too).  I'm not any more
enlightened than you are about exactly how it's making the choice.

> Oh, that's weird! Nevertheless I like you suggestion to call "xcrun"
> from "configure".

Anyway, after re-reading the previous thread, something I like about
the current behavior is that it tends to produce a version-numbered
sysroot path, ie something ending in "MacOSX11.1.sdk" or whatever.
One of the hazards we're trying to avoid is some parts of a PG
installation being built against one SDK version while other parts are
built against another.  The typical behavior of "xcrun --show-sdk-path"
seems to be to produce a path ending in "MacOSX.sdk", which defeats that.
So I think we should accept the path only if it contains a version number,
and otherwise move on to the other probe commands.

Hence, I propose the attached.  This works as far as I can tell
to fix the problem you're seeing.

                        regards, tom lane


diff --git a/src/template/darwin b/src/template/darwin
index 32414d21a9..0c890482fe 100644
--- a/src/template/darwin
+++ b/src/template/darwin
@@ -3,11 +3,26 @@
 # Note: Darwin is the original code name for macOS, also known as OS X.
 # We still use "darwin" as the port name, partly because config.guess does.
 
-# Select where system include files should be sought.
+# Select where system include files should be sought, if user didn't say.
 if test x"$PG_SYSROOT" = x"" ; then
-  PG_SYSROOT=`xcodebuild -version -sdk macosx Path 2>/dev/null`
+  # This is far more complicated than it ought to be.  We first ask
+  # "xcrun --show-sdk-path", which seems to match the default -isysroot
+  # setting of Apple's compilers.  However, that may produce no result or
+  # a result that is not version-specific (i.e., just ".../SDKs/MacOSX.sdk").
+  # Having a version-specific sysroot seems desirable, so if there are not
+  # digits in the result string, try "xcrun --sdk macosx --show-sdk-path";
+  # and if that still doesn't work, fall back to asking xcodebuild.
+  PG_SYSROOT=`xcrun --show-sdk-path 2>/dev/null`
+  if expr x"$PG_SYSROOT" : '.*[0-9]\.[0-9]' >/dev/null ; then : okay
+  else
+    PG_SYSROOT=`xcrun --sdk macosx --show-sdk-path 2>/dev/null`
+    if expr x"$PG_SYSROOT" : '.*[0-9]\.[0-9]' >/dev/null ; then : okay
+    else
+      PG_SYSROOT=`xcodebuild -version -sdk macosx Path 2>/dev/null`
+    fi
+  fi
 fi
-# Old xcodebuild versions may produce garbage, so validate the result.
+# Validate the result: if it doesn't point at a directory, ignore it.
 if test x"$PG_SYSROOT" != x"" ; then
   if test -d "$PG_SYSROOT" ; then
     CPPFLAGS="-isysroot $PG_SYSROOT $CPPFLAGS"
Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
In reply to this post by Tom Lane-2
On 15.01.2021 01:13, Tom Lane wrote:

> than relying entirely on xcodebuild.  Maybe there's a case for trying
> "xcrun --sdk macosx --show-sdk-path" in between; in my tests that
> seemed noticeably faster than invoking xcodebuild, and I've not yet
> seen a case where it gave a different answer.
>

I see that "xcrun --sdk macosx --show-sdk-path" really calls
"/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
macosx -version Path" under the hood.


% lldb -- xcrun --no-cache --sdk macosx --show-sdk-path
(lldb) target create "xcrun"
Current executable set to 'xcrun' (x86_64).
(lldb) settings set -- target.run-args  "--no-cache" "--sdk" "macosx"
"--show-sdk-path"
(lldb) settings set target.unset-env-vars SDKROOT
(lldb) b popen
Breakpoint 1: where = libsystem_c.dylib`popen, address = 0x00007fff67265607
(lldb) r
Process 26857 launched: '/usr/bin/xcrun' (x86_64)
Process 26857 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
     frame #0: 0x00007fff6e313607 libsystem_c.dylib`popen
libsystem_c.dylib`popen:
->  0x7fff6e313607 <+0>: pushq  %rbp
     0x7fff6e313608 <+1>: movq   %rsp, %rbp
     0x7fff6e31360b <+4>: pushq  %r15
     0x7fff6e31360d <+6>: pushq  %r14
Target 0: (xcrun) stopped.
(lldb) p (char *)$arg1
(char *) $1 = 0x0000000100406960
"/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
macosx -version Path"


% sudo dtrace -n 'pid$target::popen:entry { trace(copyinstr(arg0)) }' -c
'xcrun --sdk macosx --show-sdk-path'
dtrace: description 'pid$target::popen:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
   0 413269                      popen:entry
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
macosx -version Path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
dtrace: pid 26905 has exited


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Tom Lane-2
Sergey Shinderuk <[hidden email]> writes:
> I see that "xcrun --sdk macosx --show-sdk-path" really calls
> "/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
> macosx -version Path" under the hood.

Hmm.  I found something odd on my wife's Mac: although on my other
machines, I get something like

$ xcrun --verbose --no-cache --show-sdk-path
xcrun: note: looking up SDK with '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -version PlatformPath'
xcrun: note: PATH = '/Users/tgl/testversion/bin:/usr/local/autoconf-2.69/bin:/Users/tgl/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/Library/Tcl/bin:/opt/X11/bin'
xcrun: note: SDKROOT = '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk'
xcrun: note: TOOLCHAINS = ''
xcrun: note: DEVELOPER_DIR = '/Applications/Xcode.app/Contents/Developer'
xcrun: note: XCODE_DEVELOPER_USR_PATH = ''
xcrun: note: xcrun_db = '/var/folders/3p/2bnrmypd17jcqbtzw79t9blw0000gn/T/xcrun_db'
xcrun: note: lookup resolved to: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform'
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

on her machine there's no detail at all:

% xcrun --verbose --no-cache --show-sdk-path
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

So I'm not sure what to make of that.  But I'm hesitant to assume
that xcrun is just a wrapper around xcodebuild.

                        regards, tom lane


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
In reply to this post by Sergey Shinderuk
On 15.01.2021 04:53, Sergey Shinderuk wrote:

> I see that "xcrun --sdk macosx --show-sdk-path" really calls
> "/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk
> macosx -version Path" under the hood.
>

... and caches the result. xcodebuild not called without --no-cache.
So it still make sense to fall back on xcodebuild.

% sudo dtrace -n 'pid$target::popen:entry { trace(copyinstr(arg0)) }' -c
'xcrun --sdk macosx --show-sdk-path'
dtrace: description 'pid$target::popen:entry ' matched 1 probe
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
dtrace: pid 26981 has exited


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
In reply to this post by Tom Lane-2
On 15.01.2021 05:04, Tom Lane wrote:
>
> on her machine there's no detail at all:
>
> % xcrun --verbose --no-cache --show-sdk-path
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
>

The same on my machine. I get details for --find, but not for
--show-sdk-path.


> So I'm not sure what to make of that.  But I'm hesitant to assume
> that xcrun is just a wrapper around xcodebuild.
>

I agree.


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Sergey Shinderuk
In reply to this post by Tom Lane-2
On 15.01.2021 04:45, Tom Lane wrote:
> Hence, I propose the attached.  This works as far as I can tell
> to fix the problem you're seeing.
Yes, it works fine. Thank you very much.


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

Tom Lane-2
Sergey Shinderuk <[hidden email]> writes:
> On 15.01.2021 04:45, Tom Lane wrote:
>> Hence, I propose the attached.  This works as far as I can tell
>> to fix the problem you're seeing.

> Yes, it works fine. Thank you very much.

Great.  Pushed with a little more polishing.

                        regards, tom lane


Reply | Threaded
Open this post in threaded view
|

Re: pg_preadv() and pg_pwritev()

James Hilliard
In reply to this post by Tom Lane-2
On Thu, Jan 14, 2021 at 6:45 PM Tom Lane <[hidden email]> wrote:

>
> Sergey Shinderuk <[hidden email]> writes:
> > On 15.01.2021 01:13, Tom Lane wrote:
> >> Also, after re-reading [1] I am not at all excited about trying to
> >> remove the -isysroot switches from our *FLAGS.  What I propose to do
> >> is keep that, but improve our mechanism for choosing a default value
> >> for PG_SYSROOT.  It looks like first trying "xcrun --show-sdk-path",
> >> and falling back to "xcodebuild -version -sdk macosx Path" if that
> >> doesn't yield a valid path, is more likely to give a working build
> >> than relying entirely on xcodebuild.  Maybe there's a case for trying
> >> "xcrun --sdk macosx --show-sdk-path" in between; in my tests that
> >> seemed noticeably faster than invoking xcodebuild, and I've not yet
> >> seen a case where it gave a different answer.
>
> > I spent quite some time trying to understand / reverse engineer the
> > logic behind xcrun's default SDK selection.
>
> Yeah, I wasted a fair amount of time on that too, going so far as
> to ktrace xcrun (as I gather you did too).  I'm not any more
> enlightened than you are about exactly how it's making the choice.
>
> > Oh, that's weird! Nevertheless I like you suggestion to call "xcrun"
> > from "configure".
>
> Anyway, after re-reading the previous thread, something I like about
> the current behavior is that it tends to produce a version-numbered
> sysroot path, ie something ending in "MacOSX11.1.sdk" or whatever.
> One of the hazards we're trying to avoid is some parts of a PG
> installation being built against one SDK version while other parts are
> built against another.  The typical behavior of "xcrun --show-sdk-path"
> seems to be to produce a path ending in "MacOSX.sdk", which defeats that.
> So I think we should accept the path only if it contains a version number,
> and otherwise move on to the other probe commands.
I don't think enforcing a specific naming scheme makes sense, the minimum
OSX runtime version is effectively entirely separate from the SDK version.

The pwritev issue just seems to be caused by a broken configure check,
I've fixed that here:
https://postgr.es/m/20210119111625.20435-1-james.hilliard1%40gmail.com
>
> Hence, I propose the attached.  This works as far as I can tell
> to fix the problem you're seeing.
>
>                         regards, tom lane
>


12