upcoming API changes for LLVM 12

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

upcoming API changes for LLVM 12

Andres Freund
Hi,

There will be a breaking API change for JIT related API in LLVM
12. Mostly about making control over various aspects easier, and then
building on top of that providing new features (like JIT compiling in
the background and making it easier to share JIT compiled output between
processes).

I've worked with Lang Hames to ensure that the new C API has feature
parity...

The postgres changes are fairly localized, all in llvmjit.c - it's just
a few #ifdefs to support both LLVM 12 and before.

The two questions I have are:

1) Which versions do we want to add LLVM 12 support?  It'd be fairly
   easy to backport all the way. But it's not quite a bugfix... OTOH,
   it'd probably painful for packagers to have dependencies on different
   versions of LLVM for different versions of postgres.

2) When do we want to add LLVM 12 support? PG will soon stop compiling
   against LLVM 12, which will be released in about 6 months. I worked
   with Lang to make most of the breaking changes in a branch (to be
   merged in the next few days), but it's possible that there will be a
   few smaller changes.

I'd be inclined to add support for LLVM 12 to master soon, and then
backpatch that support around LLVM 12's release.

Comments?

Greetings,

Andres Freund


Reply | Threaded
Open this post in threaded view
|

Re: upcoming API changes for LLVM 12

Thomas Munro-5
On Fri, Oct 16, 2020 at 2:12 PM Andres Freund <[hidden email]> wrote:
> There will be a breaking API change for JIT related API in LLVM
> 12. Mostly about making control over various aspects easier, and then
> building on top of that providing new features (like JIT compiling in
> the background and making it easier to share JIT compiled output between
> processes).
>
> I've worked with Lang Hames to ensure that the new C API has feature
> parity...

Cool!

> I'd be inclined to add support for LLVM 12 to master soon, and then
> backpatch that support around LLVM 12's release.

+1.  I guess Fabien's animal "seawasp" will turn red next week.
Apparently it rebuilds bleeding edge LLVM weekly (though strangely
last week it went backwards... huh).


Reply | Threaded
Open this post in threaded view
|

Re: upcoming API changes for LLVM 12

Álvaro Herrera
In reply to this post by Andres Freund
On 2020-Oct-15, Andres Freund wrote:

> There will be a breaking API change for JIT related API in LLVM
> 12. Mostly about making control over various aspects easier, and then
> building on top of that providing new features (like JIT compiling in
> the background and making it easier to share JIT compiled output between
> processes).
>
> I've worked with Lang Hames to ensure that the new C API has feature
> parity...

Whee, sounds pretty good ... (am I dreaming too much if I hope execution
starts with non-jitted and switches on the fly to jitted once
background compilation finishes?)

> 2) When do we want to add LLVM 12 support? PG will soon stop compiling
>    against LLVM 12, which will be released in about 6 months. I worked
>    with Lang to make most of the breaking changes in a branch (to be
>    merged in the next few days), but it's possible that there will be a
>    few smaller changes.

hmm, how regular are LLVM releases?  I mean, what if pg14 ends up being
released sooner than LLVM12 – would there be a problem?


Reply | Threaded
Open this post in threaded view
|

Re: upcoming API changes for LLVM 12

Andres Freund
Hi,

On 2020-10-16 02:45:51 -0300, Alvaro Herrera wrote:
> Whee, sounds pretty good ... (am I dreaming too much if I hope
> execution starts with non-jitted and switches on the fly to jitted
> once background compilation finishes?)

There's some more work needed to get there, but yes, the basics for that
are there now. It'd perhaps be doable with threads now, but it's not
clear we want that... We probably could build it with processes too -
it'd require some memory management fun, but it's doable.


> > 2) When do we want to add LLVM 12 support? PG will soon stop compiling
> >    against LLVM 12, which will be released in about 6 months. I worked
> >    with Lang to make most of the breaking changes in a branch (to be
> >    merged in the next few days), but it's possible that there will be a
> >    few smaller changes.
>
> hmm, how regular are LLVM releases?  I mean, what if pg14 ends up being
> released sooner than LLVM12 – would there be a problem?

Pretty unlikely - they're half yearly releases, and come out on a
somewhat regular schedule. They've moved a few weeks but not more. And
even if they did - having a few #ifdefs for LLVM 12 would be ok anyway.

Greetings,

Andres Freund


Reply | Threaded
Open this post in threaded view
|

Re: upcoming API changes for LLVM 12

Tom Lane-2
Andres Freund <[hidden email]> writes:
> On 2020-10-16 02:45:51 -0300, Alvaro Herrera wrote:
>>> 2) When do we want to add LLVM 12 support? PG will soon stop compiling
>>> against LLVM 12, which will be released in about 6 months. I worked
>>> with Lang to make most of the breaking changes in a branch (to be
>>> merged in the next few days), but it's possible that there will be a
>>> few smaller changes.

>> hmm, how regular are LLVM releases?  I mean, what if pg14 ends up being
>> released sooner than LLVM12 – would there be a problem?

> Pretty unlikely - they're half yearly releases, and come out on a
> somewhat regular schedule. They've moved a few weeks but not more. And
> even if they did - having a few #ifdefs for LLVM 12 would be ok anyway.

Yeah.  As long as we're not breaking the ability to build against older
LLVM, I can't see a reason not to apply and back-patch these changes.
We usually want all supported PG versions to build against newer tool
chains, and this seems to fall into that category.

                        regards, tom lane


Reply | Threaded
Open this post in threaded view
|

Re: upcoming API changes for LLVM 12

Andres Freund
Hi,

On 2020-10-16 10:22:57 -0400, Tom Lane wrote:
> Yeah.  As long as we're not breaking the ability to build against older
> LLVM, I can't see a reason not to apply and back-patch these changes.
> We usually want all supported PG versions to build against newer tool
> chains, and this seems to fall into that category.

Cool! I just ran that branch against 3.9 (the currently oldest supported
version), and that still works.


A related question is whether it'd be time to prune the oldest supported
LLVM version. 3.9.0 was released 2016-08-31 (and 3.9.1, the only point
release, was 2016-12-13). There's currently no *pressing* reason to
reduce it, but it is the cause of few #ifdefs - but more importantly it
increases the test matrix.

I'm inclined to just have a deterministic policy that we apply around
release time going forward. E.g. don't support versions that are newer
than the newest available LLVM version in the second newest
long-term-supported distribution release of RHEL, Ubuntu, Debian?

Regards,

Andres


Reply | Threaded
Open this post in threaded view
|

Re: upcoming API changes for LLVM 12

Tom Lane-2
Andres Freund <[hidden email]> writes:
> A related question is whether it'd be time to prune the oldest supported
> LLVM version. 3.9.0 was released 2016-08-31 (and 3.9.1, the only point
> release, was 2016-12-13). There's currently no *pressing* reason to
> reduce it, but it is the cause of few #ifdefs - but more importantly it
> increases the test matrix.

> I'm inclined to just have a deterministic policy that we apply around
> release time going forward. E.g. don't support versions that are newer
> than the newest available LLVM version in the second newest
> long-term-supported distribution release of RHEL, Ubuntu, Debian?

Meh.  I do not think these should be mechanistic one-size-fits-all
decisions.  A lot hinges on just how messy it is to continue support
for a given tool.  Moreover, the policy you propose above is
completely out of line with our approach to every other toolchain
we use.

I'd rather see an approach along the lines of "it's time to drop
support for LLVM version X because it can't do Y", rather than
"... because Z amount of time has passed".

                        regards, tom lane


Reply | Threaded
Open this post in threaded view
|

Re: upcoming API changes for LLVM 12

Álvaro Herrera
In reply to this post by Andres Freund
On 2020-Oct-16, Andres Freund wrote:

> A related question is whether it'd be time to prune the oldest supported
> LLVM version. 3.9.0 was released 2016-08-31 (and 3.9.1, the only point
> release, was 2016-12-13). There's currently no *pressing* reason to
> reduce it, but it is the cause of few #ifdefs - but more importantly it
> increases the test matrix.

Is there a matrix of LLVM versions supported by live distros?  It sounds
like pruning away 3.9 from branch master would be reasonable enough;
OTOH looking at the current LLVM support code in Postgres it doesn't
look like you would actually save all that much.  Maybe the picture
changes with things you're doing now, but it's not evident from what's
in the tree now.

> I'm inclined to just have a deterministic policy that we apply around
> release time going forward. E.g. don't support versions that are newer
> than the newest available LLVM version in the second newest
> long-term-supported distribution release of RHEL, Ubuntu, Debian?

It seems fair to think that new Postgres releases should be put in
production only with the newest LTS release of each OS -- no need to go
back to the second newest.  But I think we should use such a criteria to
drive discussion rather than as a battle axe chopping stuff away.