Do we really want to migrate plproxy and citext into PG core distribution?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
82 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Do we really want to migrate plproxy and citext into PG core distribution?

Tom Lane-2
The current commitfest queue has two entries that propose to migrate
existing pgfoundry projects (or improved versions thereof) into our
core distribution.  The more I think about this the less happy I am
with it.  From a maintenance point of view there seems little need
for either project to get integrated: they don't appear to have much
of any code that is tightly tied to backend innards.  From a features
point of view, yeah they're cool, but there are scads of cool things
out there.  From a project-management point of view, it's insanity
to set a presumption that pgfoundry is just a proving ground for code
that should eventually get into core once it's mature enough or popular
enough or whatever.  We *have to* encourage the development of a cloud
of subprojects around the core, or core will eventually collapse of
its own weight.  We have not got the manpower to deal with an
ever-inflating collection of allegedly "core" code.  If anything,
we ought to be working to push more stuff out of the core distro so
that we can focus on the functionality that has to be there.

So my feeling is that we should not accept either of these patches.

Now, there is some value in submitting the code for review --- certainly
citext is a whole lot better than it was a few weeks ago.  I think it
would be a good idea to be open to reviewing pgfoundry code with the
same standards we'd use if we were going to integrate it.  Perhaps
commitfest is not the right venue for that, though, if only because
of the possibility of confusion over what's supposed to happen.

Comments?

                        regards, tom lane

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Josh berkus
Tom,

> Comments?

Well, in the *general* case, I think if we're going to have "first
class" pgfoundry projects, then having a unified "official" Kitchen Sink
Package will all of these add-ins becomes an imperative priority for
8.4.   EDB's recent open sourcing of their installer might help with this.

Futher, we would need to come up with some organized way to subject
pgFoundry projects to the same level of general scrutiny which core code
gets.  Or at least close.

In the specific cases of pl/proxy and citext, they are very much in line
with what we already package with the core code, including things like
dblink, ISN, and CIDR.  citext in particular would eliminate a long-time
newbie complaint about Postgres, but not if it's in an add-in package
which the user can't find binaries for.

So I would argue "maybe" on pl/proxy, but that citext does belong in core.

--Josh Berkus




--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

David E. Wheeler
In reply to this post by Tom Lane-2
On Jul 21, 2008, at 12:43, Tom Lane wrote:

> From a maintenance point of view there seems little need
> for either project to get integrated: they don't appear to have much
> of any code that is tightly tied to backend innards.

Well, citext against CVS HEAD is quite different from the other  
version I maintain for 8.3. The latter copies the str_toloer() code  
out of formatting.c from CVS and adds a number of includes in order to  
get things to work the same as against HEAD. I could probably work  
around this, though, if there was a macro with the version number in it.

> Now, there is some value in submitting the code for review ---  
> certainly
> citext is a whole lot better than it was a few weeks ago.

Absolutely. I really appreciate the feedback and comments I've  
received. Thank you!

> I think it
> would be a good idea to be open to reviewing pgfoundry code with the
> same standards we'd use if we were going to integrate it.  Perhaps
> commitfest is not the right venue for that, though, if only because
> of the possibility of confusion over what's supposed to happen.
>
> Comments?

I think that this is a very good idea. But you might have trouble  
motivating people to review code that won't be in core unless it's  
managed very diligently. An official extended library distribution, as  
Josh suggests, would probably help with this, as it then becomes a  
project alongside PostgreSQL that bundles a lot of great add-ons,  
rather than just leaving all the add-ons to themselves on pgFoundry.

Best,

David


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Marc Munro
In reply to this post by Tom Lane-2
On Mon, 2008-07-21 at 17:03 -0300, Tom Lane wrote:
> [. . .]  I think it
> would be a good idea to be open to reviewing pgfoundry code with the
> same standards we'd use if we were going to integrate it.  Perhaps
> commitfest is not the right venue for that, though, if only because
> of the possibility of confusion over what's supposed to happen.

I think this would be a great idea.  I would be overjoyed to have veil
http://pgfoundry.org/projects/veil/ ´╗┐reviewed by postgres developers.


__
Marc

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

David E. Wheeler
In reply to this post by Josh berkus
On Jul 21, 2008, at 12:53, Josh Berkus wrote:

> In the specific cases of pl/proxy and citext, they are very much in  
> line with what we already package with the core code, including  
> things like dblink, ISN, and CIDR.  citext in particular would  
> eliminate a long-time newbie complaint about Postgres, but not if  
> it's in an add-in package which the user can't find binaries for.
>
> So I would argue "maybe" on pl/proxy, but that citext does belong in  
> core.

This is my view, as well. If it was in contrib, it'd go a long way  
toward addressing a commonly-requested feature, whereas things are  
much more difficult to find on pgFoundry. pgFoundry ain't the CPAN,  
alas. Even if users do find it in pgFoundry, the fact that it isn't in  
core is more likely to be seen as a red flag at this point. One might  
ask, why isn't it in core? What's wrong with it? Why is something that  
seems so useful relegated to pgFoundry? What's the usual quality of  
code on pgFoundry?

Thanks for your consideration!

Best,

David

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Tom Lane-2
In reply to this post by Josh berkus
Josh Berkus <[hidden email]> writes:
> So I would argue "maybe" on pl/proxy, but that citext does belong in core.

Well, at least citext is pretty tiny ...

                        regards, tom lane

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Andrew Dunstan
In reply to this post by Tom Lane-2


Tom Lane wrote:

> The current commitfest queue has two entries that propose to migrate
> existing pgfoundry projects (or improved versions thereof) into our
> core distribution.  The more I think about this the less happy I am
> with it.  From a maintenance point of view there seems little need
> for either project to get integrated: they don't appear to have much
> of any code that is tightly tied to backend innards.  From a features
> point of view, yeah they're cool, but there are scads of cool things
> out there.  From a project-management point of view, it's insanity
> to set a presumption that pgfoundry is just a proving ground for code
> that should eventually get into core once it's mature enough or popular
> enough or whatever.


I think there is a case to say that modules that are sufficiently
popular have a case to be in core.

That's not necessarily a large number, but there might well be a case
for citext at least to be among the number at some stage. Surely a case
insensitive text type has more general use than, say, the seg module.



>  We *have to* encourage the development of a cloud
> of subprojects around the core, or core will eventually collapse of
> its own weight.  We have not got the manpower to deal with an
> ever-inflating collection of allegedly "core" code.  If anything,
> we ought to be working to push more stuff out of the core distro so
> that we can focus on the functionality that has to be there.
>  


When we can get the much discussed module infrastructure that will make
more sense. We will still need to keep enough modules to make sure that
the infrastructure is working. In general I feel that the number of
modules we have in core is about right. Maybe a small number should be
pushed out.

> So my feeling is that we should not accept either of these patches.
>
> Now, there is some value in submitting the code for review --- certainly
> citext is a whole lot better than it was a few weeks ago.  I think it
> would be a good idea to be open to reviewing pgfoundry code with the
> same standards we'd use if we were going to integrate it.  Perhaps
> commitfest is not the right venue for that, though, if only because
> of the possibility of confusion over what's supposed to happen.
>
> Comments?
>
>


If we don't have enough resources to maintain them do we have enough to review them?



I was going to write some stuff about citext anyway. Quite apart from
the above considerations I'm still a bit concerned about its performance
characteristics. And I'm not sure we really want all the baggage that
David is proposing to bring along with it. Is it an advance to force the
regex_replace "i" flag for such a type? I can imagine cases where I
might want it to sort insensitively, but be able to do case sensitive
regex ops on it. It's not as if the user can't supply the flag. So right
now I don't think citext should be included, because there are still
issues to sort out, if for no other reason.


cheers

andrew

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Andrew Sullivan-5
In reply to this post by David E. Wheeler
On Mon, Jul 21, 2008 at 01:17:39PM -0700, David E. Wheeler wrote:
>  pgFoundry ain't the CPAN, alas.

Maybe that's the problem that really needs solving?

One of the big Postgres features is its extensibility.  I agree that
the extensions can sometimes be hard to find, but surely the answer to
that is not an infinitely large source tarball?

A

--
Andrew Sullivan
[hidden email]
+1 503 667 4564 x104
http://www.commandprompt.com/

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

David E. Wheeler
In reply to this post by Andrew Dunstan
On Jul 21, 2008, at 13:19, Andrew Dunstan wrote:

> I was going to write some stuff about citext anyway. Quite apart  
> from the above considerations I'm still a bit concerned about its  
> performance characteristics. And I'm not sure we really want all the  
> baggage that David is proposing to bring along with it. Is it an  
> advance to force the regex_replace "i" flag for such a type? I can  
> imagine cases where I might want it to sort insensitively, but be  
> able to do case sensitive regex ops on it. It's not as if the user  
> can't supply the flag. So right now I don't think citext should be  
> included, because there are still issues to sort out, if for no  
> other reason.

I'm happy to work with folks to get them figured out, but at the end,  
there may be some differing opinions. If there's a reference  
implementation that could be checked (how does a case-insensitive  
collation work in another database?), that would be fine.

You can use the "c" flag to get case-sensitive comparison with the  
regex functions, though not with the operators. (Maybe this should be  
moved to a separate thread?)

Best,

David


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

David E. Wheeler
In reply to this post by Andrew Sullivan-5
On Jul 21, 2008, at 13:28, Andrew Sullivan wrote:

> Maybe that's the problem that really needs solving?
>
> One of the big Postgres features is its extensibility.  I agree that
> the extensions can sometimes be hard to find, but surely the answer to
> that is not an infinitely large source tarball?

Oh, of course. But one thing at a time. I'm in complete agreement that  
what goes into core should be pretty conservative, and that the module  
problem should be addressed. But even given that, I think that there  
is a strong case to be made that citext should be in contrib.

Best,

David

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Dave Cramer-8
In reply to this post by Andrew Sullivan-5

On 21-Jul-08, at 4:28 PM, Andrew Sullivan wrote:

> On Mon, Jul 21, 2008 at 01:17:39PM -0700, David E. Wheeler wrote:
>> pgFoundry ain't the CPAN, alas.
>
> Maybe that's the problem that really needs solving?
>
> One of the big Postgres features is its extensibility.  I agree that
> the extensions can sometimes be hard to find, but surely the answer to
> that is not an infinitely large source tarball?
>
> A
>
I'd have to agree with Andrew here. Making it easy to get extensions  
would solve lots of problems.

Dave

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Peter Eisentraut-2
In reply to this post by Tom Lane-2
Am Monday, 21. July 2008 schrieb Tom Lane:
> So my feeling is that we should not accept either of these patches.

My feeling as well.

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Gregory Stark-2
In reply to this post by Tom Lane-2

"Tom Lane" <[hidden email]> writes:

> From a project-management point of view, it's insanity to set a presumption
> that pgfoundry is just a proving ground for code that should eventually get
> into core once it's mature enough or popular enough or whatever. We *have
> to* encourage the development of a cloud of subprojects around the core, or
> core will eventually collapse of its own weight.

One option might be the Perl approach of having separately developed projects
which are snapshotted at stable points and included in the release. It has the
chance to offer the best of both worlds by offloading development outside of
core but provide users with a perceived "complete" system.

For perl this is important because they want programmers to be able to assume
certain modules are present. For postgres the case is less compelling since
there isn't an interoperability issue.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's Slony Replication support!

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Dave Page-7
On Tue, Jul 22, 2008 at 2:39 PM, Gregory Stark <[hidden email]> wrote:

>
> "Tom Lane" <[hidden email]> writes:
>
>> From a project-management point of view, it's insanity to set a presumption
>> that pgfoundry is just a proving ground for code that should eventually get
>> into core once it's mature enough or popular enough or whatever. We *have
>> to* encourage the development of a cloud of subprojects around the core, or
>> core will eventually collapse of its own weight.
>
> One option might be the Perl approach of having separately developed projects
> which are snapshotted at stable points and included in the release. It has the
> chance to offer the best of both worlds by offloading development outside of
> core but provide users with a perceived "complete" system.

Yeah, but then what happens when the offloaded development/maintenance
doesn't happen? We'd end up pulling the package or having to maintain
it ourselves anyway.

/D

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Gregory Stark-2
"Dave Page" <[hidden email]> writes:

> On Tue, Jul 22, 2008 at 2:39 PM, Gregory Stark <[hidden email]> wrote:
>>
>> "Tom Lane" <[hidden email]> writes:
>>
>>> From a project-management point of view, it's insanity to set a presumption
>>> that pgfoundry is just a proving ground for code that should eventually get
>>> into core once it's mature enough or popular enough or whatever. We *have
>>> to* encourage the development of a cloud of subprojects around the core, or
>>> core will eventually collapse of its own weight.
>>
>> One option might be the Perl approach of having separately developed projects
>> which are snapshotted at stable points and included in the release. It has the
>> chance to offer the best of both worlds by offloading development outside of
>> core but provide users with a perceived "complete" system.
>
> Yeah, but then what happens when the offloaded development/maintenance
> doesn't happen? We'd end up pulling the package or having to maintain
> it ourselves anyway.

Yeah, it's probably a plan which would work better once there's some solidly
maintained external projects for an extended period of time.

I suppose it's not entirely unlike the history of tsearch.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's RemoteDBA services!

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Shane Ambler-4
In reply to this post by Dave Cramer-8
Dave Cramer wrote:

>
> On 21-Jul-08, at 4:28 PM, Andrew Sullivan wrote:
>
>> On Mon, Jul 21, 2008 at 01:17:39PM -0700, David E. Wheeler wrote:
>>> pgFoundry ain't the CPAN, alas.
>>
>> Maybe that's the problem that really needs solving?
>>
>> One of the big Postgres features is its extensibility.  I agree
>> that the extensions can sometimes be hard to find, but surely the
>> answer to that is not an infinitely large source tarball?
>>
>>
> I'd have to agree with Andrew here. Making it easy to get extensions
> would solve lots of problems.

What about starting a secondary team that would review extensions?
Projects on pgfoundry could be identified as reviewed and approved as a
type of recommendation that they are of acceptable quality to use in
production - maybe against certain versions.

What I would see is current core developers teaching a new group of
developers to do the add-on code reviews to a point where they could
continue on by themselves - one or two from core may wish to stay in
this group - with core checking in from time to time to ensure the
quality doesn't slip. Thereby giving some confidence in the use of the
add-ons that get *certified*.

A new add-on would be presented to this group and maybe voted on in one
of the lists (General or Admin?) to get acceptance into the review process.

Anyone interested in starting this?



I do agree that the main code doesn't need to contain every feature that
is available. But we do need to improve the perception of add-ons.
Hardly anyone thinks twice about adding an extension to firefox, perl,
gimp or oscommerce or even drivers to the os, and we need to aim for a
similar thought here.

I do think that having a list of reviewed and approved add-ons that is
easily found on the main site along with release downloads will help
along those lines.

We need to promote that postgresql isn't a one-size-fits-all solution,
it is a solid product that can be customised to suite your needs.




--

Shane Ambler
pgSQL (at) Sheeky (dot) Biz

Get Sheeky @ http://Sheeky.Biz

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Simon Riggs
In reply to this post by Tom Lane-2

On Mon, 2008-07-21 at 15:43 -0400, Tom Lane wrote:
> From a maintenance point of view there seems little need
> for either project to get integrated: they don't appear to have much
> of any code that is tightly tied to backend innards.

This is a slightly circular argument. They have had to be written with
no linkage to core to allow them to be created outside of it.

I agree with your general principles on inclusion of features and also
agree that in this specific case the patches should be rejected. Growing
up outside of core cannot be a reason to exclude new capabilities from
core, but it is probably a reason to reject specific code.

In both these cases, I can see that the capability could be provided in
a different way and benefit from tighter integration.

I think we should return them with comments that if you integrate them
more with core *and* can justify having done so, then we might include
those features later.

--
 Simon Riggs           www.2ndQuadrant.com
 PostgreSQL Training, Services and Support


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

David E. Wheeler
On Jul 22, 2008, at 12:51, Simon Riggs wrote:

> I think we should return them with comments that if you integrate them
> more with core *and* can justify having done so, then we might include
> those features later

I believe I've done both these things for citext, though if there is  
more to be done, I'm glad to do it.

New patch coming later today, BTW.

Thanks,

David

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Josh berkus
In reply to this post by Simon Riggs
Tom, Simon, etc.:

Of the several things which "PostgreSQL could learn from MySQL" which we
covered at pgCon was that the requirement to hunt hither and yon for
popular add-ins is one of the primary reasons for developers not using
PostgreSQL.

Further, one of the main reasons why people do use PostgreSQL is our
advanced functionality.  If we focus only on core SQL features, there
are few reasons to use us over MySQL, Oracle express, SQL Server, or
Firebird.

Minimalism isn't its own reward.  Obviously Tom has reason to worry
about the overall maintenance effort for the PostgreSQL code.  But we
need to balance that against the need to add features that users want
and will keep our community growing.

If the way to do this is by packaging stuff together but maintaining
separate CVS trees, then ok -- but then we need a plan for how we're
going to do that, rather than just rejecting patches.

The general case aside, I really feel strongly that citext belongs in
core unless we come up with some other means to do case-insensitive
text. It's one of the top 10 newbie questions.

--Josh


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: Do we really want to migrate plproxy and citext into PG core distribution?

Tom Lane-2
In reply to this post by Simon Riggs
Simon Riggs <[hidden email]> writes:
> On Mon, 2008-07-21 at 15:43 -0400, Tom Lane wrote:
>> From a maintenance point of view there seems little need
>> for either project to get integrated: they don't appear to have much
>> of any code that is tightly tied to backend innards.

> This is a slightly circular argument. They have had to be written with
> no linkage to core to allow them to be created outside of it.

True, but in the form in which they are currently presented there is no
(technical) reason to integrate them: no new capability would be
provided thereby.  Contrast with, say, text search, which we integrated
mainly because we could provide easier configuration and a better
dump/restore experience than the contrib module provided.

> In both these cases, I can see that the capability could be provided in
> a different way and benefit from tighter integration.

Perhaps.  I think a lot of the dump/restore issue could be solved
generically if we had better "module" support ... but there's no need
to go over that turf again right now.

In the case of citext, I think an "integrated" solution would involve
some way of creating case-insensitive collations, which would certainly
be cool but it requires a whole lot of infrastructure we don't have yet.
And it wouldn't look even a little bit like the present citext, nor
be upward compatible with it.

In the case of plproxy, I think an integrated solution is pronounced
"SQL-MED", and likewise plproxy in its present form doesn't move us
toward that goal.

An important point here is that acceptance of a feature into core (or
even contrib) puts us on the hook to worry about upward compatibility
for it, maybe not forever but for a long time into the future.
I don't think I want to buy into that for either of these as presently
constituted --- they don't match up with what I think the long-term
goals ought to be in these areas.

                        regards, tom lane

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
12345