Palle Girgensohn's ICU patch

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Palle Girgensohn's ICU patch

Jakob Egger-2
When packaging PostgreSQL for Postgres.app, I discovered a problem: strcoll doesn't work for multibyte encodings on OS X. As a consequence, text sorting in PostgreSQL doesn't work. The only workaround seemed to be to use a legacy encoding like latin1, which is inacceptable.

I discovered that OS X shares this limitation with FreeBSD, and there exists a patch written by Palle Girgensohn that uses the ICU library for collating strings instead of the std-c strcoll function. You can find it at http://people.freebsd.org/~girgen/postgresql-icu/README.html

I applied the patch, and according to preliminary testing with 9.4rc1 it seems to work flawlessly on OS X as well.

I have two questions:

1) Does anybody else have experience with this patch? Is it safe to release PostgreSQL binaries with this patch applied to the public?

2) Is there a reason why this patch hasn't been merged into core over the years? Since it requires setting a configure switch (--with-icu) it shouldn't break anything?

Best regards,
Jakob Egger


Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Palle
Hi!

This is indeed a very well tested patch as we've run it in production for 8+ years on 20+ systems. 

It is not included upstreams mainly because I did ask for it to happen. I've been aiming to do it but haven't got around to it. Also, since 9.2 (?) there is support in PostgreSQL for setting collate locale per column. This is not yet supported by the patch, which makes it non-complete. You could argue that this is not as important as supporting the primary locale, but it would be hard to argue about that, it would have to be added for it to reach inclusion upstreams. 

So, I can vouch for it, it does the job just fine. Upstreams support will happen eventually. 

Palle



26 nov 2014 kl. 08yt:31 skrev Jakob Egger <[hidden email]>:

When packaging PostgreSQL for Postgres.app, I discovered a problem: strcoll doesn't work for multibyte encodings on OS X. As a consequence, text sorting in PostgreSQL doesn't work. The only workaround seemed to be to use a legacy encoding like latin1, which is inacceptable.

I discovered that OS X shares this limitation with FreeBSD, and there exists a patch written by Palle Girgensohn that uses the ICU library for collating strings instead of the std-c strcoll function. You can find it at http://people.freebsd.org/~girgen/postgresql-icu/README.html

I applied the patch, and according to preliminary testing with 9.4rc1 it seems to work flawlessly on OS X as well.

I have two questions:

1) Does anybody else have experience with this patch? Is it safe to release PostgreSQL binaries with this patch applied to the public?

2) Is there a reason why this patch hasn't been merged into core over the years? Since it requires setting a configure switch (--with-icu) it shouldn't break anything?

Best regards,
Jakob Egger


Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Jakob Egger-2
Hi Palle,

thanks for the extremely quick response!

In that case I will include the patch in Postgres.app. Missing support for per-column collations is preferable to missing support for the standard locale!

I'll have a look at the per-column collation support, it would be great if PostgreSQL on OS X would work out of the box at some point.

Best regards,
Jakob


Am 26.11.2014 um 08:41 schrieb Palle Girgensohn <[hidden email]>:

Hi!

This is indeed a very well tested patch as we've run it in production for 8+ years on 20+ systems. 

It is not included upstreams mainly because I did ask for it to happen. I've been aiming to do it but haven't got around to it. Also, since 9.2 (?) there is support in PostgreSQL for setting collate locale per column. This is not yet supported by the patch, which makes it non-complete. You could argue that this is not as important as supporting the primary locale, but it would be hard to argue about that, it would have to be added for it to reach inclusion upstreams. 

So, I can vouch for it, it does the job just fine. Upstreams support will happen eventually. 

Palle



26 nov 2014 kl. 08yt:31 skrev Jakob Egger <[hidden email]>:

When packaging PostgreSQL for Postgres.app, I discovered a problem: strcoll doesn't work for multibyte encodings on OS X. As a consequence, text sorting in PostgreSQL doesn't work. The only workaround seemed to be to use a legacy encoding like latin1, which is inacceptable.

I discovered that OS X shares this limitation with FreeBSD, and there exists a patch written by Palle Girgensohn that uses the ICU library for collating strings instead of the std-c strcoll function. You can find it at http://people.freebsd.org/~girgen/postgresql-icu/README.html

I applied the patch, and according to preliminary testing with 9.4rc1 it seems to work flawlessly on OS X as well.

I have two questions:

1) Does anybody else have experience with this patch? Is it safe to release PostgreSQL binaries with this patch applied to the public?

2) Is there a reason why this patch hasn't been merged into core over the years? Since it requires setting a configure switch (--with-icu) it shouldn't break anything?

Best regards,
Jakob Egger



Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Magnus Hagander-2
In reply to this post by Palle
On Wed, Nov 26, 2014 at 8:41 AM, Palle Girgensohn <[hidden email]> wrote:

> Hi!
>
> This is indeed a very well tested patch as we've run it in production for 8+
> years on 20+ systems.
>
> It is not included upstreams mainly because I did ask for it to happen. I've
> been aiming to do it but haven't got around to it. Also, since 9.2 (?) there
> is support in PostgreSQL for setting collate locale per column. This is not
> yet supported by the patch, which makes it non-complete. You could argue
> that this is not as important as supporting the primary locale, but it would
> be hard to argue about that, it would have to be added for it to reach
> inclusion upstreams.
>
> So, I can vouch for it, it does the job just fine. Upstreams support will
> happen eventually.


We did also discuss this back when we did the Windows port. One of the
big arguments against bringing it in then (because it worked) was that
we'd bring in another compile time dependency that's actually larger
than PostgreSQL itself. For example,the ICU .tgz file of the latest
version is 24.3Mb, and the latest postgresql .tgz is 21.8Mb. If we add
it as a requirement, we more than double the size of PostgreSQL. (Part
of that was specifically a concern on Windows of course, since no
dependencies can be expected to exist there - icu is a lot more likely
to already exist packaged up on linux/bsd)



--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Palle

> 26 nov 2014 kl. 09:58 skrev Magnus Hagander <[hidden email]>:
>
> On Wed, Nov 26, 2014 at 8:41 AM, Palle Girgensohn <[hidden email]> wrote:
>> Hi!
>>
>> This is indeed a very well tested patch as we've run it in production for 8+
>> years on 20+ systems.
>>
>> It is not included upstreams mainly because I did ask for it to happen. I've
>> been aiming to do it but haven't got around to it. Also, since 9.2 (?) there
>> is support in PostgreSQL for setting collate locale per column. This is not
>> yet supported by the patch, which makes it non-complete. You could argue
>> that this is not as important as supporting the primary locale, but it would
>> be hard to argue about that, it would have to be added for it to reach
>> inclusion upstreams.
>>
>> So, I can vouch for it, it does the job just fine. Upstreams support will
>> happen eventually.
>
>
> We did also discuss this back when we did the Windows port. One of the
> big arguments against bringing it in then (because it worked) was that
> we'd bring in another compile time dependency that's actually larger
> than PostgreSQL itself. For example,the ICU .tgz file of the latest
> version is 24.3Mb, and the latest postgresql .tgz is 21.8Mb. If we add
> it as a requirement, we more than double the size of PostgreSQL. (Part
> of that was specifically a concern on Windows of course, since no
> dependencies can be expected to exist there - icu is a lot more likely
> to already exist packaged up on linux/bsd)

For windows, that is very good argument. ICU is huge and takes forever to build. But as you say, it is a lot more likely to already be installed or at least packaged.

Also, you where, rightly, reluctant to use the ICU patch at that time because it required a memcopy (from utf-8 to ICUs internal utf-16) of every column it was to compare. This requirement is of course long gone, as ICU soon after fixed built in optimizations for utf-8, a very reasonable development step for the ICU platform... :-)

Jakob, including the patch in PostgreSQL.app seems pretty reasonable. There's is only a small fraction of ICU that is used, a couple of libraries I believe.

As I said, the missing feature will probably be fixed some time in the future, after which I will suggest the patch for inclusion. But it is not even near the top of my to-do list. :-/

Cheers,
Palle



--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Jakob Egger-2
One of the
big arguments against bringing it in then (because it worked) was that
we'd bring in another compile time dependency that's actually larger
than PostgreSQL itself.

Magnus: I don't see how this is a problem as long as using ICU is *optional*. On systems with a working strcoll there is no problem with using the stdc functions (except that ICU might offer more collations).


Jakob, including the patch in PostgreSQL.app seems pretty reasonable. There's is only a small fraction of ICU that is used, a couple of libraries I believe.

Palle: The ICU libraries themselves aren't that big, but the required data files (also packaged as a dynamic library) are big (around 25MB uncompressed). However, I'd rather increase the download size by 30% than ship a broken database. 


Best regards,
Jakob
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Palle

> 26 nov 2014 kl. 10:36 skrev Jakob Egger <[hidden email]>:
>
>> One of the
>> big arguments against bringing it in then (because it worked) was that
>> we'd bring in another compile time dependency that's actually larger
>> than PostgreSQL itself.
>
> Magnus: I don't see how this is a problem as long as using ICU is *optional*. On systems with a working strcoll there is no problem with using the stdc functions (except that ICU might offer more collations).
>

In windows, it was primarily about packaging, I believe. Mind you, this was many years ago... ;)

>
>> Jakob, including the patch in PostgreSQL.app seems pretty reasonable. There's is only a small fraction of ICU that is used, a couple of libraries I believe.
>
> Palle: The ICU libraries themselves aren't that big, but the required data files (also packaged as a dynamic library) are big (around 25MB uncompressed). However, I'd rather increase the download size by 30% than ship a broken database.


Bear in mind that this might alter the way indexes are built. From the top of my head, I just can't remember if this is true or not. I'm probably wrong? Magnus? You would have to try.

It does change the order by to properly handle utf-8 *AND* order by becomes case insensitve. I'm not sure this is correct SQL? I know that in Oracle, this is optional (NLS_COMP=LINGUISTIC and/or NLS_SORT=BINARY_CI), and SQL Server has something similar.




--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Jakob Egger-2

> Bear in mind that this might alter the way indexes are built. From the top of my head, I just can't remember if this is true or not. I'm probably wrong? Magnus? You would have to try.

That's why I want to include it in the first version of 9.4, when people need to dump & reload their database anyway (I'll make a note not to use pg_upgrade)

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Palle

> 26 nov 2014 kl. 10:48 skrev Jakob Egger <[hidden email]>:
>
>
>> Bear in mind that this might alter the way indexes are built. From the top of my head, I just can't remember if this is true or not. I'm probably wrong? Magnus? You would have to try.
>
> That's why I want to include it in the first version of 9.4, when people need to dump & reload their database anyway (I'll make a note not to use pg_upgrade)

Good point.



--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Dave Page-3
In reply to this post by Jakob Egger-2
On Wed, Nov 26, 2014 at 9:48 AM, Jakob Egger <[hidden email]> wrote:
>
>> Bear in mind that this might alter the way indexes are built. From the top of my head, I just can't remember if this is true or not. I'm probably wrong? Magnus? You would have to try.
>
> That's why I want to include it in the first version of 9.4, when people need to dump & reload their database anyway (I'll make a note not to use pg_upgrade)

You may want to bear in mind that postgres.app is on the main PG
downloads page on the website. If you're patching Postgres to add a
feature like this, it would become a fork and would have to be moved
out of the "PostgreSQL Core Distribution" section of the download area
as we only include "pure" distributions there.

--
Dave Page
PostgreSQL Core Team
http://www.postgresql.org/


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Jakob Egger-2
Am 26.11.2014 um 11:05 schrieb Dave Page <[hidden email]>:
You may want to bear in mind that postgres.app is on the main PG
downloads page on the website. If you're patching Postgres to add a
feature like this, it would become a fork and would have to be moved
out of the "PostgreSQL Core Distribution" section of the download area
as we only include "pure" distributions there.

I wasn't aware of this. I'll have to bring this up on the Postgres.app Github page.

Personally, I don't think that shipping a database with broken text sorting is acceptable; but I can't speak on behalf of the other contributors to Postgres.app without consulting them first.
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Dave Page-3
On Wed, Nov 26, 2014 at 10:13 AM, Jakob Egger <[hidden email]> wrote:

> Am 26.11.2014 um 11:05 schrieb Dave Page <[hidden email]>:
>
> You may want to bear in mind that postgres.app is on the main PG
> downloads page on the website. If you're patching Postgres to add a
> feature like this, it would become a fork and would have to be moved
> out of the "PostgreSQL Core Distribution" section of the download area
> as we only include "pure" distributions there.
>
>
> I wasn't aware of this. I'll have to bring this up on the Postgres.app
> Github page.
>
> Personally, I don't think that shipping a database with broken text sorting
> is acceptable; but I can't speak on behalf of the other contributors to
> Postgres.app without consulting them first.

Right - but the correct course of action would be to get the problem
fixed in PostgreSQL itself, not to fork the code which could lead to
other problems for users.

--
Dave Page
PostgreSQL Core Team
http://www.postgresql.org/


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Magnus Hagander-2
In reply to this post by Jakob Egger-2
On Wed, Nov 26, 2014 at 11:13 AM, Jakob Egger <[hidden email]> wrote:

> Am 26.11.2014 um 11:05 schrieb Dave Page <[hidden email]>:
>
> You may want to bear in mind that postgres.app is on the main PG
> downloads page on the website. If you're patching Postgres to add a
> feature like this, it would become a fork and would have to be moved
> out of the "PostgreSQL Core Distribution" section of the download area
> as we only include "pure" distributions there.
>
>
> I wasn't aware of this. I'll have to bring this up on the Postgres.app
> Github page.
>
> Personally, I don't think that shipping a database with broken text sorting
> is acceptable; but I can't speak on behalf of the other contributors to
> Postgres.app without consulting them first.

Is it broken *worse* in 9.4 than it was in previous versions?


--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Jakob Egger-2
In reply to this post by Dave Page-3

Am 26.11.2014 um 11:20 schrieb Dave Page <[hidden email]>:

On Wed, Nov 26, 2014 at 10:13 AM, Jakob Egger <[hidden email]> wrote:
Am 26.11.2014 um 11:05 schrieb Dave Page <[hidden email]>:

You may want to bear in mind that postgres.app is on the main PG
downloads page on the website. If you're patching Postgres to add a
feature like this, it would become a fork and would have to be moved
out of the "PostgreSQL Core Distribution" section of the download area
as we only include "pure" distributions there.


I wasn't aware of this. I'll have to bring this up on the Postgres.app
Github page.

Personally, I don't think that shipping a database with broken text sorting
is acceptable; but I can't speak on behalf of the other contributors to
Postgres.app without consulting them first.

Right - but the correct course of action would be to get the problem
fixed in PostgreSQL itself, not to fork the code which could lead to
other problems for users.

Agreed. Since this isn't a priority for Palle I'll have a look at the patch to see if I can extend it to make it suitable for submitting it, but since I have never contributed source to PostgreSQL I don't know yet if I can handle it.

I've opened an issue on Github to discuss what to do about Postgres.app and the upcoming 9.4 release:

Best regards,
Jakob
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Jakob Egger-2
In reply to this post by Magnus Hagander-2
> Is it broken *worse* in 9.4 than it was in previous versions?

No.

Because the indices need to be rebuilt, the only realistic opportunity for applying this patch to Postgres.app is when releasing a major new version, since then people need to migrate their data anyway. That's why I wanted to apply the patch when 9.4 is released.

I'm starting to see that maybe not all bugs can be fixed right now; I'm now waiting for input from the other contributors on Github.

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Pavel Stehule
In reply to this post by Magnus Hagander-2


2014-11-26 9:58 GMT+01:00 Magnus Hagander <[hidden email]>:
On Wed, Nov 26, 2014 at 8:41 AM, Palle Girgensohn <[hidden email]> wrote:
> Hi!
>
> This is indeed a very well tested patch as we've run it in production for 8+
> years on 20+ systems.
>
> It is not included upstreams mainly because I did ask for it to happen. I've
> been aiming to do it but haven't got around to it. Also, since 9.2 (?) there
> is support in PostgreSQL for setting collate locale per column. This is not
> yet supported by the patch, which makes it non-complete. You could argue
> that this is not as important as supporting the primary locale, but it would
> be hard to argue about that, it would have to be added for it to reach
> inclusion upstreams.
>
> So, I can vouch for it, it does the job just fine. Upstreams support will
> happen eventually.


We did also discuss this back when we did the Windows port. One of the
big arguments against bringing it in then (because it worked) was that
we'd bring in another compile time dependency that's actually larger
than PostgreSQL itself. For example,the ICU .tgz file of the latest
version is 24.3Mb, and the latest postgresql .tgz is 21.8Mb. If we add
it as a requirement, we more than double the size of PostgreSQL. (Part
of that was specifically a concern on Windows of course, since no
dependencies can be expected to exist there - icu is a lot more likely
to already exist packaged up on linux/bsd)

24MB is not problem for mostly Windows users. I don't propose ICU as main solution for us, but it can be good alternative for some companies, that should to fix inconsistency in collation implementation between Windows and Linux. Czech collation in Windows and Linux can produces different results in some corner cases.

Regards

Pavel

 



--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Palle
In reply to this post by Jakob Egger-2

> 26 nov 2014 kl. 11:44 skrev Jakob Egger <[hidden email]>:
>
>
>> Am 26.11.2014 um 11:20 schrieb Dave Page <[hidden email]>:
>>
>> On Wed, Nov 26, 2014 at 10:13 AM, Jakob Egger <[hidden email]> wrote:
>>> Am 26.11.2014 um 11:05 schrieb Dave Page <[hidden email]>:
>>>
>>> You may want to bear in mind that postgres.app is on the main PG
>>> downloads page on the website. If you're patching Postgres to add a
>>> feature like this, it would become a fork and would have to be moved
>>> out of the "PostgreSQL Core Distribution" section of the download area
>>> as we only include "pure" distributions there.
>>>
>>>
>>> I wasn't aware of this. I'll have to bring this up on the Postgres.app
>>> Github page.
>>>
>>> Personally, I don't think that shipping a database with broken text sorting
>>> is acceptable; but I can't speak on behalf of the other contributors to
>>> Postgres.app without consulting them first.
>>
>> Right - but the correct course of action would be to get the problem
>> fixed in PostgreSQL itself, not to fork the code which could lead to
>> other problems for users.
>
> Agreed. Since this isn't a priority for Palle I'll have a look at the patch to see if I can extend it to make it suitable for submitting it, but since I have never contributed source to PostgreSQL I don't know yet if I can handle it.

Well, this discussion actually pushes the priority quite a bit for me -- someone else actually beeing interested about the patch... I thought it was just me... :)=

Just for reference, the Linux collation is actaully also broken wrt to utf-8. It is better than others, but not correct. And lower()/upper() has many rather common cases where it is not working with "wide characters". For example, the towupper only looks at one character at the time, but proper handling needs to look at adjacent characters in some languages.

Either way, getting it into core would not happen before 9.5 anyway.

>
> I've opened an issue on Github to discuss what to do about Postgres.app and the upcoming 9.4 release:
> https://github.com/PostgresApp/PostgresApp/issues/233
>
> Best regards,
> Jakob



--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Greg Stark
I find it hard to believe the original premise of this thread. We knew
there were some problems with OSX and FreeBSD but surely they can't be
completely broken? What happens if you run "ls" with your locale set
to something like fr_FR.UTF8 ? Does Apple not sell Macs in countries
other than the US?

There were a number of problems with using ICU including the large
dependency and the limitations of the iterator model but the main
issue was that it's fundamentally a choice between being consistent
with every other application on your system and being consistent with
other Postgres databases running on other OSes. Most people run
multiple applications on one OS, not many databases on many OSes on
their own with no other applications. If Postgres used ICU then its
output would be inconsistent with things like "sort" or "ls" or your
application programming language's comparison operators.


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Neil Tiffin-3

On Nov 26, 2014, at 8:21 AM, Greg Stark <[hidden email]> wrote:

> I find it hard to believe the original premise of this thread. We knew
> there were some problems with OSX and FreeBSD but surely they can't be
> completely broken?

Ever tried to use Spotlight for searching (English) on the Mac, not completely broken, just not reliable.  This does not surprise me in the least for OSX.  The Mac has, in recent history, become a “looks good", but the details may or may not be really correct platform.

I thought FreeBSD was a preferred OS for PostgreSQL?  This does surprise me.

> What happens if you run "ls" with your locale set
> to something like fr_FR.UTF8 ? Does Apple not sell Macs in countries
> other than the US?

Neil
Daily Mac user for a long time.

--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [pgsql-packagers] Palle Girgensohn's ICU patch

Palle
In reply to this post by Greg Stark

> 26 nov 2014 kl. 15:21 skrev Greg Stark <[hidden email]>:
>
> I find it hard to believe the original premise of this thread. We knew
> there were some problems with OSX and FreeBSD but surely they can't be
> completely broken? What happens if you run "ls" with your locale set
> to something like fr_FR.UTF8 ? Does Apple not sell Macs in countries
> other than the US?

Hi,

On Mac OS X, ls -l is completely broken wrt utf-8 collation. Really. Horribly broken. The sorting it produces for the Swedish locale is just nonexisting, completely unaccetable, unusable. Compare it to sorting Z just after S or something, just to get the scale of how bad it is.

Application languages like Java have their own sorting. C based stuff like perl have their own way to do it. python, well depends on the version, haven't checked. C applications, well, it depends on if they use ICU or not, I guess. :)

Apples sells computers, but does not really promote using locales in Terminal.app... :)=

>
> There were a number of problems with using ICU including the large
> dependency and the limitations of the iterator model but the main
> issue was that it's fundamentally a choice between being consistent
> with every other application on your system and being consistent with
> other Postgres databases running on other OSes. Most people run
> multiple applications on one OS, not many databases on many OSes on
> their own with no other applications. If Postgres used ICU then its
> output would be inconsistent with things like "sort" or "ls" or your
> application programming language's comparison operators.

I think most people don't care about getting postgresql collation consistent with sort or ls, they just want it to work properly for real life applications, so users who really don't care about ls or sort get the result they expect. Or, they give up and sort it in the application instead (=fail). But I guess that depends on which applications you use. We've used the patch for 8+ years. For us, Linux built-in collation would not have been enough either -- if memory serves it fails to sort 'ß' together with 'ss', and also fails to upper('ß') => 'SS', which would be expected in the real world.




--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
12