Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Peter Geoghegan-4
I've realized that my patch to make nbtree keys unique by treating
heap TID as a tie-breaker attribute must use ASC ordering, for reasons
that I won't go into here. Now that I'm not using DESC ordering, there
are changes to a small number of DROP...CASCADE messages that leave
users with something much less useful than what they'll see today --
see attached patch for full details. Some of these problematic cases
involve partitioning:

"""
 create table trigpart (a int, b int) partition by range (a);
 create table trigpart1 partition of trigpart for values from (0) to (1000);
 create trigger trg1 after insert on trigpart for each row execute
procedure trigger_nothing();
 ...
 drop trigger trg1 on trigpart1; -- fail
-ERROR:  cannot drop trigger trg1 on table trigpart1 because trigger
trg1 on table trigpart requires it
-HINT:  You can drop trigger trg1 on table trigpart instead.
+ERROR:  cannot drop trigger trg1 on table trigpart1 because table
trigpart1 requires it
+HINT:  You can drop table trigpart1 instead.
"""

As you can see, the original hint suggests "you need to drop the
object on the partition parent instead of its child", which is useful.
The new hint suggests "instead of dropping the trigger on the
partition child, maybe drop the child itself!", which is less than
useless. This is a problem that needs to be treated as a prerequisite
to committing the nbtree patch, so I'd like to get it out of the way
soon.

The high level issue is that findDependentObjects() relies on the scan
order of duplicates within the
DependDependerIndexId/pg_depend_depender_index index in a way that
nbtree doesn't actually guarantee, and never has guaranteed. As I've
shown, findDependentObjects()'s assumptions around where nbtree will
leave duplicates accidentally affects the quality of various
diagnostic messages. My example also breaks with
ignore_system_indexes=on, even on the master branch, so technically
this isn't a new problem.

I've looked into a way to fix findDependentObjects(). As far as I can
tell, I can fix issues by adding a kludgey special case along these
lines:

  1 diff --git a/src/backend/catalog/dependency.c
b/src/backend/catalog/dependency.c
  2 index 7dfa3278a5..7454d4e6f8 100644
  3 --- a/src/backend/catalog/dependency.c
  4 +++ b/src/backend/catalog/dependency.c
  5 @@ -605,6 +605,15 @@ findDependentObjects(const ObjectAddress *object,
  6                          ReleaseDeletionLock(object);
  7                          return;
  8                      }
  9 +                    /*
 10 +                     * Assume that another pg_depend entry more suitably
 11 +                     * represents dependency when an entry for a partition
 12 +                     * child's index references a column of the partition
 13 +                     * itself.
 14 +                     */
 15 +                    if (foundDep->deptype == DEPENDENCY_INTERNAL_AUTO &&
 16 +                        otherObject.objectSubId != 0)
 17 +                        break;

This is obviously brittle, but maybe it hints at a better approach.
Notably, it doesn't fix other similar issues, such as this:

--- a/contrib/earthdistance/expected/earthdistance.out
+++ b/contrib/earthdistance/expected/earthdistance.out
@@ -972,7 +972,7 @@ SELECT abs(cube_distance(ll_to_earth(-30,-90),
'(0)'::cube) / earth() - 1) <

 drop extension cube;  -- fail, earthdistance requires it
 ERROR:  cannot drop extension cube because other objects depend on it
-DETAIL:  extension earthdistance depends on extension cube
+DETAIL:  extension earthdistance depends on function cube_out(cube)

Can anyone think of a workable, scalable approach to fixing the
processing order of this findDependentObjects() pg_depend scan so that
we reliably get the user-visible behavior we already tacitly expect?

--
Peter Geoghegan

0008-Questionable-tentative-ASC-regress-output-fixes.patch (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Andrey Lepikhov
In my opinion, your patch detected three problems:
1. Unsteady order of query results/system messages ('DROP...CASCADE'
detects it).
2. Hide info about a child object dropping ('drop cascades to 62
other objects' detects it).
3. Possible non-informative messages about dependencies ('drop trigger
trg1' detects it)

Problem No. 1 will be amplified with new asynchronous operations,
background workers and distributing query execution. It is not problem
of DBMS. The solution is change the tests: include sorting of query
results, sorting of system messages before diff operation.
If steady order of messages is critical for users we can sort
targetObjects array in the begin of reportDependentObjects() routine by
classId, objectId and objectSubId fields.

Problem No. 2: we can suppress some output optimizations in
object_address_present_add_flags() routine and print all deleted objects.

Problem No. 3: I suppose we can go one of two ways:
a) print all depended objects returned by scan of DependDependerIndexId
relation, not only the first.
b) search a root of dependence and print only it.

On 06.11.2018 5:04, Peter Geoghegan wrote:

> I've realized that my patch to make nbtree keys unique by treating
> heap TID as a tie-breaker attribute must use ASC ordering, for reasons
> that I won't go into here. Now that I'm not using DESC ordering, there
> are changes to a small number of DROP...CASCADE messages that leave
> users with something much less useful than what they'll see today --
> see attached patch for full details. Some of these problematic cases
> involve partitioning:
>
> """
>   create table trigpart (a int, b int) partition by range (a);
>   create table trigpart1 partition of trigpart for values from (0) to (1000);
>   create trigger trg1 after insert on trigpart for each row execute
> procedure trigger_nothing();
>   ...
>   drop trigger trg1 on trigpart1; -- fail
> -ERROR:  cannot drop trigger trg1 on table trigpart1 because trigger
> trg1 on table trigpart requires it
> -HINT:  You can drop trigger trg1 on table trigpart instead.
> +ERROR:  cannot drop trigger trg1 on table trigpart1 because table
> trigpart1 requires it
> +HINT:  You can drop table trigpart1 instead.
> """
>
> As you can see, the original hint suggests "you need to drop the
> object on the partition parent instead of its child", which is useful.
> The new hint suggests "instead of dropping the trigger on the
> partition child, maybe drop the child itself!", which is less than
> useless. This is a problem that needs to be treated as a prerequisite
> to committing the nbtree patch, so I'd like to get it out of the way
> soon.
>
> The high level issue is that findDependentObjects() relies on the scan
> order of duplicates within the
> DependDependerIndexId/pg_depend_depender_index index in a way that
> nbtree doesn't actually guarantee, and never has guaranteed. As I've
> shown, findDependentObjects()'s assumptions around where nbtree will
> leave duplicates accidentally affects the quality of various
> diagnostic messages. My example also breaks with
> ignore_system_indexes=on, even on the master branch, so technically
> this isn't a new problem.
>
> I've looked into a way to fix findDependentObjects(). As far as I can
> tell, I can fix issues by adding a kludgey special case along these
> lines:
>
>    1 diff --git a/src/backend/catalog/dependency.c
> b/src/backend/catalog/dependency.c
>    2 index 7dfa3278a5..7454d4e6f8 100644
>    3 --- a/src/backend/catalog/dependency.c
>    4 +++ b/src/backend/catalog/dependency.c
>    5 @@ -605,6 +605,15 @@ findDependentObjects(const ObjectAddress *object,
>    6                          ReleaseDeletionLock(object);
>    7                          return;
>    8                      }
>    9 +                    /*
>   10 +                     * Assume that another pg_depend entry more suitably
>   11 +                     * represents dependency when an entry for a partition
>   12 +                     * child's index references a column of the partition
>   13 +                     * itself.
>   14 +                     */
>   15 +                    if (foundDep->deptype == DEPENDENCY_INTERNAL_AUTO &&
>   16 +                        otherObject.objectSubId != 0)
>   17 +                        break;
>
> This is obviously brittle, but maybe it hints at a better approach.
> Notably, it doesn't fix other similar issues, such as this:
>
> --- a/contrib/earthdistance/expected/earthdistance.out
> +++ b/contrib/earthdistance/expected/earthdistance.out
> @@ -972,7 +972,7 @@ SELECT abs(cube_distance(ll_to_earth(-30,-90),
> '(0)'::cube) / earth() - 1) <
>
>   drop extension cube;  -- fail, earthdistance requires it
>   ERROR:  cannot drop extension cube because other objects depend on it
> -DETAIL:  extension earthdistance depends on extension cube
> +DETAIL:  extension earthdistance depends on function cube_out(cube)
>
> Can anyone think of a workable, scalable approach to fixing the
> processing order of this findDependentObjects() pg_depend scan so that
> we reliably get the user-visible behavior we already tacitly expect?
>
> --
> Peter Geoghegan
>

--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company

Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Peter Geoghegan-4
On Mon, Nov 5, 2018 at 7:46 PM Andrey Lepikhov
<[hidden email]> wrote:
> Problem No. 1 will be amplified with new asynchronous operations,
> background workers and distributing query execution. It is not problem
> of DBMS. The solution is change the tests: include sorting of query
> results, sorting of system messages before diff operation.
> If steady order of messages is critical for users we can sort
> targetObjects array in the begin of reportDependentObjects() routine by
> classId, objectId and objectSubId fields.

Yeah, maybe. I'm not that worried about the order of objects; we can
probably continue to get away with suppressing the list of objects by
placing a "\set VERBOSITY terse" where needed -- that's something
we've already been doing for some time [1]. I accept that it would be
better to sort the output, but I'm concerned that that would be a
difficult, risky project. What if there was a huge number of dependent
objects? What if a memory allocation fails?

> Problem No. 2: we can suppress some output optimizations in
> object_address_present_add_flags() routine and print all deleted objects.

If there is anything that makes it necessary to sort, it's this -- the
order of visitation can affect whether or not
object_address_present_add_flags() suppresses redundant entries. But I
still prefer to fix the problem by changing the scan order to be what
we actually want it to be.

> Problem No. 3: I suppose we can go one of two ways:
> a) print all depended objects returned by scan of DependDependerIndexId
> relation, not only the first.
> b) search a root of dependence and print only it.

A solution does occur to me that I'm kind of embarrassed to suggest,
but that would nonetheless probably do the job:

The general problem here is that the scan order of a scan that uses
pg_depend_depender_index doesn't reliably give us the order we
actually want among duplicate index entries (at least, you could
choose to characterize it in that narrow way). The index
pg_depend_depender_index looks like this:

  Column  │  Type   │ Key? │ Definition
──────────┼─────────┼──────┼────────────
 classid  │ oid     │ yes  │ classid
 objid    │ oid     │ yes  │ objid
 objsubid │ integer │ yes  │ objsubid
btree, for table "pg_catalog.pg_depend"

Note that this isn't a unique index -- there are no unique indexes on
pg_depend at all, in fact. (Note also that the same observations apply
to pg_shdepend.)

The objsubid (not refobjsubid) is 0 in all cases that we have problems
with -- we're never going to have a problem with multiple
dependent-object-is-column entries that have the same '(classid,
objid, objsubid)' value, as far as I can tell:

pg@regression[2201]=# SELECT count(*), classid, objid, objsubid FROM
pg_depend WHERE objsubid != 0 GROUP BY classid, objid, objsubid HAVING
count(*) > 1;
 count │ classid │ objid │ objsubid
───────┼─────────┼───────┼──────────
(0 rows)

(i.e. We're only having problems with some of the entries/values that
you'll see when the above query is changed to have "bjsubid = 0" in
its WHERE clause.)

Why not vary the objsubid value among entries that don't use it
anyway, so that they have a negative integer objsubid values rather
than 0? That would have the effect of forcing the scan order of
pg_depend_depender_index scans to be whatever we want it to be.
dependent-object-is-column entries would not have their sort order
affected anyway, because in practice there is only one entry with the
same  '(classid, objid, objsubid)' value when objsubid > 0.

We could invent some scheme that made pg_depend/pg_shdepend entries
use fixed objsubid negative integer codes according to the
*referenced* object type (refobjid), and/or the entry's deptype.
Places that look at ObjectAddress.objectSubId would look for entries
that were > 0 rather than != 0, so we wouldn't really be changing the
meaning of objsubid (though it would need to be documented in the user
docs, and have a release note compatibility entry).
findDependentObjects() would still use "nkeys = 2" for these entries;
it would be one of the places that would be taught to check
objectSubId > 0 rather than objectSubId != 0. But it would get the
scan order it actually needs.

I haven't attempted to put this together just yet, because I want to
see if it passes the laugh test. My sense is that the best way to fix
the problem is to force the scan order to be the one that we
accidentally depend on using some mechanism or other. Though not
necessarily the one I've sketched out.

[1] https://postgr.es/m/24279.1525401104@...
--
Peter Geoghegan

Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Peter Geoghegan-4
On Tue, Nov 13, 2018 at 1:29 PM Peter Geoghegan <[hidden email]> wrote:
> A solution does occur to me that I'm kind of embarrassed to suggest,
> but that would nonetheless probably do the job:

> Why not vary the objsubid value among entries that don't use it
> anyway, so that they have a negative integer objsubid values rather
> than 0? That would have the effect of forcing the scan order of
> pg_depend_depender_index scans to be whatever we want it to be.

I tried to implement this today, and it almost worked. However, what I
came up with was even more brittle than I thought it would be, because
knowledge of objsubid is all over the place. It's even in pg_dump.

Then I had a better idea: add a monotonically decreasing column to
pg_depend, and make it the last attribute in both
pg_depend_depender_index and pg_depend_reference_index (it has to be
both indexes). This makes almost all the regression test output go
back to the master branch output, since index-wise duplicates are now
accessed in a well defined order: reverse chronological order (the
reverse of the order of the creation of a pg_depend entry). There is a
tiny amount of stuff that still has a different order in the
regression test output, but the changes are trivial, and obviously
still correct.

Is this approach really so bad? I might want to find a way to make it
a monotonically increasing DESC column (there is a question around bki
grammar support for that), but that's just a detail. Some scheme
involving a new column seems like it will work reasonably well. It's
much more maintainable than anything else I'm likely to come up with.

The indexes are actually smaller with the change following a run of
the regression tests, provided the entire patch series is applied. So
even if the only thing you care about is system catalog size, you
still win (the table is slightly bigger, though). This approach will
fix most or all of the test flappiness that we have been papering over
with "\set VERBOSITY terse" before now. I can't promise that just
making this one pg_depend change will reliably make that class of
problem go away forever, since you could still see something like that
elsewhere, but it looks like the vast majority of problem cases
involve pg_depend. So we might just end up killing the "\set VERBOSITY
terse" hack completely this way.

We're already relying on the scan order being in reverse chronological
order, so we might as well formalize the dependency. I don't think
that it's possible to sort the pg_depend entries as a way of fixing
the breakage while avoiding storing this extra information -- what is
there to sort on that's there already? You'd have to infer a whole
bunch of things about the object types associated with pg_depend
entries to do that, and teach dependency.c about its callers. That
seems pretty brittle to me.

--
Peter Geoghegan

Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Robert Haas
On Wed, Nov 14, 2018 at 1:28 AM Peter Geoghegan <[hidden email]> wrote:
> We're already relying on the scan order being in reverse chronological
> order, so we might as well formalize the dependency. I don't think
> that it's possible to sort the pg_depend entries as a way of fixing
> the breakage while avoiding storing this extra information -- what is
> there to sort on that's there already? You'd have to infer a whole
> bunch of things about the object types associated with pg_depend
> entries to do that, and teach dependency.c about its callers. That
> seems pretty brittle to me.

I think that the solution that you are proposing sorta sucks, but it's
better than hacking objsubid to do it, which did in fact pass the
laugh test, in that I laughed when I read it.  :-)

In a perfect world, it seems to me that what we ought to do is have
some real logic in the server that figures out which of the various
things we could report would be most likely to be useful to the user
... but that's probably a non-trivial project involving a fair amount
of human judgement.  Reasonable people may differ about what is best,
never mind unreasonable people.  I am inclined to think that your
proposal here is good enough for now, and somebody who dislikes it
(surely such a person will exist) can decide to look for ways to make
it better.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Peter Geoghegan-4
On Thu, Nov 15, 2018 at 10:52 AM Robert Haas <[hidden email]> wrote:
> I think that the solution that you are proposing sorta sucks, but it's
> better than hacking objsubid to do it, which did in fact pass the
> laugh test, in that I laughed when I read it.  :-)

Probably a good idea to get another cup of coffee if I'm
pre-apologizing for my ideas.

> In a perfect world, it seems to me that what we ought to do is have
> some real logic in the server that figures out which of the various
> things we could report would be most likely to be useful to the user
> ... but that's probably a non-trivial project involving a fair amount
> of human judgement.  Reasonable people may differ about what is best,
> never mind unreasonable people.  I am inclined to think that your
> proposal here is good enough for now, and somebody who dislikes it
> (surely such a person will exist) can decide to look for ways to make
> it better.

Great. Actually, the on-disk size of the pg_depend heap relation is
*unchanged* in the attached WIP patch, since it fits in a hole
previously lost to alignment. And, as I said, the indexes end up
smaller with suffix truncation. Even if the only thing you care about
is the on-disk size of system catalogs, you'll still pretty reliably
come out ahead. The design here is squirrelly, but we're already
relying on scan order to reach objsubid = 0 entries first.

There is a single tiny behavioral change to the regression test output
with this patch applied. I think that that's just because there is one
place where this dependency management stuff interacts with pages full
of duplicates, and therefore stops putting duplicates in pg_depend
indexes in exactly DESC TID order. My other patches add a couple of
more tiny changes along similar lines, since of course I'm only doing
this with the pg_depend indexes, and not for every system catalog
index.

--
Peter Geoghegan

0001-Add-pg_depend-index-scan-tiebreaker-column.patch (20K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Andrey Lepikhov
In reply to this post by Peter Geoghegan-4
On 14.11.2018 11:28, Peter Geoghegan wrote:
> We're already relying on the scan order being in reverse chronological
> order, so we might as well formalize the dependency. I don't think
> that it's possible to sort the pg_depend entries as a way of fixing
> the breakage while avoiding storing this extra information -- what is
> there to sort on that's there already? You'd have to infer a whole
> bunch of things about the object types associated with pg_depend
> entries to do that, and teach dependency.c about its callers. That
> seems pretty brittle to me.

This solution changes pg_depend relation for solve a problem, which
exists only in regression tests.  Very rarely it can be in the
partitioning cases. Or is it not?
I think this decision is some excessive.
May be you consider another approach:
1. Order of dependencies in 'DROP ... CASCADE' case is a problem of test
tools, not DBMS. And here we can use 'verbose terse'.
2. Print all dependencies in findDependentObjects() on a drop error (see
attachment as a prototype).

--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company

0001-Print-All-dependencies-on-error.patch (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Peter Geoghegan-4
On Wed, Dec 5, 2018 at 10:35 PM Andrey Lepikhov
<[hidden email]> wrote:
> This solution changes pg_depend relation for solve a problem, which
> exists only in regression tests.  Very rarely it can be in the
> partitioning cases. Or is it not?

I don't think it's a matter of how rarely this will happen. We're
trying to avoid these diagnostic message changes because they are
wrong. I don't think that there is much ambiguity about that.  Still,
it will happen however often the user drops objects belonging to
partition children, which could be quite often.

> I think this decision is some excessive.
> May be you consider another approach:
> 1. Order of dependencies in 'DROP ... CASCADE' case is a problem of test
> tools, not DBMS. And here we can use 'verbose terse'.
> 2. Print all dependencies in findDependentObjects() on a drop error (see
> attachment as a prototype).

You didn't include changes to the regression test output, which seems
like a big oversight, given that this has a lot to do with diagnostic
messages that are represented in the regression tests. Anyway, I don't
think it's acceptable to change the messages like this. It makes them
much less useful.

These stability issue keeps coming up, which makes a comprehensive
approach seem attractive to me. At least 95% of the test instability
comes from pg_depend.
--
Peter Geoghegan

Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Andrey Lepikhov


On 06.12.2018 11:52, Peter Geoghegan wrote:

> On Wed, Dec 5, 2018 at 10:35 PM Andrey Lepikhov
> <[hidden email]> wrote:
>> This solution changes pg_depend relation for solve a problem, which
>> exists only in regression tests.  Very rarely it can be in the
>> partitioning cases. Or is it not?
>
> I don't think it's a matter of how rarely this will happen. We're
> trying to avoid these diagnostic message changes because they are
> wrong. I don't think that there is much ambiguity about that.  Still,
> it will happen however often the user drops objects belonging to
> partition children, which could be quite often.

I want to say that we need to localize the rules for the order of the
diagnostic messages as much as possible in dependency.c.

>
>> I think this decision is some excessive.
>> May be you consider another approach:
>> 1. Order of dependencies in 'DROP ... CASCADE' case is a problem of test
>> tools, not DBMS. And here we can use 'verbose terse'.
>> 2. Print all dependencies in findDependentObjects() on a drop error (see
>> attachment as a prototype).
>
> You didn't include changes to the regression test output, which seems
> like a big oversight...

It was done knowingly to show the differences in messages that
introduces this approach.

> messages that are represented in the regression tests. Anyway, I don't
> think it's acceptable to change the messages like this. It makes them
> much less useful.

May you clarify this? If I understand correctly, your solution is that
user receives a report about the last inserted internal dependency on
the object. Why the full info about internal dependencies of the object
much less useful?
>
> These stability issue keeps coming up, which makes a comprehensive
> approach seem attractive to me. At least 95% of the test instability
> comes from pg_depend.
>

During the retail index tuple deletion project (heap cleaner subsystem)
we have non-fixed order of tuples at database relations. This caused to
unsteady order of text strings in some check-world test results.
Thus, I realized that the order of messages in the test results is
mostly a game of chance. For this reason I think it is necessary to find
more general solution of the messages ordering problem.

--
Andrey Lepikhov
Postgres Professional
https://postgrespro.com
The Russian Postgres Company

Reply | Threaded
Open this post in threaded view
|

Re: Fixing findDependentObjects()'s dependency on scan order (regressions in DROP diagnostic messages)

Peter Geoghegan-4
On Thu, Dec 6, 2018 at 8:52 PM Andrey Lepikhov
<[hidden email]> wrote:
> I want to say that we need to localize the rules for the order of the
> diagnostic messages as much as possible in dependency.c.

But the issue *isn't* confined to dependency.c, anyway. It bleeds into
a couple of other modules, like extension.c and tablecmds.c.

> > messages that are represented in the regression tests. Anyway, I don't
> > think it's acceptable to change the messages like this. It makes them
> > much less useful.
>
> May you clarify this? If I understand correctly, your solution is that
> user receives a report about the last inserted internal dependency on
> the object. Why the full info about internal dependencies of the object
> much less useful?

I don't know why for sure -- I just know that it is. It must have
evolved that way, as software often does. It's not surprising that the
most recently entered dependency is usually the most marginal among
dependencies of equal precedence in the real world, and therefore the
best suggestion. I've shown this with two examples.

To return to one of the examples: are you arguing that there is no
practical difference between "you need to drop the object on the
partition parent instead of its child" and "instead of dropping the
trigger on the partition child, maybe drop the child itself"? I think
it's *extremely* obvious that there is a big practical difference to
users in that particular case. Now, I agree that there are many other
examples where it doesn't really matter to users, but I don't think
that that's very relevant. Yes, these are the exceptions, but the
exceptions are often the interesting cases.

> During the retail index tuple deletion project (heap cleaner subsystem)
> we have non-fixed order of tuples at database relations. This caused to
> unsteady order of text strings in some check-world test results.
> Thus, I realized that the order of messages in the test results is
> mostly a game of chance. For this reason I think it is necessary to find
> more general solution of the messages ordering problem.

 I have no idea what you mean here. I'm proposing a patch that stops
it being a game of chance, while preserving the existing
slightly-random behavior to the greatest extent possible. I think that
my patch would have stopped that problem altogether. Are you
suggesting that it wouldn't have?

--
Peter Geoghegan