BUG #15929: logical decoding can not write down the analyse result when the output file is touched.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

BUG #15929: logical decoding can not write down the analyse result when the output file is touched.

PG Bug reporting form
The following bug has been logged on the website:

Bug reference:      15929
Logged by:          movead li
Email address:      [hidden email]
PostgreSQL version: 12beta1
Operating system:   centos7
Description:        

I was doing some testing of logical decoding and ran into a scenario that
may be a bug. The following steps will lead to reproducing the behaviour :


1.Start a logical decoding use below command:

pg_recvlogical -d postgres --slot=test2 --create-slot
pg_recvlogical -d postgres --slot=test2 --start -f ld.log

2. When I do a DML on the database it writes down the info into ld.log
file,
so far so good.

3. Next I open the log file. “Id.log” with vi and make some arbitrary
changes
in the log file.

4. If I run a DML command now the DML info will not appear in the ld.log
file.
Refer to step 2, when I did a DML it was appearing in the log file.

5. If I do some manipulation with the log file using the OS commands below :


mv ld.log ld.log1;
cp ld.log1 ld.log;

Then the DML info will start appearing in file ld.log1.

This seems like a bug to be because the user might accidentally some data
without noticing.

I have looked at the code to debug this behaviour and find a solution

In file ‘pg_recvlogical.c’ the ‘StreamLogicalLog()’ function seem to have
a
issue with handling of DML info. It uses open() to open the file and it does
not
check the file state when writing to the file after a long wait.

Reply | Threaded
Open this post in threaded view
|

Re: BUG #15929: logical decoding can not write down the analyse result when the output file is touched.

movead li
hello

There be a problem about pg_recvlogical tool as I described last mail,
and I have made a patch to solve the problem.

It stores the 'st_ino' while opening the output file, and check it when the
process is waked up. It will try to reopen the output file if the output file
stat has changed or disappeared.

What do you think about?

---
Highgo Software (Canada/China/Pakistan) 
URL : www.highgo.ca 
EMAIL: mailto:movead(dot)li(at)highgo(dot)ca

pg_recvlogical_problem_20190925.patch (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: BUG #15929: logical decoding can not write down the analyse result when the output file is touched.

Andres Freund
In reply to this post by PG Bug reporting form
Hi,

On 2019-07-27 09:39:47 +0000, PG Bug reporting form wrote:

> I was doing some testing of logical decoding and ran into a scenario that
> may be a bug. The following steps will lead to reproducing the behaviour :
>
>
> 1.Start a logical decoding use below command:
>
> pg_recvlogical -d postgres --slot=test2 --create-slot
> pg_recvlogical -d postgres --slot=test2 --start -f ld.log
>
> 2. When I do a DML on the database it writes down the info into ld.log
> file,
> so far so good.
>
> 3. Next I open the log file. “Id.log” with vi and make some arbitrary
> changes
> in the log file.
>
> 4. If I run a DML command now the DML info will not appear in the ld.log
> file.
> Refer to step 2, when I did a DML it was appearing in the log file.
>
> 5. If I do some manipulation with the log file using the OS commands below :
>
>
> mv ld.log ld.log1;
> cp ld.log1 ld.log;
>
> Then the DML info will start appearing in file ld.log1.
>
> This seems like a bug to be because the user might accidentally some data
> without noticing.
>
> I have looked at the code to debug this behaviour and find a solution
>
> In file ‘pg_recvlogical.c’ the ‘StreamLogicalLog()’ function seem to have
> a
> issue with handling of DML info. It uses open() to open the file and it does
> not
> check the file state when writing to the file after a long wait.

I don't see a bug here. It's completely normal for a tool that writes to
a file to continue writing into the file it started to write to. The
reason that vi breaks that is presumably just that it renames the
"edited" version into the place of the file that pg_recvlogical is still
writing to.

Greetings,

Andres Freund


Reply | Threaded
Open this post in threaded view
|

Re: Re: BUG #15929: logical decoding can not write down the analyse result when the output file is touched.

movead li

 
>I don't see a bug here. It's completely normal for a tool that writes to
>a file to continue writing into the file it started to write to. The
>reason that vi breaks that is presumably just that it renames the
>"edited" version into the place of the file that pg_recvlogical is still
>writing to.

It is the issue that the target file is changed but the 'pg_recvlogical'
process doesn't know it at all. So it streams all wal changes but writes 
down nothing at last.

So I think we can recreate or reopen the changed file when there be
something wrong with the target file. And we can choose to kill the 
'pg_recvlogical' process because it's no use at all now, or we should
give a warning to users at lease.

---
Highgo Software (Canada/China/Pakistan) 
URL : www.highgo.ca 
EMAIL: mailto:movead(dot)li(at)highgo(dot)ca