pgbackrest creating new directories (messing up cron jobs).

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

pgbackrest creating new directories (messing up cron jobs).

mimble9
Hello,

My pgbackrest creates the files in
/var/lib/pgbackrest/archive/main/10/0000000100000000

Each file looks something like:
000000010000000100000022-033bbe600913c068e89c3d063afb1f0527bf513d.gz.

However, when the last two digits of the section before the hyphen become
FF, pgbackrest creates a new directory:
/var/lib/pgbackrest/archive/main/10/0000000100000001.

This is annoying as I have a cron job set up to focus on files in the
0000000100000000 directory.

Is there a way to force bgbackrest to only use one directory (e.g.
0000000100000000)?

Thanks.



Reply | Threaded
Open this post in threaded view
|

Re: pgbackrest creating new directories (messing up cron jobs).

Stephen Frost
Greetings,

* [hidden email] ([hidden email]) wrote:
> My pgbackrest creates the files in
> /var/lib/pgbackrest/archive/main/10/0000000100000000

Yes, those are files which are created in the repo.

> Each file looks something like:
> 000000010000000100000022-033bbe600913c068e89c3d063afb1f0527bf513d.gz.

That's currently the naming, but we anticipate changing it in the future
and so I strongly recommend that you do not make assumptions or depend
on that exact naming.

> However, when the last two digits of the section before the hyphen become
> FF, pgbackrest creates a new directory:
> /var/lib/pgbackrest/archive/main/10/0000000100000001.

Yes.

> This is annoying as I have a cron job set up to focus on files in the
> 0000000100000000 directory.

What is the cronjob doing..?  You really shouldn't be hacking around
with things in the repo- there's commands available like archive-get to
extract out WAL files from the repo.

> Is there a way to force bgbackrest to only use one directory (e.g.
> 0000000100000000)?

No.

Thanks,

Stephen

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pgbackrest creating new directories (messing up cron jobs).

mimble9
Thank you for your answer!

>
>> This is annoying as I have a cron job set up to focus on files in the
>> 0000000100000000 directory.
>>
>
> What is the cronjob doing..?  You really shouldn't be hacking around
> with things in the repo- there's commands available like archive-get to
> extract out WAL files from the repo.
>

The cronjob is using 'mutt' to mail me the .gz files at a set period.

It says:

cd /var/lib/pgbackrest/archive/main/10/0000000100000000/
echo "" | mutt -s "Ttile" [hidden email] -a *.gz

Of course, once 0000000100000000 becomes 0000000100000001, this no longer
works.

Is there, perhaps, a better way to achieve my goal?

Thanks again - I do appreciate all your help.



Reply | Threaded
Open this post in threaded view
|

Re: pgbackrest creating new directories (messing up cron jobs).

Stephen Frost
Greetings,

* [hidden email] ([hidden email]) wrote:

> >> This is annoying as I have a cron job set up to focus on files in the
> >> 0000000100000000 directory.
> >
> > What is the cronjob doing..?  You really shouldn't be hacking around
> > with things in the repo- there's commands available like archive-get to
> > extract out WAL files from the repo.
>
> The cronjob is using 'mutt' to mail me the .gz files at a set period.
>
> It says:
>
> cd /var/lib/pgbackrest/archive/main/10/0000000100000000/
> echo "" | mutt -s "Ttile" [hidden email] -a *.gz
Isn't that going to mail you the same WAL over and over again if you set
it up as a cronjob..?  Is that really what you want?

> Of course, once 0000000100000000 becomes 0000000100000001, this no longer
> works.

That doesn't seem like the worst of the problems here.

> Is there, perhaps, a better way to achieve my goal?

Well, you could use archive-get with pgbackrest to pull out the files by
requesting each segment number, but I'm not sure what the idea here is
exactly- *why* are you email'ing them?

Thanks,

Stephen

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pgbackrest creating new directories (messing up cron jobs).

mimble9

I think I need to take a step backwards and ask something else before
moving forward.

>
> Isn't that going to mail you the same WAL over and over again if you set
> it up as a cronjob..?  Is that really what you want?
>

I took a full backup when I started. Then I used type=diff. This creates
two archives every x hours (depending on the time period set in cron).

For example:

-rw-r----- 1 postgres postgres 27145 Mar 11 00:00
0000000100000001000000B9-6f3902fe5c3bdebc3c1c124ec6821c7206e350da.gz

-rw-r----- 1 postgres postgres 27126 Mar 11 00:00
0000000100000001000000BA-214e7142c6eda0a350577f6bd624c3db203e184f.gz

Only one of these relates to the database I setup in PostgreSQL. I don't
know what the other one is but it might be related to the "default"
database 'postgres'. (This is just a guess).

Irrespective, I would only want a new archive file if something has
changed in the database. Yet I seem to always receive two new files every
x hours even if nothing has changed.

So I tried type-incr but that seems to produce the same results as type=diff.

I wonder what I am doing wrong?



Reply | Threaded
Open this post in threaded view
|

Re: pgbackrest creating new directories (messing up cron jobs).

Stephen Frost
Greetings,

* [hidden email] ([hidden email]) wrote:
> I think I need to take a step backwards and ask something else before
> moving forward.

Good idea.

> > Isn't that going to mail you the same WAL over and over again if you set
> > it up as a cronjob..?  Is that really what you want?
>
> I took a full backup when I started. Then I used type=diff. This creates
> two archives every x hours (depending on the time period set in cron).

WAL files are not "archives" in the sense that they are a complete
database backup- they're just the write-ahead-logs from the ongoing
running of PG.  You need the actual data files as well as the WAL to
perform a restore.

> For example:
>
> -rw-r----- 1 postgres postgres 27145 Mar 11 00:00
> 0000000100000001000000B9-6f3902fe5c3bdebc3c1c124ec6821c7206e350da.gz
>
> -rw-r----- 1 postgres postgres 27126 Mar 11 00:00
> 0000000100000001000000BA-214e7142c6eda0a350577f6bd624c3db203e184f.gz
>
> Only one of these relates to the database I setup in PostgreSQL. I don't
> know what the other one is but it might be related to the "default"
> database 'postgres'. (This is just a guess).
No, that's not how WAL works.  Those are two WAL files and they're both
generated as part of running the PG system- they are not specific to one
database or another in PG.

> Irrespective, I would only want a new archive file if something has
> changed in the database. Yet I seem to always receive two new files every
> x hours even if nothing has changed.

These individuals files are not archives of the entire PG system or
anything you can use to reconstruct a running PG system with just one
file.

> So I tried type-incr but that seems to produce the same results as type=diff.

The difference between incremental and differential backups is
documented at https://pgbackrest.org

> I wonder what I am doing wrong?

I'm afraid there are some pretty fundamental misunderstandings that you
have about the PG WAL, PG backups, and what you can do with a PG cluster
andf file-level backup/restore.

Documentation about the WAL can be found here:

https://www.postgresql.org/docs/current/wal-intro.html

If what you're actually looking for is a single file that has a backup
of your entire database, you can get that by running:

pgbackrest backup --archive-copy

and then do:

pgbackrest restore --pg-path=/path/to/somewhere
tar -czf pgdata.tar.gz /path/to/somewhere

or so.  Check the pgbackrest docs for the specific command syntax and
such.

Thanks,

Stephen

signature.asc (836 bytes) Download Attachment