PostgreSQL replication failover

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

PostgreSQL replication failover

Jan Peters
Hello,

we are running postgresqlserver on s390 zLinux machines. The distribution is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.

We always run 2 instances with a replication (streaming) async mode, the replica is in hot_standby and we use it for read-only accesses. About the setup we have the following question:

How is an orderly failover accomplished? Our current procedure is.

1. primary stop
2. promote replica to primary
3. create standby.signal on old primary
4. change primary_conninfo on old primary
5. start old primary as new replica

Is this processing correct? Are there any other steps that simplify a failover? How can we be sure that all changes have been transferred from the old master to the replica?

thanks in advance



Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL replication failover

Laurenz Albe
On Wed, 2021-01-13 at 09:27 +0100, Jan Peters wrote:

> we are running postgresqlserver on s390 zLinux machines. The distribution
>  is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.
>
> We always run 2 instances with a replication (streaming) async mode, the replica
>  is in hot_standby and we use it for read-only accesses. About the setup we have the following question:
>
> How is an orderly failover accomplished? Our current procedure is.
>
> 1. primary stop
> 2. promote replica to primary
> 3. create standby.signal on old primary
> 4. change primary_conninfo on old primary
> 5. start old primary as new replica
>
> Is this processing correct? Are there any other steps that simplify a failover?
>  How can we be sure that all changes have been transferred from the old master to the replica?

What you describe is not a failover, but a switchover.

If you shut down the primary server cleanly, all changes will be replicated,
so you should be good.

During a failover, that is, if the primary suddenly fails, there is always
the possibility that you lose some transactions, unless you use synchronous
replication.

Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com



bvo
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL replication failover

bvo
>>If you shut down the primary server cleanly, all changes will be replicated,so you should be good.

>>During a failover, that is, if the primary suddenly fails, there is always
the possibility that you lose some transactions, unless you use synchronous

you said above which I don't need to run promote to make it failover as long as I set synchronous on?   The last couple of weeks I have a failure on the primary server and can't run on a slave. It picks up as reading mode only.

Bach-Nga

No one in this world is pure and perfect.  If you avoid people for their mistakes you will be alone. So judge less, love, and forgive more.EmojiEmojiEmoji
To call him a dog hardly seems to do him justice though in as much as he had four legs, a tail, and barked, I admit he was, to all outward appearances. But to those who knew him well, he was a perfect gentleman (Hermione Gingold)

**Live simply **Love generously **Care deeply **Speak kindly.
*** Genuinely rich *** Faithful talent *** Sharing success




On Wednesday, January 13, 2021, 06:25:53 AM EST, Laurenz Albe <[hidden email]> wrote:


On Wed, 2021-01-13 at 09:27 +0100, Jan Peters wrote:

> we are running postgresqlserver on s390 zLinux machines. The distribution
>  is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.
>
> We always run 2 instances with a replication (streaming) async mode, the replica
>  is in hot_standby and we use it for read-only accesses. About the setup we have the following question:
>
> How is an orderly failover accomplished? Our current procedure is.
>
> 1. primary stop
> 2. promote replica to primary
> 3. create standby.signal on old primary
> 4. change primary_conninfo on old primary
> 5. start old primary as new replica
>
> Is this processing correct? Are there any other steps that simplify a failover?
>  How can we be sure that all changes have been transferred from the old master to the replica?

What you describe is not a failover, but a switchover.

If you shut down the primary server cleanly, all changes will be replicated,
so you should be good.

During a failover, that is, if the primary suddenly fails, there is always
the possibility that you lose some transactions, unless you use synchronous

replication.


Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com




Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL replication failover

Laurenz Albe
On Wed, 2021-01-13 at 13:24 +0000, Pepe TD Vo wrote:
> you said above which I don't need to run promote to make it failover as long as I set synchronous on?

That's not what I meant.  Of course you have to promote the standby after shutting down
the old primary server.

Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com



Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL replication failover

ganeshakorde
In reply to this post by bvo
You can use different tools which detects if primary fails and automatically promotes standby. 

To assure all data on standby you should use synchronous replication.

On Wed, 13 Jan 2021, 6:54 pm Pepe TD Vo, <[hidden email]> wrote:
>>If you shut down the primary server cleanly, all changes will be replicated,so you should be good.

>>During a failover, that is, if the primary suddenly fails, there is always
the possibility that you lose some transactions, unless you use synchronous

you said above which I don't need to run promote to make it failover as long as I set synchronous on?   The last couple of weeks I have a failure on the primary server and can't run on a slave. It picks up as reading mode only.

Bach-Nga

No one in this world is pure and perfect.  If you avoid people for their mistakes you will be alone. So judge less, love, and forgive more.EmojiEmojiEmoji
To call him a dog hardly seems to do him justice though in as much as he had four legs, a tail, and barked, I admit he was, to all outward appearances. But to those who knew him well, he was a perfect gentleman (Hermione Gingold)

**Live simply **Love generously **Care deeply **Speak kindly.
*** Genuinely rich *** Faithful talent *** Sharing success




On Wednesday, January 13, 2021, 06:25:53 AM EST, Laurenz Albe <[hidden email]> wrote:


On Wed, 2021-01-13 at 09:27 +0100, Jan Peters wrote:

> we are running postgresqlserver on s390 zLinux machines. The distribution
>  is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.
>
> We always run 2 instances with a replication (streaming) async mode, the replica
>  is in hot_standby and we use it for read-only accesses. About the setup we have the following question:
>
> How is an orderly failover accomplished? Our current procedure is.
>
> 1. primary stop
> 2. promote replica to primary
> 3. create standby.signal on old primary
> 4. change primary_conninfo on old primary
> 5. start old primary as new replica
>
> Is this processing correct? Are there any other steps that simplify a failover?
>  How can we be sure that all changes have been transferred from the old master to the replica?

What you describe is not a failover, but a switchover.

If you shut down the primary server cleanly, all changes will be replicated,
so you should be good.

During a failover, that is, if the primary suddenly fails, there is always
the possibility that you lose some transactions, unless you use synchronous

replication.


Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com




Reply | Threaded
Open this post in threaded view
|

Aw: Re: PostgreSQL replication failover

Jan Peters
Hello,

thank you very much for the answers.
Can you tell me some tools, but they must be available for s390 ZLinux.
For our purposes in redhat linux
 

Gesendet: Mittwoch, 13. Januar 2021 um 19:46 Uhr
Von: "Ganesh Korde" <[hidden email]>
An: "Pepe TD Vo" <[hidden email]>
Cc: "Jan Peters" <[hidden email]>, [hidden email], "Laurenz Albe" <[hidden email]>
Betreff: Re: PostgreSQL replication failover

You can use different tools which detects if primary fails and automatically promotes standby. 
 
To assure all data on standby you should use synchronous replication. 

On Wed, 13 Jan 2021, 6:54 pm Pepe TD Vo, <[hidden email][mailto:[hidden email]]> wrote:

>>If you shut down the primary server cleanly, all changes will be replicated,so you should be good.

>>During a failover, that is, if the primary suddenly fails, there is always
the possibility that you lose some transactions, unless you use synchronous
you said above which I don't need to run promote to make it failover as long as I set synchronous on?   The last couple of weeks I have a failure on the primary server and can't run on a slave. It picks up as reading mode only.
 

Bach-Nga

No one in this world is pure and perfect.  If you avoid people for their mistakes you will be alone. So judge less, love, and forgive more.
To call him a dog hardly seems to do him justice though in as much as he had four legs, a tail, and barked, I admit he was, to all outward appearances. But to those who knew him well, he was a perfect gentleman (Hermione Gingold)
**Live simply **Love generously **Care deeply **Speak kindly.
*** Genuinely rich *** Faithful talent *** Sharing success
 
 
 
 

On Wednesday, January 13, 2021, 06:25:53 AM EST, Laurenz Albe <[hidden email][mailto:[hidden email]]> wrote:
 
 

On Wed, 2021-01-13 at 09:27 +0100, Jan Peters wrote:

> we are running postgresqlserver on s390 zLinux machines. The distribution
>  is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.
>
> We always run 2 instances with a replication (streaming) async mode, the replica
>  is in hot_standby and we use it for read-only accesses. About the setup we have the following question:
>
> How is an orderly failover accomplished? Our current procedure is.
>
> 1. primary stop
> 2. promote replica to primary
> 3. create standby.signal on old primary
> 4. change primary_conninfo on old primary
> 5. start old primary as new replica
>
> Is this processing correct? Are there any other steps that simplify a failover?
>  How can we be sure that all changes have been transferred from the old master to the replica?

What you describe is not a failover, but a switchover.

If you shut down the primary server cleanly, all changes will be replicated,
so you should be good.

During a failover, that is, if the primary suddenly fails, there is always
the possibility that you lose some transactions, unless you use synchronous
replication.

Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com[https://www.cybertec-postgresql.com]


 


Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL replication failover

jwiencek3
Hi

You can use repmgr, which is free,  or EFM which requires a subscription.

John

> On Jan 14, 2021, at 2:16 AM, Jan Peters <[hidden email]> wrote:
>
> Hello,
>
> thank you very much for the answers.
> Can you tell me some tools, but they must be available for s390 ZLinux.
> For our purposes in redhat linux
>
>
> Gesendet: Mittwoch, 13. Januar 2021 um 19:46 Uhr
> Von: "Ganesh Korde" <[hidden email]>
> An: "Pepe TD Vo" <[hidden email]>
> Cc: "Jan Peters" <[hidden email]>, [hidden email], "Laurenz Albe" <[hidden email]>
> Betreff: Re: PostgreSQL replication failover
>
> You can use different tools which detects if primary fails and automatically promotes standby.
>  
> To assure all data on standby you should use synchronous replication.
>
> On Wed, 13 Jan 2021, 6:54 pm Pepe TD Vo, <[hidden email][mailto:[hidden email]]> wrote:
>
>>> If you shut down the primary server cleanly, all changes will be replicated,so you should be good.
>
>>> During a failover, that is, if the primary suddenly fails, there is always
> the possibility that you lose some transactions, unless you use synchronous
> you said above which I don't need to run promote to make it failover as long as I set synchronous on?   The last couple of weeks I have a failure on the primary server and can't run on a slave. It picks up as reading mode only.
>  
>
> Bach-Nga
>
> No one in this world is pure and perfect.  If you avoid people for their mistakes you will be alone. So judge less, love, and forgive more.
> To call him a dog hardly seems to do him justice though in as much as he had four legs, a tail, and barked, I admit he was, to all outward appearances. But to those who knew him well, he was a perfect gentleman (Hermione Gingold)
> **Live simply **Love generously **Care deeply **Speak kindly.
> *** Genuinely rich *** Faithful talent *** Sharing success
>  
>  
>  
>  
>
> On Wednesday, January 13, 2021, 06:25:53 AM EST, Laurenz Albe <[hidden email][mailto:[hidden email]]> wrote:
>  
>  
>
> On Wed, 2021-01-13 at 09:27 +0100, Jan Peters wrote:
>> we are running postgresqlserver on s390 zLinux machines. The distribution
>>   is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.
>>
>> We always run 2 instances with a replication (streaming) async mode, the replica
>>   is in hot_standby and we use it for read-only accesses. About the setup we have the following question:
>>
>> How is an orderly failover accomplished? Our current procedure is.
>>
>> 1. primary stop
>> 2. promote replica to primary
>> 3. create standby.signal on old primary
>> 4. change primary_conninfo on old primary
>> 5. start old primary as new replica
>>
>> Is this processing correct? Are there any other steps that simplify a failover?
>>   How can we be sure that all changes have been transferred from the old master to the replica?
>
> What you describe is not a failover, but a switchover.
>
> If you shut down the primary server cleanly, all changes will be replicated,
> so you should be good.
>
> During a failover, that is, if the primary suddenly fails, there is always
> the possibility that you lose some transactions, unless you use synchronous
> replication.
>
> Yours,
> Laurenz Albe
> --
> Cybertec | https://www.cybertec-postgresql.com[https://www.cybertec-postgresql.com]
>
>
>  
>
>



Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL replication failover

franklin ferreira de lima
Fundação de apoio à escola técnica 



franklin ferreira de lima

Em Qui, 14 de jan de 2021 12:03, John Wiencek <[hidden email]> escreveu:
Hi

You can use repmgr, which is free,  or EFM which requires a subscription.

John

> On Jan 14, 2021, at 2:16 AM, Jan Peters <[hidden email]> wrote:
>
> Hello,
>
> thank you very much for the answers.
> Can you tell me some tools, but they must be available for s390 ZLinux.
> For our purposes in redhat linux
>
>
> Gesendet: Mittwoch, 13. Januar 2021 um 19:46 Uhr
> Von: "Ganesh Korde" <[hidden email]>
> An: "Pepe TD Vo" <[hidden email]>
> Cc: "Jan Peters" <[hidden email]>, [hidden email], "Laurenz Albe" <[hidden email]>
> Betreff: Re: PostgreSQL replication failover
>
> You can use different tools which detects if primary fails and automatically promotes standby.

> To assure all data on standby you should use synchronous replication.
>
> On Wed, 13 Jan 2021, 6:54 pm Pepe TD Vo, <[hidden email][mailto:[hidden email]]> wrote:
>
>>> If you shut down the primary server cleanly, all changes will be replicated,so you should be good.
>
>>> During a failover, that is, if the primary suddenly fails, there is always
> the possibility that you lose some transactions, unless you use synchronous
> you said above which I don't need to run promote to make it failover as long as I set synchronous on?   The last couple of weeks I have a failure on the primary server and can't run on a slave. It picks up as reading mode only.

>
> Bach-Nga
>
> No one in this world is pure and perfect.  If you avoid people for their mistakes you will be alone. So judge less, love, and forgive more.
> To call him a dog hardly seems to do him justice though in as much as he had four legs, a tail, and barked, I admit he was, to all outward appearances. But to those who knew him well, he was a perfect gentleman (Hermione Gingold)
> **Live simply **Love generously **Care deeply **Speak kindly.
> *** Genuinely rich *** Faithful talent *** Sharing success




>
> On Wednesday, January 13, 2021, 06:25:53 AM EST, Laurenz Albe <[hidden email][mailto:[hidden email]]> wrote:


>
> On Wed, 2021-01-13 at 09:27 +0100, Jan Peters wrote:
>> we are running postgresqlserver on s390 zLinux machines. The distribution
>>   is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.
>>
>> We always run 2 instances with a replication (streaming) async mode, the replica
>>   is in hot_standby and we use it for read-only accesses. About the setup we have the following question:
>>
>> How is an orderly failover accomplished? Our current procedure is.
>>
>> 1. primary stop
>> 2. promote replica to primary
>> 3. create standby.signal on old primary
>> 4. change primary_conninfo on old primary
>> 5. start old primary as new replica
>>
>> Is this processing correct? Are there any other steps that simplify a failover?
>>   How can we be sure that all changes have been transferred from the old master to the replica?
>
> What you describe is not a failover, but a switchover.
>
> If you shut down the primary server cleanly, all changes will be replicated,
> so you should be good.
>
> During a failover, that is, if the primary suddenly fails, there is always
> the possibility that you lose some transactions, unless you use synchronous
> replication.
>
> Yours,
> Laurenz Albe
> --
> Cybertec | https://www.cybertec-postgresql.com[https://www.cybertec-postgresql.com]
>
>

>
>




CV franklin ferreira de lima 75221386.rtf (83K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Aw: Re: PostgreSQL replication failover

Jan Peters
In reply to this post by jwiencek3
Hi John,

I can't find any s390 packages for repmgr :-(



> Gesendet: Donnerstag, 14. Januar 2021 um 16:03 Uhr
> Von: "John Wiencek" <[hidden email]>
> An: "Jan Peters" <[hidden email]>
> Cc: "Ganesh Korde" <[hidden email]>, "Pepe TD Vo" <[hidden email]>, "Pgsql-admin" <[hidden email]>, "Laurenz Albe" <[hidden email]>
> Betreff: Re: PostgreSQL replication failover
>
> Hi
>
> You can use repmgr, which is free,  or EFM which requires a subscription.
>
> John
>
> > On Jan 14, 2021, at 2:16 AM, Jan Peters <[hidden email]> wrote:
> >
> > Hello,
> >
> > thank you very much for the answers.
> > Can you tell me some tools, but they must be available for s390 ZLinux.
> > For our purposes in redhat linux
> >
> >
> > Gesendet: Mittwoch, 13. Januar 2021 um 19:46 Uhr
> > Von: "Ganesh Korde" <[hidden email]>
> > An: "Pepe TD Vo" <[hidden email]>
> > Cc: "Jan Peters" <[hidden email]>, [hidden email], "Laurenz Albe" <[hidden email]>
> > Betreff: Re: PostgreSQL replication failover
> >
> > You can use different tools which detects if primary fails and automatically promotes standby.
> >
> > To assure all data on standby you should use synchronous replication.
> >
> > On Wed, 13 Jan 2021, 6:54 pm Pepe TD Vo, <[hidden email][mailto:[hidden email]]> wrote:
> >
> >>> If you shut down the primary server cleanly, all changes will be replicated,so you should be good.
> >
> >>> During a failover, that is, if the primary suddenly fails, there is always
> > the possibility that you lose some transactions, unless you use synchronous
> > you said above which I don't need to run promote to make it failover as long as I set synchronous on?   The last couple of weeks I have a failure on the primary server and can't run on a slave. It picks up as reading mode only.
> >
> >
> > Bach-Nga
> >
> > No one in this world is pure and perfect.  If you avoid people for their mistakes you will be alone. So judge less, love, and forgive more.
> > To call him a dog hardly seems to do him justice though in as much as he had four legs, a tail, and barked, I admit he was, to all outward appearances. But to those who knew him well, he was a perfect gentleman (Hermione Gingold)
> > **Live simply **Love generously **Care deeply **Speak kindly.
> > *** Genuinely rich *** Faithful talent *** Sharing success
> >
> >
> >
> >
> >
> > On Wednesday, January 13, 2021, 06:25:53 AM EST, Laurenz Albe <[hidden email][mailto:[hidden email]]> wrote:
> >
> >
> >
> > On Wed, 2021-01-13 at 09:27 +0100, Jan Peters wrote:
> >> we are running postgresqlserver on s390 zLinux machines. The distribution
> >>   is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.
> >>
> >> We always run 2 instances with a replication (streaming) async mode, the replica
> >>   is in hot_standby and we use it for read-only accesses. About the setup we have the following question:
> >>
> >> How is an orderly failover accomplished? Our current procedure is.
> >>
> >> 1. primary stop
> >> 2. promote replica to primary
> >> 3. create standby.signal on old primary
> >> 4. change primary_conninfo on old primary
> >> 5. start old primary as new replica
> >>
> >> Is this processing correct? Are there any other steps that simplify a failover?
> >>   How can we be sure that all changes have been transferred from the old master to the replica?
> >
> > What you describe is not a failover, but a switchover.
> >
> > If you shut down the primary server cleanly, all changes will be replicated,
> > so you should be good.
> >
> > During a failover, that is, if the primary suddenly fails, there is always
> > the possibility that you lose some transactions, unless you use synchronous
> > replication.
> >
> > Yours,
> > Laurenz Albe
> > --
> > Cybertec | https://www.cybertec-postgresql.com[https://www.cybertec-postgresql.com]
> >
> >
> >
> >
> >
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Re: PostgreSQL replication failover

ganeshakorde

On Tue, 19 Jan 2021, 1:42 pm Jan Peters, <[hidden email]> wrote:
Hi John,

I can't find any s390 packages for repmgr :-(



> Gesendet: Donnerstag, 14. Januar 2021 um 16:03 Uhr
> Von: "John Wiencek" <[hidden email]>
> An: "Jan Peters" <[hidden email]>
> Cc: "Ganesh Korde" <[hidden email]>, "Pepe TD Vo" <[hidden email]>, "Pgsql-admin" <[hidden email]>, "Laurenz Albe" <[hidden email]>
> Betreff: Re: PostgreSQL replication failover
>
> Hi
>
> You can use repmgr, which is free,  or EFM which requires a subscription.
>
> John
>
> > On Jan 14, 2021, at 2:16 AM, Jan Peters <[hidden email]> wrote:
> >
> > Hello,
> >
> > thank you very much for the answers.
> > Can you tell me some tools, but they must be available for s390 ZLinux.
> > For our purposes in redhat linux
> >
> >
> > Gesendet: Mittwoch, 13. Januar 2021 um 19:46 Uhr
> > Von: "Ganesh Korde" <[hidden email]>
> > An: "Pepe TD Vo" <[hidden email]>
> > Cc: "Jan Peters" <[hidden email]>, [hidden email], "Laurenz Albe" <[hidden email]>
> > Betreff: Re: PostgreSQL replication failover
> >
> > You can use different tools which detects if primary fails and automatically promotes standby.
> >
> > To assure all data on standby you should use synchronous replication.
> >
> > On Wed, 13 Jan 2021, 6:54 pm Pepe TD Vo, <[hidden email][mailto:[hidden email]]> wrote:
> >
> >>> If you shut down the primary server cleanly, all changes will be replicated,so you should be good.
> >
> >>> During a failover, that is, if the primary suddenly fails, there is always
> > the possibility that you lose some transactions, unless you use synchronous
> > you said above which I don't need to run promote to make it failover as long as I set synchronous on?   The last couple of weeks I have a failure on the primary server and can't run on a slave. It picks up as reading mode only.
> >
> >
> > Bach-Nga
> >
> > No one in this world is pure and perfect.  If you avoid people for their mistakes you will be alone. So judge less, love, and forgive more.
> > To call him a dog hardly seems to do him justice though in as much as he had four legs, a tail, and barked, I admit he was, to all outward appearances. But to those who knew him well, he was a perfect gentleman (Hermione Gingold)
> > **Live simply **Love generously **Care deeply **Speak kindly.
> > *** Genuinely rich *** Faithful talent *** Sharing success
> >
> >
> >
> >
> >
> > On Wednesday, January 13, 2021, 06:25:53 AM EST, Laurenz Albe <[hidden email][mailto:[hidden email]]> wrote:
> >
> >
> >
> > On Wed, 2021-01-13 at 09:27 +0100, Jan Peters wrote:
> >> we are running postgresqlserver on s390 zLinux machines. The distribution
> >>   is RedHat 7 and RedHat 8, so we do not have the many x86 tools available.
> >>
> >> We always run 2 instances with a replication (streaming) async mode, the replica
> >>   is in hot_standby and we use it for read-only accesses. About the setup we have the following question:
> >>
> >> How is an orderly failover accomplished? Our current procedure is.
> >>
> >> 1. primary stop
> >> 2. promote replica to primary
> >> 3. create standby.signal on old primary
> >> 4. change primary_conninfo on old primary
> >> 5. start old primary as new replica
> >>
> >> Is this processing correct? Are there any other steps that simplify a failover?
> >>   How can we be sure that all changes have been transferred from the old master to the replica?
> >
> > What you describe is not a failover, but a switchover.
> >
> > If you shut down the primary server cleanly, all changes will be replicated,
> > so you should be good.
> >
> > During a failover, that is, if the primary suddenly fails, there is always
> > the possibility that you lose some transactions, unless you use synchronous
> > replication.
> >
> > Yours,
> > Laurenz Albe
> > --
> > Cybertec | https://www.cybertec-postgresql.com[https://www.cybertec-postgresql.com]
> >
> >
> >
> >
> >
>
>