Postgresql and Software RAID/LVM

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Postgresql and Software RAID/LVM

Steve Poe
I have a small business client that cannot afford high-end/high quality
RAID cards for their next server. That's a seperate argument/issue right
there for me, but what the client wants is what the client wants.

Has anyone ran Postgres with software RAID or LVM on a production box?
What have been your experience?

I don't forsee more 10-15 concurrent sessions running for an their OLTP
application.

Thanks.

Steve Poe


---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

               http://archives.postgresql.org
Reply | Threaded
Open this post in threaded view
|

Re: Postgresql and Software RAID/LVM

Joshua D. Drake
Steve Poe wrote:
> I have a small business client that cannot afford high-end/high quality
> RAID cards for their next server. That's a seperate argument/issue right
> there for me, but what the client wants is what the client wants.
>
> Has anyone ran Postgres with software RAID or LVM on a production box?
> What have been your experience?

I would not run RAID + LVM in a software scenario. Software RAID is fine
however.

Sincerely,

Joshua D. Drake


>
> I don't forsee more 10-15 concurrent sessions running for an their OLTP
> application.
>
> Thanks.
>
> Steve Poe
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
>                http://archives.postgresql.org


--
Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240
PostgreSQL Replication, Consulting, Custom Programming, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

               http://archives.postgresql.org
Reply | Threaded
Open this post in threaded view
|

Re: Postgresql and Software RAID/LVM

Andrew McMillan-3
In reply to this post by Steve Poe
On Fri, 2005-06-03 at 11:45 -0700, Steve Poe wrote:
> I have a small business client that cannot afford high-end/high quality
> RAID cards for their next server. That's a seperate argument/issue right
> there for me, but what the client wants is what the client wants.
>
> Has anyone ran Postgres with software RAID or LVM on a production box?
> What have been your experience?

Hi,

We regularly run LVM on top of software raid for our PostgreSQL servers
(and our other servers, for that matter).

As far as I can see these systems have not had issues related to either
software RAID or LVM - that's around 30 systems all up, maybe 8 running
PostgreSQL, in production.  The database servers are a variety of
dual-Xeon (older) and dual-Opteron (newer) systems.

The Xeons are all running Debian "Woody" with 2.4.xx kernels and the
Opterons are all running "Sarge" with 2.6.x kernels.

Regards,
                                        Andrew.

-------------------------------------------------------------------------
Andrew @ Catalyst .Net .NZ  Ltd,  PO Box 11-053, Manners St,  Wellington
WEB: http://catalyst.net.nz/            PHYS: Level 2, 150-154 Willis St
DDI: +64(4)803-2201      MOB: +64(272)DEBIAN      OFFICE: +64(4)499-2267
       The secret of being a bore is to say everything -- Voltaire
-------------------------------------------------------------------------


signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Postgresql and Software RAID/LVM

Marty Scholes
In reply to this post by Steve Poe
 > Has anyone ran Postgres with software RAID or LVM on a production box?
 > What have been your experience?

Yes, we have run for a couple years Pg with software LVM (mirroring)
against two hardware RAID5 arrays.  We host a production Sun box that
runs 24/7.

My experience:
* Software RAID (other than mirroring) is a disaster waiting to happen.
  If the metadata for the RAID set gives out for any reason (CMOS
scrambles, card dies, power spike, etc.) then you are hosed beyond
belief.  In most cases it is almost impossible to recover.  With
mirroring, however, you can always boot and operate on a single mirror,
pretending that no LVM/RAID is underway.  In other words, each mirror is
a fully functional copy of the data which will operate your server.

* Hardware RAID5 is a terrific way to boost performance via write
caching and spreading I/O across multiple spindles.  Each of our
external arrays operates 14 drives (12 data, 1 parity and 1 hot spare).
  While RAID5 protects against single spindle failure, it will not hedge
against multiple failures in a short time period, SCSI contoller
failure, SCSI cable problems or even wholesale failure of the RAID
controller.  All of these things happen in a 24/7 operation.  Using
software RAID1 against the hardware RAID5 arrays hedges against any
single failure.

* Software mirroring gives you tremendous ability to change the system
while it is running, by taking offline the mirror you wish to change and
then synchronizing it after the change.

On a fully operational production server, we have:
* restriped the RAID5 array
* replaced all RAID5 media with higher capacity drives
* upgraded RAID5 controller
* moved all data from an old RAID5 array to a newer one
* replaced host SCSI controller
* uncabled and physically moved storage to a different part of data center

Again, all of this has taken place (over the years) while our machine
was fully operational.


---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
      subscribe-nomail command to [hidden email] so that your
      message can get through to the mailing list cleanly
Reply | Threaded
Open this post in threaded view
|

Re: Postgresql and Software RAID/LVM

John Arbash Meinel
Marty Scholes wrote:

>> Has anyone ran Postgres with software RAID or LVM on a production box?
>> What have been your experience?
>
> Yes, we have run for a couple years Pg with software LVM (mirroring)
> against two hardware RAID5 arrays.  We host a production Sun box that
> runs 24/7.
>
> My experience:
> * Software RAID (other than mirroring) is a disaster waiting to happen.
>  If the metadata for the RAID set gives out for any reason (CMOS
> scrambles, card dies, power spike, etc.) then you are hosed beyond
> belief.  In most cases it is almost impossible to recover.  With
> mirroring, however, you can always boot and operate on a single mirror,
> pretending that no LVM/RAID is underway.  In other words, each mirror is
> a fully functional copy of the data which will operate your server.
Isn't this actually more of a problem for the meta-data to give out in a
hardware situation? I mean, if the card you are using dies, you can't
just get another one.
With software raid, because the meta-data is on the drives, you can pull
it out of that machine, and put it into any machine that has a
controller which can read the drives, and a similar kernel, and you are
back up and running.

>
> * Hardware RAID5 is a terrific way to boost performance via write
> caching and spreading I/O across multiple spindles.  Each of our
> external arrays operates 14 drives (12 data, 1 parity and 1 hot spare).
>  While RAID5 protects against single spindle failure, it will not hedge
> against multiple failures in a short time period, SCSI contoller
> failure, SCSI cable problems or even wholesale failure of the RAID
> controller.  All of these things happen in a 24/7 operation.  Using
> software RAID1 against the hardware RAID5 arrays hedges against any
> single failure.
No, it hedges against *more* than one failure. But you can also do a
RAID1 over a RAID5 in software. But if you are honestly willing to
create a full RAID1, just create a RAID1 over RAID0. The performance is
much better. And since you have a full RAID1, as long as both drives of
a pairing don't give out, you can lose half of your drives.

If you want the space, but you feel that RAID5 isn't redundant enough,
go to RAID6, which uses 2 parity locations, each with a different method
of storing parity, so not only is it more redundant, you have a better
chance of finding problems.

>
> * Software mirroring gives you tremendous ability to change the system
> while it is running, by taking offline the mirror you wish to change and
> then synchronizing it after the change.
>

That certainly is a nice ability. But remember that LVM also has the
idea of "snapshot"ing a running system. I don't know the exact details,
just that there is a way to have some processes see the filesystem as it
existed at an exact point in time. Which is also a great way to handle
backups.

> On a fully operational production server, we have:
> * restriped the RAID5 array
> * replaced all RAID5 media with higher capacity drives
> * upgraded RAID5 controller
> * moved all data from an old RAID5 array to a newer one
> * replaced host SCSI controller
> * uncabled and physically moved storage to a different part of data center
>
> Again, all of this has taken place (over the years) while our machine
> was fully operational.
>
So you are saying that you were able to replace the RAID controller
without turning off the machine? I realize there does exist
hot-swappable PCI cards, but I think you are overstating what you mean
by "fully operational". For instance, it's not like you can access your
data while it is being physically moved.

I do think you had some nice hardware. But I know you can do all of this
in software as well. It is usually a price/performance tradeoff. You
spend quite a bit to get a hardware RAID card that can keep up with a
modern CPU. I know we have an FC raid box at work which has a full 512MB
of cache on it, but it wasn't that much cheaper than buying a dedicated
server.

John
=:->

signature.asc (259 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Postgresql and Software RAID/LVM

Marty Scholes
John A Meinel wrote:
>
> Isn't this actually more of a problem for the meta-data to give out in a
> hardware situation? I mean, if the card you are using dies, you can't
> just get another one.
> With software raid, because the meta-data is on the drives, you can pull
> it out of that machine, and put it into any machine that has a
> controller which can read the drives, and a similar kernel, and you are
> back up and running.

Probably true.  If you have a similar kernel and hardware and if you can
recover the state information, knowing where the state information is
stored.  Those are some very big "ifs" during a hectic disaster.

> No, it hedges against *more* than one failure. But you can also do a
> RAID1 over a RAID5 in software. But if you are honestly willing to
> create a full RAID1, just create a RAID1 over RAID0. The performance is
> much better. And since you have a full RAID1, as long as both drives of
> a pairing don't give out, you can lose half of your drives.

True as well.  The problem with RAID1 over RAID0 is that, during a drive
failure, you are one bad sector from disaster.  Further, RAID5 does
automatic rebuild, whereas most RAID1 setups do not.  RAID5 reduces the
amount of time that things are degraded, reducing the time that your
data is in danger.

> If you want the space, but you feel that RAID5 isn't redundant enough,
> go to RAID6, which uses 2 parity locations, each with a different method
> of storing parity, so not only is it more redundant, you have a better
> chance of finding problems.

Agreed, RAID6 is the future, but still won't keep the server running
when the RAID controller dies, or the SCSI/FC host adapter goes, or you
want to upgrade controller firmware, or you want to replace the media, or...

> So you are saying that you were able to replace the RAID controller
> without turning off the machine? I realize there does exist
> hot-swappable PCI cards, but I think you are overstating what you mean
> by "fully operational". For instance, it's not like you can access your
> data while it is being physically moved.

Detach mirror 1, uncable and move, recable and resync.  Detach mirror 2,
uncable and move, recable and resync.

>
> I do think you had some nice hardware. But I know you can do all of this
> in software as well. It is usually a price/performance tradeoff. You
> spend quite a bit to get a hardware RAID card that can keep up with a
> modern CPU. I know we have an FC raid box at work which has a full 512MB
> of cache on it, but it wasn't that much cheaper than buying a dedicated
> server.

We run two Nexsan ATABoy2 arrays.  These can be found in 1 TB
configurations for about $3,000 each, putting mirrored RAID5 storage at
$6 per GB.  Is that a lot of money for storage?  Maybe.  In our case,
that's dirt cheap protection against storage-related downtime.

Marty


---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
      subscribe-nomail command to [hidden email] so that your
      message can get through to the mailing list cleanly