Pre-Allocate tablespace on disk

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Pre-Allocate tablespace on disk

Chris Ruprecht
Hello,

I was wondering if there is a way to pre-allocate tablespace on disk before adding data and indexes.
My understanding is:
PG writes data into files sequentially. If more space is needed, disk space is requested from the OS and if there is space, the OS will give PG a file system block and PG will write data/indexes until that block is full and so on.
Other database that I have worked with before and that I'm still working with, allow you to pre-allocate disk space so you get huge chunks of contiguous space at one, which has major impacts on database performance.

Can this be done with PG at all? If not, is this planned for any time in the future?

best regards,
chris
--
chris ruprecht
database grunt and bit pusher extraordinaíre 


--
Sent via pgsql-admin mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
Reply | Threaded
Open this post in threaded view
|

Re: Pre-Allocate tablespace on disk

Guillaume Lelarge-3
Le 15/11/2010 20:42, Chris Ruprecht a écrit :
> Hello,
>
> I was wondering if there is a way to pre-allocate tablespace on disk before adding data and indexes.
> My understanding is:
> PG writes data into files sequentially. If more space is needed, disk space is requested from the OS and if there is space, the OS will give PG a file system block and PG will write data/indexes until that block is full and so on.
> Other database that I have worked with before and that I'm still working with, allow you to pre-allocate disk space so you get huge chunks of contiguous space at one, which has major impacts on database performance.
>
> Can this be done with PG at all?

No. A tablespace is a directory on PostgreSQL. And you can't allocate
space for a directory.

> If not, is this planned for any time in the future?

AFAICT, nope.


--
Guillaume
 http://www.postgresql.fr
 http://dalibo.com

--
Sent via pgsql-admin mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
Reply | Threaded
Open this post in threaded view
|

Re: Pre-Allocate tablespace on disk

Craig James
In reply to this post by Chris Ruprecht
On 11/15/10 11:42 AM, Chris Ruprecht wrote:
> I was wondering if there is a way to pre-allocate tablespace on disk before
>  adding data and indexes.
> My understanding is:
> PG writes data into files sequentially. If more space is needed, disk space
> is requested from the OS and if there is space, the OS will give PG a file
>  system block and PG will write data/indexes until that block is full and so on.
> Other database that I have worked with before and that I'm still working with,
> allow you to pre-allocate disk space so you get huge chunks of contiguous
> space at one, which has major impacts on database performance.

This shouldn't really be a performance issue on modern Unix-type systems.  Most file systems have fairly smart block-allocation algorithms that tend to allocate large blocks of contiguous space to files even when lots of processes are asking for little chunks in interleaved requests.  It doesn't just allocate a block to the next guy who happens to ask.  It's a lot smarter than that.  In most cases, you'll barely be able to tell the difference between a file that was allocated all at once and one that was allocated on an as-needed basis.  The only time you get bad fragmentation is when your disk gets nearly full, which takes away the file system's ability to be clever.

If you really want to preallocate a file, build a separate partition on your disk and only put one database (or even just one table) on that file system.

--
Sent via pgsql-admin mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
Reply | Threaded
Open this post in threaded view
|

ignoring data type

Octavio
Hello

I have migrate a application from a server with postgres 8.1 to another with 8.3
in the new server there are some page that doesn't work because seems to want to compare different data type

does anyone know if it is a version issue or there is something Im missing?

Thanks

Octavio
the error in spanish
2010-11-15 19:03:21 PET UBICACIÃN:  op_error, parse_oper.c:896
2010-11-15 19:03:21 PET SENTENCIA:  update clamcartga  set  codusuario=$1,fechapro=$2,horapro=$3,fechaenv=$4,flenvmail=$5  wh
ere corrabon=$6 and tipoafil=$7 and numcaso=$8 and corpacien=$9  and tipcarta=$10 and corrcart=$11
2010-11-15 19:06:16 PET ERROR:  42883: el operador no existe: character = integer en carácter 383
2010-11-15 19:06:16 PET HINT:  Ningún operador coincide con el nombre y el tipo de los argumentos. Puede desear agregar convesiones explícitas de tipos.




 
Reply | Threaded
Open this post in threaded view
|

Re: ignoring data type

Samuel Stearns

What does a \d on the table ‘clamcartga’ in both environments say?  Can you spot any differences between the 8.1 and 8.3 environments?

 

Sam

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Octavio
Sent: Tuesday, 16 November 2010 3:58 PM
To: [hidden email]
Subject: [ADMIN] ignoring data type

 

Hello

I have migrate a application from a server with postgres 8.1 to another with 8.3
in the new server there are some page that doesn't work because seems to want to compare different data type

does anyone know if it is a version issue or there is something Im missing?

Thanks

Octavio
the error in spanish
2010-11-15 19:03:21 PET UBICACIÃN:  op_error, parse_oper.c:896
2010-11-15 19:03:21 PET SENTENCIA:  update clamcartga  set  codusuario=$1,fechapro=$2,horapro=$3,fechaenv=$4,flenvmail=$5  wh
ere corrabon=$6 and tipoafil=$7 and numcaso=$8 and corpacien=$9  and tipcarta=$10 and corrcart=$11
2010-11-15 19:06:16 PET ERROR:  42883: el operador no existe: character = integer en carácter 383
2010-11-15 19:06:16 PET HINT:  Ningún operador coincide con el nombre y el tipo de los argumentos. Puede desear agregar convesiones explícitas de tipos.



 

Reply | Threaded
Open this post in threaded view
|

Re: ignoring data type

Scott Marlowe-2
In reply to this post by Octavio
On Mon, Nov 15, 2010 at 10:28 PM, Octavio <[hidden email]> wrote:
>
> Hello
>
> I have migrate a application from a server with postgres 8.1 to another with 8.3
> in the new server there are some page that doesn't work because seems to want to compare different data type
>
> does anyone know if it is a version issue or there is something Im missing?

Starting in 8.3 many automatic conversions from one type to another
were removed because it wasn't always obvious if this was the right
thing to do.  So, you might need to either change some column types,
or cast them explicitly to the type you want to be used for
comparison.

--
Sent via pgsql-admin mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
Reply | Threaded
Open this post in threaded view
|

Re: ignoring data type

Octavio
Thanks Scott

In fact checking the application there are a lots of queries that doesn't make the conversion, in order to temporally skip this issue,I like to try to import those conversions from the 8.1 version to the actual postgres, how can I do that?

Octavio

--- El mar, 11/16/10, Scott Marlowe <[hidden email]> escribió:

De: Scott Marlowe <[hidden email]>
Asunto: Re: [ADMIN] ignoring data type
A: "Octavio" <[hidden email]>
Cc: [hidden email]
Fecha: martes, 16 de noviembre de 2010, 01:22 am

On Mon, Nov 15, 2010 at 10:28 PM, Octavio <octaviomaiden@...> wrote:
>
> Hello
>
> I have migrate a application from a server with postgres 8.1 to another with 8.3
> in the new server there are some page that doesn't work because seems to want to compare different data type
>
> does anyone know if it is a version issue or there is something Im missing?

Starting in 8.3 many automatic conversions from one type to another
were removed because it wasn't always obvious if this was the right
thing to do.  So, you might need to either change some column types,
or cast them explicitly to the type you want to be used for
comparison.

 
Reply | Threaded
Open this post in threaded view
|

Re: Pre-Allocate tablespace on disk

Florian Weimer-2
In reply to this post by Chris Ruprecht
* Chris Ruprecht:

> Other database that I have worked with before and that I'm still
> working with, allow you to pre-allocate disk space so you get huge
> chunks of contiguous space at one, which has major impacts on
> database performance.

PostgreSQL's write patterns do not trigger significant fragmentation
with most file systems, even when other database systems (Oracle
Berkeley DB comes to my mind) would create heavily fragmented files.

--
Florian Weimer                <[hidden email]>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

--
Sent via pgsql-admin mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
Reply | Threaded
Open this post in threaded view
|

Re: Pre-Allocate tablespace on disk

Steve Francis
Not only that, but if your "disk" is really a bunch of disks behind a hardware RAID controller, or SAN, or iSCSI or NFS server - what is contiguous for the OS may not be contiguous on the physical drives (and vice versa).
Which is why performance can (but may not) improve if you disable linux kernel IO schedulers that attempt to optimize blocks to be sequential.



On 11/19/2010 5:01 AM, Florian Weimer wrote:
* Chris Ruprecht:

Other database that I have worked with before and that I'm still
working with, allow you to pre-allocate disk space so you get huge
chunks of contiguous space at one, which has major impacts on
database performance.
PostgreSQL's write patterns do not trigger significant fragmentation
with most file systems, even when other database systems (Oracle
Berkeley DB comes to my mind) would create heavily fragmented files.



--
 Steve Francis
LogicMonitor LLC



[hidden email]
www.logicmonitor.com

Ph: 1 888 41 LOGIC x500
Ph: 1 805 698 0770

 

Reply | Threaded
Open this post in threaded view
|

Re: Pre-Allocate tablespace on disk

Dimitri Fontaine-7
In reply to this post by Chris Ruprecht
Chris Ruprecht <[hidden email]> writes:
> I was wondering if there is a way to pre-allocate tablespace on disk
> before adding data and indexes.

As others have already said, it's not clear there's a point in doing
that with PostgreSQL. Now, you can achieve it and see for yourself by
relying on OS tools, such as LVM of ZFS, or some other tricks.

For example, create a big enough file, loop mount it as a device, and
create your tablespace in there. Of course what to do exactly when the
file is full is left as an exercise to the reader. Bonus points for any
solution not involving downtime.

Baseline: it seems that PostgreSQL works differently enough from your
usual software than you may want to reconsider your habits :)

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

--
Sent via pgsql-admin mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin