FreeBSD UFS & fsync

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

FreeBSD UFS & fsync

Luca Ferrari-2
Hi all,
I'm running a virtual machine with FreeBSD 12.2, PostgreSQL 12.5 and
UFS as filesystem.
I was experimenting with fsync = off and pgbench, and I see no
particular difference in tps having fsync enabled or disabled.
Now, the same tiny test on a linux box provides a 10x tps, while on
FreeBSD is a 1% increase.
I'm trying to figure out why, and I suspect there is something related
to how UFS handles writes.

Any particular advice about tuning and parameters that can be
affecting the "no difference" with fsync turned off?


% sudo tunefs -p /dev/gpt/DATA
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                    disabled
tunefs: MAC multilabel: (-l)                                 disabled
tunefs: soft updates: (-n)                                    disabled
tunefs: soft update journaling: (-j)                      disabled
tunefs: gjournal: (-J)                                         disabled
tunefs: trim: (-t)                                                enabled
tunefs: maximum blocks per file in a cylinder group: (-e)  8192
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            6408
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)                                 DATA


Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD UFS & fsync

Thomas Munro-5
On Tue, Feb 23, 2021 at 5:49 AM Luca Ferrari <[hidden email]> wrote:
> I'm running a virtual machine with FreeBSD 12.2, PostgreSQL 12.5 and
> UFS as filesystem.
> I was experimenting with fsync = off and pgbench, and I see no
> particular difference in tps having fsync enabled or disabled.
> Now, the same tiny test on a linux box provides a 10x tps, while on
> FreeBSD is a 1% increase.
> I'm trying to figure out why, and I suspect there is something related
> to how UFS handles writes.

Do you have WCE enabled?  In that case, modern Linux file systems
would do a synchronous SYNCHRONIZE CACHE for our WAL fdatasync(), but
FreeBSD UFS wouldn't as far as I know.  It does know how to do that
(there's a BIO_FLUSH operation, also used by ZFS), but as far as I can
see UFS uses it just for its own file system meta-data crash safety
currently (see softdep_synchronize()).  (There is also no FUA flag for
O_[D]SYNC writes, an even more modern invention.)


Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD UFS & fsync

Luca Ferrari-2
On Mon, Feb 22, 2021 at 10:38 PM Thomas Munro <[hidden email]> wrote:
> Do you have WCE enabled?  In that case, modern Linux file systems
> would do a synchronous SYNCHRONIZE CACHE for our WAL fdatasync(), but
> FreeBSD UFS wouldn't as far as I know.  It does know how to do that
> (there's a BIO_FLUSH operation, also used by ZFS), but as far as I can
> see UFS uses it just for its own file system meta-data crash safety
> currently (see softdep_synchronize()).  (There is also no FUA flag for
> O_[D]SYNC writes, an even more modern invention.)

Apparently no WCE, but I could be looking at the wrong piece:

% sysctl kern.cam.ada | grep write_cache
kern.cam.ada.2.write_cache: -1
kern.cam.ada.1.write_cache: -1
kern.cam.ada.0.write_cache: -1
kern.cam.ada.write_cache: -1

I'm using sata disks, not scsi. Assuming I'm not looking at the wrong
parameter, I wil attach a scsi disk to do the same test and see if
something changes.
Or if you have any other suggestion about what to inspect, please advice.

Thanks,
Luca


Reply | Threaded
Open this post in threaded view
|

Re: FreeBSD UFS & fsync

Luca Ferrari-2
On Tue, Feb 23, 2021 at 8:46 AM Luca Ferrari <[hidden email]> wrote:
> I'm using sata disks, not scsi. Assuming I'm not looking at the wrong
> parameter, I wil attach a scsi disk to do the same test and see if
> something changes.

I've tested the same version of PostgreSQL, same benchmark, on a scsi
disk. However, turning off fsync does not provide any increment at all
(something that spans in less than 1% tps).
I've checked and I have WCE enabled on such disk, but apparently I
cannot modify (I suspect this is due to the virtualization of the
disk):

# echo "WCE: 0" | camcontrol modepage da0 -m 0x08 -e
camcontrol: error sending mode select command
# camcontrol modepage da0 -m 0x08 | grep WCE
WCE:  1

and the filesystem has everything disabled:

# tunefs -p da0p1
tunefs: Can't stat da0p1: No such file or directory
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 disabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            6408
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)

I think I will not be able to test in a virtual environment, unless
I'm missing something.

Thanks,
Luca