ZFS benchmark
I have a 64-bit box with a ZFS raidx2 array. It now has 4GB of RAM and I recently purchased another 4GB.
I thought I might do some benchmarks on it before and after I install the additional RAM. I will be using
the benchmarks/bonnie++ port.
The intial tests
In this section I will run two intial tests. One will write to a zfs dataset which has compression enabled,
and the other will be to a dataset which does not enable compresion. The latter will be first:
$ bonnie++ Writing a byte at a time...done Writing intelligently...done Rewriting...done Reading a byte at a time...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP kraken.example.o 8G 137 99 142356 29 72266 16 407 86 167074 20 249.1 6 Latency 44147us 4784ms 3045ms 85628us 191ms 872ms Version 1.96 ------Sequential Create------ --------Random Create-------- kraken.example.org -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 29919 89 +++++ +++ 29088 85 29552 88 +++++ +++ 19775 59 Latency 8709us 61us 100us 17430us 35us 120us 1.96,1.96,kraken.example.org,1,1294512706,8G,,137,99,142356,29,72266,16,407,86, 167074,20,249.1,6,16,,,,,29919,89,+++++,+++,29088,85,29552,88,+++++,+++,19775,59, 44147us,4784ms,3045ms,85628us,191ms,872ms,8709us,61us,100us,17430us,35us,120us
This test writes to the compression-enabled dataset:
$ bonnie++ Writing a byte at a time...done Writing intelligently...done Rewriting...done Reading a byte at a time...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP kraken.example.o 8G 146 99 280903 54 106387 25 422 90 181832 21 268.2 6 Latency 83050us 2199ms 781ms 77343us 164ms 719ms Version 1.96 ------Sequential Create------ --------Random Create-------- kraken.example.org -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 29140 85 +++++ +++ 29447 89 31300 93 +++++ +++ 29590 89 Latency 8942us 59us 80us 17767us 32us 57us 1.96,1.96,kraken.example.org,1,1294511752,8G,,146,99,280903,54,106387,25,422,90, 181832,21,268.2,6,16,,,,,29140,85,+++++,+++,29447,89,31300,93,+++++,+++,29590,89, 83050us,2199ms,781ms,77343us,164ms,719ms,8942us,59us,80us,17767us,32us,57usThe system
This is the dmesg output for this machine:Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-PRERELEASE #1: Tue Nov 30 22:07:59 EST 2010 dan@kraken.example.org:/usr/obj/usr/src/sys/KRAKEN amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) II X4 945 Processor (3010.17-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f42 Family = 10 Model = 4 Stepping = 2 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x802009<SSE3,MON,CX16,POPCNT> AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!> AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT> TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 4101513216 (3911 MB) ACPI APIC Table: <111909 APIC1708> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ACPI Warning: Optional field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20101013/tbfadt-655) ioapic0 <Version 2.1> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <111909 RSDT1708> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of fec10000, 20 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dfe00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci0 pci7: <ACPI PCI bus> on pcib1 em0: <Intel(R) PRO/1000 Network Connection 7.1.8> port 0xec00-0xec1f mem 0xfbfe0000-0xfbffffff,0xfbf00000-0xfbf7ffff,0xfbfdc000-0xfbfdffff irq 18 at device 0.0 on pci7 em0: Using MSIX interrupts with 3 vectors em0: [ITHREAD] em0: [ITHREAD] em0: [ITHREAD] em0: Ethernet address: 00:1b:21:51:ab:2d pcib2: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0 pci5: <ACPI PCI bus> on pcib2 pcib3: <PCI-PCI bridge> irq 17 at device 0.0 on pci5 pci6: <PCI bus> on pcib3 siis0: <SiI3124 SATA controller> port 0xdc00-0xdc0f mem 0xfbeffc00-0xfbeffc7f,0xfbef0000-0xfbef7fff irq 17 at device 4.0 on pci6 siis0: [ITHREAD] siisch0: <SIIS channel> at channel 0 on siis0 siisch0: [ITHREAD] siisch1: <SIIS channel> at channel 1 on siis0 siisch1: [ITHREAD] siisch2: <SIIS channel> at channel 2 on siis0 siisch2: [ITHREAD] siisch3: <SIIS channel> at channel 3 on siis0 siisch3: [ITHREAD] pcib4: <ACPI PCI-PCI bridge> irq 18 at device 6.0 on pci0 pci4: <ACPI PCI bus> on pcib4 re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xc800-0xc8ff mem 0xfbdff000-0xfbdfffff irq 18 at device 0.0 on pci4 re0: Using 1 MSI messages re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: <MII bus> on re0 rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: e0:cb:4e:42:f0:ff re0: [FILTER] pcib5: <ACPI PCI-PCI bridge> irq 19 at device 11.0 on pci0 pci2: <ACPI PCI bus> on pcib5 pcib6: <PCI-PCI bridge> irq 19 at device 0.0 on pci2 pci3: <PCI bus> on pcib6 siis1: <SiI3124 SATA controller> port 0xbc00-0xbc0f mem 0xfbcffc00-0xfbcffc7f,0xfbcf0000-0xfbcf7fff irq 19 at device 4.0 on pci3 siis1: [ITHREAD] siisch4: <SIIS channel> at channel 0 on siis1 siisch4: [ITHREAD] siisch5: <SIIS channel> at channel 1 on siis1 siisch5: [ITHREAD] siisch6: <SIIS channel> at channel 2 on siis1 siisch6: [ITHREAD] siisch7: <SIIS channel> at channel 3 on siis1 siisch7: [ITHREAD] ahci0: <ATI IXP700 AHCI SATA controller> port 0x9000-0x9007,0x8000-0x8003,0x7000-0x7007,0x6000-0x6003,0x5000-0x500f mem 0xfb6fe400-0xfb6fe7ff irq 22 at device 17.0 on pci0 ahci0: [ITHREAD] ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich1: [ITHREAD] ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich2: [ITHREAD] ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich3: [ITHREAD] ohci0: <OHCI (generic) USB controller> mem 0xfb6fa000-0xfb6fafff irq 16 at device 18.0 on pci0 ohci0: [ITHREAD] usbus0: <OHCI (generic) USB controller> on ohci0 ohci1: <OHCI (generic) USB controller> mem 0xfb6fb000-0xfb6fbfff irq 16 at device 18.1 on pci0 ohci1: [ITHREAD] usbus1: <OHCI (generic) USB controller> on ohci1 ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfb6fe800-0xfb6fe8ff irq 17 at device 18.2 on pci0 ehci0: [ITHREAD] ehci0: AMD SB600/700 quirk applied usbus2: EHCI version 1.0 usbus2: <EHCI (generic) USB 2.0 controller> on ehci0 ohci2: <OHCI (generic) USB controller> mem 0xfb6fc000-0xfb6fcfff irq 18 at device 19.0 on pci0 ohci2: [ITHREAD] usbus3: <OHCI (generic) USB controller> on ohci2 ohci3: <OHCI (generic) USB controller> mem 0xfb6fd000-0xfb6fdfff irq 18 at device 19.1 on pci0 ohci3: [ITHREAD] usbus4: <OHCI (generic) USB controller> on ohci3 ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfb6fec00-0xfb6fecff irq 19 at device 19.2 on pci0 ehci1: [ITHREAD] ehci1: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5: <EHCI (generic) USB 2.0 controller> on ehci1 pci0: <serial bus, SMBus> at device 20.0 (no driver attached) atapci0: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] isab0: <PCI-ISA bridge> at device 20.3 on pci0 isa0: <ISA bus> on isab0 pcib7: <ACPI PCI-PCI bridge> at device 20.4 on pci0 pci1: <ACPI PCI bus> on pcib7 ahc0: <Adaptec 2944 Ultra SCSI adapter> port 0xa800-0xa8ff mem 0xfb7df000-0xfb7dffff irq 20 at device 5.0 on pci1 ahc0: [ITHREAD] aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs vgapci0: <VGA-compatible display> mem 0xfb800000-0xfbbfffff,0xfb7f0000-0xfb7fffff irq 21 at device 6.0 on pci1 ohci4: <OHCI (generic) USB controller> mem 0xfb6ff000-0xfb6fffff irq 18 at device 20.5 on pci0 ohci4: [ITHREAD] usbus6: <OHCI (generic) USB controller> on ohci4 acpi_button0: <Power Button> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 acpi_hpet1: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 device_attach: acpi_hpet1 attach returned 12 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc87ff,0xc8800-0xc97ff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range acpi_throttle0: <ACPI CPU Throttling> on cpu0 hwpstate0: <Cool`n'Quiet 2.0> on cpu0 Timecounters tick every 1.000 msec (noperiph:siisch0:0:-1:-1): rescan already queued (noperiph:siisch2:0:-1:-1): rescan already queued (noperiph:siisch3:0:-1:-1): rescan already queued (noperiph:siisch4:0:-1:-1): rescan already queued (noperiph:siisch5:0:-1:-1): rescan already queued (noperiph:siisch6:0:-1:-1): rescan already queued (noperiph:siisch7:0:-1:-1): rescan already queued usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 usbus6: 12Mbps Full Speed USB v1.0 ugen0.1: <ATI> at usbus0 uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen1.1: <ATI> at usbus1 uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 ugen2.1: <ATI> at usbus2 uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 ugen3.1: <ATI> at usbus3 uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 ugen4.1: <ATI> at usbus4 uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 ugen5.1: <ATI> at usbus5 uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5 ugen6.1: <ATI> at usbus6 uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6 uhub6: 2 ports with 2 removable, self powered uhub0: 3 ports with 3 removable, self powered uhub1: 3 ports with 3 removable, self powered uhub3: 3 ports with 3 removable, self powered uhub4: 3 ports with 3 removable, self powered uhub2: 6 ports with 6 removable, self powered uhub5: 6 ports with 6 removable, self powered (probe0:ahc0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 (probe0:ahc0:0:0:0): CAM status: SCSI Status Error (probe0:ahc0:0:0:0): SCSI status: Check Condition (probe0:ahc0:0:0:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (probe5:ahc0:0:5:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 (probe5:ahc0:0:5:0): CAM status: SCSI Status Error (probe5:ahc0:0:5:0): SCSI status: Check Condition (probe5:ahc0:0:5:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (probe5:ahc0:0:5:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 (probe5:ahc0:0:5:0): CAM status: SCSI Status Error (probe5:ahc0:0:5:0): SCSI status: Check Condition (probe5:ahc0:0:5:0): SCSI sense: NOT READY asc:3a,0 (Medium not present) ada0 at siisch0 bus 0 scbus0 target 0 lun 0 ada0: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada1 at siisch2 bus 0 scbus2 target 0 lun 0 ada1: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2 at siisch3 bus 0 scbus3 target 0 lun 0 ada2: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3 at siisch4 bus 0 scbus4 target 0 lun 0 ada3: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada4 at siisch5 bus 0 scbus5 target 0 lun 0 ada4: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada5 at siisch6 bus 0 scbus6 target 0 lun 0 ada5: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada6 at siisch7 bus 0 scbus7 target 0 lun 0 ada6: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada7 at ahcich0 bus 0 scbus8 target 0 lun 0 ada7: <ST380815AS 4.AAB> ATA-7 SATA 2.x device ada7: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada7: Command Queueing enabled ada7: 76319MB (156301488 512 byte sectors: 16H 63S/T 16383C) ada8 at ahcich2 bus 0 scbus10 target 0 lun 0 ada8: <WDC WD1600AAJS-75M0A0 02.03E02> ATA-8 SATA 2.x device ada8: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada8: Command Queueing enabled ada8: 152587MB (312500000 512 byte sectors: 16H 63S/T 16383C) sa0 at ahc0 bus 0 scbus12 target 5 lun 0 sa0: <DEC TZ89 (C) DEC 2561> Removable Sequential Access SCSI-2 device sa0: 20.000MB/s transfers (10.000MHz, offset 8, 16bit) SMP: AP CPU #1 Launched! cd0 at ahcich1 bus 0 scbus9 target 0 lun 0SMP: AP CPU #3 Launched! cd0: <TSSTcorp CDDVDW SH-S223C SB01> Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)SMP: AP CPU #2 Launched! cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed ch0 at ahc0 bus 0 scbus12 target 0 lun 0 ch0: <DEC TL800 (C) DEC 0326> Removable Changer SCSI-2 device ch0: 20.000MB/s transfers (10.000MHz, offset 8, 16bit) ch0: 10 slots, 1 drive, 1 picker, 0 portals GEOM_MIRROR: Device mirror/gm0 launched (2/2). GEOM: mirror/gm0s1: geometry does not match label (16h,63s != 255h,63s). Trying to mount root from ufs:/dev/mirror/gm0s1a ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 4 ZFS storage pool version 15The zpools are:
$ zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT storage 12.7T 4.46T 8.23T 35% ONLINE -The status is:
$ zpool status pool: storage state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 raidz2 ONLINE 0 0 0 gpt/disk01-live ONLINE 0 0 0 gpt/disk02-live ONLINE 0 0 0 gpt/disk03-live ONLINE 0 0 0 gpt/disk04-live ONLINE 0 0 0 gpt/disk05-live ONLINE 0 0 0 gpt/disk06-live ONLINE 0 0 0 gpt/disk07-live ONLINE 0 0 0 errors: No known data errorsOther items:
$ cat /boot/loader.conf geom_mirror_load="YES" ahci_load="YES" siis_load="YES" vm.kmem_size=7G $ zpool upgrade This system is currently running ZFS pool version 15.Are there any other configuration settings you'd like to see?
After talking with pjd regarding benchmarks, the following ideas came out:
* benchmarks/filebench
* sysutils/fio
* collect data from ‘sysctl kstat’
* I might also run sysutils/zfs-stats
—
The Man Behind The Curtain