Saturday, December 4, 2010

Benchmarking EON ZFS NAS performance (Part I)

At some point you may want to know if your EON ZFS storage is performing on par with your hardware's specifications/setup or if it is doing otherwise. Using various (prerequisite) binary kit tools we can identify if network and I/O performance are optimal or working within expectations. We will use the following tools for our testing
Iperf (Mac, Win)
IOMeter [Part II]
In most cases your storage is accessed/utilized across a network using a client PC. So testing will involve a source, destination and the network that connects them. The source will be your client PC and the destination will be your EON ZFS storage. This will help identify what the network is capable of, is it working properly or if it is a bottleneck to your storage.

On the EON ZFS storage (destination), connect 2 terminals using ssh or putty sessions from your client (Win XP in this case). In one terminal, run
iperf -s
In the other terminal, run (substitute your interface name for bge0)
dladm show-link bge0 -s -i 1
On the source or client PC we will run repetitions of below, changing TCP window size
iperf -c IP_of_EON -P 1 -i 1 -w 8k -f m -t 20
The above says, run 1 process thread P=1 for a test duration of t=20secs with interval i=1sec updates and a TCP window size w=8 in kilobytes. The results will be formatted in (m) megabytes.

This will produce an output showing the upper bounds of your network between the source and destination. It may also help identify the optimal TCP window size for your network. The output below will show the different transfer rates with each change of the TCP window size from 8k to 64k, on the client on a gigabit network. So lets begin.
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 8k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.01 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3448 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  25.2 MBytes   211 Mbits/sec
[1912]  1.0- 2.0 sec  24.7 MBytes   207 Mbits/sec
[1912]  2.0- 3.0 sec  25.4 MBytes   213 Mbits/sec
[1912]  3.0- 4.0 sec  24.1 MBytes   202 Mbits/sec
[1912]  4.0- 5.0 sec  23.8 MBytes   200 Mbits/sec
[1912]  5.0- 6.0 sec  24.0 MBytes   201 Mbits/sec
[1912]  6.0- 7.0 sec  25.0 MBytes   210 Mbits/sec
[1912]  7.0- 8.0 sec  25.0 MBytes   210 Mbits/sec
[1912]  8.0- 9.0 sec  25.5 MBytes   214 Mbits/sec
[1912]  9.0-10.0 sec  24.1 MBytes   202 Mbits/sec
[1912] 10.0-11.0 sec  24.8 MBytes   208 Mbits/sec
[1912] 11.0-12.0 sec  25.1 MBytes   211 Mbits/sec
[1912] 12.0-13.0 sec  25.3 MBytes   212 Mbits/sec
[1912] 13.0-14.0 sec  23.1 MBytes   194 Mbits/sec
[1912] 14.0-15.0 sec  23.6 MBytes   198 Mbits/sec
[1912] 15.0-16.0 sec  23.6 MBytes   198 Mbits/sec
[1912] 16.0-17.0 sec  25.4 MBytes   213 Mbits/sec
[1912] 17.0-18.0 sec  25.6 MBytes   215 Mbits/sec
[1912] 18.0-19.0 sec  25.2 MBytes   211 Mbits/sec
[1912] 19.0-20.0 sec  25.0 MBytes   210 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  494 MBytes    207 Mbits/sec
Optional: Run on EON to compare network interface statistics
dladm show-link bge0 -s -i 1
LINK           IPACKETS  RBYTES   IERRORS  OPACKETS    OBYTES      OERRORS
bge0           19447     27628795 0        2432        155854      0
bge0           19089     27123001 0        2382        152546      0
bge0           19638     27886729 0        2450        156898      0
bge0           18652     26493270 0        2326        148962      0
bge0           18454     26205800 0        2301        147362      0
bge0           18561     26358926 0        2315        148258      0
bge0           19268     27372463 0        2404        153954      0
bge0           19379     27533495 0        2418        154850      0
bge0           19687     27977035 0        2457        157346      0
bge0           18681     26527515 0        2330        149218      0
bge0           19155     27208916 0        2390        153058      0
bge0           19419     27593183 0        2423        155170      0
bge0           19542     27757561 0        2438        156130      0
bge0           17840     25347308 0        2226        142562      0
bge0           18148     25776206 0        2264        144994      0
bge0           18353     26072730 0        2290        146658      0
bge0           19657     27919866 0        2452        157106      0
bge0           19756     28065895 0        2465        157858      0
bge0           19390     27542355 0        2419        154914      0
bge0           17889     25407434 0        2236        143362      0
bge0           38        8169     0        1           162         0
With TCP window set to 8K, Max throughput is approx 207Mbps/8bps=25.87MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 16k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.02 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3487 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  40.1 MBytes   336 Mbits/sec
[1912]  1.0- 2.0 sec  31.1 MBytes   261 Mbits/sec
[1912]  2.0- 3.0 sec  36.3 MBytes   305 Mbits/sec
[1912]  3.0- 4.0 sec  43.2 MBytes   363 Mbits/sec
[1912]  4.0- 5.0 sec  43.2 MBytes   363 Mbits/sec
[1912]  5.0- 6.0 sec  41.5 MBytes   348 Mbits/sec
[1912]  6.0- 7.0 sec  41.9 MBytes   352 Mbits/sec
[1912]  7.0- 8.0 sec  29.6 MBytes   249 Mbits/sec
[1912]  8.0- 9.0 sec  33.2 MBytes   278 Mbits/sec
[1912]  9.0-10.0 sec  34.4 MBytes   288 Mbits/sec
[1912] 10.0-11.0 sec  37.5 MBytes   315 Mbits/sec
[1912] 11.0-12.0 sec  41.4 MBytes   348 Mbits/sec
[1912] 12.0-13.0 sec  43.0 MBytes   360 Mbits/sec
[1912] 13.0-14.0 sec  30.9 MBytes   259 Mbits/sec
[1912] 14.0-15.0 sec  39.2 MBytes   329 Mbits/sec
[1912] 15.0-16.0 sec  42.9 MBytes   360 Mbits/sec
[1912] 16.0-17.0 sec  42.9 MBytes   360 Mbits/sec
[1912] 17.0-18.0 sec  40.6 MBytes   340 Mbits/sec
[1912] 18.0-19.0 sec  37.2 MBytes   312 Mbits/sec
[1912] 19.0-20.0 sec  24.7 MBytes   207 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  755 MBytes    317 Mbits/sec
With TCP window set to 16K, Max throughput is approx 317Mbps/8bps=39.63MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 24k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.02 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3497 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  53.4 MBytes   448 Mbits/sec
[1912]  1.0- 2.0 sec  50.2 MBytes   421 Mbits/sec
[1912]  2.0- 3.0 sec  40.8 MBytes   343 Mbits/sec
[1912]  3.0- 4.0 sec  62.5 MBytes   524 Mbits/sec
[1912]  4.0- 5.0 sec  63.9 MBytes   536 Mbits/sec
[1912]  5.0- 6.0 sec  63.4 MBytes   532 Mbits/sec
[1912]  6.0- 7.0 sec  64.4 MBytes   540 Mbits/sec
[1912]  7.0- 8.0 sec  58.7 MBytes   493 Mbits/sec
[1912]  8.0- 9.0 sec  48.0 MBytes   403 Mbits/sec
[1912]  9.0-10.0 sec  62.2 MBytes   522 Mbits/sec
[1912] 10.0-11.0 sec  54.1 MBytes   454 Mbits/sec
[1912] 11.0-12.0 sec  53.9 MBytes   452 Mbits/sec
[1912] 12.0-13.0 sec  56.8 MBytes   477 Mbits/sec
[1912] 13.0-14.0 sec  57.6 MBytes   483 Mbits/sec
[1912] 14.0-15.0 sec  49.1 MBytes   412 Mbits/sec
[1912] 15.0-16.0 sec  65.0 MBytes   545 Mbits/sec
[1912] 16.0-17.0 sec  64.2 MBytes   539 Mbits/sec
[1912] 17.0-18.0 sec  64.6 MBytes   542 Mbits/sec
[1912] 18.0-19.0 sec  64.0 MBytes   537 Mbits/sec
[1912] 19.0-20.0 sec  57.5 MBytes   483 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  1154 MBytes   484 Mbits/sec
With TCP window set to 24K, Max throughput is approx 484Mbps/8bps=60.5MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 32k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.03 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3500 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  77.0 MBytes   646 Mbits/sec
[1912]  1.0- 2.0 sec  78.2 MBytes   656 Mbits/sec
[1912]  2.0- 3.0 sec  48.9 MBytes   410 Mbits/sec
[1912]  3.0- 4.0 sec  76.6 MBytes   642 Mbits/sec
[1912]  4.0- 5.0 sec  75.8 MBytes   636 Mbits/sec
[1912]  5.0- 6.0 sec  79.4 MBytes   666 Mbits/sec
[1912]  6.0- 7.0 sec  78.8 MBytes   661 Mbits/sec
[1912]  7.0- 8.0 sec  79.0 MBytes   663 Mbits/sec
[1912]  8.0- 9.0 sec  66.7 MBytes   559 Mbits/sec
[1912]  9.0-10.0 sec  81.0 MBytes   679 Mbits/sec
[1912] 10.0-11.0 sec  76.8 MBytes   644 Mbits/sec
[1912] 11.0-12.0 sec  79.5 MBytes   667 Mbits/sec
[1912] 12.0-13.0 sec  74.1 MBytes   621 Mbits/sec
[1912] 13.0-14.0 sec  72.9 MBytes   612 Mbits/sec
[1912] 14.0-15.0 sec  70.0 MBytes   587 Mbits/sec
[1912] 15.0-16.0 sec  79.1 MBytes   663 Mbits/sec
[1912] 16.0-17.0 sec  79.6 MBytes   668 Mbits/sec
[1912] 17.0-18.0 sec  79.6 MBytes   668 Mbits/sec
[1912] 18.0-19.0 sec  79.4 MBytes   666 Mbits/sec
[1912] 19.0-20.0 sec  77.7 MBytes   652 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  1510 MBytes   633 Mbits/sec
With TCP window set to 32K, Max throughput is approx 633Mbps/8bps=79.13MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 48k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.05 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3506 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec   101 MBytes   846 Mbits/sec
[1912]  1.0- 2.0 sec   102 MBytes   852 Mbits/sec
[1912]  2.0- 3.0 sec  84.9 MBytes   712 Mbits/sec
[1912]  3.0- 4.0 sec  81.7 MBytes   686 Mbits/sec
[1912]  4.0- 5.0 sec  94.7 MBytes   794 Mbits/sec
[1912]  5.0- 6.0 sec   103 MBytes   860 Mbits/sec
[1912]  6.0- 7.0 sec   102 MBytes   859 Mbits/sec
[1912]  7.0- 8.0 sec   102 MBytes   852 Mbits/sec
[1912]  8.0- 9.0 sec   101 MBytes   850 Mbits/sec
[1912]  9.0-10.0 sec  90.6 MBytes   760 Mbits/sec
[1912] 10.0-11.0 sec  82.6 MBytes   693 Mbits/sec
[1912] 11.0-12.0 sec   101 MBytes   848 Mbits/sec
[1912] 12.0-13.0 sec  85.5 MBytes   717 Mbits/sec
[1912] 13.0-14.0 sec  85.8 MBytes   719 Mbits/sec
[1912] 14.0-15.0 sec  91.5 MBytes   767 Mbits/sec
[1912] 15.0-16.0 sec  91.8 MBytes   770 Mbits/sec
[1912] 16.0-17.0 sec  89.7 MBytes   753 Mbits/sec
[1912] 17.0-18.0 sec   102 MBytes   858 Mbits/sec
[1912] 18.0-19.0 sec  97.9 MBytes   821 Mbits/sec
[1912] 19.0-20.0 sec   100 MBytes   841 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  1891 MBytes   793 Mbits/sec
With TCP window set to 48K, Max throughput is approx 793Mbps/8bps=99.13MB/s
C:\download>iperf -c 10.72.100.148 -P 1 -i 1 -w 64k -f m -t 20
------------------------------------------------------------
Client connecting to 10.72.100.148, TCP port 5001
TCP window size: 0.06 MByte
------------------------------------------------------------
[1912] local 10.72.100.128 port 3538 connected with 10.72.100.148 port 5001
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0- 1.0 sec  98.2 MBytes   823 Mbits/sec
[1912]  1.0- 2.0 sec  66.3 MBytes   556 Mbits/sec
[1912]  2.0- 3.0 sec   101 MBytes   844 Mbits/sec
[1912]  3.0- 4.0 sec  99.8 MBytes   837 Mbits/sec
[1912]  4.0- 5.0 sec  99.5 MBytes   835 Mbits/sec
[1912]  5.0- 6.0 sec  79.7 MBytes   668 Mbits/sec
[1912]  6.0- 7.0 sec  76.6 MBytes   643 Mbits/sec
[1912]  7.0- 8.0 sec  94.1 MBytes   790 Mbits/sec
[1912]  8.0- 9.0 sec   101 MBytes   846 Mbits/sec
[1912]  9.0-10.0 sec   100 MBytes   840 Mbits/sec
[1912] 10.0-11.0 sec  96.7 MBytes   811 Mbits/sec
[1912] 11.0-12.0 sec  96.0 MBytes   805 Mbits/sec
[1912] 12.0-13.0 sec  94.0 MBytes   789 Mbits/sec
[1912] 13.0-14.0 sec  97.2 MBytes   816 Mbits/sec
[1912] 14.0-15.0 sec  98.5 MBytes   827 Mbits/sec
[1912] 15.0-16.0 sec  81.7 MBytes   686 Mbits/sec
[1912] 16.0-17.0 sec  74.6 MBytes   626 Mbits/sec
[1912] 17.0-18.0 sec  79.8 MBytes   669 Mbits/sec
[1912] 18.0-19.0 sec  98.1 MBytes   823 Mbits/sec
[1912] 19.0-20.0 sec  99.7 MBytes   837 Mbits/sec
[ ID] Interval        Transfer      Bandwidth
[1912]  0.0-20.0 sec  1832 MBytes   768 Mbits/sec
With TCP window set to 64K, Max throughput is approx 768Mbps/8bps=96MB/s.

The important things to note from these tests between a client (Win XP) and EON ZFS NAS server on this network are:
  • Possible upper limit near 99.13MBytes/sec (not the theoretical 1000Mbps/125MBytes/sec).
  • The optimal TCP window size for this network is probably 48K (should have tested 56K also).
  • A larger TCP window size of 64K did not provide the highest transfer rate.