Lately, I am spending quite some time verifying Open-Cannel SSD hardware prototypes in the context of LightNVM. A great deal of these tests is about benchmarking performance (iops) and latency, for which I use fio. Fio is a great tool to test specific I/O patterns. However, when wanting to benchmark a range of I/O configurations in different setups, it is very convenient to automatize the process. In this post, I present a simple python script that allows to automatize fio tests as a function of block size, number of jobs, I/O depth, etc. and stores the test results in csv format. Also, in order to account for variances, the script supports replicating each fio test a configurable number of times, being the average of these the one being stored.

As mentioned, the script is very simple, which makes it easy to adapt. The base script allows to cover a wide range of if configurations, eliminates the environment variances by executing each configuration several times, and stores the average result for each configuration in csv format. The different options supported by fio can be found here (which is an html version of the fio manual: $ man fio). A couple of options worth emphasizing are:

• –minimal: fio results will be printed in a semicolon-delimited format. This makes it simpler to script the results.
• –time_based –runtime=X: In order to also minimize workload variances (e.g., caching, I/O size) we make sure that each test executes a predetermined number of seconds represented by X.
• –group_reporting: When executing several jobs (threads) fio will report information for each one separately. This option makes the output to be grouped, making it easier to compare different I/O patters in different setups.

This is the current version of the script. The updated version of the python script can be found in my GitHub (here). In the future, I intend to complete the script with parameters and new fio options. I also plan to add gnuplot support in order to automatically generate graphics from the csv file.

import os
import time
import subprocess

device = 'nvme0n1'

fio_size="1G" # size in fio
fio_runtime="5" # runtime in fio for time_based tests

# fio --minimal hardcoded positions
fio_iops_pos=7
fio_slat_pos_start=9
fio_clat_pos_start=13
fio_lat_pos_start=37

kernel_version = os.uname()[2]
columns="iotype;bs;njobs;iodepth;iops;slatmin;slatmax;slatavg;clatmin;clatmax;clatavg;latmin;latmax;latavg"

# Eliminate noise by running each test n times and calculating average.
n_iterations=3

f = open(kernel_version + time.strftime("%H%M%S") + "-fio.csv", "w+")
f.write(columns+"\n")

for run in ('write', 'randwrite', 'read', 'randread'):
  for blocksize in ('512', '1k', '4k', '512k'):
     for numjobs in (1, 32, 64):
        for iodepth in (1, 8, 32, 64, 128):
           fio_type_offset = 0
           iops = 0.0
           slat = [0.0 for i in range(3)]
           clat = [0.0 for i in range(3)]
           lat = [0.0 for i in range(3)]

           result = "" + str(run) + ";" + str(blocksize) + ";" + str(numjobs) + ";" + str(iodepth) + ";"
           command = "sudo fio --minimal -name=temp-fio --bs="+str(blocksize)+" --ioengine=libaio --iodepth="+str(iodepth)+" --size="+fio_size+" --direct=1 --rw="+str(run)+" --filename=/dev/"+str(device)+" --numjobs="+str(numjobs)+" --time_based --runtime="+fio_runtime+" --group_reporting"
           print (command)

           for i in range (0, n_iterations):
             os.system("sleep 2") #Give time to finish inflight IOs
             output = subprocess.check_output(command, shell=True)
             if "write" in run:
               fio_type_offset=41

             # fio is called with --group_reporting. This means that all
             # statistics are group for different jobs.

             # iops
             iops = iops + float(output.split(";")[fio_type_offset + fio_iops_pos])

             # slat
             for j in range (0, 3):
               slat[j] = slat[j] + float(output.split(";")[fio_type_offset+fio_slat_pos_start+j])
             # clat
             for j in range (0, 3):
               clat[j] = clat[j] + float(output.split(";")[fio_type_offset+fio_clat_pos_start+j])
             # lat
             for j in range (0, 3):
               lat[j] = lat[j] + float(output.split(";")[fio_type_offset+fio_lat_pos_start+j])

           # iops
           result = result+str(iops / n_iterations)
           # slat
           for i in range (0, 3):
             result = result+";"+str(slat[i] / n_iterations)
           # clat
           for i in range (0, 3):
             result = result+";"+str(clat[i] / n_iterations)
           # lat
           for i in range (0, 3):
             result = result+";"+str(lat[i] / n_iterations)

           print (result)
           f.write(result+"\n")
           f.flush()

f.closed

Enjoy! 🙂

Javier.