Benchmarking

  • Austin C Bullock

Compared with plain Fortran

benchmarks/benchmark-compare.f90 times the library's common operations against plain Fortran that does the same work, so you can see what the library costs. Each operation checks that it got the values back exactly, and each time is the best of three runs.

The comparisons are:

Operation Library Plain Fortran
Write a CSV file to_file formatted write, a row at a time
Read a CSV file from_file list-directed read, a row at a time
Write and read a raw binary file to_file, from_file (.dat) unformatted stream write, read
Write and read a NumPy file to_file, from_file (.npy, .npz) unformatted stream write, read
Numbers to text cast into a String array internal write into a character array
Text to numbers cast from a String array internal read
Split a CSV file into cells String%read_file read each line, copy the text between commas
Build a string String%push a buffer that doubles when full

The text operations use 1,000,000 real64 values, as 20,000 rows of 50, written with 17 digits after the point (es25.17e3), as the library writes them. The binary operations use 225 real64 values (256 MiB), because they are much faster. The plain versions have no error handling, so a ratio near 1 means the library's checks and messages cost little.

Results

Run on: - an Intel Xeon w5-3435X (16 cores, 32 threads, 45 MiB L3 cache), using one thread; - 118 GiB of memory; - Fedora Linux 46 under WSL2 (kernel 6.18) on Windows, with the files on WSL2's ext4 virtual disk.

The files fit in the page cache, so the times are the CPU and memory work of each operation, not the disk's speed.

The compilers were gfortran 16.2.1, ifx 2026.2.0 and flang 23.1.5, each with -O2 through fpm 0.13 (fpm run benchmark-compare --flag -O2).

The ratio of the library's time to plain Fortran's, in version 2.2 and in 2.3 (lower is better, and 1.0 is the same speed):

Operation gfortran 2.2 2.3 ifx 2.2 2.3 flang 2.2 2.3
Write a CSV file 5.1 1.2 5.7 2.0 4.4 1.7
Read a CSV file 2.8 1.9 3.6 1.5 2.1 1.3
Write a raw binary file 5.1 1.0 1.0 1.0 1.2 0.8
Read a raw binary file 2.9 1.0 1.0 1.0 1.0 1.0
Write a .npy file 5.6 1.0 1.0 1.2 0.9 0.8
Read a .npy file 2.7 0.6 0.7 0.7 0.8 0.8
Write a .npz file 12.2 2.0 7.8 3.0 3.8 1.6
Read a .npz file 2.6 0.6 0.7 0.7 0.8 0.8
Numbers to text 2.4 1.2 3.4 1.6 3.4 1.7
Text to numbers 1.2 1.2 2.6 1.7 1.2 1.2
Split a CSV file into cells 1.2 1.0 4.3 4.4 7.3 4.7
Build a string 0.8 0.8 0.9 1.0 0.9 0.6

The 2.3 times, in milliseconds, as library / plain:

Operation gfortran ifx flang
Write a CSV file 370 / 302 386 / 197 413 / 244
Read a CSV file 485 / 255 413 / 272 440 / 342
Write a raw binary file 107 / 103 83 / 84 208 / 259
Read a raw binary file 154 / 152 143 / 141 288 / 278
Write a .npy file 106 / 103 100 / 84 213 / 259
Read a .npy file 94 / 152 100 / 141 224 / 278
Write a .npz file 208 / 103 249 / 84 403 / 259
Read a .npz file 98 / 152 100 / 141 223 / 278
Numbers to text 721 / 612 505 / 315 478 / 279
Text to numbers 596 / 497 501 / 304 531 / 451
Split a CSV file into cells 65 / 64 106 / 24 78 / 17
Build a string 14 / 18 17 / 18 9 / 16

The two versions were timed one after the other on the same machine, with nothing else running. Times vary by about 10% from run to run, so ratios within 0.1–0.2 of each other are the same.

What the numbers say: - Binary and NumPy files are as fast as plain stream I/O on all three compilers. In 2.2, gfortran took 3–5× as long, because it moves an array inside select rank one element at a time (#93). Reading is sometimes faster than the plain version, which reads into an array it allocated beforehand. - .npz writes cost 1.6–3× a plain write, for the CRC-32 that a ZIP archive needs, which is now computed eight bytes at a time (#94). Reading an array from an archive is as fast as reading a .npy file, because the CRC isn't checked on read. - Writing numbers as text costs 1.2–2× plain formatted output, for to_file and for cast (#91). Each number still goes to its own String, which plain Fortran doesn't need. - Reading a CSV file costs 1.3–1.9× a list-directed read, which reports neither where a bad value is nor reads empty cells as NaN (#92). - read_file matches splitting lines by hand with gfortran, and takes about 4.5× as long with ifx and flang. Most of that is allocating a String for each cell, against the plain version's fixed-length character array (#95). read_file also handles quoted cells, CR LF line endings and rows of different lengths, which the plain version doesn't. - String%push matches a buffer you grow yourself.

Running it

fpm run benchmark-compare --flag -O2                      # with gfortran
fpm run benchmark-compare --compiler ifx --flag -O2
fpm run benchmark-compare --flag -O2 -- 0.1               # a tenth of the data, for a quick run

It prints the tables above in Markdown, with the compiler and its options, and deletes its files at the end. It needs under 1 GiB of memory at full size. CI runs it at a quarter of the size with gfortran, and puts the tables in the job summary. Hosted machines vary too much for those times to be compared run to run, so they never fail the job, but a large change would show.

Big data I/O

The IO-Fortran-Library is capable of reading and writing very large text files with efficiency, even those whose data size exceeds the 2,147,483,647 byte upper limit of the 32-bit signed integer. benchmarks/benchmark-array.f90 (fpm run benchmark-array) times the major internal and external text I/O routines of the IO-Fortran-Library:

program main
  use, intrinsic :: iso_fortran_env, only: int64, rk=>real32, dp=>real64, compiler_version, compiler_options
  use io_fortran_lib, only: String, cast, str, LF, operator(+)
  implicit none (type, external)

  type(String)              :: csv
  type(String), allocatable :: cells(:,:)

  integer(int64) :: t1, t2
  real(dp)       :: wall_time, rate

  integer,  parameter   :: n = 15000
  real(rk), allocatable :: x(:,:), y(:,:)

  allocate( x(n,n), cells(n,n) ); call random_number(x); x = x - 0.5_rk
  write(*,"(a)")  "Compiler version: " + compiler_version()
  write(*,"(a)")  "Compiler options: " + compiler_options() + LF

  call system_clock(t1)
  call cast(x, into=cells, fmt="z")
  call system_clock(t2, count_rate=rate); wall_time = real(t2-t1,dp)/rate

  write(*,"(a)")  "Wall time for cast: " + str(wall_time, fmt="f", decimals=3) + " s"
  write(*,"(a)")  "Number of string conversions/second: " + str(nint(size(x)/wall_time)) + LF

  call system_clock(t1)
  call csv%write_file(cells, file="bigx.csv")
  call system_clock(t2, count_rate=rate); wall_time = real(t2-t1,dp)/rate

  write(*,"(a)")  "Wall time for write_file: " + str(wall_time, fmt="f", decimals=3) + " s"
  write(*,"(a)")  "Estimated file size: " + str(csv%len64()/1e9, fmt="f", decimals=6) + " GB" + LF

  call system_clock(t1)
  call csv%read_file("bigx.csv", cell_array=cells)
  call system_clock(t2, count_rate=rate); wall_time = real(t2-t1,dp)/rate

  write(*,"(a)")  "Wall time for read_file: " + str(wall_time, fmt="f", decimals=3) + " s" + LF

  call csv%empty(); allocate( y(n,n) )

  call system_clock(t1)
  call cast(cells, into=y, fmt="z")
  call system_clock(t2, count_rate=rate); wall_time = real(t2-t1,dp)/rate

  write(*,"(a)")   "Wall time for cast: " + str(wall_time, fmt="f", decimals=3) + " s"
  write(*,"(a)")   "Number of string casts/second: " + str(nint(size(x)/wall_time))
  write(*,"(a,l)") "Data is exact match: ", all(x == y)
end program main

Here, we populate an n-by-n single-precision array x with samples from the uniform distribution on [-0.5, 0.5) and convert each to a hexadecimal string to populate a cell array, write the cell array to a text file "bigx.csv", read the file back into the program to re-populate the cell array, then finally cast the cell data into y and compare with x to observe an exact match. For n = 15000, the resulting csv file size is 2.47 GB.

Note

With the Intel Fortran compiler ifx, we must specify -heap-arrays 0 to avoid a segmentation fault when reading a file of this size, as noted in compiler-dependent behavior.

For a more extreme example, benchmarks/benchmark-ints.f90 (fpm run benchmark-ints) writes every 32-bit integer, in decimal and as a hexadecimal string, to a text file int32.csv:

program main
  use, intrinsic :: iso_fortran_env, only: i64=>int64, dp=>real64
  use io_fortran_lib, only: String, str, operator(+)
  implicit none (type, external)

  integer, parameter :: largest = huge(0), smallest = -largest - 1, nsteps = 128
  integer, parameter :: step_size = int( (int(largest,i64) - int(smallest,i64) )/int(nsteps,i64))

  type(String) :: csv
  type(String), allocatable, target :: cells(:,:)
  type(String), pointer, contiguous :: cells_p(:,:)

  integer :: lower, upper, i, j

  integer(i64) :: t1, t2, total_length
  real(dp)     :: wall_time, rate

  write(*,"(a)") "Writing integers from " + str(smallest) + " to " + str(largest)

  allocate( cells(step_size, 2) )
  total_length = 0_i64
  call system_clock(t1)

  do j = 1, nsteps
    lower = smallest + (j-1)*step_size
    upper = lower + step_size - 1

    cells_p(lower:upper,1:2) => cells(:,:)

    do concurrent ( i = lower:upper )
      cells_p(i,1) = String(i, fmt="i")
      cells_p(i,2) = String(i, fmt="z")
    end do

    call csv%write_file(cells_p, file="int32.csv", append=.true.)
    total_length = total_length + csv%len64()

    write(*,"(a)")  "File length: " + str(total_length/1e9, fmt="f", decimals=3) + " GB in cycle " + str(j)
  end do

  lower = upper + 1; upper = largest
  nullify(cells_p); deallocate(cells); allocate( cells(lower:upper,2) )

  do concurrent ( i = lower:upper )
    cells(i,1) = String(i, fmt="i")
    cells(i,2) = String(i, fmt="z")
  end do

  call csv%write_file(cells, file="int32.csv", append=.true.)
  total_length = total_length + csv%len64()

  write(*,"(a)")  "File length: " + str(total_length/1e9, fmt="f", decimals=3) + " GB"

  call system_clock(t2, count_rate=rate); wall_time = real(t2-t1,dp)/rate
  write(*,"(a)")  "Total time for write: " + str(wall_time/60, fmt="f", decimals=3) + " minutes"
end program main

The resulting file size is 94.128 GB.