A benchmark is a standard or point of reference against which things may be compared or assessed, and the process of benchmarking is to evaluate or check (something) by comparison with a standard. Benchmarking in the classical computing world typically involves running some code on one or more systems, measuring some property of its execution profile, and using that as a point of comparison, usually for some sort of “my system is better than yours” statement.
As a reference, Table 1 summarizes some of the more well-known benchmarking efforts, and websites that document their results.
| Name | Year | References | Area | Architecture | Type | Code | Metric |
|---|---|---|---|---|---|---|---|
| Gibson Mix | 1959 | Gibson Mix | general | single core | technical | none | instr/s |
| Whetstone | 1972 | Whetstone | numeric | single core | competitive | synthetic | Whetstone instr/s |
| Linpack | 1976 | Linpack | linear algebra | parallel | competitive | aplication | flops/s |
| Dhrystone | 1984 | Dhrystone | non-numeric | single core | competitive | synthetic | Dhrystone instr/s |
| TPC | 1988 | TPC | database | general | competitive | app suit | varied |
| SPEC | 1988 | SPEC | general | any | technical | app suite | varied |
| SLALOM | 1990 | SLALOM | finite element | any | competitive | kernel | problem size |
| NASA | 1991 | NASA | CFD | parallel | competitive | suite | varied |
| HPL | 1993 | HPL | linear algebra | parallel | competitive | application | flops/s |
| NASA | 1994 | NASA | fluid dynamics | parallel | technical | suite | flops/s |
| Cinebench | 2000 | Cinebench | Rendering | general | competitive | application | composite |
| HPC Challenge | 2004 | HPC Challenge | memory-intensive | general | technical | kernel suite | varied rates |
| BFS | 2010 | BFS | graphs | parallel | competitive | kernel | TEP/s |
| Firehose | 2015 | Firehose | streaming | piplined | technical | kernel suite | datums/s |
| HPCG | 2017 | HPCG | linear algebra | parallel | competitive | suite | composite |
| IO500 | 2018 | IO500 | I/O | parallel | competitive | suite | composite |
| HPL-MxP | 2019 | HPL-MxP | linear algebra | parallel | competitive | kernel | flops/s |
| MLPerf | 2019 | MLPerf | machine learning | general | technical | suite | varied |
The landscape of computing, however, is changing. In conventional computing we see the explosive arrival of heterogeneous computing, with GPUs becoming a dominant part of the computing landscape. The idea of heterogeneity is going even further, with near-, and even in-, memory and network accelerators becoming practical. New classes of memory and file storage are changing the whole memory hierarchy. What is missing from conventional benchmarking is an attempt to use such information to not just compare, but to understand more deeply exactly why we are getting the numbers we see, where and what in the architecture is most or least responsible for gains, and how do changes in the data set characteristics affect the expected outcomes.
Perhaps even more challenging is the emergence of radically new models of computation, especially quantum. Here we not only don't have well-understood applications to benchmark, we aren't even sure what “performance” is, and how to measure it. While modern classical computers are far more complex than their predecessors of 80 years ago, the fundamentals of how they work haven't changed much. In biological terms, they are of the same species, and thus “performance” means essentially the same thing for any of them, and it is possible to make reasonable comparisons. That is not true of quantum computing; benchmarking such systems is a significantly different beast. There is a belief in at least part of the quantum community that we have, or shortly will, demonstrate quantum advantage (where some combination of quantum and classical computing can outperform classical alone) or quantum supremacy (where a quantum computer can compute something not feasible to compute on a classical computer). In such circumstances, “comparisons” give us little insight into the underlying systems.
This website is an attempt to serve as an aggregation point for any reported performance of any quantum algorithm on any kind of quantum hardware, without any attempt to rank the results. Instead, an attempt is made to capture significantly more data than, for example, in a report for the Top500. Further, this data is then stored in a hierarchical form, shown in Figure 1, so that queries can be made with different goals in mind, and a tailorable query function is available to look at the data in different ways. This is all to better understand the relationship between quantum algorithms, quantum hardware technologies and architectures, and performance.