How do you measure and improve database performance when you don’t have a quarter of a billion dollars to spend on hardware?
This isn’t an exaggeration. The highest-performing published TPC-C benchmark, Alibaba Cloud PolarDB Limitless, achieved over two billion transactions per minute using 2,340 database nodes, with a total system cost of CNY 1.63 billion, approximately US$225–250 million.
Most database engineers don’t have access to anything approaching that infrastructure. Nor should they need it to investigate and improve database performance.
At HammerDB, we’ve been working with MariaDB’s engineers to improve performance across both CPU-intensive and I/O-intensive workloads. The latest independent testing by McKnight Consulting Group demonstrates the progress, with MariaDB Enterprise Server 12.3 achieving 56% higher OLTP throughput than 11.8, reaching 2.65 million New Orders Per Minute on Google Cloud.
Understanding these results starts with understanding what we’re measuring.
It’s a Hardware Benchmark, Folks
We have often quoted Kevin Closson’s observation about audited TPC-C results from his OakTable-era blog in 2007:
“It’s a hardware benchmark folks.”
Kevin was discussing the performance of commercial databases including Oracle, SQL Server and DB2, observing that their highly optimised TPC-C implementations achieved broadly comparable transaction throughput per processor core.
At that level, the benchmark increasingly demonstrated the capabilities of the underlying hardware rather than differences in database engine performance.
If a database engine can efficiently use the available CPU resources, achieving higher throughput becomes a question of providing sufficient compute, memory and storage capacity. And if you have $250 million to spend, you can demonstrate just how far that scales. For the rest of us, the challenge is improving database performance with the hardware we have.
But what happens when a database stops scaling while substantial CPU capacity remains unused?
That is where database engineering comes in.
CPU-Intensive and I/O-Intensive Benchmarking
HammerDB allows us to use the same TPROC-C transactional workload to investigate different components of database performance.
With CPU-intensive benchmarking, we focus on how efficiently the database engine processes transactions, uses available processors and scales with increasing concurrency.
We minimise the influence of storage so that we can investigate transaction processing, locking, memory management and internal contention.
With I/O-intensive benchmarking, we measure the database working with its storage infrastructure. Storage latency, transaction logging, read and write performance, caching and I/O concurrency all influence the result.
For our I/O-intensive HammerDB testing, we enable the relevant database durability settings along with the HammerDB Use All Warehouses (UAW) option. UAW distributes transaction activity across the warehouse population instead of concentrating it on a small working set, increasing physical database I/O.
For CPU-intensive testing, we configure the database to minimise storage waits, including relaxing synchronous commit requirements where appropriate.
These are deliberate choices. We configure the same transactional workload to stress different components of the database system.
If storage cannot sustain the transaction rate, improvements to CPU efficiency may not appear in the measured throughput. Conversely, adding faster storage may achieve little if the database engine cannot efficiently use the available processor cores.
To improve performance, we first need to establish what is limiting it.
MariaDB: Performance Engineering
Our collaboration with MariaDB’s engineers has focused on improving performance across both CPU-intensive and I/O-intensive workloads. HammerDB testing has helped identify opportunities for optimisation and measure the effects of changes to the database engine.
From MariaDB 10.6 to 11.8, performance testing demonstrated approximately 150% higher OLTP throughput.
Following the 11.8 release, our testing identified further opportunities, and we anticipated another 50% improvement was possible. MariaDB’s engineers exceeded that expectation.
The latest independent testing by McKnight Consulting Group compared MariaDB Enterprise Server 11.8.3-1 and 12.3.3 on the same Google Cloud c4d-highmem-192-lssd instance, powered by a 96-core AMD EPYC processor.
Using HammerDB 6.0 TPROC-C with 768 warehouses, the results were:
| Metric | MariaDB 11.8 | MariaDB 12.3 |
|---|---|---|
| Peak New Orders Per Minute | 1,702,758 | 2,653,370 |
| Virtual users at peak | 88 | 152 |
| New Order p99 latency | 6.54 ms | 3.60 ms |
| CPU utilisation at peak | ~40% | ~70% |
The 56% throughput improvement is accompanied by considerably better CPU utilisation and lower transaction latency.
MariaDB 11.8 reached its peak throughput with substantial processing capacity still available. MariaDB 12.3 continued scaling to 152 virtual users, making much better use of the available processors.
These improvements came from database engineering, including changes to index locking, buffer-pool synchronisation and internal contention.
The published benchmark used a CPU-intensive configuration to measure these improvements without storage performance dominating the result.
This establishes what the database engine can achieve with the available processors. With greater storage capacity and I/O performance to support the increased transaction rate, those CPU-side improvements can translate into higher throughput across the complete database system, including fully durable workloads.
In other words, improving the database engine raises the performance potential of the system. Providing sufficient storage infrastructure allows more of that potential to be realised.
And because MariaDB is open source, anyone can examine the code changes and engineering discussions behind these improvements.
Open Source Databases Are Getting Faster
The progress is not limited to MariaDB.
Across the open-source database ecosystem, expert engineers are continuing to improve performance by addressing the internal limits of database engines and making better use of modern hardware.
PostgreSQL is another example, with ongoing work on concurrency, memory management and storage performance. Projects such as OrioleDB are exploring new approaches to the PostgreSQL storage engine, including buffer management, logging and transaction processing resulting in a 2.74x increase in performance when measured with HammerDB.
These are substantial engineering challenges, and the improvements come from understanding exactly where performance is limited.
The MariaDB results demonstrate what focused engineering can achieve. HammerDB helped identify opportunities for improvement, MariaDB’s engineers delivered the changes, and independent testing measured a further 56% gain in throughput.
As processors and storage systems continue to advance, there will be more opportunities to improve how databases use them.
And because these databases are open source, the engineering work is visible. We can examine the changes, understand how the improvements were achieved and build on them. That’s the work we want HammerDB to help support.
References
- MariaDB Enterprise Server 12.3 Performance Analysis
- McKnight Consulting Group Benchmark
- HammerDB Documentation: Use All Warehouses
- Kevin Closson: Audited TPC-C Proves SQL Server is Better Than Oracle
- TPC-C: Alibaba Cloud PolarDB Limitless
- What Is the Fastest Database? Open-Source Benchmarking with HammerDB
- OrioleDB Documentation