Cloud-Native-Bench: an Extensible Benchmarking Framework to Streamline Cloud Performance Tests

The shift to the cloud among organizations has surged enormously over the last decade. As a result, an overwhelming amount of new cloud-based technologies emerged, making it increasingly more challenging to compare the different technologies and to identify the ideal technology that aligns best with a specific use case. The performance and resource usage of a system or software technology can be assessed by running benchmarks. Performing benchmarks has proven to be a time-consuming and error-prone task, especially when executing multiple consecutive tests on the same system. This sparked curiosity in exploring the feasibility of automating this manual benchmarking process. This paper proposes Cloud-NativeBench (CNB), a novel open source benchmarking framework implemented as a Kubernetes operator that fully automates the benchmarking cycle. The entire process, including benchmark deployment, the consecutive execution of benchmarks in a queue, results collection, and statistical data analysis, is fully automated. The framework is designed to be extensible without the need to adapt the operator itself, enabling users to develop fine-tuned custom benchmarks according to their specific use cases. A detailed evaluation shows the ease-of-use of Cloud-Native-Bench and how it streamlines the process of running benchmarks in cloud-native environments. Experiments show the importance of running benchmarks on cloud technologies. For example, employing a different web server technology can increase the mean throughput by 252%.