Skip to content

Automatic Parameter Tuning for Big Data Pipelines

Tuning big data frameworks is a very important task to get the best performance for a given application. However, these frameworks are rarely used individually, they generally constitute a pipeline, each having a different role. This makes tuning big data pipelines an important yet difficult task given the size of the search space. Moreover, we have to consider the interaction between these frameworks when tuning the configuration parameters of the big data pipeline. A trade-off is then required to achieve the best end-to-end performance.

Machine learning-based methods have shown great success in automatic tuning systems, but they rely on a large number of high-quality learning examples that are rather difficult to obtain. In this context, we propose to use a deep reinforcement learning algorithm, namely Twin Delayed Deep Deterministic Policy Gradient, TD3, to tune a fraud detection big data pipeline.

Houssem Sagaama, Nourchene Ben Slimane, Maher Marwani, Sabri Skhiri, Automatic Parameter Tuning for Big Data Pipelines, In Proc. of The 26th IEEE Symposium on Computers and Communications (ISCC 2021), September 2021.

Click here to access the paper.

Releated Posts

Development & Evaluation of Automated Tumour Monitoring by Image Registration Based on 3D (PET/CT) Images

Tumor tracking in PET/CT is essential for monitoring cancer progression and guiding treatment strategies. Traditionally, nuclear physicians manually track tumors, focusing on the five largest ones (PERCIST criteria), which is both time-consuming and imprecise. Automated tumor tracking can allow matching of the numerous metastatic lesions across scans, enhancing tumor change monitoring.
Read More

Insights from Data & AI Tech Summit Warsaw 2025

11 editions later, one of the biggest technological conferences in Central Europe changed its name to reflect the latest technological advancements. The BIG DATA TECHNOLOGY WARSAW SUMMIT became the DATA & AI WARSAW TECH SUMMIT, and the conference provided a rich platform for gaining fresh perspectives on data and AI. Our CTO, Sabri Skhiri, was present to gather the insights. Here’s a rundown of the key trends, keynotes and talks that took place.
Read More