Machine Learning for Predicting Job States and CPU Power on a Supercomputer

Efficient resource management in high-performance computing (HPC) is essential for optimizing costs, reducing energy consumption, and improving system productivity. However, job variability and failures introduce uncertainties that complicate scheduling and resource allocation. Accurately predicting job failures and estimating energy consumption can enhance planning and operational efficiency. This study analyzes data from the Simple Linux Utility for Resource Management (SLURM) on the Kabré supercomputer at Costa Rica’s National High Technology Center (CeNAT). After selecting and preprocessing relevant variables, a dataset was created to train a two-stage machine learning model comprising a binary classifier and a regression model. Using 10-fold cross-validation, multiple models were evaluated, with Random Forest emerging as the best performer in both stages. The classification model was assessed using the confusion matrix and ROC curve, while the regression model was evaluated through residual analysis and metrics such as Root Mean Square Error (RMSE) and the Coefficient of Determination (R²). This approach can support users and administrators by improving job scheduling decisions and reducing energy waste in HPC systems.