Investigating HPC Job Resource Requests and Job Efficiency Reporting
Résumé
High Performance Computing (HPC) systems are ever-evolving, increasing in complexity and heterogeneity, while providing high computing power to scientific and data analysis applications. They are attracting a broad range of users that execute applications from computational chemistry, physics, digital humanities, life sciences, and machine learning, to name a few. Not all users possess the knowledge and expertise to harness the full performance capabilities of complex HPC systems. These users tend to submit inaccurate resource requests for their jobs, especially regarding time limits. This leads to inefficient scheduling of jobs, increased wait times for other users, overall inefficient system utilization, and wasted computing resources and energy, ultimately slowing down scientific discovery. This situation motivates the analysis of the accuracy of resource requests and its reporting to users to raise awareness about job efficiency, improve future job submissions, and reduce job wait times. Existing analyses of job wait times often neglect the connection between the causes of job wait times that can range from user-given and Quality of Service (QoS) limits, to unavailable resources, and approaches of reducing wait times that can be reported to users in an understandable way. In this work, we analyze almost 350’000 jobs collected over 2 months on a local university HPC cluster. Our analysis placed an emphasis on time limit accuracy, QoS characteristics, and reasons for long wait times, as well as how to engage users to improve resource requests. This work shows the importance of analyzing reasons for wait times, approaches to reduce wait times, and motivates further research into supporting users to improve resource requests and system utilization, and to reduce avoidable resource waste.