Optimizing priority scheduling in hadoop for resource utilization using quantum particle swarm optimization technique

Abstract

Efficient resource scheduling in Hadoop remains a challenging problem due to the presence of diverse workloads and varying priority requirements in cluster environments. Traditional YARN schedulers such as FIFO, Fair, and Capacity often struggle to simultaneously balance data locality, responsiveness, fairness, and priority handling, which can lead to increased waiting times and inefficient resource utilization. To address these limitations, this study proposes a Quantum-Inspired Priority Scheduler (QIPS) that integrates a Quantum-behaved Particle Swarm Optimization (QPSO) mechanism within the YARN Resource Manager to enhance job–node assignment decisions. The proposed scheduler considers multiple performance criteria, including latency, resource utilization, data locality, and priority awareness, enabling more adaptive and balanced scheduling under dynamic workload conditions. A hybrid implementation combining Java and Python is developed, where YARN handles job execution while the QPSO module performs optimization. Experimental evaluation on a multi-node Hadoop 3.3.4 cluster using synthetic workloads shows that QIPS effectively reduces deadline penalties and improves data locality, while maintaining competitive performance across other scheduling metrics. These findings indicate that quantum-inspired optimization offers a promising direction for achieving efficient and balanced resource scheduling in distributed systems.