Distributed data processing systems have emerged as a necessity for processing large-scale data volumes in reasonable time. Apache Hadoop is the most popular implementation of a distributed data processing system. It provides strong support to fault tolerance, reliability, and scalability for distributed data processing scenarios (e.g., MapReduce jobs).

The problem that yet remains is how to optimally schedule distributed tasks, such as MapReduce or Spark jobs, and distribute their input data inside the cluster. Current solutions typically uniformly distribute data of individual jobs to balance their load over the cluster nodes. However, with increasing number of different jobs (typical for shared, multi-tenant clusters) the skewness in data and thus the workload over the cluster nodes becomes more significant and harmful to performances. As a result, the execution of some tasks may be deferred, either due to waiting at the overloaded resources, or large data transfers that occur if the execution of a task is far from its input data. Effectively predicting the workload of distributed jobs over different resources in a cluster would enable smarter scheduling of tasks, as well as timely bringing the input data closer, hence not deferring the tasks execution. Such data collocation would eventually converge and produce a data distribution that better suits the workload executing in the cluster, and thus decrease the number of large data transfers. All in all, we aim at balancing processing (i.e., maximizing the use of available resources) and bring data to the resources that will consume it before it is needed. The aim of this project is to: (1) analyze how the existing prediction techniques can help predicting distributed processing workloads over Hadoop clusters, (2) develop scheduling algorithms that employ such prediction techniques for scheduling jobs and collocating data inside the cluster, (3) develop monitoring techniques that would allow to timely react and reschedule in the case of cluster failures, or inaccurate/obsolete predictions, (4) experimentally evaluate developed algorithms with regard to the optimality of the makespan of scheduled jobs.

Main Advisor at Universitat Politècnica de Catalunya (UPC)
Co-advisor at Technische Universität Dresden (TUD)