Capacity scheduler

The capacity scheduler makes sure that users get the guaranteed minimum amount of configured resources in the YARN cluster. The use of the Hadoop cluster increases with the use cases in the organization and it is very unlikely that organization creates separate Hadoop clusters for each use case because this will increase maintenance. One use case may be that different users in the same organization want to have a certain amount of resources reserved when they want to execute their tasks. The capacity scheduler helps in sharing the cluster resources in a cost-effective manner across different users in the same organization to meet the SLA by ensuring no other user uses resources configured for some other user in the cluster. In short, the cluster resources are shared across multiple user groups.
The capacity scheduler works on the concept of queues. A cluster is divided into partitions known as queues and each queue is assigned with a certain percentage of resources. The capacity scheduler is responsible for making sure that each queue gets its resource share from the cluster resource pool when a job is submitted to that queue.

Let us try to explain this with an example. Suppose we created two queues; A and B. Queue A has been allocated with 60% and queue B has been allocated 40% of resource share. When we submit first job to queue A, the capacity scheduler will allocate all 100% of the available resources to queue A because there are no other tasks or jobs running in the cluster at that moment. Now suppose while the first job is running another user submits a second job to queue B, than the capacity scheduler will kill a few tasks of first job and assign it to the second job to ensure that queue B gets its guaranteed minimum share from the cluster resources.