#YARN DISK HEALTH CHECK DRIVER#

The threshold is configured by YARN node manager configurable property: -disk-utilization-per-disk-percentage. The following error message is a common one you can notice in YARN nodes portal:ДЎ/2 local-dirs usable space is below configured utilization percentage/no more usable space 1/1 log-dirs usable space is below configured utilization percentage/no more usable space. If the attached disk are all used, the node will become unhealthy and eventually lost. For example, if memory storage is not sufficient, Spark can persist the data to disk. It can also be used to persist data depends on the configuration of Spark storage level. In EMR Spark applications, EBS volume (disk attached to the cluster nodes) are important when handling big volume of data as it can be used to store temporary data, cache files, cache user files, store logs, HDFS, etc. In many projects, we use EMR to read and write data from S3 directly.

Typical case - Disk space usage is above threshold To ensure stability of your EMR cluster, you could potentially change from spot instances to on-demand instances with bigger cost. Usually this will not cause your Spark application to fail as long as there are sufficient resources in the cluster overall. However it can be taken away at anytime that causes interruptions to your applications. The spot instancesВ will be cheaper than on-demanded ones. When configuring instance fleets, you can choose Spot units.

Node VMs were deleted by someone accidentally.

The nodes become unhealthy due to exceeding log and disk space ratio.

You are using Spot units for your Core or Task nodes and the instances were taken.

Thus to debug this issue, we can look into the possible root causes for losing nodes.

#YARN DISK HEALTH CHECK DRIVER#

These containers are created by NodeManager with coordination from Application Master (Spark driver container).ДЇrom the error message itself, we can tell the nodes were lost. About containers and nodesДЄs shown in the following diagram, containers (Spark executors) run in nodes. Hopefully that will provide some insights to debug your problem. I'm documenting some common approaches I've taken to fix these issues based on my previous experience in Spark on on-premise systems (CDH) and other cloud big data frameworks (for example, HDInsights on Azure and Dataproc on GCP). Diagnostics: Container release on a *lost* node.

One of the common frequent issue was due to containers released on nodes - Container marked as failed. Recently I've been helping my team to debug EMR Spark applications on AWS.

Yarn disk health check