Disk Space Issues in Hadoop

This topic addresses about how to troubleshoot a few common Hadoop disk space issues.

Handling a Disk Space Issue When Creating a Directory

While running Hadoop jobs, you can hit this exception: cannot create directory :No space left on device.

This exception usually appears when the disk space in the HDFS is full. In Qubole, only temporary/intermittent data to the HDFS is deleted as a cron job would be running to delete the temp files regularly. This issue would be seen in cases such as:

• Long running jobs where the jobs may be writing lots of intermediate data and the cron could not delete the data as the jobs are still running.
• Long running clusters where in rare cases, the data written from failed or killed tasks may not get deleted.

Solution: Verify the actual cause by checking the HDFS disk usage from one of these methods:

• On the Qubole UI, through the DFS Status from the running cluster’s UI page.

• By logging into the cluster node and running this command:

  hadoop dfsadmin -report

  A sample response is mentioned here.

  Configured Capacity: 153668681728 (143.12 GB)
  Present Capacity: 153668681728 (143.12 GB)
  DFS Remaining: 153555091456 (143.01 GB)
  DFS Used: 113590272 (108.33 MB)
  DFS Used%: 0.07%
  Under replicated blocks: 33
  Blocks with corrupt replicas: 0
  Missing blocks: 0
  
  -------------------------------------------------
  Live datanodes (2):
  
  Name:x.x.x.x:50010 (ip-x-x-x-x.ec2.internal)
  Hostname:ip-x-x-x-x.ec2.internal
  Decommission Status : Normal
  Configured Capacity: 76834340864 (71.56 GB)
  DFS Used: 56795136 (54.16 MB)
  Non DFS Used: 0 (0 B)
  DFS Remaining: 76777545728 (71.50 GB)
  DFS Used%: 0.07%
  DFS Remaining%: 99.93%
  Configured Cache Capacity: 0 (0 B)
  Cache Used: 0 (0 B)
  Cache Remaining: 0 (0 B)
  Cache Used%: 100.00%
  Cache Remaining%: 0.00%
  Xceivers: 2
  Last contact: Tue Dec 26 11:21:19 UTC 2017
  
  
  Name:x.x.x.x:50010((ip-x-x-x-x.ec2.internal)
  Hostname: ip-x-x-x-x.ec2.internal
  Decommission Status : Normal
  Configured Capacity: 76834340864 (71.56 GB)
  DFS Used: 56795136 (54.16 MB)
  Non DFS Used: 0 (0 B)
  DFS Remaining: 76777545728 (71.50 GB)
  DFS Used%: 0.07%
  DFS Remaining%: 99.93%
  Configured Cache Capacity: 0 (0 B)
  Cache Used: 0 (0 B)
  Cache Remaining: 0 (0 B)
  Cache Used%: 100.00%
  Cache Remaining%: 0.00%
  Xceivers: 2
  Last contact: Tue Dec 26 11:21:21 UTC 2017
  

Handling a Device Disk Space Error

While running jobs, you may hit this exception - java.io.IOException: No space left on device.

Cause: This exception usually appears when there is no disk space on the worker or coordinator nodes. You can confirm this by logging into the corresponding node and running a df -h on the node when the query is still running.

Solution: You can avoid this error by one of these solutions:

1. Enable EBS autoscaling. After enabling, you can attach additional EBS volumes based on the query’s requirement.
2. You can also try using cluster instance types with larger disk space.