Disk full on a production server: recovery runbook Pro

A disk full event on a production server rarely stays a disk problem. MySQL stops writing and sites show database errors, Exim cannot create spool files, PHP cannot save sessions so logins fail, and backups die halfway. The fix is almost always a few gigabytes away, but the fastest-looking deletions (a MySQL file, an open log, the mail spool) are exactly the ones that turn an outage into data loss. This runbook frees space in order of safety, then brings each service back properly.

Symptoms

• No space left on device in shells, PHP errors or cron mail; MySQL errors with Errcode: 28.
• Exim logs Failed to create spool file or refuses mail; the queue stops moving.
• Logins to cPanel, DirectAdmin or SSH fail oddly; PHP session writes fail.
• df -h shows 100%, or shows free space while df -i shows 100% inodes.

First 5 minutes

Find out which filesystem is full, and whether it is blocks or inodes: