When a production machine drops offline or stops responding, guessing wastes precious minutes. Here is the exact five-step triage sequence I run to find the root cause and bring systems back online.
It was 3:15 AM on a Saturday morning when my phone vibrated with an urgent Prometheus alert.
Our primary application server had stopped replying to health check probes. HTTP requests were timing out, API clients were dropping connections, and Slack was lighting up with red automated incident notifications.
When an alert like that wakes you up, your first instinct is often panic. You want to rush in and hit the reboot button. You want to restart every service in sight.
That is the biggest mistake you can make.
Rebooting an unresponsive server without looking at its state destroys vital debugging evidence. Ephemeral kernel buffers, process core dumps, open file handles, and volatile memory allocations disappear the moment you cut the power. If you reboot blindly, you might fix the immediate symptom for ten minutes, only to have the exact same crash strike again during peak business hours.
Over years of managing Linux systems, building open-source infrastructure tools, and debugging messy production incidents, I learned to rely on a calm, systematic triage sequence.
Instead of guessing, I follow a strict 5-step checklist. It takes less than five minutes to run, works on almost every modern Linux distribution, and immediately points you directly to the culprit.
Here are the 5 things I check first whenever a Linux server goes down, crashes, or stops responding.
1. Reachability and Network Connectivity
Before you do anything else, you must figure out if the operating system actually crashed or if the machine is just cut off from the network.
A server can be running perfectly fine, but if a default gateway dropped, a firewall rule blocked incoming traffic, or a network interface lost its IP address, it will look completely dead to the outside world.
The Quick Ping Test
Start from your local workstation or a jump host and send a few ICMP packets to the server:
ping -c 4 192.168.1.50
If you receive replies with consistent round-trip times, the network layer and IP stack are alive. If you see Destination Host Unreachable or Request timed out, you are dealing with a routing drop, a physical link failure, or an aggressive packet filter.
Testing SSH Access and Response Codes
Next, attempt an SSH connection with verbose output enabled. This reveals whether the port is open, closed, or silently dropping packets:
ssh -vvv user@192.168.1.50
Watch where the connection stalls:
-
Connection refused: The network path works, but the SSH service (
sshd) on the server is stopped or crashed. - Connection timed out: Packets are being dropped somewhere along the path, often by a network firewall, cloud security group, or local iptables rule.
- Host key verification or banner displayed, then frozen: The SSH daemon is accepting connections, but system resources are so exhausted that the server cannot spawn a new login shell.
If You Have Out-of-Band Console Access
If SSH is completely dead, open your cloud web console (such as AWS EC2 Serial Console, DigitalOcean Web Console, or a hardware IPMI / iLO / KVM terminal) to log directly into the virtual tty.
Once you are in the console, check your network interface status:
ip addr show
Check if the main network interface is in the UP state:
ip link show eth0
If the link state says DOWN, bring it back up:
sudo ip link set eth0 up
sudo systemctl restart systemd-networkd
Checking Routing and Local Firewalls
Verify that your default gateway is configured properly:
ip route show
You should see a default route line similar to this:
default via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.50 metric 100
If the default route is missing, traffic cannot leave the server. You can add it back temporarily:
sudo ip route add default via 192.168.1.1 dev eth0
Finally, check if local firewall rules are blocking traffic:
sudo iptables -L -n -v --line-numbers
sudo ufw status verbose
If a recent script or bad deployment added an accidental drop-all rule, you can disable the firewall temporarily to regain access:
sudo ufw disable
2. Resource Starvation, System Load, and the OOM Killer
If you can log in, but commands take thirty seconds to execute or web services fail to reply, the system is likely suffering from resource starvation.
When CPU, memory, or process tables hit 100 percent capacity, Linux slows down to a crawl. In severe cases, the Linux kernel triggers emergency safety mechanisms that terminate critical processes.
Checking System Load and Uptime
Run uptime or w to see how hard the system is working:
uptime
Terminal output will look something like this:
03:22:14 up 42 days, 6:18, 2 users, load average: 28.45, 18.12, 9.80
The three numbers at the end represent the average system load over the past 1 minute, 5 minutes, and 15 minutes.
On Linux, the load average counts processes that are actively using the CPU, waiting for CPU time, or blocked waiting for uninterruptible disk I/O.
To interpret this number correctly, compare it to your total CPU core count:
nproc
If nproc reports 4 cores and your 1-minute load average is 28.45, your server has roughly 7 times more work queued up than it can process in real time. The CPU is completely saturated, or processes are stuck waiting on slow storage.
Checking Memory and Swap Pressures
Next, check physical memory and swap space with free:
free -h
Look closely at the available and swap columns:
total used free shared buff/cache available
Mem: 15Gi 14Gi 210Mi 1.2Gi 1.1Gi 480Mi
Swap: 4.0Gi 3.9Gi 100Mi
If available memory drops below a few hundred megabytes and Swap used is near maximum, the system is thrashing. The kernel is spending all its time moving pages between RAM and disk swap rather than doing actual application work.
Run vmstat to check for active memory paging and I/O wait:
vmstat 1 5
Pay attention to these columns:
- si / so (swap in / swap out): Numbers consistently above zero mean active swapping is choking performance.
- wa (I/O wait): High percentages (above 20-30%) mean the CPU is idle because it is waiting on slow disk operations.
- b (blocked processes): A high count indicates processes stuck waiting for disk or network I/O.
Hunting the Linux Out-Of-Memory (OOM) Killer
When a server runs out of physical RAM and swap space, the Linux kernel invokes the Out-Of-Memory (OOM) Killer.
The OOM Killer scans active processes, calculates a badness score based on memory footprint, and sends a ruthless SIGKILL to the biggest offender to keep the kernel itself from crashing.
Very often, a server goes down because the OOM Killer silently terminated PostgreSQL, MySQL, Redis, or an application worker.
Check the kernel log immediately for OOM Killer events:
sudo dmesg -T | grep -i "oom-killer\|out of memory\|killed process"
Or check system logs using journalctl:
sudo journalctl -k --grep="Out of memory" --since "1 hour ago"
If you see an entry like this:
[Sat Aug 16 03:14:02 2026] Out of memory: Killed process 28419 (node) total-vm:8452140kB, anon-rss:6210440kB, file-rss:0kB, shmem-rss:0kB
You have found your culprit. Your application leaked memory or was overwhelmed by a surge of traffic, and the kernel terminated it to save the rest of the operating system.
3. Storage Failures, Full Disks, and Inode Exhaustion
Storage issues are responsible for an enormous percentage of silent server failures.
When a disk fills to 100 percent capacity, databases cannot write transaction logs, web servers cannot create temporary session files, systemd cannot write journal logs, and authentication services cannot write lockfiles. The server stays on, but every service running on it crashes or refuses new connections.
Checking Disk Space Usage
Run df -h to see how much space is left across all mounted partitions:
df -h
Check the Use% column:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 49G 0 100% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda2 100G 42G 53G 45% /data
If the root filesystem (/) or /var shows 100%, new writes will fail immediately.
The Hidden Trap: Inode Exhaustion
Here is a sneaky problem that catches even experienced administrators off guard: your disk shows 50 gigabytes of free space, but every write operation fails with No space left on device.
How is that possible?
Every file and directory in Linux requires an inode to store metadata. If an application or cron job creates millions of tiny micro-files (such as uncleaned PHP session files or temporary cache items), you will run out of inodes long before you run out of raw gigabytes.
Check inode usage with the -i flag:
df -i
Look at the IUse% column:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 3276800 3276800 0 100% /
If IUse% is at 100 percent, the filesystem cannot create a single new file, even if you have hundreds of gigabytes of empty storage.
Finding Bloated Directories and Large Files
If disk space is full, locate the largest directories on the root partition:
sudo du -h --max-depth=1 / 2>/dev/null | sort -hr | head -n 10
Once you identify the problematic directory (frequently /var/log, /var/lib/docker, or /tmp), drill down further:
sudo du -h --max-depth=1 /var/log 2>/dev/null | sort -hr | head -n 10
Find individual files larger than 500 megabytes:
sudo find /var -type f -size +500M -exec ls -lh {} +
Detecting Deleted Files Still Locked by Processes
Sometimes you delete a massive 20 gigabyte log file with rm, but df -h still shows the partition as 100 percent full.
In Linux, when a file is deleted while an active process still holds an open file descriptor to it, the disk space is not freed. The filesystem keeps the data blocks allocated until the process closes the file or restarts.
Check for deleted files that are still held open in memory:
sudo lsof +L1
Or filter by deleted files directly:
sudo lsof | grep deleted
You will see output indicating which process is holding the ghost file:
nginx 14201 www-data 4w REG 253,1 21474836480 0 131075 /var/log/nginx/access.log (deleted)
To free that space immediately without crashing the application, reload or restart the specific process:
sudo systemctl reload nginx
Checking for Read-Only Filesystem Remounts
When the Linux kernel detects physical disk errors, bad storage blocks, or severe filesystem corruption, its default safety behavior is to instantly remount the filesystem as read-only (ro).
This protects existing data from being scrambled, but it stops all running services dead in their tracks.
Check if your filesystems have been remounted read-only:
mount | grep "ro,"
Check the kernel buffer for filesystem error logs:
sudo dmesg -T | grep -i "ext4-fs error\|xfs_error\|i/o error\|buffer i/o error"
If you see read-only mounts or I/O errors, the physical drive or virtual cloud volume may be failing, requiring a filesystem check (fsck) or immediate snapshot backup.
4. Service States, Process Lifecycles, and Port Conflicts
If the network is healthy, memory is plentiful, and disk space is clear, the problem usually comes down to failed application daemons, crashed systemd services, or network port collisions.
Listing All Failed Systemd Services
Modern Linux distributions use systemd to manage system daemons and background tasks. When services crash or exit unexpectedly, systemd tracks their failure state.
Run this command to see every failed unit on the machine:
systemctl --failed
If a background daemon crashed, it will show up in clean red text:
UNIT LOAD ACTIVE SUB DESCRIPTION
● nginx.service loaded failed failed A high performance web server
● postgresql.service loaded failed failed PostgreSQL RDBMS
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state.
SUB = The low-level unit activation state.
2 loaded units listed.
Inspecting Failed Service Details
To understand why a specific service died, check its status and exit code:
sudo systemctl status nginx.service
Look at the Process and Active lines:
● nginx.service - A high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2026-08-16 03:14:10 UTC; 8min ago
Process: 31024 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=1/FAILURE)
In this example, nginx -t failed before the server could start, meaning someone deployed a syntax error inside the configuration file.
Checking Active Ports and Listening Sockets
When an application fails to start, it is often because another background process is already bound to its required TCP port.
Use ss (the modern replacement for netstat) to check all active listening TCP and UDP sockets:
sudo ss -tulpn
Filter for a specific port (such as 80, 443, or 3000):
sudo ss -tulpn | grep -E ":(80|443|3000)"
Or use lsof to find the exact Process ID (PID) holding a port:
sudo lsof -i :80
When building networked applications and local streaming services, like my open-source media engine AiroShare, managing socket lifecycles and avoiding port locking on sudden restarts is one of the most critical engineering challenges. If an old orphaned process holds onto port 80 or 8080 after a crash, new instances will fail to launch with EADDRINUSE errors until the lingering socket is killed.
If an orphaned process is locking your port, terminate it cleanly:
sudo kill -15 <PID>
If it ignores SIGTERM after several seconds, force it down:
sudo kill -9 <PID>
Checking for Application Core Dumps
If a compiled binary (written in C, C++, Rust, or Go) crashes due to a segmentation fault (SIGSEGV), systemd-coredump captures the event.
List recent application crashes:
coredumpctl list
View the stack trace for the most recent crash:
coredumpctl info
This tells you the exact library, memory address, or function that caused the process to collapse.
5. System Logs, Kernel Panics, and Security Events
When standard diagnostic commands do not reveal an obvious cause, the answers are always written in the system logs.
Linux records almost everything that happens on the machine. By reading logs in chronological order starting from the exact minute the outage began, you can reconstruct the entire timeline of the failure.
Inspecting Systemd Journal Logs by Priority
Instead of scrolling through thousands of lines of unformatted text, use journalctl filtered by log level priority.
In syslog standards, errors range from priority 0 (emerg) to priority 3 (err). Filter for high-priority errors from the current boot session:
sudo journalctl -p err..emerg -b
Filter logs for a specific time window surrounding the crash:
sudo journalctl --since "2026-08-16 03:00:00" --until "2026-08-16 03:30:00"
Follow live logs for a specific failing unit in real time:
sudo journalctl -u nginx.service -f
Checking the Kernel Ring Buffer
The Linux kernel maintains an in-memory message buffer that records hardware failures, driver crashes, network stack anomalies, and segmentation faults.
Inspect the kernel ring buffer with human-readable timestamps:
sudo dmesg -T --level=err,crit,alert,emerg
Common red flags to watch for in dmesg include:
- Hardware Machine Check Exceptions (MCE): CPU hardware errors or failing memory channels.
- Kernel Panics / Null Pointer Dereferences: Driver bugs or corrupted kernel modules.
- Link speed renegotiation drops: Flapping network cables or bad switch ports.
- EXT4/XFS journal commit errors: Underlying storage drive timeouts or disk corruption.
Checking Authentication and Security Logs
Sometimes a server goes down not because of a hardware or software glitch, but because of malicious activity or an automated brute-force login attack that exhausted connection pools.
On Ubuntu and Debian systems, inspect /var/log/auth.log:
sudo tail -n 50 /var/log/auth.log
On RHEL, Rocky Linux, and CentOS systems, inspect /var/log/secure:
sudo tail -n 50 /var/log/secure
Look for massive bursts of failed SSH attempts from unfamiliar IP addresses:
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head -n 10
If thousands of bots are hammering your SSH port simultaneously, they can saturate connection limits and starve legitimate administration sessions.
An Interesting Historical Fact About Linux Failures
Did you know why the Linux kernel has an Out-Of-Memory Killer instead of simply failing memory allocation requests?
In early computing systems, when an application asked the operating system for memory with malloc() and physical RAM was exhausted, the operating system returned NULL (failure). The program was supposed to handle that error cleanly and exit gracefully.
However, in real-world software, programmers rarely checked if memory allocation failed. When memory ran out, applications tried to write to null pointers and crashed abruptly anyway.
To improve performance and multi-tasking efficiency, Linux adopted an aggressive strategy called memory overcommit. The kernel promises memory to applications before it physically allocates the underlying RAM pages, betting that most programs never use 100 percent of the memory they request.
When programs actually touch all that memory at once and the system runs out of physical RAM, the kernel finds itself in an impossible situation: it already promised memory it does not have.
Rather than letting the entire operating system freeze or panicking the kernel, Linus Torvalds and the core kernel developers designed the OOM Killer heuristic. The kernel sacrifices one high-memory process to preserve stability for the rest of the machine. It is a controversial design choice that sparked endless debates in the Unix community, but it keeps millions of production servers from locking up completely when memory spikes.
The 5-Minute Incident Triage Cheat Sheet
Here is a quick summary checklist you can keep handy during production emergencies:
-
Step 1: Network Connectivity
-
ping -c 4 <IP>(Test basic ICMP network reachability) -
ssh -vvv user@<IP>(Identify connection stalls or refused ports) -
ip addr showandip route show(Check interface and gateway status) -
sudo ufw statusorsudo iptables -L -n(Check firewall rules)
-
-
Step 2: Resource Starvation
-
uptimeandnproc(Compare load average against CPU cores) -
free -handvmstat 1 5(Check RAM, swap thrashing, and I/O wait) -
sudo dmesg -T | grep -i oom(Verify if OOM Killer struck)
-
-
Step 3: Storage and Inodes
-
df -h(Check disk space utilization) -
df -i(Check inode exhaustion percentage) -
sudo lsof | grep deleted(Find deleted files held open by processes) -
mount | grep "ro,"(Detect emergency read-only remounts)
-
-
Step 4: Services and Sockets
-
systemctl --failed(List all crashed system daemons) -
sudo systemctl status <service>(Inspect service exit codes) -
sudo ss -tulpn(Check listening TCP/UDP ports and conflicts) -
coredumpctl list(Inspect recent binary crashes)
-
-
Step 5: System Logs
-
sudo journalctl -p err..emerg -b(Inspect high-priority systemd logs) -
sudo dmesg -T --level=err,crit(Check kernel ring buffer for hardware/disk errors) -
sudo tail -n 50 /var/log/auth.log(Check for brute force authentication attacks)
-
What Do You Check First During an Outage?
Every system administrator and DevOps engineer develops their own debugging habits over time. When your servers drop offline, what is the very first terminal command you type? Do you jump straight to top, check journalctl, or look at network routes first? Let me know in the comments below!
About the Author
Asep Sayyad is a Linux and DevOps engineer passionate about Linux administration, automation, cloud technologies, containers, and open-source software. He enjoys solving real-world infrastructure challenges and sharing practical knowledge through in-depth technical articles, tutorials, and hands-on guides.
His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.
Connect with Me
Portfolio: https://asepsayyad007.in
GitHub: https://github.com/asepsayyad007
LinkedIn: https://www.linkedin.com/in/asepsayyad
Medium: https://asepsayyad007.medium.com
Enjoyed this article?
If you found this guide helpful, consider:
- Starring my open-source projects on GitHub.
- Sharing this article with fellow Linux and DevOps engineers.
You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!
© 2026 Asep Sayyad
