10 Least-Known Linux Commands Useful for DevOps
10 lesser-known Linux commands for DevOps incidents: mtr, strace, lsof, pidstat, sar, nc, dig, findmnt, systemd-analyze and watch, with examples.
The ten Linux commands below are less famous than ls, grep or top, and each one answers a question DevOps engineers face during incidents: where packets are lost (mtr), what a process is doing (strace), who holds a file or port (lsof), what used CPU last night (sar), and why boot is slow (systemd-analyze). Every command here comes with a real example and its limits.
Command options were checked against the current man pages (man7.org, the sysstat project, the OpenBSD nc manual and the BIND documentation). Some tools need installing and some behave differently across distributions, so each section says so.
Key takeaways
- mtr combines ping and traceroute into one live view, which makes per-hop packet loss easy to spot.
- strace shows system calls, so a hung or failing process can be explained without reading its source code. It slows the traced process, so use it carefully in production.
- lsof answers "who is using this port or file", including files that were deleted but are still held open and filling a disk.
- pidstat and sar come from the sysstat package. pidstat reports per process, and sar keeps history so you can look back at last night's CPU or memory.
- nc, dig and findmnt are quick tests for connectivity, DNS and mounts when a service is "up" but unreachable.
- systemd-analyze and watch speed up boot investigations and repeated checks.
The 10 commands at a glance
| # | Command | Question it answers | Package |
|---|---|---|---|
| 1 | mtr | Where on the network path is traffic slow or lost? | mtr |
| 2 | strace | What system calls is this process making, and which one fails? | strace |
| 3 | lsof | Which process has this file, port or deleted file open? | lsof |
| 4 | pidstat | Which process is using CPU, memory, disk or context switches? | sysstat |
| 5 | sar | What did the system do over the last hours or days? | sysstat |
| 6 | nc | Does this TCP port accept connections? | netcat or ncat |
| 7 | dig | What does DNS return, and from which server? | bind-utils or dnsutils |
| 8 | findmnt | What is mounted where, with which options? | util-linux |
| 9 | systemd-analyze | Why is boot slow, or is this unit file valid? | systemd |
| 10 | watch | How does this command's output change over time? | procps |
For the everyday commands that surround these, see our lists of top 20 Linux commands for DevOps engineers and 10 Linux shell commands to master.
1. mtr: live path analysis for network problems
mtr (My Traceroute) sends probes continuously and shows loss and latency for every hop between you and a target. It replaces the habit of running ping and traceroute side by side.
mtr -rwn -c 50 example.com
The flags: -r prints a report and exits instead of opening the live screen, -w uses wide output so hostnames are not cut, -n skips reverse DNS lookups, and -c 50 sends 50 probes per hop. The report lists loss percentage, last, average, best and worst round trip times for each hop.
Reading it takes care. Loss that appears at one middle hop but not at later hops is usually a router that deprioritises ICMP replies and is not a real fault. Loss that starts at a hop and continues to the destination is the one to report to your provider. By default mtr uses ICMP echo. If firewalls drop ICMP, -T sends TCP SYN packets instead, and -P 443 sets the port.
mtr -rwn -T -P 443 -c 50 api.example.com
Use it when users report slowness only from one office, one cloud region or one ISP. Attach the report to the ticket you send to the network team.
2. strace: see what a process asks the kernel to do
strace prints every system call a process makes, with arguments and return values. For a process that fails with a vague message, the failing call (for example openat(...) = -1 ENOENT (No such file or directory)) shows the exact missing file or permission.
# start a command under strace and log to a file
strace -f -tt -T -o /tmp/app.strace ./myapp --config /etc/myapp.conf
# attach to a running process, files only
sudo strace -f -tt -p 2481 -e trace=%file
# summary of where time and errors go
sudo strace -c -p 2481
The options: -f follows child processes and threads, -tt adds microsecond timestamps, -T shows time spent in each call, -p attaches to a process ID, and -e trace=%file limits output to calls that take a file name. -c counts calls, errors and time per system call and prints a table when you stop it. Use -s 200 to print longer strings, because the default cuts them short, and -yy to decode file descriptors into paths and socket addresses.
The limit is overhead. strace slows the traced process noticeably, so attach to a busy production service for a short window, or reproduce the problem in staging. Attaching also needs ptrace permission, so expect to use sudo, and some hardened systems restrict it.
3. lsof: who has this file, port or deleted file open
In Linux many things are files, so lsof ("list open files") can show open files, directories, libraries, network sockets and pipes, and the process holding each one.
# who is listening on port 8080?
sudo lsof -nP -iTCP:8080 -sTCP:LISTEN
# everything a process has open
sudo lsof -p 2481
# files held by one user
sudo lsof -u deploy
# open files with zero links: deleted but still held open
sudo lsof +L1
The flags: -i selects network files, -n and -P stop lookups of host and port names so output is fast, -p selects a process ID, -u a user, and +L1 lists files whose link count is below one.
The last one solves a classic puzzle. A disk shows 100% full, but du cannot find the space. A process still holds a large deleted log file open, so the space is not released until the process closes it or restarts. lsof +L1 names the process and the file size. Restarting the service, or truncating the file through /proc/PID/fd where that is acceptable, frees the space.
lsof only shows full details for processes you own unless you run it as root, so use sudo on shared servers.
4. pidstat: per process CPU, memory, disk and context switches
top shows a snapshot. pidstat, from the sysstat package, prints a row every interval for each active process, so you can see which one is responsible over time.
# CPU per process, every 2 seconds, 5 samples
pidstat -u 2 5
# memory (page faults and RSS) for one process
pidstat -r -p 2481 2 5
# disk read and write per process
pidstat -d 2 5
# context switches per process, with threads
pidstat -w -t 2 5
Use -u for CPU, -r for memory, -d for disk I/O, -w for context switches and -t to include threads. -p takes one or more process IDs, or ALL. A process with high voluntary context switches is waiting on I/O or locks, while one with high involuntary switches is competing for CPU. That difference helps decide between "add CPU" and "fix the code or storage". Install with sysstat on both RHEL family and Debian family systems. For process monitoring basics, see monitoring processes with ps, top and htop.
5. sar: historical system activity
sar, from the same package, collects and reports system activity. Its strength is history: when sysstat data collection is enabled, you can answer "what was the CPU doing at 02:30 last night?" after the fact.
sar -u 1 5 # CPU, live: 5 samples, 1 second apart
sar -r # memory use, from today's data file
sar -q # run queue length and load averages
sar -n DEV 1 3 # per interface network throughput
sar -u -s 02:00:00 -e 03:00:00 # CPU between 02:00 and 03:00 today
sar -u -f /var/log/sa/sa16 # CPU from the 16th (path varies)
-u is CPU, -r memory, -q queue and load, -n DEV network devices, -s and -e set a start and end time, and -f reads an older data file. The data directory depends on the distribution: /var/log/sa is typical on RHEL family systems and /var/log/sysstat on Debian family systems. On some distributions collection is switched off until you enable the sysstat service or timer, so check that before an incident, not during one. For a wider tool tour, see our post on tools for troubleshooting Linux performance and the guide to analysing CPU and memory use.
6. nc: test a TCP port in one line
nc (netcat) opens raw TCP and UDP connections. DevOps engineers use it as a port checker when curl is overkill or the protocol is not HTTP.
nc -zv db.internal.example.com 5432
nc -zv -w 3 cache.internal.example.com 6379
In the OpenBSD version, -z scans for a listening service without sending data, -v prints the result and -w 3 sets a 3 second timeout. A "succeeded" message means the TCP handshake completed, so routing, security groups and firewall rules allow the connection. "Connection refused" means the host answered but nothing listens on that port. A timeout usually means a firewall is dropping packets.
Be aware that several programs share the name nc: the OpenBSD version, the traditional netcat and Ncat from the Nmap project. Options differ, for example how listen mode takes a port, so check man nc on your system. Inside Kubernetes pods, nc is often present in a busy debug image even when curl is not. Only test hosts and ports you are authorised to test.
7. dig: ask DNS directly
When "the site is down" turns out to be DNS, dig shows what a resolver actually returns. It is part of the BIND utilities (bind-utils on RHEL family, dnsutils or bind9-dnsutils on Debian family).
dig +short A www.example.com # just the answer
dig @8.8.8.8 www.example.com # ask a specific resolver
dig +noall +answer www.example.com MX # clean answer section for MX records
dig +trace www.example.com # follow delegation from the root
dig -x 192.0.2.10 # reverse lookup
dig +tcp www.example.com # force TCP instead of UDP
Comparing the answer from your internal resolver with the answer from @8.8.8.8 quickly shows split-horizon DNS or stale caches. +trace walks from the root servers down and exposes a wrong delegation or a name server that is not answering. The answer section also shows the TTL, which tells you how long a bad record will survive in caches. For wider network checks, see ping, traceroute and netstat for diagnostics.
8. findmnt: a readable view of mounts
findmnt, from util-linux, lists mounted filesystems as a tree and can filter, format and validate them. It is clearer than reading /proc/mounts or the long output of mount.
findmnt # tree of all mounts
findmnt -T /var/lib/docker # which filesystem holds this path?
findmnt -t nfs,nfs4 # only NFS mounts
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS # choose columns
findmnt --verify # check /etc/fstab for errors
-T is the one to remember: given any path, it shows which mount contains it, which solves "why is this directory full when the root disk is fine". --verify checks that /etc/fstab is parsable and usable, so run it before a reboot after editing the file. A mistake in fstab can drop a server into emergency mode at boot.
9. systemd-analyze: boot time and unit file checks
On systemd based distributions, systemd-analyze shows where boot time goes and validates unit files.
systemd-analyze # total boot time split into kernel and userspace
systemd-analyze blame # units sorted by time taken to start
systemd-analyze critical-chain # the chain of units that delayed boot
systemd-analyze verify /etc/systemd/system/myapp.service
systemd-analyze security myapp.service # sandboxing exposure score for a unit
blame lists units by initialisation time, but the manual warns that a slow unit is not always a blocker, because services start in parallel. critical-chain shows the dependency chain that actually held boot back, so read both. verify catches typos in a unit file before you enable it. security scores how much of the system a service can reach and suggests directives such as ProtectSystem= and NoNewPrivileges= that reduce that exposure. See our post on managing services with systemctl for the service side.
10. watch: repeat a command and highlight changes
watch runs a command at a fixed interval and redraws the screen. It turns any one-shot command into a live view without writing a loop.
watch -n 2 -d 'ss -s' # socket totals, changes highlighted
watch -n 5 'df -h /var' # disk use of /var
watch -d -n 1 'kubectl get pods -n shop' # rollout progress
watch -g 'systemctl is-active myapp' # exit when the output changes
-n sets the interval in seconds (the default is 2), -d highlights what changed between updates, -g exits when the output changes (useful in scripts that wait for a state change), and -e stops if the command returns an error. Quote the whole command so your shell does not interpret pipes before watch sees them.
Putting them together: three incident walk-throughs
A service is reachable from one office only
Run dig +short from both locations to rule out DNS, then nc -zv to the port from each, then mtr -rwn -c 50 from the failing office. The report shows whether loss starts at a hop inside that office's ISP.
A disk is full but the files add up to less
Use findmnt -T /path to confirm which filesystem you are looking at, then lsof +L1 to find deleted files still held open. Restart or reload the process that holds the file, and fix log rotation so the service reopens its log after rotation.
A process hangs after a deploy
Use pidstat -u -w -p PID 1 5 to see whether it burns CPU or sits waiting. Then attach strace -f -tt -T -p PID briefly to see whether it is stuck in a call such as futex, read or connect. A call blocked on connect to a database address sends you back to nc and dig.
Practice these on a real system
Commands like these are easier to remember when you have used them on a broken system. Our Linux administration training in Pune (classroom in Kharadi, plus live online across India) practises troubleshooting on live servers, and the RHCSA training covers storage, services and boot targets that several of these commands inspect. If you are preparing for interviews, the Linux interview preparation guide gives a study plan.
Frequently asked questions
What is the difference between mtr and traceroute?
traceroute sends one round of probes and prints the path. mtr keeps sending probes and updates loss and latency per hop, so intermittent problems show up. mtr also has a report mode (-r) for sharing results.
Is it safe to run strace in production?
It is safe in the sense that it does not change the program, but it slows the traced process, sometimes heavily. Attach for a short time, filter with -e trace= to reduce output, and prefer staging when you can reproduce the issue there.
How do I find which process is using a port?
Run sudo lsof -nP -iTCP:PORT -sTCP:LISTEN. The ss command with -ltnp also shows listening sockets with process names. Without sudo you only see processes you own.
Why does sar show no data for yesterday?
Collection may not be enabled. sar reads files written by the sysstat collector, and on some distributions it must be enabled first. File locations also differ: /var/log/sa is common on RHEL family systems and /var/log/sysstat on Debian family systems.
Is nc a good way to scan ports?
It is fine for checking one known port on a host you manage, using nc -zv host port. For broader discovery use a dedicated scanner, and scan only systems you are authorised to test.
Why does a deleted file still use disk space?
Linux frees the blocks only when the last open handle is closed. If a process still has the file open, the space stays allocated. lsof +L1 lists such files with their owners.
Do these commands work on every distribution?
The commands are widely available, but package names differ and some are not installed by default. strace, mtr, lsof and sysstat usually need installing. dig comes from bind-utils or dnsutils. systemd-analyze exists only on systemd based systems.
How can I get better at Linux troubleshooting?
Break things on purpose in a lab: fill a disk, kill a DNS record, mistype fstab, then diagnose with these commands. Repeating the cycle builds the habit of asking a specific question and choosing the command that answers it.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0