41. Listen queue overflows
During traffic spikes clients see connect timeouts, but the server process is not CPU bound. netstat -s shows the counters below rising. What does this indicate?
48213 times the listen queue of a socket overflowed
48213 SYNs to LISTEN sockets dropped
- A.The accept backlog is full because the app accepts connections too slowly or the backlog is too small
- B.The TLS handshake is failing because of an expired certificate
- C.The disk is full so new sockets cannot be created
- D.The client is using the wrong port
Show answer
Answer: A. The accept backlog is full because the app accepts connections too slowly or the backlog is too small
Completed connections wait in the accept queue until the app calls accept(). If it is full, the kernel drops new SYNs or ACKs and clients retry after a delay, which looks like timeouts. Fixes are raising the app backlog together with net.core.somaxconn and making the accept loop faster or adding workers.
42. Invalid cross-device link
ln /data/big.iso /home/me/big.iso fails with "Invalid cross-device link". /data and /home are separate filesystems. Why?
- A.Hard links are allowed only for files smaller than 4 GB
- B.Inode numbers only mean something inside one filesystem
- C.Hard links need both folders to have the same owner
- D.The target folder must be mounted with the "links" option
Show answer
Answer: B. Inode numbers only mean something inside one filesystem
Each filesystem numbers its own inodes, so a directory entry on /home cannot point at an inode on /data. Use a symbolic link instead, which stores a path and can cross filesystems. The same rule is why mv between filesystems must copy and delete.
43. Disk 100% for users but not for root
df shows /data at 100% Use and normal users get "No space left", yet Used is clearly less than Size and root can still write. What is going on?
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 1.0T 950G 0 100% /data
- A.The filesystem has run out of inodes, not blocks
- B.A deleted file is still open and holding the space
- C.ext4 keeps about 5% of blocks reserved for root by default
- D.df counts the journal as user data on ext4
Show answer
Answer: C. ext4 keeps about 5% of blocks reserved for root by default
mkfs.ext4 reserves 5% of blocks for root so the system can still log and recover when full. Avail excludes that reserve, so Use% hits 100% early. On a pure data volume you can lower it with tune2fs -m 1, but first free real space.
44. Setuid on a Bash script
You set chmod u+s on a root-owned Bash script so users can run one admin task. It still runs as the calling user. Why?
- A.The setuid bit only works for files in /usr/sbin
- B.Bash drops the bit unless the script ends in .sh
- C.Linux ignores the setuid bit on interpreted scripts
- D.The bit only applies after the next reboot
Show answer
Answer: C. Linux ignores the setuid bit on interpreted scripts
Setuid scripts had race conditions between the kernel opening the script and the interpreter reading it, so Linux honours setuid only on binaries. Use a narrow sudoers rule for that one script instead.
45. Root denied on an NFS share
On an NFS client, root gets "Permission denied" writing to a share that a normal user with the matching UID can write to. What is the usual reason?
- A.NFS only allows writes from users without a shell
- B.The client kernel blocks root from all network filesystems
- C.The server's root_squash maps client root to an unprivileged user
- D.The share is mounted with noexec, which also blocks root writes
Show answer
Answer: C. The server's root_squash maps client root to an unprivileged user
By default NFS exports use root_squash, so requests from UID 0 arrive as nobody (nfsnobody). That stops a client admin from owning every file on the server. Use the proper user, or no_root_squash only when you trust the client.
46. Root cannot delete a file
As root, rm /etc/resolv.conf says "Operation not permitted". The filesystem is mounted read-write and permissions look normal. What should you check next?
- A.ls -Z for a missing SELinux label on the file
- B.getfacl for an ACL that denies root
- C.lsattr for the immutable flag set with chattr +i
- D.stat for a hard link count above one
Show answer
Answer: C. lsattr for the immutable flag set with chattr +i
The immutable attribute stops any change, even by root, until chattr -i removes it. Some admins set it to stop DHCP clients rewriting resolv.conf. A link count above one does not block rm, and ACLs do not restrict root.
47. Bad interpreter with a ^M
A script edited on Windows fails with "/bin/bash^M: bad interpreter: No such file or directory". What is wrong?
- A.The file has CRLF line endings, so the shebang names "bash\r"
- B.Bash is not installed in /bin on this distribution
- C.The script lacks execute permission for its owner
- D.The first line must be #!/usr/bin/env sh on Linux
Show answer
Answer: A. The file has CRLF line endings, so the shebang names "bash\r"
Windows editors end lines with carriage return plus line feed. The kernel reads the shebang up to the line feed, so it looks for a program literally named "bash\r". Convert with dos2unix or sed -i 's/\r$//'.
48. %util at 100% on an NVMe drive
iostat shows an NVMe volume at 100% util, but latency (await) is still low and throughput keeps rising as load grows. How should you read this?
- A.The disk is saturated and must be replaced at once
- B.The figure is wrong because NVMe has no queue
- C.%util means "busy at least once"; parallel devices can do far more
- D.It means the page cache is full and writes are blocked
Show answer
Answer: C. %util means "busy at least once"; parallel devices can do far more
%util is the share of time with at least one request in flight. Devices that serve many requests in parallel, like NVMe or RAID arrays, can show 100% while far from their limit. Judge saturation by latency and queue growth instead.
49. Uneven speed on a two-socket server
On a two-socket server, a latency-sensitive process sometimes runs 30% slower. numastat shows most of its memory on node 1 while it runs on CPUs of node 0. What is happening?
- A.Node 1 memory is faulty and the kernel retries reads
- B.The kernel is swapping its memory between the two nodes
- C.The process lacks permission to use node 0 memory
- D.It pays the cost of remote NUMA memory; bind CPU and memory to one node
Show answer
Answer: D. It pays the cost of remote NUMA memory; bind CPU and memory to one node
Each socket has local memory; reaching the other socket's memory crosses the interconnect and is slower. numactl --cpunodebind and --membind (or NUMA-aware settings in the app) keep threads and memory together.
50. malloc succeeded, then OOM kill
A program's malloc of 20 GB succeeds on a 16 GB box, but the process is OOM-killed later while filling the buffer. Why did malloc not fail?
- A.malloc always reserves swap first, then RAM
- B.The kernel moved the buffer to another NUMA node
- C.Linux overcommits; memory is only taken when pages are touched
- D.malloc returns a fake pointer when memory is short
Show answer
Answer: C. Linux overcommits; memory is only taken when pages are touched
With the default heuristic (vm.overcommit_memory=0), the kernel grants address space and allocates physical pages on first write. If too many pages are touched, the OOM killer steps in. Mode 2 enforces a strict commit limit, making allocations fail early instead.
51. Running strace on a busy service
Why is attaching strace to a busy production database risky, and what is a lower-overhead option?
- A.strace restarts the process; use gdb instead
- B.strace needs a reboot to detach; use lsof instead
- C.ptrace stops it at every syscall; eBPF tools cost far less
- D.strace writes to the database files; use ltrace instead
Show answer
Answer: C. ptrace stops it at every syscall; eBPF tools cost far less
strace uses ptrace, which switches to the tracer on every system call and can slow a busy process many times over. perf trace and eBPF tools (bpftrace, bcc) gather similar data in the kernel with far lower overhead.
52. Latency improves under load
A low-latency trading gateway is faster when the box is busy than when it is idle. cpupower frequency-info shows the "powersave" governor. What is the likely explanation?
- A.Idle cores clock down and take time to ramp up
- B.The kernel batches network packets only when idle
- C.Busy servers get priority on the network switch
- D.The page cache is cleared when CPU usage is low
Show answer
Answer: A. Idle cores clock down and take time to ramp up
Power-saving governors and deep C-states lower frequency and sleep idle cores, so the first work after a quiet period runs slowly. Latency-critical hosts often use the performance governor and limit deep C-states, trading power for steady response.
53. RAM used but no process owns it
A 64 GB server shows 40 GB used, yet process RSS adds up to under 10 GB and cache is small. /proc/meminfo shows HugePages_Total: 15000 with Hugepagesize 2048 kB. What explains it?
- A.About 30 GB is reserved as huge pages, used or not
- B.The kernel has leaked 30 GB and needs a reboot
- C.The huge page count is in kB, so it is only 15 MB
- D.RSS excludes stack memory, which holds the rest
Show answer
Answer: A. About 30 GB is reserved as huge pages, used or not
15,000 × 2 MB is about 30 GB set aside by vm.nr_hugepages, often for a database that is not running or is configured smaller. Pre-allocated huge pages are removed from normal use even when idle. Size the pool to what the app actually maps.
54. Start request repeated too quickly
After you fix a config bug, "systemctl start api" still fails with "start request repeated too quickly". What is happening?
- A.The fixed config has not been compiled into the unit
- B.systemd blocks a unit for 24 hours after any crash
- C.It hit the start limit; run reset-failed, then start
- D.The api binary is locked until the next reboot
Show answer
Answer: C. It hit the start limit; run reset-failed, then start
If a unit starts more than StartLimitBurst times within StartLimitIntervalSec (5 in 10 s by default), systemd refuses further starts. reset-failed clears the counter. Setting RestartSec helps avoid tripping the limit in the first place.
55. Service writes to /tmp but nothing is there
A service logs that it wrote /tmp/export.csv, but ls /tmp on the host shows no such file. The unit has PrivateTmp=yes. Where is the file?
- A.Nowhere; PrivateTmp makes /tmp writes fail silently
- B.In /var/tmp, where systemd redirects all /tmp writes
- C.In RAM only, and it was lost when the shell started
- D.In a private /tmp under /tmp/systemd-private-*-<unit>-*/tmp
Show answer
Answer: D. In a private /tmp under /tmp/systemd-private-*-<unit>-*/tmp
PrivateTmp gives the service its own mount namespace with separate /tmp and /var/tmp, stored under paths like /tmp/systemd-private-<id>-<unit>-<rand>/tmp on the host. Other processes cannot see it, and it is removed when the service stops.
56. Cannot mount root after a hardware change
After moving a VM disk to a different virtual storage controller, boot fails with "unable to mount root fs" while the disk is fine. What is the likely cause?
- A.The initramfs lacks the new controller's driver; rebuild it
- B.The root filesystem was corrupted by the move
- C.GRUB cannot read disks bigger than 2 TB
- D.The kernel needs a new licence key for the hardware
Show answer
Answer: A. The initramfs lacks the new controller's driver; rebuild it
The initramfs holds the drivers needed to find and mount root before the real system loads. Many builds include only drivers for the hardware present when they were made. Boot with the old controller or a rescue image and run dracut -f or update-initramfs -u.
57. nginx cannot bind to port 8081
On RHEL with SELinux enforcing, nginx fails to start on port 8081 with "Permission denied" on bind, while port 80 works. What is the proper fix?
- A.setsebool -P httpd_can_network_connect 1
- B.chmod 755 /usr/sbin/nginx
- C.semanage port -a -t http_port_t -p tcp 8081
- D.Run nginx as root with setenforce 0
Show answer
Answer: C. semanage port -a -t http_port_t -p tcp 8081
SELinux policy lists which ports the httpd_t domain may bind, labelled http_port_t. Adding 8081 to that type allows it. httpd_can_network_connect controls outbound connections, not binding a listening port.
58. Failure hidden inside a function
Inside a function, curl fails, but $? is 0 and set -e does not stop the script. Why?
fetch() {
local body=$(curl -fsS "$URL")
echo "status: $?"
}- A.curl -f always returns 0 on HTTP errors
- B.Command substitution never reports errors in Bash
- C.Functions reset $? to 0 when they start
- D.$? is the exit status of local, which succeeded
Show answer
Answer: D. $? is the exit status of local, which succeeded
The line runs the builtin local after the substitution, and $? reflects local, which succeeds. Declare first and assign separately: local body; body=$(curl ...). Then $? and set -e see curl's status. ShellCheck warns about this (SC2155).
59. Cron job runs twice at once
A sync script runs every 5 minutes from cron. Sometimes it takes 8 minutes and two copies corrupt the data. What is a robust fix?
- A.Check for a PID file with [ -f ] and create it if absent
- B.Run it every 10 minutes instead of every 5
- C.Wrap it with flock -n /run/sync.lock so a second copy exits
- D.Add sleep 300 at the end of the script
Show answer
Answer: C. Wrap it with flock -n /run/sync.lock so a second copy exits
flock takes a kernel lock on the file, which is released automatically when the process exits, even after a crash. A manual "check then create" PID file has a race and leaves stale locks. Changing the schedule only hides the problem until a slower run.
60. Command found, but not under sudo
kubectl is in /opt/tools/bin, which is on your PATH. Running "sudo kubectl get nodes" gives "command not found". Why?
- A.kubectl refuses to run as the root user
- B.sudo uses secure_path from sudoers, not your PATH
- C.sudo only runs commands in /usr/sbin
- D.Root needs its own copy of the kubeconfig first
Show answer
Answer: B. sudo uses secure_path from sudoers, not your PATH
Many distros set "Defaults secure_path=..." in sudoers, so sudo replaces PATH with a fixed safe list. Use the full path, add the folder to secure_path, or install the tool into /usr/local/bin. The kubeconfig issue would show a different error.