When logs do not help and an application behaves unexpectedly, strace and ltrace are the safety net. Strace captures system calls — every interaction between a process and the kernel, from opening files to network communication to memory allocation. Ltrace traces library function calls. These tools reveal problems invisible in logs: missing files, denied permissions, network timeouts, or unexpected system behavior.
strace¶
strace ls /tmp # trace an entire command
strace -p 1234 # attach to a running process
strace -e open,read,write ls /tmp # filter specific syscalls only
strace -c ls /tmp # summary — syscall statistics
Examples¶
strace -e openat ./app 2>&1 | grep error # find file open errors
strace -e write -p $(pgrep myapp) # watch what the process writes
strace -c -p $(pgrep myapp) # where it spends time — syscall profiling
strace -e network -p $(pgrep myapp) # network operations only
strace -T -e read,write -p $(pgrep myapp) # with timestamps for each syscall
A typical debugging workflow: first run strace -c to identify which syscalls consume the most time, then use strace -e with a specific filter for detailed analysis. When investigating network issues, the connect,sendto,recvfrom filter shows connection establishment and communication.
ltrace¶
ltrace ./my-program # trace library calls
ltrace -e malloc+free ./my-program # track memory allocations
ltrace is useful for debugging memory leaks (tracking malloc/free pairs), analyzing library call performance, and understanding how an application interacts with external libraries.
- -f = trace child processes (fork/exec)
- -s 1000 = display longer strings in arguments
- -T = show time spent in each call
- Use briefly in production — strace significantly slows down the traced process
Ultimate debugging¶
strace saves hours of troubleshooting by showing exactly what a process does at the operating system level. Combine with perf for performance analysis and gdb for code-level debugging.