Published on

Sysadmin Linux Troubleshooting Cheatsheet


This is a one pager of all the commands I use when I have to troubleshoot problems. This post will be updated as time goes by.

Curl / Web Response Times

Template file:

$ cat curl-format.txt
time_namelookup:  %{time_namelookup}\n
time_connect:  %{time_connect}\n
time_appconnect:  %{time_appconnect}\n
time_pretransfer:  %{time_pretransfer}\n
time_redirect:  %{time_redirect}\n
time_starttransfer:  %{time_starttransfer}\n
time_total:  %{time_total}\n

The host header, source addres, destination address:

$ curl -sk -w "@curl-format.txt" -o /dev/null -H "Host:" -L

time_namelookup:  0.012178
time_connect:  0.012225
time_appconnect:  0.062149
time_pretransfer:  0.062175
time_redirect:  0.000172
time_starttransfer:  0.125631
time_total:  0.125849

MTR / Network Latencies / Packetloss

No dns, TCP, counts, port, source address, destination address:

$ mtr -n -T -c 10 --port 443 --report
Start: Sun Feb 10 19:04:50 2019
HOST: my-internet-gatewewy         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|--              0.0%    10    0.3   0.3   0.3   0.3   0.0
  2.|--              0.0%    10    0.3   0.3   0.3   0.3   0.0
  3.|--              0.0%    10    0.3   0.3   0.3   0.3   0.0


No dns, TCP, port, source address, destination address:

$ traceroute -T -n -p 443 -s; done
traceroute to (, 30 hops max, 60 byte packets
 1   0.322 ms  0.291 ms  0.224 ms
 2  179.090 ms  179.022 ms  179.023 ms

Connection flow: Thanks to

Consider two programs attempting a socket connection (call them a and b). Both set up sockets and transition to the LISTEN state. Then one program (say a) tries to connect to the other (b). a sends a request and enters the SYN_SENT state, and b receives the request and enters the SYN_RECV state. When b acknowledges the request, they enter the ESTABLISHED state, and do their business. Now a couple of things can happen:

    a wishes to close the connection, and enters FIN_WAIT1. b receives the FIN request, sends an ACK (then a enters FIN_WAIT2), enters CLOSE_WAIT, tells a it is closing down and the enters LAST_ACK. Once a acknowledges this (and enters TIME_WAIT), b enters CLOSE. a waits a bit to see if anythings is left, then enters CLOSE.
    a and b have finished their business and decide to close the connection (simultaneous closing). When a is in FIN_WAIT, and instead of receiving an ACK from b, it receives a FIN (as b wishes to close it as well), a enters CLOSING. But there are still some messages to send (the ACK that a is supposed to get for its original FIN), and once this ACK arrives, a enters TIME_WAIT as usual.

Active Connections:

$ netstat -n -A  inet | grep -v ""

Established Connections:

$ netstat -nputw | grep ESTABLISHED
$ netstat -antp | grep :3306 | grep ESTABLISHED

Time Wait Connections:

$ netstat -antp | grep TIME_WAIT

How many connections:

$ wc -l /proc/net/tcp

Listing Open files per Port:

$ lsof -i:3306

Listing Open files per User:

$ lsof -u glassfish

Network Throughput

You can test the network throughput between two linux hosts with iperf:

On side-a we will start the server in TCP mode:

$ iperf -s
Server listening on TCP port 5001
TCP window size:  128 KByte (default)

On side-b we will start the client, which connects to the server:

$ iperf -c
Client connecting to, TCP port 5001
TCP window size: 43.8 KByte (default)
[  3] local port 43870 connected with port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.4 MBytes  9.54 Mbits/sec

We can also run this in UDP mode where the server will run iperf -s -u and the client will run iperf -c host-address -u


Thank You

Thanks for reading, if you like my content, feel free to check out my website, and subscrube to my newsletter or follow me at @ruanbekker on Twitter.

Buy Me A Coffee