awkProcess text by columns: filter, sum, count and report
awk splits every line into fields ($1, $2 ... $NF) and runs a small program on them: print chosen columns, keep lines where a field meets a condition, sum or average a column, or count by key with arrays. Most 'answer this from the logs' interview questions are one awk line.
awk [-F SEP] [-v VAR=VALUE] 'PATTERN { ACTION } ...' [FILE]...nginx's combined log is long and noisy. You want the client, status and path, the 5xx lines only, and the busiest clients.
awkA small language for line-and-field processing. On Ubuntu it is mawk, on Amazon Linux and RHEL gawk.
awk runs a loop you don't have to write. Before any input it runs the BEGIN blocks. Then for each record (a line, unless RS says otherwise) it stores the text in $0, increments NR, splits it on FS into $1 ... $NF, and tests each pattern in order; when a pattern is true (or absent) its action runs. A pattern with no action prints the line. Fields are strings that become numbers when used as numbers, which is why $5+0 turns '92%' into 92. Variables need no declaration and start as 0 or empty; arrays are hash maps keyed by strings, so counting by IP or status is just n[$1]++. After the last record the END blocks run, which is where totals and averages are printed. Assigning to a field rebuilds $0 with OFS between fields. The whole program is one process reading the file once, so awk is often faster than a chain of grep | cut | sort for the same job.
awk reads one record at a time, splits it into fields, runs every pattern-action pair, and prints totals in END.
BEGIN blocks run once, before the first line is read. It's where you set FS or OFS, initialise counters or print a header.
-F SEPField separator: -F: for /etc/passwd, -F, for CSV, -F'\t' for TSV; can be a regex-v name=valPass a shell value in safely: awk -v limit="$MAX" '$3 > limit'$0, $1..$NFThe whole line, each field, and the last fieldNR / NFRecord (line) number overall / number of fields on this lineBEGIN { }Runs once before input: set FS, OFS, print a headerEND { }Runs once after input: print totals, averages, arrays/regex/Pattern: lines matching the regex; $7 ~ /re/ tests one fieldprintfFormatted output: printf "%-14s %6.1f\n", $1, $2arr[key]++Associative array: count or sum by any keyOFSOutput field separator used between print arguments (default space)length() / substr()String functions; also split(), sub(), gsub(), tolower(), index()nextStop processing this line and move to the next oneawk "{print $2}" fileIn double quotes the shell replaces $2 (usually with nothing) before awk starts, so awk prints whole lines or fails in confusing ways.
awk '{print $2}' fileawk '{print $1 $2}' file (expecting two columns)Without a comma the two fields are concatenated into one string: web-0110.0.2.11.
awk '{print $1, $2}' file # or printf "%s\t%s\n", $1, $2awk '$3 > $LIMIT' file to use a shell variableSplicing shell variables into the program breaks on quotes and is an injection risk with untrusted input. Inside awk, $LIMIT means 'field number LIMIT'.
awk -v limit="$LIMIT" '$3 > limit' fileawk -F, '{print $3}' export.csv on a real CSVQuoted fields like "Lopez, Maria" contain commas, so every column after them shifts. awk has no CSV parser (gawk 5.3 added --csv).
python3 -c 'import csv,sys; [print(r[2]) for r in csv.reader(sys.stdin)]' < export.csvTry these on a Linux machine of your own. Each task has a hint and a solution.
Print the usernames and home directories of all users in /etc/passwd.
Show only the filesystems in df -h output that are 50% full or more, keeping the header.
Count lines per log level in ~/logs/app.log (the level is the second field).
Print the total size in MB of all files in ~/logs using ls -l and awk.
Print every line of ~/data/servers.txt with a line number and only the first and last fields.
awkProcess text by columns: filter, sum, count and report
awk splits every line into fields ($1, $2 ... $NF) and runs a small program on them: print chosen columns, keep lines where a field meets a condition, sum or average a column, or count by key with arrays. Most 'answer this from the logs' interview questions are one awk line.
awk [-F SEP] [-v VAR=VALUE] 'PATTERN { ACTION } ...' [FILE]...nginx's combined log is long and noisy. You want the client, status and path, the 5xx lines only, and the busiest clients.
awkA small language for line-and-field processing. On Ubuntu it is mawk, on Amazon Linux and RHEL gawk.
awk runs a loop you don't have to write. Before any input it runs the BEGIN blocks. Then for each record (a line, unless RS says otherwise) it stores the text in $0, increments NR, splits it on FS into $1 ... $NF, and tests each pattern in order; when a pattern is true (or absent) its action runs. A pattern with no action prints the line. Fields are strings that become numbers when used as numbers, which is why $5+0 turns '92%' into 92. Variables need no declaration and start as 0 or empty; arrays are hash maps keyed by strings, so counting by IP or status is just n[$1]++. After the last record the END blocks run, which is where totals and averages are printed. Assigning to a field rebuilds $0 with OFS between fields. The whole program is one process reading the file once, so awk is often faster than a chain of grep | cut | sort for the same job.
awk reads one record at a time, splits it into fields, runs every pattern-action pair, and prints totals in END.
BEGIN blocks run once, before the first line is read. It's where you set FS or OFS, initialise counters or print a header.
-F SEPField separator: -F: for /etc/passwd, -F, for CSV, -F'\t' for TSV; can be a regex-v name=valPass a shell value in safely: awk -v limit="$MAX" '$3 > limit'$0, $1..$NFThe whole line, each field, and the last fieldNR / NFRecord (line) number overall / number of fields on this lineBEGIN { }Runs once before input: set FS, OFS, print a headerEND { }Runs once after input: print totals, averages, arrays/regex/Pattern: lines matching the regex; $7 ~ /re/ tests one fieldprintfFormatted output: printf "%-14s %6.1f\n", $1, $2arr[key]++Associative array: count or sum by any keyOFSOutput field separator used between print arguments (default space)length() / substr()String functions; also split(), sub(), gsub(), tolower(), index()nextStop processing this line and move to the next oneawk "{print $2}" fileIn double quotes the shell replaces $2 (usually with nothing) before awk starts, so awk prints whole lines or fails in confusing ways.
awk '{print $2}' fileawk '{print $1 $2}' file (expecting two columns)Without a comma the two fields are concatenated into one string: web-0110.0.2.11.
awk '{print $1, $2}' file # or printf "%s\t%s\n", $1, $2awk '$3 > $LIMIT' file to use a shell variableSplicing shell variables into the program breaks on quotes and is an injection risk with untrusted input. Inside awk, $LIMIT means 'field number LIMIT'.
awk -v limit="$LIMIT" '$3 > limit' fileawk -F, '{print $3}' export.csv on a real CSVQuoted fields like "Lopez, Maria" contain commas, so every column after them shifts. awk has no CSV parser (gawk 5.3 added --csv).
python3 -c 'import csv,sys; [print(r[2]) for r in csv.reader(sys.stdin)]' < export.csvTry these on a Linux machine of your own. Each task has a hint and a solution.
Print the usernames and home directories of all users in /etc/passwd.
Show only the filesystems in df -h output that are 50% full or more, keeping the header.
Count lines per log level in ~/logs/app.log (the level is the second field).
Print the total size in MB of all files in ~/logs using ls -l and awk.
Print every line of ~/data/servers.txt with a line number and only the first and last fields.