uniqCollapse and count repeated adjacent lines
uniq collapses runs of identical adjacent lines into one, and with -c it counts them. Put sort in front and it becomes a group-by: sort | uniq -c | sort -rn is the most famous pipeline in log analysis.
uniq [OPTION]... [INPUT [OUTPUT]]There were 164 failed logins (HTTP 401). Is one client brute-forcing the login endpoint?
awk '{print $1}' access.logExtract the key you want to count, one per line: here the client IP.
uniq reads its input line by line and keeps exactly one previous line in memory. For each new line it compares with the previous one: if equal, it increments a counter; if different, it prints the previous line (with the count for -c) and starts a new run. That's why it uses constant memory on any size of input, and also why it cannot find duplicates that aren't next to each other: it never looks further back than one line. sort solves that by making all equal lines adjacent, at the cost of reading everything first. Options change what 'equal' means: -i ignores case, -f N skips the first N fields, -s N skips N characters and -w N compares only the first N characters. -d prints only lines that were repeated, -u only those that never were. If the input is already grouped, like a log in time order, you can skip the sort.
Five processes connected by pipes run at the same time; sort is the one step that must read everything before it can write.
The shell creates four pipes and starts all five commands at once. Each process's stdout is wired to the next one's stdin; none of them knows about the others.
-cPrefix each line with the number of times it occurred in a row-dOnly print lines that were repeated (one copy each)-uOnly print lines that were never repeated-iIgnore case when comparing-f NSkip the first N fields when comparing (e.g. a timestamp)-s NSkip the first N characters when comparing-w NCompare only the first N characters-DPrint every copy of the repeated lines (GNU)uniq -c access_ips.txt on unsorted inputuniq only merges adjacent duplicates, so the counts are split across many runs and mostly 1.
sort access_ips.txt | uniq -cgrep ERROR app.log | sort | uniq -c (every count is 1)Each line has a unique timestamp and request ID, so no two lines are identical. Remove the varying parts before counting.
grep ERROR app.log | cut -d' ' -f2- | sed 's/req=[^ ]* //' | sort | uniq -csort | uniq -c | sort -n | head for the top itemsAscending order puts the rarest lines first, so head shows the bottom of the list.
sort | uniq -c | sort -rn | headuniq input.txt input.txt to dedupe in placeThe second argument is the OUTPUT file. uniq opens it for writing, truncating the input before it has been read.
sort -u -o input.txt input.txtTry these on a Linux machine of your own. Each task has a hint and a solution.
Count how many times each log level (the 2nd field) appears in ~/logs/app.log, most frequent first.
Print only the lines that appear more than once in ~/data/hosts.txt.
Count the distinct login shells in /etc/passwd and how many users use each.
Show how many requests per minute were logged in the last 5 minutes of ~/logs/app.log.
uniqCollapse and count repeated adjacent lines
uniq collapses runs of identical adjacent lines into one, and with -c it counts them. Put sort in front and it becomes a group-by: sort | uniq -c | sort -rn is the most famous pipeline in log analysis.
uniq [OPTION]... [INPUT [OUTPUT]]There were 164 failed logins (HTTP 401). Is one client brute-forcing the login endpoint?
awk '{print $1}' access.logExtract the key you want to count, one per line: here the client IP.
uniq reads its input line by line and keeps exactly one previous line in memory. For each new line it compares with the previous one: if equal, it increments a counter; if different, it prints the previous line (with the count for -c) and starts a new run. That's why it uses constant memory on any size of input, and also why it cannot find duplicates that aren't next to each other: it never looks further back than one line. sort solves that by making all equal lines adjacent, at the cost of reading everything first. Options change what 'equal' means: -i ignores case, -f N skips the first N fields, -s N skips N characters and -w N compares only the first N characters. -d prints only lines that were repeated, -u only those that never were. If the input is already grouped, like a log in time order, you can skip the sort.
Five processes connected by pipes run at the same time; sort is the one step that must read everything before it can write.
The shell creates four pipes and starts all five commands at once. Each process's stdout is wired to the next one's stdin; none of them knows about the others.
-cPrefix each line with the number of times it occurred in a row-dOnly print lines that were repeated (one copy each)-uOnly print lines that were never repeated-iIgnore case when comparing-f NSkip the first N fields when comparing (e.g. a timestamp)-s NSkip the first N characters when comparing-w NCompare only the first N characters-DPrint every copy of the repeated lines (GNU)uniq -c access_ips.txt on unsorted inputuniq only merges adjacent duplicates, so the counts are split across many runs and mostly 1.
sort access_ips.txt | uniq -cgrep ERROR app.log | sort | uniq -c (every count is 1)Each line has a unique timestamp and request ID, so no two lines are identical. Remove the varying parts before counting.
grep ERROR app.log | cut -d' ' -f2- | sed 's/req=[^ ]* //' | sort | uniq -csort | uniq -c | sort -n | head for the top itemsAscending order puts the rarest lines first, so head shows the bottom of the list.
sort | uniq -c | sort -rn | headuniq input.txt input.txt to dedupe in placeThe second argument is the OUTPUT file. uniq opens it for writing, truncating the input before it has been read.
sort -u -o input.txt input.txtTry these on a Linux machine of your own. Each task has a hint and a solution.
Count how many times each log level (the 2nd field) appears in ~/logs/app.log, most frequent first.
Print only the lines that appear more than once in ~/data/hosts.txt.
Count the distinct login shells in /etc/passwd and how many users use each.
Show how many requests per minute were logged in the last 5 minutes of ~/logs/app.log.