AWK Notes

AWK fits naturally after grep and sed. Grep finds matching lines, sed edits streams, and AWK lets you work with columns, conditions, totals, and reports without opening a spreadsheet or writing a full script.

1. What is AWK?

AWK = Pattern scanning and text processing language.

Full Form: Named after creators Aho, Weinberger, and Kernighan (1977, Bell Labs).

One-liner definition:
AWK reads files line-by-line, splits each line into fields, and lets you process/filter/transform data using patterns and actions.

2. Why AWK?

3. When to Use AWK?

• Parsing structured files (CSV, TSV, logs, /etc/passwd)
• Extracting specific columns or fields
• Filtering rows based on conditions
• Quick stats (sum, count, avg, min, max)
• Reformatting output
• Preprocessing data before piping to other tools

4. Who Needs AWK?

5. AWK vs Others

6. General Syntax

awk [OPTIONS] 'PATTERN {ACTION}' filename

If no PATTERN: Action runs on every line.

If no ACTION: Prints entire matching line.

7. Key Options

8. Field Separator Deep Dive

Default separator: Whitespace (space/tab).

# CSV (comma)
awk -F, '{print $1}' file.csv

# TSV (tab)
awk -F'\t' '{print $1}' file. tsv

# Colon (like /etc/passwd)
awk -F:  '{print $1}' /etc/passwd

# Pipe
awk -F'|' '{print $1}' file. psv

# Multiple separators (comma OR semicolon)
awk -F'[,;]' '{print $1}' file.txt

9. Built-in Variables

10. Special Blocks

BEGIN Block

Runs before processing any lines.

awk 'BEGIN {print "Starting..."} {print $0}' file.txt

Use cases: Print headers, initialize variables, set FS.

awk 'BEGIN {FS=","; print "Name,Salary"} NR>1 {print $2,$4}' emp.csv

END Block

Runs after all lines processed.

awk '{sum+=$1} END {print "Total:", sum}' numbers.txt

Use cases: Print totals, averages, summaries.

11. Operators

Comparison

Logical

Regex Match

# Name contains "a"
awk -F, '$2 ~ /a/' emp.csv

# Department NOT marketing
awk -F, '$3 ! ~ /marketing/' emp.csv

12. Associative Arrays

Key-value storage - like Python dictionaries.

# Count per department
awk -F, 'NR>1 {dept[$3]++} END {for(d in dept) print d, dept[d]}' emp.csv

# Sum salary per city
awk -F, 'NR>1 {city[$5]+=$4} END {for(c in city) print c, city[c]}' emp.csv

Syntax:

array[key] = value    # assign
array[key]++          # increment
for(var in array)     # loop through keys

13. Useful Functions

String Functions

Math Functions

14. Output Formatting

print (simple)

awk -F, '{print $2, $4}' emp.csv
# Output: john 75000

printf (formatted)

awk -F, '{printf "%-10s %d\n", $2, $4}' emp.csv
# Output: john       75000

15. Sample Data: employees.csv

16. Practice Drills - Q&A

Basic Extraction

awk -F, '{print $2}' emp.csv

awk -F, '{print $2, $4}' emp.csv

awk -F, 'NR==1' emp.csv

awk -F, 'NR>1' emp.csv

awk -F, '{print $NF}' emp.csv

awk -F, '{print $(NF-1)}' emp.csv

Filtering

awk -F, '$3=="engineering"' emp.csv

awk -F, 'NR>1 && $4>70000 {print $2}' emp.csv

awk -F, '$6=="inactive"' emp.csv

awk -F, '$5=="chicago" || $5=="boston"' emp.csv

awk -F, '$3=="finance" && $4>70000 {print $2}' emp.csv

awk -F, '$2 ~ /a/ {print $2}' emp.csv

awk -F, '$3 ! ~ /marketing/' emp.csv

Formatting Output

awk -F, 'NR>1 {print $2":"$4}' emp.csv

awk -F, 'NR>1 {print $2"\t"$3}' emp.csv

awk -F, 'NR>1 {print toupper($2)}' emp.csv

awk -F, 'NR>1 {printf "%-12s %8d\n", $2, $4}' emp.csv

Counting

awk -F, 'NR>1 {count++} END {print count}' emp.csv

awk -F, 'NR>1 {dept[$3]++} END {for(d in dept) print d, dept[d]}' emp.csv

awk -F, 'NR>1 {city[$5]++} END {for(c in city) print c, city[c]}' emp.csv

awk -F, 'NR>1 {status[$6]++} END {for(s in status) print s, status[s]}' emp.csv

Math & Aggregation

awk -F, 'NR>1 {sum+=$4} END {print sum}' emp.csv

awk -F, 'NR>1 {sum+=$4; count++} END {print sum/count}' emp.csv

awk -F, 'NR>1 && $4>max {max=$4} END {print max}' emp.csv

awk -F, 'NR>1 {if($4>max){max=$4; name=$2}} END {print name}' emp.csv

awk -F, 'NR==2 {min=$4} NR>2 && $4

awk -F, 'NR>1 {dept[$3]+=$4} END {for(d in dept) print d, dept[d]}' emp.csv

awk -F, 'NR>1 {sum[$5]+=$4; count[$5]++} END {for(c in sum) print c, sum[c]/count[c]}' emp.csv

awk -F, 'NR>1 {sal[NR]=$4; name[NR]=$2; sum+=$4; count++} END {avg=sum/count; for(i in sal) if(sal[i]>avg) print name[i], sal[i]}' emp.csv

awk -F, 'NR>1 {print $4, $2}' emp.csv | sort -rn | head -3

awk -F, 'NR>1 {dept[$3]+=$4} END {for(d in dept) if(dept[d]>max){max=dept[d]; top=d} print top, max}' emp.csv

awk -F, 'NR>1 {print NR-1, $2, $4}' emp.csv

awk -F, 'NR>1 && ! seen[$3]++ {print $3}' emp.csv

awk -F, 'NR>1 {print $2, ($4>75000 ? "HIGH" :  "LOW")}' emp.csv

awk -F, 'NR>1 {print $3, $2}' emp.csv

# List all usernames
awk -F:  '{print $1}' /etc/passwd

# Find users with bash shell
awk -F: '$7 ~ /bash/ {print $1}' /etc/passwd

# Count requests per IP
awk '{ip[$1]++} END {for(i in ip) print ip[i], i}' access.log | sort -rn | head -10

# Find 404 errors
awk '$9==404 {print $7}' access.log

# Sum sales column
awk -F, 'NR>1 {sum+=$5} END {print "Total Sales:", sum}' sales.csv

# Find above-average performers
awk -F, 'NR>1 {sum+=$5; count++; data[NR]=$0} END {avg=sum/count; for(i in data) {split(data[i],a,","); if(a[5]>avg) print a[1], a[5]}}' sales.csv