A SIEM alert points to unusual Apache requests with encoded paths and payload-like strings. Before you open heavier tools, you need a quick first pass: which IPs are involved, what URLs they touched, and where the timeline starts.
Why this works: Filter the massive log for attack signatures: path traversal (%2f), SQL injection (union select), XSS (<script>), and LFI (/etc/passwd). This immediately narrows millions of lines to just the malicious requests.
Why this works: Extract source IPs from each suspicious line, count occurrences, and sort descending. The top IPs become your first review list because normal users should not trigger these signatures repeatedly.
Why this works: Focus on one source IP and extract the URLs it requested. This gives you a readable sequence of reconnaissance, exploit attempts, and possible data access.
Why this works: Extract timestamps ($4), URLs ($7), and status codes ($9) for the attacker. Status 200 on sensitive paths = successful data access. This defines your breach scope for the incident report.
Step 5sed
sed -n '/203.0.113.42/s/.*\[\(.*\)\].*/\1/p' /var/log/apache2/access.log | head -1
Why this works: Extract the first timestamp tied to the suspicious source. That gives the incident report a clear starting point for log preservation, notification checks, and follow-up review.
Preserve original logs before analysis - forensic integrity matters
Use grep -c for quick counts before deep-diving
Document every command during IR for the post-incident report
Correlate timestamps across multiple log sources for full timeline
Scenario 2
Credential Leak Sweep Across Codebase
Situation
A developer accidentally pushed credentials to a public repository. After the keys are revoked, you still need to scan the codebase for hardcoded secrets such as API keys, passwords, tokens, and private keys.
Why this works: Recursively search config and source files for secret patterns. The -i catches case variations. The regex matches both = and : assignment styles across all common languages.
Step 2grep
grep -rnE "AKIA[0-9A-Z]{16}" ./
Why this works: AWS access key IDs commonly start with AKIA followed by 16 uppercase alphanumeric characters. It is a strong signal, but still review each match before reporting it.
Why this works: Extract unique filenames from matches using awk's field separator. This creates your remediation checklist - the exact files that need secrets removed or rotated.
Step 4sed
sed -i 's/AKIA[0-9A-Z]\{16\}/REDACTED_AWS_KEY/g' config.py
Why this works: The -i flag edits files in place, which is useful after you have confirmed the matches and created a backup. Redaction helps clean the repository, but exposed credentials still need rotation.
Why this works: Find hardcoded secrets while excluding comments and env-var references. The negative lookahead skips legitimate patterns like os.environ - reducing false positives.
Use AWS IAM Access Analyzer to find unused or exposed keys
Scenario 3
DDoS Attack Traffic Analysis
Situation
Your e-commerce platform is crawling during Black Friday. The NOC suspects a DDoS attack mixed with legitimate traffic. You need to analyze nginx access logs to separate bots from real customers, identify attack patterns, and generate a blocklist - without taking the server offline.
Why this works: Count requests per IP and sort descending. Normal users might make dozens of requests in a session, while automated traffic can produce thousands. This gives you a first review list instead of guessing.
Step 2awk
awk '{ips[$1]++} END {for (ip in ips) if (ips[ip] > 1000) print ips[ip], ip}' /var/log/nginx/access.log | sort -rn
Why this works: Pure awk single-pass solution - faster on massive files. Builds an associative array counting hits per IP, then filters for 1000+ threshold. Your DDoS source list, generated quickly.
Why this works: Analyze User-Agent strings from suspects. DDoS bots use identical or empty User-Agents. Legitimate users show diverse browser strings. This confirms bot vs human traffic.
Step 4awk
awk '{split($4,t,":"); hours[t[2]]++} END {for (h in hours) printf "%s:00 - %d requests\n", h, hours[h]}' /var/log/nginx/access.log | sort
Why this works: Build hourly traffic histogram. DDoS attacks show dramatic spikes in specific hours. This identifies exact attack windows for your incident timeline and ISP abuse reports.
Step 5awk+sed
awk '{ips[$1]++} END {for (ip in ips) if (ips[ip] > 1000) print ip}' /var/log/nginx/access.log | sed 's/^/deny /;s/$/;/' > /etc/nginx/blocklist.conf
Why this works: Generate nginx blocklist directly from analysis. Each offending IP formatted as deny IP; - ready to include in your config. This is analysis-to-mitigation in one pipeline.
Full pipeline
awk '{ips[$1]++} END {for (ip in ips) if (ips[ip]>500) print ip}' access.log | sed 's/^/deny /;s/$/;/' > blocklist.conf
Practical checks
Set up rate limiting in nginx (limit_req_zone) proactively
Use fail2ban jails to auto-block repeat offenders
Keep 30+ days of access logs for forensic depth
Prefer awk over grep for large files - single-pass processing is faster
Scenario 4
SSH Brute-Force Review and Response
Situation
Your dashboard shows 50,000+ failed SSH logins in 24 hours across your fleet. Management wants a full forensic report: attacker IPs, targeted usernames, timing patterns, and an automated defense mechanism - using grep, sed, and awk.
Why this works: Extract source IPs from failed attempts. PCRE's \K resets the match start, extracting only the IP after 'from'. Top attackers identified clearly - your threat actor list.
Why this works: Extract targeted usernames. Awk iterates fields to find the word after 'for'. Common targets: root, admin, ubuntu, test. If they're trying YOUR specific usernames - this is targeted, not opportunistic.
Step 3awk
awk '/Failed password/ {split($0,a,":"); hour=substr(a[1],length(a[1])-4); hours[hour]++} END {for(h in hours) print h, hours[h]}' /var/log/auth.log | sort
Why this works: Hourly attack profile. Botnets attack in waves - identifying peak hours helps you decide on stricter controls during high-risk windows or detect coordinated campaigns.
Why this works: Attack intensity by date. Clean formatting with sed for reporting. Spikes on specific dates correlate with new botnet campaigns or your IP appearing on attacker target lists.
Step 5awk
awk '/Failed password/ {for(i=1;i<=NF;i++) if($i=="from") ips[$(i+1)]++} END {for(ip in ips) if(ips[ip]>50) printf "iptables -A INPUT -s %s -j DROP\n", ip}' /var/log/auth.log
Why this works: This parses logs, counts failures per IP, and formats block rules for sources over the threshold. Review the generated rules before applying them, especially on shared or customer-facing systems.
Full pipeline
grep "Failed password" /var/log/auth.log | awk '{for(i=1;i<=NF;i++) if($i=="from") ips[$(i+1)]++} END {for(ip in ips) if(ips[ip]>100) print ip}' | while read ip; do iptables -A INPUT -s $ip -j DROP; echo "Blocked: $ip"; done
Practical checks
Disable password auth entirely - use SSH keys only
Deploy fail2ban for automatic brute-force mitigation
Move SSH to a non-standard port to reduce noise
Use AllowUsers/AllowGroups in sshd_config to limit access
Scenario 5
Compliance Audit: PII Detection in Logs
Situation
GDPR/CCPA audit incoming. The compliance team suspects application logs contain PII - emails, phone numbers, credit cards, SSNs - violating data protection regulations. You need to scan 200GB+ of log archives across hundreds of files.
Why this works: Scan for email addresses. The -o prints only matched emails, -n gives line numbers, -P enables Perl regex for accuracy. Every match is a potential GDPR violation that needs redaction.
Why this works: Detect SSN patterns (123-45-6789, 123.45.6789, 123456789). Word boundaries \b prevent matching random number strings. Every hit requires immediate redaction and breach assessment.
Why this works: Find potential credit card numbers (13-16 digits) while filtering out timestamps and session IDs. This dramatically reduces false positives while catching PCI-DSS violations.
Step 4sed
sed -i 's/[a-zA-Z0-9._%+-]\+@[a-zA-Z0-9.-]\+\.[a-zA-Z]\{2,\}/[EMAIL_REDACTED]/g' /var/log/app/app.log
Why this works: In-place PII redaction. Replaces all emails with [EMAIL_REDACTED], bringing logs into compliance without losing the surrounding context. Audit-safe remediation.
Why this works: Compliance report: PII count per file, sorted by severity. This gives the audit team a prioritized remediation list - fix worst offenders first.
Full pipeline
grep -rlE "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}|[0-9]{3}-[0-9]{2}-[0-9]{4}" /var/log/app/ | tee /tmp/pii_audit.log
Practical checks
Sanitize PII at the application level - never log it in the first place
Implement log masking middleware in your stack
Schedule regular automated PII scans as part of CI/CD
Maintain audit trail of when PII was discovered and redacted
Scenario 6
WAF Log Analysis: New Attack Pattern Review
Situation
Your WAF logs show suspicious requests that don't match known signatures. The security team suspects a new exploit attempt targeting your custom API. You need to identify the new attack pattern, decode the payload, and create custom detection rules - fast.
Why this works: Filter for error responses (4xx/5xx excluding noise) and extract URL + status code. Unusual error concentrations on specific endpoints reveal exploitation targets.
Why this works: Find abnormally long URIs. Zero-day exploits stuff encoded payloads into URLs. Normal: <200 chars. Anything significantly longer is suspicious and needs inspection.
Why this works: Isolate URL-encoded payloads on the targeted endpoint. Sed breaks URLs at each percent-encoded sequence, which makes repeated payload structure easier to inspect.
Step 4awk
awk '/api\/v2\/upload/ {split($4,t,":"); mins[t[2]":"t[3]]++} END {for(m in mins) if(mins[m]>10) print mins[m], m}' /var/log/waf/requests.log | sort -rn
Why this works: Minute-by-minute timeline on the target endpoint. Exploit tools produce bursts in tight windows - this confirms automated exploitation vs manual probing.
Why this works: Generate a draft ModSecurity rule list from the analysis. Treat the output as a review artifact first, then test any block rule before putting it in front of production traffic.
Start WAF in detection mode, switch to blocking after tuning
Correlate WAF logs with app logs for full attack context
Use OWASP ModSecurity CRS as your baseline ruleset
Share IOCs with your ISAC community to help the industry
Scenario 7
Mass Server Config Hardening
Situation
A pentest report reveals 150 servers have insecure SSH configs: root login enabled, password auth on, weak ciphers. You need to audit and fix every sshd_config simultaneously - without breaking access for your team.
Why this works: Audit current state. Grep shows which lines contain security directives and their values. The -n flag gives line numbers for precise manual review before making changes.
Step 2sed
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
Why this works: Disable root login. The ^#* regex handles both commented and active lines, replacing with the secure value. One command handles every possible state of the directive.
Step 3sed
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
Why this works: Force key-based auth by disabling passwords. The same repeatable pattern handles both commented and active settings, but test access before restarting SSH on remote systems.
Why this works: Audit cipher configs before modifying. The BEFORE: prefix creates clear audit trail documentation. Essential for change management compliance and rollback planning.
Why this works: Remove old cipher and MAC lines, then append stronger suites. Treat the exact list as policy-dependent and verify it against your OpenSSH version and the benchmark your team follows.
Always sshd -t to validate config before reloading
Keep an active SSH session open while making changes - your safety net
Use Ansible/Puppet for fleet-wide config management
Follow CIS Benchmark for SSH - it covers everything systematically
Scenario 8
Kubernetes Pod Failure Triage
Situation
Your K8s cluster is experiencing cascading pod failures during a rolling deployment. As the on-call SRE, you need to identify failing pods, parse their JSON logs, determine if it's a code bug or infrastructure issue - all from kubectl output processed with grep, sed, and awk.
Step-by-step walkthrough
Step 1awk
kubectl get pods -A | awk '$4 != "Running" && $4 != "Completed" && NR>1 {print $1, $2, $4, $5}'
Why this works: Filter non-healthy pods across all namespaces. Awk skips healthy states and shows the pods that need attention first, including restart counts.
Why this works: Parse simple JSON log lines with awk by using a quote as the field separator. For deeply nested JSON, move to jq, but this works for quick flat log messages.
Why this works: Find resource-hungry pods causing cluster pressure. Parsing metrics with awk filters for pods consuming >500m CPU or >1Gi memory - these are your optimization targets or OOM culprits.
Step 5sed+grep
kubectl describe pod api-server-7d9f | sed -n '/Events:/,$ p' | grep -v 'Normal' | awk '{$1=$2=""; print}'
Why this works: Extract Warning events from pod description. Sed captures the Events section, grep removes Normal noise. What remains are actual problems - ImagePullBackOff, CrashLoopBackOff, OOMKilled.
Full pipeline
kubectl get pods -A | awk '$4=="CrashLoopBackOff" {print $1, $2}' | while read ns pod; do echo "=== $ns/$pod ==="; kubectl logs -n $ns $pod --tail=5 2>&1 | grep -iE 'error|fatal'; done
Practical checks
Use structured JSON logging in all containerized apps
Set CPU/memory requests AND limits on every pod
Use kubectl logs --previous for crashed container logs
Implement liveness + readiness probes for early failure detection
Scenario 9
Malware Persistence Mechanism Hunt
Situation
An EDR tool flagged a server for suspicious outbound connections. The IR team wants a first pass across common persistence locations: cron jobs, modified binaries, unusual services, and unauthorized SSH keys.
Why this works: Search systemd units for execution directives. Filtering out /usr/ removes legitimate system services. What remains are custom or recently added units - potential malware persistence.
Step 2awk+grep
for user in $(awk -F: '$7 !~ /(nologin|false)/ {print $1}' /etc/passwd); do echo "=== $user ==="; crontab -u $user -l 2>/dev/null | grep -v '^#'; done
Why this works: Check cron jobs for all users with login shells. Awk extracts usernames from /etc/passwd, then each user's crontab is shown without comments for easier review.
Why this works: Search all cron locations for suspicious commands: downloaders (curl/wget), reverse shells (nc -e, bash -i, /dev/tcp), remote execution. These are classic attack persistence patterns.
Step 4awk
find /usr/bin /usr/sbin -mtime -7 -type f | while read f; do rpm -qf "$f" 2>/dev/null || dpkg -S "$f" 2>/dev/null || echo "UNPACKAGED: $f"; done | awk '/UNPACKAGED/'
Why this works: Find recently modified system binaries not belonging to any package. Unpackaged binaries in system paths are strong indicators of trojanized tools replacing legitimate ones.
Why this works: Extract SSH key comments from authorized_keys files. Unknown identities should be investigated, then removed or rotated through the normal incident process.
Deploy AIDE or OSSEC for file integrity monitoring
Compare against a known-good baseline image
Check /tmp, /dev/shm, /var/tmp for malware staging areas
Every finding needs timestamps for the forensic chain of evidence
Scenario 10
CI/CD Pipeline Log Debugging and Optimization
Situation
Your CI/CD pipeline fails intermittently - sometimes green, sometimes red. Each run produces 50,000+ lines of output. You need to find failure patterns, identify flaky tests, extract timing data, and optimize the slowest build stages.
Why this works: Find real errors in 50K lines. The second grep excludes false positives ("0 errors", error handling code). This cuts noise to just the 5-10 lines that matter.
Step 2awk
awk '/FAIL/ {test=$NF; fails[test]++} END {for(t in fails) printf "%3d failures: %s\n", fails[t], t}' test_results_*.log | sort -rn
Why this works: Aggregate failures across builds to find flaky tests. Awk counts each test's fail frequency. Tests failing intermittently are your pipeline stability killers.
Why this works: Calculate step durations. Sorting by time reveals your slowest stages - the biggest optimization wins. Often one slow step dominates total build time.
Why this works: Extract the dependency install section and filter for issues. Deprecation warnings and vulnerability alerts reveal supply chain risks causing intermittent builds.
Why this works: Build failure rate report. Shows which runs had failures and how many. Correlating failure counts with code changes reveals which commits introduced instability.
Implement structured build logging with timestamps
Quarantine flaky tests rather than retrying until green
Cache dependencies to speed up install stages
Set up pipeline analytics dashboards to track failure trends
Wrapup complete. You have now seen how grep, sed, and awk work together in realistic log review, config audit, incident response, and build troubleshooting tasks. The next step is practice: take one familiar log file, ask one clear question about it, and build the smallest pipeline that answers it.