AI-Powered Ethical Hacking (GitHub + Agent Skills), Part 6: Triage Lab Logs and Write a Sigma Detection Rule
Series: AI-Powered Ethical Hacking (GitHub + Agent Skills), Part 6. Previous: Part 5, retest checklists and a reporting skill. Start: Part 1, AI meets Kali.
In this part you use a local Ollama model to read an SSH log from a lab you own or are authorized to test, count the failed logins yourself, and turn the pattern into a Sigma detection rule. Then you check that rule with sigma-cli. It is AI log triage and detection rules in one offline loop.
First, the legal bit
Disclaimer: This post is for education and authorized security work only. Scanning, auditing or accessing a computer, network or account without permission is illegal, including under India’s Information Technology Act, 2000 and similar laws in other countries. Only work with logs from systems you own, or systems where you have clear written permission from the owner before you start.
Apart from installing sigma-cli and its Splunk plugin, every step reads or writes files on your own machine. Nothing scans or contacts another host. The log lines below are a fake sample I wrote by hand, with a documentation address (203.0.113.45) and a host only lab address (192.168.56.102). If you want to use real logs, take them only from a VM in a lab you own or are authorized to test, set up like the Kali Linux safe practice lab.
Why turn log triage into a detection rule?
In Part 4 we triaged findings, and in Part 5 we checked that fixes held. Both looked at a moment in time. Logs are different. They keep coming, and the same pattern shows up again next week.
My day job sits close to network detection and response, and the habit I trust is simple. When you spot a pattern once by hand, write it down as a rule, so the next time a machine spots it for you. Sigma is a good format for that. It is a plain YAML file that describes what to look for, and tools like sigma-cli can turn it into a query for Splunk, Elasticsearch and other backends.
A local model helps with two dull jobs: summarising messy log lines and writing the first version of the rule. It should never be the one who decides the counts or says the rule is correct. You check both.
What do you need before you start?
- Local Ollama from Part 2. The latest release on GitHub is v0.40.11 (24 Oct 2026). Anything from v0.40.7, which Parts 4 and 5 used, works for these steps. The steps use the small
qwen3:4bmodel. - Python 3.10 or newer and
pipx, for sigma-cli 3.1.0. - A terminal and a plain text file program. No network access is needed after the installs.
- About 30 minutes.
How do you triage lab logs with a local model?
Step 1: Create a working folder
LOGS=~/lab-notes/2026-10-26/logs
mkdir -p "$LOGS"
cd "$LOGS"
Step 2: Save the sample auth log
This is a SAMPLE with fake lines. Nobody logged in anywhere to make it.
cat > sample-auth.log <<'LOG'
Oct 25 09:12:01 lab-web-01 sshd[2101]: Invalid user admin from 203.0.113.45 port 51120
Oct 25 09:12:03 lab-web-01 sshd[2101]: Failed password for invalid user admin from 203.0.113.45 port 51120 ssh2
Oct 25 09:12:06 lab-web-01 sshd[2104]: Invalid user test from 203.0.113.45 port 51134
Oct 25 09:12:08 lab-web-01 sshd[2104]: Failed password for invalid user test from 203.0.113.45 port 51134 ssh2
Oct 25 09:12:11 lab-web-01 sshd[2107]: Failed password for root from 203.0.113.45 port 51150 ssh2
Oct 25 09:12:14 lab-web-01 sshd[2107]: Failed password for root from 203.0.113.45 port 51150 ssh2
Oct 25 09:20:40 lab-web-01 sshd[2190]: Failed password for labuser from 192.168.56.102 port 40118 ssh2
Oct 25 09:20:47 lab-web-01 sshd[2190]: Accepted password for labuser from 192.168.56.102 port 40118 ssh2
Oct 25 09:31:02 lab-web-01 sshd[2233]: Accepted publickey for labuser from 192.168.56.102 port 40160 ssh2
LOG
If you would rather use a log from a lab VM you own or are authorized to test, copy it into this folder instead. Newer Debian based systems may not write /var/log/auth.log at all, so export the SSH journal from that VM:
sudo journalctl -u ssh --since today --no-pager > my-lab-auth.log
Before any real log goes near a model, replace real usernames and any address outside your lab with placeholders, as we did in Part 4.
Step 3: Count the failures yourself first
This gives you the ground truth. The model’s summary has to match it.
grep -E 'Failed password|Invalid user' sample-auth.log | grep -oE 'from [0-9.]+' | sort | uniq -c | sort -rn
Expected output (sample data):
6 from 203.0.113.45
1 from 192.168.56.102
Six failure lines from one address in 13 seconds, and one failure from the Kali address 192.168.56.102 that was followed by a good login, which looks like a typo. Keep this output. You will compare against it in Step 6.
Step 4: Save the triage prompt
As in Parts 4 and 5, the prompt lives in a file and goes to the model as one argument. Pasting many lines into the interactive ollama run chat sends each line on its own unless you wrap them in triple quotes, according to Ollama’s CLI docs.
Save this as triage-prompt.txt:
You are helping me triage an SSH auth log from an authorized lab VM.
Do not suggest scanning, attacking or contacting any host.
Use only the lines in the LOG block. If something is unclear, write "needs evidence".
For each source address, give:
Source | Failed or invalid user lines | Usernames tried | First and last time | Any accepted login after it | Plain English note
Then one line: "Pattern worth a rule:" followed by one sentence.
LOG:
Step 5: Run it and save the reply
ollama run --hidethinking qwen3:4b "$(cat triage-prompt.txt sample-auth.log)" > triage-notes.txt
cat triage-notes.txt
Expected output (SAMPLE, fake lab data, your model’s wording will differ):
203.0.113.45 | 6 | admin, test, root | 09:12:01 to 09:12:14 | none | Several usernames tried in a few seconds, no success.
192.168.56.102 | 1 | labuser | 09:20:40 to 09:20:40 | yes, 09:20:47 | One failure then a good login, looks like a typo.
Pattern worth a rule: repeated failed or invalid user SSH logins from one source.
Step 6: Check the model against your own counts
Put the Step 3 output next to triage-notes.txt. The counts for each source must match. If the model says 5 where you counted 6, or invents a third address, fix the notes by hand and trust grep. Small models can miscount, which is why Step 3 comes first.
How do you turn the pattern into a Sigma rule?
Step 7: Save the rule prompt
The prompt asks for flow style lists in square brackets. They are valid YAML, and they keep the file short and easy to read.
Save this as rule-prompt.txt:
Write one Sigma detection rule in YAML for the pattern below. It is for authorized lab logs.
Use these fields only: title, id, status, description, author, date, tags, logsource, detection, falsepositives, level.
Rules:
1. logsource is {product: linux, service: auth}.
2. Use a keywords search for the exact log phrases in the pattern. Write every list in square brackets on one line.
3. status is experimental, level is medium, date is 2026-10-26, author is my name.
4. Put "REPLACE_WITH_UUID" in the id field.
5. Output only the YAML, no explanation.
PATTERN:
Step 8: Run it and save the rule
ollama run --hidethinking qwen3:4b "$(cat rule-prompt.txt triage-notes.txt)" > lab-ssh-failed-logins.yml
sed -i "s/REPLACE_WITH_UUID/$(python3 -c 'import uuid; print(uuid.uuid4())')/" lab-ssh-failed-logins.yml
cat lab-ssh-failed-logins.yml
If the model wraps the YAML in a fenced code block, delete those fence lines by hand. Then read the file and fix anything that does not match your log.
Expected output (SAMPLE rule, after hand fixes, your id will differ):
title: Repeated SSH login failures in a lab auth log
id: 6b1f0c2e-3d4a-4e8b-9a51-2f7c8d9e0a13
status: experimental
description: Flags failed SSH password and invalid user lines in the auth log of an authorized lab VM.
author: Deepu S
date: 2026-10-26
tags: [attack.credential-access, attack.t1110]
logsource: {product: linux, service: auth}
detection:
keywords: ['Failed password for', 'Invalid user']
condition: keywords
falsepositives: ['A user mistyping their password on a lab VM']
level: medium
The keywords list means “match any of these phrases anywhere in the line”. attack.t1110 is the MITRE ATT&CK tag for brute force, and the falsepositives line covers the typo from Step 3.
How do you check the rule with sigma-cli?
Step 9: Install sigma-cli
The sigma-cli README lists pipx as the easiest way in:
python3 -m pipx install sigma-cli
sigma version
Expected output (as of 25 Oct 2026):
3.1.0 (online pypi.org: 3.1.0)
On 25 Oct 2026, a fresh install of sigma-cli 3.1.0 pulled in pySigma 1.5.1, even though pySigma 2.0.1 is out on PyPI. That is fine. The CLI picks the library version it supports.
Step 10: Run sigma check
sigma check lab-ssh-failed-logins.yml
Expected output, last lines (sample rule above):
=== Summary ===
Found 0 errors, 0 condition errors and 0 issues.
No rule errors found.
No condition errors found.
No validation issues found.
A common model mistake is writing condition: selection when the search is called keywords. sigma check catches that and ends with “Check failure”. The key line looks like this (sample, your path will differ):
Condition error in /home/you/lab-notes/2026-10-26/logs/lab-ssh-failed-logins.yml:Detection 'selection' not defined in detections in /home/you/lab-notes/2026-10-26/logs/lab-ssh-failed-logins.yml
Fix the name, run the check again, and only move on when it is clean.
Step 11: See what the rule matches
First, test it against the sample with plain grep, using the same two phrases:
grep -ciE 'Failed password for|Invalid user' sample-auth.log
Expected output (sample data): 7. That is the six lines from 203.0.113.45 plus the one typo from 192.168.56.102. The two accepted logins do not match, which is what you want.
If you use Splunk, you can also turn the rule into a query to read it in a familiar shape:
sigma plugin install splunk
sigma convert -t splunk --without-pipeline lab-ssh-failed-logins.yml
Expected output:
"Failed password for" OR "Invalid user"
Step 12: Keep the rule with its evidence
Treat the rule like a finding from Part 5. Keep it next to the sample it was built from, and hash both:
cd "$LOGS"
sha256sum sample-auth.log triage-notes.txt lab-ssh-failed-logins.yml > SHA256SUMS.txt
sha256sum -c SHA256SUMS.txt
Expected output: one OK line for each of the three files.
What should stay out of the cloud?
Auth logs hold usernames, addresses and login times, which is exactly the stuff you don’t want to hand out. Ollama’s FAQ says that when you run a model locally, Ollama doesn’t see your prompts or data. Keep real logs on that local path. If you want a cloud model’s opinion on a rule, send it only the rule and fully fake log lines like the sample above. For how prompts and logs leak in LLM apps, see my LLM observability guide for 2026.
What to do next
Your plan for this week:
- Save the sample log and run Step 3 so you know the true counts.
- Run the triage prompt and correct any count the model got wrong.
- Write the rule with the rule prompt and fix it by hand.
- Get a clean
sigma check, then keep the rule and its sample together with a hash list. - Add the rules from Part 5’s
reporting-rules.txtin front of both prompts, so the model keeps the same habits.
This rule matches single lines. Sigma also has correlation rules that can count events, such as many failures from one source in a short window, and the SigmaHQ specification covers them. That is a good next step once your single line rule is clean.
Leave a comment