AI-Powered Ethical Hacking (GitHub + Agent Skills), Part 5: Retest Checklists, Evidence Folders and a Reusable Reporting Skill

Series: AI-Powered Ethical Hacking (GitHub + Agent Skills), Part 5. Previous: Part 4, AI for defense, triage and reporting. Start: Part 1, AI meets Kali.

In this post you take the fix list from Part 4 and turn it into three things you can reuse: an evidence folder for your lab findings, a retest checklist written by a local model, and a reporting skill file. You also run a retest against your own authorized lab hosts. The model runs locally, on notes you already have.

First, the legal bit

Disclaimer: This post is for education and authorized security testing only. Scanning, auditing or accessing a computer, network or account without permission is illegal, including under India’s Information Technology Act, 2000 and similar laws in other countries. Only work with notes and tool output from systems you own, or systems where you have clear written permission from the owner before you start.

I ended Part 4 by saying it closed the AI side of this series. A fix list still leaves one question open, though: did the fix hold? So here is one more defensive part. It assumes your findings came from your own practice target, set up like the Kali Linux safe practice lab. Most steps only work with files. The retest steps do touch hosts, so run them only against lab machines you own or are authorized to test, on a host only network.

Why does a retest pack matter after triage?

A fix list feels finished, but it only says what should change. A week later, someone asks a simple question: did the fix hold? If your answer is “I think so”, the report loses its value.

My day job sits close to network detection and response, and the pattern I trust is boring. Each finding gets an ID, a folder of evidence, a fix owner, and a plain retest line that says what “fixed” looks like. A retest pack is just those four things kept together.

A local model is useful for the dull part. It can turn ten messy fix notes into ten tidy checklist rows, in the same shape every time. It should never decide whether something is fixed. You decide that by comparing fresh evidence with the old evidence.

How do you build an evidence folder for your own findings?

Prerequisites

• Local Ollama from Part 2, v0.40.7 or newer, with a small chat model such as qwen3:4b.
• The Part 4 files in a dated reporting folder: findings-redacted.txt, triage-summary.txt and manager-note.txt.
• A plain text editor and a terminal. Steps 1 to 5 and 10 to 12 need no network access.
• For the retest in Steps 6 to 9: your Kali VM and lab-web-01 on the same host only lab network, with sudo on both. No internet target, ever.

Step 1: Create the folder layout

Pick the Part 4 date you want to work from. I use a variable so the commands stay copyable.

NOTES=~/lab-notes/2026-10-23
mkdir -p "$NOTES/evidence/F01" "$NOTES/evidence/F02" "$NOTES/retest"
cd "$NOTES"

One subfolder for each finding is enough. F01 and F02 match the two items from the Part 4 sample.

Step 2: Copy the original evidence in

Copy only saved files from your own session. Do not run anything new here.

cp reporting/findings-redacted.txt evidence/F01/before-notes.txt
cp reporting/findings-redacted.txt evidence/F02/before-notes.txt
cp reporting/triage-summary.txt retest/

Then open each before-notes.txt and delete the lines that belong to the other finding. Each folder should hold only its own facts.

Step 3: Add a hash list so files can’t quietly change

sha256sum evidence/*/* > evidence/SHA256SUMS.txt
cat evidence/SHA256SUMS.txt

Expected output (sample, fake hashes):

3f1a9c...e07b  evidence/F01/before-notes.txt
8d42b0...91ac  evidence/F02/before-notes.txt

Later, sha256sum -c evidence/SHA256SUMS.txt should print OK for each file. If one says FAILED, someone edited the evidence after the fact, and you should find out why before you write anything.

How do you write a retest checklist with a local model?

Step 4: Save the checklist prompt

As in Part 4, the prompt lives in a file and goes to the model as one argument. Pasting many lines into the interactive ollama run chat sends each line separately unless you wrap them in triple quotes, according to Ollama’s CLI docs.

Save this as retest/retest-prompt.txt:

You are helping me write a defensive retest checklist from my own authorized lab triage.
Do not suggest scanning, exploiting or contacting any host.
Do not include commands, tool names or flags.
Use only facts in the TRIAGE block. If a fact is missing, write "needs evidence".

For each finding, return one row with these fields:
ID (F01, F02 and so on) | Finding | Fix owner | What "fixed" looks like | Evidence I must attach | Status (always "open")

After the rows, add one line: "Retest date: ____ Tester: ____".

TRIAGE:

Step 5: Run it and save the reply

cd "$NOTES/retest"
ollama run --hidethinking qwen3:4b "$(cat retest-prompt.txt triage-summary.txt)" > retest-checklist.txt
cat retest-checklist.txt

Expected output (SAMPLE checklist, fake lab data, not from a real engagement):

F01 | lab-web-01 port 3000 shows a default page | me | Service on 3000 is removed or bound to localhost only | After notes from the authorized retest, saved in evidence/F01 | open
F02 | Kali SSH listens on all interfaces | me | SSH listens only on the lab interface, or is stopped | Copy of the SSH listen setting and after notes, saved in evidence/F02 | open

Retest date: ____ Tester: ____

Read every row against triage-summary.txt. If the model added a finding, a CVE or a command, delete that line. Status stays “open” until you have attached after evidence yourself.

How do you run the authorized retest in your lab?

The checklist says what “fixed” looks like. Steps 6 to 9 collect the after evidence for F01 and F02. Unlike the rest of this post, they do touch hosts, so run them only against lab-web-01 and your own Kali VM on the host only lab network, never against anything you don’t own or aren’t authorized to test. The addresses below are samples: 192.168.56.101 for lab-web-01 and 192.168.56.102 for Kali. Replace them with yours.

Step 6: Re run Nmap against lab-web-01 for F01

From Kali:

cd "$NOTES"
TARGET=192.168.56.101   # lab-web-01, your own lab host only
sudo nmap -Pn -p 3000 "$TARGET" -oA evidence/F01/after-nmap-3000
{ echo "F01 retest $(date '+%F %R') target lab-web-01 ($TARGET)"; grep "3000/tcp" evidence/F01/after-nmap-3000.nmap; } >> evidence/F01/after-notes.txt
cat evidence/F01/after-notes.txt

Expected output (sample, fake lab data):

F01 retest 2026-10-24 10:15 target lab-web-01 (192.168.56.101)
3000/tcp closed ppp

closed (or filtered) from Kali means the service no longer answers on the lab network. If you still see open, F01 stays open. Go to Step 7.

Step 7: Find what binds port 3000 on lab-web-01 and move it to localhost

Log in to lab-web-01 itself and check what is listening:

sudo ss -ltnp | grep ':3000'

Expected output before the fix (sample):

LISTEN 0  4096  0.0.0.0:3000  0.0.0.0:*  users:(("docker-proxy",pid=1234,fd=4))

0.0.0.0:3000 means every interface. The process name tells you where to change it. In general, look for a listen or bind address setting in the app’s config file or service unit, set it to 127.0.0.1, and restart the service. If it is the Juice Shop container from this series, stop it and start it again bound to localhost:

sudo docker ps --filter publish=3000
sudo docker stop <container id>
sudo docker run -d --rm -p 127.0.0.1:3000:3000 bkimminich/juice-shop
sudo ss -ltnp | grep ':3000'

Expected output after the fix (sample):

LISTEN 0  4096  127.0.0.1:3000  0.0.0.0:*  users:(("docker-proxy",pid=5678,fd=4))

Then repeat Step 6 from Kali. A closed result there, plus this 127.0.0.1:3000 line, is your F01 after evidence. Add the ss line to evidence/F01/after-notes.txt too.

Step 8: Confirm Kali’s SSH listens only on the lab interface for F02

On Kali, check what SSH listens on and what the config says:

sudo ss -ltnp | grep sshd
sudo grep -E '^\s*ListenAddress' /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*.conf 2>/dev/null

Expected output before the fix (sample):

LISTEN 0  128  0.0.0.0:22  0.0.0.0:*  users:(("sshd",pid=812,fd=6))

No ListenAddress line means SSH listens everywhere. To limit it to the lab interface, add one line with your Kali lab address, check the config, and restart:

echo "ListenAddress 192.168.56.102" | sudo tee /etc/ssh/sshd_config.d/10-lab-only.conf
sudo sshd -t && sudo systemctl restart ssh
sudo ss -ltnp | grep sshd

Expected output after the fix (sample):

ListenAddress 192.168.56.102
LISTEN 0  128  192.168.56.102:22  0.0.0.0:*  users:(("sshd",pid=1490,fd=3))

sshd -t prints nothing when the config is valid. Keep the host only adapter enabled at boot, because sshd can’t start if the ListenAddress IP is missing. If it prints an error, fix that before restarting, or you may lock yourself out.

Now check reachability from another lab host, such as lab-web-01:

nc -zv -w 3 192.168.56.102 22

Expected output (sample):

Connection to 192.168.56.102 22 port [tcp/ssh] succeeded!

That confirms SSH still answers on the lab interface. Then record it on Kali:

cd "$NOTES"
{ echo "F02 retest $(date '+%F %R') host kali"; sudo ss -ltnp | grep sshd; cat /etc/ssh/sshd_config.d/10-lab-only.conf; echo "lab reachability from lab-web-01: succeeded"; } >> evidence/F02/after-notes.txt
cat evidence/F02/after-notes.txt

Expected output (sample): the date line, the 192.168.56.102:22 listen line, the ListenAddress line and the reachability note.

Step 9: Let your local agent double check the fix, in scope only

If you set up Ollama with mcp-kali-server and an MCP client in Part 2, you can ask the agent to repeat the F01 check. Keep approval on for every tool call. Paste this into the client, with your own lab address:

Scope: 192.168.56.101 (lab-web-01) only. It is a lab host I am authorized to test. Refuse any other host, range or domain.
Before each tool call, show the exact command and wait for my approval.
Task: run one nmap check of TCP port 3000 on 192.168.56.101 to confirm finding F01 is fixed.
Show the raw output first. Then say "closed", "filtered" or "open", quoting the exact line.
Do not change anything on the host and do not mark the finding as fixed.

Expected behaviour (sample): the agent proposes a single command such as nmap -Pn -p 3000 192.168.56.101, waits for your approval, then shows a raw line like 3000/tcp closed ppp and answers “closed”. Compare that with your own Step 6 result. The agent’s answer is a second opinion, not evidence on its own. The status change in Step 10 stays with you.

Step 10: Close a row by hand

When your authorized retest is done and the after notes are saved in the right folder, edit the row yourself. Change open to fixed or still open, and add the file name of the after notes. Then refresh the hash list:

cd "$NOTES"
sha256sum evidence/*/* > evidence/SHA256SUMS.txt

I keep the model out of this step on purpose. A status change is a claim, and claims need a human who looked at the files.

How do you save this as a reusable reporting skill?

In Part 3 we wrote a scope first skill for lab work. The same idea works for reporting. You write your rules once, keep them in one file, and send them with every request so the shape never drifts.

Step 11: Write the skill file

mkdir -p ~/skills/lab-reporting
cat > ~/skills/lab-reporting/reporting-rules.txt <<'RULES'
Name: lab-reporting
Purpose: turn my own authorized lab notes into triage, retest rows and a manager note.

Rules:
1. Work only from text I paste. Never suggest scanning, exploiting or contacting a host.
2. No commands, tool names or flags in any output.
3. Keep finding IDs (F01, F02) exactly as given.
4. If evidence is missing, write "needs evidence". Never guess.
5. Status values allowed: open, fixed, still open. Never change a status yourself.
6. Plain English, short lines, no marketing words.
RULES
cat ~/skills/lab-reporting/reporting-rules.txt

Expected output: the name, purpose and six rules printed back to you.

If your client supports Anthropic’s Agent Skills format, Part 3 shows how to wrap this as a SKILL.md with a short name and description header. With plain Ollama, a text file works fine.

Step 12: Use the skill with any task prompt

Put the rules first, then the task, then the notes:

cd "$NOTES/retest"
ollama run --hidethinking qwen3:4b "$(cat ~/skills/lab-reporting/reporting-rules.txt retest-prompt.txt triage-summary.txt)" > retest-checklist-v2.txt
diff retest-checklist.txt retest-checklist-v2.txt

Expected output (sample): either no output, or a few small wording changes. If the rows, IDs or owners differ, read both against the triage file and keep the version that matches it.

What stays out of the cloud?

The short version from Part 4 still applies. Ollama’s FAQ says that when you run a model locally, Ollama doesn’t see your prompts or data. Keep it that way for evidence folders, hash lists and retest notes. If you want help from a cloud model, give it a fully fake example with made up hosts and ports, never the real folder. For how prompts and logs leak in LLM apps, see my LLM observability guide for 2026.

What to do next

Your plan for this week:

1. Create an evidence folder with one subfolder for each finding from your Part 4 notes.
2. Add a SHA256SUMS.txt file and check it with sha256sum -c.
3. Run the retest prompt on local Ollama and delete any row it invented.
4. Save reporting-rules.txt and send it first with every reporting prompt.
5. After your own authorized retest, close each row by hand and refresh the hashes.

Next time you open a lab notes folder, you will know what was found, what fixed means, and where the proof lives.

If your agents touch real systems, start with the checklist in what to do this week if you run AI agents.

Next, you point the same local model at a lab SSH log and turn what it finds into a checked rule: AI log triage and Sigma detection rules.

Leave a comment

Cancel reply

Your email stays private. First-time comments are held for a quick review.

Comment *

Name *

Email *

Website

Save my name, email, and website in this browser for the next time I comment.