[Pelis Agent Factory Advisor] Pelis Agent Factory Advisor - Agentic Workflow Maturity Report (2026-03-09)

[github-actions[bot]]

📊 Executive Summary

The gh-aw-firewall repository has a strong and growing agentic workflow foundation (28 workflow .md files, 16+ distinct automations) with excellent coverage of security, CI health, and code quality domains — appropriate for a security-critical firewall product. However, several high-value workflow categories from the Pelis Agent Factory remain unexplored: issue triage/labeling, code simplification, meta-agent observability, and PR assistance workflows. Additionally, a cluster of recent Secret Digger failures (#1172, #1171, #1168) and a CI Doctor failure (#1164) suggest some workflows need attention.

---

🎓 Patterns Learned from Pelis Agent Factory

The Pelis Agent Factory blog series documents 100+ automated agentic workflows across 13 categories. The key patterns and best practices that stand out:

Key Patterns from the Factory

Pattern	Description	Pelis Factory Examples
Causal Chains	Agents create issues → Issue Monster assigns → Copilot fixes → CI Doctor verifies	CI Doctor → 9 merged PRs (69% rate)
Meta-Agents	Workflows that watch and audit other workflows	Audit Workflows (93 discussions!), Workflow Health Manager
Skip-if-match	Prevents duplicate work using GitHub search queries	Prevents multiple open PRs of same type
Cache Memory	Persistent state across workflow runs for deduplication	Issue Duplication Detector (already using this!)
Observability First	Metrics collection before optimization	Metrics Collector → Portfolio Analyst
Specialized > Monolithic	Many focused agents beat one "do everything" agent	100+ specialized workflows
Continuous Quality	Daily agents cleaning code, not "cleanup sprints"	Code Simplifier (83% merge rate), Duplicate Detector (79%)

How This Repo Compares

What it does well compared to Pelis patterns:

• ✅ Issue Monster (task dispatch to Copilot coding agent) — matching best practice
• ✅ CI Doctor (CI failure investigation) — matching best practice
• ✅ Issue Duplication Detector with cache memory — sophisticated pattern
• ✅ Strong security coverage (security-guard, security-review, dependency-security-monitor, secret-diggers)
• ✅ Domain-specific adaptation (build-test for 8 languages tests the firewall's actual use cases)

Gaps vs Pelis patterns:

• ❌ No issue triage/labeling (Pelis: every repo starts with this)
• ❌ No code quality agents (simplifier, duplicate detector)
• ❌ No meta-agent observability (Audit Workflows, Metrics Collector)
• ❌ No PR assistance (PR Fix, Grumpy Reviewer)
• ❌ No breaking change detection
• ❌ No weekly summary/status report

---

📋 Current Agentic Workflow Inventory

Workflow	Purpose	Trigger	Assessment
build-test-{bun,cpp,deno,dotnet,go,java,node,rust}	Smoke-test AWF with each language ecosystem	workflow_dispatch, schedule	✅ Excellent domain-specific coverage
ci-doctor	Investigate CI failures, create diagnostic issues	workflow_run (completed)	✅ Good, but CI Doctor itself recently failed (#1164)
ci-cd-gaps-assessment	Assess CI/CD pipeline gaps	Daily	✅ Good analysis tool
cli-flag-consistency-checker	CLI flags vs docs sync	Weekly	✅ Solid, should track merge rate
dependency-security-monitor	Dependency CVE monitoring, auto-PRs	Daily	✅ Good, had recent failure (#1135)
doc-maintainer	Docs sync with code changes	Daily	✅ Well-configured with skip-if-match
issue-duplication-detector	Detect duplicate issues	issues.opened	✅ Excellent use of cache memory
issue-monster	Assign issues to Copilot coding agent	issues.opened, hourly	✅ Central task dispatcher
pelis-agent-factory-advisor	This workflow	Daily	🔄 Running now
plan	Plan command via comment	issue_comment	✅ Useful ChatOps
secret-digger-{claude,codex,copilot}	Secret scanning	Scheduled	⚠️ All 3 recently failed (#1172, #1171, #1168)
security-guard	PR security review	pull_request	✅ Critical guard for this repo
security-review	Daily comprehensive security review	Daily	✅ Thorough, evidence-based
smoke-{chroot,claude,codex,copilot}	End-to-end smoke tests	Scheduled	✅ Good integration coverage
test-coverage-improver	Find and fill test coverage gaps	Weekly	✅ Focused on security-critical paths
update-release-notes	Enhance release notes	release.published	✅ Good release automation

---

🚀 Actionable Recommendations

P0 — Implement Immediately

P0.1: Fix the Secret Digger Cluster Failure

What: Three secret digger workflows (claude, codex, copilot) all recently failed (#1172, #1171, #1168). This is a critical security automation gap.

Why: Secret scanning is a core security practice and having all three engines fail simultaneously suggests a systemic issue (likely a common dependency, API change, or configuration problem). Every day these are broken, potential credentials could go undetected.

How: Investigate the failures in those three issues, identify the common root cause, and fix the shared configuration. The CI Doctor should have caught this, but it itself also failed (#1164) — so check both.

Effort: Low (diagnostic + targeted fix)

---

P0.2: Add Issue Triage / Labeling Workflow

What: An automated agent that labels new issues with bug, feature, enhancement, documentation, question, help-wanted, or good-first-issue and leaves a comment explaining the label.

Why: The current open issues have inconsistent labeling (e.g., #1139 "[Security Review]" discussion has no labels, #1136 has no labels). Issue Monster relies on unlabeled issues to find work — but without triage labels, it can't prioritize. In Pelis Factory, this is the "hello world" of agentic workflows and the foundation of the causal chain.

How: Add an issue triage workflow triggered on issues.opened:

---
name: Issue Triage
on:
  issues:
    types: [opened, reopened]
permissions:
  issues: read
tools:
  github:
    toolsets: [issues, labels]
    lockdown: false
safe-outputs:
  add-labels:
    allowed: [bug, feature, enhancement, documentation, question, help-wanted, good-first-issue, security]
  add-comment: {}
timeout-minutes: 5
---
# Issue Triage Agent
Analyze the new issue in the gh-aw-firewall repository (a network firewall for AI agents). 
Apply one label: bug, feature, enhancement, documentation, question, help-wanted, good-first-issue, or security.
Skip issues already labeled. Comment to the author explaining the label and how the issue may be addressed.

Effort: Low (10 min to implement, following the Pelis pattern exactly)

---

P1 — Plan for Near-Term

P1.1: Breaking Change Checker

What: A workflow triggered on PRs that analyzes changes for backward-incompatible API or CLI flag changes, configuration format changes, or Docker image interface changes, and creates an alert issue or comment.

Why: This firewall is used by CI/CD pipelines. Breaking changes to CLI flags, Docker configurations, or network rules can silently break downstream users. The Pelis Factory's Breaking Change Checker caught issues before they hit production.

How: PR-triggered workflow that examines src/cli.ts, src/types.ts, and container entrypoints for interface changes. Add as a PR check (safe-outputs: add-comment).

Effort: Low-Medium (adapt from Pelis Factory pattern)

---

P1.2: Workflow Audit / Meta-Agent

What: A weekly or daily workflow that audits all other agentic workflow runs — analyzing costs, error patterns, success rates, and identifying workflows that are failing, expensive, or producing low-quality output.

Why: In the Pelis Factory, the Audit Workflows meta-agent was their "most prolific discussion-creating agent" with 93 reports and 9 issues. Currently this repo has no observability into workflow performance. The cluster of recent failures (Secret Diggers, CI Doctor) would have been caught faster by a meta-agent.

How: Using tools: agentic-workflows: (already used in security-review), analyze recent workflow run results via agenticworkflows-logs and agenticworkflows-audit. Create discussions with [Workflow Audit] prefix.

Effort: Medium (uses existing agentic-workflows tool already in the repo)

---

P1.3: Code Simplifier (TypeScript-focused)

What: A daily workflow that analyzes recently modified TypeScript files in src/ and creates PRs with simplifications — shorter code, better idiomatic patterns, extracted helpers for repeated patterns.

Why: The Pelis Factory's Code Simplifier achieved an 83% PR merge rate. This TypeScript codebase has patterns that could benefit (e.g., repeated docker exec logic in src/docker-manager.ts which is ~1500 lines). The firewall's security-critical nature makes clean, readable code especially important for security reviews.

How: Daily workflow on recently-committed .ts files in src/, with skip-if-match to prevent stacked simplification PRs:

---
on:
  schedule: daily
  workflow_dispatch:
  skip-if-match:
    query: 'is:pr is:open in:title "[Simplify]"'
    max: 2
safe-outputs:
  create-pull-request:
    title-prefix: "[Simplify] "
    labels: [refactor, ai-generated]
    draft: true
````

**Effort**: Low (follows Pelis pattern, adapt to TypeScript)

---

#### P1.4: PR Fix Assistant

**What**: A workflow triggered when a PR's CI checks fail that analyzes the failure, investigates root cause, and either comments with a fix suggestion or directly pushes a fix commit.

**Why**: The Pelis Factory's "pr-fix" workflow (from agentics repo) and CI Coach (9/9 merged PRs, 100% rate!) automated CI fixes. For a repository where contributors must navigate Docker, iptables, and Squid config — CI failures can be opaque and fixing them requires deep domain knowledge that an agent can provide.

**How**: Adapt the `pr-fix.md` from `githubnext/agentics` for this repo:

````
gh aw add-wizard githubnext/agentics/pr-fix
````

Then customize to understand AWF-specific failure patterns.

**Effort**: Low (direct add-wizard from agentics)

---

#### P1.5: Weekly Repository Status / Summary

**What**: A weekly report summarizing repository health: open issues count by category, recent PR activity, test coverage trends, recently added domains to the allowlist, and workflow execution health.

**Why**: The `daily-repo-status.md` and `weekly-repo-chronicle.md` patterns from the agentics repository provide stakeholders with a regular health pulse. For a security tool, a weekly status that includes "issues opened/closed this week", "security reviews completed", and "coverage trends" gives maintainers a quick overview without digging through individual workflow discussions.

**How**: Adapt `githubnext/agentics/daily-repo-status.md` for weekly cadence:

````
gh aw add-wizard githubnext/agentics/daily-repo-status
````

**Effort**: Low (add-wizard + minor customization)

---

### P2 — Consider for Roadmap

#### P2.1: Grumpy Reviewer / PR Nitpick Reviewer

**What**: A PR review agent that checks for common issues: TypeScript best practices, security anti-patterns specific to iptables/Squid config, missing tests for new code paths, and documentation gaps.

**Why**: The `grumpy-reviewer.md` and `pr-nitpick-reviewer.md` in the agentics repository are high-signal reviewers that catch issues before human review. For a security tool, automated review of PRs that touch `setup-iptables.sh`, `squid-config.ts`, or host iptables code is particularly valuable.

**How**: Customize the grumpy reviewer to understand AWF security boundaries (the security-guard already does this partially, but a complementary code-quality reviewer would be additive).

**Effort**: Medium

---

#### P2.2: Duplicate Code Detector

**What**: Detect duplicate logic across the TypeScript codebase, particularly in the large `src/docker-manager.ts` file and the container scripts.

**Why**: The Pelis Factory's Duplicate Code Detector achieved a 79% PR merge rate over 76 merged PRs. `docker-manager.ts` is growing (mentioned as ~1500 lines in docs) and likely has repeated patterns in container lifecycle management code.

**How**: Adapt `githubnext/agentics/duplicate-code-detector.md` for TypeScript focus. Note the Serena MCP tool may need to be added.

**Effort**: Medium-High (requires Serena setup)

---

#### P2.3: Documentation Link Checker

**What**: A weekly workflow that checks all links in `docs/`, `README.md`, `docs-site/`, and `AGENTS.md` for broken links, redirects, and outdated URLs.

**Why**: The `githubnext/agentics/link-checker.md` pattern is simple and high-value. This repo has extensive documentation with links to GitHub Actions runs, Docker registries, and external tools. Link rot is common in active projects.

**How**: 

````
gh aw add-wizard githubnext/agentics/link-checker
````

**Effort**: Very Low

---

#### P2.4: CI Coach for Workflow Optimization

**What**: A periodic analysis of CI/CD pipeline efficiency — identifying slow workflows, redundant steps, and optimization opportunities.

**Why**: The Pelis Factory's CI Coach achieved 9/9 merged PRs (100% rate!). The `build-test-*` workflows for 8 languages run the full Docker stack for each language — potentially costly and slow. A CI Coach could identify consolidation opportunities.

**How**:

````
gh aw add-wizard https://github.com/github/gh-aw/blob/v0.45.5/.github/workflows/ci-coach.md

Effort: Very Low (add-wizard)

---

P2.5: Firewall-Specific Daily Health Check Discussion

What: A daily discussion summarizing yesterday's firewall usage — which domains were accessed in CI runs, which were blocked, and whether any unexpected domain access patterns appeared.

Why: This is a firewall repository — dog-fooding its own firewall (via smoke tests) should produce observable network access data. A daily summary of [Firewall Health] showing "10 smoke runs, 0 blocked legitimate domains, 3 blocked unauthorized domains" would demonstrate the product working and catch configuration regressions.

How: Post-process squid logs from smoke runs and compile into a discussion. Uses the repo's own awf logs summary command!

Effort: Medium (requires log artifact collection across workflow runs)

---

P3 — Future Ideas

P3.1: Contribution Guidelines Checker

What: Check new PRs against CONTRIBUTING.md rules — conventional commits, test coverage requirements, documentation updates.

How: gh aw add-wizard githubnext/agentics/contribution-guidelines-checker

P3.2: Glossary Maintainer

What: Keep a technical glossary of AWF-specific terms (Squid, iptables NAT, DNAT, etc.) synchronized with the docs-site.

How: gh aw add-wizard githubnext/agentics/glossary-maintainer

P3.3: Issue Arborist (Sub-issue Linking)

What: Automatically link related issues as sub-issues when they share themes (e.g., all issues about --allow-domains flag becoming sub-issues of a parent issue).

How: Adapt the Pelis Factory Issue Arborist for this repo's issue taxonomy.

P3.4: Changeset / Release Automation

What: Automate version bump determination and CHANGELOG generation based on commit analysis (semver: feat→minor, fix→patch, breaking→major).

Why: The current update-release-notes runs after a release is published, but a changeset workflow would prepare releases by auto-generating PRs with the appropriate version bump.

How: gh aw add-wizard https://github.com/github/gh-aw/blob/v0.45.5/.github/workflows/changeset.md

---

📈 Maturity Assessment

Current Level: 3.5 / 5 — Established & Domain-Focused

This repository has moved well beyond basic CI/CD. It has specialized agentic workflows for security review, CI investigation, dependency monitoring, documentation maintenance, and test coverage improvement. The domain-specific adaptations (8 language build-tests, secret-digger variants) show sophisticated application of agentic patterns.

Target Level: 4.5 / 5 — Comprehensive Ecosystem

Gap Analysis to reach 4.5:

Gap	Value	Effort
Issue triage/labeling	High	Very Low
Meta-agent observability	High	Medium
Code quality agents	Medium	Low
PR assistance (fix + review)	Medium	Low
Release automation	Medium	Low
Breaking change detection	High	Medium

---

🔄 Comparison with Pelis Factory Best Practices

What this repo does particularly well:

• Domain adaptation: Build-test workflows for 8 languages specifically test the firewall's real use cases — a sophisticated domain-specific pattern not seen in generic repos
• Security depth: 6+ security-related workflows (security-review, security-guard, secret-diggers × 3, dependency-security-monitor) matches the Pelis Factory's security category
• Causal chain readiness: Issue Monster is already configured as the task dispatcher, so adding triage + more issue-creating workflows would immediately activate the full causal chain
• Cache memory usage: Issue Duplication Detector already uses this advanced pattern

Unique opportunities given the domain:

• Dog-fooding: As a firewall tool, smoke tests generate actual firewall logs that could feed automated analysis workflows
• Security validation: A dedicated workflow that verifies the firewall's own network rules haven't been weakened would be unique to this repo
• Multi-engine testing: Running the same test across claude/codex/copilot engines already — could add a "comparative analysis" workflow that tracks which engine handles certain prompts better

---

Report generated by Pelis Agent Factory Advisor (2026-03-09). Previous report: #1136 (2026-03-03). Cache memory updated at /tmp/gh-aw/cache-memory/advisor-notes.md.

AI generated by Pelis Agent Factory Advisor

• expires on Mar 16, 2026, 3:29 AM UTC

[github-actions[bot]]

🔮 The ancient spirits stir, and the oracle bears witness: the smoke-test agent has passed through these halls and left its mark.

🔮 The oracle has spoken through Smoke Codex