AI Agents CI/CD Pipeline: 3 Configs to Ship Now

Most “AI in DevOps” articles end with an architecture diagram and a list of tools. You’ve read them. You still don’t have a working config.

This post is different. You’ll leave with three copy-paste GitHub Actions workflows — self-healing builds, automated AI code review, and LLM-as-a-Judge quality gates — plus the security model and approximate cost for each. No enterprise budget required. No platform team prerequisite.

These patterns solve the core AI agents CI/CD pipeline problem: 80% of teams read the theory and never ship the implementation (Markaicode, 2025). That ends here.

—

The Three Patterns — What You’re Actually Building

Before touching YAML, here’s where each pattern fires in a standard PR workflow:

| Pattern | Trigger | What the Agent Does |

|—|—|—|

| Self-Healing Builds | Build fails | Reads logs, opens a fix PR |

| AI Code Review | PR opened | Layered review: AI + security scanner + linter |

| LLM-as-a-Judge Gate | Tests pass | Scores your AI feature outputs, blocks merge if quality drops |

These three patterns augment the edges of your pipeline. Your core build, test, and deploy steps stay deterministic — and they must.

—

Before You Start — The Continuous AI Mental Model

GitHub’s Continuous AI concept draws a hard line between two types of pipeline work:

• Deterministic CI/CD: Build, test, lint, deploy. These steps must produce the same output every time. Auditability is non-negotiable. AI agents don’t belong here.
• Agentic workflows: Log triage, code review suggestions, quality scoring. These are judgment-heavy tasks where LLM reasoning adds value precisely because the right answer isn’t a binary pass/fail.

The rule: If a pipeline failure would block a release or affect production, keep it deterministic. Use AI agents only where human judgment was already required.

This boundary is also your security model. Every pattern below follows three principles:

1. Read-only tokens by default — agents read logs and code; they never push directly to `main`
2. No auto-merge — every AI-generated fix opens a PR for human review
3. Sandboxed execution — agents run in isolated containers with outbound network access controlled by the Agent Workflow Firewall

With that foundation in place, let’s build.

—

Pattern 1 — Self-Healing Builds with GitHub Agentic Workflows

How the Log Doctor agent works

When your build breaks, a developer’s first job is log triage — reading stderr, finding the root cause, and tracing it to a line of code. Engineering teams report losing up to 20% of weekly time to tooling inefficiencies and manual pipeline triage (JetBrains State of CI/CD 2025). That’s context-switching tax disguised as “just part of the job.”

GitHub shipped Agentic Workflows in technical preview on February 13, 2026, enabling AI agents to run inside GitHub Actions triggered by repository events. The “Log Doctor” pattern uses this to:

1. Detect a failed build via a `workflow_run` trigger
2. Feed the failed job’s stderr to an LLM
3. Generate a targeted fix as a unified diff
4. Commit to a new branch and open a pull request automatically

Elastic validated this at production scale: their self-healing PR plugin fixed 24 initially broken PRs in its first month and saved an estimated 20 days of active development — while covering only 45% of the dependency surface (Elasticsearch Labs).

The YAML

“`yaml

# .github/workflows/self-heal.yml

name: Self-Healing Build

on:

workflow_run:

workflows: [“CI”]

types: [completed]

permissions:

contents: write

pull-requests: write

jobs:

log-doctor:

if: ${{ github.event.workflow_run.conclusion == ‘failure’ }}

runs-on: ubuntu-latest

steps:

• uses: actions/checkout@v4

• name: Download failed job logs

uses: actions/github-script@v7

id: get-logs

with:

script: |

const logs = await github.rest.actions.downloadWorkflowRunLogs({

owner: context.repo.owner,

repo: context.repo.repo,

run_id: ${{ github.event.workflow_run.id }}

});

return Buffer.from(logs.data).toString(‘utf-8’).slice(-8000);

• name: Run Log Doctor agent

uses: github/agentic-workflow@v1

with:

agent-model: gpt-4o-mini

system-prompt: |

You are a CI repair agent. Analyze the build log, identify the root cause,

and output a minimal code fix. Output ONLY the changed file path and a

unified diff. Do not modify tests or configuration files.

user-input: ${{ steps.get-logs.outputs.result }}

safe-outputs: true # Sanitizes output, blocks prompt injection

token: ${{ secrets.GITHUB_TOKEN }}

• name: Apply fix and open PR

uses: peter-evans/create-pull-request@v6

with:

token: ${{ secrets.GITHUB_TOKEN }}

branch: fix/auto-heal-${{ github.run_id }}

title: “[Auto] Fix build failure from run #${{ github.event.workflow_run.id }}”

body: “🤖 Generated by Log Doctor. Review carefully before merging.“

labels: auto-fix, needs-review

“`

The `safe-outputs: true` flag is critical. It tells the Agentic Workflows runtime to sanitize agent output before it reaches downstream steps — blocking prompt injection via crafted log messages. Agentic Workflows is currently free within your included Actions minutes.

—

Pattern 2 — AI Code Review with PR-Agent and GitHub Actions

Why one tool isn’t enough

Most “AI code review” setups install a single GitHub App and call it done. The problem: AI reviewers catch logic and design issues well, but they’re inconsistent on security vulnerabilities and language-specific style. You need layers.

The three-layer approach:

• PR-Agent (AI judgment layer) — understands intent, suggests refactors, and explains trade-offs. Open source and self-hostable with 10,500+ GitHub stars, it gained Claude Opus 4.6 and Gemini-3-pro-preview support in its v0.32 February 2026 release.
• Semgrep (static security layer) — deterministic SAST rules for injection, secrets, and CVEs. Never LLM-influenced, always auditable.
• Ruff or ESLint (language linting layer) — enforces style and catches obvious bugs before human reviewers spend time on them.

This layering gives you AI judgment where it adds value without replacing the deterministic security checks that compliance requires.

The YAML

“`yaml

# .github/workflows/ai-review.yml

name: AI Code Review

on:

pull_request:

types: [opened, synchronize]

permissions:

pull-requests: write

contents: read

jobs:

semgrep:

runs-on: ubuntu-latest

steps:

• uses: actions/checkout@v4
• uses: returntocorp/semgrep-action@v1

with:

config: p/owasp-top-ten

lint:

runs-on: ubuntu-latest

steps:

• uses: actions/checkout@v4
• uses: astral-sh/ruff-action@v1 # swap for eslint-action if JS/TS

pr-agent:

runs-on: ubuntu-latest

needs: [semgrep, lint] # AI review only runs if static checks pass

steps:

• uses: actions/checkout@v4

• name: PR-Agent review

uses: Codium-ai/pr-agent@v0.32

env:

OPENAI_KEY: ${{ secrets.OPENAI_KEY }} # or ANTHROPIC_KEY for Claude

GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

with:

command: review

args: >

–pr_reviewer.require_focused_review=true

–pr_reviewer.num_code_suggestions=5

–pr_reviewer.inline_code_comments=true

“`

Setting `needs: [semgrep, lint]` means PR-Agent fires only after your deterministic checks pass. Reviewers see a clean security and style signal before the AI weighs in on logic.

Self-hosted vs. GitHub App

If you’d rather skip YAML entirely, CodeRabbit connects as a GitHub App — no workflow file, zero configuration to start. It has connected over 2 million repositories and processed 13 million+ pull requests. The trade-off: you can’t gate other jobs on its output, so it works as an async advisor rather than a blocking quality gate.

PR-Agent self-hosted gives you the blocking gate. CodeRabbit gives you faster onboarding. Pick based on whether you need CI integration or better PR comments.

—

Pattern 3 — LLM-as-a-Judge Quality Gate

Why traditional assertions break for AI features

Your product’s AI chatbot, recommendation engine, or summarizer doesn’t have a boolean correct answer. A unit test can check that your function returns a string. It can’t check whether that string is helpful, accurate, or safe. Manual QA doesn’t scale when you’re shipping daily.

The LLM evaluation pipeline pattern solves this: a separate judge model scores your feature’s outputs against a fixed eval dataset and fails the build if average quality drops below a threshold.

The 2026 production consensus is a 90/10 split: LLM-as-Judge handles thousands of automated evaluations per day while humans handle calibration, edge cases, and regulatory-defensible decisions (Label Your Data).

Two solid open-source options:

• DeepEval — Python-native, rich built-in metrics (G-Eval, hallucination detection, RAG faithfulness), integrates directly with pytest
• Promptfoo — YAML-first config, ideal for teams managing prompts as configuration, supports assertion chaining

The YAML

“`yaml

# .github/workflows/llm-eval.yml

name: LLM Quality Gate

on:

push:

branches: [main, “feature/**”]

permissions:

contents: read

jobs:

eval:

runs-on: ubuntu-latest

steps:

• uses: actions/checkout@v4

• name: Set up Python

uses: actions/setup-python@v5

with:

python-version: “3.12”

• name: Install DeepEval

run: pip install deepeval

• name: Run LLM eval suite

env:

OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

run: |

deepeval test run tests/eval_suite.py \

–min-success-rate 0.85

# Job fails and blocks merge if success rate < 0.85

“`

Your `eval_suite.py` loads a fixed dataset of 50 input/expected-output pairs and scores each with a judge model. Score below 0.85 — the PR can’t merge.

Setting your threshold and managing cost

Start at 0.75 and calibrate up as you build confidence. To set a defensible threshold: run your eval suite on 20–30 human-labeled examples, require ≥80% agreement between the judge and your human raters, then set the threshold 5 points below your baseline to absorb natural variance.

Cost is more approachable than most teams expect. Running a 50-example eval suite with GPT-4o-mini costs approximately $0.02 per CI run (Traceloop). At 50 PRs per month, that’s $1/month for continuous AI quality coverage.

—

Security, Cost, and What Can Go Wrong

Security model across all three patterns

All three patterns share the same architecture:

• Scoped `GITHUB_TOKEN` with minimum required permissions — read for eval jobs, write only for PR creation
• Agent Workflow Firewall — a Squid proxy allowlist restricting outbound network access to approved endpoints only
• Threat-detection scan — GitHub’s built-in job scans agent outputs for prompt injection before publishing comments or commits
• No auto-merge — every AI-generated change is a PR; a human approves it

Known failure modes and fixes

Flaky judges: LLM judges exhibit a systematic verbosity bias of ~15% score inflation and position bias causing ~40% inconsistency in pairwise comparisons (Vadim’s blog). Fix: calibrate on human-labeled examples, use normalized scoring rubrics, and consider Claude Haiku if you see consistent inflation with GPT-4o-mini.

Noisy PR-Agent comments: Out of the box, PR-Agent can flood a PR with low-signal suggestions. The flags `–pr_reviewer.num_code_suggestions=5` and `–pr_reviewer.require_focused_review=true` keep comments actionable. Tune these in your first week.

Harmful self-healing fixes: An agent that can write to your repo can write bad code. The `safe-outputs` flag and mandatory human PR review are your safeguards. Never grant the agent merge permissions — not even temporarily.

When NOT to use AI agents in CI/CD

This is the question most articles skip. AI agents are the wrong tool when:

• The step must be auditable: Compliance-gated release checks, security scans, and license validation — keep these deterministic.
• Failure has blast radius: Deploy jobs, database migrations, infrastructure changes. An agent proposing a fix to a migration script is a high-risk proposition.
• You haven’t calibrated your eval yet: If your LLM judge has no human-labeled baseline, it will produce unreliable gates. Calibrate first, automate second.

The boundary: Agentic judgment at the edges. Deterministic execution at the core. Never mix both in the same job.

Cost per pattern at a glance

| Pattern | Tool | Approx. Cost per PR |

|—|—|—|

| Self-Healing Builds | GitHub Agentic Workflows (preview) | Free (within Actions minutes) |

| AI Code Review | PR-Agent self-hosted | ~$0.01–0.05 (LLM API calls) |

| LLM-as-a-Judge Gate | DeepEval + GPT-4o-mini (50 examples) | ~$0.02 |

—

Putting It All Together — A Full PR Lifecycle with AI Agents

Here’s how all three patterns sequence in a single pull request:

“`

1. Developer pushes → PR opened

└─ [ai-review.yml fires]

├─ Semgrep scans for security issues (deterministic)

├─ Ruff/ESLint lints the code (deterministic)

└─ PR-Agent posts review comments (agentic, only if static checks pass)

1. Developer pushes follow-up commit → CI runs

└─ Build fails?

└─ [self-heal.yml fires]

└─ Log Doctor opens a fix PR (agentic, human reviews)

1. All checks pass → PR ready for eval

└─ [llm-eval.yml fires]

└─ LLM judge scores AI feature outputs

├─ Score ≥ 0.85 → merge unblocked

└─ Score < 0.85 → build fails, merge blocked

1. Human reviews all AI suggestions and approves

└─ Merge to main

“`

The AI agents never merge. They inform, triage, and surface quality signals. Every merge is a human decision — that’s the design, not a limitation.

—

Start Your AI Agents CI/CD Pipeline Today

You don’t need all three on day one. If build failures are your biggest pain point, start with Pattern 1. If your team spends more review cycles than it should, start with Pattern 2. If you’re shipping an AI agents CI/CD pipeline for a product feature with no quality baseline, Pattern 3 is non-negotiable.

Gartner forecasts that applications embedding agentic capabilities will leap from under 5% in 2025 to 40% in 2026. The teams moving first aren’t waiting for enterprise budgets — they’re dropping YAML files into their repos and iterating.

Copy the config that solves your biggest friction today. Set a scoped token. Open a PR. The best AI CI/CD setup is the one that’s running.

—

Found a failure mode or variation that worked for your team? Share it in the comments — these configs improve with real-world feedback.