Champion/Challenger/Shadow: Modern Deployment Strategies for AI Agents

Deploying AI agents to production is fundamentally different from deploying traditional software. With deterministic systems, you validate behavior with tests. With AI agents, behavior emerges from prompts, model selection, and context - making traditional blue-green deployments inadequate.

At Cortex, we needed a deployment strategy that could:

1. Test new agent versions without impacting production
2. Validate behavioral changes with real traffic
3. Rollback instantly if something goes wrong
4. Maintain statistical rigor for decision-making

The answer? Champion/Challenger/Shadow - a three-tier deployment pattern designed specifically for AI systems.

Why Blue-Green Isn’t Enough

Traditional blue-green deployment works for web services:

• Deploy to blue environment
• Smoke test
• Switch traffic
• Done

But AI agents are different:

The Non-Determinism Problem: The same prompt with the same inputs can produce different outputs. You can’t validate with unit tests alone.

The Behavior Drift Problem: Small prompt changes can cause subtle behavioral shifts that only manifest under production load patterns.

The Context Problem: Agents build context over time. Switching all traffic instantly loses accumulated learning.

The Rollback Risk: If you deploy a bad version and switch 100% of traffic, you’re fully exposed before you know there’s a problem.

We needed something better.

The Three-Tier Pattern

Champion/Challenger/Shadow gives us progressive rollout with built-in safety:

graph TD
    A[Production Traffic<br/>100%] --> B{Traffic Split}

    B -->|100%| C[Champion v1.0<br/>Production Stable]
    B -->|Copy 100%| D[Shadow v1.1<br/>Observe Only]

    D --> E{Shadow<br/>Validation}
    E -->|Pass| F[Promote to<br/>Challenger]
    E -->|Fail| G[Iterate on v1.1]

    F --> H{New Traffic Split}
    H -->|90%| C
    H -->|10%| I[Challenger v1.1<br/>Real Traffic]
    H -->|Copy 100%| J[Shadow v1.2<br/>Next Version]

    I --> K{Statistical<br/>Validation}
    K -->|Pass| L[Promote to<br/>Champion]
    K -->|Fail| M[Rollback]

    L --> N[Champion v1.1<br/>100% Traffic]
    M --> C

    style A fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style B fill:#30363d,stroke:#f85149,stroke-width:2px
    style C fill:#30363d,stroke:#00d084,stroke-width:3px
    style D fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style E fill:#30363d,stroke:#f85149,stroke-width:2px
    style F fill:#30363d,stroke:#00d084,stroke-width:2px
    style G fill:#30363d,stroke:#f85149,stroke-width:2px
    style H fill:#30363d,stroke:#f85149,stroke-width:2px
    style I fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style J fill:#30363d,stroke:#8b949e,stroke-width:2px
    style K fill:#30363d,stroke:#f85149,stroke-width:2px
    style L fill:#30363d,stroke:#00d084,stroke-width:2px
    style M fill:#30363d,stroke:#f85149,stroke-width:2px
    style N fill:#30363d,stroke:#00d084,stroke-width:3px

Champion: The Production Version

The champion is your active production version. It handles 100% of user traffic (or 90% during canary testing). This is the version you trust.

Key Properties:

• Fully vetted through shadow and challenger stages
• Handles all production traffic by default
• Can be rolled back to instantly
• Performance baseline for comparisons

Challenger: The Canary Version

The challenger receives a small percentage of production traffic (typically 10%) for validation. Responses are used, so users see them, but exposure is limited.

Key Properties:

• Receives real production traffic
• Responses impact actual users (limited blast radius)
• Statistical comparison against champion
• Can be promoted to champion or rolled back

Shadow: The Monitoring Version

Shadow receives a copy of production traffic but responses are discarded. This is pure observation mode.

Key Properties:

• Zero production impact
• Full production traffic patterns
• Error detection without user exposure
• First stage for all new versions

Implementation in Cortex

Here’s how we implemented this at the file system level.

Directory Structure

coordination/masters/coordinator/
├── versions/
│   ├── v1.0.0/              # Stable production version
│   ├── v1.1.0/              # New candidate version
│   ├── v2.0.0/              # Experimental version
│   └── aliases.json         # Alias configuration
├── context/
├── knowledge-base/
└── workers/

Each master (security, development, CI/CD, etc.) has a versions directory with:

• Individual version directories (semantic versioning)
• An aliases.json file mapping aliases to versions

Alias Configuration

{
  "master_id": "coordinator",
  "aliases": {
    "champion": {
      "version": "v1.0.0",
      "description": "Production version (primary active)",
      "promoted_at": "2025-11-27T00:00:00Z",
      "status": "active"
    },
    "challenger": {
      "version": "v1.1.0",
      "description": "Canary version (10% traffic)",
      "promoted_at": "2025-11-27T12:00:00Z",
      "status": "active"
    },
    "shadow": {
      "version": null,
      "description": "Shadow version (monitoring only)",
      "promoted_at": null,
      "status": "inactive"
    }
  },
  "version_history": [
    {
      "version": "v1.1.0",
      "created_at": "2025-11-27T12:00:00Z",
      "description": "Promoted from shadow to challenger",
      "status": "challenger"
    },
    {
      "version": "v1.0.0",
      "created_at": "2025-11-27T00:00:00Z",
      "description": "Initial production version",
      "status": "champion"
    }
  ],
  "last_updated": "2025-11-27T12:00:00Z"
}

Version Library

We built a bash library for version management:

# scripts/lib/read-alias.sh

# Get the active champion version
get_champion_version() {
    local master_id="$1"
    local aliases_file="coordination/masters/${master_id}/versions/aliases.json"

    jq -r '.aliases.champion.version' "$aliases_file"
}

# Get full path to version directory
get_master_version_path() {
    local master_id="$1"
    local alias_name="$2"  # champion, challenger, or shadow

    local version=$(get_master_version "$master_id" "$alias_name")
    echo "coordination/masters/${master_id}/versions/${version}"
}

# Validate champion exists and is valid
validate_champion() {
    local master_id="$1"
    local champion_version=$(get_champion_version "$master_id")

    if [ -z "$champion_version" ]; then
        echo "ERROR: No champion version set"
        return 1
    fi

    if [ ! -d "coordination/masters/${master_id}/versions/${champion_version}" ]; then
        echo "ERROR: Champion version directory missing"
        return 1
    fi

    return 0
}

Runtime Integration

Every master agent validates its version at startup:

# scripts/run-coordinator-master.sh

source "$SCRIPT_DIR/lib/read-alias.sh"

main() {
    log_section "Starting Coordinator Master"

    # Validate champion version exists
    if ! validate_champion "coordinator"; then
        log_error "Champion version validation failed"
        exit 1
    fi

    MASTER_VERSION=$(get_champion_version "coordinator")
    log_info "Running champion version: $MASTER_VERSION"

    # Load version-specific configuration
    VERSION_PATH=$(get_master_version_path "coordinator" "champion")
    source "${VERSION_PATH}/config.sh"

    # Continue with normal execution...
}

This ensures:

1. Invalid versions never run
2. Version is logged for observability
3. Rollback is immediate (update alias, restart)

Progressive Deployment Workflow

Here’s a real production deployment sequence with decision gates at each stage:

graph TD
    A[New Version v1.1.0<br/>Created] --> B[Deploy to Shadow]
    B --> C[Monitor 1-2 Hours]
    C --> D{Decision Gate 1}

    D -->|Error Rate OK?<br/>Response Time OK?<br/>No Crashes?| E[Promote to<br/>Challenger]
    D -->|Issues Found| F[Fix & Iterate]
    F --> B

    E --> G[Monitor 24+ Hours<br/>Min 30 Tasks]
    G --> H{Decision Gate 2}

    H -->|Completion ≥ Champion?<br/>Quality ≥ Champion?<br/>No New Errors?| I[Promote to<br/>Champion]
    H -->|Issues Found| J[Rollback or Iterate]
    J --> K{Severity}
    K -->|Critical| L[Immediate Rollback]
    K -->|Minor| F

    I --> M[Monitor First Hour<br/>Closely]
    M --> N{Decision Gate 3}

    N -->|All Metrics Stable?| O[Deployment Complete]
    N -->|Issues Detected| L

    L --> P[Revert to Previous<br/>Champion]

    style A fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style B fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style C fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style D fill:#30363d,stroke:#f85149,stroke-width:3px
    style E fill:#30363d,stroke:#00d084,stroke-width:2px
    style F fill:#30363d,stroke:#f85149,stroke-width:2px
    style G fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style H fill:#30363d,stroke:#f85149,stroke-width:3px
    style I fill:#30363d,stroke:#00d084,stroke-width:2px
    style J fill:#30363d,stroke:#f85149,stroke-width:2px
    style K fill:#30363d,stroke:#f85149,stroke-width:2px
    style L fill:#30363d,stroke:#f85149,stroke-width:3px
    style M fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style N fill:#30363d,stroke:#f85149,stroke-width:3px
    style O fill:#30363d,stroke:#00d084,stroke-width:3px
    style P fill:#30363d,stroke:#f85149,stroke-width:2px

Step 1: Deploy to Shadow

# Create new version directory
mkdir coordination/masters/coordinator/versions/v1.1.0

# Copy champion as starting point
cp -r coordination/masters/coordinator/versions/v1.0.0/* \
      coordination/masters/coordinator/versions/v1.1.0/

# Make your changes in v1.1.0/
# ... edit prompts, configs, logic ...

# Deploy to shadow
./scripts/promote-master.sh coordinator deploy-shadow v1.1.0

What happens:

• v1.1.0 becomes the shadow version
• It receives a copy of all traffic
• Responses are discarded
• Errors are logged but don’t impact users

What to watch:

• Error rates
• Response times
• Unexpected behaviors
• Resource usage

Duration: 1-2 hours minimum. Look for obvious issues.

Step 2: Promote to Challenger

# After shadow validation looks good
./scripts/promote-master.sh coordinator promote-to-challenger

What happens:

• v1.1.0 moves from shadow to challenger
• It now receives 10% of real traffic
• Responses are used in production
• Champion handles remaining 90%

What to watch:

• Completion rate vs champion
• Quality score vs champion
• User-reported issues
• Statistical significance

Duration: 24 hours minimum. Need enough samples for statistical confidence.

Step 3: Promote to Champion

# After challenger validates successfully
./scripts/promote-master.sh coordinator promote-to-champion

What happens:

• v1.1.0 becomes the new champion
• It receives 100% of traffic
• v1.0.0 remains available for rollback
• Version history records the promotion

What to watch:

• First hour critically important
• Watch for edge cases at full load
• Monitor system-wide metrics
• Keep rollback plan ready

Metrics and Decision Making

The challenger stage is where we gather data for promotion decisions. Here’s what we track:

Core Metrics

{
  "version": "v1.1.0",
  "role": "challenger",
  "metrics": {
    "tasks_assigned": 127,
    "tasks_completed": 121,
    "tasks_failed": 6,
    "completion_rate": 95.3,
    "avg_duration_seconds": 38.7,
    "avg_quality_score": 4.4
  }
}

Statistical Comparison

We need statistical significance before promotion:

Minimum Sample Size: 30 tasks per version Minimum Duration: 24 hours Confidence Level: 95%

Example comparison:

Champion (v1.0.0):
  Completion Rate: 92.5%
  Avg Duration: 45.2s
  Quality Score: 4.1/5

Challenger (v1.1.0):
  Completion Rate: 95.8%
  Avg Duration: 38.7s
  Quality Score: 4.4/5

Decision: PROMOTE
Reasoning:
  - 3.3% improvement in completion rate (statistically significant)
  - 14.4% faster (6.5s improvement)
  - Higher quality scores
  - No new error patterns

Decision Framework

We promote when challenger meets all criteria:

graph TD
    A[Challenger Metrics<br/>Collected] --> B{Performance<br/>Check}
    B -->|Completion ≥ Champion| C{Quality<br/>Check}
    B -->|Completion < Champion| X[ITERATE]

    C -->|Quality ≥ Champion| D{Reliability<br/>Check}
    C -->|Quality < Champion| X

    D -->|No New Errors| E{Efficiency<br/>Check}
    D -->|New Errors Found| X

    E -->|Resource Usage OK| F{Time<br/>Check}
    E -->|Resource Issues| X

    F -->|≥ 24 Hours| G{Volume<br/>Check}
    F -->|< 24 Hours| Y[CONTINUE<br/>MONITORING]

    G -->|≥ 30 Tasks| H[PROMOTE]
    G -->|< 30 Tasks| Y

    X --> Z1[Fix Issues<br/>New Version]
    Y --> A
    H --> Z2[Deploy to<br/>Champion]

    style A fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style B fill:#30363d,stroke:#f85149,stroke-width:2px
    style C fill:#30363d,stroke:#f85149,stroke-width:2px
    style D fill:#30363d,stroke:#f85149,stroke-width:2px
    style E fill:#30363d,stroke:#f85149,stroke-width:2px
    style F fill:#30363d,stroke:#f85149,stroke-width:2px
    style G fill:#30363d,stroke:#f85149,stroke-width:2px
    style H fill:#30363d,stroke:#00d084,stroke-width:3px
    style X fill:#30363d,stroke:#f85149,stroke-width:2px
    style Y fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style Z1 fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style Z2 fill:#30363d,stroke:#00d084,stroke-width:2px

Promotion Criteria:

1. Performance: Equal or better completion rate
2. Quality: Equal or better quality scores
3. Reliability: No new error patterns
4. Efficiency: Acceptable resource usage
5. Time: Minimum 24 hours of data
6. Volume: Minimum 30 completed tasks

Outcomes:

• PROMOTE: All criteria met, deploy to champion
• CONTINUE MONITORING: Need more time or data
• ITERATE: Issues found, fix and redeploy
• ABANDON: Fundamental issues, discard version

Rollback Strategies

The beauty of this system is instant, zero-downtime rollback:

graph TD
    A[Issue Detected] --> B{Severity<br/>Assessment}

    B -->|Critical<br/>Production Impact| C[Emergency Rollback]
    B -->|Moderate<br/>Degraded Performance| D[Planned Rollback]
    B -->|Minor<br/>Edge Cases| E[Continue Monitoring]

    C --> F[Immediate Action]
    F --> G[Set Champion to<br/>Previous Version]
    G --> H[Restart Active<br/>Workers]
    H --> I[Monitor Recovery]
    I --> J{Stable?}
    J -->|Yes| K[Rollback Complete<br/>< 5 minutes]
    J -->|No| L[Escalate Issue]

    D --> M[Schedule Rollback<br/>Low Traffic Window]
    M --> G

    E --> N[Document Issue]
    N --> O{Fix Available?}
    O -->|Yes| P[Deploy Fix<br/>to Shadow]
    O -->|No| D

    style A fill:#30363d,stroke:#f85149,stroke-width:2px
    style B fill:#30363d,stroke:#f85149,stroke-width:3px
    style C fill:#30363d,stroke:#f85149,stroke-width:3px
    style D fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style E fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style F fill:#30363d,stroke:#f85149,stroke-width:2px
    style G fill:#30363d,stroke:#f85149,stroke-width:2px
    style H fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style I fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style J fill:#30363d,stroke:#f85149,stroke-width:2px
    style K fill:#30363d,stroke:#00d084,stroke-width:3px
    style L fill:#30363d,stroke:#f85149,stroke-width:2px
    style M fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style N fill:#30363d,stroke:#58a6ff,stroke-width:2px
    style O fill:#30363d,stroke:#f85149,stroke-width:2px
    style P fill:#30363d,stroke:#58a6ff,stroke-width:2px

Simple Rollback

./scripts/promote-master.sh coordinator rollback

What happens:

• Champion reverts to previous version from history
• Change is atomic (JSON update)
• Next master spawn uses old version
• Active workers continue until task completion

Rollback Speed:

• Alias update: < 100ms
• New spawns use old version: Immediate
• Full fleet rollback: < 5 minutes (as workers cycle)

Emergency Rollback

For critical issues, bypass the flow:

./scripts/promote-master.sh coordinator set-champion v1.0.0

This directly sets the champion version. Use only when:

• Production is significantly impacted
• Challenger is causing cascading failures
• Need instant recovery

Warning: This bypasses safety checks. The script requires manual confirmation.

Rollback Decision Triggers

We automatically consider rollback when:

Error Rate: > 5% higher than champion baseline Response Time: > 2x champion average Quality Score: Drops below acceptable threshold Critical Functionality: Core features broken

Example:

11:23 AM - Challenger v1.2.0 deployed
11:45 AM - Error rate spike detected: 8.2% (baseline: 2.1%)
11:47 AM - Manual investigation begins
11:52 AM - Root cause identified: prompt regression
11:53 AM - Rollback initiated
11:54 AM - Error rate returning to baseline
11:58 AM - Rollback complete, all workers on v1.1.0

Production Learnings

After deploying this system across 5 master agents, here’s what we learned.

What Works Well

Progressive Risk Reduction: The three stages catch issues at increasing levels of exposure. Shadow catches obvious bugs. Challenger catches subtle behavioral issues. Champion is thoroughly vetted.

Statistical Rigor: Forcing 24+ hours and 30+ tasks prevents premature promotion. We’ve caught several issues that only appeared after sustained load.

Instant Rollback: JSON-based aliases mean rollback is atomic. No deployment pipeline, no build process - just update the file and restart.

Audit Trail: Version history provides complete lineage. We can trace every promotion, rollback, and reason.

What Was Tricky

Traffic Splitting Complexity: We haven’t implemented true 10% traffic splitting yet. Currently, it’s all-or-nothing per worker spawn. Next iteration will add probabilistic routing.

Shadow Cost: Running shadow versions doubles infrastructure for that master. We’re selective about what runs in shadow mode.

Version Proliferation: Need active cleanup of old versions. We keep last 3 versions per master maximum.

Context Loss: When rolling back, accumulated context is lost. We’re exploring context migration strategies.

Best Practices We’ve Established

Never Skip Stages: Always go shadow → challenger → champion. Skipping stages has bitten us every time.

Document Everything: Each version directory has a README explaining what changed and why.

Monitor Actively: First hour of challenger is critical. Watch metrics closely.

Small Changes: Each version changes one thing. Prompt update OR config change OR logic fix - not all three.

Rollback Test: Verify rollback works in staging before production promotion.

Integration with A/B Testing

We also built an A/B testing framework on top of this infrastructure:

# Create A/B test: 80% champion, 20% challenger
create_ab_test "security-v2-test" \
    "security-master" \
    "v1.0.0" \
    "v2.0.0" \
    80

# Get variant for specific task
variant=$(get_variant "security-v2-test" "task-12345")

# Route based on variant
if [[ "$variant" == "a" ]]; then
    version="v1.0.0"
else
    version="v2.0.0"
fi

This enables:

• Gradual rollout (80/20, then 50/50, then 20/80)
• Statistical comparison under identical conditions
• Per-task variant consistency (same task always same version)
• Multi-version testing (more than 2 versions)

Comparison with Traditional Patterns

How does Champion/Challenger/Shadow compare to other deployment strategies?

vs. Blue-Green

Blue-Green: Two identical environments. Switch traffic instantly.

Champion/Challenger/Shadow: Three exposure levels with progressive rollout.

Winner: C/C/S for AI agents. Blue-green is too binary - you’re either 0% or 100% exposed. AI needs gradual validation.

vs. Canary

Canary: Route small percentage to new version, gradually increase.

Champion/Challenger/Shadow: Canary + shadow mode for zero-impact testing.

Winner: C/C/S is superset of canary. Shadow stage adds safety. Champion/challenger are canary implementation.

vs. Feature Flags

Feature Flags: Toggle features on/off at runtime.

Champion/Challenger/Shadow: Version-level switching with full isolation.

Winner: Different use cases. Feature flags for runtime toggling. C/C/S for version deployment. Both useful.

vs. Rolling Update

Rolling Update: Replace instances one at a time.

Champion/Challenger/Shadow: All instances run same version, version determined by alias.

Winner: Rolling for infrastructure. C/C/S for AI behavior validation. Often used together.

Real-World Scenario

Here’s how we deployed a major coordinator upgrade.

Context: New MoE routing algorithm. Significant change to how tasks are assigned.

Week 1: Shadow Deployment

Monday 9 AM: Deploy v2.0.0 to shadow

./scripts/promote-master.sh coordinator deploy-shadow v2.0.0

Monday 9 AM - Friday 5 PM: Monitor shadow version

• Error rate: 1.8% (baseline: 2.1%) ✓
• Response time: -15% (faster) ✓
• Resource usage: +5% (acceptable) ✓
• Routing decisions: Validated by security team ✓

Friday 5 PM: Shadow validation complete

Week 2: Challenger Validation

Monday 9 AM: Promote to challenger (10% traffic)

./scripts/promote-master.sh coordinator promote-to-challenger

Monday-Thursday: Statistical gathering

• Tasks assigned: 89 challenger, 834 champion
• Completion rate: 96.6% challenger, 94.2% champion ✓
• Quality scores: 4.5 challenger, 4.3 champion ✓
• User feedback: Positive (no complaints)

Thursday 2 PM: Challenger metrics strongly positive

Thursday 3 PM: Promote to champion

./scripts/promote-master.sh coordinator promote-to-champion

Week 2 (continued): Champion Validation

Thursday 3 PM - Friday 5 PM: Full load monitoring

• First hour: Watch closely for edge cases
• Hours 2-6: Metrics stable
• Next day: Confirm sustained performance

Friday 5 PM: v2.0.0 fully validated in production

Result: Major routing algorithm change deployed with zero downtime, zero user impact, and complete statistical validation.

Tooling and Scripts

The promotion script is the primary interface:

# View current configuration
./scripts/promote-master.sh coordinator show

# Output:
# === Master Version Aliases: coordinator ===
#
# Champion (Production):    v1.0.0
# Challenger (Canary):      v1.1.0
# Shadow (Monitoring):      none
#
# Available Versions:
#   - v1.0.0
#   - v1.1.0
#   - v1.2.0
#
# Version History (last 5):
#   2025-11-27T21:50:54Z: v1.1.0 (champion)
#   2025-11-27T21:50:50Z: v1.1.0 (challenger)
#   2025-11-27T21:50:45Z: v1.1.0 (shadow)
#   2025-11-27T00:00:00Z: v1.0.0 (champion)

Complete workflow:

# 1. Create version
mkdir coordination/masters/coordinator/versions/v1.1.0

# 2. Deploy to shadow
./scripts/promote-master.sh coordinator deploy-shadow v1.1.0

# 3. Monitor, then promote to challenger
./scripts/promote-master.sh coordinator promote-to-challenger

# 4. Validate, then promote to champion
./scripts/promote-master.sh coordinator promote-to-champion

# 5. If issues arise, rollback
./scripts/promote-master.sh coordinator rollback

Future Enhancements

We’re planning several improvements:

Automated Promotion

Use statistical analysis to automatically promote when thresholds are met:

# Auto-promote if metrics exceed thresholds
if [[ $completion_rate_improvement > 3 ]] && \
   [[ $quality_score >= 4.0 ]] && \
   [[ $error_rate < $baseline_error_rate ]]; then
    ./scripts/promote-master.sh coordinator promote-to-champion
fi

Gradual Traffic Shifting

Instead of fixed 10%, gradually increase:

# Day 1: 5%
# Day 2: 10%
# Day 3: 25%
# Day 4: 50%
# Day 5: 100%

Multi-Shadow Testing

Run multiple shadow versions simultaneously to compare candidates before selecting challenger.

Context Migration

When promoting versions, migrate learned context from champion to new champion.

Conclusion

Champion/Challenger/Shadow is purpose-built for AI agents. Traditional deployment patterns assume deterministic behavior - AI agents don’t have that luxury.

The three-tier approach gives us:

• Safety: Shadow testing with zero production impact
• Validation: Challenger provides statistical confidence
• Speed: Instant rollback when needed
• Rigor: Version history and audit trails

We’ve used this system to deploy 23 master agent versions across 5 agents with zero production incidents.

If you’re deploying AI agents to production, consider this pattern. The progressive rollout and instant rollback capabilities are worth the infrastructure investment.

The code is in production at Cortex. We’ve learned a lot - and we’re still iterating.

---

Next in the Cortex series: How we built statistical evaluation into the deployment pipeline to automatically validate agent behavior.