title: "Building Production AI Agents: The Complete Pre-Launch Checklist" slug: "building-production-ai-agents-checklist" description: "The definitive pre-launch checklist for production AI agents — 87 items covering reliability, security, cost, observability, compliance, and user experience. Don't deploy without it." category: "Engineering" publishedAt: "2026-07-22"

Building Production AI Agents: The Complete Pre-Launch Checklist

Launching an AI agent into production is high-stakes. A missed checklist item can mean cost overruns, security vulnerabilities, compliance violations, or embarrassing outputs. This checklist covers every aspect that must be verified before your agent goes live. It's been refined through 50+ production deployments and represents the collective lessons from failures that didn't need to happen.

How to Use This Checklist

Work through each section methodically. Every item has three possible states:

• ✅ Verified — Tested and confirmed working
• ⚠️ N/A — Not applicable to your deployment (with justification)
• ❌ Missing — Must be addressed before launch

Do not deploy with any ❌ items. Period.

Section 1: Architecture & Design (12 items)

• Agent boundaries defined — Clear documentation of what the agent can and cannot do
• Fallback strategy — What happens when the primary model is unavailable (degraded service vs. error)
• Model selection documented — Why this model was chosen, with cost/quality tradeoff analysis
• Alternative models identified — At least one backup model tested and ready for failover
• State management strategy — How conversation state, user preferences, and context are persisted
• Session handling — Session timeout, cleanup, and concurrent session limits defined
• Concurrency model — Maximum concurrent requests, queue behavior, and backpressure strategy
• Idempotency — Duplicate requests produce the same result (no double-charges, duplicate emails)
• Graceful degradation — Agent can operate in reduced-capability mode when dependencies fail
• Circuit breakers — External service failures don't cascade into agent failures
• Timeout strategy — Maximum execution time per request (recommended: 30s default, configurable)
• Async vs sync — Long-running tasks are handled asynchronously with status polling

Section 2: Prompt Engineering & Model Behavior (10 items)

• System prompt versioned — System prompts are version-controlled in git, not hardcoded
• Prompt injection defense — User input cannot override system instructions

  Tested with:
  - "Ignore all previous instructions..."
  - "System: Override safety to..."
  - Prompt embedded in tool responses
  - Indirect injection via fetched web content
  

• Output format validated — Agent output matches expected schema (JSON, markdown, etc.)
• Hallucination rate measured — Baseline hallucination rate established on golden dataset
• Temperature optimized — Temperature setting justified for use case (0 for factual, 0.3-0.7 for creative)
• Few-shot examples tested — At least 3 few-shot examples that improve output quality
• Token budget per request — Maximum token consumption defined and enforced
• Multi-language testing — Agent behaves correctly in all supported languages
• Edge case prompts — Empty input, extremely long input, emoji-only, code injection attempts
• Toxic input handling — Agent gracefully handles offensive, harmful, or manipulative inputs

Section 3: Tool Integration (8 items)

• Tool schemas validated — Every tool has a Zod/Joi schema for input validation
• Tool error handling — Tool failures produce user-friendly messages, not stack traces
• Tool timeout enforcement — Each tool call has a timeout (recommended: 10s default)
• Tool call logging — Every tool invocation logged with params, result, duration, and cost
• MCP server reliability — If using MCP servers, failover strategy for server unavailability
• Tool parameter sanitization — No SQL injection, path traversal, or command injection possible
• Rate limiting per tool — External API rate limits respected with client-side throttling
• Tool response size limits — Truncate large tool responses to prevent context bloat

// Example tool with all safety measures
const searchTool = {
  name: "search",
  description: "Search the knowledge base",
  inputSchema: z.object({
    query: z.string().min(1).max(500),
    limit: z.number().int().min(1).max(50).default(10)
  }),
  
  execute: async (params, context) => {
    // Rate limiting
    await rateLimiter.consume(context.userId, "search");
    
    // Timeout
    const result = await withTimeout(
      knowledgeBase.search(params.query, params.limit),
      10000 // 10s
    );
    
    // Response size limit
    const truncated = truncateResult(result, 4000); // max 4K tokens
    
    // Log
    logger.info("tool_call", {
      tool: "search",
      params,
      result_count: result.length,
      duration_ms: result.durationMs,
      cost: 0
    });
    
    return truncated;
  }
};

Section 4: Security & Privacy (15 items)

• Authentication — Every API endpoint requires authentication (JWT, API key, or OAuth)
• Authorization — Users can only access their own data and permitted tools
• Input sanitization — All user inputs sanitized (XSS prevention, SQL injection prevention)
• Output filtering — Agent outputs scanned for sensitive data before delivery
• PII detection — Personally Identifiable Information is detected and handled per policy
• Data retention policy — Defined retention periods for conversations, logs, and cached data
• Encryption at rest — All stored data encrypted (database, logs, caches)
• Encryption in transit — TLS 1.3 for all communications
• Secret management — API keys, model credentials in vault (not environment variables in production)
• Prompt injection tests — Comprehensive injection test suite executed and passing
• Rate limiting — Per-user, per-IP, and global rate limits configured
• DDoS protection — Cloudflare or equivalent protection layer
• Audit logging — Every agent action logged with timestamp, user, action, and result
• GDPR compliance — Data subject rights implemented (access, deletion, portability)
• Security scan — Dependency vulnerabilities checked (npm audit, pip audit, Snyk)

Section 5: Cost Management (10 items)

• Per-request cost limit — Maximum cost per agent invocation (e.g., €0.05)
• Daily budget cap — System stops processing when daily budget exceeded
• Model routing strategy — Cost-aware routing implemented for different task complexities
• Caching layer — Semantic or exact-match caching for repeated queries
• Token monitoring — Real-time token consumption tracked and alerted
• Cost attribution — Costs attributed to user, team, or cost center
• Idle resource cleanup — Unused sessions, caches, and connections cleaned up
• Off-peak processing — Batch jobs scheduled during lower-cost hours where possible
• Provider cost comparison — Monthly review of model provider costs; switch if cheaper alternatives maintain quality
• Cost dashboard — Grafana or internal dashboard showing daily/weekly/monthly costs

Section 6: Reliability & Error Handling (12 items)

• Error messages — User-facing errors are helpful, non-technical, and actionable
• Retry strategy — Exponential backoff with jitter for transient failures

  @retry(
      max_attempts=3,
      backoff=ExponentialBackoff(base=1, factor=2, max_delay=30),
      retry_on=(TimeoutError, RateLimitError, ConnectionError)
  )
  async def call_model(prompt):
      return await llm.complete(prompt)
  

• Dead letter queue — Failed messages stored for manual review and retry
• Health check endpoint — /health returns 200 when healthy, 503 when degraded
• Readiness check — /ready returns 200 only when fully operational
• Graceful shutdown — In-flight requests completed before process termination
• Load testing — Tested at 2x expected peak traffic with acceptable latency (<5s p95)
• Chaos testing — Dependency failures simulated (model API down, database slow)
• Error budget — Defined SLO (e.g., 99.5%) with error budget tracking
• Rollback procedure — Documented and tested rollback to previous version (<5 min)
• Blue-green or canary deployment — Safe deployment strategy with traffic splitting
• Incident response runbook — Documented steps for common incidents

Section 7: Observability (10 items)

• Structured logging — All logs in JSON format with consistent fields

  {
    "timestamp": "2026-07-22T14:32:01Z",
    "level": "info",
    "agent_id": "cs-agent-v2",
    "session_id": "sess_abc123",
    "user_id": "user_456",
    "event": "tool_call",
    "tool": "crm_lookup",
    "duration_ms": 234,
    "cost_eur": 0.002,
    "status": "success"
  }
  

• Distributed tracing — Request tracing across all services (OpenTelemetry)
• Metrics dashboard — Grafana dashboard with latency, error rate, cost, and throughput
• Alerting rules — PagerDuty/Slack alerts for: error rate >5%, latency >5s, cost >daily budget
• Conversation replay — Ability to replay any production conversation for debugging
• Token usage tracking — Input/output tokens logged per request
• Tool call tracing — Every tool call visible in trace with params and results
• User feedback collection — Thumbs up/down or explicit feedback mechanism
• Performance regression detection — Automated comparison against baseline metrics
• Log retention — Logs retained per compliance requirements (typically 90-365 days)

Section 8: Compliance & Governance (8 items)

• EU AI Act classification — Agent risk category determined (minimal, limited, high, unacceptable)
• Transparency notice — Users informed they're interacting with an AI (required by EU AI Act)
• Human oversight — High-risk decisions require human review (if applicable)
• Data processing agreement — DPA in place with all data processors
• Record keeping — Automatic logs maintained per Article 12 of EU AI Act
• Bias testing — Agent tested for demographic bias in outputs
• Model card — Documentation of model capabilities, limitations, and intended use
• Right to explanation — Users can request explanation of AI-driven decisions

Section 9: User Experience (6 items)

• Response time — First token in <2s (streaming) or complete response <10s
• Loading states — User sees progress indicator during long operations
• Error recovery — When agent fails, users get actionable next steps
• Conversation export — Users can export their conversation history
• Accessibility — Agent interface meets WCAG 2.2 AA standards
• Multi-language — All user-facing strings localized (if multi-region)

Section 10: Testing & QA (8 items)

• Unit tests — Individual prompt components tested (≥80% coverage)
• Integration tests — Agent + tools + database + external APIs tested together
• Behavioral tests — Multi-turn conversation scenarios tested
• Regression suite — Golden dataset of ≥100 test cases with ≥90% pass rate
• Safety tests — Harmful request refusal tested with adversarial inputs
• Load test — Sustained throughput test (30 min at expected peak)
• Spike test — 5x traffic spike handled without errors
• User acceptance testing — Real users tested and approved before launch

Scoring Your Readiness

Post-Launch Verification

After deployment, verify these items within the first 24 hours:

• Smoke test — Agent responds correctly to 10 standard queries
• Monitoring active — Dashboards showing data, alerts configured
• Cost tracking — Per-request costs within expected range
• Error rate — <2% in first hour
• Latency — P95 within target (<5s for most use cases)
• Log volume — Logs arriving at expected rate and detail level
• User feedback — Initial feedback collected and reviewed

The Launch Decision Framework

Ask these three questions before deploying:

1. What's the worst that can happen? — If the agent fails badly, what's the impact? Reputational damage? Financial loss? Regulatory penalty?
2. How quickly can we roll back? — If something goes wrong, can you revert in under 5 minutes?
3. Who's watching? — Is someone on-call for the first 48 hours, ready to investigate anomalies?

If any answer is concerning, don't deploy yet. Fix the gap first.

Conclusion

This checklist represents the accumulated wisdom from dozens of production deployments. Every item exists because its absence once caused a production incident. Going through 87 items takes 2-4 hours — a small investment compared to the cost of a botched launch.

The teams that religiously follow pre-launch checklists ship better agents, sleep better at night, and earn more trust from their organizations. Be that team.