> ## Documentation Index
> Fetch the complete documentation index at: https://aitutorial.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Production Considerations

> Prompt versioning, monitoring, observability, and deployment checklists for production LLM systems

export const QuizQuestion = ({question, options, answer, explanation}) => {
  const [selected, setSelected] = useState(null);
  const [revealed, setRevealed] = useState(false);
  const handleSelect = index => {
    if (revealed) return;
    setSelected(index);
    setRevealed(true);
  };
  const isCorrect = selected === answer;
  const getOptionClass = i => {
    const classes = ['quiz-option'];
    if (revealed) {
      classes.push('quiz-option-disabled');
      if (i === answer) classes.push('quiz-option-correct'); else if (i === selected && !isCorrect) classes.push('quiz-option-wrong');
    }
    return classes.join(' ');
  };
  return <div className="quiz-card">
      <p className="quiz-question">{question}</p>
      <div className="quiz-options">
        {options.map((option, i) => <button key={i} onClick={() => handleSelect(i)} className={getOptionClass(i)}>
            <span className="quiz-letter">{String.fromCharCode(65 + i)}</span>
            {option}
          </button>)}
      </div>
      {revealed && <div className={`quiz-feedback ${isCorrect ? 'quiz-feedback-correct' : 'quiz-feedback-wrong'}`}>
          <strong>{isCorrect ? 'Correct!' : 'Incorrect.'}</strong> {explanation}
        </div>}
    </div>;
};

export const Quiz = ({title = "Check Your Understanding", children}) => {
  return <div style={{
    marginTop: '24px'
  }}>
      <div className="quiz-title">{title}</div>
      {children}
    </div>;
};

Shipping a prompt to production is where most projects fail. This page covers versioning, monitoring, and the checklist you need before deploying any LLM feature.

## Versioning and Change Management

**The Problem:**
You improve your prompt. It works great in testing. You deploy it. Customer complaints spike.

**Why It Happens:**

* Test set doesn't cover real distribution
* Edge cases appear in production
* Model updates can break prompts

**Production Pattern: Prompt Registry**

<CodeGroup>
  ```ts Pseudocode theme={null}
  function getPrompt(name: string, userId?: string): string {
      /**
       * Load prompt with A/B testing support.
       */
      const config = loadPrompts()[name];
      
      // Determine version
      let version: string;
      if (userId && shouldTest(userId, config.v3.rollout)) {
          version = "v3";
      } else {
          version = "v2";
      }
      
      return config[version].prompt;
  }
  ```

  ```yaml prompts.yaml theme={null}
  prompts:
    contract_summarizer:
      v1:
        created: "2025-01-01"
        prompt: "..."
        status: "deprecated"
        metrics:
          accuracy: 0.78
          avg_cost: 0.12
      
      v2:
        created: "2025-01-15"
        prompt: "..."
        status: "active"
        rollout: 100%
        metrics:
          accuracy: 0.92
          avg_cost: 0.08
      
      v3:
        created: "2025-02-01"
        prompt: "..."
        status: "testing"
        rollout: 10%
        metrics:
          accuracy: 0.94  # Looking good!
          avg_cost: 0.09
  ```
</CodeGroup>

## Monitoring and Observability

**What to Track:**

<CodeGroup>
  ```ts Pseudocode theme={null}
  async function generateResponse(prompt: string, userId: string): Promise<string> {
      const startTime = Date.now();
      
      const response = await llm.generate(prompt);
      
      // Log metrics
      logMetrics({
          user_id: userId,
          prompt_version: getVersion(prompt),
          latency: Date.now() - startTime,
          input_tokens: countTokens(prompt),
          output_tokens: countTokens(response),
          cost: calculateCost(prompt, response),
          timestamp: new Date()
      });
      
      // Check for issues
      if (detectHallucination(response)) {
          alert("Possible hallucination detected");
      }
      
      if (detectPromptInjection(prompt)) {
          alert("Prompt injection attempt");
      }
      
      return response;
  }
  ```
</CodeGroup>

**Dashboard Metrics:**

* Requests per minute
* Average latency (p50, p95, p99)
* Cost per request
* Error rate
* User satisfaction (thumbs up/down)
* Cache hit rate
* Model distribution (if cascading)

**AI Observability Tools:**

Several tools can help you implement comprehensive monitoring:

* **Open Source:** Phoenix, LangFuse, Opik
* **Commercial:** Arize, AgentOps
* **Product-Specific:** Langsmith (for LangChain and LangGraph applications)

These tools provide features like prompt versioning, cost tracking, latency monitoring, and quality metrics out of the box.

## The Production Checklist

Before deploying any LLM feature:

**Testing:**

* [ ] Evaluation dataset created (100+ examples)
* [ ] Accuracy meets requirements (>90%)
* [ ] Edge cases tested
* [ ] Failure modes documented

**Cost:**

* [ ] Cost per request measured
* [ ] Caching implemented where possible
* [ ] Model selection optimized
* [ ] Budget alerts configured

**Safety:**

* [ ] Input validation in place
* [ ] Output validation in place
* [ ] Prompt injection defenses tested
* [ ] Fallback behavior defined

**Observability:**

* [ ] Metrics logging configured
* [ ] Alerts set up
* [ ] Dashboard created
* [ ] On-call runbook written

**Deployment:**

* [ ] A/B testing framework ready
* [ ] Rollout plan defined (10% → 50% → 100%)
* [ ] Rollback procedure documented
* [ ] Customer communication prepared

***

<Quiz>
  <QuizQuestion question="You deploy prompt v3 to 10% of traffic. Accuracy is 94% (vs 92% for v2). Should you immediately roll to 100%?" options={["Yes — 94% beats 92%, ship it", "No — 10% traffic may not cover all edge cases; monitor for at least a week and check segment-level metrics before full rollout", "Only if latency is also lower"]} answer={1} explanation="10% rollout might miss low-frequency query types. Gradual rollout with monitoring catches regressions that small samples miss." />

  <QuizQuestion question="Your LLM system's p50 latency is 200ms but p99 is 8 seconds. What does this tell you?" options={["The system is fine — average latency is fast", "1% of users experience 40x worse latency, likely due to long outputs or retries — you need to investigate the tail", "You should only optimize p50 since it affects most users"]} answer={1} explanation="P99 reveals the worst user experience. A 40x gap between p50 and p99 suggests some queries trigger much more work (long outputs, retries, or timeouts) that need investigation." />
</Quiz>