Hands-On Exercise

Build a Complete, Production-Grade RAG System and Evaluate It Rigorously

Project Requirements

1. Document Corpus Selection (Choose one):

Option A: Use provided corpus (technical documentation, 50 docs)
Option B: Bring your own (company docs, research papers, etc.)

2. Implementation Requirements: Must implement:

Two-stage retrieval (first-pass + reranking)
Hybrid search (BM25 + semantic)
Appropriate chunking strategy with metadata
Error handling and fallbacks
Logging and observability

Bonus (choose 1+):

Advanced pattern (GraphRAG, iterative, or agentic)
Unstructured data handling (PDFs, images, tables)
Query preprocessing/expansion
Result caching

3. Evaluation Requirements: Must include:

Golden dataset (minimum 20 test cases)
Retrieval metrics (Recall@5, Precision@5, NDCG@10) — prefer LlamaIndex evaluations
Generation metrics (Faithfulness, Relevance) — prefer LlamaIndex evaluations
Component-level analysis (retrieval vs. generation failures)
Failure case analysis (identify and document 3 worst queries)

4. Optimization: Must demonstrate:

Chunking strategy comparison (test 2+ strategies)
Search strategy comparison (BM25, semantic, hybrid)
Document optimization decisions with metrics

Module Exercises

Chunking Strategies

Experiment with chunking strategies:

Pseudocode

// Your task:
// 1. Load 'Assets/paul_graham_essay.txt'
// 2. Chunk with: fixed-size (200 char), semantic (500 char), structure-aware (if converted to HTML/Markdown)
// 3. For each strategy, evaluate:
//    - How many chunks produced?
//    - Are section boundaries respected?
//    - Does any important info get split awkwardly?
// 4. Pick best strategy for this document type and justify

// Expected insight: Semantic chunking preserves paragraph/context well for essays;
// structure-aware helps if headings are available

Search Strategy Selection

Implement and compare all three approaches on the same dataset:

Pseudocode

// Pseudocode: Comparison Task Logic
async function compareSearchStrategies() {
  // 1. Data Preparation
  const essay = await loadFile('Assets/paul_graham_essay.txt');
  const corpus = chunkDocument(essay);

  // 2. Define Benchmark
  const queries = [
    "exact phrases",        // Targets BM25
    "thematic concepts",    // Targets Semantic
    "mixed queries"         // Targets Hybrid
  ];

  // 3. Execution & Comparison
  for (const query of queries) {
    const lexical = searchLexical(corpus, query);
    const semantic = searchSemantic(corpus, query);
    const hybrid = searchHybrid(corpus, query);

    // 4. Analysis
    logStrategyWins({ lexical, semantic, hybrid });
  }
}

Reranking

Pseudocode

// Your task:
// 0) Corpus: use chunks from 'Assets/paul_graham_essay.txt' (treat each chunk as a doc)
// 1) Use HybridRetriever to get top-50 for 5 queries.
// 2) Rerank with CrossEncoderReranker to top-5.
// 3) Compute NDCG@10, MRR before vs. after reranking.
// 4) Record end-to-end latency; note candidate pool size impact.
//
// Expected insights:
// - Reranking lifts precision-focused metrics.
// - Most gains come from better ordering of already-relevant docs.
// - Larger candidate pools help recall but increase latency.

Unstructured Data

Process a mixed document corpus:

Pseudocode

async function processMixedCorpus(documents: File[]) {
  // 1. Process text document
  const textDoc = documents.find(d => d.name === "paul_graham_essay.txt");
  const textResult = await processText(textDoc);

  // 2. Apply chunking strategy (see Chunking Strategies)
  const chunks = chunkDocument(textResult.content, { strategy: "semantic" });
  storeMetadata(chunks);

  // 3. (Optional) Compare with PDF/Image processing
  if (hasMultimedia(documents)) {
    const visionResult = await processVision(documents.image);
    const pdfResult = await processPDF(documents.pdf);

    compareLatencyAndQuality(textResult, visionResult, pdfResult);
  }

  // 4. Analyze results
  // - Measure processing time
  // - Check for quality risks (short chunks, OCR noise)
  // - Validate confidence scores
}

// Expected Insights:
// - Text path is high-confidence & fast
// - Vision/OCR adds latency & uncertainty

RAG Evaluation

Evaluate your RAG system:

// Your task:
// 0. Corpus: use chunks from 'Assets/paul_graham_essay.txt' for retrieval
// 1. Load test cases (or create 20 Q/A pairs about the essay if missing)
// 2. Run RAG system on all queries
// 3. Use LlamaIndex evaluations to compute retrieval metrics (Recall@5, Precision@5, NDCG@10)
// 4. Use LlamaIndex evaluations for generation (Faithfulness, Relevance)
// 5. Identify 3 worst-performing queries
// 6. Debug: Is failure in retrieval or generation?

// Expected insights:
// - Retrieval usually bottleneck (70% of failures)
// - Multi-hop queries hardest
// - Ambiguous queries need query refinement

Home

Context Engineering & Prompt Design

Retrieval Augmented Generation (RAG)

AI Agents

Computer Vision

Coming Soon

Build a Complete, Production-Grade RAG System and Evaluate It Rigorously

Project Requirements

Module Exercises

Chunking Strategies

Search Strategy Selection

Reranking

Unstructured Data

RAG Evaluation

​Build a Complete, Production-Grade RAG System and Evaluate It Rigorously

​Project Requirements

​Module Exercises

​Chunking Strategies

​Search Strategy Selection

​Reranking

​Unstructured Data

​RAG Evaluation

Build a Complete, Production-Grade RAG System and Evaluate It Rigorously

Project Requirements

Module Exercises

Chunking Strategies

Search Strategy Selection

Reranking

Unstructured Data

RAG Evaluation