> ## Documentation Index
> Fetch the complete documentation index at: https://aitutorial.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Segmentation with U-Net

> Pixel-level masks, encoder-decoder architectures, and training a U-Net end to end

export const QuizQuestion = ({question, options, answer, explanation}) => {
  const [selected, setSelected] = useState(null);
  const [revealed, setRevealed] = useState(false);
  const handleSelect = index => {
    if (revealed) return;
    setSelected(index);
    setRevealed(true);
  };
  const isCorrect = selected === answer;
  const getOptionClass = i => {
    const classes = ['quiz-option'];
    if (revealed) {
      classes.push('quiz-option-disabled');
      if (i === answer) classes.push('quiz-option-correct'); else if (i === selected && !isCorrect) classes.push('quiz-option-wrong');
    }
    return classes.join(' ');
  };
  return <div className="quiz-card">
      <p className="quiz-question">{question}</p>
      <div className="quiz-options">
        {options.map((option, i) => <button key={i} onClick={() => handleSelect(i)} className={getOptionClass(i)}>
            <span className="quiz-letter">{String.fromCharCode(65 + i)}</span>
            {option}
          </button>)}
      </div>
      {revealed && <div className={`quiz-feedback ${isCorrect ? 'quiz-feedback-correct' : 'quiz-feedback-wrong'}`}>
          <strong>{isCorrect ? 'Correct!' : 'Incorrect.'}</strong> {explanation}
        </div>}
    </div>;
};

export const Quiz = ({title = "Check Your Understanding", children}) => {
  return <div style={{
    marginTop: '24px'
  }}>
      <div className="quiz-title">{title}</div>
      {children}
    </div>;
};

export const CodeEditor = ({file = 'src/hello_world.ts', lines, title = 'Code Example', repo = 'ai-tutorial/typescript-examples', height = '650px', functionName, theme: userTheme}) => {
  const STORAGE_KEY = 'openai_api_key';
  const GEMINI_STORAGE_KEY = 'gemini_api_key';
  const ANTHROPIC_STORAGE_KEY = 'anthropic_api_key';
  const PROVIDER_STORAGE_KEY = 'llm_playground_provider';
  if (!functionName) {
    console.warn('CodeEditor: functionName parameter is required');
  }
  const hasCreatedEnvRef = useRef(false);
  const vmRef = useRef(null);
  const [isMaximized, setIsMaximized] = useState(false);
  const [isCollapsed, setIsCollapsed] = useState(false);
  const [isStuck, setIsStuck] = useState(false);
  const [iframeKey, setIframeKey] = useState(0);
  const [showApiKeyDialog, setShowApiKeyDialog] = useState(false);
  const [apiKey, setApiKey] = useState('');
  const [error, setError] = useState('');
  const [success, setSuccess] = useState(false);
  const [isSubmitting, setIsSubmitting] = useState(false);
  const [isValidating, setIsValidating] = useState(false);
  const [detectedTheme, setDetectedTheme] = useState('dark');
  useEffect(() => {
    if (typeof window === 'undefined') return;
    const checkTheme = () => {
      const isDark = document.documentElement.classList.contains('dark');
      setDetectedTheme(isDark ? 'dark' : 'light');
    };
    checkTheme();
    const observer = new MutationObserver(checkTheme);
    observer.observe(document.documentElement, {
      attributes: true,
      attributeFilter: ['class']
    });
    return () => observer.disconnect();
  }, []);
  const theme = userTheme || detectedTheme;
  const [selectedProvider, setSelectedProvider] = useState(() => {
    if (typeof window === 'undefined') return 'gemini';
    return localStorage.getItem(PROVIDER_STORAGE_KEY) || 'gemini';
  });
  const isApiKeyConfigured = () => {
    const openaiKey = localStorage.getItem(STORAGE_KEY);
    const geminiKey = localStorage.getItem(GEMINI_STORAGE_KEY);
    const anthropicKey = localStorage.getItem(ANTHROPIC_STORAGE_KEY);
    return openaiKey !== null && openaiKey.trim().length > 0 || geminiKey !== null && geminiKey.trim().length > 0 || anthropicKey !== null && anthropicKey.trim().length > 0;
  };
  const dispatchApiKeyChanged = () => {
    if (typeof window !== 'undefined' && window.dispatchEvent) {
      window.dispatchEvent(new CustomEvent('apiKeyChanged', {
        detail: {
          configured: isApiKeyConfigured()
        }
      }));
    }
  };
  const saveApiKey = apiKey => {
    if (apiKey && apiKey.trim()) {
      const trimmedKey = apiKey.trim();
      localStorage.setItem(STORAGE_KEY, trimmedKey);
      dispatchApiKeyChanged();
      return true;
    }
    return false;
  };
  const buildEnvContent = () => {
    const openaiKey = localStorage.getItem(STORAGE_KEY)?.trim();
    const geminiKey = localStorage.getItem(GEMINI_STORAGE_KEY)?.trim();
    const anthropicKey = localStorage.getItem(ANTHROPIC_STORAGE_KEY)?.trim();
    if (!openaiKey && !geminiKey && !anthropicKey) {
      return `OPENAI_MODEL=gpt-4.1-nano
OPENAI_API_KEY=sk-mock-key-1234567890abcdef
GEMINI_MODEL=gemini-2.5-flash-lite
GOOGLE_GENERATIVE_AI_API_KEY=
GOOGLE_API_KEY=
ANTHROPIC_API_KEY=
AI_PROVIDER=openai
# API key not found in browser storage
# To configure your API key:
# 1. For Gemini (free): Go to https://aistudio.google.com/apikey
# 2. For OpenAI: Go to https://platform.openai.com/api-keys
# 3. For Claude: Go to https://console.anthropic.com/settings/keys
# 4. Enter it in the configuration form above this editor
# 5. The .env file will be automatically updated with your key`;
    }
    const envLines = ['# Using the API key(s) you configured. This file will be created when the dialog is loaded.'];
    if (openaiKey) {
      envLines.push(`OPENAI_MODEL=gpt-4.1-nano`);
      envLines.push(`OPENAI_API_KEY=${openaiKey}`);
    }
    if (geminiKey) {
      envLines.push(`GEMINI_MODEL=gemini-2.5-flash-lite`);
      envLines.push(`# Vercel AI SDK uses GOOGLE_GENERATIVE_AI_API_KEY, LangChain uses GOOGLE_API_KEY`);
      envLines.push(`GOOGLE_GENERATIVE_AI_API_KEY=${geminiKey}`);
      envLines.push(`GOOGLE_API_KEY=${geminiKey}`);
    }
    if (anthropicKey) {
      envLines.push(`ANTHROPIC_API_KEY=${anthropicKey}`);
    }
    const provider = anthropicKey ? 'anthropic' : geminiKey ? 'gemini' : 'openai';
    envLines.push(`AI_PROVIDER=${provider}`);
    return envLines.join('\n');
  };
  const updateEnvFile = async vm => {
    if (!vm) return;
    try {
      await vm.applyFsDiff({
        create: {
          'env/.env': buildEnvContent(),
          'env/run.conf': `file=${file}`
        },
        destroy: []
      });
      hasCreatedEnvRef.current = true;
    } catch (error) {
      console.error('Failed to write env files:', error);
      hasCreatedEnvRef.current = false;
    }
  };
  useEffect(() => {
    if (!isApiKeyConfigured()) {
      setShowApiKeyDialog(true);
    }
    const handleApiKeyChanged = () => {
      if (isApiKeyConfigured()) {
        setShowApiKeyDialog(false);
      }
    };
    if (typeof window !== 'undefined') {
      window.addEventListener('apiKeyChanged', handleApiKeyChanged);
      return () => {
        window.removeEventListener('apiKeyChanged', handleApiKeyChanged);
      };
    }
  }, []);
  const validateApiKey = async (key, provider) => {
    try {
      const urls = {
        gemini: 'https://generativelanguage.googleapis.com/v1beta/models?key=' + encodeURIComponent(key.trim()),
        openai: 'https://api.openai.com/v1/models',
        anthropic: 'https://api.anthropic.com/v1/models'
      };
      const headerMap = {
        gemini: {
          'Content-Type': 'application/json'
        },
        openai: {
          'Authorization': `Bearer ${key.trim()}`,
          'Content-Type': 'application/json'
        },
        anthropic: {
          'x-api-key': key.trim(),
          'anthropic-version': '2023-06-01',
          'Content-Type': 'application/json'
        }
      };
      const url = urls[provider];
      const headers = headerMap[provider];
      const response = await fetch(url, {
        method: 'GET',
        headers
      });
      if (response.ok) {
        return {
          valid: true
        };
      } else if (response.status === 401 || response.status === 403) {
        return {
          valid: false,
          error: 'Invalid API key. Please check your key and try again.'
        };
      } else if (response.status === 429) {
        return {
          valid: false,
          error: 'Rate limit exceeded. Please try again later.'
        };
      } else {
        const errorData = await response.json().catch(() => ({}));
        return {
          valid: false,
          error: errorData.error?.message || `API request failed with status ${response.status}`
        };
      }
    } catch (err) {
      if (err.name === 'TypeError' && err.message.includes('fetch')) {
        return {
          valid: false,
          error: 'Network error. Please check your connection and try again.'
        };
      }
      return {
        valid: false,
        error: err.message || 'Failed to validate API key. Please try again.'
      };
    }
  };
  const handleSkipConfiguration = () => {
    const skipKey = 'sk-<configure-your-key>';
    saveApiKey(skipKey);
    setShowApiKeyDialog(false);
  };
  const handleApiKeySubmit = async e => {
    e.preventDefault();
    setError('');
    setSuccess(false);
    setIsSubmitting(true);
    const providerNames = {
      gemini: 'Gemini',
      openai: 'OpenAI',
      anthropic: 'Claude'
    };
    if (!apiKey || !apiKey.trim()) {
      setError(`Please enter your ${providerNames[selectedProvider]} API key`);
      setIsSubmitting(false);
      return;
    }
    const trimmedKey = apiKey.trim();
    if (selectedProvider === 'openai' && !trimmedKey.startsWith('sk-')) {
      setError('Invalid API key format. OpenAI API keys should start with "sk-"');
      setIsSubmitting(false);
      return;
    }
    if (selectedProvider === 'anthropic' && !trimmedKey.startsWith('sk-ant-')) {
      setError('Invalid API key format. Anthropic API keys should start with "sk-ant-"');
      setIsSubmitting(false);
      return;
    }
    setIsValidating(true);
    setError('');
    const validation = await validateApiKey(trimmedKey, selectedProvider);
    setIsValidating(false);
    if (!validation.valid) {
      setError(validation.error || 'Invalid API key. Please check your key and try again.');
      setIsSubmitting(false);
      return;
    }
    try {
      const storageKeys = {
        gemini: GEMINI_STORAGE_KEY,
        openai: STORAGE_KEY,
        anthropic: ANTHROPIC_STORAGE_KEY
      };
      localStorage.setItem(storageKeys[selectedProvider], trimmedKey);
      localStorage.setItem(PROVIDER_STORAGE_KEY, selectedProvider);
      dispatchApiKeyChanged();
      setSuccess(true);
      setApiKey('');
      setTimeout(() => {
        window.location.reload();
      }, 1000);
    } catch (err) {
      setError(err.message || 'Failed to save API key. Please try again.');
      setIsSubmitting(false);
    }
  };
  const baseFilePath = file || 'src/hello_world.ts';
  let filePath = baseFilePath;
  if (typeof lines === 'string' && lines.trim()) {
    const lineParts = lines.split('-');
    if (lineParts.length === 2) {
      filePath = `${filePath}:L${lineParts[0].trim()}-L${lineParts[1].trim()}`;
    } else {
      filePath = `${filePath}:L${lineParts[0].trim()}`;
    }
  } else if (typeof lines === 'object' && lines.start !== undefined) {
    filePath = lines.end !== undefined ? `${filePath}:L${lines.start}-L${lines.end}` : `${filePath}:L${lines.start}`;
  }
  const stackblitzUrl = `https://stackblitz.com/github/${repo}?file=${encodeURIComponent(filePath)}&embed=1&view=editor&theme=${theme}`;
  const loadSDK = () => {
    return new Promise((resolve, reject) => {
      if (window.StackBlitzSDK || window.stackblitzSDK) {
        resolve(window.StackBlitzSDK || window.stackblitzSDK);
        return;
      }
      if (document.querySelector('script[data-stackblitz-sdk]')) {
        const checkInterval = setInterval(() => {
          if (window.StackBlitzSDK || window.stackblitzSDK) {
            clearInterval(checkInterval);
            resolve(window.StackBlitzSDK || window.stackblitzSDK);
          }
        }, 100);
        setTimeout(() => {
          clearInterval(checkInterval);
          reject(new Error('SDK loading timeout'));
        }, 10000);
        return;
      }
      const script = document.createElement('script');
      script.src = 'https://unpkg.com/@stackblitz/sdk/bundles/sdk.umd.js';
      script.async = true;
      script.setAttribute('data-stackblitz-sdk', 'true');
      script.onload = () => {
        const sdk = window.StackBlitzSDK || window.stackblitzSDK;
        if (sdk) {
          resolve(sdk);
        } else {
          reject(new Error('SDK loaded but not available on window'));
        }
      };
      script.onerror = () => {
        reject(new Error('Failed to load StackBlitz SDK'));
      };
      document.head.appendChild(script);
    });
  };
  const LOAD_TIMEOUT_MS = 10000;
  const iframeElRef = useRef(null);
  const reloadCountRef = useRef(0);
  const handleRetry = () => {
    vmRef.current = null;
    hasCreatedEnvRef.current = false;
    reloadCountRef.current = 0;
    setIsStuck(false);
    setIframeKey(prev => prev + 1);
  };
  const iframeRef = iframe => {
    iframeElRef.current = iframe;
  };
  const connectToVM = async iframe => {
    const sdk = await loadSDK();
    return sdk.connect(iframe);
  };
  const handleIframeLoad = async () => {
    const iframe = iframeElRef.current;
    if (!iframe) return;
    if (reloadCountRef.current > 0) {
      try {
        const vm = await connectToVM(iframe);
        vmRef.current = vm;
        await updateEnvFile(vm);
      } catch (_) {}
      return;
    }
    try {
      if (vmRef.current) return;
      const vm = await Promise.race([connectToVM(iframe), new Promise((_, reject) => setTimeout(() => reject(new Error('connect timeout')), LOAD_TIMEOUT_MS))]);
      vmRef.current = vm;
      await updateEnvFile(vm);
    } catch (error) {
      console.error('Failed to connect to StackBlitz VM:', error);
      if (typeof window !== 'undefined' && window.gtag) {
        window.gtag('event', 'load_refresh_error', {
          event_category: 'stackblitz',
          event_label: file,
          error_message: error.message
        });
      }
      reloadCountRef.current = 1;
      setTimeout(() => {
        setIframeKey(prev => prev + 1);
      }, 2000);
    }
  };
  const isSafari = typeof navigator !== 'undefined' && (/^((?!chrome|android).)*safari/i).test(navigator.userAgent);
  if (isSafari) {
    return <div className="code-editor-dialog-container" style={{
      height: height
    }}>
        <div className="code-editor-dialog-box">
          <h2 className="code-editor-dialog-title">
            <svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#f59e0b" strokeWidth="2">
              <path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"></path>
              <line x1="12" y1="9" x2="12" y2="13"></line>
              <line x1="12" y1="17" x2="12.01" y2="17"></line>
            </svg>
            Browser Not Supported
          </h2>
          <p className="code-editor-dialog-description">
            The interactive code editor is not supported on Safari. Please use <strong>Chrome</strong>, <strong>Edge</strong>, or <strong>Firefox</strong> to run the examples.
          </p>
        </div>
      </div>;
  }
  if (showApiKeyDialog) {
    return <div className="code-editor-dialog-container" style={{
      height: height
    }}>
        <div className="code-editor-dialog-box">
          <h2 className="code-editor-dialog-title">
            <svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#f59e0b" strokeWidth="2">
              <path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"></path>
              <line x1="12" y1="9" x2="12" y2="13"></line>
              <line x1="12" y1="17" x2="12.01" y2="17"></line>
            </svg>
            Configure API Key
          </h2>

          <p className="code-editor-dialog-description">
            All interactive examples execute entirely within your browser environment, ensuring complete security and privacy.
            Your API key is stored locally in your browser's storage and is never transmitted to external servers.
          </p>

          <div className="llm-provider-tabs" style={{
      marginBottom: '16px'
    }}>
            <button type="button" onClick={() => {
      setSelectedProvider('gemini');
      setError('');
      setApiKey('');
    }} className={`llm-provider-tab ${selectedProvider === 'gemini' ? 'llm-provider-tab-active' : ''}`}>
              Gemini <span className="llm-provider-tab-badge">Free</span>
            </button>
            <button type="button" onClick={() => {
      setSelectedProvider('openai');
      setError('');
      setApiKey('');
    }} className={`llm-provider-tab ${selectedProvider === 'openai' ? 'llm-provider-tab-active' : ''}`}>
              OpenAI
            </button>
            <button type="button" onClick={() => {
      setSelectedProvider('anthropic');
      setError('');
      setApiKey('');
    }} className={`llm-provider-tab ${selectedProvider === 'anthropic' ? 'llm-provider-tab-active' : ''}`}>
              Claude
            </button>
          </div>

          {selectedProvider === 'gemini' && <div className="llm-gemini-recommendation" style={{
      marginBottom: '16px'
    }}>
              Gemini offers a generous free tier — great for learning! Get your free API key at{' '}
              <a href="https://aistudio.google.com/apikey" target="_blank" rel="noopener noreferrer" className="code-editor-link">
                aistudio.google.com/apikey
              </a>
            </div>}

          {selectedProvider === 'openai' && <div className="code-editor-info-box">
              <p className="code-editor-info-box-title">
                Don't have an API key?
              </p>
              <p className="code-editor-info-box-text">
                Get one at{' '}
                <a href="https://platform.openai.com/api-keys" target="_blank" rel="noopener noreferrer" className="code-editor-link">
                  platform.openai.com/api-keys
                </a>
              </p>
            </div>}

          {selectedProvider === 'anthropic' && <div className="code-editor-info-box">
              <p className="code-editor-info-box-title">
                Don't have an API key?
              </p>
              <p className="code-editor-info-box-text">
                Get one at{' '}
                <a href="https://console.anthropic.com/settings/keys" target="_blank" rel="noopener noreferrer" className="code-editor-link">
                  console.anthropic.com/settings/keys
                </a>
              </p>
            </div>}

          <form onSubmit={handleApiKeySubmit}>
            <div className="code-editor-form-group">
              <label htmlFor="api-key-input" className="code-editor-label">
                {selectedProvider === 'gemini' ? 'Gemini' : 'OpenAI'} API Key
              </label>
              <input id="api-key-input" type="password" value={apiKey} onChange={e => {
      setApiKey(e.target.value);
      setError('');
      setSuccess(false);
    }} placeholder={selectedProvider === 'openai' ? 'sk-...' : 'Gemini API Key'} disabled={isSubmitting} className={`code-editor-input ${error ? 'code-editor-input-error' : ''}`} />
            </div>

            {isValidating && <div className="code-editor-message code-editor-message-info">
                <span className="code-editor-message-icon">⏳</span>
                <span>Validating API key...</span>
              </div>}

            {error && !isValidating && <div className="code-editor-message code-editor-message-error">
                <span className="code-editor-message-icon">⚠️</span>
                <span>{error}</span>
              </div>}

            {success && <div className="code-editor-message code-editor-message-success">
                <span className="code-editor-message-icon">✓</span>
                <span>API key saved successfully! Loading editor...</span>
              </div>}

            <button type="submit" disabled={isSubmitting || isValidating || !apiKey.trim()} className="code-editor-button">
              {isValidating ? 'Validating...' : isSubmitting ? 'Saving...' : 'Save API Key'}
            </button>
          </form>

          <button type="button" onClick={handleSkipConfiguration} disabled={isSubmitting || isValidating} className="code-editor-button-secondary">
            Skip Configuration
          </button>

          <div className="code-editor-footer">
            <p className="code-editor-footer-text">
              Alternatively, you may checkout the source code from{' '}
              <a href="https://github.com/ai-tutorial/typescript-examples" target="_blank" rel="noopener noreferrer" className="code-editor-link code-editor-link-break">
                https://github.com/ai-tutorial/typescript-examples
              </a>
              {' '}and run the examples locally.
            </p>
          </div>
        </div>
      </div>;
  }
  const toggleMaximize = () => setIsMaximized(!isMaximized);
  const toggleCollapse = () => setIsCollapsed(!isCollapsed);
  return <div className={`code-editor-wrapper ${isMaximized ? 'maximized' : ''} ${isCollapsed ? 'collapsed' : ''}`} data-theme={theme}>
      <div className="code-editor-header">
        <div className="code-editor-title">
          <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
            <path d="M11 4H4a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h14a2 2 0 0 0 2-2v-7"></path>
            <path d="M18.5 2.5a2.121 2.121 0 0 1 3 3L12 15l-4 1 1-4 9.5-9.5z"></path>
          </svg>
          {title}
        </div>
        <div className="code-editor-controls">
          {!isMaximized && <button className="code-editor-collapse-button" onClick={toggleCollapse} title={isCollapsed ? "Expand" : "Collapse"} type="button">
              <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
                {isCollapsed ? <polyline points="6 9 12 15 18 9" /> : <polyline points="6 15 12 9 18 15" />}
              </svg>
            </button>}
          <button className="code-editor-maximize-button" onClick={toggleMaximize} title={isMaximized ? "Minimize" : "Maximize (Focus Mode)"} type="button">
            <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
              {isMaximized ? <><path d="M4 14h6v6" /><path d="M20 10h-6V4" /><path d="M14 10l7-7" /><path d="M3 21l7-7" /></> : <><path d="M15 3h6v6" /><path d="M9 21H3v-6" /><path d="M21 3l-7 7" /><path d="M3 21l7-7" /></>}
            </svg>
          </button>
        </div>
      </div>

      {!isCollapsed && <div style={{
    position: 'relative',
    height: isMaximized ? 'auto' : height,
    flex: isMaximized ? 1 : 'none'
  }}>
          <iframe key={iframeKey} ref={iframeRef} onLoad={handleIframeLoad} src={stackblitzUrl} className="code-editor-iframe" style={{
    height: '100%',
    flex: isMaximized ? 1 : 'none'
  }} title={title || 'Code Example'} allow="accelerometer; camera; encrypted-media; geolocation; gyroscope; hid; microphone; midi; payment; usb; xr-spatial-tracking" sandbox="allow-forms allow-modals allow-popups allow-presentation allow-same-origin allow-scripts" />

          {isStuck && <div className="code-editor-stuck-overlay">
              <div className="code-editor-stuck-box">
                <svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#f59e0b" strokeWidth="2">
                  <path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"></path>
                  <line x1="12" y1="9" x2="12" y2="13"></line>
                  <line x1="12" y1="17" x2="12.01" y2="17"></line>
                </svg>
                <p>StackBlitz is taking too long to load. This can happen when the repository was recently updated.</p>
                <button type="button" className="code-editor-button" onClick={handleRetry} style={{
    marginTop: '8px'
  }}>
                  Retry
                </button>
              </div>
            </div>}
        </div>}
    </div>;
};

Classification gives one label per image. Detection draws a box around each object. Segmentation goes one level deeper and labels **every pixel** — it's what powers background removal, medical organ delineation, autonomous-driving road masks, and every "blur the background" feature on your phone. The canonical architecture for this job is the **U-Net**, and this page is built around it.

## Three Flavors of Segmentation

Worth keeping straight because papers and APIs all assume you know the difference.

| Type                      | What it labels                                                                                | Example task                                    |
| ------------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| **Semantic segmentation** | Each pixel gets a class label — no instance distinction                                       | "this pixel is road / sidewalk / building"      |
| **Instance segmentation** | Each pixel gets a class label **plus** an instance ID                                         | "this pixel is car #3"                          |
| **Panoptic segmentation** | Semantic labels for `stuff` (sky, road), instance labels for `things` (car, person) — unified | "this pixel is road", "this pixel is person #2" |

Output shape is always a mask of the same spatial size as the input: `(H, W)` of class IDs for semantic, or `(N, H, W)` of binary masks for instance-level.

## Why Classification Networks Don't Work Here

A classifier maps `(C, H, W)` → a vector of class scores. A segmenter has to produce `(K, H, W)` — one score per class *per pixel*. Three things break when you try to reuse a standard CNN:

1. **Spatial resolution collapses.** A classification backbone aggressively downsamples with pooling and strides to produce a compact feature vector. If you simply upsample that final feature map back to the input size, you get blurry, low-frequency output — no fine detail.
2. **The fully-connected head expects a flattened vector**, not a grid. You need to remove it and replace it with something spatial.
3. **Every pixel contributes to the loss.** A single "is this a cat?" gradient doesn't carry enough signal to teach the network where the cat's ears end.

So the architecture needs two things: a way to **recover spatial resolution** on the output side, and a way to **preserve the spatial detail** that the encoder normally throws away.

## U-Net: The Encoder-Decoder That Defined Segmentation

U-Net (Ronneberger et al., 2015) solves both problems with a symmetric encoder-decoder plus **skip connections** from encoder to decoder at every resolution. It was originally designed for biomedical microscopy — where it is still the workhorse — and has since spread to essentially every field that does segmentation (and, famously, it is the denoiser sitting inside every latent diffusion model).

```text theme={null}
        Encoder (contracting)                  Decoder (expanding)
        ─────────────────────                  ────────────────────

  Input (C, 256, 256)                          Output (K, 256, 256)
          │                                             ▲
          v                                             │
    ┌─ Conv block ─┐─────── skip ──────────────▶ Conv block (1/1)
    │              │                                    ▲
    │   pool ↓                             upsample ↑   │
    │              │                                    │
    └─ Conv block ─┘─────── skip ──────────▶ Conv block (1/2)
                   │                          ▲
       pool ↓                        upsample ↑
                   │                          │
         Conv block ─── skip ─────▶ Conv block (1/4)
                   │                ▲
       pool ↓              upsample ↑
                   │                │
              Bottleneck (deep features, low resolution)
```

**What each part does:**

* **Encoder (left side)** — stack of conv + pool blocks, same shape as a classification CNN. Each step halves the spatial resolution and roughly doubles the channel count. Learns progressively abstract features.
* **Bottleneck** — the lowest-resolution, highest-semantic feature map. *What* is in the image is encoded here; *where* is mostly lost.
* **Decoder (right side)** — a mirror of the encoder. At each stage it upsamples (via transposed convolution or bilinear-plus-conv) and applies a couple of conv layers.
* **Skip connections** — at each resolution, the encoder's feature map is **concatenated** into the corresponding decoder stage *before* its conv layers run. This is the whole trick: the decoder receives both **semantic context** from the bottleneck and **precise spatial detail** from the skips.

### Output and Loss

* **Output** shape is `(K, H, W)` — one score per class per pixel. Take `argmax` along the class axis to get the predicted mask.
* **Loss** is per-pixel cross-entropy, averaged over every pixel — exactly like classification, just `H × W` times per image. For imbalanced datasets (tiny foreground, huge background — common in medical and in defect detection) **Dice loss** or a Dice + cross-entropy hybrid is the default, because plain cross-entropy is dominated by the easy background pixels.

### Training a U-Net

The training loop is identical to a classifier's — forward pass, loss, backward pass, parameter update — just with a spatial loss. The worked example trains a small U-Net on a toy segmentation dataset so you can see the pipeline end to end.

<CodeEditor file="src/computer-vision/unet_training.ts" functionName="main" lines="1-120" title="Training a U-Net on a small segmentation dataset (placeholder)" />

On a modest dataset (a few thousand images), a 4- or 5-stage U-Net reaches usable accuracy in minutes on a single GPU. That efficiency — strong results from a small network and little data — is exactly why it still dominates applied work.

## Beyond Vanilla U-Net

U-Net is the starting point; the family has grown and branched:

* **nnU-Net (2021)** — a self-configuring U-Net pipeline that inspects the dataset and picks spacing, patch size, normalization, and architecture hyperparameters automatically. Wins medical segmentation benchmarks year after year with almost no tuning.
* **U-Net++, Attention U-Net** — refinements to the skip connections (nested or attention-gated) that squeeze a few points of Dice out of hard datasets.
* **Swin UNETR, UNETR** — replace the convolutional encoder with a vision transformer; stronger on large or complex volumes at the cost of more compute.
* **DeepLab v3+ (2018)** — atrous (dilated) convolutions for large receptive fields without downsampling; strong on outdoor scenes.
* **Mask R-CNN (2017)** — adds a per-ROI mask head to a detector; the standard move when you need **instance** segmentation with class labels.
* **Mask2Former, OneFormer (2022)** — transformer decoders with mask queries that unify semantic / instance / panoptic in a single model.

For most applied work the decision tree is small: if you need semantic masks, start with a U-Net; if you need instance IDs, use Mask R-CNN or Mask2Former; if you're in medical imaging, go straight to nnU-Net.

## Segmentation Foundation Models: A Note on Recent Work

Recent work has reframed segmentation as "prompting for pixels". **Segment Anything (SAM, Meta 2023)** is a large pretrained encoder plus a prompt-aware mask decoder: click a point or draw a box, get a zero-shot mask with no fine-tuning. **SAM 2 (2024)** extends the idea to video with memory attention. **Grounded-SAM** chains an open-vocabulary detector (Grounding DINO) into SAM so you can "segment everything labeled 'hard hat'" from a text prompt.

These models don't replace U-Net — they complement it. SAM shines for interactive tools and for segmenting objects a custom model was never trained on; U-Net (and descendants) still dominate when you have a labeled dataset and a specific target domain. The rule of thumb: **if you have labels, train a U-Net; if you don't, prompt SAM.**

## When To Use What

| Situation                                                                | Recommendation                      |
| ------------------------------------------------------------------------ | ----------------------------------- |
| Specialized domain with labeled data (medical, industrial QA, satellite) | U-Net / nnU-Net / Swin UNETR        |
| Instance-level masks with class labels                                   | Mask R-CNN or Mask2Former           |
| Panoptic scene understanding                                             | Mask2Former / OneFormer             |
| "Segment the thing I clicked" in an interactive tool                     | SAM                                 |
| "Segment everything matching this text" with no fine-tuning              | Grounded-SAM (Grounding DINO → SAM) |
| Video object segmentation / rotoscoping                                  | SAM 2                               |

### Practical Implication

Segmentation's production bottleneck is almost always **labeled data**, not architecture. A well-curated 500-image labeled dataset trained with a small U-Net will usually outperform a fancier model trained on 50 poorly-labeled ones. Spend the time on label quality first; swap architectures second.

#### ❌ Antipattern

Reaching for a transformer-based Swin UNETR on a dataset of 200 images. The model has tens of millions of parameters, your dataset won't regularize it, and training Dice will look great while test Dice collapses.

#### ✅ Best Practice

Start with a small U-Net plus standard augmentation (random flips, small rotations, and — for medical — elastic deformations). Check whether accuracy plateaus because of the model or because of the data. If it's the data, labeling is the highest-leverage next step. If it's the model, *then* scale up.

<Quiz>
  <QuizQuestion question="You're training a U-Net to segment thin vessels on retinal images. Training loss goes down but predictions completely miss the vessels at test time. What's most likely going wrong?" options={["The optimizer is stuck in a local minimum", "Class imbalance — the background dwarfs the vessel, so per-pixel cross-entropy is dominated by easy negatives. Switch to Dice loss or a Dice + cross-entropy hybrid", "The encoder isn't deep enough"]} answer={1} explanation="Thin structures on a large background make cross-entropy a lopsided loss — predicting 'background everywhere' is already ~99% correct per pixel. Dice or Focal loss weight minority-class errors much more heavily and are the standard fix." />

  <QuizQuestion question="You remove the skip connections from a U-Net to simplify it. Training converges but predicted masks look like fuzzy blobs with no sharp edges. Why?" options={["The decoder needs more layers", "Without skip connections the decoder has only the low-resolution bottleneck to work from — it has lost the fine spatial detail the encoder threw away during pooling", "The loss function needs to change"]} answer={1} explanation="Skip connections are U-Net's whole point: they give the decoder access to the high-resolution encoder features that would otherwise be downsampled away. Without them you get semantically correct but spatially blurry masks." />
</Quiz>