<arch.design/>
← Lessons/

AI Infrastructure Design

1 / 8

API vs self-hosted inference

Managed API (OpenAI, Anthropic)

  • —No infrastructure to manage
  • —Pay per token — expensive at scale
  • —Latest models immediately available
  • —Data leaves your infrastructure
  • —Use for: prototyping, low-volume, cutting-edge models

Self-hosted (Ollama, vLLM)

  • —Full data privacy — nothing leaves your network
  • —High fixed cost (GPU servers), low marginal cost
  • —You manage updates, scaling, reliability
  • —Slower to access new models
  • —Use for: production at scale, sensitive data, cost optimization

$0.002

GPT-4o per 1K output tokens

$0.0004

self-hosted llama3 per 1K

5×

cost reduction self-hosted

★

API for speed to market; self-hosted for cost at scale and data privacy — most orgs run both.

← → navigate · S recap