Compare Architectures
Side-by-side: use case, requirements, latency, cost, strengths, weaknesses, failure modes, example — and when to choose each.
Workflow vs AgentSingle agent vs Multi-agentRAG vs Fine-tuningVector search vs BM25Function calling vs MCPDirect API integration vs MCPPlan-then-execute vs ReActShort-term memory vs Long-term memoryLLM-as-judge vs Deterministic evaluatorsHuman-in-the-loop vs Human-on-the-loopSupervisor vs PipelinePrompt caching vs Semantic cachingRouter vs SupervisorFramework vs No framework
| Prompt caching | Semantic caching | |
|---|---|---|
| Use case | Large repeated prefix: system prompt, tool schemas, documents | Repeated or near-duplicate user questions |
| Requirements | Stable prefix ordering; provider cache support | Embedding of queries, similarity threshold, response store, invalidation |
| Latency | Faster prefill on cache hits | Skips the model entirely on hits |
| Cost | Cached input tokens billed at a fraction (often ~10%) | Zero model cost on hits; embedding cost on every request |
| Strengths | Transparent, exact, no correctness risk | Large savings on FAQ-like traffic |
| Weaknesses | Only helps the prefix; any change invalidates | Threshold too loose returns wrong answers to similar questions |
| Failure modes | Dynamic content placed before static content kills hit rate | Serving a stale or personalized answer to the wrong user |
| Example | 20k-token system prompt reused across every call | "What is your refund policy?" asked 5,000 times a day |
| Choose this when | Choose prompt caching always when you have a long stable prefix; order static content first. | Choose semantic caching for high-volume, non-personalized questions, with a strict threshold and invalidation. |