This week’s system design refresher:
How OpenAI Built its Data Agent (Youtube video)
Top 7 Ways to Reduce LLM Costs
8 Frontier Open Models I’m Most Excited About
Why Most AI Demos Fail in Production
Short-Term vs Long-Term Memory in AI Agents: What’s the difference
How OpenAI Built Its Data Agent
Top 7 Ways to Reduce LLM Costs
Prompt compression: Reduce input token count by removing redundant instructions, compressing few-shot examples, and trimming RAG chunks before sending to the model.
Semantic caching: Return stored responses for semantically similar queries.
Prompt caching: Use native prompt/KV caching to skip reprocessing repeated static context.
Model routing: Automatically direct simple tasks to small distilled models and complex reasoning to frontier models - the cost difference can be 10–50x per call.
RAG context trimming: Retrieve only the most relevant chunks, rerank by similarity, and inject only top-K into context - directly reducing input token cost on every call.
Batch & async processing: Group non-urgent requests into large batches - fully loading the GPU on each batch maximizes hardware utilization. Providers pass ~50% cost discount on batch API calls.
Output constraints: Instruct the model explicitly to respond concisely or in structured format (JSON, labels) - shorter outputs mean fewer output tokens billed.
8 Frontier Open Models I’m Most Excited About
Inkling (Thinking Machines): released this week, now the strongest American open model. Text, image, and audio input.
Nemotron 3 Ultra (NVIDIA): a solid choice for long-running agents. The Mamba hybrid keeps long-context inference cheap.
GLM-5.2 (Z.ai): currently the best open model for coding.
Kimi K2.6 (Moonshot): strong on long agent tasks. Holds up over hundreds of tool calls.
DeepSeek-V4 Pro: a very cheap way to get frontier-level quality over an API.
Qwen3.6-35B (Alibaba): the best model to run on your own machine. A single 24 GB GPU is enough.
Gemma 4 31B (Google): the best choice for on-device multimodal. Takes image and audio input on a gaming GPU.
MiniMax M3: the only open model with native video input.
Over to you: Which open model are you most excited about?
Why Most AI Demos Fail in Production
Here are some of the most common reasons:
Clean demo data vs. messy production reality: Demos use clean datasets but production data arrives from multiple streams with inconsistent formats and missing fields. Validate schemas at ingestion and run quality checks before any input reaches the model.
Retrieval failures cause hallucinations: Wrong chunks or missing context cause failures that are hard to trace. Measure retrieval quality separately and verify answers are grounded in what was retrieved.
Outdated evals create false confidence: Test cases from day one do not reflect how real users ask questions months later. Sample live traffic weekly and use that as the benchmark.
Standard monitoring misses AI failures: A service can be operational while the model silently returns wrong answers. Log every prompt, tool call, and response and run quality checks on live traffic.
Token costs compound at scale: What costs pennies in a demo compounds fast when agents retry and chain calls. Set hard token limits, enable prompt caching, and route lighter models for routine tasks.
Data drift causes regressions: User behavior and language patterns shift causing model to drift. Quality drops without warning. Monitor input distributions and wire thresholds to trigger retraining or rollback.
Prompt coupling causes deployment risks: When prompts and application logic share the same codebase, every change is a deployment risk. Store prompts in version-controlled config and add quality gates to the pipeline.
Cascading failures break pipelines: One bad output silently feeds the next step in a multi-step pipeline. Add validation checks between steps and design each stage to reject bad inputs.
Over to you: Which of these has your team run into?
Short-Term vs Long-Term Memory in AI Agents: What’s the difference
An AI agent handles a current conversation well but upon the next session, it starts from scratch with no context, no history. This is a memory problem, and it comes down to two types.
Short-Term Memory (Context Window)
It is the working memory the model uses during a single call. The system prompt, conversation history, tool outputs, and retrieved content are placed into the context window so the model can reason over them and generate a response.
But once the call ends, that window resets. As the conversation grows longer, the model may also lose track of earlier details inside the same window.
Long-Term Memory (External Storage)
LTM lives outside the model and persists across sessions. In practice, it often shows up as episodic memory for past interactions, semantic memory for facts and preferences, and procedural memory for rules or workflows.
When a new call starts, the agent retrieves relevant memory records from the LTM store and injects them into the context window through RAG so the model can reason over them. After a session ends, key information can be extracted and written back to LTM storage for future use.
In production, LTM does not replace the context window. It feeds it. Even a 1M token window resets after every call. For isolated single-call tasks STM is enough. Agents that need to remember users, past decisions, and outcomes across sessions need both.
Over to you: How are you handling memory in your agents today?