Start Here
This series documents the engineering patterns behind production GenAI systems I've built in enterprise consulting work — technique-first and generalized, the way a conference talk would teach them. The through-line: an LLM feature becomes a platform the moment someone else's workflow depends on it, and the patterns that get you there (tool discovery, grounding, evals, human-in-the-loop) are the actual engineering.
Context: multi-provider LLM platform work (Anthropic, OpenAI/Azure, Google/Vertex, AWS Bedrock), agent tooling over the Model Context Protocol, applied transformer fine-tuning, and the front ends that make it client-ready. All examples are generic and representative; no client or proprietary detail.
Prerequisites: comfort with APIs and Python/TypeScript; no prior LLM platform experience assumed.
Reading Order
1. How Do You Give an AI Agent 75 Tools Without Wrecking Its Judgment?
What: Tool-selection collapse is real: dump 75+ tools into a context window and the agent's judgment degrades. The fix is a tool-RAG discovery surface — list → search → describe → call — plus a sandboxed code-composition mode, with the reliability layer (idempotency, confirm-guards, error envelopes, circuit breakers) that makes agent actions safe to retry.
Why it matters: every serious agent deployment hits this wall; the discovery-surface pattern is the difference between a demo and a platform.
2. How Do You Make an LLM Cite Its Sources — and Prove It?
What: Provider-native citation grounding: model output tied to page-anchored source quotes rendered as click-to-source footnotes, an evidence-first pipeline (gather → prompt → validate → score), knowledge-graph validation via JSON-LD/RDF + W3C SHACL, and the honest "no evidence" state instead of fabricated citations.
Why it matters: trust is the product. A wrong answer with a checkable source beats a right answer with none.
3. How Do You Pick Which Model to Ship?
What: The offline eval harness — frontier vs. baseline models measured on latency, token cost, and output quality with automated hallucination scoring — and how a ship/no-ship decision actually gets structured, including the LLM-as-judge pitfalls.
Why it matters: model selection by vibes is how teams ship regressions; eval infrastructure is what the strongest AI teams hire for.
4. How Do You Lift a Classifier's Hardest Level From 4% to 77%?
What: A hierarchical DeBERTa/RoBERTa classifier over a 4-level, 137-path taxonomy — one fine-tuned model per branch, parents constraining children — and the class-imbalance toolkit (class weights, focal loss, branch oversampling) that lifted deepest-level accuracy from ~4% to 77%+. Plus the MLOps loop: CI-triggered SageMaker fine-tuning, ECS serving, symlink-rollout model artifacts.
Why it matters: the fine-tuning-under-imbalance playbook transfers to any deep-taxonomy problem, and the equation everyone cites () finally gets a plain-English treatment.
5. What Does a Trustworthy AI Analyst UI Actually Look Like?
What: The client-ready React 19/TypeScript front end: SSE streaming chat with inline charts, click-to-source citation markers, constrained tool-calling with stable chart ids (editing beats regenerating), web-search-grounded autofill with human accept/reject review, and export gates with human-in-the-loop approval.
Why it matters: AI UIs fail on trust, not on polish — the streaming state machine and the provenance contract are the hard parts.
Series Status
| Post | Topic | Status |
|---|---|---|
| 1. Agent tool discovery | MCP · tool-RAG · reliability patterns | ✅ Published |
| 2. Grounded citations | Gateway · XAI · SHACL validation | ✅ Published |
| 3. Eval harness | Model selection · hallucination scoring | ✅ Published |
| 4. Hierarchical classifier | Fine-tuning · imbalance · MLOps | ✅ Published |
| 5. Trustworthy AI UI | React 19 · streaming · HITL | ✅ Published |
Related: the applied-ML research side of this portfolio lives in The Memorial Bot: The Complete Story — behavioral cloning and style-constrained RL on a single machine.