Building with LLMs: A Pragmatic Approach

There's no shortage of hype around LLMs. Here's a pragmatic framework for actually shipping products with them.

Start with the user experience, not the model. What does the user need? A summary? A classification? A generated draft? Pick the simplest AI capability that solves the problem — don't default to a chatbot.

Prompt engineering is real engineering. Treat your prompts like code: version them, test them, and review changes. A prompt change can improve or destroy output quality overnight.

Build evaluation before you build features. Create 20–50 test cases from real user scenarios. Run them automatically on every prompt or model change. This is your safety net.

Latency and cost are features. A 10-second response feels broken even if the answer is perfect. Use streaming, cache common responses, and choose smaller models for simpler tasks.

Ship incrementally. Launch with a narrow use case, gather feedback, and expand. The teams that try to build 'ChatGPT for X' in v1 usually never ship.