Start with a workflow, not a model
The most successful AI features we see are narrow. They take one step in an existing workflow that is slow, repetitive and text-heavy — summarizing a case history, drafting a reply, extracting fields from a document, answering questions about internal policies — and make it faster while keeping a person in the loop.
Starting from the model (“we should use AI”) tends to produce chat windows nobody uses. Starting from the workflow (“support agents spend twenty minutes reading ticket history before replying”) produces features people adopt because they remove real effort.
What makes a good first AI use case
- The task is mostly reading, writing, classifying or summarizing text
- A person currently does it and can judge whether the output is right
- Mistakes are recoverable — the output is a draft or suggestion, not an irreversible action
- The data needed is already available in your systems
- The volume is high enough that saved minutes add up
The architecture: AI as a service beside your app
You rarely need to rewrite anything. The common pattern is a small AI service that your existing application calls through an internal API. That service assembles context from your data, calls a hosted model, validates the response and returns structured output your application can display or act on.
Keeping AI logic in its own service means you can change models, prompts and retrieval strategies without touching the core product, and you can apply consistent logging, rate limits and access control in one place.
Retrieval-augmented generation (RAG)
Models don’t know your data. Retrieval-augmented generation solves this by searching your documents or records for relevant passages — usually with vector search — and including them in the prompt. The model answers from that context, and you can show users the sources it used. For most business use cases, RAG is more practical and cheaper to maintain than fine-tuning a model.
Structured output
When the result feeds into your application rather than straight to a person, ask the model for structured output such as JSON that matches a schema, and validate it before use. This turns free-form text generation into a dependable component.
A pragmatic rollout plan
- 01
Prototype with real data1–3 weeks
Test the use case against a representative sample of real inputs to see whether quality is good enough to be useful.
- 02
Build an evaluation set1–2 weeks
Collect example inputs with known good outputs so you can measure quality objectively whenever prompts or models change.
- 03
Integrate behind a feature flag2–6 weeks
Add the AI service and UI to the product for a small group of internal users first.
- 04
Measure and iterateOngoing
Track acceptance rate of suggestions, edits made, time saved and cost per request; improve prompts and retrieval.
- 05
Expand gradually
Roll out to more users and adjacent workflows once the first one proves its value.
Risks and how to control them
| Risk | Control |
|---|---|
| Incorrect or invented answers | Ground responses in retrieved sources, show citations, keep a human review step |
| Sensitive data exposure | Filter what is sent to the model, apply user permissions to retrieval, review provider data-use terms |
| Prompt injection from user content | Treat retrieved and user content as untrusted, limit tool permissions, validate outputs |
| Unpredictable cost | Cache results, use smaller models for simpler steps, set per-user limits and monitor spend |
| Quality drift after changes | Re-run the evaluation set before every prompt or model change |
Managing cost
Model usage is billed per token, so cost scales with how much context you send and how often. Retrieval that selects only the most relevant passages, caching repeated requests, and routing simple tasks to smaller, cheaper models keep costs predictable. Track cost per completed task, not just total spend, so you can compare it with the time saved.