ChatGPT style assistant
Users chat with an AI model. Answers stream word by word, conversations are saved, and the service runs on expensive, limited GPUs.
In plain words
You ask a question and the answer starts typing itself out almost at once. The model really does make the answer one word at a time, and the app shows each word as it arrives, so you aren't staring at a spinner for 10 seconds.
The model can only read so much at once. If your chat has gone on for an hour, the app keeps the instructions and your most recent messages, and leaves out the oldest ones, like a friend who remembers the last half hour of a conversation but not the start. Behind all of it are GPUs, which are expensive and limited, so requests wait in a queue and are run together in batches.
Examples
- Streaming uses server sent events: one request, many small chunks back.
- Limits are counted in tokens, not requests: a 5,000 word question costs more than a hello.
- Easy questions can go to a smaller, cheaper model.
- Close the tab and generation stops, so no GPU time is wasted.
1. Requirements
Functional
- Chat with a model; answers stream as they are made.
- Conversations are saved and listed.
- Stop and regenerate.
- Free and paid tiers with different limits.
Non functional
- First token in under a second.
- GPUs are scarce: share them fairly.
- Cost per message under control.
- Private: users can delete their data.
Questions to ask
- How many users and messages per second?
- Do we run the models, or call a model provider?
- File uploads, tools, web search in scope?
- Free and paid tiers with different limits?
2. Estimates
| messages | 100M a day, about 1.2k per second, peaks 3x |
| tokens | about 1k tokens in and 500 out per message |
| GPU time | a few seconds per answer; GPUs are the bottleneck and the bill |
3. API
POST /conversations/{id}/messages { text } stream of tokens (server sent events)
GET /conversations?cursor=... [conversation titles]
GET /conversations/{id} messages
POST /conversations/{id}/stop stops generating
4. Data model
conversations(id, user_id, title, created_at) messages(conversation_id, seq, role, text, tokens) usage(user_id, day, tokens) for limits and billing files: object storage; embeddings in a vector index for retrieval
5. High level design
Follow one request at a time, top to bottom.
Send a message
- BrowserPOST message, keeps the stream open
- API serverauth, token based rate limit by tier
- Context buildersystem prompt + newest messages that fit the budget
- Queueper tier priority
- Model servers (GPU)batch requests, generate token by token
- Browsershows tokens as server sent events arrive
Save
- API serverwrites the user message and the final answer with seq numbers
- Usage tabletokens per user per day
- Title jobnames the conversation in the background
6. Deep dives
Why stream?
A full answer takes seconds. Streaming tokens as they are made shows the first word in under a second. Use server sent events: one request, many small chunks.
The conversation is longer than the context window?
Keep the system prompt and the newest messages that fit the token budget, and drop or summarise older ones. Long term facts go in a separate memory store.
How do you share scarce GPUs?
A queue in front of the model servers, per tier priority, and batching: the server runs many requests together. Rate limit by tokens per user, not just requests.
How do you keep costs down?
Cache answers to identical prompts, route easy questions to a smaller model, and cache the shared system prompt prefix on the GPU.
See the core idea run
Core logic (JavaScript, tested)
// Fit a conversation into the model's context window: keep the system prompt and the newest
// messages, drop the oldest middle ones. Tokens are estimated as about 4 characters each.
const tokens = s => Math.ceil(s.length / 4);
function fitContext(system, messages, budget) {
let used = tokens(system);
const kept = [];
for (let i = messages.length - 1; i >= 0; i--) {
const t = tokens(messages[i].text);
if (used + t > budget) break;
used += t;
kept.unshift(messages[i]);
}
return { kept, dropped: messages.length - kept.length, used };
}
// Stream the answer: the model makes tokens one by one; the client shows each as it arrives.
async function* streamTokens(text) {
for (const word of text.split(" ")) yield word;
}
async function collect(stream) {
const parts = [];
for await (const t of stream) parts.push(t);
return parts.join(" ");
}7. Failures
| Model server crashes mid answer | The client sees the stream end; offer "regenerate". Save partial answers as partial. |
| GPU capacity full | Queue with a visible wait, fall back to a smaller model for free users, or return 429 with retry time. |
| User closes the tab | Stop generating to save GPU time, unless it is a background task. |
8. Trade-offs
- Bigger models answer better but cost more and are slower; routing picks per question.
- Saving everything enables history and training opt in, but needs strict privacy controls and deletion.