WhyUAscii

The demo works. Production is a different problem

We help teams get AI features past the prototype. Retrieval that returns the right thing, evaluation that catches regressions, isolation that holds, and cost you can forecast.

Get an AI readiness review
  • Use-case audit that names what is worth building and what is not, with reasoning attached
  • Hybrid keyword and vector retrieval, structure-aware chunking, reranking, and measured recall
  • Evaluation suites that run in CI, so prompt changes are reviewed and rollable like code
  • Tenant-scoped retrieval, isolated vector namespaces, per-request prompt assembly, full call logs
  • Typed tools executed under your permissions, with human confirmation on irreversible actions
  • Per-request cost and latency measured from day one, with model routing and hard token ceilings

What an AI consultant does

AI consulting comes down to two questions. Where is AI genuinely the right tool, and what has to be true before shipping it is safe?

The first question kills more projects than the second, and that's the point. Once a use case survives it, the rest is concrete engineering. Choosing models and vendors. Designing the retrieval pipeline. Building an evaluation harness. Enforcing tenant isolation below the model. Instrumenting cost and latency. Then handing the whole thing to your engineers with enough documentation that they can change it without calling us.

We're not a place to buy a chatbot. If a database query, a rules engine, a search index, or a better-designed form would serve you better, we'll say so and explain why. Most of the durable value in AI work comes from picking the right three problems. Not from picking the right model.

Why AI demos fail in production

A demo proves a model can do a task once, on an input someone chose. Production asks it to do that task thousands of times, for people you've never met, on data you've never seen.

That gap is where projects die, and the failures are boring and predictable. The demo got tested on the happy path. Real inputs are misspelled, empty, in another language, four hundred pages long, or written by someone trying to break your system. Nobody can tell whether a change helped, because the workflow is edit the prompt, check three examples, ship it. Quality degrades quietly on the other nine hundred.

Retrieval returns plausible but wrong context, and the model faithfully summarizes the wrong document with total confidence. Cost per request was never measured, so the feature is fine at a hundred users and ruinous at ten thousand. Latency was never budgeted, so an eight-second wait that felt acceptable in a demo becomes a product people close. And nobody scoped what the model is allowed to do, so a hostile instruction buried in an uploaded PDF turns into an action your system takes.

None of that is a model problem. Those are engineering problems arranged around a model, which is why the fix is engineering and not a better prompt.

Deciding whether to use AI at all

We open every AI engagement by trying to disqualify the use case. If it survives that, it's worth building.

It probably fits if the task is language-shaped, meaning the input or the output is unstructured text. If there are many acceptable answers instead of one exact correct one. If a human does it today, and a wrong answer gets noticed and can be walked back. And if the volume is high enough that even partial automation pays for the engineering wrapped around it.

It probably doesn't fit if the answer has to be exactly right and independently verifiable. That's a query or a calculation. If the rules are small and stable, write the rules. If a confident wrong answer carries legal, financial, or physical consequences, slow down hard. And if the information the model would need in order to be correct doesn't exist anywhere in your systems, no model compensates for that. None of them.

The most valuable output of a first engagement is often a short list. Three things worth building, seven that aren't, and the reasoning attached so the conversation doesn't restart every quarter.

Retrieval is a search problem before it is a model problem

Most retrieval-augmented generation systems fail at retrieval, not generation. If the right passage never enters the context window, no model recovers from that. Worse, the better the model writes, the more convincing the wrong answer sounds.

So we treat retrieval as its own system with its own quality bar. Chunking respects document structure instead of cutting every five hundred characters through the middle of a table. Metadata rides along with every chunk: source document, section, date, and the permissions that govern it, so you can show a citation and enforce access. Retrieval is hybrid, keyword and vector together, because part numbers, error codes, proper names, and exact identifiers are precisely where embedding similarity falls apart. Results get reranked before they reach the model, since the top match by embedding distance often isn't the top match by relevance.

Then we measure it. Before anyone reads a generated answer, we build a labeled set of real questions along with the passages that should be found, and we check whether retrieval actually finds them. You can't measure answer quality while retrieval quality is unknown.

We also make the system able to say it doesn't know. A pipeline that always returns its top five chunks no matter how weak the match guarantees that vague questions produce confident answers assembled from irrelevant context. A relevance floor and an explicit no-answer path beat another paragraph of prompt asking the model to be honest.

And freshness. Documents change, and an index that reflects last quarter's policy is worse than no index at all, because people trust it.

Evaluation: knowing it works before your customers do

You can't ship an AI feature you can't measure, and you can't measure one on impressions.

We build the evaluation set out of real inputs, never invented ones. It mixes a few kinds of check. Exact-match cases, where a correct answer genuinely exists. Retrieval recall, measured separately from generation. Model-graded rubrics for open-ended output, where the rubric is written down and agreed on beforehand and not improvised during grading. And regression cases pulled from actual failures, so the same mistake can't come back quietly.

Evaluations run in CI next to your tests. A prompt change is a code change and gets handled like one. It lives in version control, goes through review, runs the eval suite, and can be rolled back. Prompts edited directly in a production console are how quality collapses without anyone being able to say when it started.

After launch, the work keeps going. We sample real traffic and log the input, the retrieved context, the model output, and what happened next, then review a slice on a schedule. Give users a way to flag a bad answer, too. The failures that matter most are the ones nobody thought to write a test for.

The point isn't a score on a dashboard. The point is that when someone edits a prompt at four on a Thursday afternoon, you find out before your customers do.

Tool use and agents: scope before capability

An agent is only as safe as the narrowest tool you hand it. Capability is easy to add and very hard to take back, so we scope first.

Each tool is a typed function with a validated input schema, executed by your code under your permissions. The model proposes. Your system authorizes. That distinction is the whole security model, and it collapses the moment a tool takes free-form input and passes it somewhere consequential.

The rules we apply don't move much. Read tools are cheap, so they get broad use. Write tools pass the same authorization checks as your regular API, because the model calling them isn't a privileged user. Irreversible actions get a human confirmation step or a reversible staging step, so a mistake becomes a rollback and not an incident. Every tool call is logged with its arguments, its result, and the conversation that produced it. Step limits and token budgets cap runaway loops, because an agent that retries forever is both an outage and an invoice.

Treat model output as untrusted input, because that's exactly what it is. Anything the model reads, whether a customer document, a web page, an email, or a support ticket, might carry instructions written for your system and not for a person. Your defenses belong at the boundary where actions get authorized. Not in a paragraph of the prompt asking the model to be careful.

Multi-tenant AI: isolation is architecture, not a prompt

You can't instruct a model not to leak. Isolation gets enforced underneath it, at the point where data is fetched.

So every retrieval is scoped by tenant at query time, down in the data layer, never fetched broadly and filtered afterward. Vector data lives in separate namespaces or partitioned indexes, so one tenant's embeddings are never even candidates in another tenant's search. Conversation memory and caches are keyed by tenant and never shared. System prompts get assembled per request from that tenant's configuration, so you're not maintaining one global prompt full of conditional branches for different customers.

Logging is part of isolation and not a separate concern. Every call records the tenant, the user, the model, the token counts, and the identifiers of the documents retrieved. When a customer asks what their data was used for, the answer is a query and not a guess. That same log is what lets you investigate a bad answer six months later.

Only answer using documents belonging to this customer is a request, not a control. If the wrong document can reach the context window, one day it will.

Cost and latency you can predict

Cost and latency are design constraints you choose before the architecture. They aren't parameters you tune after the first surprising invoice.

We measure cost per request from day one and put it on a dashboard next to latency, because a number nobody watches is a number that grows. Work gets routed by difficulty. A small fast model handles classification, extraction, and routing. The bigger one is reserved for the step that genuinely needs it. Caching applies at several layers, including identical retrievals and stable prompt prefixes. Responses stream, so perceived latency drops even when total time doesn't. Anything no user is waiting on moves to a background job, which changes both your latency budget and the model you can afford to run.

Hard ceilings matter too. A maximum token budget per request. A maximum number of steps per agent run. Alerts on cost per active user, not only on total spend. Total spend rising is ambiguous. Cost per user rising is a bug.

A feature whose unit economics only work at today's volume is a problem waiting for success to show up.

Staying portable across models and vendors

We build so that changing model provider is a configuration change plus a rerun of the evaluation suite. Not a rewrite.

Model calls sit behind one internal interface and don't get scattered through the application. Prompts live in version control as data, with their own history. Provider-specific features get used where they earn their place, but deliberately, with a note about what depends on them, so the cost of leaving is known ahead of time and not discovered on the way out.

The evaluation suite is what actually makes switching safe. It answers whether a new model is better at your task, on your data. A benchmark someone else ran on someone else's problem can't tell you that. Public benchmark position and performance on your specific workload are only loosely related.

This field moves faster than anyone's procurement cycle. The teams that handle it well aren't the ones who picked correctly at the start. They're the ones who made switching cheap before they needed to.

Common questions

What is RAG and does my product actually need it?
RAG, or retrieval-augmented generation, means searching your own data for relevant passages and handing them to a language model as context before it answers. You need it when correct answers depend on information the model was never trained on, like your documentation, your customer records, or your policies. You don't need it when the task is classification, extraction, rewriting, or summarizing text the user already handed you.
Our AI prototype works but we are afraid to put it in front of customers. What is missing?
Usually four things. An evaluation set built from real inputs, so quality can be measured and not sensed. Retrieval quality measured separately from answer quality. Isolation enforced in the data layer, so one customer's information can't reach another's context. And cost and latency instrumented per request. A prototype proves the task is possible once. Those four are what let you change it on a Thursday without holding your breath.
How do you test an AI feature when there is no single correct answer?
You mix evaluation types instead of hunting for one metric. Exact-match tests cover the subset where a correct answer exists. Retrieval recall gets measured on its own, since most failures happen before generation. Open-ended output is graded against a written rubric, often by a model, with human review on a sample. Every production failure becomes a permanent regression case. WhyUAscii LLC runs the whole suite in CI, so a prompt change can't ship without passing it.
Which AI model should we use?
Whichever one wins on your evaluation suite, on your data, at a cost and latency you can live with. Public benchmark rankings correlate only loosely with performance on a specific task. The more durable decision is architectural: put model calls behind one internal interface, keep prompts in version control, and maintain an evaluation suite. Then changing providers is a config change and a rerun, not a project.
How do you stop one customer's data from appearing in another customer's AI results?
You enforce isolation below the model, never in the prompt. Every retrieval is scoped by tenant at query time in the data layer, not filtered after fetching. Vector data lives in separate namespaces, so one tenant's embeddings are never candidates in another's search. Caches and conversation memory are keyed by tenant. Every call is logged with tenant, user, and retrieved document identifiers. Asking a model politely to respect boundaries is not a control.
Should we fine-tune a model or use retrieval?
Retrieval first, most of the time. Fine-tuning teaches a model form, tone, and task shape. It doesn't reliably teach facts, and facts baked into weights can't be updated when the underlying document changes. If the model lacks your information, use retrieval. If it won't follow your format or your domain conventions even after clear instruction and good examples, then fine-tuning is worth evaluating.
Our AI feature is too slow and costs too much per request. Can that be fixed?
Usually, and rarely by swapping in a cheaper model alone. The reliable levers are routing work by difficulty so small models handle classification and extraction, caching identical retrievals and stable prompt prefixes, trimming context that's being sent without earning its place, streaming responses to cut perceived latency, and moving anything no user is waiting on into a background job. Measure cost per request first. That's what makes the right lever obvious.
Can you build AI agents that take actions instead of only answering questions?
Yes, with the scope defined before the capability. Each tool is a typed function with a validated schema, executed by your code under your permissions, so the model proposes and your system authorizes. Write actions pass the same authorization checks as your regular API. Irreversible actions get a human confirmation or a reversible staging step. Every call is logged, and step and token ceilings stop runaway loops before they become an invoice.
Do we need a data scientist or ML engineer on staff to maintain this after you leave?
No, for most applied LLM work. What these systems need is ordinary software engineering discipline: version-controlled prompts, an evaluation suite in CI, logging, cost dashboards, and someone reviewing sampled outputs on a schedule. WhyUAscii LLC builds AI features so a competent application engineering team can operate and change them, and writes the documentation with that team as the intended reader.
What if AI turns out to be the wrong solution for our problem?
Then that's the finding, and the engagement says so out loud. Plenty of problems brought to AI consultants are better solved by a database query, a search index, a rules engine, or a redesigned form, and those are cheaper, faster, and correct every single time. WhyUAscii LLC opens engagements by trying to disqualify the use case, because a short honest list of what's worth building beats a proof of concept nobody can ship.

Bring us the demo that is not shipping

Tell us what the prototype does, what's stopping it from reaching customers, and what data it depends on. We'll tell you what has to be true before it's safe to ship, and whether AI is the right tool at all. Email hello@whyuascii.com.

Get an AI readiness reviewhello@whyuascii.com