Build a real-time voice assistant

Five-stage workflow

Recommended components

Start with one component per capability and replace it only when evaluation data shows a clear gap.

From proof of concept to production

1. 01Define the user journey, input and output schemas, permission boundaries, and success metrics.
2. 02Build one end-to-end happy path with the smallest viable tool stack and a fixed evaluation set.
3. 03Add authentication, timeouts, retries, fallbacks, audit logs, and human review for risky actions.
4. 04Release to a limited audience, measure quality, latency, and cost, then expand with evidence.

Framework-neutral orchestration skeleton

type WorkflowInput = { request: string; userId: string };
type StageAdapter = (context: unknown) => Promise<unknown>;

const stages = [
  "Capture audio and detect noise, endpoints, and language",
  "Stream transcription and maintain conversation context",
  "Resolve intent and call external tools safely",
  "Generate concise replies and stream synthesized speech",
  "Monitor end-to-end latency, interruption, and task success"
];
const selectedTools = [
  "Hugging Face",
  "LangChain.js"
];

export async function runWorkflow(input: WorkflowInput, adapters: Record<string, StageAdapter>) {
  let context: unknown = { ...input, selectedTools };

  for (const stage of stages) {
    const adapter = adapters[stage];
    if (!adapter) throw new Error(`Missing adapter: ${stage}`);
    context = await adapter(context);
  }

  return context;
}

Implement each adapter on the server with the selected component’s official SDK or API. Never expose provider credentials in browser code.

Production checklist

• ✓Every stage has a defined owner, input, output, timeout, and failure state.
• ✓Credentials and sensitive data stay server-side with least-privilege access.
• ✓Model output is validated before it reaches a database, user, or external action.
• ✓Sources, tool calls, prompts, versions, and human decisions are traceable.
• ✓A fixed evaluation set covers normal, ambiguous, adversarial, and unavailable-tool cases.
• ✓The workflow has a visible fallback and a manual recovery path.