5 Patterns Every Production Voice Agent Needs

The worst bug in voice AI has no error message.

No exception. No crash. Just silence. Someone says "Hey Ferni" and gets nothing back, and the cause is something as small as a missing reconnection handler.

That's the core lesson: voice AI has failure modes you've never seen in traditional web development. Network blips that would be invisible in HTTP become conversation-ending silences. Rate limits don't show error pages - they just make your AI seem stupid.

Here are five patterns that guard against those failures. Copy-paste them into your codebase before you need them.

---

1. The Reconnection Handler

Our original code looked innocent enough:

const client = new FerniClient({ apiKey: process.env.FERNI_API_KEY });
await client.start();
// Done! ...right?

The problem? WebSocket connections drop. A lot. Mobile users walk through dead zones. Coffee shop WiFi hiccups. Home routers restart at 3 AM. And when that connection drops mid-sentence, your user gets silence.

What to use instead:

const client = new FerniClient({
  apiKey: process.env.FERNI_API_KEY,
  reconnect: true,
  reconnectAttempts: 3,
  reconnectDelay: 1000,  // Exponential backoff starts here
});

// This saved us on launch night
client.on('disconnected', (reason) => {
  analytics.track('connection_lost', { reason });
  // Show "reconnecting..." indicator
  ui.showReconnecting();
});

client.on('reconnecting', (attempt) => {
  if (attempt > 2) {
    ui.showMessage("Hold tight, I'm reconnecting...");
  }
});

client.on('connected', () => {
  ui.hideReconnecting();
  // User doesn't even know it happened
});

The key insight: users forgive reconnection delays if they know what's happening. What they won't forgive is unexplained silence.

---

2. The Webhook Signature Check

Picture a burst of "subscription upgraded" webhooks, all from the same IP, all with suspiciously similar timing. Someone has found your webhook endpoint and is spoofing upgrade events to get free premium access.

With signature verification, every forged request fails validation and gets logged. Without it, you hand out free subscriptions to every spoofed account.

This is not optional:

import { verifyWebhookSignature } from '@ferni/sdk';

app.post('/webhook', (req, res) => {
  const signature = req.headers['x-ferni-signature'];
  const timestamp = req.headers['x-ferni-timestamp'];

  // This check stopped the fraud attempt
  const isValid = verifyWebhookSignature({
    payload: req.body,
    signature,
    timestamp,
    secret: process.env.WEBHOOK_SECRET,
    maxAge: 300,  // Reject anything older than 5 minutes
  });

  if (!isValid) {
    // Log it - you'll want to know about these
    security.log('invalid_webhook', { ip: req.ip });
    return res.status(401).json({ error: 'Invalid signature' });
  }

  // Safe to process
  handleWebhook(req.body);
  res.json({ received: true });
});

Pro tip: Set maxAge to prevent replay attacks. A valid signature doesn't help if someone can capture a legitimate webhook and send it again a thousand times.

---

3. The Context-Preserving Handoff

A conversation starts with Ferni, then turns into "actually, can I talk to someone who knows about fitness?" We have Maya (our coaching persona), but the naive approach - just switching - feels jarring.

"Hi! I'm Maya! How can I help?"

...when Maya had no idea what the user had just spent 5 minutes explaining to Ferni.

The context-preserving handoff:

async function handoffToSpecialist(context, targetPersona) {
  // Don't just switch - bring the conversation with you
  const summary = await context.summarizeConversation({
    maxLength: 500,
    focus: 'user_concerns_and_preferences',
  });

  return {
    action: 'handoff',
    target: targetPersona,
    context: {
      summary,
      userMood: context.detectedMood,
      pendingTopics: context.unaddressedTopics,
      // This is the magic
      continuationPrompt: `Continue from: "${context.lastUserMessage}"`,
    },
  };
}

Now Maya would say: "Ferni mentioned you're looking to start a morning workout routine and that you've struggled with consistency before. I totally get that - let's figure out something that actually fits your life."

It feels like the AI was listening, because it was. It just remembered what was said.

---

4. The Streaming Handler

The naive implementation waits for the complete response before doing anything. The LLM thinks, then suddenly dumps the entire response. It feels sluggish and robotic.

The fix was streaming - but not just for text. Voice AI streams emotions, actions, and transcript simultaneously.

const stream = await client.streamResponse(userInput);

let fullTranscript = '';
for await (const chunk of stream) {
  switch (chunk.type) {
    case 'transcript':
      // Text starts appearing instantly
      fullTranscript += chunk.text;
      ui.updateTranscript(fullTranscript);
      break;

    case 'emotion':
      // Avatar reacts BEFORE words finish
      // This creates the "anticipation" effect
      updateAvatarExpression(chunk.emotion);
      break;

    case 'action':
      // Tools execute mid-sentence
      // "Let me check your calendar..." [calendar actually opens]
      executeAction(chunk.action);
      break;
  }
}

The result: responses feel faster even though the total latency is the same, because the first words arrive sooner. Perception matters.

---

5. The Error Boundary That Turned Failures Into Features

Rate limits and context limits are brutal in voice. The LLM hits a limit mid-conversation, and the person on the other end gets a generic "Something went wrong."

Turn it into a feature instead of a bug:

async function safeVoiceInteraction(input) {
  try {
    return await client.process(input);
  } catch (error) {
    if (error.code === 'CONTEXT_LIMIT') {
      // Don't show an error - compress and continue
      await client.compactContext({
        preserveRecent: 10,  // Keep last 10 turns
        summarizeRest: true,  // Summarize older context
      });

      return await client.process(input);
    }

    if (error.code === 'RATE_LIMITED') {
      // Make the delay feel intentional
      return {
        response: "Give me just a second to think about that...",
        retry: true,
        retryAfter: error.retryAfter,
      };
    }

    // Unknown error - log it, show graceful message
    errorTracker.capture(error);
    return {
      response: "I got a bit confused there. Could you say that again?",
      retry: false,
    };
  }
}

The user never hears about a rate limit. They just hear an AI taking a moment to think.

---

The Pattern Behind the Patterns

All five of these have something in common: they anticipate failure modes that break the illusion of conversation.

Traditional web apps can show loading spinners. They can display error messages. They can retry in the background while users read content.

Voice AI can't do any of that. When something goes wrong, there's just silence. And silence is the enemy of good conversation.

Build for the network dropping. Build for the API timing out. Build for the context overflowing. Build for the bad actor.

Your users will never know. They'll just think your AI is remarkably good at conversation.

---

Start Here

1. Copy the reconnection handler - you will have network issues
2. Implement webhook signatures - you will have security issues
3. Test the handoff flow - your users will switch contexts
4. Enable streaming - perceived speed matters more than actual speed
5. Write error boundaries - graceful degradation is a feature

Each of these is cheaper to add before launch than after.

---

Have a pattern that saved your production deployment? Share it on Discord - we're building a community cookbook.