Smart Turn Detection

Advanced conversational turn detection using Fal-hosted smart-turn model

Overview

Smart Turn Detection uses an advanced machine learning model to determine when a user has finished speaking and your bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection recognizes natural conversational cues like intonation patterns and linguistic signals for more natural conversations.

On Pipecat Cloud, Smart Turn Detection is powered by Fal.ai’s hosted smart-turn model, providing scalable inference without any setup required.

Smart Turn Model

Try the model in Fal’s interactive playground

GitHub Repository

Open source model for conversational turn detection

Key Benefits

• Natural conversations: More human-like turn-taking patterns
• Zero setup: A Fal API key is automatically provisioned for you
• Free to use: Included at no additional cost
• Scalable: Powered by Fal.ai’s cloud infrastructure

Quick Start

To enable Smart Turn Detection in your Pipecat Cloud bot, add the FalSmartTurnAnalyzer to your transport configuration.

Use an environment variable of FAL_API_KEY which automatically receives a Fal API key at runtime when deployed to Pipecat Cloud.

import os
import aiohttp
from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.transports.services.daily import DailyParams, DailyTransport

async def main(room_url: str, token: str):
    async with aiohttp.ClientSession() as session:
        transport = DailyTransport(
            room_url,
            token,
            "Voice AI Bot",
            DailyParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
                # Set VAD to 0.2 seconds for optimal Smart Turn performance
                vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
                # Enable Smart Turn Detection using FAL_API_KEY, which is automatically provisioned
                turn_analyzer=FalSmartTurnAnalyzer(
                    api_key=os.getenv("FAL_API_KEY"), aiohttp_session=session
                ),
            ),
        )

        # Continue with your pipeline setup...

Smart Turn Detection requires VAD to be enabled with stop_secs=0.2. This value mimics the training data and allows Smart Turn to dynamically adjust timing based on the model’s predictions.

How It Works

1. Audio Analysis: The system continuously analyzes incoming audio for speech patterns
2. VAD Processing: Voice Activity Detection segments audio into speech and silence
3. Turn Classification: When VAD detects a pause, the ML model analyzes the speech segment for natural completion cues
4. Smart Response: The model determines if the turn is complete or if the user is likely to continue speaking

Training Data Collection

The smart-turn model is trained on real conversational data collected through these applications. Help us improve the model by contributing your own data or classifying existing data:

Data Collector

Contribute conversational data to improve the model

Data Classifier

Help classify turn completion patterns in conversations

Deployment Requirements

• Automatic API key provisioning: When deployed to Pipecat Cloud, a Fal API key is automatically provided via the FAL_API_KEY environment variable
• Manual setup for other deployments: You can use Smart Turn Detection locally or on other platforms by obtaining your own Fal API key from fal.ai
• Internet connectivity: Requires connection to Fal.ai’s inference servers

On Pipecat Cloud, the FAL_API_KEY environment variable is automatically provided at no cost. For local development or other deployment platforms, you’ll need to sign up for your own Fal account and API key.

Performance Notes

• The model is optimized for English conversations
• Network latency may vary based on geographic location
• Fallback VAD-based turn detection if the service is unavailable
• Best results with clear audio input and minimal background noise

Assistant

Responses are generated using AI and may contain mistakes.