The “Tutorial for LiveKit Cloud & Twilio (Step by Step Guide)” helps you deploy a LiveKit cloud agent to your mobile device from scratch. It walks you through setting up Twilio, Deepgram, Cartesia, and OpenAI keys, configuring SIP trunks, and using the command line to deploy a voice agent that can handle real inbound calls.
The guide follows a clear sequence—SOP, Part 1 and Part 2, local testing, cloud deployment, Twilio setup, and live testing—with timestamps so you can jump to what you need. You’ll also learn how to run the stack cost-effectively using free credits and service tiers, ending with a voice agent capable of handling high-concurrency sessions and free minutes on LiveKit.
Prerequisites and system requirements
Before you begin, make sure you have a developer machine or cloud environment where you can run command-line tools, install SDKs, and deploy services. You’ll need basic familiarity with terminal commands, Git, and editing environment files. Expect to spend time configuring accounts and verifying network access for SIP and real-time media. Plan for both local testing and eventual cloud deployment so you can iterate quickly and then scale.
Supported operating systems and command-line tools required
You can run the agent and tooling on Linux, macOS, or Windows (Windows Subsystem for Linux recommended). You’ll need a shell (bash, zsh, or PowerShell), Git, and a package/runtime manager for your chosen language (Node.js with npm or pnpm, Python with pip, or Go). Install CLIs for LiveKit, Twilio, and any SDKs you choose to use. Common tools include curl or HTTPie for API testing, and a code editor like VS Code. Make sure your OS network settings allow RTP/UDP traffic for media testing and that you can adjust firewall rules if needed.
Accounts to create beforehand: LiveKit Cloud, Twilio, Deepgram, Cartesia, OpenAI
Create accounts before you start so you can obtain API keys and configure services. You’ll need a LiveKit Cloud project for the media plane and agent hosting, a Twilio account for phone numbers and SIP trunks, a Deepgram account for real-time speech-to-text, a Cartesia account if you plan to use their tooling or analytics, and an OpenAI account for language model responses. Having these accounts ready prevents interruptions as you wire services together during the tutorial.
Recommended quota and free tiers available including LiveKit free minutes and Deepgram credit
Take advantage of free tiers to test without immediate cost. LiveKit typically provides developer free minutes and a “Mini” tier you can use to run small agents and test media; in practice you can get around 1,000 free minutes and support for dozens to a hundred concurrent sessions depending on the plan. Deepgram usually provides promotional credits (commonly $200) for new users to test transcription. Cartesia often includes free minutes or trial analytics credits, and OpenAI has usage-based billing and may include initial credits depending on promotions. For production readiness, plan a budget for additional minutes, transcription usage, and model tokens.
Hardware and network considerations for running a mobile agent locally and in cloud
When running a mobile agent locally, a modern laptop or small server with at least 4 CPU cores and 8 GB RAM is fine for development; more CPU and memory will help if you run multiple concurrent sessions. For cloud deployment, choose an instance sized for your expected concurrency and CPU-bound model inference tasks. Network-wise, ensure low-latency uplinks (preferably under 100 ms to your Twilio region) and an upload bandwidth that supports multiple simultaneous audio streams (each call may require 64–256 kbps depending on codec and signaling). Verify NAT traversal with STUN/TURN if you expect clients behind restrictive firewalls.
Permissions and billing settings to verify in cloud and Twilio accounts
Before testing live calls, confirm billing is enabled on Twilio and LiveKit accounts so phone number purchases and outbound connection attempts aren’t blocked. Ensure your Twilio account is out of trial limitations if you need unrestricted calling or PSTN access. Configure IAM roles or API key scopes in LiveKit and any cloud provider so the agent can create rooms, manage participants, and upload logs. For Deepgram and OpenAI, monitor quotas and set usage limits or alerts so you don’t incur unexpected charges during testing.
Architecture overview and data flow
Understanding how components connect will help you debug and optimize. At a high level, your architecture will include Twilio handling PSTN phone numbers and SIP trunks, LiveKit as the SIP endpoint or media broker, a voice agent that processes audio and integrates with Deepgram for transcription, OpenAI for AI responses, and Cartesia optionally providing analytics or tooling. The voice agent sits at the center, routing media and events between these services while maintaining session state.
High-level diagram describing LiveKit, Twilio SIP trunk, voice agent, and transcription services
Imagine a diagram where PSTN callers connect to Twilio phone numbers. Twilio forwards media via a SIP trunk to LiveKit or directly to your SIP agent. LiveKit hosts the media room and can route audio to your voice agent, which may run as a worker inside LiveKit Cloud or a separate service connected through the SIP interface. The voice agent streams audio to Deepgram for real-time transcription and uses OpenAI to generate contextual replies. Cartesia can tap into logs and transcripts for analytics and monitoring. Each arrow in the diagram represents a media stream or API call with clear directionality.
How inbound phone calls flow through Twilio into SIP/LiveKit and reach the voice agent
When a PSTN caller dials your Twilio number, Twilio applies your configured voice webhook or SIP trunk mapping. If using a SIP trunk, Twilio takes the call media and SIP-signals it to the SIP URI you defined (which can point to LiveKit’s SIP endpoint or your SIP proxy). LiveKit receives the SIP INVITE, creates or joins a room, and either bridges the call to the voice agent participant or forwards media to your agent service. The voice agent then receives RTP audio, processes that audio for transcription and intent detection, and sends audio responses back into the room so the caller hears the agent.
Where Deepgram and OpenAI fit in for speech-to-text and AI responses
Deepgram is responsible for converting the live audio streams into text in real time. Your voice agent will stream audio to Deepgram and receive partial and final transcripts. The agent feeds these transcripts, along with session context and possibly prior conversation state, into OpenAI models to produce natural responses. OpenAI returns text that the agent converts back into audio (via a TTS service or an audio generation pipeline) and plays back to the caller. Deepgram can also handle diarization or confidence scores that help decide whether to reprompt or escalate to a human.
Roles of Cartesia if it is used for additional tooling or analytics
Cartesia can provide observability, session analytics, or attached tooling for your voice conversations. If you integrate Cartesia, it can consume transcripts, call metadata, sentiment scores, and event logs to visualize agent performance, highlight keywords, and produce call summaries. You might use Cartesia for post-call analytics, searching across transcripts, or building dashboards that track concurrency, latency, and conversion metrics.
Latency, concurrency, and session limits to be aware of
Measure end-to-end latency from caller audio to AI response. Transcription and model inference add delay: Deepgram streaming is low-latency (tens to hundreds of milliseconds) but OpenAI response time depends on model and prompt size (hundreds of milliseconds to seconds). Factor in network round trips and audio encoding/decoding overhead. Concurrency limits come from LiveKit project quotas, Deepgram connection limits, and OpenAI rate limits; ensure you’ve provisioned capacity for peak sessions. Monitor session caps and use backpressure or queueing in your agent to protect system stability.
Create and manage API keys
Properly creating and storing keys is essential for secure, stable operation. You’ll collect keys from LiveKit, Twilio, Deepgram, OpenAI, and Cartesia and use them in configuration files or secret stores. Limit scope when possible and rotate keys periodically.
Generate LiveKit Cloud API keys and configure project settings
In LiveKit Cloud, create a project and generate API keys (API key and secret). Configure project-level settings such as allowed origins, room defaults, and any quota or retention policies. If you plan to deploy agents in the cloud, create a service key or role with permissions to create rooms and manage participants. Note the project ID and any region settings that affect media latency.
Obtain Twilio account SID, auth token, and configure programmable voice resources
From Twilio, copy your Account SID and Auth Token to a secure location (treat them like passwords). In Twilio Console, enable Programmable Voice, purchase a phone number for inbound calls, and set up a SIP trunk or voice webhook. Create any required credential lists or IP access control if you use credential-based SIP authentication. Ensure that your Twilio settings (voice URLs or SIP mappings) point to your LiveKit or SIP endpoint.
Create Deepgram API key and verify $200 free credit availability
Sign into Deepgram and generate an API key for real-time streaming. Confirm your account shows the promotional credit balance (commonly $200 for new users) and understand how transcription billing is calculated (per minute or per second). Restrict the key so it is used only by your voice agent services or set per-key quotas if Deepgram supports that.
Create OpenAI API key and configure usage limits and models
Generate an OpenAI API key and decide which models you’ll use for agent responses. Configure rate limits or usage caps in your account to avoid unexpected spend. Choose faster, lower-cost models for short interactive responses and larger models only where more complex reasoning is needed. Store the key securely.
Store keys securely using environment variables or a secret manager
Never hard-code keys in source. Use environment variables for local development (.env files that are .gitignored), and use a secret manager (cloud provider secrets, HashiCorp Vault, or similar) in production. Reference secret names in deployment manifests or CI/CD pipelines and grant minimum permissions to services that need them.
Install CLI tools and SDKs
You’ll install the command-line tools and SDKs required to interact with LiveKit, Twilio, Deepgram, Cartesia, and your chosen runtime. This keeps local development consistent and allows you to script tests and deployments.
Install LiveKit CLI or any required LiveKit developer tooling
Install the LiveKit CLI to create projects, manage rooms, and inspect media sessions. The CLI also helps with deploying or debugging LiveKit Cloud agents. After installing, verify by running the version command and authenticate the CLI against your LiveKit account using your API key.
Install Twilio CLI and optionally Twilio helper libraries for your language
Install the Twilio CLI to manage phone numbers, SIP trunks, and test calls from your terminal. For application code, install Twilio helper libraries in your language (Node, Python, Go) to make API calls for phone number configuration, calls, and SIP trunk management.
Install Deepgram CLI or SDK and any Cartesia client libraries if needed
Install Deepgram’s SDK for streaming audio to the transcription service from your agent. If Cartesia offers an SDK for analytics or instrumentation, add that to your dependencies so you can submit transcripts and metrics. Verify installation with a simple transcript test against a sample audio file.
Install Node/Python/Go runtime and dependencies for the voice agent project
Install the runtime for the sample voice agent (Node.js with npm or yarn, Python with virtualenv and pip, or Go). Install project dependencies, and run package manager diagnostics to confirm everything is resolved. For Node projects, run npm ci or install; for Python, create a venv and pip install -r requirements.txt.
Verify installations with version checks and test commands
Run version checks for each CLI and runtime to ensure compatibility. Execute small test commands: list LiveKit rooms, fetch Twilio phone numbers, send a sample audio to Deepgram, and run a unit test from the repository. These checks prevent surprises when you start wiring services together.
Clone, configure, and inspect the voice agent repository
You’ll work from an example repository or template that integrates SIP, media handling, and AI hooks. Inspecting the structure helps you find where to place keys and tune audio parameters.
Clone the example repository used in the tutorial or a template voice agent
Use Git to clone the provided voice agent template. Choose the branch that matches your runtime and read the README for runtime-specific setup. Having the template locally lets you modify prompts, adjust retry behavior, and instrument logging.
Review project structure to locate SIP, media, and AI integration files
Open the repository and find directories for SIP handling, media codecs, Deepgram integration, and OpenAI prompts. Typical files include the SIP session handler, RTP adapter, transcription pipeline, and an AI controller that constructs prompts and handles TTS. Understanding this layout lets you quickly change behavior or add logging.
Update configuration files with LiveKit and third-party API keys
Edit the configuration or .env file to include LiveKit project ID and secret, Twilio credentials, Deepgram key, OpenAI key, and Cartesia token if applicable. Keep example .env.sample files for reference and never commit secrets. Some repos include a config.json or YAML file for codec and session settings—update those too.
Set environment variables and example .env file entries for local testing
Create a .env file with entries like LIVEKIT_API_KEY, LIVEKIT_API_SECRET, TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, DEEPGRAM_API_KEY, OPENAI_API_KEY, and CARTESIA_API_KEY. For local testing, you may also set DEBUG flags, local port numbers, and TURN/STUN endpoints. Document any optional flags for tracing or mock mode.
Explain key configuration options such as audio codecs, sample rates, and session limits
Key options include the audio codec (PCMU/PCMA for telephony compatibility, or Opus for higher fidelity), sample rates (8 kHz for classic telephony, 16 kHz or 48 kHz for better ASR), and audio channels. Session limits in config govern max concurrent calls, buffer sizes for streaming to Deepgram, and timeouts for AI responses. Tune these to balance latency, transcription accuracy, and cost.
Local testing: run the voice agent on your machine
Testing locally allows rapid iteration before opening to PSTN traffic. You’ll verify media flows, transcription accuracy, and AI prompts with simulated calls.
Start LiveKit server or use LiveKit Cloud dev mode for local testing
If you prefer a local LiveKit server, run it on your machine and point the agent to localhost. Alternatively, use LiveKit Cloud’s dev mode to avoid local server setup. Ensure the agent’s connection parameters (API keys and region) match the LiveKit instance you use.
Run the voice agent locally and confirm it registers with LiveKit
Start your agent process and observe logs verifying it connects to LiveKit, registers as a participant or service, and is ready to accept media. Confirm the agent appears in the LiveKit room list or via the CLI.
Simulate inbound calls locally by using Twilio test credentials or SIP tools
Use Twilio test credentials or SIP softphone tools to generate SIP INVITE messages to your configured SIP endpoint. You can also replay pre-recorded audio into the agent using RTP injectors or SIP clients to simulate caller audio. Verify the agent accepts the call and audio flows are established.
Test Deepgram transcription and OpenAI response flows from a sample audio file
Feed a sample audio file through the pipeline to Deepgram and ensure you receive partial and final transcripts. Pass those transcripts into your OpenAI prompt logic and verify you get sensible replies. Check that TTS or audio playback works and that the synthesized response is played back into the simulated call.
Common local troubleshooting steps including port, firewall, and codec mismatches
If things fail, check that required ports (SIP signaling and RTP ports) are open, that NAT or firewall rules aren’t blocking traffic, and that sample rates and codecs match across components. Look at logs for SIP negotiation failures, codec negotiation errors, or transcription timeouts. Enabling debug logging often reveals mismatched payload types or dropped packets.
Setting up Twilio for SIP and phone number handling
Twilio will be your gateway to the PSTN, so set up trunks, numbers, and secure mappings carefully.
Create a Twilio SIP trunk or configure Programmable Voice depending on architecture
Decide whether to use a SIP trunk (recommended for direct SIP integration with LiveKit or a SIP proxy) or Programmable Voice webhooks if you want TwiML-based control. Create a SIP trunk in Twilio, and add an Origination URI that points to your SIP endpoint. Configure the trunk settings to handle codecs and session timers.
Purchase and configure a Twilio phone number to receive inbound calls
Purchase an inbound-capable phone number in the Twilio console and assign it to route calls to your SIP trunk or voice webhook. Set the voice configuration to either forward calls to the SIP trunk or call a webhook that uses TwiML to instruct call forwarding. Ensure the number’s voice capabilities match your needs (PSTN inbound/outbound).
Configure SIP domain, authentication methods, and credential lists for secure SIP
Create credential lists and attach them to your trunk to use username/password authentication if needed. Alternatively, use IP access control to restrict which IPs can originate calls into your SIP trunk. Configure SIP domains and enforce TLS for signaling to protect call setup metadata.
Set up voice webhook or SIP URI mapping to forward incoming calls to LiveKit/SIP endpoint
If you use a webhook, configure the TwiML to dial your SIP URI that points to LiveKit or your SIP proxy. If using a trunk, set the trunk’s origination and termination URIs appropriately. Make sure the SIP URI includes the correct transport parameter (e.g., transport=tls) if required.
Verify Twilio console settings and TwiML configuration for proper media negotiation
Use Twilio’s debugging tools and logs to confirm SIP INVITEs are sent and that Twilio receives 200 OK responses. Check media codec negotiation to ensure Twilio and LiveKit agree on a codec like PCMU or Opus. Use Twilio’s diagnostics to inspect signaling and media problems and iterate.
Connecting Twilio and LiveKit: SIP trunk configuration details
Connecting both systems requires attention to SIP URI formats, transport, and authentication.
Define the exact SIP URI and transport protocol (UDP/TCP/TLS) used by LiveKit
Decide on the SIP URI format your LiveKit or proxy expects (for example, sip:user@host:port) and whether to use UDP, TCP, or TLS. TLS is preferred for signaling security. Ensure the URI is reachable and resolves to the LiveKit ingress or proxy that accepts SIP calls.
Configure Twilio trunk origination URI to point to LiveKit Cloud agent or proxy
In the Twilio trunk settings, add the LiveKit SIP URI as an Origination URI. Specify transport and port, and if using TLS you may need to provide or trust certificates. Confirm the URI’s hostname matches the certificate subject when using TLS.
Set up authentication mechanism such as IP access control or credential-based auth
For security, prefer IP access control lists that only permit Twilio’s egress IPs, or set up credential lists with scoped usernames and strong passwords. Store credentials in Twilio’s credential store and bind them to the trunk. Audit these credentials regularly.
Testing SIP registration and call flow using Twilio’s SIP diagnostics and logs
Place test calls and consult Twilio logs to trace SIP messaging. Twilio provides detailed SIP traces that show INVITEs, 200 OKs, and RTP negotiation. Use these traces to pinpoint header mismatches, authentication failures, or codec negotiation issues.
Handle NAT, STUN/TURN, and TLS certificate considerations for reliable media
RTP may fail across NAT boundaries if STUN/TURN aren’t configured. Ensure your LiveKit or proxy has proper STUN/TURN servers and that TURN credentials are available if needed. Maintain valid TLS certificates on your SIP endpoint and rotate them before expiration to avoid signaling errors.
Integrating Deepgram for real-time transcription
Deepgram provides the speech-to-text layer; integrate it carefully to handle partials, punctuation, and robustness.
Enable Deepgram real-time streaming and link it to the voice agent
Enable streaming in your Deepgram account and use the SDK to create WebSocket or gRPC streams from your agent. Stream microphone or RTP-decoded audio with the correct sample rate and encoding type. Authenticate the stream using your Deepgram API key.
Configure audio format and sample rates to match Deepgram requirements
Choose audio formats Deepgram supports (16-bit PCM, Opus, etc.) and match the sample rate (8 kHz for telephony or 16 kHz/48 kHz for higher fidelity). Ensure your agent resamples audio if necessary before sending to Deepgram to avoid transcription degradation.
Process Deepgram transcription results and feed them into OpenAI for contextual responses
Handle partial transcripts by buffering partials and only sending final transcripts or intelligently using partials for low-latency responses. Add conversation context, metadata, and recent turns to the prompt when calling OpenAI so the model can produce coherent replies. Sanitize transcripts for PII if required.
Handle partial transcripts, punctuation, and speaker diarization considerations
Decide whether to wait for final transcripts or act on partials to minimize response latency. Use Deepgram’s auto-punctuation features to improve prompt quality. If multiple speakers are present, use diarization to attribute speech segments properly; this helps your agent understand who asked what and whether to hand off.
Retry and error handling strategies for transcription failures
Implement exponential backoff and retry strategies for Deepgram stream interruptions. On repeated failures, fallback to a different transcription mode or place a prompt to inform the caller there’s a temporary issue. Log failures and surface metrics to Cartesia or your monitoring to detect systemic problems.
Conclusion
You’ve seen the end-to-end components and steps required to build a voice AI agent that connects PSTN callers to LiveKit, uses Deepgram for speech-to-text, and OpenAI for responses. With careful account setup, key management, codec tuning, and testing, you can get a functioning agent that handles real phone calls.
Recap of steps to get a voice AI agent running with LiveKit Cloud and Twilio
Start by creating LiveKit, Twilio, Deepgram, Cartesia, and OpenAI accounts and collecting API keys. Install CLIs and SDKs, clone the voice agent template, configure keys and audio settings, and run locally. Test Deepgram transcription and OpenAI responses with sample audio, then configure Twilio phone numbers and SIP trunks to route live calls to LiveKit. Verify and iterate until the flow is robust.
Key tips to prioritize during development, testing, and production rollout
Prioritize secure key storage and least-privilege permissions, instrument end-to-end latency and error metrics, and test with realistic audio and concurrency. Use STUN/TURN to solve NAT issues and prefer TLS for signaling. Configure usage limits or alerts for Deepgram and OpenAI to control costs.
Resources and links to docs, example repos, and community channels
Look for provider documentation and community channels for sample code, troubleshooting tips, and architecture patterns. Example repositories and official SDKs accelerate integration and show best practices for encoding, retry, and security.
Next steps for advanced features such as analytics, multi-language support, and agent handoff
After basic functionality works, add analytics via Cartesia, support additional languages by configuring Deepgram and model prompts, and implement intelligent handoff to human agents when needed. Consider session recording, sentiment analysis, and compliance logging for regulated environments.
Encouragement to iterate, measure, and optimize based on real call data
Treat the first deployment as an experiment: gather real call data, measure transcription accuracy, latency, and business outcomes, then iterate on prompts, resourcing, and infrastructure. With continuous measurement and tuning, you’ll improve the agent’s usefulness and reliability as it handles more live calls. Good luck — enjoy building your voice AI agent!
If you want to implement Chat and Voice Agents into your business to reduce missed calls, book more appointments, save time, and make more revenue, book a discovery call here: https://brand.eliteaienterprises.com/widget/bookings/elite-ai-30-min-demo-call