How Lemon Slice builds AI characters with Pipecat and Daily

Lemon Slice is building the next generation of video foundation models focused on humans. Their platform allows anyone to create videos of expressive, talking characters, and has been used to generate over 1 million clips that range in style from photorealism to cartoons.

Lemon Slice envisions AI video not just as a creator tool, but as the future of interactive media and embodied AI.

---

After becoming one of the top creator tools for talking head videos, we recognized that Generative AI is at an inflection point. Eventually a new type of media and entertainment will emerge that is centered around interactivity. Real-time AI video marks the beginning of that era.

Lina Colucci, CEO LemonSlice

To bring this vision to life, Lemon Slice first had to make their existing video transformer model 10 times faster. Through extensive research and experimentation, they developed a zero-shot model that supports 25fps streaming. What sets Lemon Slice apart from other real-time video chat experiences is that their model doesn't rely on pre-existing videos of real people, 3D game engines, or human-driven motions. With just a single image of a character and audio input, it can generate streaming video of that character speaking in real-time.

With their real-time video transformer ready, Lemon Slice faced a new challenge: how to productize it as a complete interactive experience. Building this would require them to incorporate their model into a broader framework that handles user input, AI voice processing (TTS, STT), and LLM inference. Of course, minimizing latency would be critical.

"It was a no-brainer to work with Daily from Day 1", Colucci notes. "They were the only vendor that could support us across the full tech stack, from audio and video transport to building an entirely new type of AI application."

Lemon Slice went from concept to functioning prototype in just a few days by leveraging Pipecat, the open-source framework developed by Daily for building multi-modal AI applications. The team seamlessly integrated their video model service into existing Pipecat workflows without ever worrying about low-level video transport details.

For their initial MVP and internal demos, Lemon Slice used Daily Prebuilt, Daily's turnkey interface for video conferencing. This let them quickly go end-to-end during early development and validate that real-time interactive video calls with their model were actually possible.

Before launching, they wanted to migrate the video call UI to their own website for greater control over user experience and layout design. What they expected would take at least two days was completed in under three hours. "It came together really fast. Integrating Daily React into our Next.js application was dead simple," says Colucci.

Built-in support for video call recording. Lemon Slice viewed recording as a core part of the user experience they wanted to build. "Talking to AI characters is fun and entertaining. A recording would allow our users to share the experience with others or replay later for themselves," Colucci explains. Using Daily, they were able to add that functionality with just a single line of code.

Built-in support for chat (text messaging). Lemon Slice wanted to support both audio and text input since a user might prefer one over the other depending on context. Daily and Pipecat's built-in support for messaging made this an easy extension of their original prototype.

Third-party integrations. Pipecat's extensive support for third-party providers for services like transcription and LLMs made it easy for Lemon Slice to experiment with different options. "This flexibility allowed us to evaluate various providers and identify the optimal combination of latency and performance for our unique needs," notes Colucci. "We could easily swap out components without rewriting our entire pipeline."

Lemon Slice and Daily worked hand-in-hand throughout the development process.

"We were initially excited to work with Daily because of their expertise in WebRTC and video streaming. However, Daily's API and SDKs were so easy to use, and their video infrastructure so robust, that we rarely had questions on that front."

Where Lemon Slice found the most value was in Daily's expertise with Pipecat and orchestrating complex multi-modal AI applications. Daily's engineers were responsive to questions in the Pipecat developer forums, eager to help, and played a pivotal role in helping Lemon Slice launch on an aggressive timeline.

The collaboration enabled Lemon Slice to focus on what they do best: pushing the boundaries of AI video generation, while Daily and Pipecat handled the complex infrastructure and architectural challenges of delivering a unique real-time AI experience at scale. Today, Lemon Slice's interactive AI characters are live and engaging users in real-time conversations - try it yourself here: lemonslice.com