Talktree

Overview

A short engagement to fix the response speed of Talktree, a real-time AI English conversation app for junior learners. The product worked; it simply felt slow enough that children stopped talking to it.

Core Problem: Serial Stages Add Up

A voice pipeline runs speech recognition, then the language model, then speech synthesis. Run strictly in sequence, the user waits for the sum of all three — and no single stage looks bad enough in isolation to explain the delay. That is why this class of latency survives so long: every component passes its own benchmark.

The fix was structural rather than component-level — overlap the stages instead of shortening them. Recognition streams partial results, generation begins before the utterance completes, and synthesis starts on the first phrase rather than the finished sentence. What was a sum collapses toward the single longest stage.

Key Contributions

• Profiled the STT-LLM-TTS pipeline end to end — locating where time was actually spent rather than where it was assumed to be spent
• Achieved 70%+ reduction in end-to-end response time — measured on the full user-perceived path rather than per-component benchmarks
• Optimized real-time streaming — downstream stages starting on partial input instead of awaiting completion
• Introduced async processing and caching — removing from the live path work that never needed to be on it

Tech Stack

• AI Pipeline: STT, LLM, TTS Integration
• Optimization: Streaming, Async Processing, Caching
• Monitoring: Performance Profiling, Latency Analysis