Speaking Tutor App
English speaking tutor app - a real-time voice AI learning engine with nine modules, built end-to-end under contract
Overview
An English speaking tutor app delivered under contract for Oizi. The system listens to a learner’s speech in real time, scores it, and returns coaching. Sole ownership of the AI engine and backend: voice pipeline, module engine, assessment system, and the integration contract with the partner learning platform.
Core Problem: Real-Time Voice at Sustainable Cost
A voice tutor must respond within a beat, sound natural, and stay cheap enough per session to sustain a business. These three constraints pull against one another. The architecture resolves the tension by spending LLM calls only where they buy something:
| Tier | Where applied | LLM cost |
|---|---|---|
| Seed | Guidance and prompts at fixed steps | none (pre-generated) |
| Slot | Feedback on correct/incorrect answers | none (template variable substitution) |
| LLM | Applied coaching, free role-play, presentation critique | live calls |
Most modules run at zero LLM calls per session; only the free-conversation modules incur per-turn cost. Perceived quality holds while unit cost drops sharply.
Key Contributions
- Designed and built the real-time voice pipeline (STT → LLM → TTS) — including comparative evaluation of speech engines and migration of the TTS stack as stronger options emerged
- Built a nine-module learning engine — study, vocabulary, shadowing, fill-in-the-blank, sentence production, scripted role-play, free role-play, and one-minute presentation, unified under a shared six-stage state machine
- Designed the speech assessment system — pronunciation, fluency, grammar, pragmatics, and speech output across five axes plus internal metrics; held to a principle of reporting raw units (%, WPM, errors/sentence) rather than a synthesized 0-100 score, so learners can locate what to fix
- Implemented persona-constrained real-time role-play — free conversation held inside scenario and role via guardrails, with tracking of target-expression usage
- Reconciled a single authoritative spec — three conflicting source documents (planning doc, scenario doc, data sample) merged into one implementation standard, with every conflict resolved and recorded as a numbered decision
- Defined the partner-platform integration contract — a content-receiving contract consumed at session entry, paired with a bundled standalone path enabling demo and QA before integration
How It Was Built
Nearly all implementation was carried out by coding agents; the human role was direction and intent, contact with reality (client feedback, real-device behavior), and the coordination that keeps delegation safe (verification, permissions, arbitration between conflicting decisions). The full record of this engagement — commits, human interventions, discarded approaches — was later analyzed end to end and became the empirical foundation for UyooPack, an infrastructure product for keeping an organization’s standards and its actual output in sync.
Tech Stack
- AI: Whisper (STT), LLM orchestration, Google Chirp 3 HD (TTS), pronunciation assessment
- Backend: Node.js, TypeScript, monorepo
- Workflow: Claude Code, Basecamp, GitHub