Conversatio: an AI interview simulator
Practice behavioral and technical interviews with an AI and get realtime feedback.
Coversatio is an AI interview simulator I'm currently building together with Carl Liu. The goal is to let anyone practice interviews with an AI and get realtime, structured feedback. You can try a behavioral interview yourself.
The idea
Interview practice is hard to get on demand. You need someone to sit across from you, ask good questions, and give honest feedback. Coversatio simulates a realistic conversation with an AI interviewer that adapts to your answers and tells you what to work on.
Built with evals first
This was the first time I built with evals and benchmarks in mind from the start. I wrote a set of evals that tested different models on two axes: how well they simulated a real interview, and how fast they were. I chose our model lineup from those results. Right now the interview agent and manager run on Gemma-32b.
Making the model stop being helpful
By default, an LLM wants to be a helpful assistant. An interviewer isn't supposed to be helpful, it's supposed to interview, push, and hold the line even when the user struggles. Getting the model to behave like an interview agent instead of an assistant turned out to be one of the real challenges.
We built a “manager LLM” that reviews the “interviewer LLM”, can steer it in specific directions, and manages timing. It keeps the interview on track. We're still working on bringing latency down.
Voice agents are tricky
From building Hablo.pro, I learned that voice agents can get expensive quickly. I saw this project as an opportunity to experiment with a custom STT (with VAD) > LLM > TTS pipeline. The evals were key in choosing the right models for the job.
I had to also debug a difficult issue with UDP. The connection was often finicky. I then found out, that in a university WiFi a direct connection between client and server often cannot be directly established. This makes the initial connection take time. I added a CoTURN VM, that relays the communication between client and server, and makes sure that there is *always* an available path.
Try it
The behavioral interview demo is live at coversatio.com/demo/behavioral
What I learned
- My first time building with evals and benchmarks driving the model choices.
- How o bend an LLM away from “helpful assistant” into a purpose-built interview agent.
- Using a manager LLM to supervise and steer another LLM, and to manage timing. There is much that can be done with multiple LLMs working together.
- Some of the realtime latency, cost, and complexity trade-offs of running conversational models in production.