Joico Soft Skill Training

Joico wanted a way to train employees on client conversations: both onboarding new staff and preparing experienced staff for difficult situations, like a client who is unhappy or asking questions that are easy to accidentally answer with too much technical jargon. The goal was to let people practice de-escalating and explaining things clearly, in a safe environment, as often as they want.

The technical ask made it tricky: it had to run in the browser on a Meta Quest 2 with working VR controllers, and the conversation had to feel responsive even though every reply travels through three different AI services (speech-to-text, a language model, and text-to-speech).

The trainee plays a hairdresser doing an intake consultation with Barbera, an AI-driven virtual client. The whole loop is voice-based and hands-free: you just talk. The app detects when you start and stop speaking (no push-to-talk button), transcribes your words with Whisper, has ChatGPT respond in character as the client, and speaks the reply back through ElevenLabs. Barbera reacts visually too: she has eight emotional states (happy, angry, insecure, confused, and so on) that the AI picks based on how the conversation is going, each with a talking and silent variant so she appears to speak.

The voice pipeline builds on a base I developed for an earlier training sim (Cosis), evolved and expanded here. Latency was the main battle, since every turn chains three AI services. I built a Node.js backend (Express, running on an EC2 instance) with one job: return the audio reply as fast as possible. Instead of the headset calling each AI service separately, it sends the recording once and the backend handles the whole chain in one round trip. That is both faster than doing it on the frontend and more secure, since the API keys never leave the server. On top of that, the ElevenLabs audio starts playing while it is still being generated instead of waiting for the full file. I also had to guard against race conditions: if the user interrupts or moves on, any late responses from a previous turn are detected and thrown away instead of playing over the new ones.

A few other things I'm proud of in there:

• Whisper sometimes mishears domain terms or hallucinates text on silence, so there is a correction table and a filter that cleans up known bad transcriptions before they reach the AI.
• All the AI prompts, the client's persona, tutorial scripts, and every line of UI text (Dutch and English) load at runtime from Google Sheets. I set the sheets up with the base prompts, and we guided Joico in editing them, so they could tune Barbera's behavior and the feedback rubric themselves without us redeploying the app. It saved everyone time.
• The same build runs on desktop, mobile, and in VR on the Quest 2, with controller pointing, teleporting, and smooth movement in VR and a drag-to-look camera everywhere else.

When the conversation ends, ChatGPT reviews the full transcript and the trainee gets scored feedback on four competencies: communication skills, clarity, solution-oriented thinking, and technical knowledge.

I handled the development; a colleague at Enversed created the art, including Barbera's emotion illustrations and the illustrated 360 salon that forms the environment.

A browser-based VR training app in which Joico employees practice a full client consultation by actually talking, out loud, with an AI client who listens, responds in character, shows emotion, and can escalate if handled poorly. Every session ends with a personal feedback report scored on four competencies. The app works in Dutch and English, runs on desktop, mobile, and Meta Quest 2 from a single build, and was delivered to Joico together with a set of Quest 2 headsets as part of a broader learning program, alongside a webinar we produced for them.

The training is live and still being used by Joico. For me, the big step in this project was the backend: it was not my first time working with LLMs and voice AI, but it was the first time I optimized the whole pipeline by moving it server-side. Building the Node.js service that chains transcription, the language model, and text-to-speech in a single round trip made the conversation noticeably faster and kept all API keys off the headset, and it is an approach I have carried into projects since.