Ondertitelbril: Live Captions on Smart Glasses

Together with Dare Health Innovation I joined the Hackathon Inclusive Technology, organised by UWV and Platform Inclusieve Technologie. One of my teammates went deaf later in life and already uses glasses with live captions. Those glasses send every conversation to the cloud to be transcribed. That made me curious whether it could be done locally. After the hackathon I started digging and found out that for Dutch, a local version for this type of smart glasses didn't exist yet. So I built one.

For deaf and hard of hearing people a big part of the workday gets lost. In the hallway, at the coffee machine, when a colleague drops by and during meetings. Those are exactly the moments where a phone screen gets in the way and glasses don't.

Speech recognition runs on the phone itself. The text goes to the glasses and the audio is gone within seconds. No data or audio ever leaves the app, so an organisation keeps full control over its own information. The app doesn't even ask for internet permission and works in airplane mode.

The app is native Android in Kotlin with Jetpack Compose. Speech recognition uses the Parakeet model via sherpa-onnx, with speaker recognition to filter out the wearer's own voice. A lot of time went into measuring instead of guessing. I benchmarked models, noise conditions and quantisation on Dutch test sentences. Quantising only the right layers of the model brought the word error rate on clean speech down from 5.3% to 1.8%, without losing speed on the phone.

What I like most: during a conversation the technology almost disappears. You look at the other person instead of at a screen, so communication feels natural right away. Ondertitelbril is a working prototype, not a finished product. Do you recognise this problem in your organisation? Send me a message and let's talk.