XR Auto Service

The app

XR Auto Service shows a technician repair procedures on a Meta Quest 3, one step at a time. Each step has reference photos and safety warnings, and the technician takes a confirmation photo when it’s done. They can ask questions out loud about the step they’re on without putting their tools down.

I built it end to end, from the headset app to the search behind the assistant. The app starts a job by scanning the VIN, using a burst of frames and the VIN’s check digit to correct misreads. It works offline and syncs when the headset reconnects.

Answering questions

The assistant answers from a knowledge graph I built over licensed service data. The graph breaks each procedure into its steps and links it to the vehicles it applies to. Every search is filtered to the year, model and engine of the car on the lift.

Search scores passages by keyword match, then walks the graph from the best hits to the steps and procedures they link to. The answer is read aloud with the source it came from.

The data

Before building search, I wrote a toolkit that pulls only service data the company has the rights to use. It draws from several sources, checks each one’s license, and writes a provenance record for every document. Each answer cites the document it came from.

Testing

Every change to search runs in continuous integration against questions written by repair experts. Each question comes with a ranked list of the passages the expert expected, and a model judge also grades the answers.

I also tested the test. Weakening the ranker on purpose lowered the score, so the test does see the ranker. Several standard ranking methods scored about the same.