Overview
Project details
Players complete missions straight through the phone camera in a PWA for iOS and Android: recognizing objects and paintings in the video stream, matching images, mimicking poses, connecting objects and completing voice-based tasks.
All computer vision runs on-device in the browser, so video never leaves the phone. Object and painting recognition uses OpenCV.js, pose matching uses MediaPipe and TensorFlow.js, and voice tasks use Deepgram speech recognition.
A no-code admin panel lets content managers configure missions, reference photos and poses, train recognition, generate voiceovers and manage geolocation on a map.
- Two large events and pilots in Seoul, Dubai and Berkeley
- 100% on-device vision, no video upload
- No-code mission editor for content teams