Experiments with MediaPipe
Exploring the overlaps between creativity and technology, experimenting with MediaPipe's machine-learning vision models.
Tools + Tech
Claude Code
MediaPipe
ReactI've been exploring interfaces you operate with your body instead of a mouse, keyboard, or screen — real-time experiments that read your hands, face, and breath through a plain webcam and turn them into sound and image.
They're built with MediaPipe, an open-source library from Google that runs machine-learning vision models live in the browser. From a normal camera feed it tracks the human body in real time — the 21 joints of each hand, a mesh of facial features, full-body pose — and reports where each of those points is, frame by frame, as a live stream of coordinates.
The pieces below pair that tracking with sound and visuals. They each took a couple of hours of experimentation with Claude Code. I'm curious and excited about the overlaps between creativity and technology — and how new tools are making it easier to experiment at these boundaries.