AirSynth — Play Piano With Your Hands
A gesture-driven piano in the browser. Right hand points at a chord on a radial reel; left hand holds a hand-shape (fist · peace · thumbs up · etc) to pick how the chord is played. Built on MediaPipe + Tone.js + Salamander piano + Tonal.js.
AirSynth — gesture-driven piano
A musical instrument you play with your hands in front of a webcam. Live at airsynth.carlfung.dev.
The idea
Pick a key (or a vibe preset). Raise your hands. The right hand points its index finger at a chord slice on a radial reel — that chord starts looping. The left hand holds a gesture — fist, peace, open hand, thumbs up, rock-on — and that gesture picks how the chord is played (block chord, arpeggio, walking bass, classical broken, etc).
No pinch, no click. Move your fingers, hear music.
What's in v1
Chord reel (right hand · 6 slices) — diatonic I, ii, iii, IV, V, vi in any key. Empty center = rest / silence. Toggle Triads ↔ 7ths in the header to swap the entire palette to Cmaj7, Dm7, Em7, Fmaj7, G7, Am7-style voicings with proper octave voicing.
Pattern reel (left hand · 8 gestures):
| Gesture | Pattern | Sound |
|---|---|---|
| 👊 Fist | Block | Full chord on beats 1 & 3 |
| 🖐 Open hand | Stride | (1,3) dyad alternating with low 5 — pop |
| ☝️ Index | Cascade | Chord stab then falling arpeggio + low 5 |
| ✌️ Peace | Up Arp | 1 3 5 1↑ rising |
| 🤟 Three | Alberti | 1 5 3 5 classical broken |
| 👍 Thumbs up | Hit | Chord stab + 5 3 melodic fill |
| 🤘 Rock-on | Bounce | 1 5 1↑ 5 bouncy bass |
| 🤙 Hang loose | Down Arp | 1↑ 5 3 1 falling |
Vibe presets: Lo-fi (D dorian), Anthem (E major), Sad Disney (F major), Cyberpunk (F♯ minor), Sunrise (G mixolydian), K-Drama (F major + 7ths + Cascade by default — full Fmaj7-Gm7-Am7-Bbmaj7-C7-Dm7 OST palette).
Drone fill: when both hands leave the camera, a soft held tonic in the current key fills the silence (toggleable).
How it works
A Tone.js transport runs at 110 BPM. A Tone.Loop fires every 8th note. Every step looks up the current chord (set by the right hand) and the current pattern (set by the left hand's gesture), maps the pattern's degree tokens (0 = full chord, 1/3/5 = chord tones, 8 = octave up, -1/-3/-5 = octave down) to actual pitches, and triggers the Salamander Grand Piano samples through a reverb send.
Chord and scale harmony comes from Tonal.js. MediaPipe HandLandmarker provides 21 landmarks per hand at 30–60 FPS via WASM. Gesture detection is pure geometry — for each finger, compare tip-y vs PIP-y; for thumb, distance from tip to index-MCP. No ML classifier.
A canvas particle visualizer reacts to the audio analyser's FFT bins — each hand emits particles in its own color (purple for left, cyan for right), with pinch causing a burst.
Tech Stack
| Layer | Tech |
|---|---|
| Framework | Next.js 16 (App Router, React 19, Turbopack) |
| Audio | Tone.js · Salamander Grand Piano (CDN samples) |
| Theory | Tonal.js (scales, chords, voicing) |
| Vision | @mediapipe/tasks-vision (HandLandmarker, GPU delegate) |
| Animation | framer-motion, canvas particle visualizer |
| Styling | Tailwind CSS 4 |
| Hosting | Vercel + carlfung.dev wildcard subdomain |
Try it
- Open airsynth.carlfung.dev on a desktop browser.
- Click ✋ Enable hand tracking (bottom-right) and grant camera permission.
- Raise your right hand with index finger extended — point at a chord slice. The loop starts.
- While the chord plays, make a different shape with your left hand — feel the pattern morph from stride to arpeggio to chord-stab in real time.
- Click K-Drama in the header for the moody ballad palette.
Built as the second project in the gesture-engine lineage — same MediaPipe + radial-reel foundation, completely different application.