AirSynth — gesture-driven piano & guitar

A musical instrument you play with your hands in front of a webcam. Live at airsynth.carlfung.dev.

The idea

Pick a key, a vibe preset, or load a song. Raise your hands. The right hand points its index finger at a chord on a radial reel — that chord starts looping. The left hand makes a shape — fist, peace, open hand, thumbs up, rock-on — and the shape picks how the chord plays (block stab, arpeggio, walking bass, classical broken, etc).

No pinch, no click. Move your fingers, hear music.

The meta image

The whole project is exactly that picture: two hands, one pointing into harmony, one shaping rhythm, both half-organic and half-tracked. The site looks like a music recital and reads like a computer vision demo.

Song mode

Load a song and AirSynth swaps from a generic 7-chord diatonic reel to the song's actual palette — Someone Like You gives you A · E · F♯m · D · C♯m in first-appearance order. Time-synced lyrics fetch from LRClib and float across the bottom of the screen with chord markers above the right words. Each lyric line is a phrase in the song data, so chord changes land on syllables, not bars.

Songs shipped:

New songs come in via a Claude Code skill — drop screenshots of an Ultimate Guitar printable tab and the skill produces a phrase-aligned Song entry with degree mapping, slash-chord handling, and the right default pattern for the song's feel.

Practice mode

The practice-mode controls live in the header whenever a song is loaded:

• Tempo slider (50%–150%) scales the song's authored BPM. Backing tracks (when shipped) stretch in lock-step via playbackRate, so the band stays glued to the live chord loop even when you slow Marry You down to 70% to learn the changes.
• Section loop wraps the cursor inside the current section instead of advancing — run a chorus until the chord shapes are automatic.
• Transpose ±6 semitones shifts the active song's chord palette + engine root live so singers can drop a song into their range without rewriting the source data. The header readout shows original → effective key with the offset chip. Sharps stay sharps (F♯ / C♯ / G♯) for chord-chart consistency.
• Clickable section navigation — the chord ribbon's top row shows every section in the song's structure as a chip; click any verse / bridge / chorus / break to jump the cursor instantly.
• Two-hand commands (hold ~200ms with both hands in the same shape):
  • ✊ ✊ both fists → toggle pause
  • 👍 👍 both thumbs → jump to next section
  • ☝️ ☝️ both index fingers → jump to previous section
  • A bottom-center progress bar fills while the gesture is held; single-hand pattern selection is suppressed during the hold.
• Chord diagrams pop up below the chord ribbon whenever a chord is held.
  • Piano mode — universal mini-keyboard with chord-tone pitch classes lit cyan and the root highlighted amber. Derives directly from the resolved voicing, so it works for diatonic chords, 7ths, slash chords, and altered extensions (♭9, ♯11, etc.).
  • Guitar mode — open-position fretboard with a barre rectangle, finger dots, and open/muted markers. Hardcoded shapes cover the standard pop vocabulary plus the slash chords used by shipped songs; anything outside the table falls back to the piano diagram.

Audio engine

Piano: smplr's SplendidGrandPiano — Steinway D, four velocity layers — replacing the old synth. There's a 10-flavor picker: Grand · Kawai · Steinway B (1895) · Upright Knight · Upright Yamaha · Bright · Honky-Tonk · Rhodes · Wurlitzer · CP80. Mix of smplr's SplendidGrandPiano, Versilian VCSL, ElectricPiano, and Soundfont.

Guitar: real sampled acoustic guitar instead of PluckSynth.

Plate reverb (Dattorro) wired in parallel dry/wet routing — at 0% reverb the dry signal bypasses the processor entirely, so it doesn't get colored by the input filter and allpass network. The 'hollow' feeling at wet=0 with algorithmic reverbs vanishes. Sustain pedal (CC64) fires on chord changes for natural ring-out.

Dual AudioContext: Tone keeps its own (a standardized-audio-context wrapper) while smplr gets a fresh native AudioContext. Sharing one breaks smplr's AudioWorkletNode typecheck.

Pattern library — 16 piano + 8 guitar

Each pattern describes what each 8th note plays. Tokens are scale steps from the chord root when a scale is set (always true in song mode), so the same phrase adapts naturally to every chord without manual transposition.

Piano: Stride · Block · Arpeggio · Alberti · Slow Stride · Wave · Down Arp · Roll · Folk · Lift · Stairs · Pop · R&B · Hold · Half Push · Float.

Guitar: Strum · Stab · Travis · Pluck · Classical · Country · Alt Bass · Falling.

Most piano patterns carry a bassDrone — octave-doubled low root, both -1 and -2 octaves under the chord, held for the whole bar. A sustained low bass anchors the rhythmic figure on top.

Right-hand fillers — 10 + Random

A fill plays the last four eighth-notes of each bar. All resolve relative to the tonic of the key, not the next chord — feels in-key rather than mechanical.

Tension tokens (♭9, 9, ♯11, 13) resolve via Tonal.js against the active chord, not just the scale.

Vibe presets (free-play mode)

Lo-fi (D dorian) · Anthem (E major) · Sad Disney (F major) · Cyberpunk (F♯ minor) · Sunrise (G mixolydian) · K-Drama (F major + 7ths + Roll by default — full Fmaj7-Gm7-Am7-B♭maj7-C7-Dm7 OST palette).

How it works

A Tone.js transport runs at the song's BPM (scaled by the tempo slider). A Tone.Loop fires every 8th note. Every step looks up the current chord (set by the right hand or the song cursor) and the current pattern (set by the left hand's gesture or chosen via keyboard), maps the pattern's degree tokens to actual pitches via the active scale, and triggers the smplr piano samples through the parallel reverb send.

Chord and scale harmony comes from Tonal.js. MediaPipe HandLandmarker provides 21 landmarks per hand at 30–60 FPS via WASM. Single-hand gesture detection is pure geometry — for each finger, compare tip-y vs PIP-y; for thumb, distance from tip to index-MCP. Two-hand commands sustain the same shape for ~200ms across both hands then enter a 1s cooldown.

Tech Stack

Try it

1. Open airsynth.carlfung.dev on a desktop browser.
2. Click ✋ Enable hand tracking (bottom-right) and grant camera permission.
3. Pick a song from the top bar, or stay in free-play mode.
4. Raise your right hand with index finger extended — point at a chord. The loop starts.
5. While the chord plays, make a different shape with your left hand — feel the pattern morph from stride to arpeggio to chord-stab in real time.
6. Drop the tempo to 70%, flip on Loop, and run a verse until it sticks. Hit the Key ± buttons to shift into your singing range.
7. Try the two-hand commands — both 👍 to jump to the next section, both ☝️ to go back, both ✊ to pause.
8. Click K-Drama in the header for the moody ballad palette, or pick a piano flavor to swap the timbre.

Built as the second project in the gesture-engine lineage — same MediaPipe + radial-reel foundation, completely different application.