Audio-Visual Looper in p5.js
I built a browser-based instrument this weekend. Twelve audio samples mapped to keyboard keys, each with a corresponding animation. p5.js for visuals, Tone.js for audio.
Basic structure:
Each sound-visual pair is a class that handles both the audio playback and the animation. The base class loads an audio buffer and manages a pool of buffer sources (more on why later). Subclasses implement onStart(), update(), and draw() for their specific animations—spirals, expanding squares, bouncing dots, etc.
The canvas is divided into a 3×4 grid. Each animation draws in its assigned cell.
The looper:
Press L to start recording. A metronome ticks at 120 BPM. Notes are stored as {key, progress, pitchIndex} where progress is a 0-1 value representing position within the 4-second loop. Each frame, we check if any recorded notes should play based on current loop position.
Undo (U) pops the last note off the array.
Pitch shifting:
Arrow keys move through a major scale. The pitch is converted to a playback rate using 2^(semitones/12). Shift+arrow transposes all recorded notes, not just future ones.
Problem encountered:
Tone.js kept throwing “time must be greater than or equal to last scheduled time” errors when multiple notes triggered simultaneously. The issue was with Tone.Player—it has internal scheduling state that conflicts when you try to change playback rates rapidly. Fix was to switch to Tone.BufferSource, which is stateless. You create a new source each time, set the rate, play it, dispose when done.
Other features:
- Press P to unpin animations from the grid. They float around and bounce off walls.
- Press O to export loop as JSON, I to import.
I built this with Claude. I described features, Claude wrote code, I tested and reported bugs. The Tone.js issue took a few iterations to solve.
https://editor.p5js.org/noahblack/full/EXg3eTqQX
Controls:
- A S D F G H J K W E R T — trigger sounds
- L — toggle looper
- U — undo last note
- ↑↓ — change pitch
- Shift + ↑↓ — transpose all notes
- P — toggle floating mode
- O — export loop
- I — import loop