Portable Piano

Co-developed a projection-based virtual instrument that tracks fingertips in 3D and turns any flat surface into a responsive piano, xylophone, or DJ pad.

Jan 2026 - Mar 2026

Technologies

PythonC++PygameLeap MotionLattePanda MuGesture TrackingReal-Time SystemsAudio SynthesisJSONUSBHDMI3D Printing

Completed Portable Piano system projecting a colorful xylophone interface onto a table from its custom black enclosure — The completed portable instrument projecting its touch-free xylophone interface onto a tabletop. Select the image to view it at full size.

Front view of the assembled black 3D-printed Portable Piano enclosure with projector and Leap Motion sensor — Front view of the custom 3D-printed frame that aligns the projector and Leap Motion sensor.

Angled view of the assembled Portable Piano enclosure and integrated electronics — The finished enclosure packages the projection, sensing, and computing hardware into one movable unit.

Portable Piano system projecting its eight-note rainbow xylophone interface onto a table — Full xylophone-mode setup during integrated testing.

Close view of the projected rainbow xylophone controls and tracking hardware — The xylophone interface maps eight projected bars to individual notes.

Portable Piano system projecting its DJ sample-pad interface onto a table — DJ mode expands the prototype beyond tonal instruments with a 3×3 sample board.

Close view of the projected DJ board with nine labeled sample pads and a hand-tracking graphic — Close-up of the projected DJ pad and live hand-tracking feedback.

A user selecting an instrument from the projected Portable Piano home screen — Touch-free menu interaction during a user demonstration.

A user holding a hand above the projected Portable Piano instrument-selection menu — The projected menu lets players switch instruments without a separate display or controller.

Side view of a user moving a hand over the projected instrument menu while fingertip tracking is displayed — Live fingertip visualization helped the team validate projected control alignment.

Overview

Co-developed a portable, projection-based musical instrument with Welldone Matanga and Justin Ansell for Northwestern University's CE 347 Microprocessor System Projects course. The system projects an interactive interface onto a flat surface and lets a player trigger sounds by tapping the projected controls—without physical keys or a touchscreen.

The final architecture combines a Leap Motion Controller, a LattePanda Mu single-board computer, and a compact projector with an integrated speaker. A compiled C++ program reads 3D fingertip coordinates from the Leap Motion SDK and streams them as JSON to a Python application. The software calibrates the sensor's horizontal and depth coordinates to the projected interface, uses a 15 mm tap threshold and per-key debounce timing to reject accidental triggers, and keeps the interaction responsive.

Players can switch among three complete instruments: a one-octave piano spanning C4 through B4, a colorful xylophone, and a 3×3 DJ sample board. Pygame renders the projected interface at 30 frames per second and manages stereo audio, including synthesized tonal notes with ADSR envelopes and pre-cached samples for low-latency playback.

We packaged the projector, computer, and tracking sensor in a custom 3D-printed frame, then iteratively tuned sensor placement and coordinate mapping so the physical tap region aligned with the projected controls. The finished prototype demonstrated real-time multi-finger tracking, mode selection, audible feedback, and portable operation, with all interaction data processed locally on the device.

Project Context

This three-person CE 347 project combined 3D hand tracking, cross-language software, real-time graphics and audio, and mechanical prototyping. The goal was to create a portable musical experience that could transform an ordinary flat surface into several playable instruments.

Key Features

Leap Motion 3D fingertip tracking with locally processed interaction data
C++ sensor bridge streaming structured JSON coordinates to Python
Calibrated horizontal and depth mapping from physical space to projected controls
15 mm tap detection threshold with 80 ms per-key debounce
Three playable modes: one-octave piano, xylophone, and 3×3 DJ sample board
Pygame interface rendered at 30 FPS with stereo sound output
ADSR-based tonal synthesis and pre-cached samples for responsive playback
Custom 3D-printed frame aligning the projector, computer, and tracking sensor

Challenges

Installing and connecting the Leap Motion SDK across the C++ and Python components
Mapping 3D sensor coordinates accurately onto a 2D projected interface
Reducing false taps and repeated notes without making the instrument feel sluggish
Keeping graphics, tracking, and audio responsive on the embedded computer
Maintaining alignment while integrating the electronics into a portable enclosure

Results

Delivered a functional, hardware-integrated projection instrument
Demonstrated real-time fingertip tracking, tap detection, visual feedback, and sound generation
Implemented and tested piano, xylophone, and DJ performance modes
Reduced perceived audio latency through pre-generated notes and pre-cached samples
Completed a portable 3D-printed assembly and calibrated it for tabletop demonstrations