Lancelot Blanchard, Perry Naseck, Stephen Brade, Kimaya Lecamwasam, Jordan Rudess, Cheng-Zhi Anna Huang, & Joseph Paradiso. (2025). The jam_bot, a Real-Time System for Collaborative Free Improvisation with Music Language Models. Proceedings of the 26th International Society for Music Information Retrieval Conference, 755–762. https://doi.org/10.5281/zenodo.17811478

Abstract

In order to design a Generative AI system that could improvise on stage with GRAMMY-winning keyboard virtuoso Jordan Rudess, we developed the “JAM_BOT”, a real-time performance system that could match his eclectic improvisational aesthetics. We debuted the JAM_BOT at a high-stakes sold-out concert to critical acclaim, realizing a series of virtuosic tightly-coupled Human-AI free improvisations in varying musical styles. Reflecting on our year-long collaboration, we summarize learnings for AI researchers and musicians on the adaptations needed to turn state-of-the-art symbolic music Language Models (LMs) into JAM_BOTS and the engineering required to make them performance-ready. We focus on three aspects: (1) enabling JAM_BOTS to take on different musical roles by adapting music LMs to employ different interaction strategies by modifying the context and conditioning signals; (2) describing how Rudess intentionally structures his improvisation in order to finetune JAM_BOTS to match the style needed for each piece; and (3) showing the optimizations needed to run music LMs in real-time and embed them in a low-latency multi-threaded system that listens, prompts, and schedules model generations seamlessly. We hope these insights enable more musician-AI symbiotic virtuosity.