Gestural Media Framework



We are all equipped with two extremely expressive instruments for performance: the body and the voice. By using computer systems to sense and analyze human movement and voices, artists can take advantage of technology to augment the body's communicative powers. However, the sophistication, emotional content, and variety of expression possible through original physical channels is often not captured by the technologies used for analyzing them, and thus cannot be intuitively transferred from body to digital media. To address these issues, we are developing systems that use machine learning to map continuous input data, whether of gesture or voice, to a space of expressive, qualitative parameters. We are also developing a new framework for expressive performance augmentation, allowing users to create clear, intuitive, and comprehensible mappings by using high-level qualitative movement descriptions, rather than low-level descriptions of sensor data streams.