In January 2024, several members of the MIT Media Lab traveled to Fribourg, Switzerland, to conduct a workshop on sensing, interactions, and augmentation. This MISTI-funded workshop was jointly led by PhD candidates Patrick Chwalek of the Responsive Environments group, and Sailin Zhong from the Human-IST Institute at the University of Fribourg. The event featured a series of presentations from members of the Responsive Environments and Fluid Interfaces groups, and welcomed participation from PhD students affiliated with the Smart Living Lab at the University of Fribourg and École Polytechnique Fédérale de Lausanne (EPFL). Bringing together students, researchers, and professors from diverse disciplines, the workshop focused on interdisciplinary collaboration for projects aimed at enhancing research for the built environment.

Keynote Speakers

• Prof. Joseph A. Paradiso (Associate Academic Head and Director of the Responsive Environments group, MIT Media Lab)
• Prof. Denis Lalanne (Professor in the Department of Informatics and Director of the Human-IST Institute)
• Prof. Dusan Licina (Associate Professor at EPFL and Director of the Human-Oriented Built Environment Lab (HOBEL))

Organizational Committee

• Patrick Chwalek (Responsive Environments, MIT Media Lab)
• Sailin Zhong (Human-IST Institute, University of Fribourg)
  
• Mengying (Cathy) Fang (Fluid Interfaces, MIT Media Lab)
• Sam Chin (Responsive Environments, MIT Media Lab)
• Abhinandan Jain (Fluid Interfaces, MIT Media Lab)
• Yong-Joon Thoo (Human-IST Institute, University of Fribourg)
• Kanaha Shoji (Civil and Environmental Engineering, EPFL)

Highlights

Day 1: Talks

Professor Joe Paradiso initiated the workshop with an insightful overview of the MIT Media Lab, touching upon both historical and current projects. This was followed by a session where attendees introduced themselves, sharing their diverse backgrounds and areas of interest, which transitioned into collaborative brainstorming discussions through lunch.

Post-lunch, the attendees had an informative tour led by members of the Smart Living Lab and the Human-IST Institute, showcasing a variety of research laboratories and facilities within the vicinity. Highlights included an exhibition of innovative tools developed by the Building2050 group, designed for the monitoring and digital manipulation of physical spaces, and a visit to the Human-Oriented Built Environment Lab where extensive environmental sensing research is conducted within their specialized chambers.

The afternoon agenda featured Patrick Chwalek and Sailin Zhong, who delivered an engaging presentation on crafting robust hardware systems and mitigating deployment risks in real-world applications, supplemented by case studies drawn from their previous experiences, such as Arctic sensor deployment and collaborations with both academia and industry. Patrick also introduced the AirSpecs project, an intelligent eyewear system conceived to monitor environmental and physiological parameters in realistic settings. Concurrently, Sailin shared insights from her comprehensive research on environmental conditions affecting building occupants. Cathy Fang then articulated her concept for novel data interaction paradigms, utilizing advanced language models to facilitate a more intuitive user experience in data queries.

Subsequently, Abhi Jain discussed the enigmatic nature of sleep and dreams, approached from the perspectives of neuroscience and psychology. He explained how foundational research in these domains is propelling forward hardware development, thereby enabling groundbreaking studies. The afternoon sessions were rounded out with Sam Chin and Yong-Joon Thoo's exposition on sensory augmentation, including its various interpretations and practical applications in rehabilitation.

The day concluded with discussions, reflecting on the day's presentations, and exploring potential synergies with the research interests of the attendees.

Day 2: Talks

On the second day, we started with a lecture from Professor Dusan Licina, Associate Professor at EPFL and Director of the Human-Oriented Built Environment Lab (HOBEL). Professor Licina discussed the contemporary landscape of environmental sensing and emphasized the significance of embedding continuous environmental sensing within our environment. We then heard from Abhi on his insights into designing interfaces to enhance sleep quality. Abhi delineated his involvement with sleep-focused research in the Fluid Interfaces group and his contributions to hardware development aimed at sleep improvement.

Post-lunch, which was marked by dynamic group discussions on research topics, we delved into further presentations from the organizational committee and Professor Denis Lalanne, a professor in the Department of Informatics and the director of the Human-IST Institute. Professor Lalanne delivered an illuminating presentation on the Human-IST Institute's endeavors and expounded on innovative human-computer interaction techniques. His discussion traversed various modalities of communication and dissected the intricacies of multi-modal fusion across data, feature, and decision levels.

The afternoon sessions featured Sam and Yong-Joon, who presented on the intersection of multi-modal rehabilitation and gamification, with a particular focus on employing gamification as a motivational tool within rehabilitative systems. Sailin and Cathy then took the stage to discuss the nexus of data, interfaces, and their end-users. Sailin shared her expertise in user interface design and her approach to presenting environmental sensing data to a diverse audience. Cathy, delving deeper, explored the application of Large Language Model (LLM) Agent Systems in the intelligent querying of environmental and physiological data, highlighting how these agents can facilitate more intuitive user interactions.

Kanaha Shoji concluded the series of talks with an engaging session on walkability, examining the interplay between walking behavior, social cohesion, and the built environment.

Day 3-4: Hands-on Workshop

During the third and fourth days, we transitioned from the Smart Living Lab to laboratory facilities at the University of Fribourg. Our discussions began with an overview of the hardware devices developed by researchers from the MIT Media Lab. Patrick led the session by delving into the AirSpecs project, along with the suite of software tools designed to assist researchers from various fields in utilizing the system for scientific inquiry. Sam followed with her multi-modal haptic device, elaborating on the collaboration with Yong-Joon to adapt the device for virtual reality applications.

Abhi later presented his cutting-edge vestibular stimulation device and a smart sleep mask equipped to measure a range of physiological parameters and induce sleep optimization through stimulation. The subsequent hours, and the majority of the fourth day, were dedicated to in-depth research discussions and demonstrations.

We drew the workshop to a close by recapping the myriad discussions and outlining the fresh research avenues that had emerged. In addition, Sam Chin left one of her devices with the group, and Patrick contributed ten AirSpecs devices in collaboration with Sailin and the Human-IST Institute. These tools will support ongoing collaborative projects, including virtual reality applications, controlled environmental studies within HOBEL, and extended environmental research in real-world settings.

In the evening, a majority of the workshop participants had the opportunity to explore the Swiss Museum for Electronic Music Instruments (SMEM), home to one of the world's most extensive collections of synthesizers, organs, keyboards, and studio equipment. Professor Paradiso graced the venue with a keynote address entitled “A Synth Born Swiss Rig,” in which he shared his lifelong enthusiasm for synthesizers and electronic music.

Day 5: EPFL

On the fifth day, the organizing committee, accompanied by Professor Joseph Paradiso and several workshop participants, went to the École Polytechnique Fédérale de Lausanne (EPFL). The day at EPFL started with an engaging lunch with the Computer-Human Interaction in Learning and Instruction (CHILI) lab, where we discussed hardware-centric research for learning applications. Subsequently, we toured EPFL's diverse maker spaces and prototyping labs, including the Discovery Learning Labs. This facility boasts an extensive collection of tools and equipment, serving as a vibrant hub for student-driven projects and research objectives. The visit concluded with a visit to the "Cyber Physical: Architecture in Real Time" exhibition, which featured four dynamic installations artfully converting virtual data into palpable, interactive kinetic sculptures.

Concluding Thoughts

We’d like to thank everyone from the organizational committee for working tirelessly in making the workshop possible. We’d also like to thank all the attendees and professors who joined us throughout the workshop, allowing for energized discussions and intriguing cross-disciplinary research directions. Finally, thank you MISTI for the financial support in making this workshop possible.