Dec. 14, 2019
Narain, J.*, Johnson, K.T.*, Picard, R.W., Maes, P. "Zero-Shot Transfer Learning to Enhance Communication for Minimally Verbal Individuals with Autism using Naturalistic Data," NeurIPS Workshop on AI for Social Good, December 2019. (*Co-first authors/Equal contribution)
We applied zero-shot transfer learning to classify vocalizations from a nonverbal individual with autism using captured audio. Data were recorded in natural environments using a small wearable camera and sparsely labeled in real-time with a custom-built open-source app. We then trained LSTM models on VGGish audio embeddings from the generic AudioSet database for three categories of vocalizations: laughter, negative affect, and self-soothing sounds. We applied these models to the unique audio recordings of a young autistic boy with no spoken words. The models identified laughter and negative affect with 70% and 69% accuracy, respectively, but classification of the self-soothing sounds produced accuracies around chance. This work highlights both the need and potential for specialized, naturalistic databases and novel computational methods to enhance translational communication technologies in underserved populations.