Project Abstract/Summary
Music is an integral part of our lives and there is a lot of evidence that giving students opportunities to instrumental music education is important, because of its advantages for developing cognitive, social, and physical skills, and its benefits for mental health. Learning to play a musical instrument, however, constitutes a lengthy and complex process. Private music instruction is costly, afforded only by a privileged part of the world population. The recently developed tools in AI and online education provide an opportunity to create new technology for music education which can reach a very diverse student population. Little is known, however, on how to do intelligent online teaching of musical instruments, and how to keep students engaged through individualized instructions in a discipline that involves motor instructions. In this project, a team with expertise in violin pedagogy and visual and auditory signal analysis aims to develop an AI platform to analyze a student’s playing during individual practice time. Video and audio input recorded by cameras and microphones on handheld devices are used to provide feedback on posture, bow movements, sound quality and selection of curricular materials. This project advances our understanding of the learning and teaching of motor tasks, and provides new insights for computational perception of human movements from multi-modal data. It supports digital humanities in the field of music; from a humanity perspective, it democratizes music education, and provides the benefits of instrumental instruction to a large and more diverse population.
Inspired by cognitive studies on educational feedback intervention, this project develops a prototype AI system that acts as a virtual assistance to violin students, teachers, and supervising parents, and like a human teacher it also provides explanations. This is accomplished through two unique components: 1) a feedback system that provides advice based on visual and auditory analysis and causal relationships of errors, i.e., why a movement error caused non-ideal sound; and 2) assignment of educational materials specific to the students based on the perceptual evaluation and a corpus of recorded materials. The technical work involves the collection of multi-modal data from violin players, the development of machine learning algorithms for analyzing the players’ performance, the recording and curation of a corpus of educational music pieces from videos and created sheet music, the design of a gamified user interface, and feedback instructions. The combination of developed AI software, data collected through observation of violin students, and the music corpuses to be digitized, recorded, and categorized constitute a major step forward into the 21st century for the field of music pedagogy and innovation of tools for studying human motor learning through perception.
This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.
Principal Investigator
Cornelia Fermuller – University of Maryland, College Park located in COLLEGE PARK, MD
Co-Principal Investigators
Shihab Shamma, Irina Muresanu
Funders
Funding Amount
$900,000.00
Project Start Date
09/15/2023
Project End Date
08/31/2026
Will the project remain active for the next two years?
The project has more than two years remaining
Source: National Science Foundation
Please be advised that recent changes in federal funding schemes may have impacted the project’s scope and status.
Updated: April, 2025