A multimodal music recommender with physiological signals

Abstract

Music is a healthy, inexpensive and popular medium that is listened to by people. It is also reported as being effective in modulating people’s emotions. Therefore, it is promising to use music to help students adjust their emotions in their daily life, which is intensely occupied with challenging learning tasks. Especially in today’s “big data” era, the quantity of available music resources is enormous and continue increasing, which provide a large variety of music to satisfy different needs of emotion optimization. However, it also brings a great challenge for university students to discover suitable music in massive music resources. In addition, despite using the physiological signal to recognize human emotion has been researched in many years, the effectiveness of considering the physiological signal as emotion features into a music recommendation system is yet to be explored. To fill the research gaps and attain a higher accuracy of the music recommendation system to personalizing music playlists for students, this research describes a multimodal hybrid recommendation approach that combines music features and student’s information, particularly physiological signals. To achieve the aims of the research, a mixed research method is applied to gather qualitative and quantitative data through questionnaire, user experiment and a short interview. In particular, we first develop a music information retrieval system with recommendation function (Moody v4) based on the previous system version and experimental data. Second, we designed a series of questionnaires to obtain students’ properties while the user experiment is conducted to gather the physiological signals from students by using Moody v4 to listen to music. Also, a brief interview was made for gathering the feedback to the Moody v4 after the user experiment. To evaluate the performance of the proposed approach, the Root Mean Square Error (RMSE) and Mean Absolute error (MAE) were used in this research. Results reveal 1) physiological signals could have positive effect on the music recommendation system, 2) apart from the physiological signals, features from user properties and user context could be are the more effective than music content and context in music recommender, and 3) lightGBM model could perform more better than other recommendation algorithms with the multimodal features while SVD and SVD++ performs well in the collaborative filtering model. The findings can contribute to the research and practice of music recommendation and academic emotion.