File Download
There are no files associated with this item.
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1145/2232817.2232842
- Scopus: eid_2-s2.0-84863549107
- Find via
Supplementary
-
Citations:
- Scopus: 0
- Appears in Collections:
Conference Paper: Generating ground truth for music mood classification using mechanical turk
Title | Generating ground truth for music mood classification using mechanical turk |
---|---|
Authors | |
Keywords | Crowdsourcing Evaluation Gold Standard Ground Truth Mechanical Turk Mood Music Information Retrieval |
Issue Date | 2012 |
Publisher | I E E E. The Journal's web site is located at http://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000206 |
Citation | Proceedings Of The Acm/Ieee Joint Conference On Digital Libraries, 2012, p. 129-138 How to Cite? |
Abstract | Mood is an important access point in music digital libraries and online music repositories, but generating ground truth for evaluating various music mood classification algorithms is a challenging problem. This is because collecting enough human judgments is time-consuming and costly due to the subjectivity of music mood. In this study, we explore the viability of crowdsourcing music mood classification judgments using Amazon Mechanical Turk (MTurk). Specifically, we compare the mood classification judgments collected for the annual Music Information Retrieval Evaluation eXchange (MIREX) with judgments collected using MTurk. Our data show that the overall distribution of mood clusters and agreement rates from MIREX and MTurk were comparable. However, Turkers tended to agree less with the pre-labeled mood clusters than MIREX evaluators. The system evaluation results generated using both sets of data were mostly the same except for detecting one statistically significant pair using Friedman's test. We conclude that MTurk can potentially serve as a viable alternative for ground truth collection, with some reservation with regards to particular mood clusters. © 2012 ACM. |
Persistent Identifier | http://hdl.handle.net/10722/180714 |
ISSN | 2020 SCImago Journal Rankings: 0.264 |
References |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Lee, JH | en_US |
dc.contributor.author | Hu, X | en_US |
dc.date.accessioned | 2013-01-28T01:41:33Z | - |
dc.date.available | 2013-01-28T01:41:33Z | - |
dc.date.issued | 2012 | en_US |
dc.identifier.citation | Proceedings Of The Acm/Ieee Joint Conference On Digital Libraries, 2012, p. 129-138 | en_US |
dc.identifier.issn | 1552-5996 | en_US |
dc.identifier.uri | http://hdl.handle.net/10722/180714 | - |
dc.description.abstract | Mood is an important access point in music digital libraries and online music repositories, but generating ground truth for evaluating various music mood classification algorithms is a challenging problem. This is because collecting enough human judgments is time-consuming and costly due to the subjectivity of music mood. In this study, we explore the viability of crowdsourcing music mood classification judgments using Amazon Mechanical Turk (MTurk). Specifically, we compare the mood classification judgments collected for the annual Music Information Retrieval Evaluation eXchange (MIREX) with judgments collected using MTurk. Our data show that the overall distribution of mood clusters and agreement rates from MIREX and MTurk were comparable. However, Turkers tended to agree less with the pre-labeled mood clusters than MIREX evaluators. The system evaluation results generated using both sets of data were mostly the same except for detecting one statistically significant pair using Friedman's test. We conclude that MTurk can potentially serve as a viable alternative for ground truth collection, with some reservation with regards to particular mood clusters. © 2012 ACM. | en_US |
dc.language | eng | en_US |
dc.publisher | I E E E. The Journal's web site is located at http://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000206 | en_US |
dc.relation.ispartof | Proceedings of the ACM/IEEE Joint Conference on Digital Libraries | en_US |
dc.subject | Crowdsourcing | en_US |
dc.subject | Evaluation | en_US |
dc.subject | Gold Standard | en_US |
dc.subject | Ground Truth | en_US |
dc.subject | Mechanical Turk | en_US |
dc.subject | Mood | en_US |
dc.subject | Music Information Retrieval | en_US |
dc.title | Generating ground truth for music mood classification using mechanical turk | en_US |
dc.type | Conference_Paper | en_US |
dc.identifier.email | Hu, X: xiaoxhu@hku.hk | en_US |
dc.identifier.authority | Hu, X=rp01711 | en_US |
dc.description.nature | link_to_subscribed_fulltext | en_US |
dc.identifier.doi | 10.1145/2232817.2232842 | en_US |
dc.identifier.scopus | eid_2-s2.0-84863549107 | en_US |
dc.relation.references | http://www.scopus.com/mlt/select.url?eid=2-s2.0-84863549107&selection=ref&src=s&origin=recordpage | en_US |
dc.identifier.spage | 129 | en_US |
dc.identifier.epage | 138 | en_US |
dc.publisher.place | United States | en_US |
dc.identifier.scopusauthorid | Lee, JH=36064170000 | en_US |
dc.identifier.scopusauthorid | Hu, X=55496358400 | en_US |
dc.identifier.issnl | 1552-5996 | - |