File Download
There are no files associated with this item.
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1002/meet.14504701144
- Scopus: eid_2-s2.0-84861422067
- Find via
Supplementary
-
Citations:
- Scopus: 0
- Appears in Collections:
Conference Paper: New measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness
Title | New measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness |
---|---|
Authors | |
Keywords | Experimentation Human factors Measurement |
Issue Date | 2010 |
Citation | ASIS&T 2010 Annual Meeting, Pittsburgh, PA, 22-27 October 2010. In Proceedings of the ASIST Annual Meeting, 2010, v. 47 How to Cite? |
Abstract | User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system. |
Persistent Identifier | http://hdl.handle.net/10722/180707 |
ISSN | 2019 SCImago Journal Rankings: 0.125 |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Cheng, J | en_US |
dc.contributor.author | Hu, X | en_US |
dc.contributor.author | Heidorn, PB | en_US |
dc.date.accessioned | 2013-01-28T01:41:31Z | - |
dc.date.available | 2013-01-28T01:41:31Z | - |
dc.date.issued | 2010 | en_US |
dc.identifier.citation | ASIS&T 2010 Annual Meeting, Pittsburgh, PA, 22-27 October 2010. In Proceedings of the ASIST Annual Meeting, 2010, v. 47 | en_US |
dc.identifier.issn | 1550-8390 | en_US |
dc.identifier.uri | http://hdl.handle.net/10722/180707 | - |
dc.description.abstract | User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system. | en_US |
dc.language | eng | en_US |
dc.relation.ispartof | Proceedings of the ASIST Annual Meeting | en_US |
dc.subject | Experimentation | en_US |
dc.subject | Human factors | en_US |
dc.subject | Measurement | en_US |
dc.title | New measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness | en_US |
dc.type | Conference_Paper | en_US |
dc.identifier.email | Hu, X: xiaoxhu@hku.hk | en_US |
dc.identifier.authority | Hu, X=rp01711 | en_US |
dc.description.nature | link_to_subscribed_fulltext | en_US |
dc.identifier.doi | 10.1002/meet.14504701144 | en_US |
dc.identifier.scopus | eid_2-s2.0-84861422067 | - |
dc.identifier.volume | 47 | en_US |
dc.identifier.issnl | 1550-8390 | - |