Download full text
(371.4Kb)
Citation Suggestion
Please use the following Persistent Identifier (PID) to cite this document:
https://nbn-resolving.org/urn:nbn:de:0168-ssoar-46667-7
Exports for your reference manager
Implications of inter-rater agreement on a student information retrieval evaluation
[collection article]
Corporate Editor
Universität Kassel, Knowledge and Data Engineering Group
Abstract This paper is about an information retrieval evaluation on three different retrieval-supporting
services. All three services were designed to
compensate typical problems that arise in metadata-
driven Digital Libraries, which are not adequately
handled by a simple tf-idf based retrieval.
The se... view more
This paper is about an information retrieval evaluation on three different retrieval-supporting
services. All three services were designed to
compensate typical problems that arise in metadata-
driven Digital Libraries, which are not adequately
handled by a simple tf-idf based retrieval.
The services are: (1) a co-word analysis based
query expansion mechanism and re-ranking via
(2) Bradfordizing and (3) author centrality. The
services are evaluated with relevance assessments
conducted by 73 information science students.
Since the students are neither information
professionals nor domain experts the question of
inter-rater agreement is taken into consideration.
Two important implications emerge: (1) the inter-
rater agreement rates were mainly fair to
moderate and (2) after a data-cleaning step which
erased the assessments with poor agreement rates
the evaluation data shows that the three retrieval
services returned disjoint but still relevant result sets. (author's abstract)... view less
Keywords
data bank; information retrieval; library; information system; information capture; information; information management
Classification
Information and Documentation, Libraries, Archives
Information Science
Free Keywords
Digitale Bibliothek; Digital Library
Collection Title
LWA 2010 : Lernen, Wissen & Adaptivität ; workshop proceedings
Editor
Atzmüller, Martin; Benz, Dominik; Hotho, Andreas; Stumme, Gerd
Conference
Lernen, Wissen & Adaptivität (LWA). Kassel, 2010
Document language
English
Publication Year
2010
City
Kassel
Page/Pages
p. 1-7
Series
Kasseler Informatikschriften (KIS), 2010/5
Status
Postprint; reviewed
Licence
Deposit Licence - No Redistribution, No Modifications