A Component-level Analysis of an Academic Search Test Collection

Abstract

This study analyzes causes for successful and unsuccessful search performance in an academic search test collection. Based on a component-level evaluation setting presented in a parallel paper, analyses of the recall base and the semantic heterogeneity of queries and documents were used for performance prediction. The study finds that neither the recall base, query specificity nor ambiguity can predict the overall search performance or identify badly performing queries. A detailed query analysis finds patterns for negative effects (e.g. non-content-bearing terms in topics), but none are overly significant.