Toward the Scientific Evaluation of Music Information Retrieval Systems.pdf

Toward the Scientific Evaluation of Music Information Retrieval Systems J. Stephen Downie Graduate School of Library and Information Science University of Illinois at Urbana-Champaign jdownie@ ABSTRACT facing MIR research. Sections 2-3 report upon the findings-to- date of the “MIR/MDL Evaluation Project,” with issues This paper outlines the findings-to-date of a project surrounding the creation of a TREC-like evaluation paradigm to assist in the efforts being made to establish a for MIR as the central focus. Section 4 highlights the progress TREC-like evaluation paradigm within the Music being made concerning the establishment of the necessary test Information Retrieval (MIR) research community. collection(s). Section 5 concludes with a summary and The findings and recommendations are based upon outlines some of the key challenges uncovered that require expert opinion garnered from members of the further investigation. Information Retrieval (IR), Music Digital Library (MDL) and MIR communities with regard to the 1.1 Current Scientific Problem construction and implementation of scientifically Notwithstanding the promising tech