Best Task 2021: Task 8, MeasEval - Extracting Counts and Measurements and their Related Contexts
Corey Harper, Jessica Cox, Curt Kohler, Antony Scerri, Ron Daniel Jr., and Paul Groth MeasEval is an original information extraction task focused on quantitative measurements in scientific text, with spans for the quantity, units, item measured, and other mentioned attributes, as well as relations between them. The task setup featured a carefully developed annotation schema, guidelines, and dataset, and an evaluation metric with score components for the various kinds of spans and relations. 19 teams participated, and baseline systems developed by the organizers were evaluated as well. The task paper surveys the system papers and includes strong analysis of results, with breakdowns by span/relation type and genre, thoughtful investigation of possible artifacts of the evaluation metric, main conclusions, and ideas for future work.