Abstract Carlos Santos Armendariz 3 June 2020
Title: CoSimLex: Graded Effects of Context in Similarity Perception
State of the art natural language processing tools are built on context-dependent word embeddings, but no direct method for evaluating these representations currently exists. Standard tasks and datasets for intrinsic evaluation of embeddings are based on judgements of similarity, but ignore context; standard tasks for word sense disambiguation take account of context but focus on discrete differences, and do not provide continuous measures of similarity. In this talk we have a look at CoSimlex, a new dataset containing context-dependent similarity measures, designed to provide not only discrete differences in word sense but more subtle, graded changes in meaning. We will explain the psychological/cognitive motivations behind our design choices, and present lessons learned during annotation and the running of a SemEval task based on this dataset. Finally we will offer a very preliminary analysis of the results, and ideas for future endeavours.