WinoReg: A New Faster and More Accurate Metric of Hardness for Winograd Schemas

13 pages•Published: April 27, 2020

Abstract

The Winograd Schema Challenge (WSC), the task of resolving pronouns in certain carefully-structured sentences, has received considerable interest in the past few years as an alternative to the Turing Test. In our recent work we demonstrated the plausibility of using commonsense knowledge, automatically acquired from raw text in English Wikipedia, towards computing a metric of hardness for a limited number of Winograd Schemas.
In this work we present WinoReg, a new system to compute hardness of Winograd Schemas, by training a Random Forest classifier over a rich set of features identified in relevant WSC works in the literature. Our empirical study shows that this new system is considerably faster and more accurate compared to the system proposed in our earlier work, making its use as part of other WSC-based systems feasible.

Keyphrases: evaluation of ai systems, knowledge representation, machine learning, natural language systems and linguistics, winograd schema challenge

In: Gregoire Danoy, Jun Pang and Geoff Sutcliffe (editors). GCAI 2020. 6th Global Conference on Artificial Intelligence (GCAI 2020), vol 72, pages 46-58.

Links:	https://easychair.org/publications/paper/qvGz
	https://doi.org/10.29007/wl4b

BibTeX entry

@inproceedings{GCAI2020:WinoReg_New_Faster_More,
  author    = {Nicos Isaak and Loizos Michael},
  title     = {WinoReg: A New Faster and More Accurate Metric of Hardness for Winograd Schemas},
  booktitle = {GCAI 2020. 6th Global Conference on Artificial Intelligence (GCAI 2020)},
  editor    = {Gregoire Danoy and Jun Pang and Geoff Sutcliffe},
  series    = {EPiC Series in Computing},
  volume    = {72},
  publisher = {EasyChair},
  bibsource = {EasyChair, https://easychair.org},
  issn      = {2398-7340},
  url       = {/publications/paper/qvGz},
  doi       = {10.29007/wl4b},
  pages     = {46-58},
  year      = {2020}}

Download PDF Open PDF in browser