RICA is a logically-grounded inference challenge with focus on the ability to make robust commonsense inferences despite textual perturbations.
RICA consists of sets of natural language statements in the "premise-conclusion" format that require reasoning using latent (implicit) commonsense relationships. We generate 257k commonsense statements capturing 43k axioms comprising different types of commonsense, such as physical, material, and social properties.
Submit to this leaderboard: You can submit your prediction by sending email to peiz@usc.edu with the title "RICA submission (your model name)" and the same format of this example prediction file.
Rank | Model | Average Accuracy |
|---|---|---|
|
Human Performance |
91.7 |
1 |
RoBERTa-Large Radford et. al. 2019 |
50.3 |
2 |
ERNIE Zhang et. al. 2019 |
50.2 |
3 |
BART Lewis et. al. 2019 |
50.2 |
4 |
GPT-2 Radford et. al. 2019 |
50.1 |
5 |
BERT-Large Devlin et. al. 2018 |
49.4 |
Rank | Model | Average Accuracy |
|---|---|---|
|
Human Performance |
91.7 |
1 |
RoBERTa-Large Radford et. al. 2019 |
52.3 |
2 |
BART Lewis et. al. 2019 |
50.2 |
3 |
ERNIE Zhang et. al. 2019 |
50.1 |
4 |
GPT-2 Radford et. al. 2019 |
50.1 |
5 |
BERT-Large Devlin et. al. 2018 |
49.9 |
@inproceedings{zhou2021rica,
title={RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms},
author={Zhou, Pei and Khanna, Rahul and Lee, Seyeon and Lin, Bill Yuchen and Ho, Daniel and Pujara, Jay and Ren, Xiang},
booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
pages={7560--7579},
year={2021}
}