takehika/xlm-roberta-en-squadv2-qa

xlm-roberta-base-squad-v2-qa

Fine-tuned xlm-roberta-base on the English SQuAD v2 dataset for extractive question answering with unanswerable questions.

• Task: extractive QA (SQuAD v2)
• Language: English
• License: CC BY-SA 4.0
• Base model: xlm-roberta-base

Usage

from transformers import pipeline

model_id = "takehika/xlm-roberta-en-squadv2-qa"
qa = pipeline("question-answering", model=model_id, tokenizer=model_id)

context = "I am a digital assistant created to support humans. I do not sleep or eat like people do. I live inside computers and servers around the world. I enjoy processing information, and my favorite energy source is electricity."
questions = [
    "What was the assistant created to do?",
    "Where does it live?",
    "What does it enjoy?",
    "What is its favorite energy source?",
    "What is the capital of France?"
]

threshold = 0.15
results = qa(question=questions, context=context)

for r in results:
    if r["score"] < threshold:
        r["answer"] = ""  # no-answer
    print(r)
    

Data

• Dataset: SQuAD v2 (rajpurkar/squad_v2)
• Task: extractive QA with unanswerable questions in English

Training

• Base: xlm-roberta-base
• Epochs: 2
• Learning rate: 2e-5
• Warmup ratio: 0.1
• Batch size: 8 per device, grad accumulation 2 (effective 16)
• Evaluation/save: every 1000 steps
• Best model selection: f1

Evaluation

• Checkpoint selected for push: checkpoint-14000
  • Exact: 75.38
  • F1: 78.08
  • HasAns Exact/F1: 71.00 / 76.42
  • NoAns Exact/F1: 79.75 / 79.75

Intended Use & Limitations

• Intended for English extractive QA on SQuAD v2 style data.
• Long contexts (>512 tokens) need sliding windows.
• Threshold tuning for no-answer can shift EM/F1 without retraining.
• Domain shifts or ambiguous/sensitive questions may degrade quality.

Attribution & Licenses

• License: CC BY-SA 4.0
• Base model: xlm-roberta-base by Meta AI - MIT License
  • Model card: https://huggingface.co/xlm-roberta-base
• Dataset: SQuAD v2 (rajpurkar/squad_v2) - CC BY-SA 4.0
  • Dataset card: https://huggingface.co/datasets/rajpurkar/squad_v2

This model modifies the base model by fine-tuning on the above dataset.

Base Model Citation

Please cite the following when using the XLM-R base model:

@article{DBLP:journals/corr/abs-1911-02116,
  author    = {Alexis Conneau and
               Kartikay Khandelwal and
               Naman Goyal and
               Vishrav Chaudhary and
               Guillaume Wenzek and
               Francisco Guzm{\'{a}}n and
               Edouard Grave and
               Myle Ott and
               Luke Zettlemoyer and
               Veselin Stoyanov},
  title     = {Unsupervised Cross-lingual Representation Learning at Scale},
  journal   = {CoRR},
  volume    = {abs/1911.02116},
  year      = {2019},
  url       = {http://arxiv.org/abs/1911.02116},
  eprinttype = {arXiv},
  eprint    = {1911.02116},
  timestamp = {Mon, 11 Nov 2019 18:38:09 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1911-02116.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

Dataset Citation

Please cite the following when using the SQuAD v2 datasets:

@inproceedings{rajpurkar-etal-2018-know,
    title = "Know What You Don{'}t Know: Unanswerable Questions for {SQ}u{AD}",
    author = "Rajpurkar, Pranav  and
      Jia, Robin  and
      Liang, Percy",
    editor = "Gurevych, Iryna  and
      Miyao, Yusuke",
    booktitle = "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    month = jul,
    year = "2018",
    address = "Melbourne, Australia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P18-2124",
    doi = "10.18653/v1/P18-2124",
    pages = "784--789",
    eprint={1806.03822},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

@inproceedings{rajpurkar-etal-2016-squad,
    title = "{SQ}u{AD}: 100,000+ Questions for Machine Comprehension of Text",
    author = "Rajpurkar, Pranav  and
      Zhang, Jian  and
      Lopyrev, Konstantin  and
      Liang, Percy",
    editor = "Su, Jian  and
      Duh, Kevin  and
      Carreras, Xavier",
    booktitle = "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2016",
    address = "Austin, Texas",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/D16-1264",
    doi = "10.18653/v1/D16-1264",
    pages = "2383--2392",
    eprint={1606.05250},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
}