-
Generation based MRCNLP/MRC 2021. 10. 13. 11:20
주어진 지문과 질의를 보고 답변을 생성
MRC 모델 구조
- Seq-to-seq PLM 구조 (generation) vs. PLM + classifier 구조 (extraction)
- loss 계산을 위한 답의 형태 Prediction의 형태
- Free form text 형태 (generation) vs. 지문 내 답의 위치 (extraction)
입력 표현
- Special Token
- 학습 시에만 사용되며 단어 자체의 의미는 가지지 않는 특별한 토큰
- Attention mask
- Token type ids
- BART에는 입력시퀀스에 대한 구분이 없어 token_type_ids가 존재하지 않음
- 따라서 입력에 token_type_ids가 안들어감
BART
- 기계독해, 기계번역, 요약, 대화 등 seq2seq 문제 pre-trainingdmf 위한 denoising autoencoder
- 텍스트에 노이즈를 주고 원래 텍스트를 복구하는 문제를 푸는 것으로 pre-training
Post processing
- Search methods → 일반적으로 beam search 사용 (k 개의 candidate를 유지)
https://sshleifer.github.io/blog_v2/jupyter/2020/03/12/bart.html
Introducing BART
Episode 1 – a mysterious new Seq2Seq model with state of the art summarization performance visits a popular open source library
sshleifer.github.io
https://arxiv.org/abs/1910.13461
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based
arxiv.org
https://arxiv.org/abs/1910.10683
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a div
arxiv.org
'NLP > MRC' 카테고리의 다른 글
Passage Retrieval - Scaling Up (0) 2021.10.13 Dense Embedding (0) 2021.10.13 Passage Retrieval (0) 2021.10.13 Extraction-Based MRC (0) 2021.10.12 Introduction to MRC (0) 2021.10.12