Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models 事件

Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models arXiv:2303.15619v2 Announce Type: replace Abstract: The choice of \emph{which} tokens to mask is a central, under-examined design decision in masked language modeling (MLM). Standard pretraining masks tokens uniformly at random, but several studies show that more informative masking targets can improve downstream performance. We study masking as a \emph{task-adaptive} component of the fine-tuning pipeli