Entropy-aware Masking for Masked Language Modeling 事件

Entropy-aware Masking for Masked Language Modeling arXiv:2605.28526v1 Announce Type: cross Abstract: Masked language modeling has become a standard pretraining objective for training encoder-based language models. In this approach, certain tokens in the input are masked, and the model learns to predict them using the surrounding context. This process enables the model to capture both syntactic and semantic properties of language. Conventionally, the tokens selected for masking are chosen at ran