Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning 事件

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning arXiv:2606.05173v1 Announce Type: new Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure. Inspired by the success of Joint Embedding Predictive Architectures (JEPA) (LeCun, 2022) in vision and audio, we propo