BEiT: BERT Pre-Training of Image Transformers - OpenReview After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods
BEIT: RE-TRAINING OF IMAGE TRANSFORMERS - OpenReview We pretrain BEIT and conduct extensive fine-tuning experiments on downstream tasks, such as image classification, and semantic segmentation We present that the self-attention mechanism of self-supervised BEIT learns to distinguish semantic regions and object boundaries, although without using any human annotation