In ProtTrans, the author says that:
No auxiliary tasks like BERT's next-sentence prediction were used for any model described here.
But in the PEER, the [CLS] token is used for ProtBert as a protein-level embedding representation. In this case the [CLS] token may not have the ability to represent sequence embedding.
For ProtBert, should we use the same strategy as for ESM (i.e., mean pooling over all residues) to get a fairer comparison?
In ProtTrans, the author says that:
But in the PEER, the
[CLS]token is used forProtBertas a protein-level embedding representation. In this case the[CLS]token may not have the ability to represent sequence embedding.For
ProtBert, should we use the same strategy as forESM(i.e., mean pooling over all residues) to get a fairer comparison?