Any example for non-clip VLM?

[Ash-one]

Currently used VLMs such as qwen2.5-vl do not have the CLS EOS SOS token from CLIP. How can the results in the paper be reproduced under these circumstances?

[childult-programmer]

Hello, @Ash-one.
Thank you for your interest in our work!

For models without a [CLS] token, we use the average attention over visual tokens as a proxy for [CLS]. For the [EOS] token, we use the mean of all text token embeddings. The [SOS] token is used only for analysis and is not involved in token pruning. Thank you.

[Ash-one]

Thank you for your reply. I would like to ask some details. For qwen2.5-vl, is it correct to select the attention score in the last layer of its visual encoder, calculate the average attention received by each visual token from other tokens, and then take the index corresponding to the maximum value as the [CLS] proxy? For text, is it correct to extract the embedding of the referent text instead of the full user prompt?

[childult-programmer]

Thank you for the follow-up question.

For the visual side, we do not select a single visual token with the maximum average attention for the [CLS] token. For models without a [CLS] token, we calculate the average attention each visual token receives from all other tokens at the final layer.

For the input text, we use all text tokens. Although RefCOCO-family datasets provide the referent text, we assume that such information is not generally available, and therefore use the average of all text token embeddings as a proxy for the [EOS] token.