We're using LM-Harness to evaluate Qwen/Qwen2-VL-2B-Instruct on task_name: mmmu_val using 4 A100 GPUs w/ 40GB VRAM.

The evaluation starts and progresses OK until the middle of the dataset where it fails with CUDA OOM (out-of-memory). It looks like it may be caused by a single example in the dataset which requires more GPU memory than usual, leading to OOM.

This Qwen2-VL OOM problem was resolved for training, by passing the following params as Qwen2-VL processor_kwargs:

{
  "min_pixels": 256 * 28 * 28,
  "max_pixels": 1280 * 28 * 28,
}

Context: https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct#image-resolution-for-performance-boost

For LH-Harness evaluation, I haven't found a way to configure processor_kwargs for hf-multimodal (only model_args seem configurable):

Could you please advise if setting processor_kwargs is supported? If not, would it be possible to add it for hf-multimodal ?