ValueError: rope_scaling's type field must be one of ['su', 'yarn'], got longrope

[jzyee]

attempted: https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/integrate_phi3_experts.ipynb

when loading the model with:

import torch
from mergoo.models.modeling_phi3 import Phi3ForCausalLM

model_id = 'data/checkpoint_demo'
// Define the device (use cuda:0 or another GPU if necessary)
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
model = Phi3ForCausalLM.from_pretrained(
model_id,
device_map=device,
torch_dtype=torch.bfloat16
)

and got the following error:
ValueError: rope_scaling's type field must be one of ['su', 'yarn'], got longrope

transformers==4.44.2
torch==2.0.1

[Shigerello]

Same here.
It seems mergoo's Phi-3 implementation is a bit outdated, and is not compatible with the latest model revision.

mergoo

mergoo/mergoo/models/modeling_phi3.py

Lines 236 to 238 in 8dec73f

	rope_scaling_long_factor = self.rope_scaling.get("long_factor", None)
	if rope_scaling_type is None or rope_scaling_type not in ["su", "yarn"]:
	raise ValueError(f"`rope_scaling`'s type field must be one of ['su', 'yarn'], got {rope_scaling_type}")

Phi-3
https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/blob/10d25dfa1593265daf4a3bac573ab76ccf61d60f/configuration_phi3.py#L203-L205

        rope_scaling_long_factor = self.rope_scaling.get("long_factor", None)
        if rope_scaling_type is None or rope_scaling_type not in ["longrope"]:
            raise ValueError(f"`rope_scaling`'s type field must be one of ['longrope'], got {rope_scaling_type}")