Importing a local module in a task included with include_path

[joaormfsilva]

I'm having problems importing a local module in a custom task that is included from a separate folder via --include_path.

The ifeval task contains code that imports other Python modules from that task. For instance, ifeval's utils.py has the following import: from lm_eval.tasks.ifeval import instructions_registry.

Now, I'm working on a new task, let's call it newtask, and I need to do something similar, but this new task is not in the harness' tasks folder, but in a separate folder, let's call it newtasksfolder, which is included using --include_path newtasksfolder in the lm_eval command line.

For other tasks, with no local imports, this works fine. However, I haven't been able to find the proper way to do an import similar to the one above. The most straightforwad option, from . import instructions_registry, fails with ImportError: attempted relative import with no known parent package.

Ultimately, I guess I can just move the new task to the normal tasks folder of the harness and use from lm_eval.tasks.newtask import instructions_registry, but first I'd like to try to have the new task totally separate from the harness, as it is now.

Am I missing something or is the module loading (sys.path and so on) not dealing well with tasks that are loaded from a separate folder via include_path?