Support for namespaced pipelines

[derluke]

Like it says in the title. I see a lot of output in the debug console like:

2024-07-24 18:37:34.755 [info] [Trace - 6:37:34 PM] Received response 'textDocument/hover - (38)' in 7ms. Request failed: kedro.io.core.DatasetNotFoundError: Dataset 'params:custom_model.base_environment_id' not found in the catalog - did you mean one of these instead: params:deploy_llm.custom_model.base_environment_id, params:deploy_vanilla_llm.custom_model.base_environment_id, params:deploy_app.base_environment_id (-32603).

clearly it's missing the namespace on my pipeline.

We also have a hook that injects some datasets into the catalog. Not immediately important, but it causes errors in the log since it looks like the hooks are being ignored.

[noklam]

@derluke Thanks for the report. Hooks are being ignored on purpose for few reasons:

• The extension shouldn't load plugins like spark and making connection.

injects some datasets into the catalog

Can you explains how this was done in a hook? In general we don't encourage injecting datasets, and it's difficult to support dynamic change as the extension shouldn't execute a pipeline & pipeline related logic.

namespace

I think this is the most valid one and more urgent to fix.

[Calychas]

@noklam Using kedro-pandera's schema interpolation (registered by default by the hook) also breaks the extension unfortunately as the hooks aren't loaded by the extension.

Log from the extension's output:

  File "repo/venv/lib/python3.10/site-packages/omegaconf/base.py", line 702, in _evaluate_custom_resolver
    raise UnsupportedInterpolationType(
omegaconf.errors.UnsupportedInterpolationType: Unsupported interpolation type pa.yaml

A workaround is to set them yourself in settings.py the same way as they are registered in the hook, but would be nicer if I could not worry about that 😅

[noklam]

Can you show a snippets how you get around this with settings.py? What does the hook do and when is this error being thrown? (at initialisation or when you trigger some action?) It would be great if you can provide an example repo that I can use for testing, I may have an idea for a quick fix.

…

On Tue, 6 Aug 2024, 10:39 Kacper Leśniara, ***@***.***> wrote: @noklam <https://github.com/noklam> Using kedro-pandera's schema interpolation (registered by default by the hook) also breaks the extension unfortunately as the hooks aren't loaded by the extension. Log from the extension's output: File "repo/venv/lib/python3.10/site-packages/omegaconf/base.py", line 702, in _evaluate_custom_resolver raise UnsupportedInterpolationType( omegaconf.errors.UnsupportedInterpolationType: Unsupported interpolation type pa.yaml A workaround is to set them yourself in settings.py the same way as they are registered in the hook, but would be nicer if I could not worry about that 😅 — Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AELAWLZTUHUO2JVEFJIACGLZQCKUNAVCNFSM6AAAAABLM6TW5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZQHA2DEMZRHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Calychas]

Part of settings.py:

from kedro_pandera.framework.config.resolvers import (  # noqa: E402
    resolve_dataframe_model,
    resolve_interpolated_yaml_schema,
    resolve_yaml_schema,
)

CONFIG_LOADER_ARGS = {
    "base_env": "base",
    "default_run_env": "local",
    "custom_resolvers": {
        # For vscode-kedro extension to work we need to import kedro_pandera resolvers manually
        "pa.dict": resolve_yaml_schema,
        "pa.yaml": resolve_interpolated_yaml_schema,
        "pa.python": resolve_dataframe_model,
    },
}

which is basically this - https://github.com/Galileo-Galilei/kedro-pandera/blob/4fa5ca3c8bbb31f2fcbed00e3dbcd03594ff4a01/kedro_pandera/framework/hooks/pandera_hook.py#L46

The error is thrown during initialization of the extension

[noklam]

@Calychas Thanks for getting back. Do you really need kedro-pandera for the extension, or simply because the import error is causing issue?

From the VSCode extension point of view, the extension have no idea which plugin will be useful, and by default Hook are disabled because they are irrelevant to the extension.

[noklam]

Let's move the discussion to #69 since namespace and hook are two different issues.