Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Python package metadata for conda packages built using rapids-build-backend #151

Open
vyasr opened this issue Feb 11, 2025 · 0 comments

Comments

@vyasr
Copy link
Contributor

vyasr commented Feb 11, 2025

One of the principle functions of rapids-build-backend is to overwrite the project name and dependencies in pyproject.toml during the build. However, for conda builds we disable this behavior (e.g. here in rmm) because none of the package names need to be modified since CUDA suffixes for wheels are a workaround for the fact that pip does not support the richer metadata required to differentiate different types of builds of a given version of a package while conda does. Implicit in this decision is the assumption that we also do not need different sets of dependencies based on CUDA versions. Technically, this is not true. Concretely, if we consider rmm's cuda-python dependency, we see that it is set up to fall back to the CUDA 11 dependencies by default, so both the CUDA 11 and CUDA 12 builds of the conda package will both wind up with the CUDA 11 dependency in the wheel metadata file.In practice, this is largely unimportant because pip metadata is not relevant for conda packages as long as all the packages being installed are conda packages. However, having incorrect pip metadata reduces the (already limited) interoperability of these conda packages with other pip installations since pip's dependency resolution will read the wrong dependency data and try to use it and therefore lead to many complaints that dependencies are unsatisfied. It also makes these packages incompatible with pip check.

In order to fix this, I think that it would be sufficient to change the behavior of the disable_cuda option to only turn of package naming, not dependency matrixing. For backwards-compatibility reasons we could define two new options, one for disabling name suffixes and one for dependencies, but in the long run I think we only ever really need to turn off the name suffixes. When I was originally developing RBB I added the current behavior before we had dfg integration, and so at the time it made sense to group the dependency name suffixes and the dependency versions. However, in practice I think that we always want matrixed dependency versions and we only want to skip the dependency names. If a package version is identical across different CUDA matrix entries we can always leverage dfg's newer support for globs to still capture all the necessary versions in a single matrix entry in dependencies.yaml. With respect to RBB, all that should be required (up to whatever backwards-compatibility layers etc we put in) should be removing this line.

We would then have to split our dependency lists. Broadly speaking there are two groups that come to mind:

  1. Packages like cuda-python that embed the CUDA version directly in their own version, and therefore don't have different names.
  2. Packages like cupy or RAPIDS packages, which use different names for CUDA versions but produce packages with the same version

In the first case I don't think any action is needed on the dependency list because just specifying the matrix to dfg should work because the lists are already matrixed because we have used joint conda/pyproject dependency lists. In the second case we would need to convert common dependency lists for conda deps into specific matrix entries dependent on CUDA versions.

I do not think there are any cases of packages that both have different package names for different CUDA versions and embed the CUDA version in their own versioning. If that were to exist (say if cuda.core adopted that pattern for some reason) then I think we would just wind up with a multiplexed dependency list entry for that dependency, same as above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant