You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
One of the principle functions of rapids-build-backend is to overwrite the project name and dependencies in pyproject.toml during the build. However, for conda builds we disable this behavior (e.g. here in rmm) because none of the package names need to be modified since CUDA suffixes for wheels are a workaround for the fact that pip does not support the richer metadata required to differentiate different types of builds of a given version of a package while conda does. Implicit in this decision is the assumption that we also do not need different sets of dependencies based on CUDA versions. Technically, this is not true. Concretely, if we consider rmm's cuda-python dependency, we see that it is set up to fall back to the CUDA 11 dependencies by default, so both the CUDA 11 and CUDA 12 builds of the conda package will both wind up with the CUDA 11 dependency in the wheel metadata file.In practice, this is largely unimportant because pip metadata is not relevant for conda packages as long as all the packages being installed are conda packages. However, having incorrect pip metadata reduces the (already limited) interoperability of these conda packages with other pip installations since pip's dependency resolution will read the wrong dependency data and try to use it and therefore lead to many complaints that dependencies are unsatisfied. It also makes these packages incompatible with pip check.
In order to fix this, I think that it would be sufficient to change the behavior of the disable_cuda option to only turn of package naming, not dependency matrixing. For backwards-compatibility reasons we could define two new options, one for disabling name suffixes and one for dependencies, but in the long run I think we only ever really need to turn off the name suffixes. When I was originally developing RBB I added the current behavior before we had dfg integration, and so at the time it made sense to group the dependency name suffixes and the dependency versions. However, in practice I think that we always want matrixed dependency versions and we only want to skip the dependency names. If a package version is identical across different CUDA matrix entries we can always leverage dfg's newer support for globs to still capture all the necessary versions in a single matrix entry in dependencies.yaml. With respect to RBB, all that should be required (up to whatever backwards-compatibility layers etc we put in) should be removing this line.
We would then have to split our dependency lists. Broadly speaking there are two groups that come to mind:
Packages like cuda-python that embed the CUDA version directly in their own version, and therefore don't have different names.
Packages like cupy or RAPIDS packages, which use different names for CUDA versions but produce packages with the same version
In the first case I don't think any action is needed on the dependency list because just specifying the matrix to dfg should work because the lists are already matrixed because we have used joint conda/pyproject dependency lists. In the second case we would need to convert common dependency lists for conda deps into specific matrix entries dependent on CUDA versions.
I do not think there are any cases of packages that both have different package names for different CUDA versions and embed the CUDA version in their own versioning. If that were to exist (say if cuda.core adopted that pattern for some reason) then I think we would just wind up with a multiplexed dependency list entry for that dependency, same as above.
The text was updated successfully, but these errors were encountered:
One of the principle functions of
rapids-build-backend
is to overwrite the project name and dependencies in pyproject.toml during the build. However, for conda builds we disable this behavior (e.g. here in rmm) because none of the package names need to be modified since CUDA suffixes for wheels are a workaround for the fact that pip does not support the richer metadata required to differentiate different types of builds of a given version of a package while conda does. Implicit in this decision is the assumption that we also do not need different sets of dependencies based on CUDA versions. Technically, this is not true. Concretely, if we consider rmm's cuda-python dependency, we see that it is set up to fall back to the CUDA 11 dependencies by default, so both the CUDA 11 and CUDA 12 builds of the conda package will both wind up with the CUDA 11 dependency in the wheel metadata file.In practice, this is largely unimportant because pip metadata is not relevant for conda packages as long as all the packages being installed are conda packages. However, having incorrect pip metadata reduces the (already limited) interoperability of these conda packages with other pip installations since pip's dependency resolution will read the wrong dependency data and try to use it and therefore lead to many complaints that dependencies are unsatisfied. It also makes these packages incompatible withpip check
.In order to fix this, I think that it would be sufficient to change the behavior of the
disable_cuda
option to only turn of package naming, not dependency matrixing. For backwards-compatibility reasons we could define two new options, one for disabling name suffixes and one for dependencies, but in the long run I think we only ever really need to turn off the name suffixes. When I was originally developing RBB I added the current behavior before we had dfg integration, and so at the time it made sense to group the dependency name suffixes and the dependency versions. However, in practice I think that we always want matrixed dependency versions and we only want to skip the dependency names. If a package version is identical across different CUDA matrix entries we can always leverage dfg's newer support for globs to still capture all the necessary versions in a single matrix entry in dependencies.yaml. With respect to RBB, all that should be required (up to whatever backwards-compatibility layers etc we put in) should be removing this line.We would then have to split our dependency lists. Broadly speaking there are two groups that come to mind:
In the first case I don't think any action is needed on the dependency list because just specifying the matrix to dfg should work because the lists are already matrixed because we have used joint conda/pyproject dependency lists. In the second case we would need to convert
common
dependency lists for conda deps intospecific
matrix entries dependent on CUDA versions.I do not think there are any cases of packages that both have different package names for different CUDA versions and embed the CUDA version in their own versioning. If that were to exist (say if cuda.core adopted that pattern for some reason) then I think we would just wind up with a multiplexed dependency list entry for that dependency, same as above.
The text was updated successfully, but these errors were encountered: