[RESEARCH] finding libnvvm.so #441

rwgk · 2025-02-07T23:17:00Z

Creating this as a place for myself to collect information related to finding libnvvm.so

Note that libnvvm.so is unusual: while most CTK .so files live under <loc>/lib or <loc>/lib64, it lives under <loc>/nvvm/lib64.

rwgk · 2025-02-07T23:19:35Z

$ ./cuda_12.8.0_570.86.10_linux.run --toolkit --toolkitpath=$HOME/ctk_downloads/scratch

$ cd scratch
$ find . -name '*.so' | grep -v -e '^\./extras/' -e '^\./nsight-' -e '^\./compute-sanitizer/' -e '/eclipse_[0-9]*\.so$'
./nvvm/lib64/libnvvm.so
./targets/x86_64-linux/lib/libnvblas.so
./targets/x86_64-linux/lib/libcusolver.so
./targets/x86_64-linux/lib/libnppim.so
./targets/x86_64-linux/lib/libnpps.so
./targets/x86_64-linux/lib/libcudart.so
./targets/x86_64-linux/lib/libcufft.so
./targets/x86_64-linux/lib/libnvJitLink.so
./targets/x86_64-linux/lib/libcusolverMg.so
./targets/x86_64-linux/lib/libcuinj64.so
./targets/x86_64-linux/lib/libnvfatbin.so
./targets/x86_64-linux/lib/stubs/libcusolver.so
./targets/x86_64-linux/lib/stubs/libnvidia-ml.so
./targets/x86_64-linux/lib/stubs/libnppim.so
./targets/x86_64-linux/lib/stubs/libnpps.so
./targets/x86_64-linux/lib/stubs/libcufft.so
./targets/x86_64-linux/lib/stubs/libnvJitLink.so
./targets/x86_64-linux/lib/stubs/libcusolverMg.so
./targets/x86_64-linux/lib/stubs/libnvfatbin.so
./targets/x86_64-linux/lib/stubs/libnppif.so
./targets/x86_64-linux/lib/stubs/libcublas.so
./targets/x86_64-linux/lib/stubs/libnppial.so
./targets/x86_64-linux/lib/stubs/libcurand.so
./targets/x86_64-linux/lib/stubs/libcublasLt.so
./targets/x86_64-linux/lib/stubs/libnppidei.so
./targets/x86_64-linux/lib/stubs/libnvjpeg.so
./targets/x86_64-linux/lib/stubs/libcuda.so
./targets/x86_64-linux/lib/stubs/libnvrtc.alt.so
./targets/x86_64-linux/lib/stubs/libnppig.so
./targets/x86_64-linux/lib/stubs/libcusparse.so
./targets/x86_64-linux/lib/stubs/libnppisu.so
./targets/x86_64-linux/lib/stubs/libnppicc.so
./targets/x86_64-linux/lib/stubs/libnppitc.so
./targets/x86_64-linux/lib/stubs/libnppist.so
./targets/x86_64-linux/lib/stubs/libcufftw.so
./targets/x86_64-linux/lib/stubs/libnvrtc.so
./targets/x86_64-linux/lib/stubs/libnppc.so
./targets/x86_64-linux/lib/libnppif.so
./targets/x86_64-linux/lib/libcublas.so
./targets/x86_64-linux/lib/libcufile_rdma.so
./targets/x86_64-linux/lib/libcufile.so
./targets/x86_64-linux/lib/libnppial.so
./targets/x86_64-linux/lib/libaccinj64.so
./targets/x86_64-linux/lib/libcurand.so
./targets/x86_64-linux/lib/libcublasLt.so
./targets/x86_64-linux/lib/libnppidei.so
./targets/x86_64-linux/lib/libnvrtc-builtins.alt.so
./targets/x86_64-linux/lib/libnvToolsExt.so
./targets/x86_64-linux/lib/libOpenCL.so
./targets/x86_64-linux/lib/libnvjpeg.so
./targets/x86_64-linux/lib/libnvrtc.alt.so
./targets/x86_64-linux/lib/libnppig.so
./targets/x86_64-linux/lib/libcusparse.so
./targets/x86_64-linux/lib/libnvrtc-builtins.so
./targets/x86_64-linux/lib/libnppisu.so
./targets/x86_64-linux/lib/libnppicc.so
./targets/x86_64-linux/lib/libnppitc.so
./targets/x86_64-linux/lib/libnppist.so
./targets/x86_64-linux/lib/libcufftw.so
./targets/x86_64-linux/lib/libnvrtc.so
./targets/x86_64-linux/lib/libnppc.so

rwgk · 2025-02-10T18:31:44Z

Inside a cccl [Dev Container: cuda12.8-gcc13 @ ... ]

$ cd /usr/local/cuda
$ find . -name '*.so' | grep -v -e '^\./compute-sanitizer/'
./nvvm/lib64/libnvvm.so
./targets/x86_64-linux/lib/libnvperf_target.so
./targets/x86_64-linux/lib/libcupti.so
./targets/x86_64-linux/lib/libcudart.so
./targets/x86_64-linux/lib/libnvJitLink.so
./targets/x86_64-linux/lib/libcuinj64.so
./targets/x86_64-linux/lib/stubs/libnvidia-ml.so
./targets/x86_64-linux/lib/stubs/libnvJitLink.so
./targets/x86_64-linux/lib/stubs/libcurand.so
./targets/x86_64-linux/lib/stubs/libcuda.so
./targets/x86_64-linux/lib/stubs/libnvrtc.alt.so
./targets/x86_64-linux/lib/stubs/libnvrtc.so
./targets/x86_64-linux/lib/libpcsamplingutil.so
./targets/x86_64-linux/lib/libaccinj64.so
./targets/x86_64-linux/lib/libcurand.so
./targets/x86_64-linux/lib/libnvperf_host.so
./targets/x86_64-linux/lib/libnvrtc-builtins.alt.so
./targets/x86_64-linux/lib/libnvToolsExt.so
./targets/x86_64-linux/lib/libnvrtc.alt.so
./targets/x86_64-linux/lib/libnvrtc-builtins.so
./targets/x86_64-linux/lib/libcheckpoint.so
./targets/x86_64-linux/lib/libnvrtc.so

rwgk · 2025-02-10T18:35:48Z

Ubuntu 24.04 workstation with CTK 12.6.1 installed into /usr/local

$ cd /usr/local/cuda
$ find . -name '*.so' | grep -v -e '^\./compute-sanitizer/' -e '/eclipse_[0-9]*\.so$'
./nvvm/lib64/libnvvm.so
./targets/x86_64-linux/lib/libnvperf_target.so
./targets/x86_64-linux/lib/libcupti.so
./targets/x86_64-linux/lib/libnvblas.so
./targets/x86_64-linux/lib/libcusolver.so
./targets/x86_64-linux/lib/libnppim.so
./targets/x86_64-linux/lib/libnpps.so
./targets/x86_64-linux/lib/libcudart.so
./targets/x86_64-linux/lib/libcufft.so
./targets/x86_64-linux/lib/libnvJitLink.so
./targets/x86_64-linux/lib/libcusolverMg.so
./targets/x86_64-linux/lib/libcuinj64.so
./targets/x86_64-linux/lib/libnvfatbin.so
./targets/x86_64-linux/lib/stubs/libcusolver.so
./targets/x86_64-linux/lib/stubs/libnvidia-ml.so
./targets/x86_64-linux/lib/stubs/libnppim.so
./targets/x86_64-linux/lib/stubs/libnpps.so
./targets/x86_64-linux/lib/stubs/libcufft.so
./targets/x86_64-linux/lib/stubs/libnvJitLink.so
./targets/x86_64-linux/lib/stubs/libcusolverMg.so
./targets/x86_64-linux/lib/stubs/libnvfatbin.so
./targets/x86_64-linux/lib/stubs/libnppif.so
./targets/x86_64-linux/lib/stubs/libcublas.so
./targets/x86_64-linux/lib/stubs/libnppial.so
./targets/x86_64-linux/lib/stubs/libcurand.so
./targets/x86_64-linux/lib/stubs/libcublasLt.so
./targets/x86_64-linux/lib/stubs/libnppidei.so
./targets/x86_64-linux/lib/stubs/libnvjpeg.so
./targets/x86_64-linux/lib/stubs/libcuda.so
./targets/x86_64-linux/lib/stubs/libnppig.so
./targets/x86_64-linux/lib/stubs/libcusparse.so
./targets/x86_64-linux/lib/stubs/libnppisu.so
./targets/x86_64-linux/lib/stubs/libnppicc.so
./targets/x86_64-linux/lib/stubs/libnppitc.so
./targets/x86_64-linux/lib/stubs/libnppist.so
./targets/x86_64-linux/lib/stubs/libcufftw.so
./targets/x86_64-linux/lib/stubs/libnvrtc.so
./targets/x86_64-linux/lib/stubs/libnppc.so
./targets/x86_64-linux/lib/libnppif.so
./targets/x86_64-linux/lib/libcublas.so
./targets/x86_64-linux/lib/libcufile_rdma.so
./targets/x86_64-linux/lib/libcufile.so
./targets/x86_64-linux/lib/libnppial.so
./targets/x86_64-linux/lib/libpcsamplingutil.so
./targets/x86_64-linux/lib/libaccinj64.so
./targets/x86_64-linux/lib/libcurand.so
./targets/x86_64-linux/lib/libcublasLt.so
./targets/x86_64-linux/lib/libnvperf_host.so
./targets/x86_64-linux/lib/libnppidei.so
./targets/x86_64-linux/lib/libnvToolsExt.so
./targets/x86_64-linux/lib/libOpenCL.so
./targets/x86_64-linux/lib/libnvjpeg.so
./targets/x86_64-linux/lib/libnppig.so
./targets/x86_64-linux/lib/libcusparse.so
./targets/x86_64-linux/lib/libnvrtc-builtins.so
./targets/x86_64-linux/lib/libnppisu.so
./targets/x86_64-linux/lib/libnppicc.so
./targets/x86_64-linux/lib/libnppitc.so
./targets/x86_64-linux/lib/libnppist.so
./targets/x86_64-linux/lib/libcufftw.so
./targets/x86_64-linux/lib/libcheckpoint.so
./targets/x86_64-linux/lib/libnvrtc.so
./targets/x86_64-linux/lib/libnppc.so

rwgk · 2025-02-11T21:29:45Z

Using a venv (named scratchenv here):

$ pip install cuda-bindings

Collecting cuda-bindings
  Using cached cuda_bindings-12.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)
Using cached cuda_bindings-12.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.2 MB)
Installing collected packages: cuda-bindings
Successfully installed cuda-bindings-12.8.0

$ pip install nvidia-cuda-nvcc-cu12

Collecting nvidia-cuda-nvcc-cu12
  Using cached nvidia_cuda_nvcc_cu12-12.8.61-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.7 kB)
Using cached nvidia_cuda_nvcc_cu12-12.8.61-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl (40.1 MB)
Installing collected packages: nvidia-cuda-nvcc-cu12
Successfully installed nvidia-cuda-nvcc-cu12-12.8.61

$ find scratchenv -name '*.so*'

scratchenv/lib/python3.12/site-packages/cuda/ccudart.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/cudart.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/nvrtc.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/cnvrtc.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/cuda.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/_internal/nvjitlink.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/_internal/utils.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/cydriver.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/nvrtc.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/driver.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/cyruntime.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/nvjitlink.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/cynvjitlink.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/_bindings/cydriver.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/_bindings/cynvrtc.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/_lib/cyruntime/cyruntime.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/_lib/cyruntime/utils.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/_lib/utils.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/cynvrtc.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/bindings/runtime.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/cuda/ccuda.cpython-312-x86_64-linux-gnu.so
scratchenv/lib/python3.12/site-packages/nvidia/cuda_nvcc/nvvm/lib64/libnvvm.so

rwgk · 2025-02-11T22:00:47Z

When using conda (miniforge), this appears to always exist, after installing cuda-nvcc:

$CONDA_PREFIX/nvvm/lib64/

With these files:

     libnvvm.so -> libnvvm.so.4.0.0
     libnvvm.so.4 -> libnvvm.so.4.0.0
62M  libnvvm.so.4.0.0

I tried:

conda install cuda-version=12.8 cuda-nvcc into the base environment.
conda create --yes -n scratchenv python=3.12 cuda-version=12.8 cuda-nvcc

Additionally, $CONDA_PREFIX/lib/python3.12/site-packages appears to always be included in sys.path.

rwgk · 2025-02-11T23:54:59Z

For the record, I also experimented with this script:

https://github.com/rwgk/stuff/blob/2398710e6be12f65c044b91c8cb105093c720f43/cuda-python/cuda_bindings_nvvm_proc_self_maps.py

Example:

Into a scratchvenv:

pip install cuda_bindings-12.8.0-cp312-cp312-linux_x86_64.whl

Example output WITHOUT setting LD_LIBRARY_PATH:


label: after import cuda
NO DIFF


label: after import cuda.bindings
NO DIFF


label: after import cuda.bindings.nvvm
--- before
+++ after
@@ -1,8 +1,15 @@
+/home/rgrossekunst/scratchenv/lib/python3.12/site-packages/cuda/bindings/_internal/nvvm.cpython-312-x86_64-linux-gnu.so
+/home/rgrossekunst/scratchenv/lib/python3.12/site-packages/cuda/bindings/_internal/utils.cpython-312-x86_64-linux-gnu.so
+/home/rgrossekunst/scratchenv/lib/python3.12/site-packages/cuda/bindings/cynvvm.cpython-312-x86_64-linux-gnu.so
+/home/rgrossekunst/scratchenv/lib/python3.12/site-packages/cuda/bindings/nvvm.cpython-312-x86_64-linux-gnu.so
 /usr/bin/python3.12
 /usr/lib/locale/locale-archive
 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
 /usr/lib/x86_64-linux-gnu/libc.so.6
 /usr/lib/x86_64-linux-gnu/libexpat.so.1.9.1
+/usr/lib/x86_64-linux-gnu/libgcc_s.so.1
 /usr/lib/x86_64-linux-gnu/libm.so.6
+/usr/lib/x86_64-linux-gnu/libpthread.so.0
+/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33
 /usr/lib/x86_64-linux-gnu/libz.so.1.3


label: after cuda.bindings.nvvm.version() failure
--- before
+++ after
@@ -7,9 +7,12 @@
 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
 /usr/lib/x86_64-linux-gnu/libc.so.6
+/usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03
+/usr/lib/x86_64-linux-gnu/libdl.so.2
 /usr/lib/x86_64-linux-gnu/libexpat.so.1.9.1
 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
 /usr/lib/x86_64-linux-gnu/libm.so.6
 /usr/lib/x86_64-linux-gnu/libpthread.so.0
+/usr/lib/x86_64-linux-gnu/librt.so.1
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33
 /usr/lib/x86_64-linux-gnu/libz.so.1.3

After export LD_LIBRARY_PATH=/usr/local/cuda/nvvm/lib64 the output changes:

--- /home/rgrossekunst/z/vfailure       2025-02-11 15:53:15.693518916 -0800
+++ /home/rgrossekunst/z/vsuccess       2025-02-11 15:53:05.281172240 -0800
@@ -29,10 +29,10 @@
  /usr/lib/x86_64-linux-gnu/libz.so.1.3
 
 
-label: after cuda.bindings.nvvm.version() failure
+label: after cuda.bindings.nvvm.version() success
 --- before
 +++ after
-@@ -7,9 +7,12 @@
+@@ -7,9 +7,13 @@
  /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
  /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
  /usr/lib/x86_64-linux-gnu/libc.so.6
@@ -45,5 +45,6 @@
 +/usr/lib/x86_64-linux-gnu/librt.so.1
  /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33
  /usr/lib/x86_64-linux-gnu/libz.so.1.3
++/usr/local/cuda-12.6/nvvm/lib64/libnvvm.so.4.0.0

rwgk · 2025-02-12T00:09:06Z

Possible approaches for locating libnvvm.so:

Rely on the existing $ORIGIN/../../../nvidia/cuda_nvcc/nvvm/lib64 rpath addition to discover site-packages/nvidia/cuda_nvcc/nvvm/lib64.
Traverse sys.path, look for lib subdirectories, look for nvvm/lib64 in their parent directories. — This will work for conda-based installations with separate venv. — We wouldn't need the $ORIGIN/../../../../../../nvvm/lib64 rpath addition anymore, although it would be fine to keep. — Note that this is similar to looking for $CONDA_PREFIX/nvvm/lib64, but without requiring that environment variable.
Look for $CUDA_HOME/nvvm/lib64 or $CUDA_PATH/nvvm/lib64 as the last resort, e.g. to find /usr/local/cuda/nvvm/lib64.

In theory we could look for /usr/local/cuda/nvvm/lib64 as the very last resort, but is that wise?

kkraus14 · 2025-02-12T17:31:34Z

@rwgk could probably also learn from how Numba finds nvvm today as well: https://github.com/NVIDIA/numba-cuda/blob/bf487d78a40eea87f009d636882a5000a7524c95/numba_cuda/numba/cuda/cudadrv/libs.py#L54

rwgk self-assigned this Feb 7, 2025

github-actions bot added the triage Needs the team's attention label Feb 7, 2025

rwgk mentioned this issue Feb 12, 2025

[Experimental] Clone numba get_nvvm_path() #447

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RESEARCH] finding libnvvm.so #441

[RESEARCH] finding libnvvm.so #441

rwgk commented Feb 7, 2025 •

edited

Loading

rwgk commented Feb 7, 2025

rwgk commented Feb 10, 2025

rwgk commented Feb 10, 2025

rwgk commented Feb 11, 2025

rwgk commented Feb 11, 2025

rwgk commented Feb 11, 2025

rwgk commented Feb 12, 2025 •

edited

Loading

kkraus14 commented Feb 12, 2025

[RESEARCH] finding libnvvm.so #441

[RESEARCH] finding libnvvm.so #441

Comments

rwgk commented Feb 7, 2025 • edited Loading

rwgk commented Feb 7, 2025

rwgk commented Feb 10, 2025

rwgk commented Feb 10, 2025

rwgk commented Feb 11, 2025

rwgk commented Feb 11, 2025

rwgk commented Feb 11, 2025

rwgk commented Feb 12, 2025 • edited Loading

kkraus14 commented Feb 12, 2025

rwgk commented Feb 7, 2025 •

edited

Loading

rwgk commented Feb 12, 2025 •

edited

Loading