Issue with CUDA Compilation and GPU Execution after Updating to Castro 24.01

[xiaoxiangyeyu-zxy]

Hello Castro Team,

I hope this message finds you well. I am currently facing an issue with the Castro software after updating to version 24.01. In the previous version (23.10), I was able to successfully compile using CUDA and run the application on GPU. However, upon upgrading to version 24.01, I encounter an error during the initial execution.

The error message is as follows:
"amrex::Abort::0::GPU last error detected in file ../../../external/amrex/Src/Base/AMReX_GpuLauchFunctsG.H line 885: named symbol not found !!!"

I attempted to address the issue by rolling back to previous versions such as 23.11, 23.10, and 23.09, but unfortunately, the problem persists. My system configuration includes Ubuntu 22.04, GCC 11.4.0, CUDA 12.3, and NVIDIA driver version 545.23.08. The appendix is "Backtrace" and "GNUmakefile" for Sedov case.

I would appreciate any guidance or assistance in resolving this matter. Thank you for your time and support.

Best regards,

ISSUE.zip

[zingale]

can you do:

./Castro3d.gnu.CUDA.ex --describe

and post the output here?

[xiaoxiangyeyu-zxy]

Thanks for your reply.
The output is:

Castro Build Information

build date: 2024-01-17 05:58:35.236024
build machine: Linux ubuntu 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
build dir: /home/xiaoyu/Castro/Exec/hydro_tests/Sedov
AMReX dir: ../../../external/amrex

COMP: gnu
COMP version: 11.4.0

C++ compiler: nvcc
C++ flags: -ccbin=g++ -Xcompiler='-Werror=return-type -gdwarf-4 -O3 -std=c++17 -pthread -std=c++17' --std=c++17 -Wno-deprecated-gpu-targets -m64 --generate-code arch=compute_70,code=sm_70 -maxrregcount=255 --expt-relaxed-constexpr --expt-extended-lambda --forward-unknown-to-host-compiler -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -Xcudafe --diag_suppress=implicit_return_from_non_void_function --Werror cross-execution-space-call -lineinfo --ptxas-options=-O3 --use_fast_math --Werror ext-lambda-captures-this --display-error-number --diag-error 20092 -x cu -dc -DBL_NO_FORT -DAMREX_USE_CUDA -DAMREX_USE_GPU -DBL_COALESCE_FABS -DAMREX_GPU_MAX_THREADS=256 -DAMREX_USE_GPU_RDC -DBL_SPACEDIM=3 -DAMREX_SPACEDIM=3 -DBL_FORT_USE_UNDERSCORE -DAMREX_FORT_USE_UNDERSCORE -DBL_Linux -DAMREX_Linux -DAMREX_DIMENSION_AGNOSTIC -DNDEBUG -DAMREX_NO_PROBINIT -DSPONGE -DSTRANG -DSCREEN_METHOD=SCREEN_METHOD_screen5 -DNAUX_NET=0 -Itmp_build_dir/s/3d.gnu.CUDA.EXE -I. -I../../../external/amrex/Src/Base -I../../../external/amrex/Src/Base/Parser -I../../../external/amrex/Src/AmrCore -I../../../external/amrex/Src/Amr -I../../../external/amrex/Src/Boundary -I../../../external/Microphysics/util -I../../../external/Microphysics/util/gcem/include -I./ -I../../../Source/driver -I../../../Source/hydro -I../../../Source/problems -I../../../Source/sources -I../../../external/Microphysics/EOS -I../../../external/Microphysics/EOS/gamma_law -I../../../external/Microphysics/networks/general_null -I../../../external/Microphysics/EOS -I../../../external/Microphysics/networks -I../../../external/Microphysics/interfaces -I../../../external/Microphysics/constants -Itmp_build_dir/castro_sources/3d.gnu.CUDA.EXE -I../../../external/amrex/Tools/C_scripts -isystem /usr/local/cuda/include

Fortran comp: gfortran
Fortran flags: -g -O3 -ffree-line-length-none -fno-range-check -fno-second-underscore -fimplicit-none

Link flags: -L. -L/usr/local/cuda/lib64
Libraries: -lcuda -lcurand

EOS: ../../../external/Microphysics/EOS/gamma_law
NETWORK: ../../../external/Microphysics/networks/general_null

Castro git describe: 23.12-dirty
AMReX git describe: 23.12
Microphysics git describe: 23.12

[zingale]

Unfortunately, I am unable to reproduce this. You said that the older versions that used to work no longer do -- is that right?

when you test back on 23.10, are you also putting amrex and Microphysics on 23.10?

Also, your git hashes indicate that you have local changes to Castro that are not committed. What are those changes?

[xiaoxiangyeyu-zxy]

Yes, you are correct. And when rolling back Castro to version 23.10, I made sure to also roll back the versions of amrex and Microphysics accordingly. As for the local changes, I think I only modified the GNUmakefile.

There's another peculiar aspect to this issue—Castro is currently running fine on my local machine, but encounters difficulties on the server. Both the local machine and the server are running the same version of Ubuntu, and the error messages originate from the server.

[xiaoxiangyeyu-zxy]

We have successfully resolved the issue. The solution involves adding CUDA_ARCH = n to the GNUmakefile, where n corresponds to the respective GPU architecture.

Thank you for your assistance.
Best regards.

[zingale]

oh, excellent. For reference, can you tell us what value of CUDA_ARCH you needed to use?

[xiaoxiangyeyu-zxy]

Certainly! When the GPU is RTX4090, the value of CUDA_ARCH that resolves the issue is 89.
And this can be reference:https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/.

[zingale]

can you also share the output of nvidia-smi?

[xiaoxiangyeyu-zxy]

Hello, here is the output of "nvidia-smi":

[zingale]

thanks, I've updated the docs here:
#2715