New Nalu Trilinos config issues

[spdomin]

Greetings,

On 09/20/2023, our Nalu/Trilinos nightly process began to fail. As I was out of the lab, I was unable to report the issue.

The Nalu configuration error is as follows and something that is commonly seen when a configuration step failed in Trilinos and the Trilinos installation directory was not set up properly,

find: ‘./CMakeFiles’: No such file or directory
   Trilinos_DIR = /home/spdomin/gitHubWork/scratch_build/install/gcc10.3.0/Trilinos_nightly_release
CMake Error at CMakeLists.txt:20 (find_package):
  Could not find a package configuration file provided by "Trilinos" with any
  of the following names:

    TrilinosConfig.cmake
    trilinos-config.cmake

  Add the installation prefix of "Trilinos" to CMAKE_PREFIX_PATH or set
  "Trilinos_DIR" to a directory containing one of the above files.  If
  "Trilinos" provides a separate development package or SDK, be sure it has
  been installed.

However, if I look at this path, I see a legit structure:

[spdomin@hpws247 build_nightly]$ ls  /home/spdomin/gitHubWork/scratch_build/install/gcc10.3.0/Trilinos_nightly_release
bin  include  lib64  TrilinosRepoVersion.txt

Just like a lagged version:

[spdomin@hpws247 build_nightly]$ ls  /home/spdomin/gitHubWork/scratch_build/install/gcc10.3.0/Trilinos_stable_release/
bin  include  lib  TrilinosRepoVersion.txt

Moreover, the structures seem filled (/bin shown as an example):

[spdomin@hpws247 build_nightly]$ ls  /home/spdomin/gitHubWork/scratch_build/install/gcc10.3.0/Trilinos_stable_release/bin/
cth_pressure_map  epu   exodiff        hpcbind  io_modify  kokkos_launch_compiler  nem_spread    pepu          skinner
decomp            epup  getopt.seacas  io_info  io_shell   nem_slice               nvcc_wrapper  shell_to_hex  sphgen

and

[spdomin@hpws247 build_nightly]$ ls  /home/spdomin/gitHubWork/scratch_build/install/gcc10.3.0/Trilinos_nightly_release/bin/
cth_pressure_map  epu   exodiff        hpcbind  io_modify  kokkos_launch_compiler  nem_spread    pepu          skinner
decomp            epup  getopt.seacas  io_info  io_shell   nem_slice               nvcc_wrapper  shell_to_hex  sphgen

Any help would be appreciated. I can probably perform a bisect as well....

Steps to Reproduce

Bad:

NaluCFD/Nalu SHA1: 7f2be162e938da76271a4200d0aa721b0465e2ff
Trilinos/develop SHA1: dc5a72a81842f419d68820b80f20895f8f35dcba

Good:

NaluCFD/Nalu SHA1: 7f2be162e938da76271a4200d0aa721b0465e2ff
Trilinos/develop SHA1: 63f536b25579f1139d0c3c3e35f5da759a45d7fc

[spdomin]

Nalu Config: https://github.com/NaluCFD/Nalu/blob/master/build/do-configNalu_release
Trilinos Config: https://github.com/NaluCFD/Nalu/blob/master/build/do-configTrilinos_release

[spdomin]

After a bisect, I see the following (any ideas @bartlettroscoe - since I cannot locate [email protected] github user name)... Note, this bisect may be flawed as I have never tried one on a config breakage...

5cdb42c is the first bad commit
commit 5cdb42c
Author: srberna [email protected]
Date: Mon Sep 11 09:23:14 2023 -0600

Fix per suggestion on PR review

Also add release notes to RELEASE_NOTES file. Revert original commit
since I committed all the whitespace changes in RELEASE_NOTES file by
mistake.

:100644 100644 99d17f564f405d80de4078bd7d705bc22158557c d3bfc048ea2070165b65b1e00dc091203f80bdd6 M CMakeLists.txt
:100644 100644 0e25fb33c57d64d95f9048e2a1a303713002172a a2c747e5b46a480647c2e6bd96d0e71e3e24a1c2 M RELEASE_NO

[spdomin]

Adding @srbdev

[jhux2]

@trilinos/framework Can someone help @spdomin out?

[bartlettroscoe]

@spdomin, this is a known issue when moving to the usage of the standard CMake module GNUInstalDirs.cmake. See:

• Improve release note for using GNUInstallDirs by default (#12104) #12321

and the underlying issue:

• https://gitlab.kitware.com/cmake/cmake/-/issues/25157#note_1418009

NOTE: Albany had the exact same issue:

• Failing Albany configures due to incorrect Trilinos version sandialabs/Albany#982 (comment)

Is Nalu getting compilers from Trilinos by calling find_package(Triilnos ...) and then defining the compilers? If not, you can define the compilers first (e.g. project(Nalu LANGAUGES C CXX ...)) then call find_package(Trilinos ...), it should find TrilinosConfig.cmake under <trilinosInstallDir>/lib64/cmake/Trilinos/ automatically (but just adding <trilinosInstallDir> to CMAKE_PREFIX_PATH).

As for getting compilers from Trilinos, consider:

• Address providing compilers in TrilinosConfig.cmake and <Package>Config.cmake files #12306
• Address providing compilers to downstream CMake projects TriBITSPub/TriBITS#545

[spdomin]

@bartlettroscoe, yes, Nalu wants to be built after Trilinos and "inherit" all of what Trilinos used. Your suggestion above (as with the Albany suggestion of moving project() first) seems to work. I will process a push, and a nightly test process soon...

I will need to digest the other suggestions you provided above.

[bartlettroscoe]

@bartlettroscoe, yes, Nalu wants to be built after Trilinos and "inherit" all of what Trilinos used. Your suggestion above (as with the Albany suggestion of moving project() first) seems to work. I will process a push, and a nightly test process soon...

@spdomin, if Nalu wants to keep pulling compilers from installed Trilinos, you should chime in to:

• Address providing compilers in TrilinosConfig.cmake and <Package>Config.cmake files #12306

and let Trilinos know.

CUDA and GNUInstallDirs.cmake makes it unattractive to try to get compilers from Trilinos moving forward.

[spdomin]

Given: NaluCFD/Nalu#664

100% tests passed, 0 tests failed out of 84

NaluCFD/Nalu SHA1: 2de1b4268c57ed71103390954d975d595d399093
Trilinos/develop SHA1: 4ee44e19102938643860e317a93989c212620079