Use config flag for locality-aware mpi #941

geraldc-unm · 2023-06-28T15:27:18Z

Add configure option for locality-aware MPI.

When creating Comm Pkg, must set comm_pkg->use_neighbor = 1, defaults to 0.
Since we observed that node aware performs worse at finest levels, added a lvl threshold setting that controls a switchover to use node aware MPI. By default, all levels will use node_aware.

geraldc-unm · 2023-08-24T16:13:32Z

I can remove the print statements, I was using them to debug

victorapm · 2023-09-07T00:19:33Z

@geraldc-unm, this looks nice!

Which path should I pass to --with-node-aware-mpi-include to config hypre with node-aware MPI support? i.e., Which is the dependency library here?

Thanks!

victorapm · 2023-09-07T00:22:05Z

src/parcsr_mv/_hypre_parcsr_mv.h

@@ -75,6 +86,11 @@ typedef struct _hypre_ParCSRCommPkg
   HYPRE_Int                         num_recvs;
   HYPRE_Int                        *recv_procs;
   HYPRE_Int                        *recv_vec_starts;
+   HYPRE_Int                         use_neighbor;
+#ifdef HYPRE_USING_NODE_AWARE_MPI
+   long                             *global_send_indices;


Maybe it's better to use hypre_uint64, see

hypre/src/utilities/general.h

Line 33 in 36ab29b

typedef uint64_t hypre_uint64;

or hypre_longint here

src/parcsr_mv/new_commpkg.c

src/parcsr_mv/par_csr_communication.c

victorapm · 2023-09-07T00:31:30Z

src/test/ij.c

+      {
+         arg_index++;
+         hypre_HandleNodeAwareSwitchoverThreshold(hypre_handle()) = atoi(argv[arg_index++]);
+         if (hypre_HandleNodeAwareSwitchoverThreshold(hypre_handle()) > 0) {


Should hypre_HandleNodeAwareSwitchoverThreshold be an option internal to BoomerAMG instead of a global variable?

This was a first approach to have the value accessible. I'm not sure how we could have it as part of BoomerAMG but yes it should only be used for BoomerAMG.

Thank you! @liruipeng, do you support the idea of moving this switch threshold inside BoomerAMG?

Hi @victorapm I totally agreed with you. It would be much better if we can move it into the AMG struct. But a technical issue is that we need to access this value in some functions involved in the solve phase but the AMG struct is not passed along to those functions. So we ended up having a global variable for its simplicity. Any suggestions? Thanks.

@liruipeng thanks for providing the context. Now, I can see it better! See my follow-up comments at #941 (comment) and #941 (comment)
Sorry to bug with this, but I think it's better if we have less global variables

geraldc-unm · 2023-09-14T22:02:45Z

@geraldc-unm, this looks nice!

Which path should I pass to --with-node-aware-mpi-include to config hypre with node-aware MPI support? i.e., Which is the dependency library here?

Thanks!

As of right now, MPI Advance must be separately cloned and built. I run CC=mpicc CXX=mpicxx ./configure --enable-persistent --with-MPI-include=<path to mpi advance>/src --with-MPI-lib-dirs=<path to mpi advance>/build/src/ --with-MPI-libs=mpi_advance --with-node-aware-mpi

And I build MPI advance with mkdir build; cd build; CC=mpicc CXX=mpicxx cmake ..; make

victorapm · 2023-09-14T23:08:36Z

Thanks @geraldc-unm!

Where can I find the MPI advance library?

geraldc-unm · 2023-09-14T23:23:37Z

Thanks @geraldc-unm!

Where can I find the MPI advance library?

https://github.com/mpi-advance/locality_aware

Thanks so much for the review

victorapm · 2023-09-20T12:21:44Z

src/parcsr_ls/par_cycle.c

@@ -286,6 +286,15 @@ hypre_BoomerAMGCycle( void              *amg_vdata,
   hypre_GpuProfilingPushRange(nvtx_name);
   while (Not_Finished)
   {
+#ifdef HYPRE_USING_NODE_AWARE_MPI
+      if (level >= hypre_HandleNodeAwareSwitchoverThreshold(hypre_handle()))


@liruipeng This switch threshold could be moved inside the AMG struct

For this one, I think it could be in the AMG struct as long as it can still be set when parsing command line args, where the value comes from

OK. We have the access to hypre_ParAMGData in this function. Do we need hypre_HandleNodeAwareSwitchoverThreshold in other functions that don't have the access?

I can only see it being used at par_cycle.c

So I didn't check these in as permanent changes to the ij driver but to properly use node aware, you must set this hypre_HandleNodeAwareSwitchoverThreshold otherwise it will unoptimally use node aware mpi for every amg level.

This brings up 2 things:

NodeAwareSwitchoverThreshold must be accessible to set like from the ij.c driver. It's quite nice to have it as a command line arg to test different thresholds without recompiling or anything.

We probably want to document this and how to connect to MPI advance. I can compile the steps and maybe we could include it somewhere in the wiki or something? But I don't know how to contribute to the wiki.

Yes, there should be a Set routine for that, e.g., HYPRE_BoomerAMGSetNodeAwareSwitchLevel

Great idea, I'm not sure about the access but if you prepare something in markdown format, we can add it to the wiki.

Documenting it in the Wiki is a good idea.

victorapm · 2023-09-20T12:22:26Z

src/parcsr_ls/par_cycle.c

+#ifdef HYPRE_USING_NODE_AWARE_MPI
+      if (level >= hypre_HandleNodeAwareSwitchoverThreshold(hypre_handle()))
+      {
+         hypre_HandleUsingNodeAwareMPI(hypre_handle()) = 1;


@liruipeng however, this other global variable (which turns off/on node aware MPI) cannot. One idea is to move it to the comm_pkg struct.

PS: we could have it defaulted to zero, and implement something like hypre_CommPkgSetNodeAwareMPI to turn comm_pkg->node_aware_mpi on/off

I think to do that, comm_pkg would need to be accessible within hypre_BoomerAMGCycle which I don't think it is unless I'm mistaken. I don't see it in hypre_ParAMGData. Maybe it could be added as a field there but would need to be properly set and maintained.

You can use

hypre/src/parcsr_mv/par_csr_matrix.h

Line 108 in 36ab29b

#define hypre_ParCSRMatrixCommPkg(matrix) ((matrix) -> comm_pkg)

e.g., hypre_ParCSRMatrixCommPkg(A_array[lvl])

I could be wrong...but to what granularity do we want to control hypre_HandleUsingNodeAwareMPI? If we put in commPkg, it would be per commPkg/Parcsrmatrix. I thought it's a more global thing, like the whole AMG solver. Again, putting it into the AMG struct is ideal, if we can find a clean way to do it.

As far as I can see, the node aware option has been implemented only for the IJ interface. If we want it on the other interfaces in the future, we could extend their respective comm_pkg structs to handle this option.

Note I'm suggesting to put using_node_aware_mpi under the comm_pkg struct here, i.e., not under the AMG struct as for the other case :)

This may still work but it's my understanding that we have multiple matrices per level that each have a comm pkg. The using_node_aware toggle logically applies to a whole level with multiple matrices. So I think we would need to get the comm pkg for A and P and set them both to use node aware in their comm pkgs but I think that would work. What do y'all think?

Thanks @geraldc-unm! I spoke to Rui Peng and shared that's a better approach in my opinion. Let's think about this (moving the variable to inside comm_pkg) a bit more...

geraldc-unm force-pushed the locality-aware branch 2 times, most recently from a918397 to 73daa01 Compare July 3, 2023 22:28

geraldc-unm force-pushed the locality-aware branch from 73daa01 to 749dd41 Compare July 26, 2023 16:40

geraldc-unm force-pushed the locality-aware branch 2 times, most recently from ef381db to 8f70740 Compare August 15, 2023 21:55

geraldc-unm force-pushed the locality-aware branch 3 times, most recently from de4702e to fe2890a Compare August 24, 2023 16:12

geraldc-unm requested review from liruipeng, ulrikeyang and rfalgout August 24, 2023 16:12

victorapm reviewed Sep 7, 2023

View reviewed changes

src/parcsr_mv/new_commpkg.c Outdated Show resolved Hide resolved

victorapm reviewed Sep 7, 2023

View reviewed changes

src/parcsr_mv/par_csr_communication.c Outdated Show resolved Hide resolved

victorapm reviewed Sep 7, 2023

View reviewed changes

Use config flag for locality-aware mpi

316e15d

geraldc-unm force-pushed the locality-aware branch from fe2890a to 316e15d Compare September 13, 2023 05:32

victorapm reviewed Sep 20, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use config flag for locality-aware mpi #941

Use config flag for locality-aware mpi #941

geraldc-unm commented Jun 28, 2023 •

edited

Loading

geraldc-unm commented Aug 24, 2023

victorapm commented Sep 7, 2023

victorapm Sep 7, 2023

victorapm Sep 7, 2023

geraldc-unm Sep 14, 2023

victorapm Sep 14, 2023

liruipeng Sep 19, 2023

victorapm Sep 20, 2023

geraldc-unm commented Sep 14, 2023

victorapm commented Sep 14, 2023

geraldc-unm commented Sep 14, 2023 •

edited

Loading

victorapm Sep 20, 2023

geraldc-unm Sep 20, 2023

liruipeng Sep 20, 2023

victorapm Sep 20, 2023

geraldc-unm Sep 20, 2023

victorapm Sep 20, 2023

liruipeng Sep 20, 2023 •

edited

Loading

victorapm Sep 20, 2023 •

edited

Loading

geraldc-unm Sep 20, 2023

victorapm Sep 20, 2023 •

edited

Loading

liruipeng Sep 20, 2023

victorapm Sep 20, 2023

geraldc-unm Sep 20, 2023

victorapm Sep 20, 2023 •

edited

Loading

Use config flag for locality-aware mpi #941

Are you sure you want to change the base?

Use config flag for locality-aware mpi #941

Conversation

geraldc-unm commented Jun 28, 2023 • edited Loading

geraldc-unm commented Aug 24, 2023

victorapm commented Sep 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

geraldc-unm commented Sep 14, 2023

victorapm commented Sep 14, 2023

geraldc-unm commented Sep 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liruipeng Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

victorapm Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

victorapm Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

victorapm Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

geraldc-unm commented Jun 28, 2023 •

edited

Loading

geraldc-unm commented Sep 14, 2023 •

edited

Loading

liruipeng Sep 20, 2023 •

edited

Loading

victorapm Sep 20, 2023 •

edited

Loading

victorapm Sep 20, 2023 •

edited

Loading

victorapm Sep 20, 2023 •

edited

Loading