-*- text -*--------------------------------------------------
file: README
General info and compilation instructions for MASS2
-------------------------------------------------------------
-------------------------------------------------------------
Battelle Memorial Institute
Pacific Northwest Laboratory
-------------------------------------------------------------
-------------------------------------------------------------
Created November 17, 1999 by William A. Perkins
Last Change: 2017-08-25 10:45:44 d3g096
$Id$
-------------------------------------------------------------

MASS2 is now exclusively parallel.  The serial version is no longer
functional.  

-------------------------------------------------------------
Requirements
-------------------------------------------------------------

CGNS, Version 2.5, or later
---------------------------

Global Arrays, Version 5.0, or later
------------------------------------

PETSc, Version 3.0 or later
---------------------------

-------------------------------------------------------------
Example Configuration
-------------------------------------------------------------


PE10900:
set prefix = /net/flophouse/files0/perksoft/macosx

configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FCFLAGS="-g -fbounds-check -ffpe-trap=invalid,zero,overflow" \
    FC_CPPFLAG="-x f95-cpp-input" \
    PETSC_DIR="$prefix/../petsc-3.4.4" \
    PETSC_ARCH="arch-macosx-fortran-debug" \
    --enable-maintainer-mode \
    --with-cgns="$prefix/cgns2" \
    --with-global-arrays="$prefix/mass2" \
    --prefix="$prefix/mass2"

configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FCFLAGS="-O2 -g -ffpe-trap=invalid,zero,overflow" \
    FC_CPPFLAG="-x f95-cpp-input" \
    PETSC_DIR="$prefix/../petsc-3.4.4" \
    PETSC_ARCH="arch-macosx-fortran-opt" \
    --enable-maintainer-mode \
    --with-cgns="$prefix/cgns2" \
    --with-global-arrays=$prefix/mass2 \
    --prefix="$prefix/mass2"

configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FCFLAGS="-O2 -g -ffpe-trap=invalid,zero,overflow" \
    FC_CPPFLAG="-x f95-cpp-input" \
    PETSC_DIR="$prefix/../petsc-3.4.4" \
    PETSC_ARCH="arch-macosx-fortran-opt" \
    --enable-maintainer-mode \
    --with-cgns="$prefix/cgns2" \
    --with-global-arrays=$prefix/mass2 \
    --prefix="$prefix/mass2"

configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FCFLAGS="-O2 -g -ffpe-trap=invalid,zero,overflow" \
    FC_CPPFLAG="-x f95-cpp-input" \
    LDFLAGS="-L/opt/local/lib" LIBS="-lhdf5" \
    PETSC_DIR="/opt/local/lib/petsc" \
    PETSC_ARCH="" \
    --enable-maintainer-mode \
    --with-cgns="/opt/local" \
    --with-global-arrays="/opt/local"

configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FCFLAGS="-g -ffpe-trap=invalid,zero,overflow" \
    FC_CPPFLAG="-x f95-cpp-input" \
    LDFLAGS="-L/opt/local/lib" LIBS="-lhdf5" \
    PETSC_DIR="/opt/local/lib/petsc" \
    PETSC_ARCH="" \
    --enable-maintainer-mode \
    --with-cgns="/opt/local" \
    --with-global-arrays="/opt/local"

Flophouse, with Intel compiler:

source /usr/software/intel101/x86_64/bin/ifortvars.csh
set path = (/usr/local/openmpi-intel/bin $path)
configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FC_CPPFLAG="-fpp" \
    FCFLAGS="-O2 -fpe0 -heap-arrays" \
    GA_MPILIBS="-llapack -lblas" \
    PETSC_DIR="/usr/local/openmpi-intel" \
    --with-cgns="$prefix/cgns2" \
    --with-global-arrays=/usr/local/openmpi-intel \
    --prefix="$prefix/mass2"


configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FC_CPPFLAG=-fpp \
    FCFLAGS="-g -check bounds -fpe1 -heap-arrays" \
    GA_MPILIBS="-llapack -lblas" \
    PETSC_DIR="/usr/local/openmpi-intel" \
    --with-cgns="$prefix/cgns2" \
    --with-global-arrays=/usr/local/openmpi-intel

set prefix = /net/flophouse/files0/perksoft/linux64
set path = ($prefix/openmpi48/bin $path)
configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FCFLAGS="-g -fbounds-check -ffpe-trap=invalid,zero,overflow" \
    PETSC_DIR="$prefix/../petsc-3.5.2" \
    PETSC_ARCH="linux-gnu48-real-opt" \
    --with-cgns="$prefix/cgns2" \
    --with-global-arrays="$prefix/openmpi48"
    


Grapes:

sh configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FC_CPPFLAG=-fpp \
    FCFLAGS="-O2 -fpe0 -heap-arrays" \
    GA_MPILIBS="-libverbs -llapack -lblas" \
    PETSC_DIR="/home/d3g096/stuff/src/petsc-3.1-p7" \
    PETSC_ARCH="linux-gnu-c-opt" \
    LIBS="-L$HOME/stuff/lib -lhdf5 -lsz -lz" \
    --with-global-arrays=/share/apps/ga/5.0.1/intel \
    --with-cgns=$HOME/stuff


Briareus:

module load intel/11.1
module load mpi/openmpi/1.4.3/intel

sh configure \
    FC=mpif90 \
    MPIFC=mpif90 \
    FC_CPPFLAG=-fpp \
    FCFLAGS="-O2 -fpe0 -heap-arrays" \
    GA_MPILIBS="-libverbs -llapack -lblas" \
    PETSC_DIR="/home/d3g096/stuff/petsc-3.4.4" \
    PETSC_ARCH="intel-fortran-opt" \
    --with-global-arrays=/share/apps/ga/5.0.1/intel \
    --with-cgns="$HOME/stuff/cgns2"

Franklin:

module load PrgEnv-pgi
module load ga/5.0
module load petsc/3.1_O
module load hdf5
sh configure \
    FC=ftn \
    MPIFC=ftn \
    FC_CPPFLAG="-Mpreprocess" \
    FCFLAGS="-fast" \
    LDFLAGS="-L$CRAY_HDF5_DIR/hdf5-pgi/lib" \
    LIBS="-ldl" \
    --with-global-arrays=/usr/common/usg/ga/5.0.2.i4/pgi \
    --with-cgns=$HOME/stuff/$NERSC_HOST \
    --prefix=$HOME/stuff/$NERSC_HOST


-------------------------------------------------------------
The stuff below is very dated ...
-------------------------------------------------------------

-------------------------------------------------------------
CGNS
-------------------------------------------------------------

MASS2 also requires Version 2.1 of the CGNS library.  This can be
obtained from http://www.icemcfd.com/cgns.  On Linux platforms, it is
by default compiled with gcc and g77, and the -fno-second-underscore
flag is used on g77.  Compile CGNS as follows

    SGI: make SYSTEM=mips4_n32 ==> lib/libcgns.mips4_n32.a
    Intel Linux: make SYSTEM=linux ==> lib/libcgns.linux.a
    Alpha Linux: cp Makefile.linux Makefile.alinux; make SYSTEM=alinux ==> lib/libcgns.alinux.a

The second to last line in the MASS2 configuration file controls the
CGNS output.  The following is expected to be on that line:

    1. (T/F) Flag to turn CGNS output on.  CGNS output can be on at
       the same time as NetCDF, but there is really no reason to do
       that.  

    2. (T/F) Flag to cause CGNS output to be cell-centered.  If true,
       the CGNS output will have a grid like the input grid (and
       output in gridplot1.dat) and all output values will be
       cell-centered.  Otherwise, the output grid will be like that
       produced in the NetCDF file: vertexes at model cell centers
       and artifical edges.  

    3. (T/F) Flag to add descriptions to solution output fields.  If
       true, each field of each solution has descriptors for the field
       name and the units.  This is meant to duplicate the NetCDF
       behavior.  It does not take a great deal of extra space, but
       the extra nodes can cause the model to slow down.

    4. (integer) Maximum number of time planes to put in the CGNS
       output files.  The number of time planes can be limited to
       avoid the problem that CGNS has with lots of nodes in the
       file.  

CGNS has some problems when there are a lot of nodes in the file.  Is
some tests of SAC problems, putting 1050 time planes in a single CGNS
output file doubled the MASS2 execution time (and the extra time was
not spent in I/O).  Whereas, putting a maximum of 100 time planes in
11 files increased the MASS2 execution time only 3%.  

-------------------------------------------------------------
compilation via GNU autotools
-------------------------------------------------------------

MASS2 is configured and compiled using the GNU autotools.  If you
check MASS2 out of CVS, you must build the configure script on a
system that has the GNU autotools installed (Linux systems usually
do).  The configure script is built by the command

    sh bootstrap.sh

You do not have to do this on the machine you which to compile for;
the only requirement is that autotools are installed (autoconf and
automake).  

To compile MASS2, the configure script must be run on the architecture
you wish to run MASS2.  The following architectures and compilers are
supported: 

    * Silicon Graphics with SGI Fortran 90 compiler (f90),
    * OSF4 on DEC/Compaq/HP Alpha with the DEC/Compaq/HP Fortran 90
      compiler (f90, f95), 
    * Linux on DEC/Compaq/HP Alpha with the DEC/Compaq/HP Fortran 90
      compiler (fort),
    * Linux on Intel with any of 
        * Absoft Fortran 95/90 (version 7.5 or above) compiler (f90, f95),
        * Intel Fortran (version 7.0 or above) compiler (ifc) , or
        * Lahey Fortran 95 Pro (version 6.1 or above) compiler (lf95).

If you try to configure without when not working with one of these
architectures, the configure script will crash with an error.  If
however, you set the F77 environment variable to the compiler you want
(or use --with-f90, see below) configure will try to make it work with
some (probably wrong) compilation flags.  You can also use the F77
environment variable (or --with-f90) to choose the compiler when there
are multiple choices.

The configure script has several options.  To get a complete list,
issue the command

    configure --help

The options specific to the compilation of MASS2 are as follows:

    --with-f90=cmd

        Use "cmd" as the Fortran 90 compiler command.

    --enable-f90-optimize

        Optimization is turned on using flags appropriate to the
        compiler/architecture.

    --enable-f90-debug (default)

        Debugging flags are included by default.  If
        --enable-f90-optimize is specified, the debugging flags are
        not included.  Use --enable-f90-debug to include debugging
        with optimization.

    --enable-f90-profile

        Turn on profiling.

    --with-extra-f90flags=flags

        Use the specified flags in addition to the flags added by the
        above options.  This is mostly used to add CPU specific
        optimization options.  For example, with the Intel ia32
        compiler, these flags are used to compile two different
        executables specialized for the Pentium 3 and Pentium 4:

            P3: -tpp6 -xK
            P4: -tpp7 -xW

        The same thing can be accomplished by setting F90FLAGS or
        FFLAGS in the environment before running configure.

    --with-netcdf=dir

        NetCDF is required for compilation.  This option specifies
        where NetCDF libraries and headers are located.  The specified
        directory is expected to have include and lib subdirectories.

    --with-cgns=dir

        CGNS is required for compilation.  This option specifies where
        the CGNS libraries were unpacked and built.  The specified
        directory is expected to have the structure of the CGNS source
        as it is extracted.

    --with-sixpack=dir
 
    --with-aztec=dir

Some typical configurations

    configure --with-netcdf=/home/perk/Projects/ParallelMASS2/src/netcdf-3.4
    --enable-optimize --enable-profile --with-cgns=/usr/unsupported/CGNSLib
    --with-sixpack=/home/perk/Projects/ParallelMASS2/src/sixpack-2.3.7
    --with-aztec=/home/perk/Projects/ParallelMASS2/src/aztec

    configure --with-netcdf=/home/perk/Projects/ParallelMASS2/src/netcdf-3.4 \
        --with-cgns=/usr/unsupported/CGNSLib --with-f90=ifc \
        --enable-f90-optimize --enable-f90-debug
        ----with-extra-f90flags="-tpp7 -xW"

On leechong w/ Intel Compiler:

    setenv FFLAGS ""
    configure \
        --with-netcdf=/home/perk/Projects/ParallelMASS2/src/netcdf-3.4 \
        --with-cgns=/usr/unsupported/CGNSLib \
        --with-cgns-lib=cgns.LINUX \
        --with-f90=ifc \
        --enable-f90-optimize

On mcperk w/ Absoft Compiler:

setenv FCFLAGS ""
configure \
    --with-netcdf=/home/perk/Projects/ParallelMASS2/src/netcdf-3.5.1/macosx \
    --with-cgns=/usr/unsupported/CGNSLib \
    --with-cgns-lib=cgns.DARWIN \
    --with-f90=f95 \
    --enable-f90-optimize \
    --enable-f90-debug

setenv FCFLAGS "-ffpe-trap=invalid,zero,overflow"
configure \
    --with-netcdf=/home/perk/Projects/ParallelMASS2/src/netcdf-3.5.1/macosx \
    --with-cgns=/usr/unsupported/CGNSLib \
    --with-cgns-lib=cgns.DARWIN \
    --with-f90=gfortran \
    --enable-f90-debug \
    --enable-f90-optimize

On mac laptop:

    setenv FCFLAGS "-fno-second-underscore -ffpe-trap=invalid,zero,overflow"
    configure \
        --with-netcdf=/Users/d3g096/stuff \
        --with-cgns=/Users/d3g096/stuff/src/cgnslib_2.5 \
        --with-f90=gfortran-mp-4.3 

    setenv FCFLAGS "-fno-second-underscore"
    configure \
        --with-netcdf=/Users/d3g096/stuff \
        --with-cgns=/Users/d3g096/stuff/src/cgnslib_2.5 \
        --with-f90=g95

On PE10900:

    setenv FC "gfortran-mp-4.3"
    setenv FCFLAGS "-g -fbounds-check -fno-second-underscore -ffpe-trap=invalid,zero,overflow"
    configure \
        --with-netcdf=/home/d3g096/Projects/MASS2/src/netcdf-3.6.2/macosx \
        --with-cgns=/usr/unsupported/CGNSLib

    setenv FC "ifort"
    setenv FCFLAGS "-g -CB -fpe1"
    setenv FCFLAGS "-O3 -fpe0"
    configure \
        --with-netcdf=/home/d3g096/Projects/MASS2/src/netcdf-3.6.2/macosx \
        --with-cgns=/usr/unsupported/CGNSLib

On bearflag:

    setenv ABSOFT /usr/software/absoft10
    set path = ($path ${ABSOFT}/bin)
    sh bootstrap.sh
    configure \
        --with-f90=f95 \
        --with-netcdf=/home/perk/Projects/ParallelMASS2/src/netcdf-3.6.1/linux64 \
        --with-cgns=/usr/unsupported/CGNSLib \
        --with-cgns-lib=cgns.LINUX64 \
        --enable-f90-optimize \
        --enable-f90-debug
    

    sh bootstrap.sh
    set path = ( /usr/software/intel101/x86_64/bin $path)
    setenv FC "ifort"
    setenv FCFLAGS "-g -fpe1 -CB"
    setenv FCFLAGS "-O3 -fpe0"
    configure \
        --with-f90=ifort \
        --with-netcdf=/home/d3g096/Projects/MASS2/src/netcdf-3.6.2/linux64 \
        --with-cgns=/usr/unsupported/CGNSLib \
        --with-cgns-lib=cgns.LINUX64 \
        --enable-f90-debug \
        --enable-f90-optimize

    # Using the GNU compiler (does not seem to work with CGNS output)
    sh bootstrap.sh
    setenv FCFLAGS "-ffpe-trap=invalid,zero,overflow"
    configure \
        --with-f90=gfortran \
        --with-netcdf=/home/d3g096/Projects/MASS2/src/netcdf-3.6.2/linux64 \
        --with-cgns=/usr/unsupported/CGNSLib \
        --with-cgns-lib=cgns.LINUX64 \
        --enable-f90-debug \
        --enable-f90-optimize


-------------------------------------------------------------
distributing MASS2
-------------------------------------------------------------
If the MASS2 source code is to be sent to someone, or it is to be
transferred to and compiled on a machine without GNU autotools, one
can generate a "distribution" file automatically with make.  This is
to ensure all necessary files are included when the code is
transferred.  

You must prepare a MASS2 distribution on a machine that has GNU
autotools installed.  Once MASS2 is checked out of CVS, build the
configure scripts with

    sh bootstrap.sh

Then configure MASS2 appropriately, as discussed above, and make the
distribution file with

    make dist

This will make a gzip'd tar file named something like
mass2-0.30.tar.gz.  It contains the MASS2 distribution and should have
all the necessary files to build MASS2. 

-------------------------------------------------------------
old compilation  -- no longer maintained
-------------------------------------------------------------

By default all of the makefiles will produce an executable called
`mass2_v027'.  This can be changed by setting the `TARGET' macro on
the command line.  Some suggested `TARGET's are shown below (using the
condor codes for OpSys and Arch). Currently the following platforms
are supported

    IRIX:   

        make -f makefile [DEBUG=-g [-O]] [TARGET=mass2.IRIX65.SGI]

    DEC OSF/1:      

        make -f makefile.dec F90=f95 [DEBUG=-g [-O]] [TARGET=mass2_v025.dec]

        Compile on tophet.  Can run on both OSF/1 and Linux alpha.

    Linux i386:     

        make -f makefile.linux [DEBUG=-g] [TARGET=mass2.LINUX.INTEL]

        This is for the Absoft Pro Fortran 90/95 compiler, currently
        version 7.5. 

        make -f makefile.intel [DEBUG=-g] [TARGET=mass2.LINUX.INTEL]

        This is for the Intel Fortran 90 compiler, version 6.0.1. 

        make -f makefile.lahey [DEBUG=-g] [TARGET=mass2.LINUX.INTEL]

        This is for the Lahey Fortran 95 compiler, version 6.10a

    Linux alpha:    

        make -f makefile.dec [DEBUG=-g [-O]] [TARGET=mass2.LINUX.ALPHA]

        This uses the DEC Fortran 90 compiler for Linux.  This is
        installed only on gehenna. 

    Sun Solaris SPARC:

        make -f makefile.sun [DEBUG=-g] [TARGET=mass2.SOLARIS27.SUN4u]

        This is for the Sun Fortran 90 compiler.  This can be found on
        artemis. 

    Windows:        

        nmake /f mass2_v025.mak  "CFG=mass2_v025 - Win32 Release"
        nmake /f mass2_v025.mak  "CFG=mass2_v025 - Win32 Debug"

        Compile on r101243.