Friday, 8 July 2016

WRF, ORR2, Infiniband, Intel

Orr2 is the HPC cluster of the UCD School of Mathematics and Statistics.

 I'd like to install WRF as an aid in teaching the MSc Climate Change.

ORR2 architecture

Orr2 consists of 20 compute nodes on the following queues ('qstat -f' will show all queues):
  • 2x64.q
    compute-0-0.local, compute-0-1.local
    2 nodes with 64-cores: AMD Bulldozer/Opteron. 2.1GHz, 128GB.
  • 68nht.q
    compute-1-0.local
    1 node with 12 cores: Intel Nehalem. 2.66GHz, 24GB, Infiniband
    compute-1-1.local - compute-1-7.local
    7 nodes with 8 cores: Intel Nehalem. 2.4GHz, 24GB, Infiniband 
  • 6x8i.q
    compute-2-0.local - compute-2-5.local
    6 nodes with 8 cores: Intel Core2. 2.5GHz, 32GB.
  • 4x8a.q
    compute-3-0.local - compute-3-3.local
    4 nodes with 8 cores: AMD Shanghai. 2.6GHz, 32GB.
Orr2 has Intel compilers and MVAPICH (MPI over InfiniBand). The Orr2 documentation has information on using the Intel compilers with mvapich.

Download WRF, geog and libraries

  • WRFV3.8.TAR.gz (39M)
  • WPSV3.8.TAR.gz (618K)
  • geog_complete.tar.bz2 (2.3G)
  • jasper-1.701.0.tar.gz (1M)
  • libpng-1.2.12.tar.gz (812K)
  • zlib-1.2.3.tar.gz (491K)

Download NetCDF

NetCDF is available from the authors/maintainers of NetCDF, but WRF recommend using the (single file) version 4.1.3, available here:
  • netcdf-4.1.3.tar.gz (11M)

Set up Intel environment on Orr2

. /share/apps/intelPS/composer_xe_2015/bin/compilervars.sh intel64
export PATH=/share/apps/mvapich2intel/bin:$PATH # add mpif90, mpicc to path

The mvapich compilers are in /share/apps/mvapich2intel. Check intel compilers and preprocessor:
[conor@orr2 ~]$ mpif90 --version
ifort (IFORT) 15.0.3 20150407
[conor@orr2 ~]$ mpicc --version
icc (ICC) 15.0.3 20150407

GNU mpi is loaded on Orr2 by default. Unload the module to make sure you're using Intel mpi:
module unload rocks-openmpi

Build NetCDF


Extract files:
cd /home/conor/LIBRARIES
tar -xzf netcdf-4.1.3.tar.gz
cd netcdf-4.1.3

Set environment variables:
export DIR=/home/conor/LIBRARIES
export CC=icc
export CXX=icpc
export CFLAGS='-O2 -fpic'
export CXXFLAGS='-O2 -fpic'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O2 -fpic'
export CPP='icc -E'
export CXXCPP='icpc -E'
note: Intel compiler option '-fpic' generates position-independent code, and must be used when building shared objects.

 Configure, pointing to required install directory: 
./configure --prefix=$DIR/netcdf --disable-dap --disable-netcdf-4 --disable-shared

 Make and check:
make
make check
make install
export PATH=$DIR/netcdf/bin:$PATH
export NETCDF=$DIR/netcdf

Build zlib

This is a compression library necessary for compiling WPS (specifically ungrib) with GRIB2 capability. Assuming all the environment variables from the NetCDF install are already set, you can move on to the commands to install zlib.

 Extract and change into directory:
tar -xzf zlib-1.2.3.tar.gz
cd zlib-1.2.3

 Configure and make: 
export LDFLAGS=-L$DIR/zlib/lib
export CPPFLAGS=-I$DIR/zlib/include
./configure --prefix=$DIR/zlib
make
make check
make install

Build libpng

This is a compression library necessary for compiling WPS (specifically ungrib) with GRIB2 capability. Assuming all above environmental variables have been set, you can build as follows:
tar -xzf libpng-1.2.12.tar.gz 
cd libpng-1.2.12
./configure --prefix=$DIR/libpng
make
make check
make install

Build JasPer

Another compression library necessary for GRIB2 capability. Assuming all above environmental variables have been set, you can build as follows:
tar -xzf jasper-1.701.0.tar.gz
cd jasper-1.701.0
./configure --prefix=$DIR/jasper
make
make install
export JASPERINC=${DIR}/jasper/include
export JASPERLIB=${DIR}/jasper/lib

Build WRF

It is important to compile WRFV3 before WPS as WPS makes use of the external I/O libraries in the WRFV3/external directory that are built when WRF is installed. Change into the WRF directory and build:
./configure
Option: 15. (dmpar) INTEL (ifort/icc)
./compile em_real >& log.compile

...it's took AGES (1 hour 20 minutes), but it worked, producing the following executables;
ls -hl main/*.exe
43M main/ndown.exe
43M main/real.exe
42M main/tc.exe
50M main/wrf.exe

Build WPS

Make sure that the Intel compiler environment is set up, and that environment variables are set:
. /share/apps/intelPS/composer_xe_2015/bin/compilervars.sh intel64
export PATH=/share/apps/mvapich2intel/bin:$PATH # add mpif90, mpicc to path
export DIR=/home/conor/LIBRARIES
export PATH=$DIR/netcdf/bin:$PATH
export NETCDF=$DIR/netcdf
export JASPERINC=${DIR}/jasper/include
export JASPERLIB=${DIR}/jasper/lib

 Then configure and compile:
./configure
  17.  Linux x86_64, Intel compiler    (serial)
./compile >& log.compile

If all goes well, you should get the following executables:
geogrid.exe -> geogrid/src/geogrid.exe
metgrid.exe -> metgrid/src/metgrid.exe
ungrib.exe -> ungrib/src/ungrib.exe

That's it! Time to run a forecast...