GFS data for MCC WRF forecasts

The forecasts on the MCC web page (http://mathsci.ucd.ie/met/mcc-forecast.html) use global forecast data from NOAA's Global Forecast System (GFS: http://www.emc.ncep.noaa.gov/index.php?branch=GFS). GFS data are available on a global 0.25 degree grid, every hour.

GFS forecast data should be available for download 4 hours after forecast analysis time:

(ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod)

I'd like to download a subset of the GFS data to run the MCC WRF forecast at a higher resolution for Ireland. To do this, I am going to use NOMADS g2subs

http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?dir=%2Fgfs.2016062906

• Use "make subregion": lon:-55 to 25, lat: 25 to 70
• Select the option: "Show the URL only for web programming"
• Open a terminal and set the data URL, then use curl to retrieve the data file:

bash$ URL="http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl..."
bash$ curl “$URL” -o my_grib_file1

This works!

Notes:

• Don't use the analysis data file - it doesn't contain soil data.
• You can see which fields are in the GFS file by using the WPS g2print.exe command.
• Using all levels, all variables, and subregion (lon:-55 to 25, lat: 25 to 70) reduced the size of the analysis file from 176M to 24M. The file size becomes even smaller by requesting less variables.

WRF on SONIC

These are my notes on WRF timing when running on multiple nodes on sonic at UCD.

The domain used for speedup tests was a 300 x 250 grid (namelist.wps):

(jupyter notebook: PlotWPSDomain.ipynb)

SONIC has a mixture of 24-core and 40-core (hyper-threaded) nodes. Only the 40-core nodes are guaranteed to have infiniband. There are cores with more nodes (e.g. the highmem node) but these mess up the MPI messaging. So, to ensure best/consistent performance, use the infiniband queue:

qsub -q infiniband ...

#PBS -l nodes=04:ppn=40
...
module purge
module load WRF

module list
...

time mpirun -np 64 --map-by ppr:1:core wrf.exe

The map by core is required to unsure that hyper-threading is not used.

I ran WRF using different numbers of nodes and cores. The IO time becomes significant for larger number of cores, as the others have to wait during IO. To get a fairer picture of speedup with more cores, I've done the following:

• changed the history value in namelist.input to greater than the forecast range, so wrfout files are not written (apart from analysis)

• stripped out only the compute timings from the rsl.out.0000 log file, using the following command, which skips timings at and 1 minute after each half hour, as the timings are larger then.

• NC=04x64
  grep 'main:.*[2-9]:00' RUNDIR${NC}/rsl.out.0000 | awk '{print $9}' > ComputeTimings.${NC}

Here are the timings I got:

(jupyter notebook: WRFspeedupSONICcomputetimes.ipynb)

CMPI5 Data

Downloading CMPI5 data.

Go here: https://pcmdi.llnl.gov/projects/cmip5/

Click "Login" link at the top-right and use OpenID and password.

Use Search to select model, experiment, variables, then download wget script for files.

Edit wget file to remove unwanted downloads.

I've retrieved HadGEM2-ES mean sea-level pressure (psl) data every six hours for the historical (1950-2005) and RCP85 (2006-2100) ensemble member r1i1p1. I'll use this to look at some storm track behaviour.

WRF3.7 on gaia

Downloaded and installed on gaia: sweeneyc/Code/WRF3.7

module list

Currently Loaded Modulefiles:
  1) gcc/4.9.24.4.1
  2) openmpi/1.8.4
  3) netcdf/2014_10
  4) mpich2/1.5
  5) netcdf-fortran/

export JASPERLIB=/gaia/software/src/grib2/lib
export JASPERINC=/gaia/software/src/grib2/include
export NETCDF=/gaia/software/src/netcdf-fortran-4.4.1/build
export PHDF5=/gaia/software/src/hdf5-1.8.13/build
export NCARG_ROOT=/gaia/software/src/ncl_ncarg-6.2.1

./clean -a

./configure
#35
#1

./compile em_real

This should produce the following in main/*.exe:
wrf.exe -- model executable
real.exe -- real-data initialization
ndown.exe -- off-line, one-way nesting
tc.exe -- TC bogussing, serial only

cd ../WPS

./clean -a

./configure
#1

vim configure.wps
# WRF_DIR                 =       ../WRFV3p7
# NETCDF                  =       /gaia/software/src/netcdf-c/build
# NETCDFF                 =       /gaia/software/src/netcdf-fortran-4.4.1/build
#WRF_INCLUDE         =   ...
                        -I$(NETCDF)/include\
                        -I$(NETCDFF)/include
#WRF_LIB             =   ...
                        -L$(NETCDF)/lib -lnetcdf \
                        -L$(NETCDFF)/lib -lnetcdff -lgomp
# DM_FC               = mpif90 #-f90=gfortran
# DM_CC               = mpicc #-cc=gcc
#Append -lgomp to LDFLAGS

./compile >& log.compile

This should produce:

geogrid.exe -> geogrid/src/geogrid.exe
ungrib.exe -> ungrib/src/ungrib.exe
metgrid.exe -> metgrid/src/metgrid.exe

TEST RUN

In WPS directory, edit namelist.wps:

max_dom = 1,
geog_data_path = '/gaia/software/src/WPS_geog/'

Check domain:

module load ncl_ncarg
export NCARG_ROOT=/gaia/software/src/ncl_ncarg-6.2.1
ncl util/plotgrids_new.ncl

run ./geogrid.exe then make sure that the static file has been created:

geo_em.d01.nc

Download the Tutorial case study data

Link in the AWIP Vtable:

ln -sf ungrib/Variable_Tables/Vtable.AWIP Vtable

Link-in the GRIB data by making use of the script link_grib.csh

./link_grib.csh ../../../DATA/TESTWRF/2000012

Edit namelist.wps, and set the following:

start_date = '2000-01-24_12:00:00',
end_date = '2000-01-25_12:00:00',
interval_seconds = 21600,

Run ungrib to create the intermediate files:

./ungrib.exe >& ungrib_data.log

Run metgrid.exe to interpolate the input data on our model domain:

# module load netcdf-fortran/4.4.1 
./metgrid.exe

Change into WRF directory.

cd ../WRFV3p7/run

Link-in the met_em files created with metgrid.exe

ln -sf ../../WPS/met_em.d01.2000-01* .

namelist.input should already have default settings. Run real.exe. Check that the following two files have been created:

wrfinput_d01
wrfbdy_d01

Run wrf.exe. Check that the following file has been created:

wrfout_d01_2000-01-24_12:00:00

Move output file to Mac, and plot something with Python: