GFS data for MCC WRF forecasts

The forecasts on the MCC web page (http://mathsci.ucd.ie/met/mcc-forecast.html) use global forecast data from NOAA's Global Forecast System (GFS: http://www.emc.ncep.noaa.gov/index.php?branch=GFS). GFS data are available on a global 0.25 degree grid, every hour.

GFS forecast data should be available for download 4 hours after forecast analysis time:

(ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod)

I'd like to download a subset of the GFS data to run the MCC WRF forecast at a higher resolution for Ireland. To do this, I am going to use NOMADS g2subs

http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?dir=%2Fgfs.2016062906

• Use "make subregion": lon:-55 to 25, lat: 25 to 70
• Select the option: "Show the URL only for web programming"
• Open a terminal and set the data URL, then use curl to retrieve the data file:

bash$ URL="http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl..."
bash$ curl “$URL” -o my_grib_file1

This works!

Notes:

• Don't use the analysis data file - it doesn't contain soil data.
• You can see which fields are in the GFS file by using the WPS g2print.exe command.
• Using all levels, all variables, and subregion (lon:-55 to 25, lat: 25 to 70) reduced the size of the analysis file from 176M to 24M. The file size becomes even smaller by requesting less variables.

WRF on SONIC

These are my notes on WRF timing when running on multiple nodes on sonic at UCD.

The domain used for speedup tests was a 300 x 250 grid (namelist.wps):

(jupyter notebook: PlotWPSDomain.ipynb)

SONIC has a mixture of 24-core and 40-core (hyper-threaded) nodes. Only the 40-core nodes are guaranteed to have infiniband. There are cores with more nodes (e.g. the highmem node) but these mess up the MPI messaging. So, to ensure best/consistent performance, use the infiniband queue:

qsub -q infiniband ...

#PBS -l nodes=04:ppn=40
...
module purge
module load WRF

module list
...

time mpirun -np 64 --map-by ppr:1:core wrf.exe

The map by core is required to unsure that hyper-threading is not used.

I ran WRF using different numbers of nodes and cores. The IO time becomes significant for larger number of cores, as the others have to wait during IO. To get a fairer picture of speedup with more cores, I've done the following:

• changed the history value in namelist.input to greater than the forecast range, so wrfout files are not written (apart from analysis)

• stripped out only the compute timings from the rsl.out.0000 log file, using the following command, which skips timings at and 1 minute after each half hour, as the timings are larger then.

• NC=04x64
  grep 'main:.*[2-9]:00' RUNDIR${NC}/rsl.out.0000 | awk '{print $9}' > ComputeTimings.${NC}

Here are the timings I got:

(jupyter notebook: WRFspeedupSONICcomputetimes.ipynb)