Run National Water Model (WRFHydro) on HPC through CyberGIS-Compute Service

The goal of this notebook is to show the steps to run an example National Water Model (WRFhydro) model on HPC resources through the CyberGIS-Compute Service. This notebook uses wrfhydropy, a Python wrapper for WRFHydro, in model preprocessing and postprocessing, and the resulting ready-to-run model is handed over to CyberGIS-Compute Service for execution on a supported HPC resource (Virtual Roger/Keeling at UIUC or XSEDE COMET at SDSC). This example is adapted from the "ex_01_end_to_end.ipynb" notebook from wrfhydropy official github repo https://github.com/NCAR/wrf_hydro_py, and users are encouraged to refer to the tutorials there to get familair with wrfhydropy usages.

Preparations

This section is to setup environment, checkout model source code, download test case data.

import os
import sys
import time
import uuid
import pathlib
import pickle
from pprint import pprint
import wrfhydropy
%matplotlib inline

Set up environments

# local workspace to store model, data, configurations and results
workspace = "/home/jovyan/work/workspace"
# WRFHydro branch/tag name (eg: nwm-v2.0) or commit id at https://github.com/NCAR/wrf_hydro_nwm_public
model_version = "4d0c8ad534aa5f53a392008b1144170d42e7307c"
# URL to the test case data stored on HydroShare
testcase_url = "https://www.hydroshare.org/resource/33352fa9697d43718abe271d70f5ca32/data/contents/croton_river.zip"

# create a folder with a random name in the workspace
experiment_dir = pathlib.Path(
    os.path.join(workspace, "wrfhydro_{}_{}".format(str(int(time.time())), str(uuid.uuid4())[-4:])))
if not experiment_dir.exists():
    os.makedirs(str(experiment_dir))
notebook_dir = os.getcwd()
os.chdir(str(experiment_dir))
pprint("Experiment dir: {}".format(experiment_dir))

Download model source code from NCAR WRFHydro repo and check out the selected branch or commit

# path to store model sourde code
model_repo = experiment_dir / 'wrf_hydro_nwm_public'

! git clone https://github.com/NCAR/wrf_hydro_nwm_public.git {model_repo}
! cd {model_repo} && git checkout {model_version}

Retrieve testcase data from HydroShare

# Path to save test case data (domain)
domain_dir = experiment_dir / 'domain'

Speed up data downloading with multi-threaded "aria2c" if installed, otherwise just use wget

%%bash -s "$testcase_url" "$experiment_dir" "$domain_dir"
if ! type aria2c > /dev/null; then
  wget $1 -O $2/testcase.zip
else
  aria2c -x 8 -d $2 -o testcase.zip $1
fi

Unzip test case into domain folder

! unzip -d {domain_dir} {experiment_dir}/testcase.zip

Model Configurations with "wrfhydropy"

This section is to configure the model using wrfhydropy. We will have a ready-to-run model.

# set to use built-in configuration: National Water Model - Analysis & Assimilation
model_config = 'nwm_ana'

Domain

# create Domain obj
domain = wrfhydropy.Domain(
    domain_top_dir=domain_dir,
    domain_config=model_config)

Model

# create Model obj, specify source code and compiler
model = wrfhydropy.Model(
    model_repo / 'trunk/NDHMS',
    compiler='gfort',
    model_config=model_config)

Compile source code

compile_dir = experiment_dir / 'compile'
model.compile(compile_dir)

Set up model runs (Jobs)

# the mpirun command is only for local run
exe_cmd = 'mpirun -np 1 ./wrf_hydro.exe'

# 1st model run
job_24hr = wrfhydropy.Job(
    job_id='24hr',
    exe_cmd = exe_cmd,
    model_start_time = '2011-08-26',
    model_end_time='2011-08-27',
    restart_freq_hr=24,
    output_freq_hr=1)

# 2nd model run
job_145hr = wrfhydropy.Job(
    job_id='144hr',
    exe_cmd = exe_cmd,
    model_start_time = '2011-08-27',
    model_end_time='2011-09-02',
    restart_freq_hr=24,
    output_freq_hr=1,
    restart_dir='.')

Set up Simulation

simulation = wrfhydropy.Simulation()
simulation.add(model)
simulation.add(domain)
simulation.add(job_24hr)
simulation.add(job_145hr)

Save model configurations to disk

sim_dir = experiment_dir / 'simulation_cybergis'
os.mkdir(sim_dir)
os.chdir(sim_dir)
simulation.compose()

Run the model locally on Jupyter server (Optional)

Create a new Simulation object for local run

# simulation_local = wrfhydropy.Simulation()
# simulation_local.add(model)
# simulation_local.add(domain)
# simulation_local.add(job_24hr)
# simulation_local.add(job_145hr)

Save model configurations to disk

# sim_dir = experiment_dir / 'simulation_interactive'
# os.mkdir(sim_dir)
# os.chdir(sim_dir)
# simulation_local.compose()

Run it locally

# # Run the model locally on Jupyter server
# simulation_local.run()

Visualize outputs

# # Plot streamflow timeseries at forecast points
# simulation_local.collect()
# simulation_local.output.open('chanobs')
# simulation_local.output.chanobs.streamflow.plot(x='time', hue='feature_id', aspect=2, size=8)

Hand over to CyberGIS-Compute Service for Execution

! ls {sim_dir} -al

Adjustments to model folder

Before we hand the configured model to CyberGIS-Compute Service (CCS), we would need to remove the compiled executable "wrf_hydro.exe" from the model folder. The reason is the locally compiled "wrf_hydro.exe" might be incompatible with the environment on HPC due to potentially mismached dependencies. In addition, allowing users to send arbitrary executables to HPC poses a security risk. Instead, the CCS will pull the model source code from the official NCAR WRFHydro repo (https://github.com/NCAR/wrf_hydro_nwm_public.git) and compile the source code on HPC on the fly. To do this, we need to inform CCS with the correct branch name or commit id to use, which is stored in the Model object. Also, the Simulation object has important info on how to control the model runs. Therefore, we put the two 'pickled objects' (WrfHydroModel.pkl and simulation.pkl) in to the model folder.

# remove locally compiled executable "wrf_hydro.exe"
! rm -f {sim_dir}/wrf_hydro.exe
# save pickled Model object "WrfHydroModel.pkl" in model folder
! cp {compile_dir}/WrfHydroModel.pkl {sim_dir}/

# save pickled Simulation object "simulation.pkl" in model folder
simulation.pickle(sim_dir / "simulation.pkl")

Final inspection on model folder before submission

As we can see, the "wrf_hydro.exe" is removed and two new files (WrfHydroModel.pkl and simulation.pkl) are added. You may also notice there are some symbolic links such as the FORCING folder and RESTART files. The CyberGIS JJS Python client will replace those symbolic links with corresponding real files/folders.

! ls {sim_dir} -al

CyberGIS-Compute Service

The model is configured and ajustments are made to the model folder

import CyberGIS-Compute Service Python SDK

from job_supervisor_client import *

Create a new session

os.chdir(notebook_dir)
communityWRFHydroSession = Session('wrfhydro', isJupyter=True)

Create a WRFHydro Job and Upload the model data

communityWRFHydroJob = communityWRFHydroSession.job() # create new job
communityWRFHydroJob.upload(sim_dir)

Submit the job

# Specifiy CPUs and HPC resource
# node -- number of cpus to run the jobs
# machine -- which HPC: keeling(virtual roger) or comet
communityWRFHydroJob.submit(payload={
    "node": 4,
    "machine": "keeling"
})

Monitor the Job

communityWRFHydroJob.events(liveOutput=True)

Download model outputs

output_zip = communityWRFHydroJob.download(str(experiment_dir))

! mkdir -p {experiment_dir}/output
! unzip {output_zip} -d {experiment_dir}/output

Postprocessing

Once model outputs are retrieved from CyberGIS-Compute Service, we use wrfhydropy to load the output files and visualize the results.

output = wrfhydropy.core.simulation.SimulationOutput()
output.collect_output(sim_dir=os.path.join(experiment_dir, "output"))
output.chanobs.sort()
output.open("chanobs")
output.chanobs.streamflow.plot(x='time', hue='feature_id', aspect=2, size=8)

Done