Run National Water Model (WRFHydro) on HPC through CyberGIS-Compute Service

The goal of this notebook is to show the steps to run an example National Water Model (WRFhydro) model on HPC resources through the CyberGIS-Compute Service. This notebook uses wrfhydropy, a Python wrapper for WRFHydro, in model preprocessing and postprocessing, and the resulting ready-to-run model is handed over to CyberGIS-Compute Service for execution on a supported HPC resource (Virtual Roger/Keeling at UIUC or XSEDE COMET at SDSC). This example is adapted from the "ex_01_end_to_end.ipynb" notebook from wrfhydropy official github repo, and users are encouraged to refer to the tutorials there to get familair with wrfhydropy usages.


This section is to setup environment, checkout model source code, download test case data.

Set up environments

Download model source code from NCAR WRFHydro repo and check out the selected branch or commit

Retrieve testcase data from HydroShare

Speed up data downloading with multi-threaded "aria2c" if installed, otherwise just use wget

Unzip test case into domain folder

Model Configurations with "wrfhydropy"

This section is to configure the model using wrfhydropy. We will have a ready-to-run model.



Compile source code

Set up model runs (Jobs)

Set up Simulation

Save model configurations to disk

Run the model locally on Jupyter server (Optional)

Create a new Simulation object for local run

Save model configurations to disk

Run it locally

Visualize outputs

Hand over to CyberGIS-Compute Service for Execution

Adjustments to model folder

Before we hand the configured model to CyberGIS-Compute Service (CCS), we would need to remove the compiled executable "wrf_hydro.exe" from the model folder. The reason is the locally compiled "wrf_hydro.exe" might be incompatible with the environment on HPC due to potentially mismached dependencies. In addition, allowing users to send arbitrary executables to HPC poses a security risk. Instead, the CCS will pull the model source code from the official NCAR WRFHydro repo ( and compile the source code on HPC on the fly. To do this, we need to inform CCS with the correct branch name or commit id to use, which is stored in the Model object. Also, the Simulation object has important info on how to control the model runs. Therefore, we put the two 'pickled objects' (WrfHydroModel.pkl and simulation.pkl) in to the model folder.

Final inspection on model folder before submission

As we can see, the "wrf_hydro.exe" is removed and two new files (WrfHydroModel.pkl and simulation.pkl) are added. You may also notice there are some symbolic links such as the FORCING folder and RESTART files. The CyberGIS JJS Python client will replace those symbolic links with corresponding real files/folders.

CyberGIS-Compute Service

The model is configured and ajustments are made to the model folder

import CyberGIS-Compute Service Python SDK

Create a new session

Create a WRFHydro Job and Upload the model data

Submit the job

Monitor the Job

Download model outputs


Once model outputs are retrieved from CyberGIS-Compute Service, we use wrfhydropy to load the output files and visualize the results.