Retrieve a large number of model outputs using Globus within CyberGIS-Compute

Author(s): Admin

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW. In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

Keywords: CJW, cybergis-compute, Globus, HydroShare

Posted by

profile-image

zhiyuli

CyberGIS Center @ UIUC




(for viewing purpose only)


Launch on CyberGIS-Jupyter for Water

LEAVE A COMMENT

Name and email are required. Your email will not be published.

Please provide a username.
Please provide a valid email
Please input your message.