|
It is often desirable to perform large-scale multi-year simulations
on cloud computing resources.
Given the scale and complexity of such large runs in the cloud, reliable
tools and a robust, scalable workflow are essential. We describe our cloud-based
docker-centric implementation, which utilizes emerging technologies (such as
Docker, Object Storage, and cloud computing), and the benefits and
constraints that this solution brings in both a research and commercial
environment.
IBM's Deep Thunder is a high spatial and temporal-resolution weather
forecasting system, part of which is based on the ARW core of the Weather
Research and Forecasting (WRF) model. WRF, its dependencies, and the other
components that form the Deep Thunder system are non-trivial to build,
install, maintain and deploy. Therefore, we produced a development operations
(DevOps) pipeline to automate the deployment of containerized versions of
Deep Thunder.
The pipeline presented goes from code to cloud, and takes advantage of modern
agile and DevOps tools and methodologies to allow us to automate, distribute,
control and communicate more effectively.
We found that use of continuous integration coupled with automated testing,
containerization and container distribution technology has transformed the
distribution of both applications and data, and is essential for large-scale
deployments to heterogeneous environments.
|