4.3. Using wc_env_manager build, version, and sharing computing environments for WC modeling¶
WC modeling requires complex computing environments with numerous dependencies. wc_env_manager helps modelers and software developers setup customizable computing environments for developing, testing, and running whole-cell (WC) models and WC modeling software. This makes it easy for modelers and software developers to install and configure the numerous dependencies required for WC modeling. This helps modelers and software developers focus on developing WC models and software tools, rather than on installing, configuring, and maintaining complicated dependencies.
In addition, wc_env_manager facilitates collaboration by helping WC modelers and software developers share a common base computing environment with third party dependencies. Furthermore, wc_env_manager helps software developers anticipate and debug issues in deployment by enabling developers to replicate similar environments to those used to test and deploy WC models and tools in systems such as Amazon EC2, CircleCI, and Heroku.
wc_env_manager uses Docker to setup a customizable computing environment that contains all of the software packages needed to run WC models and WC modeling software. This includes
Required non-Python packages
Required Python packages from PyPI and other sources
Optionally, local packages on the user’s machine such as clones of these WC models and WC modeling tools
wc_env_manager supports both the development and deployment of WC models and WC modeling tools:
Development: wc_env_manager can run WC models and WC modeling software that is located on the user’s machine. This is useful for testing WC models and WC modeling software before committing it to GitHub.
Deployment: wc_env_manager can run WC models and WC modeling software from external sources such as GitHub.
4.3.1. How wc_env_manager works¶
wc_env_manager is based on Docker images and containers which enable virtual environments within all major operating systems including Linux, Mac OSX, and Windows, and the DockerHub repository for versioning and sharing virtual environments.
wc_env_manager creates a Docker image, wc_env_dependencies with the third-party dependencies needed for WC modeling or pulls this image from DockerHub. This image represents an Ubuntu Linux machine.
wc_env_manager uses this Docker image to create another Docker image, wc_env with the WC models, WC modeling tools, and the configuration files and authentication keys needed for WC modeling.
wc_env_manager uses this image to create a Docker container to run WC models and WC modeling tools. Optionally, the container can have volumes mounted from the host to run code on the host inside the Docker container, which is helpful for using the container to test and debug WC models and tools.
The images and containers created by wc_env_manager can be customized using a configuration file.
4.3.2. Installing wc_env_manager¶
First, install the following requirements. See Section 6 for detailed instructions.
Second, run the following command to install the latest version of wc_env_manager from GitHub:
pip install git+https://github.com/KarrLab/wc_env_manager.git#egg=wc_env_manager
4.3.4. Using wc_env_manager to create and run Docker containers for WC modeling¶
Developers should follow these steps to build and use WC modeling computing environments (Docker images and containers) to test, debug, and run WC models and WC modeling tools.
Use wc_env_manager to pull existing WC modeling Docker images
Use wc_env_manager to create Docker containers with volumes mounted from the host and installations of software packages contained on the house
Run models and tools inside the Docker containers created by wc_env_manager
4.3.4.1. Pulling existing Docker images¶
First, use the following command to pull existing WC modeling Docker images. This will pull both the base image with third part dependencies, wc_env_dependencies, and the image with WC models and modeling tools, wc_env.:
wc-env-manager pull
The following commands can also be used to pull the individual images.:
wc-env-manager base-image pull
wc-env-manager image pull
4.3.4.2. Building containers for WC modeling¶
Second, set the configuration for the containers created by wc_env_manager by creating a configuration file ./wc_env_manager.cfg following the schema outlined in /path/to/wc_env_manager/wc_env_manager/config/core.schema.cfg and the defaults in /path/to/wc_env_manager/wc_env_manager/config/core.default.cfg.
Set the host paths that should be mounted into the containers. This should include the root directory of your clones of WC models and WC modeling tools (e.g. map host:~/Documents to container:/root/Documents-Host).
Set the WC modeling packages that should be installed into wc_env. This should be specified in the pip requirements.txt format and should be specified in terms of paths within the container. The following example illustrates how install clones of wc_lang and wc_utils mounted from the host into the container.:
/root/Documents-Host/wc_lang /root/Documents-Host/wc_utils
Third, use the following command to use wc_env to construct a Docker container.:
wc-env-manager container build
This will print out the id of the created container.
4.3.4.3. Using containers to run WC models and WC modeling tools¶
Fourth, use the following command to log in the container.:
cd /path/to/wc_env_manager
docker-compose up -d
docker-compose exec wc_env bash
Fifth, use the integrated WC modeling command line program, *wc_cli*, to run WC models and WC modeling tools. For example, the following command illustrates how to get help for the wc_cli program. See the *wc_cli* documentation for more information.:
container >> wc-cli --help
4.3.5. Using WC modeling computing environments with an external IDE such as PyCharm¶
The Docker images created with wc_env_manager can be used with external integrated development environments (IDEs) such as PyCharm. See the links below for instructions on how to use these tools with Docker images created with wc_env_manager.
Other IDEs:
Install the IDE in a Docker image
Use X11 forwarding to render graphical output from a Docker container to your host. See Using GUI’s with Docker for more information.
4.3.6. Caveats and troubleshooting¶
Code run in containers created by wc_env_manager can create host files and overwrite existing host files. This is because wc_env_manager mounts host directories into containers.
Containers created by wc_env_manager can be used to run code located on your host machine. However, using different versions of Python between your host and the Docker containers can create Python caches and compiled Python files that are incompatible between your host and the Docker containers. Before switching between running code on your host your and the Docker containers, you may need to remove all
__pycache__subdirectories and*.pycfiles from host packages mounted into the containers.Code run in Docker containers will not have access to the absolute paths of your host and vice-versa. Consequently, arguments that represent absolute host paths or which contain absolute host paths must be mapped from absolute host paths to the equivalent container path. Similarly, outputs which represent or contain absolute container paths must be mapped to the equivalent host paths.
Running code in containers created with wc_env_manager will be slower than running the same code on your host. This is because wc_env_manager is based on Docker containers, which add an additional layer of abstraction between your code and your processor.