4.1. karr_lab_build_utils package¶
4.1.1. Subpackages¶
4.1.2. Submodules¶
4.1.3. karr_lab_build_utils.__main__ module¶
Karr Lab build utilities
- Author
Jonathan Karr <karr@mssm.edu>
- Date
2016-10-27
- Copyright
2018, Karr Lab
- License
MIT
-
class
karr_lab_build_utils.__main__.
AnalyzePackageController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Perform static analyses of a package using Pylint
-
class
karr_lab_build_utils.__main__.
App
(label=None, **kw)[source]¶ Bases:
cement.core.foundation.App
Command line application
-
class
Meta
[source]¶ Bases:
object
-
handlers
= [<class 'karr_lab_build_utils.__main__.BaseController'>, <class 'karr_lab_build_utils.__main__.CreatePackageController'>, <class 'karr_lab_build_utils.__main__.CreateRepositoryController'>, <class 'karr_lab_build_utils.__main__.SetupRepositoryController'>, <class 'karr_lab_build_utils.__main__.CreateDocumentationTemplateController'>, <class 'karr_lab_build_utils.__main__.DownloadInstallPackageConfigFilesController'>, <class 'karr_lab_build_utils.__main__.RunTestsController'>, <class 'karr_lab_build_utils.__main__.DockerController'>, <class 'karr_lab_build_utils.__main__.DockerCreateContainerController'>, <class 'karr_lab_build_utils.__main__.InstallPackageToDockerContainerController'>, <class 'karr_lab_build_utils.__main__.RunTestsInDockerContainerController'>, <class 'karr_lab_build_utils.__main__.DockerRemoveContainerController'>, <class 'karr_lab_build_utils.__main__.FollowCircleciBuildController'>, <class 'karr_lab_build_utils.__main__.GetCircleciEnvironmentVariablesController'>, <class 'karr_lab_build_utils.__main__.SetCircleciEnvironmentVariableController'>, <class 'karr_lab_build_utils.__main__.DeleteCircleciEnvironmentVariableController'>, <class 'karr_lab_build_utils.__main__.CreateCodeClimateGithubWebhookController'>, <class 'karr_lab_build_utils.__main__.DoPostTestTasksController'>, <class 'karr_lab_build_utils.__main__.MakeAndArchiveReportsController'>, <class 'karr_lab_build_utils.__main__.CombineCoverageReportsController'>, <class 'karr_lab_build_utils.__main__.ArchiveCoverageReportController'>, <class 'karr_lab_build_utils.__main__.UploadCoverageReportToCoverallsController'>, <class 'karr_lab_build_utils.__main__.UploadCoverageReportToCodeClimateController'>, <class 'karr_lab_build_utils.__main__.MakeDocumentationController'>, <class 'karr_lab_build_utils.__main__.CompileDownstreamDependenciesController'>, <class 'karr_lab_build_utils.__main__.ArePackageDependenciesAcyclicController'>, <class 'karr_lab_build_utils.__main__.VisualizePackageDependenciesController'>, <class 'karr_lab_build_utils.__main__.AnalyzePackageController'>, <class 'karr_lab_build_utils.__main__.FindMissingRequirementsController'>, <class 'karr_lab_build_utils.__main__.FindUnusedRequirementsController'>, <class 'karr_lab_build_utils.__main__.UploadPackageToPypiController'>][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
ArchiveCoverageReportController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Archive a coverage report:
Upload report to Coveralls and Code Climate
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--coverage-dirname'], {'type': <class 'str'>, 'default': 'tests/reports', 'help': "Directory to store coverage data; default='tests/reports'"}), (['--dry-run'], {'default': False, 'dest': 'dry_run', 'action': 'store_true', 'help': 'If set, do not send results to Coveralls and Code Climate'})][source]¶
-
-
class
karr_lab_build_utils.__main__.
ArePackageDependenciesAcyclicController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Check if the package dependencies are acyclic so they are supported by CircleCI
-
class
karr_lab_build_utils.__main__.
BaseController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Base controller for command line application
-
class
karr_lab_build_utils.__main__.
CombineCoverageReportsController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Combine coverage reports
-
class
karr_lab_build_utils.__main__.
CompileDownstreamDependenciesController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Compile the downstream dependencies of a package by analyzing the requirements files of other packages
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--dirname'], {'type': <class 'str'>, 'default': '.', 'help': 'Path to package'}), (['--packages-parent-dir'], {'type': <class 'str'>, 'default': '..', 'help': 'Path to the parent directory of the other packages'}), (['--config-filename'], {'type': <class 'str'>, 'default': None, 'help': 'Path to save the configuration including downstream dependencies in YAML format'})][source]¶
-
description
= 'Compile the downstream dependencies of a package by analyzing the requirements files of other packages'[source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
CreateCodeClimateGithubWebhookController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Create Code Climate GitHub webhook for the current repository
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--repo-type'], {'type': <class 'str'>, 'default': None, 'help': 'Repository type (e.g., github)'}), (['--repo-owner'], {'type': <class 'str'>, 'default': None, 'help': 'Repository owner'}), (['--repo-name'], {'type': <class 'str'>, 'default': None, 'help': 'Name of the repository to build. This defaults to the name of the current repository.'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
CreateDocumentationTemplateController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Create a Sphinx documentation template for a package
-
class
karr_lab_build_utils.__main__.
CreatePackageController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Create a package
Create local and remote Git repositories;
Setup the directory structure of the repository;
Add the repository to CircleCI, Coveralls, Code Climate, Read the Docs, and code.karrlab.org;
Update the downstream dependencies of the package’’s dependencies
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--pypi-repository'], {'default': 'pypi', 'type': <class 'str'>, 'help': 'Repository upload package (e.g. pypi or testpypi)'})][source]¶
-
description
= '- Create local and remote Git repositories;\n - Setup the directory structure of the repository;\n - Add the repository to CircleCI, Coveralls, Code Climate, Read the Docs, and code.karrlab.org;\n - Update the downstream dependencies of the packages dependencies'[source]¶
-
-
class
karr_lab_build_utils.__main__.
CreateRepositoryController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Create a GitHub repository and clone the repository locally
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['name'], {'type': <class 'str'>, 'help': 'Name of the repository (i.e. repo_<name>)'}), (['--description'], {'default': '', 'type': <class 'str'>, 'help': 'Description of the repository'}), (['--public'], {'default': False, 'action': 'store_true', 'help': 'if set, make the repository public'}), (['--dirname'], {'default': None, 'type': <class 'str'>, 'help': 'Path for the repository'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
DeleteCircleciEnvironmentVariableController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Delete a CircleCI environment variable for a repository
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['name'], {'type': <class 'str'>, 'help': 'Name of the environment variable.'}), (['--repo-type'], {'type': <class 'str'>, 'default': None, 'help': 'Repository type (e.g., github)'}), (['--repo-owner'], {'type': <class 'str'>, 'default': None, 'help': 'Repository owner'}), (['--repo-name'], {'type': <class 'str'>, 'default': None, 'help': 'Name of the repository to build. This defaults to the name of the current repository.'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
DoPostTestTasksController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Do all post-test tasks for CircleCI
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['installation_exit_code'], {'type': <class 'int'>, 'help': 'Exit code of the package installation tasks'}), (['tests_exit_code'], {'type': <class 'int'>, 'help': 'Exit code of the tests'}), (['--dry-run'], {'default': False, 'dest': 'dry_run', 'action': 'store_true', 'help': 'If set, do not send results to Coveralls and Code Climate'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
DockerController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Base controller for Docker tasks
-
class
karr_lab_build_utils.__main__.
DockerCreateContainerController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Create a Docker container for running tests
-
class
karr_lab_build_utils.__main__.
DockerRemoveContainerController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Remove a Docker container
-
class
karr_lab_build_utils.__main__.
DownloadInstallPackageConfigFilesController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Download and install configuration files from GitHub configuration repository
-
class
karr_lab_build_utils.__main__.
FindMissingRequirementsController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
cement.Controller for finding missing requirements
-
class
karr_lab_build_utils.__main__.
FindUnusedRequirementsController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
cement.Controller for finding unused requirements
-
class
karr_lab_build_utils.__main__.
FollowCircleciBuildController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Follow a CircleCI build for a repository
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--repo-type'], {'type': <class 'str'>, 'default': None, 'help': 'Repository type (e.g., github)'}), (['--repo-owner'], {'type': <class 'str'>, 'default': None, 'help': 'Repository owner'}), (['--repo-name'], {'type': <class 'str'>, 'default': None, 'help': 'Name of the repository to build. This defaults to the name of the current repository.'}), (['--has-private-dependencies'], {'default': False, 'action': 'store_true', 'help': 'Set if the build requires an SSH key for the Karr Lab machine user because the repository depends on another private repository'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
GetCircleciEnvironmentVariablesController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Get the CircleCI environment variables for a repository and their partial values
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--repo-type'], {'type': <class 'str'>, 'default': None, 'help': 'Repository type (e.g., github)'}), (['--repo-owner'], {'type': <class 'str'>, 'default': None, 'help': 'Repository owner'}), (['--repo-name'], {'type': <class 'str'>, 'default': None, 'help': 'Name of the repository to build. This defaults to the name of the current repository.'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
InstallPackageToDockerContainerController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Copy and install a package to a Docker container
-
class
karr_lab_build_utils.__main__.
MakeAndArchiveReportsController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Make and archive reports:
Generate HTML test history reports
Generate HTML API documentation
Archive coverage report to Coveralls and Code Climate
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--coverage-dirname'], {'type': <class 'str'>, 'default': 'tests/reports', 'help': "Directory to store coverage data; default='tests/reports'"}), (['--dry-run'], {'default': False, 'dest': 'dry_run', 'action': 'store_true', 'help': 'If set, do not send results to Coveralls and Code Climate'})][source]¶
-
-
class
karr_lab_build_utils.__main__.
MakeDocumentationController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
cement.Controller for make_documentation.
Make HTML documentation. Optionally, spell check documentation.
-
class
karr_lab_build_utils.__main__.
RunTestsController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
cement.Controller for run_tests.
Run unit tests located at test-path. Optionally, generate a coverage report. Optionally, save the results to an XML file.
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--test-path'], {'type': <class 'str'>, 'default': None, 'help': 'Path to tests to run. The path should be relative to the current directory, or an absolute path. Default: the value of the environment variable `test_path` or `./tests` if the environment variable has not been set.'}), (['--dirname'], {'type': <class 'str'>, 'default': '.', 'help': "Path to package to test; default='.'"}), (['--n-workers'], {'type': <class 'int'>, 'default': 1, 'help': 'Numbers of workers to run tests`'}), (['--i-worker'], {'type': <class 'int'>, 'default': 0, 'help': 'Index of worker within {0 .. n-workers - 1}'}), (['--verbose'], {'default': False, 'action': 'store_true', 'help': 'if set display test output'}), (['--with-xunit'], {'default': False, 'action': 'store_true', 'help': 'if set save test results to XML file'}), (['--with-coverage'], {'default': False, 'action': 'store_true', 'help': 'if set assess code coverage'}), (['--coverage-dirname'], {'type': <class 'str'>, 'default': 'tests/reports', 'help': "Directory to store coverage data; default='tests/reports'"}), (['--coverage-type'], {'type': <class 'str'>, 'default': 'branch', 'help': "Type of coverage analysis to run {statement, branch, or multiple-decision}; default='branch'"}), (['--environment'], {'type': <class 'str'>, 'default': 'local', 'help': "Environment to run tests (local, docker, or circleci); default='local'"}), (['--ssh-key-filename'], {'type': <class 'str'>, 'default': '~/.ssh/id_rsa', 'help': 'Path to GitHub SSH key'}), (['--keep-docker-container'], {'dest': 'remove_docker_container', 'action': 'store_false', 'default': True, 'help': 'Keep Docker container'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
RunTestsInDockerContainerController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Run tests in a Docker container
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['container'], {'type': <class 'str'>, 'help': 'Container id'}), (['--test-path'], {'type': <class 'str'>, 'default': None, 'help': 'Path to tests to run. The path should be relative to the current directory, or an absolute path. Default: the value of the environment variable `test_path` or `./tests` if the environment variable has not been set.'}), (['--n-workers'], {'type': <class 'int'>, 'default': 1, 'help': 'Numbers of workers to run tests`'}), (['--i-worker'], {'type': <class 'int'>, 'default': 0, 'help': 'Index of worker within {0 .. n-workers - 1}'}), (['--verbose'], {'default': False, 'action': 'store_true', 'help': 'if set display test output'}), (['--with-xunit'], {'default': False, 'action': 'store_true', 'help': 'if set save test results to XML file'}), (['--with-coverage'], {'default': False, 'action': 'store_true', 'help': 'if set assess code coverage'}), (['--coverage-dirname'], {'type': <class 'str'>, 'default': 'tests/reports', 'help': "Directory to store coverage data; default='tests/reports'"}), (['--coverage-type'], {'type': <class 'str'>, 'default': 'branch', 'help': "Type of coverage analysis to run {statement, branch, or multiple-decision}; default='branch'"})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
SetCircleciEnvironmentVariableController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Set a CircleCI environment variable for a repository
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['name'], {'type': <class 'str'>, 'help': 'Name of the environment variable.'}), (['value'], {'type': <class 'str'>, 'help': 'Value of the environment variable.'}), (['--repo-type'], {'type': <class 'str'>, 'default': None, 'help': 'Repository type (e.g., github)'}), (['--repo-owner'], {'type': <class 'str'>, 'default': None, 'help': 'Repository owner'}), (['--repo-name'], {'type': <class 'str'>, 'default': None, 'help': 'Name of the repository to build. This defaults to the name of the current repository.'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
SetupRepositoryController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Setup a local Git repository with the default directory structure
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['name'], {'type': <class 'str'>, 'help': 'Name of the repository (i.e. repo_<name>)'}), (['--description'], {'default': '', 'type': <class 'str'>, 'help': 'Description of the repository'}), (['--keyword'], {'dest': 'keywords', 'default': [], 'type': <class 'str'>, 'action': 'append', 'help': 'Keyword for the repository'}), (['--dependency'], {'dest': 'dependencies', 'default': [], 'type': <class 'str'>, 'action': 'append', 'help': 'Karr Lab package that the package depends on'}), (['--public'], {'default': False, 'action': 'store_true', 'help': 'if set, make the repository public'}), (['--build-image-version'], {'default': None, 'type': <class 'str'>, 'help': 'Build image version'}), (['--dirname'], {'default': None, 'type': <class 'str'>, 'help': 'Path for the repository'}), (['--coveralls-repo-badge-token'], {'default': None, 'type': <class 'str'>, 'help': 'Coveralls badge token for the repository'}), (['--code-climate-repo-id'], {'default': None, 'type': <class 'str'>, 'help': 'Code Climate ID the repository'}), (['--code-climate-repo-badge-token'], {'default': None, 'type': <class 'str'>, 'help': 'Code Climate badge token for the repository'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
UploadCoverageReportToCodeClimateController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Upload coverage report to Code Climate
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--coverage-dirname'], {'type': <class 'str'>, 'default': 'tests/reports', 'help': "Directory to store coverage data; default='tests/reports'"}), (['--dry-run'], {'default': False, 'dest': 'dry_run', 'action': 'store_true', 'help': 'If set, do not send results to Code Climate'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
UploadCoverageReportToCoverallsController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Upload coverage report to Code Climate
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--coverage-dirname'], {'type': <class 'str'>, 'default': 'tests/reports', 'help': "Directory to store coverage data; default='tests/reports'"}), (['--dry-run'], {'default': False, 'dest': 'dry_run', 'action': 'store_true', 'help': 'If set, do not send results to Coveralls'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
UploadPackageToPypiController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Upload package to PyPI
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--dirname'], {'type': <class 'str'>, 'default': '.', 'help': 'Path to package (e.g. parent directory of setup.py)'}), (['--repository'], {'type': <class 'str'>, 'default': 'pypi', 'help': 'Repository upload package (e.g. pypi or testpypi)'}), (['--do-not-upload-source'], {'default': False, 'action': 'store_true', 'help': 'if set, do not upload source code to PyPI'}), (['--do-not-upload-build'], {'default': False, 'action': 'store_true', 'help': 'if set, do not upload build to PyPI'})][source]¶
-
-
class
-
class
karr_lab_build_utils.__main__.
VisualizePackageDependenciesController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Visualize downstream package dependencies as a graph
-
class
Meta
[source]¶ Bases:
object
-
class
4.1.4. karr_lab_build_utils._version module¶
4.1.5. karr_lab_build_utils.core module¶
Karr Lab build utilities
- Author
Jonathan Karr <karr@mssm.edu>
- Date
2016-08-02
- Copyright
2016-2018, Karr Lab
- License
MIT
-
class
karr_lab_build_utils.core.
BuildHelper
[source]¶ Bases:
object
Utility class to help build projects:
Run tests
Archive reports to test history server, Coveralls, and Code Climate
-
proj_docs_source_dir
[source]¶ local directory of source documentation files created by sphinx-apidoc
- Type
str
-
proj_docs_build_html_dir
[source]¶ local directory where generated HTML documentation should be saved
- Type
str
-
proj_docs_build_spelling_dir
[source]¶ local directory where spell check results should be saved
- Type
str
-
email_password
[source]¶ password for
email_username
- Type
str
-
DEFAULT_PROJ_TESTS_XML_DIR
[source]¶ default local directory where the test reports generated should be saved
- Type
str
-
DEFAULT_PROJ_TESTS_XML_LATEST_FILENAME
[source]¶ default file name to store latest XML test report
- Type
str
-
DEFAULT_PROJ_DOCS_STATIC_DIR
[source]¶ default local directory of static documentation files
- Type
str
-
DEFAULT_PROJ_DOCS_SOURCE_DIR
[source]¶ default local directory of source documentation files created by sphinx-apidoc
- Type
str
-
DEFAULT_PROJ_DOCS_SPELLING_DIR
[source]¶ default local directory where spell check results should be saved
- Type
str
-
DEFAULT_PROJ_DOCS_BUILD_HTML_DIR
[source]¶ default local directory where generated HTML documentation should be saved
- Type
str
Construct build helper
-
CIRCLE_API_ENDPOINT
= 'https://circleci.com/api'[source]
-
CODE_CLIMATE_ENABLED
= True[source]
-
COVERALLS_ENABLED
= True[source]
-
DEFAULT_BUILD_IMAGE
= 'karrlab/wc_env_dependencies:latest'[source]
-
DEFAULT_BUILD_IMAGE_VERSION
= 'latest'[source]
-
DEFAULT_PROJ_DOCS_BUILD_HTML_DIR
= 'docs/_build/html'[source]
-
DEFAULT_PROJ_DOCS_DIR
= 'docs'[source]
-
DEFAULT_PROJ_DOCS_SOURCE_DIR
= 'docs/source'[source]
-
DEFAULT_PROJ_DOCS_STATIC_DIR
= 'docs/_static'[source]
-
DEFAULT_PROJ_TESTS_DIR
= 'tests'[source]
-
DEFAULT_PROJ_TESTS_XML_DIR
= 'tests/reports'[source]
-
DEFAULT_PROJ_TESTS_XML_LATEST_FILENAME
= 'latest'[source]
-
DEFAULT_TEST_RUNNER
= 'pytest'[source]
-
GITHUB_API_ENDPOINT
= 'https://api.github.com'[source]
-
INITIAL_PACKAGE_VERSION
= '0.0.1'[source]
-
analyze_package
(package_name, messages=None, config_filename=None, verbose=False)[source]¶ Perform static analyses of a package using Pylint.
The default options will identify the following issues:
Unused imported modules, classes, functions, and variables
Reimported modules, classes, functions, and variables
Wild card imports outside of __init__.py
Duplicate arguments and keys
Missing requirements
- Parameters
package_name (
str
) – name of the package to analyzemessages (
list
ofstr
, optional) – list of Pylint checks to performconfig_filename (
str
, optional) – path to Pylist configuration file (rcfile)verbose (
bool
, optional) – ifTrue
, display extra Pylint information
- Returns
pylint return code
- Return type
int
-
archive_coverage_report
(coverage_dirname='tests/reports', dry_run=False)[source]¶ Archive coverage report:
Upload report to Coveralls
Upload report to Code Climate
- Parameters
coverage_dirname (
str
, optional) – directory to save coverage datadry_run (
bool
, optional) – if true, don’t upload to the Coveralls and Code Climate servers
-
archive_test_report
()[source]¶ Upload test report to history server
- Raises
BuildHelperError – if there is an error uploading the report to the test history server
-
are_package_dependencies_acyclic
(packages_parent_dir='..')[source]¶ Check if the package dependencies are acyclic so they are supported by CircleCI
- Parameters
packages_parent_dir (
str
, optional) – path to the parent directory of the packages- Returns
True
if the package dependencies are acyclic- Return type
bool
-
combine_coverage_reports
(coverage_dirname='tests/reports')[source]¶ - Parameters
coverage_dirname (
str
, optional) – directory to merge coverage files
-
compile_downstream_dependencies
(dirname='.', packages_parent_dir='..', config_filename=None)[source]¶ Compile the downstream dependencies of a package and save them to
config_filename
- Parameters
dirname (
str
, optional) – path to packagepackages_parent_dir (
str
, optional) – path to the parent directory of the packagesconfig_filename (
str
, optional) – path to save configuration with list of downstream dependencies in YAML format
- Returns
downstream dependencies
- Return type
list
ofstr
- Raises
BuildHelperError – if a package has more than one module
-
create_code_climate_github_webhook
(repo_type=None, repo_owner=None, repo_name=None)[source]¶ Create GitHub webhook for Code Climate
- Parameters
repo_type (
str
, optional) – repository typerepo_owner (
str
, optional) – repository ownerrepo_name (
str
, optional) – repository name
- Raises
ValueError – if webhook wasn’t created and didn’t already exist
-
create_docker_container
(ssh_key_filename='~/.ssh/id_rsa')[source]¶ Create a docker container
- Parameters
ssh_key_filename (
str
, optional) – path to GitHub SSH key- Returns
container id
- Return type
str
-
create_documentation_template
(dirname='.')[source]¶ Create Sphinx documentation template for a package
- Parameters
dirname (
str
, optional) – path to package- Raises
ValueError – if no package or more than one package is specified
-
create_package
(pypi_repository=None)[source]¶ Create a package
Create a local Git repository
Create a remote GitHub repository
Add the repository to Code Climate
Add the repository to Coveralls
Add the repository to CircleCI project (by following the GitHub repository)
Add environment variable for tokens for code.karrlab.org, Coveralls, Code Climate, and CircleCI
Add environment variable for password for karr.lab.daemon@gmail.com
Generate API token for status badge
If the repository is not private, add the repository to Read the Docs
Add the package to code.karrlab.org
Add JSON-formatted file to
ssh://code.karrlab.org:/home/karrlab_code/code.karrlab.org/repo/{{ name }}.json
Add badges for Code Climate, Coveralls, CircleCI, and Read the Docs to README.md
Add package name to
downstream_dependencies
key in.karr_lab_build_utils.yml
- Parameters
pypi_repository (
str
, optional) – name of a repository defined in the PyPI configuration file or a repository URL
-
create_repository
(name, description='', private=True, dirname=None)[source]¶ Create a GitHub repository and clone the repository locally
- Parameters
( (name) – obj`str`): package name
description (
str
, optional) – package descriptionprivate (
bool
, optional) – ifFalse
, make the GitHub repository public and set up documentation generation with Read the Docsdirname (
str
, optional) – directory name for repository
-
delete_circleci_environment_variable
(var, repo_type=None, repo_owner=None, repo_name=None)[source]¶ Delete a CircleCI environment variable for a repository
- Parameters
var (
str
) – name of variable to deleterepo_type (
str
, optional) – repository typerepo_owner (
str
, optional) – repository ownerrepo_name (
str
, optional) – repository name
-
do_post_test_tasks
(installation_error, tests_error, dry_run=False)[source]¶ Do all post-test tasks for CircleCI
Make test and coverage reports
Compile documentation
Archive test and coverage reports to the Karr Lab test history server, Coveralls, and Code Climate
Trigger tests of downstream dependencies
Notify authors of new failures in downstream packages
- Parameters
installation_error (
bool
) –True
if there were other errors during the installationtests_error (
bool
) – obj:False if the tests passesdry_run (
bool
, optional) – if true, don’t upload to the Coveralls and Code Climate servers
- Returns
names of triggered packages
dict
ofstr
,str
: dictionary which maps names of untriggered packages to the reasonswhy they weren’t triggered
dict
: status of a set of resultsdict
: exception from make_and_archive_reports- Return type
list
ofstr
-
download_package_config_files
()[source]¶ Download the configuration repository
- Parameters
update (
bool
, optional) – ifTrue
, update the configuration
-
find_missing_requirements
(package_name, dirname='.')[source]¶ Finding missing requirements
- Parameters
package_name (
str
) – name of the package to analyzedirname (
str
, optional) – path to package
- Returns
list of missing dependencies and their occurences in the code
- Return type
list
-
find_unused_requirements
(package_name, dirname='.')[source]¶ Finding unused_requirements
- Parameters
package_name (
str
) – name of the package to analyzedirname (
str
, optional) – path to package
- Returns
name of the unused dependencies
- Return type
list
-
follow_circleci_build
(repo_type=None, repo_owner=None, repo_name=None, has_private_dependencies=False)[source]¶ Follow CircleCI build for a repository
- Parameters
repo_type (
str
, optional) – repository typerepo_owner (
str
, optional) – repository ownerrepo_name (
str
, optional) – repository namehas_private_dependencies (
bool
, optional) – ifTrue
, add a GitHub SSH key for the Karr Lab machine user to the build
- Raises
ValueError – if a CircleCI build wasn’t followed and didn’t already exist
-
get_circleci_environment_variables
(repo_type=None, repo_owner=None, repo_name=None)[source]¶ Get the CircleCI environment variables for a repository and their partial values
- Parameters
repo_type (
str
, optional) – repository typerepo_owner (
str
, optional) – repository ownerrepo_name (
str
, optional) – repository name
- Returns
dictionary of environment variables and their partial values
- Return type
dict
-
static
get_python_version
()[source]¶ Get the Python version
- Returns
the Python version
- Return type
str
-
get_test_results
()[source]¶ Load test results from a set of XML files
- Results:
TestResults
: test results
-
get_test_results_status
(test_results, installation_error, tests_error, other_error, dry_run=False)[source]¶ Get the status of a set of results
Old err
New error
Fixed error
New downstream error
- Parameters
test_results (
TestResults
) – test resultsinstallation_error (
bool
) –True
if there were other errors during the installationtests_error (
bool
) – obj:False if the tests passesother_error (
bool
) –True
if there were other errors during the build such as in generating and/or archiving the reportsdry_run (
bool
, optional) – if true, don’t upload to the Coveralls and Code Climate servers
- Returns
status of a set of results
- Return type
dict
-
install_package_config_files
(overwrite=False)[source]¶ Copy third party configs to their appropriate paths from the configs repository
- Parameters
overwrite (
bool
, optional) – ifTrue
, overwrite existing configuration files
-
install_package_to_docker_container
(container, dirname='.')[source]¶ Copy and install package to Docker container
- Parameters
container (
str
) – container iddirname (
str
, optional) – path to package to copy and install
-
install_requirements
(upgrade=False)[source]¶ Install requirements
- Parameters
upgrade (
bool
, optional) – ifTrue
, upgrade requirements
-
make_and_archive_reports
(coverage_dirname='tests/reports', dry_run=False)[source]¶ Make and archive reports:
Upload test report to history server
Upload coverage report to Coveralls and Code Climate
- Parameters
coverage_dirname (
str
, optional) – directory to merge coverage filesdry_run (
bool
, optional) – if true, don’t upload to the Coveralls and Code Climate servers
- Returns
analyses of missing and unused requirements
- Return type
dict
-
make_documentation
(spell_check=False)[source]¶ Make HTML documentation using Sphinx for one or more packages. Save documentation to proj_docs_build_html_dir
- Parameters
spell_check (
bool
) – ifTrue
, run spell checking- Raises
BuildHelperError – If project name not set
-
remove_docker_container
(container)[source]¶ Stop and remove a docker container
- Parameters
container (
str
) – container id
-
run_circleci_api
(command, version='1.1', method='get', repo_type=None, repo_owner=None, repo_name=None, data=None)[source]¶ Run the CircleCI API
- Parameters
command (
str
) – API commandversion (
str
, optional) – version of the API to usemethod (
str
, optional) – type of HTTP request (get, post, delete)repo_type (
str
, optional) – repository type (e.g., github)repo_owner (
str
, optional) – repository ownerrepo_name (
str
, optional) – repository namedata (
str
, optional) – data
- Returns
CircleCI result
- Return type
dict
- Raises
requests.exceptions.HTTPError – if the HTTP request to CircleCI does not succeed
-
run_method_and_capture_stderr
(func, *args, **kwargs)[source]¶ Run a method that returns a numerical error value, and exit if the return value is non-zero
- Parameters
func (
function
) – function to run*args – arguments to
func
**kwargs – keyword arguments to obj:func
-
run_method_and_capture_stdout
(func, *args, **kwargs)[source]¶ Run a method that returns a numerical error value, and exit if the return value is non-zero
- Parameters
func (
function
) – function to run*args – arguments to
func
**kwargs – keyword arguments to obj:func
- Returns
stdout
- Return type
str
-
run_tests
(dirname='.', test_path=None, n_workers=1, i_worker=0, verbose=False, with_xunit=False, with_coverage=False, coverage_dirname='tests/reports', coverage_type=<CoverageType.branch: 1>, environment=<Environment.local: 0>, exit_on_failure=True, ssh_key_filename='~/.ssh/id_rsa', remove_docker_container=True)[source]¶ Run unit tests located at test_path.
Optionally, generate a coverage report. Optionally, save the results to a file
To configure coverage, place a .coveragerc configuration file in the root directory of the repository - the same directory that holds .coverage. Documentation of coverage configuration is in https://coverage.readthedocs.io/en/coverage-4.2/config.html
- Parameters
dirname (
str
, optional) – path to package that should be testedtest_path (
str
, optional) – path to tests that should be runn_workers (
int
, optional) – number of workers to run testsi_worker (
int
, optional) – index of worker within {0 ..n_workers
- 1}verbose (
str
, optional) – ifTrue
, display stdout from testswith_xunit (
bool
, optional) – whether or not to save test resultswith_coverage (
bool
, optional) – whether or not coverage should be assessedcoverage_dirname (
str
, optional) – directory to save coverage datacoverage_type (
CoverageType
, optional) – type of coverage to run whenwith_coverage
isTrue
environment (
str
, optional) – environment to run tests (local, docker, or circleci-local-executor)exit_on_failure (
bool
, optional) – whether or not to exit on test failuressh_key_filename (
str
, optional) – path to GitHub SSH key; needed for Docker environmentremove_docker_container (
bool
, optional) – ifTrue
, remove Docker container
- Raises
BuildHelperError – If the environment is not supported or the package directory not set
-
run_tests_in_docker_container
(container, test_path=None, n_workers=1, i_worker=0, verbose=False, with_xunit=False, with_coverage=False, coverage_dirname='tests/reports', coverage_type=<CoverageType.branch: 1>)[source]¶ Test a package in a docker container
- Parameters
container (
str
) – container idtest_path (
str
, optional) – path to tests that should be runn_workers (
int
, optional) – number of workers to run testsi_worker (
int
, optional) – index of worker within {0 ..n_workers
- 1}verbose (
str
, optional) – ifTrue
, display stdout from testswith_xunit (
bool
, optional) – whether or not to save test resultswith_coverage (
bool
, optional) – whether or not coverage should be assessedcoverage_dirname (
str
, optional) – directory to save coverage datacoverage_type (
CoverageType
, optional) – type of coverage to run whenwith_coverage
isTrue
-
send_email_notifications
(installation_error, tests_error, other_error, static_analyses, dry_run=False)[source]¶ Send email notifications of failures, fixes, and downstream failures
- Parameters
installation_error (
bool
) –True
if there were other errors during the installationtests_error (
bool
) – obj:False if the tests passesother_error (
bool
) –True
if there were other errors during the build such as in generating and/or archiving the reportsstatic_analyses (
dict
) – analyses of missing and unused requirementsdry_run (
bool
, optional) – if true, don’t upload to the Coveralls and Code Climate servers
- Returns
status of a set of results
- Return type
dict
-
set_circleci_environment_variables
(vars, repo_type=None, repo_owner=None, repo_name=None)[source]¶ Set the CircleCI environment variables for a repository
- Parameters
vars (
dict
) – dictionary of environment variables to setrepo_type (
str
, optional) – repository typerepo_owner (
str
, optional) – repository ownerrepo_name (
str
, optional) – repository name
- Returns
dictionary of environment variables and their values
- Return type
dict
-
setup_repository
(name, description='', keywords=None, dependencies=None, private=True, build_image_version=None, dirname=None, circleci_repo_token=None, coveralls_repo_badge_token=None, code_climate_repo_id=None, code_climate_repo_badge_token=None)[source]¶ Setup a local Git repository with the default directory structure
- Parameters
( (name) – obj`str`): package name
description (
str
, optional) – package descriptionkeywords (
list
ofstr
, optional) – list of keywordsdependencies (
list
ofstr
, optional) – list of Karr Lab packages that the package depends onprivate (
bool
, optional) – ifFalse
, make the GitHub repository public and set up documentation generation with Read the Docsbuild_image_version (
str
, optional) – build image versiondirname (
str
, optional) – directory namecircleci_repo_token (
str
, optional) – CircleCI API token (e.g. for badges) for the repositorycoveralls_repo_badge_token (
str
, optional) – Coveralls badge token for the repositorycode_climate_repo_id (
str
, optional) – Code Climate ID for the repositorycode_climate_repo_badge_token (
str
, optional) – Code Climate for the repository
-
trigger_tests_of_downstream_dependencies
(config_filename='.karr_lab_build_utils.yml', dry_run=False)[source]¶ Trigger CircleCI to test downstream dependencies listed in
config_filename
- Parameters
config_filename (
str
, optional) – path to YAML configuration file which contains a list of downstream dependenciesdry_run (
bool
, optional) – if true, don’t upload to the Coveralls and Code Climate servers
- Returns
names of triggered packages
dict
ofstr
,str
: dictionary which maps names of untriggered packages to the reasonswhy they weren’t triggered
- Return type
list
ofstr
- Todo
support branches
-
upgrade_karr_lab_packages
()[source]¶ Upgrade the packages from the Karr Lab’s GitHub organization
- Returns
upgraded requirements from the Karr Lab’s GitHub organization
- Return type
list
ofstr
-
upload_coverage_report_to_code_climate
(coverage_dirname='tests/reports', dry_run=False)[source]¶ Upload coverage report to Code Climate
- Parameters
coverage_dirname (
str
, optional) – directory to save coverage datadry_run (
bool
, optional) – if true, don’t upload to the Coveralls server
- Raises
BuildHelperError – If error uploading code coverage to Code Climate
-
upload_coverage_report_to_coveralls
(coverage_dirname='tests/reports', dry_run=False)[source]¶ Upload coverage report to Coveralls
- Parameters
coverage_dirname (
str
, optional) – directory to save coverage datadry_run (
bool
, optional) – if true, don’t upload to the Coveralls server
-
upload_documentation_to_docs_server
(dirname='.')[source]¶ Upload compiled documentation to the lab server
- Parameters
dirname (
str
, optional) – path to package
-
upload_package_to_pypi
(dirname='.', repository=None, upload_source=True, upload_build=True)[source]¶ Upload a package to PyPI
- Parameters
dirname (
str
, optional) – path to package to uploadrepository (
str
, optional) – name of a repository defined in the PyPI configuration file or a repository URLupload_source (
bool
, optional) – ifTrue
, upload source codeupload_build (
bool
, optional) – ifTrue
-
visualize_package_dependencies
(packages_parent_dir='..', out_filename='../package_dependencies.pdf')[source]¶ Visualize downstream package dependencies as a graph
- Parameters
packages_parent_dir (
str
, optional) – path to the parent directory of the packagesout_filename (
str
, optional) – path to save visualization
-
exception
karr_lab_build_utils.core.
BuildHelperError
[source]¶ Bases:
Exception
Represents
BuildHelper
errors
-
class
karr_lab_build_utils.core.
TestCaseResult
[source]¶ Bases:
object
The result of a test case
-
class
karr_lab_build_utils.core.
TestCaseResultType
[source]¶ Bases:
enum.Enum
Type of test case result
-
class
karr_lab_build_utils.core.
TestResults
[source]¶ Bases:
object
Unit test results
-
get_num_errors
()[source]¶ Get the number of tests with errors
- Returns
number of tests with errors
- Return type
int
-
get_num_failures
()[source]¶ Get the number of tests with failures
- Returns
number of tests with failures
- Return type
int
-
get_num_passed
()[source]¶ Get the number of tests that passed
- Returns
number of tests that passed
- Return type
int
-