2.1. wc_utils package

2.1.1. Subpackages

2.1.2. Submodules

2.1.3. wc_utils._version module

2.1.4. wc_utils.cache module

Caching

Author

Jonathan Karr <jonrkarr@gmail.com>

Date

2018-03-30

Copyright

2018, Karr Lab

License

MIT

class wc_utils.cache.Cache(directory='/root/.wc/cache/', hash_block_size=65536, **kwargs)[source]

Bases: diskcache.fanout.FanoutCache

Cache that shards keys (including the file content of keys that represent file names) and values

hash_block_size[source]

block size to use for hashing the content of file arguments

Type

int

DEFAULT_DIRECTORY = '/root/.wc/cache/'[source]
memoize(name=None, typed=False, expire=None, tag=None, filename_args=None, filename_kwargs=None)[source]

Memoizing cache decorator

Parameters

  • name (str, optional) – name given for callable

  • typed (bool, optional) – cache different types separately

  • expire (float, optional) – seconds until arguments expire

  • tag (str, optional) – text to associate with arguments

  • filename_args (list, optional) – list of indices of arguments that represent filenames

  • filename_kwargs (list, optional) – list of keys of keyword arguments that represent filenames

Returns

callable decorator

Return type

types.FunctionType

wc_utils.cache.memoize(name=None, typed=False, expire=None, tag=None, filename_args=None, filename_kwargs=None)[source]

Memoizing cache decorator

Parameters

  • name (str, optional) – name given for callable

  • typed (bool, optional) – cache different types separately

  • expire (float, optional) – seconds until arguments expire

  • tag (str, optional) – text to associate with arguments

  • filename_args (list, optional) – list of indices of arguments that represent filenames

  • filename_kwargs (list, optional) – list of keys of keyword arguments that represent filenames

Returns

callable decorator

Return type

types.FunctionType

2.1.5. wc_utils.quilt module

High-level interface for the Quilt data revisioning system

Author

Jonathan Karr <jonrkarr@gmail.com>

Date

2019-10-08

Copyright

2018-2019, Karr Lab

License

MIT

class wc_utils.quilt.QuiltManager(path=None, namespace=None, package=None, hash=None, registry=None, username=None, password=None, aws_bucket=None, aws_profile=None)[source]

Bases: object

Manager for Quilt packages

Quilt credentials and configuration can be stored in a wc_utils configuration file (e.g., ~/.wc/wc_utils.cfg) or passed to the constructor:

[wc_utils]
    [[quilt]]
        username = ...
        password = ...
        aws_bucket = ...
        aws_profile = ...

AWS S3 credentials should be stored in ~/.aws/credentials:

[default]
aws_access_key_id = ...
aws_secret_access_key = ...

AWS S3 regions should be configured in ~/.aws/config:

[default]
region=us-east-1
output=json
path[source]

local path to package

Type

str

namespace[source]

namespace for package

Type

str

package[source]

name of package

Type

str

hash[source]

hash of version of package

Type

str

registry[source]

URL for Quilt registry

Type

str

username[source]

Quilt user name

Type

str

password[source]

Quilt password

Type

str

aws_bucket[source]

AWS bucket to store/access packages

Type

str

aws_profile[source]

AWS profile (credentials) to store/access packages

Type

str

config()[source]

Configure the Quilt client to the desired AWS S3 bucket (“remote Quilt registry”)

delete_file_from_bucket(key)[source]

Delete file to AWS S3 bucket

Parameters

key (str) – path within bucket to save file

delete_package(del_from_bucket=True)[source]

Delete package

Parameters

del_from_bucket (bool, optional) – if True, delete the files for the package from the AWS bucket

download_file_from_bucket(key, path)[source]

Get file from AWS S3 bucket

Parameters

  • key (str) – path within bucket to file

  • path (str) – path to save file

download_package(path=None)[source]

Download package, or a path within a package, to local directory

Parameters

path (str, optional) – path within a package to download

get_aws_bucket_uri()[source]

Get the full URI of an AWS S3 bucket (s3:// + bucket id)

Returns

full URI of an AWS S3 bucket

Return type

str

get_full_package_id()[source]

Get the full id of a package (namespace and package id)

Returns

full package id

Return type

str

get_packages()[source]

Get the names of the packages in the S3 bucket

Returns

list of package names

Return type

list of str

login(credentials='aws')[source]

Login with user or session token

upload_file_to_bucket(path, key)[source]

Upload file to AWS S3 bucket

Parameters

  • path (str) – path to file to upload

  • key (str) – path within bucket to save file

upload_package(message=None)[source]

Build and upload package from local directory, ignoring all files listed in .quiltignore

Parameters

message (str) – commit message

2.1.6. Module contents