3.1. obj_tables package¶
3.1.1. Subpackages¶
3.1.2. Submodules¶
3.1.3. obj_tables.__main__ module¶
Command line utilities for modeling data in tables (XLSX, CSV, TSV)
- Author
Jonathan Karr <karr@mssm.edu>
- Date
2019-09-11
- Copyright
2019, Karr Lab
- License
MIT
-
class
obj_tables.__main__.
App
(label=None, **kw)[source]¶ Bases:
cement.core.foundation.App
Command line application
-
class
Meta
[source]¶ Bases:
object
-
handlers
= [<class 'obj_tables.__main__.BaseController'>, <class 'obj_tables.__main__.VizSchemaController'>, <class 'obj_tables.__main__.ValidateController'>, <class 'obj_tables.__main__.NormalizeController'>, <class 'obj_tables.__main__.InitSchemaController'>, <class 'obj_tables.__main__.GenTemplateController'>, <class 'obj_tables.__main__.DiffController'>, <class 'obj_tables.__main__.ConvertController'>][source]¶
-
-
class
-
class
obj_tables.__main__.
BaseController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Base controller for command line application
-
class
obj_tables.__main__.
ConvertController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Convert a schema-encoded workbook to another format (CSV, XLSX, JSON, TSV, YAML)
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['schema_file'], {'type': <class 'str'>, 'help': 'Path to the schema (.py) or a declarative description of the schema (.csv, .tsv, .xlsx)'}), (['in_wb_file'], {'type': <class 'str'>, 'help': 'Path to the workbook (.csv, .json, .tsv, .xlsx, .yml)'}), (['out_wb_file'], {'type': <class 'str'>, 'help': 'Path to save the workbook (.csv, .json, .tsv, .xlsx, .yml)'}), (['--write-toc'], {'action': 'store_true', 'default': False, 'help': 'If set, write a table of contents with the outputted workbook'}), (['--write-schema'], {'action': 'store_true', 'default': False, 'help': 'If set, save a copy of the schema within the outputted workbook'}), (['--unprotected'], {'action': 'store_true', 'default': False, 'help': 'If set, do not protect the outputted workbook'})][source]¶
-
-
class
-
class
obj_tables.__main__.
DiffController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Calculate the difference between two workbooks according to a schema
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['schema_file'], {'type': <class 'str'>, 'help': 'Path to the schema (.py) or a declarative description of the schema (.csv, .tsv, .xlsx)'}), (['model'], {'type': <class 'str'>, 'help': 'Type of objects to compare'}), (['wb_file_1'], {'type': <class 'str'>, 'help': 'Path to the first workbook (.csv, .json, .tsv, .xlsx, .yml)'}), (['wb_file_2'], {'type': <class 'str'>, 'help': 'Path to the second workbook (.csv, .json, .tsv, .xlsx, .yml)'})][source]¶
-
-
class
-
class
obj_tables.__main__.
GenTemplateController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Generate a template workbook (XLSX, CSV, TSV) for a schema or declarative description of a schema
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['schema_file'], {'type': <class 'str'>, 'help': 'Path to the schema (.py) or declarative description of the schema (.csv, .tsv, .xlsx)'}), (['template_file'], {'type': <class 'str'>, 'help': 'Path to save the template (.csv, .tsv, .xlsx)'}), (['--write-toc'], {'action': 'store_true', 'default': False, 'help': 'If set, write a table of contents with the outputted workbook'}), (['--write-schema'], {'action': 'store_true', 'default': False, 'help': 'If set, save a copy of the schema within the template'}), (['--unprotected'], {'action': 'store_true', 'default': False, 'help': 'If set, do not protect the outputted workbook'})][source]¶
-
description
= 'Generate a template workbook (XLSX, CSV, TSV) for a schema or declarative description of a schema'[source]¶
-
-
class
-
class
obj_tables.__main__.
InitSchemaController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Initialize a Python schema from a declarative description of the schema in a table (XLSX, CSV, TSV)
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['in_file'], {'type': <class 'str'>, 'help': 'Path to the declarative description of the schema (.csv, .tsv, .xlsx)'}), (['out_file'], {'type': <class 'str'>, 'help': 'Path to save Python schema (.py)'})][source]¶
-
description
= 'Initialize a Python schema from a declarative description of the schema in a table (XLSX, CSV, TSV)'[source]¶
-
-
class
-
class
obj_tables.__main__.
NormalizeController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Normalize a workbook according to a schema
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['schema_file'], {'type': <class 'str'>, 'help': 'Path to the schema (.py) or a declarative description of the schema (.csv, .tsv, .xlsx)'}), (['model'], {'type': <class 'str'>, 'help': 'Type of objects to normalize'}), (['in_wb_file'], {'type': <class 'str'>, 'help': 'Path to the workbook (.csv, .json, .tsv, .xlsx, .yml)'}), (['out_wb_file'], {'type': <class 'str'>, 'help': 'Path to save the normalized workbook (.csv, .json, .tsv, .xlsx, .yml)'}), (['--write-toc'], {'action': 'store_true', 'default': False, 'help': 'If set, write a table of contents with the outputted workbook'}), (['--write-schema'], {'action': 'store_true', 'default': False, 'help': 'If set, save a copy of the schema within the normalized workbook'}), (['--unprotected'], {'action': 'store_true', 'default': False, 'help': 'If set, do not protect the outputted workbook'})][source]¶
-
-
class
-
class
obj_tables.__main__.
ValidateController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Validate that a workbook is consistent with a schema, and report any errors
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['schema_file'], {'type': <class 'str'>, 'help': 'Path to the schema (.py) or a declarative description of the schema (.csv, .tsv, .xlsx)'}), (['wb_file'], {'type': <class 'str'>, 'help': 'Path to the workbooks (.csv, .json, .tsv, .xlsx, .yml)'})][source]¶
-
-
class
-
class
obj_tables.__main__.
VizSchemaController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Generate a UML diagram for a schema
-
class
Meta
[source]¶ Bases:
object
-
class
3.1.4. obj_tables._version module¶
3.1.5. obj_tables.abstract module¶
Support for abstract model classes
- Author
Jonathan Karr <karr@mssm.edu>
- Date
2017-05-23
- Copyright
2016, Karr Lab
- License
MIT
-
class
obj_tables.abstract.
AbstractModel
(_comments=None, **kwargs)[source]¶ Bases:
obj_tables.core.Model
Abstract model base class
- Parameters
**kwargs – dictionary of keyword arguments with keys equal to the names of the model attributes
- Raises
TypeError – if keyword argument is not a defined attribute
-
class
obj_tables.abstract.
AbstractModelMeta
[source]¶ Bases:
obj_tables.core.ModelMeta
,abc.ABCMeta
Abstract model metaclass
- Parameters
metacls (
Model
) –Model
, or a subclass ofModel
name (
str
) –Model
class namebases (
tuple
) – tuple of superclassesnamespace (
dict
) – namespace ofModel
class definition
- Returns
a new instance of
Model
, or a subclass ofModel
- Return type
Model
3.1.6. obj_tables.core module¶
Toolkit for modeling complex datasets with collections of user-friendly tables
Many classes contain the methods serialize()
and deserialize()`, which invert each other.
serialize()
converts a python object instance into a string representation, whereas
deserialize()
parses an object’s string representation – as would be stored in a file or spreadsheet
representation of a biochemical model – into a python object instance.
deserialize()
returns an error when the string representation cannot be parsed into the
python object. Deserialization methods for related attributes (subclasses of RelatedAttribute
)
do not get called until all other attributes have been deserialized. In particular, they’re called
by obj_tables.io.WorkbookReader.link_model
. Therefore, they get passed all objects that are not inline,
which can then be referenced to deserialize the related attribute.
- Author
Jonathan Karr <karr@mssm.edu>
- Author
Arthur Goldberg <Arthur.Goldberg@mssm.edu>
- Date
2016-12-12
- Copyright
2016, Karr Lab
- License
MIT
-
class
obj_tables.core.
Attribute
(init_value=None, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False, unique_case_insensitive=False)[source]¶ Bases:
object
Model attribute
-
type
[source]¶ allowed type(s) of the values of the attribute
- Type
types.TypeType
ortuple
oftypes.TypeType
-
default_cleaned_value
[source]¶ value to replace
None
values with during cleaning, or function which computes the value to replaceNone
values- Type
object
- Parameters
init_value (
object
, optional) – initial valuedefault (
object
, optional) – default valuedefault_cleaned_value (
object
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be uniqueunique_case_insensitive (
bool
, optional) – if true, conduct case-insensitive test of uniqueness
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
abstract
copy_value
(value, objects_and_copies)[source]¶ Copy value
- Parameters
value (
object
) – valueobjects_and_copies (
dict
) – dictionary that maps objects to their copies
- Returns
copy of value
- Return type
object
-
abstract
deserialize
(value)[source]¶ Deserialize value
- Parameters
value (
object
) – semantically equivalent representation- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
abstract
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
object
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
object
-
get_default_cleaned_value
()[source]¶ Get value to replace
None
values with during cleaning- Returns
initial value
- Return type
object
-
get_init_value
(obj)[source]¶ Get initial value for attribute
- Parameters
obj (
Model
) – object whose attribute is being initialized- Returns
initial value
- Return type
object
-
abstract
merge
(left, right, right_objs_in_left, left_objs_in_right)[source]¶ Merge an attribute of elements of two models
-
abstract
serialize
(value)[source]¶ Serialize value
- Parameters
value (
object
) – Python representation- Returns
simple Python representation
- Return type
bool
,float
,str
, orNone
-
set_value
(obj, new_value)[source]¶ Set value of attribute of object
- Parameters
obj (
Model
) – objectnew_value (
object
) – new attribute value
- Returns
attribute value
- Return type
object
-
abstract
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
object
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
object
-
abstract
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, otherwise return a list
of errors as an instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
validate_unique
(objects, values)[source]¶ Determine if the attribute values are unique
- Parameters
- Returns
- None if values are unique, otherwise return a list of
errors as an instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
-
class
obj_tables.core.
BaseRelatedAttribute
[source]¶ Bases:
object
Attribute which represents a relationship with 1 or more other
Model
s
-
class
obj_tables.core.
BooleanAttribute
(default=False, default_cleaned_value=None, none_value=None, verbose_name='', description='Enter a Boolean value')[source]¶ Bases:
obj_tables.core.LiteralAttribute
Boolean attribute
- Parameters
default (
bool
, optional) – default valuedefault_cleaned_value (
bool
, optional) – value to replaceNone
values with during cleaningnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – description
- Raises
ValueError – if
default
is not abool
or ifdefault_cleaned_value
is not abool
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofbool
,InvalidAttribute
orNone
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
CellDialect
[source]¶ Bases:
str
,enum.Enum
Dialect for serializing values to a cell
-
class
obj_tables.core.
DateAttribute
(none=True, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.LiteralAttribute
Date attribute
-
default_cleaned_value
[source]¶ value to replace
None
values with during cleaning, or function which computes the value to replaceNone
values- Type
date
- Parameters
none (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
date
, optional) – default datedefault_cleaned_value (
date
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
(
date
,None
), or (None
,InvalidAttribute
) reporting error- Return type
tuple
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
str
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
date
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
serialize
(value)[source]¶ Serialize string
- Parameters
value (
date
) – Python representation- Returns
simple Python representation
- Return type
str
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
date
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
str
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
date
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
-
class
obj_tables.core.
DateTimeAttribute
(none=True, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.LiteralAttribute
Datetime attribute
-
default_cleaned_value
[source]¶ value to replace
None
values with during cleaning, or function which computes the value to replaceNone
values- Type
datetime
- Parameters
none (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
datetime
, optional) – default datetimedefault_cleaned_value (
datetime
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
tuple
ofobject
,InvalidAttribute
ifvalue
is invalid, ortuple
ofdatetime
,None
with cleaned value otherwise
- Return type
tuple
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
str
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
datetime
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
serialize
(value)[source]¶ Serialize string
- Parameters
value (
datetime
) – Python representation- Returns
simple Python representation
- Return type
str
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
datetime
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
str
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
datetime
) – value of attribute to validate
- Returns
None
if attribute is valid, otherwise return list oferrors as an instance of
InvalidAttribute
- Return type
None
orInvalidAttribute
-
-
class
obj_tables.core.
EmailAttribute
(verbose_name='', description='Enter a valid email address', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.StringAttribute
Attribute for email addresses
- Parameters
verbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
date
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
EnumAttribute
(enum_class, none=False, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False, unique_case_insensitive=False)[source]¶ Bases:
obj_tables.core.LiteralAttribute
Enumeration attribute
- Parameters
enum_class (
type
orlist
) – subclass ofEnum
,list
of enumerated names,list
of 2-tuples of each enumerated name and its value, or adict
which maps enumerated names to their valuesnone (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
object
, optional) – default valuedefault_cleaned_value (
Enum
, optional) – value to replaceNone
values with during cleaningnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be uniqueunique_case_insensitive (
bool
, optional) – if true, conduct case-insensitive test of uniqueness
- Raises
ValueError – if
enum_class
is not a subclass ofEnum
, ifdefault
is not an instance ofenum_class
, or ifdefault_cleaned_value
is not an instance ofenum_class
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofEnum
,InvalidAttribute
orNone
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
str
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
Enum
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
serialize
(value)[source]¶ Serialize enumeration
- Parameters
value (
Enum
) – Python representation- Returns
simple Python representation
- Return type
str
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
Enum
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
str
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
FloatAttribute
(min=nan, max=nan, nan=True, default=nan, default_cleaned_value=nan, none_value=nan, verbose_name='', description='', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.NumericAttribute
Float attribute
- Parameters
min (
float
, optional) – minimum valuemax (
float
, optional) – maximum valuenan (
bool
, optional) – if true, allow nan valuesdefault (
float
, optional) – default valuedefault_cleaned_value (
float
, optional) – value to replaceNone
values with during cleaningverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
- Raises
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
offloat
,InvalidAttribute
orNone
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
merge
(left, right, right_objs_in_left, left_objs_in_right)[source]¶ Merge an attribute of elements of two models
- Parameters
- Raises
ValueError – if the attributes of the elements of the models are different
-
serialize
(value)[source]¶ Serialize float
- Parameters
value (
float
) – Python representation- Returns
simple Python representation
- Return type
float
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
IntegerAttribute
(min=None, max=None, none=False, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.NumericAttribute
Integer attribute
- Parameters
min (
int
, optional) – minimum valuemax (
int
, optional) – maximum valuenone (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
int
, optional) – default valuedefault_cleaned_value (
int
, optional) – value to replaceNone
values with during cleaningnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
- Raises
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofint
,InvalidAttribute
orNone
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
float
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
int
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
serialize
(value)[source]¶ Serialize integer
- Parameters
value (
int
) – Python representation- Returns
simple Python representation
- Return type
float
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
int
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
float
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, otherwise return list of
errors as an instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
InvalidAttribute
(attribute, messages, related=False, location=None, value=None)[source]¶ Bases:
object
Represents an invalid attribute and its errors
indicates if error is about value or related value
- Type
bool
-
location
[source]¶ a string representation of the attribute’s location in an input file
- Type
str
, optional
- Parameters
attribute (
Attribute
) – invalid attributemessage (
list
ofstr
) – list of error messagesrelated (
bool
, optional) – indicates if error is about value or related valuelocation (
str
, optional) – a string representation of the attribute’s location in an input filevalue (
str
, optional) – invalid input value
-
class
obj_tables.core.
InvalidModel
(model, attributes)[source]¶ Bases:
object
Represents an invalid model, such as a model with an attribute that fails to meet specified constraints
-
attributes
[source]¶ list of invalid attributes and their errors
- Type
list
ofInvalidAttribute
- Parameters
model (
class
) –Model
classattributes (
list
ofInvalidAttribute
) – list of invalid attributes and their errors
-
-
class
obj_tables.core.
InvalidObject
(object, attributes)[source]¶ Bases:
object
Represents an invalid object and its errors
-
attributes
[source]¶ list of invalid attributes and their errors
- Type
list
ofInvalidAttribute
- Parameters
object (
Model
) – invalid objectattributes (
list
ofInvalidAttribute
) – list of invalid attributes and their errors
-
-
class
obj_tables.core.
InvalidObjectSet
(invalid_objects, invalid_models)[source]¶ Bases:
object
Represents a list of invalid objects and invalid models
-
invalid_objects
[source]¶ list of invalid objects
- Type
list
ofInvalidObject
-
invalid_models
[source]¶ list of invalid models
- Type
list
ofInvalidModel
- Parameters
invalid_objects (
list
ofInvalidObject
) – list of invalid objectsinvalid_models (
list
ofInvalidModel
) – list of invalid models
- Raises
ValueError –
invalid_models
is not unique
-
__str__
()[source]¶ Get string representation of errors
- Returns
string representation of errors
- Return type
str
-
get_model_errors_by_model
()[source]¶ Get model errors grouped by models
- Returns
InvalidModel
: dictionary of model errors, grouped by model- Return type
dict
ofModel
-
get_object_errors_by_model
()[source]¶ Get object errors grouped by model
- Returns
list
ofInvalidObject
: dictionary of object errors, grouped by model- Return type
dict
ofModel
-
-
class
obj_tables.core.
ListAttribute
(type=<class 'str'>, separator=', ', default=[], none_value=[], verbose_name='', description='A list of values')[source]¶ Bases:
obj_tables.core.LiteralAttribute
List attribute
- Parameters
type (
type
, optional) – type of elementsseparator (
str
, optional) – element separator for serializationdefault (
list
, optional) – default valuenone_value (
list
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – description
-
clean
(value)[source]¶ Deserialize value
- Parameters
value (
str
) – semantically equivalent representation- Returns
list
: cleaned valueInvalidAttribute
: cleaning error
- Return type
tuple
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
dict
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
list
-
serialize
(value)[source]¶ Serialize string
- Parameters
value (
list
) – Python representation- Returns
simple Python representation
- Return type
str
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
list
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
dict
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value- Parameters
obj (
Model
) – class being validatedvalue (
list
) – value of attribute to validate
- Returns
- None if attribute is valid, other return
list of errors as an instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
LiteralAttribute
(init_value=None, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False, unique_case_insensitive=False)[source]¶ Bases:
obj_tables.core.Attribute
Base class for literal attributes (Boolean, enumeration, float, integer, string, etc.)
- Parameters
init_value (
object
, optional) – initial valuedefault (
object
, optional) – default valuedefault_cleaned_value (
object
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be uniqueunique_case_insensitive (
bool
, optional) – if true, conduct case-insensitive test of uniqueness
-
copy_value
(value, objects_and_copies)[source]¶ Copy value
- Parameters
value (
object
) – valueobjects_and_copies (
dict
) – dictionary that maps objects to their copies
- Returns
copy of value
- Return type
object
-
deserialize
(value)[source]¶ Deserialize value
- Parameters
value (
object
) – semantically equivalent representation- Returns
- tuple
of cleaned value and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
from_builtin
(json)[source]¶ Decode a simple Python representation (
dict
,list
,str
,float
,bool
,None
) of a value of the attribute that is compatible with JSON and YAML- Parameters
json (
object
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
object
-
merge
(left, right, right_objs_in_left, left_objs_in_right)[source]¶ Merge an attribute of elements of two models
- Parameters
- Raises
ValueError – if the attributes of the elements of the models are different
-
serialize
(value)[source]¶ Serialize value
- Parameters
value (
object
) – Python representation- Returns
- simple Python
representation
- Return type
bool
,float
,str
, orNone
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (
dict
,list
,str
,float
,bool
,None
) that is compatible with JSON and YAML- Parameters
value (
object
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
object
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, otherwise return a
list of errors as an instance of
InvalidAttribute
- Return type
InvalidAttribute
orNone
-
class
obj_tables.core.
LocalAttribute
(attr, primary_class, is_primary=True)[source]¶ Bases:
object
Meta data about a local attribute in a class
other class which is related to this attribute
- Type
name of this attribute in
related_cls
- Type
str
True
if this attribute is an instance ofRelatedAttribute
- Type
bool
True
if the value of this attribute is a list (*-to-many relationship)- Type
bool
minimum number of related objects in the forward direction
- Type
int
maximum number of related objects in the forward direction
- Type
int
minimum number of related objects in the reverse direction
- Type
int
maximum number of related objects in the reverse direction
- Type
int
-
class
obj_tables.core.
LocalPathAttribute
(verbose_name='', description='Enter a path to a local file or directory', none=False, default=None, default_cleaned_value=None, none_value=None, primary=False, unique=False)[source]¶ Bases:
obj_tables.core.LongStringAttribute
Attribute to be used for paths to local files and directories
- Parameters
verbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionnone (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
str
, optional) – default valuedefault_cleaned_value (
str
, optional) – value to replaceNone
values with during cleaningprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
(
pathlib.Path
,None
), or (None
,InvalidAttribute
) reporting error- Return type
tuple
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
str
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
pathlib.Path
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
serialize
(value)[source]¶ Serialize string
- Parameters
value (
pathlib.Path
) – Python representation- Returns
simple Python representation
- Return type
str
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
pathlib.Path
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
str
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
pathlib.Path
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
LongStringAttribute
(min_length=0, max_length=4294967295, none=False, default='', default_cleaned_value='', none_value='', verbose_name='', description='', primary=False, unique=False, unique_case_insensitive=False)[source]¶ Bases:
obj_tables.core.StringAttribute
Long string attribute
- Parameters
min_length (
int
, optional) – minimum lengthmax_length (
int
, optional) – maximum lengthnone (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
str
, optional) – default valuedefault_cleaned_value (
str
, optional) – value to replaceNone
values with during cleaningverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be uniqueunique_case_insensitive (
bool
, optional) – if true, conduct case-insensitive test of uniqueness
-
class
obj_tables.core.
Manager
(cls)[source]¶ Bases:
object
Enable O(1) dictionary-based searching of a Model’s instances
This class is inspired by Django’s
Manager
class. An instance ofManger
is associated with eachModel
and accessed as the class attributeobjects
(as in Django). The tuples of attributes to index are specified by theindexed_attrs_tuples
attribute ofModel.Meta
, which contains a tuple of tuples of attributes to index.Model
s with emptyindexed_attrs_tuples
attributes incur no overhead fromManager
.Manager
maintains a dictionary for each indexed attribute tuple, and a reverse index from eachModel
instance to its indexed attribute tuple keys.These data structures support * O(1) get operations for
Model
instances indexed by a indexed attribute tuple * O(1)Model
instance insert and update operations-
_new_instances
[source]¶ set of all new instances of
cls
that have not been indexed, stored as weakrefs, soModel
’s that are otherwise unused can be garbage collected- Type
WeakSet
-
_index_dicts
[source]¶ indices that enable lookup of
Model
instances from theirMeta.indexed_attrs_tuples
mapping: <attr names tuple> -> <attr values tuple> -> WeakSet(<model_obj instances>)- Type
dict
mappingtuple
toWeakSet
-
_reverse_index
[source]¶ a reverse index that provides all of each
Model
’s indexed attribute tuple keys mapping: <model_obj instances> -> <attr names tuple> -> <attr values tuple>- Type
WeakKeyDictionary
mappingModel
instance todict
- Parameters
cls (
class
) – theModel
class which is being managed
-
get
(**kwargs)[source]¶ Get the
Model
instance(s) that match the attribute name,value pair(s) inkwargs
The keys in
kwargs
must correspond to an entry in theModel
’sindexed_attrs_tuples
. Warning: this method is non-deterministic. To obtainManager
’s O(1) performance,Model
instances in the index are stored inWeakSet
’s. Therefore, the order of elements in the list returned is not reproducible. Applications that need reproducibility must deterministically order elements in lists returned by this method.- Parameters
**kwargs – keyword args mapping from attribute name(s) to value(s)
- Returns
- a list of
Model
instances whose indexed attribute tuples have the values in
kwargs
; otherwiseNone
, indicating no match
- a list of
- Return type
list
ofModel
- Raises
ValueError – if no arguments are provided, or the attribute name(s) in
kwargs.keys()
do not correspond to an indexed attribute tuple of theModel
-
get_one
(**kwargs)[source]¶ Get one
Model
instance that matches the attribute name,value pair(s) inkwargs
Uses
get
.
-
insert_all_new
()[source]¶ Insert all new instances of this
Manager
’sModel
’s into the search indices
-
reset
()[source]¶ Reset this
Manager
Empty
Manager
’s indices. SinceManager
globally indexes all instances of aModel
, this method is useful when multiple models are loaded sequentially.
-
upsert
(model_obj)[source]¶ Update the indices for
model_obj
that are used to search on indexed attribute tuplesUpsert
means update or insert. Update the indices ifmodel_obj
is already stored, otherwise insertmodel_obj
. Users ofManager
are responsible for calling this method ifmodel_obj
changes.Costs O(I) where I is the number of indexed attribute tuples for the
Model
.
-
-
class
obj_tables.core.
ManyToManyAttribute
(related_class, related_name='', default=[], default_cleaned_value=[], related_default=[], none_value=<class 'list'>, separator=', ', min_related=0, max_related=inf, min_related_rev=0, max_related_rev=inf, verbose_name='', verbose_related_name='', description='', related_manager=<class 'obj_tables.core.ManyToManyRelatedManager'>, cell_dialect=<CellDialect.json: 'json'>)[source]¶ Bases:
obj_tables.core.ToManyAttribute
,obj_tables.core.RelatedAttribute
Represents a many-to-many relationship between two types of objects.
related manager
- Type
type
- Parameters
related_class (
class
) – related classrelated_name (
str
, optional) – name of related attribute onrelated_class
default (
callable
, optional) – function which returns the default valuesdefault_cleaned_value (
callable
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesrelated_default (
callable
, optional) – function which returns the default related valuesnone_value (
object
, optional) – none valueseparator (
str
, optional) – element separator for serializationmin_related (
int
, optional) – minimum number of related objects in the forward directionmax_related (
int
, optional) – maximum number of related objects in the forward directionmin_related_rev (
int
, optional) – minimum number of related objects in the reverse directionmax_related_rev (
int
, optional) – maximum number of related objects in the reverse directionverbose_name (
str
, optional) – verbose nameverbose_related_name (
str
, optional) – verbose related namedescription (
str
, optional) – descriptionrelated_manager (
type
, optional) – related managercell_dialect (
CellDialect
, optional) – dialect for serializing values to a cell
-
deserialize
(values, objects, decoded=None)[source]¶ Deserialize value
- Parameters
values (
object
) – String representationobjects (
dict
) – dictionary of objects, grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
get_init_value
(obj)[source]¶ Get initial value for attribute
- Parameters
obj (
Model
) – object whose attribute is being initialized- Returns
initial value
- Return type
object
Get initial related value for attribute
- Parameters
obj (
object
) – object whose attribute is being initialized- Returns
initial value
- Return type
value (
object
)- Raises
ValueError – if related property is not defined
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
merge
(left, right, right_objs_in_left, left_objs_in_right)[source]¶ Merge an attribute of elements of two models
- Parameters
- Raises
ValueError – if the attributes of the elements of the models are different
Determine if
value
is a valid value of the related attribute- Parameters
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
serialize
(value, encoded=None)[source]¶ Serialize related object
- Parameters
value (
list
ofModel
) – Python representationencoded (
dict
, optional) – dictionary of objects that have already been encoded
- Returns
simple Python representation
- Return type
str
Update the values of the related attributes of the attribute
- Parameters
obj (
object
) – object whose attribute should be setnew_values (
list
) – value of the attribute
- Returns
value of the attribute
- Return type
list
- Raises
ValueError – if related property is not defined
-
set_value
(obj, new_values)[source]¶ Get value of attribute of object
- Parameters
obj (
Model
) – objectnew_values (
list
) – new attribute value
- Returns
new attribute value
- Return type
list
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
ManyToManyRelatedManager
(object, attribute, related=True)[source]¶ Bases:
obj_tables.core.RelatedManager
Represent values and related values of related attributes
- Parameters
-
append
(value, propagate=True)[source]¶ Add value to list
- Parameters
value (
object
) – valuepropagate (
bool
, optional) – propagate change to related attribute
- Returns
self
- Return type
-
cut
(kind=None)[source]¶ Cut values and their children of kind
kind
into separate graphs.If
kind
isNone
, children are defined to be the values of the related attributes defined in each class.- Parameters
kind (
str
, optional) – kind of children to include- Returns
cut values and their children
- Return type
list
ofModel
-
class
obj_tables.core.
ManyToOneAttribute
(related_class, related_name='', default=None, default_cleaned_value=None, related_default=[], none_value=None, min_related=0, min_related_rev=0, max_related_rev=inf, verbose_name='', verbose_related_name='', description='', related_manager=<class 'obj_tables.core.ManyToOneRelatedManager'>)[source]¶ Bases:
obj_tables.core.RelatedAttribute
Represents a many-to-one relationship between two types of objects. This is analagous to a foreign key relationship in a database.
related manager
- Type
type
- Parameters
related_class (
class
) – related classrelated_name (
str
, optional) – name of related attribute onrelated_class
default (
callable
, optional) – callable which returns the default valuedefault_cleaned_value (
callable
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesrelated_default (
callable
, optional) – callable which returns the default related valuenone_value (
object
, optional) – none valuemin_related (
int
, optional) – minimum number of related objects in the forward directionmin_related_rev (
int
, optional) – minimum number of related objects in the reverse directionmax_related_rev (
int
, optional) – maximum number of related objects in the reverse directionverbose_name (
str
, optional) – verbose nameverbose_related_name (
str
, optional) – verbose related namedescription (
str
, optional) – descriptionrelated_manager (
type
, optional) – related manager
-
deserialize
(value, objects, decoded=None)[source]¶ Deserialize value
- Parameters
value (
str
) – String representationobjects (
dict
) – dictionary of objects, grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
deserialize_from_cell
(value, objects, decoded=None)[source]¶ Deserialize value from cell
- Parameters
value (
str
) – String representationobjects (
dict
) – dictionary of objects, grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
Get initial related value for attribute
- Parameters
obj (
object
) – object whose attribute is being initialized- Returns
initial value
- Return type
value (
object
)- Raises
ValueError – if related property is not defined
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
merge
(left, right, right_objs_in_left, left_objs_in_right)[source]¶ Merge an attribute of elements of two models
- Parameters
- Raises
ValueError – if the attributes of the elements of the models are different
Determine if
value
is a valid value of the related attribute- Parameters
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
serialize
(value, encoded=None)[source]¶ Serialize related object
- Parameters
value (
Model
) – Python representationencoded (
dict
, optional) – dictionary of objects that have already been encoded
- Returns
simple Python representation
- Return type
str
-
serialize_to_cell
(value, encoded=None)[source]¶ Serialize related object
- Parameters
value (
Model
) – Python representationencoded (
dict
, optional) – dictionary of objects that have already been encoded
- Returns
string representation
- Return type
str
Update the values of the related attributes of the attribute
- Parameters
obj (
object
) – object whose attribute should be setnew_values (
list
) – value of the attribute
- Returns
value of the attribute
- Return type
list
- Raises
ValueError – if related property is not defined
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
ManyToOneRelatedManager
(object, attribute)[source]¶ Bases:
obj_tables.core.RelatedManager
Represent values of related attributes
-
append
(value, propagate=True)[source]¶ Add value to list
- Parameters
value (
object
) – valuepropagate (
bool
, optional) – propagate change to related attribute
- Returns
self
- Return type
-
cut
(kind=None)[source]¶ Cut values and their children of kind
kind
into separate graphs.If
kind
isNone
, children are defined to be the values of the related attributes defined in each class.- Parameters
kind (
str
, optional) – kind of children to include- Returns
cut values and their children
- Return type
list
ofModel
-
-
class
obj_tables.core.
Model
(_comments=None, **kwargs)[source]¶ Bases:
object
Base object model
- Parameters
**kwargs – dictionary of keyword arguments with keys equal to the names of the model attributes
- Raises
TypeError – if keyword argument is not a defined attribute
-
class
Meta
[source]¶ Bases:
object
Meta data for
Model
attributes declared in related objects
- Type
collections.OrderedDict
ofstr, :obj:`Attribute
-
local_attributes
[source]¶ dictionary that maps the names of all local attributes to their instances, including attributes defined in this class and attributes defined in related classes
- Type
collections.OrderedDict
ofstr
,Attribute
-
unique_together
[source]¶ controls what tuples of attribute values must be unique
- Type
tuple
oftuple
’s of attribute names
-
indexed_attrs_tuples
[source]¶ tuples of attributes on which instances of this
Model
will be indexed by theModel
’sManager
- Type
tuple
oftuple
’s of attribute names
-
attribute_order
[source]¶ tuple of attribute names, in the order in which they should be displayed
- Type
tuple
ofstr
-
description
[source]¶ description of the model (e.g., to print in the table of contents in XLSX)
- Type
str
-
ordering
[source]¶ controls the order in which objects should be printed when serialized
- Type
tuple
of attribute names
-
children
[source]¶ dictionary that maps types of children to names of attributes which compose each type of children
- Type
dict
that mapsstr
totuple
ofstr
-
attribute_order
= ()[source]
-
attributes
= None[source]
-
children
= {}[source]
-
description
= ''[source]
-
frozen_columns
= 1[source]
-
indexed_attrs_tuples
= ()[source]
-
inheritance
= None[source]
-
merge
= 1[source]
-
ordering
= None[source]
-
primary_attribute
= None[source]
-
related_attributes
= None[source]
-
table_format
= 1[source]
-
unique_together
= ()[source]
-
verbose_name
= ''[source]
-
verbose_name_plural
= ''[source]
-
__setattr__
(attr_name, value, propagate=True)[source]¶ Set attribute and validate any unique attribute constraints
- Parameters
attr_name (
str
) – attribute namevalue (
object
) – valuepropagate (
bool
, optional) – propagate change through attributeset_value
andset_related_value
-
__str__
()[source]¶ Get the string representation of an object
- Returns
string representation of object
- Return type
str
-
classmethod
are_attr_paths_equal
(attr_path, other_attr_path)[source]¶ Determine if two attribute paths are semantically equal
- Parameters
attr_path (
list
oflist
ofobject
) – the path to an attribute or nested attribute of a modelother_attr_path (
list
oflist
ofobject
) – the path to another attribute or nested attribute of a model
- Returns
True
if the paths are semantically equal- Return type
bool
Determine if the immediate related attributes of the class can be serialized
- Returns
True
if the related attributes can be serialized- Return type
bool
-
clean
()[source]¶ Clean all of this
Model
’s attributes- Returns
None
if the object is valid,otherwise return a list of errors as an instance of
InvalidObject
- Return type
InvalidObject
or None
-
cut
(kind=None)[source]¶ Cut the object and its children from the rest of the object graph.
If
kind
isNone
, children are defined to be the values of the related attributes defined in each class.- Parameters
kind (
str
, optional) – kind of children to get- Returns
same object, but cut from the rest of the object graph
- Return type
-
cut_relations
(objs_to_keep=None)[source]¶ Cut relations to objects not in
objs
.- Parameters
objs_to_keep (
set
ofModel
, optional) – objects to retain relations to
-
classmethod
deserialize
(value, objects)[source]¶ Deserialize value
- Parameters
value (
str
) – String representationobjects (
dict
) – dictionary of objects, grouped by model
- Returns
- tuple of cleaned value
and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
static
from_dict
(json, models, decode_primary_objects=True, primary_objects=None, decoded=None, ignore_extra_models=False, validate=False, output_format=None)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of an object that is compatible with JSON and YAML, including references to objects through
__id
keys.- Parameters
json (
dict
) – simple Python representation of the objectdecode_primary_objects (
bool
, optional) – ifTrue
, decode primary classes otherwise just look up objects by their IDsprimary_objects (
list
, optional) – list of instances of primary classes (i.e. non-line classes)decoded (
dict
, optional) – dictionary of objects that have already been decodedignore_extra_models (
bool
, optional) – ifTrue
and allmodels
are found, ignore other worksheets or filesvalidate (
bool
, optional) – ifTrue
, validate the dataoutput_format (
str
, optional) –desired structure of the return value
- Returns
decoded object
- Return type
-
gen_merge_map
(other)[source]¶ Create a dictionary that maps instances of objects in another model to objects in a model
- Parameters
other (
Model
) – other model- Returns
dict
: dictionary that maps instances of objects in another model to objects in a modellist
: list of instances of objects in another model which have no parallel in the model
- Return type
tuple
-
gen_serialized_val_obj_map
()[source]¶ Generate mappings from serialized values to objects
- Returns
dictionary which maps types of models to dictionaries which serialized values to objects
- Return type
dict
- Raises
ValueError – if serialized values are not unique within each type
Optimally obtain all objects related to objects in
objs
The set of all
Model
s can be viewed as a graph whose nodes areModel
instances and whose edges are related connections. Because related edges are bi-directional, this graph is a set of strongly connected components and no edges connect the components.The algorithm here finds all
Model
s that are reachable from a set of instances in \(O(n)\), where \(n\) is the size of the reachable set. This algorithm is optimal. It achieves this performance becauseobj.get_related()
takes \(O(n(c))\) where \(n(c)\) is the number of nodes in the component containingobj
, and each component is only explored once because all of a component’s nodes are stored infound_objs
when the component is first explored.In addition, this method is deterministic because ordered dictionaries preserve insertion order.
-
classmethod
get_attr_index
(attr)[source]¶ Get the index of an attribute within
Meta.attribute_order
- Parameters
attr (
Attribute
) – attribute- Returns
index of attribute within
Meta.attribute_order
- Return type
int
-
classmethod
get_attrs
(type=None, forward=True, reverse=True)[source]¶ Get attributes of a type, optionally including attributes from related classes. By default, return all attributes.
- Parameters
type (
type
ortuple
oftype
, optional) – type of attributes to getforward (
bool
, optional) – ifTrue
, include attributes from classreverse (
bool
, optional) – ifTrue
, include attributes from related classes
- Returns
- dictionary of the names and instances
of matching attributes
- Return type
dict
ofstr
,Attribute
-
get_attrs_by_val
(type=None, reverse=True, include=None, exclude=None)[source]¶ Get attributes whose type is
type
and values are ininclude
and notexclude
, optionally including attributes from related classes. By default, get all attributes.- Parameters
type (
type
ortuple
oftype
, optional) – type of attributes to getreverse (
bool
, optional) – ifTrue
, include attributes from related classesinclude (
list
, optional) – list of values to filter forexclude (
list
, optional) – list of values to filter out
- Returns
- dictionary of the names and instances
of matching attributes
- Return type
dict
ofstr
,Attribute
-
get_children
(kind=None, _Model__type=None, recursive=True, **kwargs)[source]¶ Get a kind of children.
If
kind
isNone
, children are defined to be the values of the related attributes defined in each class.
-
get_empty_literal_attrs
()[source]¶ Get empty (
None
, ‘’, or NaN) literal attributes- Returns
- dictionary of the names and instances
of empty literal attributes
- Return type
dict
ofstr
,Attribute
Get empty (
None
or []) related attributes- Parameters
reverse (
bool
, optional) – ifTrue
, include attributes from related classes- Returns
- dictionary of the names and instances
of empty related attributes
- Return type
dict
ofstr
,Attribute
-
get_immediate_children
(kind=None, _Model__type=None, **kwargs)[source]¶ Get a kind of immediate children
If
kind
isNone
, children are defined to be the values of the related attributes defined in each class.
-
classmethod
get_literal_attrs
()[source]¶ Get literal attributes
- Returns
- dictionary of the names and instances
of literal attributes
- Return type
dict
ofstr
,Attribute
-
classmethod
get_nested_attr
(attr_path)[source]¶ Get the value of an attribute or a nested attribute of a model
- Parameters
attr_path (
list
oflist
ofstr
) – the path to an attribute or nested attribute of a model- Returns
nested attribute
- Return type
-
get_nested_attr_val
(attr_path)[source]¶ Get the value of an attribute or a nested attribute of a model
- Parameters
attr_path (
list
oflist
ofobject
) – the path to an attribute or nested attribute of a model- Returns
value of the attribute or nested attribute
- Return type
Object
-
get_non_empty_literal_attrs
()[source]¶ Get non-empty (
None
, ‘’, or NaN) literal attributes- Returns
- dictionary of the names and instances
of non-empty literal attributes
- Return type
dict
ofstr
,Attribute
Get non-empty (
None
or []) related attributes- Parameters
reverse (
bool
, optional) – ifTrue
, include attributes from related classes- Returns
- dictionary of the names and instances
of non-empty related attributes
- Return type
dict
ofstr
,Attribute
-
get_primary_attribute
()[source]¶ Get value of primary attribute
- Returns
value of primary attribute
- Return type
object
Get all related objects reachable from
self
- Parameters
forward (
bool
, optional) – ifTrue
, get all forward related objectsreverse (
bool
, optional) – ifTrue
, get all reverse related objects
- Returns
related objects, without any duplicates
- Return type
list
ofModel
Get related attributes
- Parameters
reverse (
bool
, optional) – ifTrue
, include attributes from related classes- Returns
- dictionary of the names and instances
of related attributes
- Return type
dict
ofstr
,Attribute
-
classmethod
get_sort_key
(object, attr_name)[source]¶ Get sort key for
Model
instanceobject
based oncls.Meta.ordering
-
get_source
(attr_name)[source]¶ Get file location of attribute with name
attr_name
Provide the type, filename, worksheet, row, and column of
attr_name
. Row and column use 1-based counting. Column is provided in XLSX format if the file was a spreadsheet.- Parameters
attr_name (
str
) – attribute name- Returns
type, basename, worksheet, row, columnf
- Return type
tuple
:raises ValueError if the location of
attr_name
is unknown:
-
has_attr_vals
(_Model__type=None, _Model__check_attr_defined=True, **kwargs)[source]¶ Check if the type and values of the attributes of an object match a set of conditions
- Parameters
__type (
types.TypeType
ortuple
oftypes.TypeType
) – subclass(es) ofModel
__check_attr_defined (
bool
, optional) – ifTrue
, raise an exception if the queried attribute is not defined**kwargs – dictionary of attribute name/value pairs to find matching object or create new object
- Returns
True
if the object is an instance of__type
and thethe values of the attributes of the object match
kwargs
- Return type
bool
-
is_equal
(other, tol=0.0)[source]¶ Determine whether two models are semantically equal
- Parameters
other (
Model
) – object to comparetol (
float
, optional) – equality tolerance
- Returns
True
if objects are semantically equal, elseFalse
- Return type
bool
-
classmethod
is_serializable
()[source]¶ Determine if the class (and its related classes) can be serialized
- Raises
bool –
True
if the class can be serialized
-
merge
(other, normalize=True, validate=True)[source]¶ Merge another model into a model
- Parameters
other (
Model
) – other modelnormalize (
bool
, optional) – ifTrue
, normalize models and merged modelvalidate (
bool
, optional) – ifTrue
, validate models and merged model
-
merge_attrs
(other, other_objs_in_self, self_objs_in_other)[source]¶ Merge attributes of two objects
- Parameters
other (
Model
) – other modelother_objs_in_self (
dict
) – dictionary that maps instances of objects in another model to objects in a modelself_objs_in_other (
dict
) – dictionary that maps instances of objects in a model to objects in another model
-
normalize
()[source]¶ Normalize an object into a canonical form. Specifically, this method sorts the RelatedManagers into a canonical order because their order has no semantic meaning. Importantly, this canonical form is reproducible. Thus, this canonical form facilitates reproducible computations on top of
Model
objects.
-
pformat
(max_depth=2, indent=3)[source]¶ Return a human-readable string representation of this
Model
.Follows the graph of related
Model
’s up to a depth ofmax_depth
.Model
’s at depthmax_depth+1
are represented by ‘<class name>: …’, while deeperModel
’s are not traversed or printed. Re-encountered Model’s do not get printed, and are indicated by ‘<attribute name>: –’. Attributes that are related or iterable are indented.For example, we have:
Model1_classname: # Each model starts with its classname, followed by a list of attr1: value1 # attribute names & values. attr2: value2 attr3: # Reference attributes can point to other Models; we indent these under the attribute name Model2_classname: # Reference attribute attr3 contains Model2; ... # its attributes follow. attr4: Model3_classname: # An iteration over reference attributes is a list at constant indentation: ... attr5: Model2_classname: -- # Traversing the Model network may re-encounter a Model; they're listed with '--' attr6: Model5_classname: attr7: Model5_classname: ... # The size of the output is controlled with max_depth; # models encountered at depth = max_depth+1 are shown with '...'
-
serialize
()[source]¶ Get value of primary attribute
- Returns
value of primary attribute
- Return type
str
-
set_nested_attr_val
(attr_path, value)[source]¶ Set the value of an attribute or a nested attribute of a model
- Parameters
attr_path (
list
oflist
ofobject
) – the path to an attribute or nested attribute of a modelvalue (
object
) – new value
- Returns
- the same model with the value of an attribute
modified
- Return type
-
set_source
(path_name, sheet_name, attribute_seq, row, table_id=None)[source]¶ Set metadata about source of the file, worksheet, columns, and row where the object was defined
- Parameters
path_name (
str
) – pathname of source file for objectsheet_name (
str
) – name of spreadsheet containing source data for objectattribute_seq (
list
) – sequence of attribute names in source file; blank values indicate attributes that were ignoredrow (
int
) – row number of object in its source filetable_id (
str
, optional) – id of the source table
-
static
to_dict
(object, models=None, encode_primary_objects=True, encoded=None)[source]¶ Encode a instance of
Model
or a collection of instances ofModel
using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML. Use__id
keys to avoid infinite recursion by encoding each object once and referring to objects by their __id for each repeated reference.- Parameters
object (
object
) – instance ofModel
or a collection (dict
,list
,tuple
, or nested combination ofdict
,list
, andtuple
) of instances ofModel
models (
str
, optional) – list of models to encode into JSONencode_primary_objects (
bool
, optional) – ifTrue
, encode primary classes otherwise just encode their IDsencoded (
dict
, optional) – objects that have already been encoded and their assigned JSON identifiers
- Returns
simple Python representation of the object
- Return type
dict
-
validate
()[source]¶ Determine if the object is valid
- Returns
None
if the object is valid,otherwise return a list of errors as an instance of
InvalidObject
- Return type
InvalidObject
or None
Validate attribute values
- Raises
ValueError – if related attributes are not valid (e.g. if a class that is the subject of a relationship does not have a primary attribute)
-
classmethod
validate_unique
(objects)[source]¶ Validate attribute uniqueness
- Parameters
objects (
list
ofModel
) – list of objects- Returns
list of invalid attributes and their errors
- Return type
InvalidModel
orNone
-
class
obj_tables.core.
ModelMeta
[source]¶ Bases:
type
- Parameters
- Returns
- Return type
-
create_model_manager
()[source]¶ Create a
Manager
for thisModel
The
Manager
is accessed via aModel
’sobjects
attribute- Parameters
cls (
type
) – theModel
class which is being managed
-
init_attribute_order
()[source]¶ Initialize the order in which the attributes should be printed across XLSX columns
-
init_inheritance
()[source]¶ Create tuple of this model and superclasses which are subclasses of
Model
Initialize related attributes
-
normalize_attr_tuples
(attribute)[source]¶ Normalize a tuple of tuples of attribute names
- Parameters
attribute (
str
) – the name of the attribute to validate and normalize
-
static
normalize_tuple_of_tuples_of_attribute_names
(tuple_of_tuples_of_attribute_names)[source]¶ Normalize a tuple of tuples of attribute names by sorting each member tuple
Enables simple indexing and searching of tuples
- Parameters
tuple_of_tuples_of_attribute_names (
tuple
) – a tuple of tuples of attribute names- Returns
a tuple of sorted tuples of attribute names
- Return type
tuple
-
classmethod
validate_attr_tuples
(name, bases, namespace, meta_attribute_name)[source]¶ Validate a tuple of tuples of attribute names
- Parameters
meta_attribute_name (
str
) – the name of the attribute to validate and normalize- Raises
ValueError – if attributes are not valid
-
classmethod
validate_attribute_inheritance
(name, bases, namespace)[source]¶ Check attribute inheritance
- Raises
ValueError – if subclass overrides a superclass attribute (instance of Attribute) with an incompatible attribute (i.e. an attribute that is not a subclass of the class of the super class’ attribute)
-
classmethod
validate_attributes
(name, bases, namespace)[source]¶ Validate attribute values
- Raises
ValueError – if attributes are not valid
-
classmethod
validate_primary_attribute
(name, bases, namespace)[source]¶ Check the attributes
- Raises
ValueError – if there are multiple primary attributes
Check the related attributes
- Raises
ValueError – if an
OneToManyAttribute
orManyToOneAttribute
has arelated_name
equal to itsname
-
class
obj_tables.core.
ModelSource
(path_name, sheet_name, attribute_seq, row, table_id=None)[source]¶ Bases:
object
Represents the file, sheet, columns, and row where a
Model
instance was defined-
attribute_seq
[source]¶ sequence of attribute names in source file; blank values indicate attributes that were ignored
- Type
list
- Parameters
path_name (
str
) – pathname of source file for objectsheet_name (
str
) – name of spreadsheet containing source data for objectattribute_seq (
list
) – sequence of attribute names in source file; blank values indicate attributes that were ignoredrow (
int
) – row number of object in its source filetable_id (
str
, optional) – id of the source table
-
-
class
obj_tables.core.
NumericAttribute
(init_value=None, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False, unique_case_insensitive=False)[source]¶ Bases:
obj_tables.core.LiteralAttribute
Base class for numeric literal attributes (float, integer)
- Parameters
init_value (
object
, optional) – initial valuedefault (
object
, optional) – default valuedefault_cleaned_value (
object
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be uniqueunique_case_insensitive (
bool
, optional) – if true, conduct case-insensitive test of uniqueness
-
class
obj_tables.core.
OneToManyAttribute
(related_class, related_name='', default=[], default_cleaned_value=[], related_default=None, none_value=<class 'list'>, separator=', ', min_related=0, max_related=inf, min_related_rev=0, verbose_name='', verbose_related_name='', description='', related_manager=<class 'obj_tables.core.OneToManyRelatedManager'>, cell_dialect=<CellDialect.json: 'json'>)[source]¶ Bases:
obj_tables.core.ToManyAttribute
,obj_tables.core.RelatedAttribute
Represents a one-to-many relationship between two types of objects. This is analagous to a foreign key relationship in a database.
related manager
- Type
type
- Parameters
related_class (
class
) – related classrelated_name (
str
, optional) – name of related attribute onrelated_class
default (
callable
, optional) – function which returns the default valuedefault_cleaned_value (
callable
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesrelated_default (
callable
, optional) – function which returns the default related valuenone_value (
object
, optional) – none valueseparator (
str
, optional) – element separator for serializationmin_related (
int
, optional) – minimum number of related objects in the forward directionmax_related (
int
, optional) – maximum number of related objects in the forward directionmin_related_rev (
int
, optional) – minimum number of related objects in the reverse directionverbose_name (
str
, optional) – verbose nameverbose_related_name (
str
, optional) – verbose related namedescription (
str
, optional) – descriptionrelated_manager (
type
, optional) – related managercell_dialect (
CellDialect
, optional) – dialect for serializing values to a cell
-
deserialize
(values, objects, decoded=None)[source]¶ Deserialize value
- Parameters
values (
object
) – String representationobjects (
dict
) – dictionary of objects, grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
get_init_value
(obj)[source]¶ Get initial value for attribute
- Parameters
obj (
Model
) – object whose attribute is being initialized- Returns
initial value
- Return type
object
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
merge
(left, right, right_objs_in_left, left_objs_in_right)[source]¶ Merge an attribute of elements of two models
- Parameters
- Raises
ValueError – if the attributes of the elements of the models are different
Determine if
value
is a valid value of the related attribute- Parameters
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
serialize
(value, encoded=None)[source]¶ Serialize related object
- Parameters
value (
list
ofModel
) – Python representationencoded (
dict
, optional) – dictionary of objects that have already been encoded
- Returns
simple Python representation
- Return type
str
Update the values of the related attributes of the attribute
-
set_value
(obj, new_values)[source]¶ Update the values of the related attributes of the attribute
- Parameters
obj (
object
) – object whose attribute should be setnew_values (
list
) – value of the attribute
- Returns
value of the attribute
- Return type
list
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
OneToManyRelatedManager
(object, attribute)[source]¶ Bases:
obj_tables.core.RelatedManager
Represent values of related attributes
-
append
(value, propagate=True)[source]¶ Add value to list
- Parameters
value (
object
) – valuepropagate (
bool
, optional) – propagate change to related attribute
- Returns
self
- Return type
-
cut
(kind=None)[source]¶ Cut values and their children of kind
kind
into separate graphs.If
kind
isNone
, children are defined to be the values of the related attributes defined in each class.- Parameters
kind (
str
, optional) – kind of children to include- Returns
cut values and their children
- Return type
list
ofModel
-
-
class
obj_tables.core.
OneToOneAttribute
(related_class, related_name='', default=None, default_cleaned_value=None, related_default=None, none_value=None, min_related=0, min_related_rev=0, verbose_name='', verbose_related_name='', description='')[source]¶ Bases:
obj_tables.core.RelatedAttribute
Represents a one-to-one relationship between two types of objects.
- Parameters
related_class (
class
) – related classrelated_name (
str
, optional) – name of related attribute onrelated_class
default (
callable
, optional) – callable which returns default valuedefault_cleaned_value (
callable
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesrelated_default (
callable
, optional) – callable which returns default related valuenone_value (
object
, optional) – none valuemin_related (
int
, optional) – minimum number of related objects in the forward directionmin_related_rev (
int
, optional) – minimum number of related objects in the reverse directionverbose_name (
str
, optional) – verbose nameverbose_related_name (
str
, optional) – verbose related namedescription (
str
, optional) – description
-
deserialize
(value, objects, decoded=None)[source]¶ Deserialize value
- Parameters
value (
str
) – String representationobjects (
dict
) – dictionary of objects, grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
deserialize_from_cell
(value, objects, decoded=None)[source]¶ Deserialize value from cell
- Parameters
value (
str
) – String representationobjects (
dict
) – dictionary of objects, grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
merge
(left, right, right_objs_in_left, left_objs_in_right)[source]¶ Merge an attribute of elements of two models
- Parameters
- Raises
ValueError – if the attributes of the elements of the models are different
Determine if
value
is a valid value of the related attribute- Parameters
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
serialize
(value, encoded=None)[source]¶ Serialize related object
- Parameters
value (
Model
) – Python representationencoded (
dict
, optional) – dictionary of objects that have already been encoded
- Returns
simple Python representation
- Return type
str
-
serialize_to_cell
(value, encoded=None)[source]¶ Serialize related object
- Parameters
value (
Model
) – Python representationencoded (
dict
, optional) – dictionary of objects that have already been encoded
- Returns
string representation
- Return type
str
Update the values of the related attributes of the attribute
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
PositiveFloatAttribute
(max=nan, nan=True, default=nan, default_cleaned_value=nan, none_value=nan, verbose_name='', description='', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.FloatAttribute
Positive float attribute
- Parameters
max (
float
, optional) – maximum valuenan (
bool
, optional) – if true, allow nan valuesdefault (
float
, optional) – default valuedefault_cleaned_value (
float
, optional) – value to replaceNone
values with during cleaningverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
PositiveIntegerAttribute
(max=None, none=False, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.IntegerAttribute
Positive integer attribute
- Parameters
min (
int
, optional) – minimum valuemax (
int
, optional) – maximum valuenone (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
int
, optional) – default valuedefault_cleaned_value (
int
, optional) – value to replaceNone
values with during cleaningnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
Range
(min, max)[source]¶ Bases:
object
A numerical range
- Attributes
min (
int
,float
): minimum max (int
,float
): maximum
- Parameters
min (
int
,float
) – minimummax (
int
,float
) – maximum
-
class
obj_tables.core.
RangeAttribute
(type=<class 'float'>, separator='-', separator_pattern='(?<!e) *- *', none=True, default=None, none_value=None, verbose_name='', description='A range of values')[source]¶ Bases:
obj_tables.core.LiteralAttribute
Attribute for a range of values (x-y)
- Parameters
type (
type
, optional) – type of elementsseparator (
str
, optional) – element separator for serializationseparator_pattern (
str
, optional) – element separator for deserializationnone (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
Range
, optional) – default valuenone_value (
Range
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – description
-
clean
(value)[source]¶ Deserialize value
- Parameters
value (
str
) – semantically equivalent representation- Returns
Range
: cleaned valueInvalidAttribute
: cleaning error
- Return type
tuple
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
dict
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
-
serialize
(value)[source]¶ Serialize string
- Parameters
value (
Range
) – Python representation- Returns
simple Python representation
- Return type
str
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
Range
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
dict
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value- Parameters
- Returns
- None if attribute is valid, other return
list of errors as an instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
RegexAttribute
(pattern, flags=0, min_length=0, max_length=None, none=False, default='', default_cleaned_value='', none_value='', verbose_name='', description='', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.StringAttribute
Regular expression attribute
- Parameters
pattern (
str
) – regular expression patternflags (
int
, optional) – regular expression flagsmin_length (
int
, optional) – minimum lengthmax_length (
int
, optional) – maximum lengthnone (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
str
, optional) – default valuedefault_cleaned_value (
str
, optional) – value to replaceNone
values with during cleaningverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
RelatedAttribute
(related_class, related_name='', init_value=None, default=None, default_cleaned_value=None, none_value=None, related_init_value=None, related_default=None, min_related=0, max_related=inf, min_related_rev=0, max_related_rev=inf, verbose_name='', verbose_related_name='', description='')[source]¶ Bases:
obj_tables.core.BaseRelatedAttribute
,obj_tables.core.Attribute
Attribute which represents a relationship with another
Model
allowed type(s) of the related values of the attribute
- Type
types.TypeType
ortuple
oftypes.TypeType
the type of the class that contains a related attribute
- Type
class
name of related attribute on
related_class
- Type
str
verbose related name
- Type
str
initial value of related attribute
- Type
object
default value of related attribute
- Type
object
minimum number of related objects in the forward direction
- Type
int
maximum number of related objects in the forward direction
- Type
int
minimum number of related objects in the reverse direction
- Type
int
maximum number of related objects in the reverse direction
- Type
int
- Parameters
related_class (
class
) – related classrelated_name (
str
, optional) – name of related attribute onrelated_class
init_value (
object
, optional) – initial valuedefault (
object
, optional) – default valuedefault_cleaned_value (
object
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesnone_value (
object
, optional) – none valuerelated_init_value (
object
, optional) – related initial valuerelated_default (
object
, optional) – related default valuemin_related (
int
, optional) – minimum number of related objects in the forward directionmax_related (
int
, optional) – maximum number of related objects in the forward directionmin_related_rev (
int
, optional) – minimum number of related objects in the reverse directionmax_related_rev (
int
, optional) – maximum number of related objects in the reverse directionverbose_name (
str
, optional) – verbose nameverbose_related_name (
str
, optional) – verbose related namedescription (
str
, optional) – description
- Raises
ValueError – If default or related_default is not None, an empty list, or a callable or default and related_default are both non-empty lists or callables
-
deserialize
(value, objects, decoded=None)[source]¶ Deserialize value
- Parameters
values (
object
) – String representationobjects (
dict
) – dictionary of objects, grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofobject
,InvalidAttribute
orNone
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
object
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
object
Get default related value for attribute
- Parameters
obj (
Model
) – object whose attribute is being initialized- Returns
initial value
- Return type
object
- Raises
ValueError – if related property is not defined
Get initial related value for attribute
- Parameters
obj (
object
) – object whose attribute is being initialized- Returns
initial value
- Return type
value (
object
)- Raises
ValueError – if related property is not defined
Determine if
value
is a valid value of the related attribute- Parameters
obj (
Model
) – object to validatevalue (
list
) – value to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
serialize
(value, encoded=None)[source]¶ Serialize related object
- Parameters
value (
Model
) – Python representationencoded (
dict
, optional) – dictionary of objects that have already been encoded
- Returns
simple Python representation
- Return type
str
Update the values of the related attributes of the attribute
- Parameters
obj (
object
) – object whose attribute should be setnew_values (
object
) – value of the attribute
- Returns
value of the attribute
- Return type
object
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
object
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
object
-
class
obj_tables.core.
RelatedManager
(object, attribute, related=True)[source]¶ Bases:
list
Represent values and related values of related attributes
is related attribute
- Type
bool
- Parameters
-
add
(value, **kwargs)[source]¶ Add value to list
- Parameters
value (
object
) – value- Returns
self
- Return type
-
append
(value, **kwargs)[source]¶ Add value to list
- Parameters
value (
object
) – value- Returns
self
- Return type
-
create
(_RelatedManager__type=None, **kwargs)[source]¶ Create instance of primary class and add to list
-
difference_update
(values)[source]¶ Retain only values of list not in
values
- Parameters
values (
list
) – values to difference with list- Returns
self
- Return type
-
discard
(value)[source]¶ Remove value from list if value in list
- Parameters
value (
object
) – value- Returns
self
- Return type
-
extend
(values)[source]¶ Add values to list
- Parameters
values (
list
) – values to add to list- Returns
self
- Return type
-
get
(_RelatedManager__type=None, **kwargs)[source]¶ Get related objects by attribute/value pairs and, optionally, only return matches that are also instances of
Model
subclass__type
.
-
get_one
(_RelatedManager__type=None, **kwargs)[source]¶ Get a related object by attribute/value pairs; report an error if multiple objects match and, optionally, only return matches that are also instances of
Model
subclass__type
.- Parameters
__type (
types.TypeType
ortuple
oftypes.TypeType
) – subclass(es) ofModel
**kwargs – dictionary of attribute name/value pairs to find matching objects
- Returns
matching instance of
Model
, orNone
if no matching instance- Return type
Model
orNone
- Raises
ValueError – if multiple matching objects
-
get_or_create
(_RelatedManager__type=None, **kwargs)[source]¶ Get or create a related object by attribute/value pairs. Optionally, only get or create instances of
Model
subclass__type
.
-
index
(*args, **kwargs)[source]¶ Get related object index by attribute/value pairs
- Parameters
*args (
Model
) – object to find**kwargs – dictionary of attribute name/value pairs to find matching objects
- Returns
index of matching object
- Return type
int
- Raises
ValueError – if no argument or keyword argument is provided, if argument and keyword arguments are both provided, if multiple arguments are provided, if the keyword attribute/value pairs match no object, or if the keyword attribute/value pairs match multiple objects
-
intersection_update
(values)[source]¶ Retain only intersection of list and
values
- Parameters
values (
list
) – values to intersect with list- Returns
self
- Return type
-
pop
(i=-1)[source]¶ Remove an arbitrary element from the list
- Parameters
i (
int
, optional) – index of element to remove- Returns
removed element
- Return type
-
symmetric_difference_update
(values)[source]¶ Retain values in only one of list and
values
- Parameters
values (
list
) – values to difference with list- Returns
self
- Return type
-
exception
obj_tables.core.
SchemaWarning
[source]¶ Bases:
obj_tables.core.ObjTablesWarning
Schema warning
-
class
obj_tables.core.
SlugAttribute
(verbose_name='', description=None, primary=True, unique=True)[source]¶ Bases:
obj_tables.core.RegexAttribute
Slug attribute to be used for string IDs
- Parameters
verbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate whether attribute must be unique
-
class
obj_tables.core.
StringAttribute
(min_length=0, max_length=255, none=False, default='', default_cleaned_value='', none_value='', verbose_name='', description='', primary=False, unique=False, unique_case_insensitive=False)[source]¶ Bases:
obj_tables.core.LiteralAttribute
String attribute
- Parameters
min_length (
int
, optional) – minimum lengthmax_length (
int
, optional) – maximum lengthnone (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
str
, optional) – default valuedefault_cleaned_value (
str
, optional) – value to replaceNone
values with during cleaningverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be uniqueunique_case_insensitive (
bool
, optional) – if true, conduct case-insensitive test of uniqueness
- Raises
ValueError – if
min_length
is negative,max_length
is less thanmin_length
,default
is not a string, ordefault_cleaned_value
is not a string
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofstr
,InvalidAttribute
orNone
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
serialize
(value)[source]¶ Serialize string
- Parameters
value (
str
) – Python representation- Returns
simple Python representation
- Return type
str
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value for this StringAttribute- Parameters
obj (
Model
) – class being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
TableFormat
[source]¶ Bases:
enum.Enum
Describes a table’s orientation
row
: the first row contains attribute names; subsequents rows store objectscolumn
: the first column contains attribute names; subsequents columns store objectscell
: a cell contains a table, as a comma-separated list for examplemultiple_cells
: multiple cells within a row or column
-
class
obj_tables.core.
TimeAttribute
(none=True, default=None, default_cleaned_value=None, none_value=None, verbose_name='', description='', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.LiteralAttribute
Time attribute
-
default_cleaned_value
[source]¶ value to replace
None
values with during cleaning, or function which computes the value to replaceNone
values- Type
time
- Parameters
none (
bool
, optional) – ifFalse
, the attribute is invalid if its value isNone
default (
time
, optional) – default timedefault_cleaned_value (
time
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesnone_value (
object
, optional) – none valueverbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
clean
(value)[source]¶ Convert attribute value into the appropriate type
- Parameters
value (
object
) – value of attribute to clean- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
oftime
,InvalidAttribute
orNone
-
from_builtin
(json)[source]¶ Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML
- Parameters
json (
str
) – simple Python representation of a value of the attribute- Returns
decoded value of the attribute
- Return type
time
-
get_xlsx_validation
(sheet_models=None, doc_metadata_model=None)[source]¶ Get XLSX validation
- Parameters
sheet_models (
list
ofModel
, optional) – models encoded as separate sheetsdoc_metadata_model (
type
) – model whose worksheet contains the document metadata
- Returns
validation
- Return type
wc_utils.workbook.io.FieldValidation
-
serialize
(value)[source]¶ Serialize string
- Parameters
value (
time
) – Python representation- Returns
simple Python representation
- Return type
str
-
to_builtin
(value)[source]¶ Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML
- Parameters
value (
time
) – value of the attribute- Returns
simple Python representation of a value of the attribute
- Return type
str
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
time
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
-
class
obj_tables.core.
ToManyAttribute
(related_class, related_name='', init_value=None, default=None, default_cleaned_value=None, none_value=None, related_init_value=None, related_default=None, min_related=0, max_related=inf, min_related_rev=0, max_related_rev=inf, verbose_name='', verbose_related_name='', description='')[source]¶ Bases:
obj_tables.core.RelatedAttribute
*-to-many attribute
- Parameters
related_class (
class
) – related classrelated_name (
str
, optional) – name of related attribute onrelated_class
init_value (
object
, optional) – initial valuedefault (
object
, optional) – default valuedefault_cleaned_value (
object
, optional) – value to replaceNone
values with during cleaning, or function which computes the value to replaceNone
valuesnone_value (
object
, optional) – none valuerelated_init_value (
object
, optional) – related initial valuerelated_default (
object
, optional) – related default valuemin_related (
int
, optional) – minimum number of related objects in the forward directionmax_related (
int
, optional) – maximum number of related objects in the forward directionmin_related_rev (
int
, optional) – minimum number of related objects in the reverse directionmax_related_rev (
int
, optional) – maximum number of related objects in the reverse directionverbose_name (
str
, optional) – verbose nameverbose_related_name (
str
, optional) – verbose related namedescription (
str
, optional) – description
- Raises
ValueError – If default or related_default is not None, an empty list, or a callable or default and related_default are both non-empty lists or callables
-
class
obj_tables.core.
UrlAttribute
(verbose_name='', description='Enter a valid URL', primary=False, unique=False)[source]¶ Bases:
obj_tables.core.RegexAttribute
URL attribute to be used for URLs
- Parameters
verbose_name (
str
, optional) – verbose namedescription (
str
, optional) – descriptionprimary (
bool
, optional) – indicate if attribute is primary attributeunique (
bool
, optional) – indicate if attribute value must be unique
-
validate
(obj, value)[source]¶ Determine if
value
is a valid value of the attribute- Parameters
obj (
Model
) – object being validatedvalue (
object
) – value of attribute to validate
- Returns
- None if attribute is valid, other return list of errors as an
instance of
InvalidAttribute
- Return type
InvalidAttribute
or None
-
class
obj_tables.core.
Validator
[source]¶ Bases:
object
Engine to validate sets of objects
-
clean
(objects)[source]¶ Clean a list of objects and return their errors
- Parameters
object (
list
ofModel
) – list of objects- Returns
list of invalid objects/models and their errors
- Return type
InvalidObjectSet
orNone
-
run
(objects, get_related=False)[source]¶ Validate a list of objects and return their errors
- Parameters
- Returns
list of invalid objects/models and their errors
- Return type
InvalidObjectSet
orNone
-
validate
(objects)[source]¶ Validate a list of objects and return their errors
- Parameters
object (
list
ofModel
) – list of Model instances- Returns
list of invalid objects/models and their errors
- Return type
InvalidObjectSet
orNone
-
-
obj_tables.core.
get_model
(name, module=None)[source]¶ Get first
Model
with namename
- Parameters
name (
str
) – namemodule (
Module
, optional) – module
- Returns
model class
- Return type
class
-
obj_tables.core.
get_models
(module=None, inline=True)[source]¶ Get models
- Parameters
module (
module
, optional) – moduleinline (
bool
, optional) – if true, return inline models
- Returns
list of model classes
- Return type
list
ofclass
-
obj_tables.core.
join_separated_list
(values, separator=', ')[source]¶ Parse a separator list of values into a list of values
- Parameters
values (
list
ofstr
) – valuesseparator (
str
, optional) – separator
- Returns
seperator-separated list of values
- Return type
str
-
obj_tables.core.
split_separated_list
(joined_values, separator=', ')[source]¶ Parse a separator list of values into a list of values
- Parameters
joined_values (
str
) – seperator-separated list of valuesseparator (
str
, optional) – separator
- Returns
values
- Return type
list
ofstr
-
obj_tables.core.
xlsx_col_name
(col)[source]¶ Convert column number to an XLSX-style string.
From http://stackoverflow.com/a/19169180/509882
- Parameters
col (
int
) – column number (positive integer)- Returns
alphabetic column name
- Return type
str
- Raises
ValueError – if
col
is not positive
3.1.7. obj_tables.grammar module¶
Attributes for embedding domain-specific langauges for describing *-to-many relationships into XLSX cell
- Author
Jonathan Karr <karr@mssm.edu>
- Date
2019-09-23
- Copyright
2019, Karr Lab
- License
MIT
-
class
obj_tables.grammar.
ToManyGrammarAttribute
(related_class, grammar=None, **kwargs)[source]¶ Bases:
obj_tables.core.RelatedAttribute
*-to-many attribute that can be deserialized with a grammar
Class attributes:
grammar (
str
): grammargrammar_path (
str
): path to grammarTransformer (
type
): subclass ofTransformer
which transforms parse trees into a list of instances ofcore.Model
- Parameters
related_class (
type
) – related classgrammar (
str
, optional) – grammar
-
deserialize
(values, objects, decoded=None)[source]¶ Deserialize value
- Parameters
values (
object
) – String representation of related objectsobjects (
dict
) – dictionary of objects, grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
tuple of cleaned value and cleaning error
- Return type
tuple
ofobject
,core.InvalidAttribute
orNone
-
classmethod
gen_transformer
(model)[source]¶ Generate transformer for model
- Parameters
model (
type
) – model- Returns
transformer
- Return type
type
-
grammar
= None[source]
-
class
obj_tables.grammar.
ToManyGrammarTransformer
(objects)[source]¶ Bases:
lark.visitors.Transformer
Transforms parse trees into a list of instances of
core.Model
-
objects
[source]¶ dictionary that maps types of models to dictionaries which map serialized values of instances of models to instances
- Type
dict
- Parameters
objects (
dict
) – dictionary that maps types of models to dictionaries which map serialized values of instances of models to instances
-
get_or_create_model_obj
(model, _serialized_val=None, _clean=True, **kwargs)[source]¶ Get a instance of a model with serialized value
_serialized_val
, or create an instance if there is no such instance- Parameters
model (
type
) – type of model instance to get or create_serialized_val (
str
, optional) – serialized value of instance of model_clean (
bool
, optional) – ifTrue
, clean valueskwargs (
dict
, optional) – arguments to constructor of model for instance
-
3.1.8. obj_tables.io module¶
Reading/writing schema objects to/from files
Comma separated values (.csv)
XLSX (.xlsx)
JavaScript Object Notation (.json)
Tab separated values (.tsv)
Yet Another Markup Language (.yaml, .yml)
- Author
Jonathan Karr <karr@mssm.edu>
- Author
Arthur Goldberg <Arthur.Goldberg@mssm.edu>
- Date
2019-09-19
- Copyright
2016-2019, Karr Lab
- License
MIT
-
exception
obj_tables.io.
IoWarning
[source]¶ Bases:
obj_tables.core.ObjTablesWarning
IO warning
-
class
obj_tables.io.
JsonReader
[source]¶ Bases:
obj_tables.io.ReaderBase
Read model objects from a JSON or YAML file
-
run
(path, schema_name=None, models=None, allow_multiple_sheets_per_model=False, ignore_missing_models=False, ignore_extra_models=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, ignore_empty_rows=True, group_objects_by_model=True, validate=True)[source]¶ Read model objects from file(s) and, optionally, validate them
- Parameters
path (
str
) – path to file(s)schema_name (
str
, optional) – schema namemodels (
types.TypeType
orlist
oftypes.TypeType
, optional) – type or list of type of objects to readallow_multiple_sheets_per_model (
bool
, optional) – ifTrue
, allow multiple sheets per modelignore_missing_models (
bool
, optional) – ifFalse
, report an error if a worksheet/ file is missing for one or more modelsignore_extra_models (
bool
, optional) – ifTrue
and allmodels
are found, ignore other worksheets or filesignore_sheet_order (
bool
, optional) – ifTrue
, do not require the sheets to be provided in the canonical orderinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
ignore_missing_attributes (
bool
, optional) – ifFalse
, report an error if a worksheet/file doesn’t contain all of attributes in a model inmodels
ignore_extra_attributes (
bool
, optional) – ifTrue
, do not report errors if attributes in the data are not in the modelignore_attribute_order (
bool
, optional) – ifTrue
, do not require the attributes to be provided in the canonical orderignore_empty_rows (
bool
, optional) – ifTrue
, ignore empty rowsgroup_objects_by_model (
bool
, optional) – ifTrue
, group decoded objects by their typesvalidate (
bool
, optional) – ifTrue
, validate the data
- Returns
model objects grouped by
Model
class- Return type
dict
- Raises
ValueError – if the input format is not supported, model names are not unique, or the data is invalid
-
-
class
obj_tables.io.
JsonWriter
[source]¶ Bases:
obj_tables.io.WriterBase
Write model objects to a JSON or YAML file
-
run
(path, objects, schema_name=None, doc_metadata=None, model_metadata=None, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, write_toc=False, write_schema=False, write_empty_models=True, write_empty_cols=True, extra_entries=0, group_objects_by_model=True, data_repo_metadata=False, schema_package=None, protected=False)[source]¶ Write a list of model classes to a JSON or YAML file
- Parameters
path (
str
) – path to write file(s)objects (
Model
orlist
ofModel
) – object or list of objectsschema_name (
str
, optional) – schema namedoc_metadata (
dict
, optional) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)model_metadata (
dict
, optional) – dictionary that maps models to dictionary with their metadata to be saved to header row (e.g.,!!ObjTables ...
)models (
list
ofModel
, optional) – modelsget_related (
bool
, optional) – ifTrue
, write object and all related objectsinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
validate (
bool
, optional) – ifTrue
, validate the datatitle (
str
, optional) – titledescription (
str
, optional) – descriptionkeywords (
str
, optional) – keywordsversion (
str
, optional) – versionlanguage (
str
, optional) – languagecreator (
str
, optional) – creatorwrite_toc (
bool
, optional) – ifTrue
, include additional worksheet with table of contentswrite_schema (
bool
, optional) – ifTrue
, include additional worksheet with schemawrite_empty_models (
bool
, optional) – ifTrue
, write models even when there are no instanceswrite_empty_cols (
bool
, optional) – ifTrue
, write columns even when all values areNone
extra_entries (
int
, optional) – additional entries to displaygroup_objects_by_model (
bool
, optional) – ifTrue
, group objects by modeldata_repo_metadata (
bool
, optional) – ifTrue
, try to write metadata information about the file’s Git repo; the repo must be current with origin, except for the fileschema_package (
str
, optional) – the package which defines the ObjTables schema used by the file; if notNone
, try to write metadata information about the the schema’s Git repository: the repo must be current with originprotected (
bool
, optional) – ifTrue
, protect the worksheet
- Raises
ValueError – if model names are not unique or output format is not supported
-
-
class
obj_tables.io.
MultiSeparatedValuesReader
[source]¶ Bases:
obj_tables.io.ReaderBase
Read a list of model objects from a single text file which contains multiple comma or tab-separated files
-
run
(path, schema_name=None, models=None, allow_multiple_sheets_per_model=False, ignore_missing_models=False, ignore_extra_models=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, ignore_empty_rows=True, group_objects_by_model=True, validate=True)[source]¶ Read a list of model objects from a single text file which contains multiple comma or tab-separated files
- Parameters
path (
str
) – path to file(s)schema_name (
str
, optional) – schema namemodels (
types.TypeType
orlist
oftypes.TypeType
, optional) – type or list of type of objects to readallow_multiple_sheets_per_model (
bool
, optional) – ifTrue
, allow multiple sheets per modelignore_missing_models (
bool
, optional) – ifFalse
, report an error if a worksheet/ file is missing for one or more modelsignore_extra_models (
bool
, optional) – ifTrue
and allmodels
are found, ignore other worksheets or filesignore_sheet_order (
bool
, optional) – ifTrue
, do not require the sheets to be provided in the canonical orderinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
ignore_missing_attributes (
bool
, optional) – ifFalse
, report an error if a worksheet/file doesn’t contain all of attributes in a model inmodels
ignore_extra_attributes (
bool
, optional) – ifTrue
, do not report errors if attributes in the data are not in the modelignore_attribute_order (
bool
, optional) – ifTrue
, do not require the attributes to be provided in the canonical orderignore_empty_rows (
bool
, optional) – ifTrue
, ignore empty rowsgroup_objects_by_model (
bool
, optional) – ifTrue
, group decoded objects by their typesvalidate (
bool
, optional) – ifTrue
, validate the data
- Returns
- if
group_objects_by_model
set returnsdict
: of model objects grouped byModel
class; else returns
list
: of all model objects
- if
- Return type
obj
- Raises
ValueError – if
path
contains a glob pattern
-
-
class
obj_tables.io.
MultiSeparatedValuesWriter
[source]¶ Bases:
obj_tables.io.WriterBase
Write model objects to a single text file which contains multiple comma or tab-separated tables.
-
run
(path, objects, schema_name=None, doc_metadata=None, model_metadata=None, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, write_toc=True, write_schema=False, write_empty_models=True, write_empty_cols=True, extra_entries=0, group_objects_by_model=True, data_repo_metadata=False, schema_package=None, protected=False)[source]¶ Write model objects to a single text file which contains multiple comma or tab-separated tables.
- Parameters
path (
str
) – path to write file(s)objects (
Model
orlist
ofModel
) –Model
instance or list ofModel
instancesschema_name (
str
, optional) – schema namedoc_metadata (
dict
, optional) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)model_metadata (
dict
, optional) – dictionary that maps models to dictionary with their metadata to be saved to header row (e.g.,!!ObjTables ...
)models (
list
ofModel
, optional) – models in the order that they should appear as worksheets; all models which are not inmodels
will follow in alphabetical orderget_related (
bool
, optional) – ifTrue
, writeobjects
and all their related objectsinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
validate (
bool
, optional) – ifTrue
, validate the datatitle (
str
, optional) – titledescription (
str
, optional) – descriptionkeywords (
str
, optional) – keywordsversion (
str
, optional) – versionlanguage (
str
, optional) – languagecreator (
str
, optional) – creatorwrite_schema (
bool
, optional) – ifTrue
, include additional worksheet with schemawrite_empty_models (
bool
, optional) – ifTrue
, write models even when there are no instanceswrite_empty_cols (
bool
, optional) – ifTrue
, write columns even when all values areNone
write_toc (
bool
, optional) – ifTrue
, include additional worksheet with table of contentsextra_entries (
int
, optional) – additional entries to displaygroup_objects_by_model (
bool
, optional) – ifTrue
, group objects by modeldata_repo_metadata (
bool
, optional) – ifTrue
, try to write metadata information about the file’s Git repo; the repo must be current with origin, except for the fileschema_package (
str
, optional) – the package which defines the ObjTables schema used by the file; if notNone
, try to write metadata information about the the schema’s Git repository: the repo must be current with originprotected (
bool
, optional) – ifTrue
, protect the worksheet
- Raises
ValueError – if no model is provided or a class cannot be serialized
-
-
class
obj_tables.io.
PandasWriter
[source]¶ Bases:
obj_tables.io.WorkbookWriter
Write model instances to a dictionary of
pandas.DataFrame
-
_data_frames
[source]¶ dictionary that maps models (
Model
) to their instances (pandas.DataFrame
)- Type
dict
-
run
(objects, schema_name=None, models=None, get_related=True, include_all_attributes=True, validate=True, protected=False)[source]¶ Write model instances to a dictionary of
pandas.DataFrame
- Parameters
objects (
Model
orlist
ofModel
) – object or list of objectsschema_name (
str
, optional) – schema namemodels (
list
ofModel
, optional) – models in the order that they should appear as worksheets; all models which are not inmodels
will follow in alphabetical orderget_related (
bool
, optional) – ifTrue
, writeobjects
and all their related objectsinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
validate (
bool
, optional) – ifTrue
, validate the dataprotected (
bool
, optional) – ifTrue
, protect the worksheet
- Returns
- dictionary that maps models (
Model
) to their instances (
pandas.DataFrame
)
- dictionary that maps models (
- Return type
dict
-
write_sheet
(writer, model, data, headings, metadata_headings, validation, extra_entries=0, merge_ranges=None, protected=False)[source]¶ Write data to sheet
- Parameters
writer (
wc_utils.workbook.io.Writer
) – io writermodel (
type
) – modeldata (
list
oflist
ofobject
) – list of list of cell valuesheadings (
list
oflist
ofstr
) – list of list of row headingsvalidationsmetadata_headings (
list
oflist
ofstr
) – model metadata (name, description) to print at the top of the worksheetvalidation (
WorksheetValidation
) – validationextra_entries (
int
, optional) – additional entries to displaymerge_ranges (
list
oftuple
) – list of ranges of cells to mergeprotected (
bool
, optional) – ifTrue
, protect the worksheet
-
-
class
obj_tables.io.
Reader
[source]¶ Bases:
obj_tables.io.ReaderBase
-
static
get_reader
(path)[source]¶ Get the IO class whose
run
method can read the file(s) atpath
- Parameters
path (
str
) – path to write file(s)- Returns
reader class
- Return type
type
- Raises
ValueError – if extension is not supported
-
run
(path, schema_name=None, models=None, allow_multiple_sheets_per_model=False, ignore_missing_models=False, ignore_extra_models=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, ignore_empty_rows=True, group_objects_by_model=True, validate=True)[source]¶ Read a list of model objects from file(s) and, optionally, validate them
- Parameters
path (
str
) – path to file(s)schema_name (
str
, optional) – schema namemodels (
types.TypeType
orlist
oftypes.TypeType
, optional) – type of object to read or list of types of objects to readallow_multiple_sheets_per_model (
bool
, optional) – ifTrue
, allow multiple sheets per modelignore_missing_models (
bool
, optional) – ifFalse
, report an error if a worksheet/ file is missing for one or more modelsignore_extra_models (
bool
, optional) – ifTrue
and allmodels
are found, ignore other worksheets or filesignore_sheet_order (
bool
, optional) – ifTrue
, do not require the sheets to be provided in the canonical orderinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
ignore_missing_attributes (
bool
, optional) – ifFalse
, report an error if a worksheet/file doesn’t contain all of attributes in a model inmodels
ignore_extra_attributes (
bool
, optional) – ifTrue
, do not report errors if attributes in the data are not in the modelignore_attribute_order (
bool
, optional) – ifTrue
, do not require the attributes to be provided in the canonical orderignore_empty_rows (
bool
, optional) – ifTrue
, ignore empty rowsgroup_objects_by_model (
bool
, optional) – ifTrue
, group decoded objects by their typesvalidate (
bool
, optional) – ifTrue
, validate the data
- Returns
- if
group_objects_by_model
is set returnsdict
: model objects grouped by
Model
class, otherwise returnslist
: of model objects
- if
- Return type
obj
-
static
-
class
obj_tables.io.
ReaderBase
[source]¶ Bases:
object
Interface for classes which write model objects to file(s)
-
_doc_metadata
[source]¶ dictionary of document metadata read from header row (e.g.,
!!!ObjTables ...
)- Type
dict
-
_model_metadata
[source]¶ dictionary which maps models (
Model
) to dictionaries of metadata read from a document (e.g., !!ObjTables date=’…’ …)- Type
dict
-
MODELS
[source]¶ default types of models to export and the order in which to export them
- Type
tuple
oftype
-
MODELS
= ()[source]
-
abstract
run
(path, schema_name=None, models=None, allow_multiple_sheets_per_model=False, ignore_missing_models=False, ignore_extra_models=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, ignore_empty_rows=True, group_objects_by_model=True, validate=True)[source]¶ Read a list of model objects from file(s) and, optionally, validate them
- Parameters
path (
str
) – path to file(s)schema_name (
str
, optional) – schema namemodels (
types.TypeType
orlist
oftypes.TypeType
, optional) – type of object to read or list of types of objects to readallow_multiple_sheets_per_model (
bool
, optional) – ifTrue
, allow multiple sheets per modelignore_missing_models (
bool
, optional) – ifFalse
, report an error if a worksheet/ file is missing for one or more modelsignore_extra_models (
bool
, optional) – ifTrue
and allmodels
are found, ignore other worksheets or filesignore_sheet_order (
bool
, optional) – ifTrue
, do not require the sheets to be provided in the canonical orderinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
ignore_missing_attributes (
bool
, optional) – ifFalse
, report an error if a worksheet/file doesn’t contain all of attributes in a model inmodels
ignore_extra_attributes (
bool
, optional) – ifTrue
, do not report errors if attributes in the data are not in the modelignore_attribute_order (
bool
, optional) – ifTrue
, do not require the attributes to be provided in the canonical orderignore_empty_rows (
bool
, optional) – ifTrue
, ignore empty rowsgroup_objects_by_model (
bool
, optional) – ifTrue
, group decoded objects by their typesvalidate (
bool
, optional) – ifTrue
, validate the data
- Returns
model objects grouped by
Model
class- Return type
dict
-
-
class
obj_tables.io.
WorkbookReader
[source]¶ Bases:
obj_tables.io.ReaderBase
Read model objects from an XLSX file or CSV and TSV files
-
DOC_METADATA_PATTERN
= '^!!!ObjTables( +(.*?)=(\'((?:[^\'\\\\]|\\\\.)*)\'|\\"((?:[^\\"\\\\]|\\\\.)*)\\"))* *$'[source]¶
-
MODEL_METADATA_PATTERN
= '^!!ObjTables( +(.*?)=(\'((?:[^\'\\\\]|\\\\.)*)\'|\\"((?:[^\\"\\\\]|\\\\.)*)\\"))* *$'[source]¶
-
classmethod
get_model_sheet_name
(sheet_names, model)[source]¶ Get the name of the worksheet/file which corresponds to a model
- Parameters
sheet_names (
list
ofstr
) – names of the sheets in the workbook/filesmodel (
Model
) – model
- Returns
name of sheet corresponding to the model or
None
if there is no sheet for the model- Return type
str
- Raises
ValueError – if the model matches more than one sheet
-
classmethod
get_possible_model_sheet_names
(model)[source]¶ Return set of possible sheet names for a model
- Parameters
model (
Model
) – Model- Returns
set of possible sheet names for a model
- Return type
set
-
classmethod
header_row_col_names
(index, file_ext, table_format)[source]¶ Determine row and column names for header entries.
- Parameters
index (
int
) – index in header sequencefile_ext (
str
) – extension for model filetable_format (
TableFormat
) – orientation of the stored table
- Returns
tuple
of row, column, header_entries
-
link_model
(model, attributes, data, objects, objects_by_primary_attribute, decoded=None)[source]¶ Construct object graph
- Parameters
model (
Model
) – anobj_tables.core.Model
attributes (
list
ofAttribute
) – attribute order ofdata
data (
list
oflist
ofobject
) – nested list of object dataobjects (
list
) – list of model objects in order ofdata
objects_by_primary_attribute (
dict
) – dictionary of model objects grouped by modeldecoded (
dict
, optional) – dictionary of objects that have already been decoded
- Returns
list of parsing errors
- Return type
list
ofstr
-
merge_doc_metadata
(metadata)[source]¶ Merge metadata into document metadata
- Parameters
metadata (
dict
) – meta data- Raises
ValueError – if the meta data conflicts with existing document metadata
-
classmethod
parse_worksheet_heading_metadata
(heading, sheet_name=None)[source]¶ Parse key-value pairs of metadata from heading
- Parameters
heading (
str
) – heading with key-value pairs of metadatasheet_name (
str
, optional) – sheet name
- Returns
dictionary of document metadata
- Return type
dict
- Raises
ValueError – if a key is repeated
-
read_model
(reader, sheet_name, schema_name, model, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, ignore_empty_rows=True, validate=True)[source]¶ Instantiate a list of objects from data in a table in a file
- Parameters
reader (
wc_utils.workbook.io.Reader
) – readersheet_name (
str
) – sheet nameschema_name (
str
) – schema namemodel (
type
) – the model describing the objects’ schemainclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
ignore_missing_attributes (
bool
, optional) – ifFalse
, report an error if the worksheet/files don’t have all of attributes in the modelignore_extra_attributes (
bool
, optional) – ifTrue
, do not report errors if attributes in the data are not in the modelignore_attribute_order (
bool
, optional) – ifTrue
, do not require the attributes to be provided in the canonical orderignore_empty_rows (
bool
, optional) – ifTrue
, ignore empty rowsvalidate (
bool
, optional) – ifTrue
, validate the data
- Returns
list
ofAttribute
: attribute order ofdata
list
oflist
ofobject
: a two-dimensional nested list of object datalist
ofstr
: a list of parsing errorslist
ofModel
: constructed model objects
- Return type
tuple
-
read_sheet
(model, reader, sheet_name, num_row_heading_columns=0, num_column_heading_rows=0, ignore_empty_rows=False, ignore_empty_cols=False)[source]¶ Read worksheet or file into a two-dimensional list
- Parameters
model (
type
) – the model describing the objects’ schemareader (
wc_utils.workbook.io.Reader
) – readersheet_name (
str
) – worksheet namenum_row_heading_columns (
int
, optional) – number of columns of row headingsnum_column_heading_rows (
int
, optional) – number of rows of column headingsignore_empty_rows (
bool
, optional) – ifTrue
, ignore empty rowsignore_empty_cols (
bool
, optional) – ifTrue
, ignore empty columns
- Returns
list
oflist
: two-dimensional list of table valueslist
oflist
: row headingslist
oflist
: column_headingslist
ofstr
: comments above column headings
- Return type
tuple
- Raises
ValueError – if worksheet doesn’t have header rows or columns
-
classmethod
read_worksheet_metadata
(sheet_name, rows)[source]¶ Read worksheet metadata
- Parameters
sheet_name (
str
) – sheet namerows (
list
) – rows
- Returns
dict
: dictionary of document propertiesdict
: dictionary of model propertieslist
ofstr
: comments
- Return type
tuple
-
run
(path, schema_name=None, models=None, allow_multiple_sheets_per_model=False, ignore_missing_models=False, ignore_extra_models=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, ignore_empty_rows=True, group_objects_by_model=True, validate=True)[source]¶ Read a list of model objects from file(s) and, optionally, validate them
File(s) may be a single XLSX workbook with multiple worksheets or a set of delimeter separated files encoded by a single path with a glob pattern.
- Parameters
path (
str
) – path to file(s)schema_name (
str
, optional) – schema namemodels (
types.TypeType
orlist
oftypes.TypeType
, optional) – type or list of type of objects to readallow_multiple_sheets_per_model (
bool
, optional) – ifTrue
, allow multiple sheets per modelignore_missing_models (
bool
, optional) – ifFalse
, report an error if a worksheet/ file is missing for one or more modelsignore_extra_models (
bool
, optional) – ifTrue
and allmodels
are found, ignore other worksheets or filesignore_sheet_order (
bool
, optional) – ifTrue
, do not require the sheets to be provided in the canonical orderinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
ignore_missing_attributes (
bool
, optional) – ifFalse
, report an error if a worksheet/file doesn’t contain all of attributes in a model inmodels
ignore_extra_attributes (
bool
, optional) – ifTrue
, do not report errors if attributes in the data are not in the modelignore_attribute_order (
bool
, optional) – ifTrue
, do not require the attributes to be provided in the canonical orderignore_empty_rows (
bool
, optional) – ifTrue
, ignore empty rowsgroup_objects_by_model (
bool
, optional) – ifTrue
, group decoded objects by their typesvalidate (
bool
, optional) – ifTrue
, validate the data
- Returns
- if
group_objects_by_model
set returnsdict
: of model objects grouped byModel
class; else returns
list
: of all model objects
- if
- Return type
obj
- Raises
ValueError – if * Sheets cannot be unambiguously mapped to models * The file(s) indicated by
path
is missing a sheet for a model andignore_missing_models
isFalse
* The file(s) indicated bypath
contains extra sheets that don’t correspond to one ofmodels
andignore_extra_models
isFalse
* The worksheets are file(s) indicated bypath
are not in the canonical order andignore_sheet_order
isFalse
* Some models are not serializable * The data contains parsing errors found byread_model
-
-
class
obj_tables.io.
WorkbookWriter
[source]¶ Bases:
obj_tables.io.WriterBase
Write model objects to an XLSX file or CSV or TSV file(s)
-
static
create_worksheet_style
(model, extra_entries=0)[source]¶ Create worksheet style for model
- Parameters
model (
type
) – model classextra_entries (
int
, optional) – additional entries to display
- Returns
worksheet style
- Return type
WorksheetStyle
-
run
(path, objects, schema_name=None, doc_metadata=None, model_metadata=None, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, write_toc=True, write_schema=False, write_empty_models=True, write_empty_cols=True, extra_entries=0, group_objects_by_model=True, data_repo_metadata=False, schema_package=None, protected=True)[source]¶ - Write a list of model instances to an XLSX file, with one worksheet for each model class,
or to a set of .csv or .tsv files, with one file for each model class
- Parameters
path (
str
) – path to write file(s)objects (
Model
orlist
ofModel
) –Model
instance or list ofModel
instancesschema_name (
str
, optional) – schema namedoc_metadata (
dict
, optional) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)model_metadata (
dict
, optional) – dictionary that maps models to dictionary with their metadata to be saved to header row (e.g.,!!ObjTables ...
)models (
list
ofModel
, optional) – models in the order that they should appear as worksheets; all models which are not inmodels
will follow in alphabetical orderget_related (
bool
, optional) – ifTrue
, writeobjects
and all their related objectsinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
validate (
bool
, optional) – ifTrue
, validate the datatitle (
str
, optional) – titledescription (
str
, optional) – descriptionkeywords (
str
, optional) – keywordsversion (
str
, optional) – versionlanguage (
str
, optional) – languagecreator (
str
, optional) – creatorwrite_toc (
bool
, optional) – ifTrue
, include additional worksheet with table of contentswrite_schema (
bool
, optional) – ifTrue
, include additional worksheet with schemawrite_empty_models (
bool
, optional) – ifTrue
, write models even when there are no instanceswrite_empty_cols (
bool
, optional) – ifTrue
, write columns even when all values areNone
extra_entries (
int
, optional) – additional entries to displaygroup_objects_by_model (
bool
, optional) – ifTrue
, group objects by modeldata_repo_metadata (
bool
, optional) – ifTrue
, try to write metadata information about the file’s Git repo; the repo must be current with origin, except for the fileschema_package (
str
, optional) – the package which defines the ObjTables schema used by the file; if notNone
, try to write metadata information about the the schema’s Git repository: the repo must be current with originprotected (
bool
, optional) – ifTrue
, protect the worksheet
- Raises
ValueError – if no model is provided or a class cannot be serialized
-
write_model
(writer, model, objects, schema_name, date, doc_metadata, doc_metadata_model, model_metadata, sheet_models, include_all_attributes=True, encoded=None, write_empty_models=True, write_empty_cols=True, extra_entries=0, protected=True)[source]¶ Write a list of model objects to a file
- Parameters
writer (
wc_utils.workbook.io.Writer
) – io writermodel (
type
) – modelobjects (
list
ofModel
) – list of instances ofModel
schema_name (
str
) – schema namedate (
str
) – datedoc_metadata (
dict
) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)doc_metadata_model (
type
) – model whose worksheet contains the document metadatamodel_metadata (
dict
) – dictionary of model metadatasheet_models (
list
ofModel
) – models encoded as separate sheetsinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
encoded (
dict
, optional) – objects that have already been encoded and their assigned JSON identifierswrite_empty_models (
bool
, optional) – ifTrue
, write models even when there are no instanceswrite_empty_cols (
bool
, optional) – ifTrue
, write columns even when all values areNone
extra_entries (
int
, optional) – additional entries to displayprotected (
bool
, optional) – ifTrue
, protect the worksheet
-
write_schema
(writer, models, name, date, doc_metadata, protected=True)[source]¶ Write a worksheet with a schema
- Parameters
writer (
wc_utils.workbook.io.Writer
) – io writermodels (
list
ofModel
, optional) – models in the order that they should appear in the table of contentsname (
str
) – namedate (
str
) – datedoc_metadata (
dict
) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)protected (
bool
, optional) – ifTrue
, protect the worksheet
-
write_sheet
(writer, model, data, headings, metadata_headings, validation, extra_entries=0, merge_ranges=None, protected=True)[source]¶ Write data to sheet
- Parameters
writer (
wc_utils.workbook.io.Writer
) – io writermodel (
type
) – modeldata (
list
oflist
ofobject
) – list of list of cell valuesheadings (
list
oflist
ofstr
) – list of list of row headings validationsmetadata_headings (
list
oflist
ofstr
) – model metadata (name, description) to print at the top of the worksheetvalidation (
WorksheetValidation
) – validationextra_entries (
int
, optional) – additional entries to displaymerge_ranges (
list
oftuple
) – list of ranges of cells to mergeprotected (
bool
, optional) – ifTrue
, protect the worksheet
-
write_toc
(writer, models, schema_name, date, doc_metadata, grouped_objects, write_schema=False, protected=True)[source]¶ Write a worksheet with a table of contents
- Parameters
writer (
wc_utils.workbook.io.Writer
) – io writermodels (
list
ofModel
, optional) – models in the order that they should appear in the table of contentsschema_name (
str
) – schema namedate (
str
) – datedoc_metadata (
dict
) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)grouped_objects (
dict
) – dictionary which maps models to lists of instances of each modelwrite_schema (
bool
, optional) – ifTrue
, include additional row for worksheet with schemaprotected (
bool
, optional) – ifTrue
, protect the worksheet
-
static
-
class
obj_tables.io.
Writer
[source]¶ Bases:
obj_tables.io.WriterBase
Write a list of model objects to file(s)
-
static
get_writer
(path)[source]¶ Get writer
- Parameters
path (
str
) – path to write file(s)- Returns
writer class
- Return type
type
- Raises
ValueError – if extension is not supported
-
run
(path, objects, schema_name=None, doc_metadata=None, model_metadata=None, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, write_toc=True, write_schema=False, write_empty_models=True, write_empty_cols=True, extra_entries=0, group_objects_by_model=True, data_repo_metadata=False, schema_package=None, protected=True)[source]¶ - Write a list of model classes to an XLSX file, with one worksheet for each model, or to
a set of .csv or .tsv files, with one file for each model.
- Parameters
path (
str
) – path to write file(s)objects (
Model
orlist
ofModel
) – object or list of objectsschema_name (
str
, optional) – schema namedoc_metadata (
dict
, optional) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)model_metadata (
dict
, optional) – dictionary that maps models to dictionary with their metadata to be saved to header row (e.g.,!!ObjTables ...
)models (
list
ofModel
, optional) – models in the order that they should appear as worksheets; all models which are not inmodels
will follow in alphabetical orderget_related (
bool
, optional) – ifTrue
, writeobjects
and all related objectsinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
validate (
bool
, optional) – ifTrue
, validate the datatitle (
str
, optional) – titledescription (
str
, optional) – descriptionkeywords (
str
, optional) – keywordsversion (
str
, optional) – versionlanguage (
str
, optional) – languagecreator (
str
, optional) – creatorwrite_schema (
bool
, optional) – ifTrue
, include additional worksheet with schemawrite_empty_models (
bool
, optional) – ifTrue
, write models even when there are no instanceswrite_empty_cols (
bool
, optional) – ifTrue
, write columns even when all values areNone
write_toc (
bool
, optional) – ifTrue
, include additional worksheet with table of contentsextra_entries (
int
, optional) – additional entries to displaygroup_objects_by_model (
bool
, optional) – ifTrue
, group objects by modeldata_repo_metadata (
bool
, optional) – ifTrue
, try to write metadata information about the file’s Git repo; the repo must be current with origin, except for the fileschema_package (
str
, optional) – the package which defines the ObjTables schema used by the file; if notNone
, try to write metadata information about the the schema’s Git repository: the repo must be current with originprotected (
bool
, optional) – ifTrue
, protect the worksheet
-
static
-
class
obj_tables.io.
WriterBase
[source]¶ Bases:
object
Interface for classes which write model objects to file(s)
-
MODELS
[source]¶ default types of models to export and the order in which to export them
- Type
tuple
oftype
-
MODELS
= ()[source]
-
make_metadata_objects
(data_repo_metadata, path, schema_package)[source]¶ Make models that store Git repository metadata
Metadata models can only be created from suitable Git repos. Failures to obtain metadata are reported as warnings that do not interfeer with writing data files.
- Parameters
data_repo_metadata (
bool
) – ifTrue
, try to obtain metadata information about the Git repo containingpath
; the repo must be current with origin, except for the file atpath
path (
str
) – path of the file(s) that will be writtenschema_package (
str
, optional) – the package which defines the ObjTables schema used by the file; if notNone
, try to obtain metadata information about the the schema’s Git repository from a package onsys.path
: the repo must be current with its origin
- Returns
metadata objects(s) created
- Return type
list
ofModel
-
abstract
run
(path, objects, schema_name=None, doc_metadata=None, model_metadata=None, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, write_toc=True, write_schema=False, write_empty_models=True, write_empty_cols=True, extra_entries=0, group_objects_by_model=True, data_repo_metadata=False, schema_package=None, protected=True)[source]¶ - Write a list of model classes to an XLSX file, with one worksheet for each model, or to
a set of .csv or .tsv files, with one file for each model.
- Parameters
path (
str
) – path to write file(s)objects (
Model
orlist
ofModel
) – object or list of objectsschema_name (
str
, optional) – schema namedoc_metadata (
dict
, optional) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)model_metadata (
dict
, optional) – dictionary that maps models to dictionary with their metadata to be saved to header row (e.g.,!!ObjTables ...
)models (
list
ofModel
, optional) – modelsget_related (
bool
, optional) – ifTrue
, write object and all related objectsinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
validate (
bool
, optional) – ifTrue
, validate the datatitle (
str
, optional) – titledescription (
str
, optional) – descriptionkeywords (
str
, optional) – keywordsversion (
str
, optional) – versionlanguage (
str
, optional) – languagecreator (
str
, optional) – creatorwrite_toc (
bool
, optional) – ifTrue
, include additional worksheet with table of contentswrite_schema (
bool
, optional) – ifTrue
, include additional worksheet with schemawrite_empty_models (
bool
, optional) – ifTrue
, write models even when there are no instanceswrite_empty_cols (
bool
, optional) – ifTrue
, write columns even when all values areNone
extra_entries (
int
, optional) – additional entries to displaygroup_objects_by_model (
bool
, optional) – ifTrue
, group objects by modeldata_repo_metadata (
bool
, optional) – ifTrue
, try to write metadata information about the file’s Git repo; a warning will be generated if the repo repo is not current with origin, except for the fileschema_package (
str
, optional) – the package which defines the ObjTables schema used by the file; if notNone
, try to write metadata information about the the schema’s Git repository: the repo must be current with originprotected (
bool
, optional) – ifTrue
, protect the worksheet
-
-
obj_tables.io.
convert
(source, destination, schema_name=None, models=None, allow_multiple_sheets_per_model=False, ignore_missing_models=False, ignore_extra_models=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, ignore_empty_rows=True, protected=True)[source]¶ Convert among comma-separated (.csv), XLSX (.xlsx), JavaScript Object Notation (.json), tab-separated (.tsv), and Yet Another Markup Language (.yaml, .yml) formats
- Parameters
source (
str
) – path to source filedestination (
str
) – path to save converted fileschema_name (
str
, optional) – schema namemodels (
list
oftype
) – list of modelsallow_multiple_sheets_per_model (
bool
, optional) – ifTrue
, allow multiple sheets per modelignore_missing_models (
bool
, optional) – ifFalse
, report an error if a worksheet/ file is missing for one or more modelsignore_extra_models (
bool
, optional) – ifTrue
and allmodels
are found, ignore other worksheets or filesignore_sheet_order (
bool
, optional) – ifTrue
, do not require the sheets to be provided in the canonical orderinclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
ignore_missing_attributes (
bool
, optional) – ifFalse
, report an error if a worksheet/file doesn’t contain all of attributes in a model inmodels
ignore_extra_attributes (
bool
, optional) – ifTrue
, do not report errors if attributes in the data are not in the modelignore_attribute_order (
bool
, optional) – ifTrue
, do not require the attributes to be provided in the canonical orderignore_empty_rows (
bool
, optional) – ifTrue
, ignore empty rowsprotected (
bool
, optional) – ifTrue
, protect the worksheet
-
obj_tables.io.
create_template
(path, schema_name, models, title=None, description=None, keywords=None, version=None, language=None, creator=None, write_toc=True, write_schema=False, write_empty_models=True, write_empty_cols=True, extra_entries=10, group_objects_by_model=True, protected=True)[source]¶ Create a template for a model
- Parameters
path (
str
) – path to write file(s)schema_name (
str
) – schema namemodels (
list
) – list of model, in the order that they should appear as worksheets; all models which are not inmodels
will follow in alphabetical ordertitle (
str
, optional) – titledescription (
str
, optional) – descriptionkeywords (
str
, optional) – keywordsversion (
str
, optional) – versionlanguage (
str
, optional) – languagecreator (
str
, optional) – creatorwrite_schema (
bool
, optional) – ifTrue
, include additional worksheet with schemawrite_empty_models (
bool
, optional) – ifTrue
, write models even when there are no instanceswrite_empty_cols (
bool
, optional) – ifTrue
, write columns even when all values areNone
write_toc (
bool
, optional) – ifTrue
, include additional worksheet with table of contentsextra_entries (
int
, optional) – additional entries to displaygroup_objects_by_model (
bool
, optional) – ifTrue
, group objects by modelprotected (
bool
, optional) – ifTrue
, protect the worksheet
-
obj_tables.io.
format_doc_metadata
(schema_name, metadata)[source]¶ Format document metadata as a string of key-value pairs of document metadata
- Parameters
schema_name (
str
) – schema namemetadata (
dict
) – document metadata
- Returns
string of key-value pairs of document metadata
- Return type
str
-
obj_tables.io.
get_fields
(cls, schema_name, date, doc_metadata, doc_metadata_model, model_metadata, include_all_attributes=True, sheet_models=None)[source]¶ Get the attributes, headings, and validation for a worksheet
- Parameters
cls (
type
) – Model type (subclass ofModel
)schema_name (
str
) – schema namedate (
str
) – datedoc_metadata (
dict
) – dictionary of document metadata to be saved to header row (e.g.,!!!ObjTables ...
)doc_metadata_model (
type
) – model whose worksheet contains the document metadatamodel_metadata (
dict
) – dictionary of model metadatainclude_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
sheet_models (
list
ofModel
, optional) – list of models encoded as separate worksheets; used to setup XLSX validation for related attributes
- Returns
list
ofAttribute
: Attributes ofcls
in the order they should be encoded as one or more columns in a worksheet. Attributes which define *-to-one relationships to other classes which are encoded as multiple cells (TableFormat.multiple_cells
) will be encoded as multiple columns. All other attributes will be encoded as a single column.This represents a nested tree of attributes. For classes which have *-to-one relationships to other classes which are encoded as multiple cells, the tree has two levels. For all other classes, the tree only has a single level.
list
of tuple ofAttribute
: Flattened representation of the first return value. This is a list of attributes ofcls
and attributes of classes related tocls
by *-to-one relationships that are encoded as multiple cells (TableFormat.multiple_cells
), in the order they are encoded as columns in a worksheet.Each element of the list is a tuple.
For attributes of
cls
that represent *-to-one relationships to classes encoded as multiple cells, the first element will be the attribute. This will be used to populate a merged cell in Row 1 of the worksheet which represents the heading for the multiple columns that encode the attributes of the related class. For all other attributes, the first element will beNone
, and no value will be printed in Row 1.The second element will be the attribute that should be encoded in the column. For attributes that represent *-to-one relationships to related classes encoded as multiple cells, this will be an attribute of the related class. For all other attributes, this will be an attribute of
cls
. This will be used to populate the columns headings for the worksheet. For classes that have *-to-one relationships with classes encoded as multiple columns, the column headings will appear in Row 2 (and the group headings specified by the first element of the tuple will be in Row 1). For all other classes, the column headings will appear in Row 1.
list
: field headingslist
: list of field headings to mergelist
: list of field validationslist
oflist
str
: model metadata (name and description)to print at the top of the worksheet
- Return type
tuple
-
obj_tables.io.
get_ordered_attributes
(cls, include_all_attributes=True)[source]¶ Get the attributes for a class in the order that they should be printed
- Parameters
cls (
type
) – Model type (subclass ofModel
)include_all_attributes (
bool
, optional) – ifTrue
, export all attributes including those not explictly included inModel.Meta.attribute_order
- Returns
attributes in the order they should be printed
- Return type
list
ofAttribute
3.1.9. obj_tables.migrate module¶
Support schema migration
- Author
Arthur Goldberg <Arthur.Goldberg@mssm.edu>
- Date
2018-11-18
- Copyright
2018, Karr Lab
- License
MIT
-
class
obj_tables.migrate.
CementControllers
[source]¶ Bases:
object
Cement Controllers for cement CLIs in data and schema repos involved with migrating files
Because these controllers are used by multiple schema and data repos, they’re defined here and imported into __main__.py modules in schema repos that use ObjTables to define data schemas and into __main__.py modules in data repos that contain data files to migrate. wc_lang is an example schema repo. wc_sim is an example data repo that contains data files whose schema is defined in wc_lang.
-
class
DataSchemaMigrationConfigController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Create a data-schema migration configuration file.
This controller is used by data repos.
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--data_repo_dir'], {'type': <class 'str'>, 'help': 'path of the directory of the repository storing the data file(s) to migrate; defaults to the current directory', 'default': '.'}), (['schema_url'], {'type': <class 'str'>, 'help': 'URL of the schema in its git repository, including the branch'}), (['file_to_migrate'], {'action': 'store', 'type': <class 'str'>, 'nargs': '+', 'help': 'a file to migrate'})][source]¶
-
-
class
-
class
MigrateController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Perform a migration configured by a data-schema migration config file
This controller is used by data repos.
-
class
MigrateFileController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Migrate specified data file(s)
This controller is used by data repos.
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--data_repo_dir'], {'type': <class 'str'>, 'help': 'path of the directory of the repository storing the data file(s) to migrate; defaults to the current directory', 'default': '.'}), (['schema_url'], {'type': <class 'str'>, 'help': 'URL of the schema in its git repository, including the branch'}), (['file_to_migrate'], {'action': 'store', 'type': <class 'str'>, 'nargs': '+', 'help': 'a file to migrate'})][source]¶
-
-
class
-
class
SchemaChangesTemplateController
(*args, **kw)[source]¶ Bases:
cement.ext.ext_argparse.ArgparseController
Create a template schema changes file
This controller is used by schema repos.
-
class
Meta
[source]¶ Bases:
object
-
arguments
= [(['--schema_repo_dir'], {'type': <class 'str'>, 'help': "path of the directory of the schema's repository; defaults to the current directory", 'default': '.'}), (['--commit'], {'type': <class 'str'>, 'help': 'hash of a commit containing the changes; default is most recent commit'})][source]¶
-
-
class
-
class
-
class
obj_tables.migrate.
DataSchemaMigration
(**kwargs)[source]¶ Bases:
object
Automate the migration of the data files in a repo
A data repo stores the data files that need to be migrated. A schema repo contains the schemas that provide the data models for these data files. The data and schema repos may be the same repo or two different repos.
DataSchemaMigration uses configuration information in the data and schema repos to migrate data files in the data repo to the latest version of the schema repo.
The data repo must contain a migrations directory that has:
Data-schema migration configuration files, written in YAML
A data-schema migration configuration file contains the attributes described in DataSchemaMigration._CONFIG_ATTRIBUTES:
files_to_migrate: a list of files to be migrated
schema_repo_url: the URL of the schema repo
branch: the branch of the schema repo
schema_file: the relative path of the schema file in the schema repo
The schema repo contains a migrations directory that contains schema changes files, which may refer to associated transformations files. Hashes in the changes files must refer to commits in the schema repo. These files are managed by SchemaChanges objects. Migration will not work if changes to the schema are not documented in schema changes files.
-
data_config_file_basename
[source]¶ the basename of the YAML configuration file for the migration, which is stored in the data repo’s migrations directory
- Type
str
-
io_classes
[source]¶ custom schema repo specific reader and/or writer for I/O of existing and migrated files, respectively
- Type
dict
oftype
, optional
-
__str__
()[source]¶ Provide a string representation
- Returns
a string representation of this DataSchemaMigration
- Return type
str
-
automated_migrate
(tmp_dir=None)[source]¶ Migrate the data repo’s data files
Migrate to the latest commit referenced by a schema changes file in the schema repo, and migrate data files in place. If the data repo passed to DataSchemaMigration was a directory, then the migrated data files will be stored in that directory.
- Parameters
tmp_dir (
str
, optional) – if the data repo passed to DataSchemaMigration was an URL, then the migrated files will be returned in a temporary directory. If tmp_dir is provided then it will contain the migrated files; if not, then a temporary directory is created to hold them, and the caller is responsible for deleting it.- Returns
the migrated files, and the value of tmp_dir
- Return type
tuple
oflist
,str
-
generate_migration_spec
(data_file, schema_changes)[source]¶ Generate a MigrationSpec from a sequence of schema changes
The migration will migrate data_file in place.
- Parameters
data_file (
str
) – the existing data file that will be migratedschema_changes (
list
ofSchemaChanges
) – a sequence of schema changes instances
- Returns
- a MigrationSpec that specifies a migration of the file through
the sequence of schema changes
- Return type
- Raises
MigratorError – if the MigrationSpec that’s created doesn’t validate
-
get_data_file_git_commit_hash
(data_file)[source]¶ Get the git commit hash of the schema repo that describes a data file
- Parameters
data_file (
str
) – pathname of a data file- Returns
the hash
- Return type
str
- Raises
MigratorError – if data_file does not contain a schema repo metadata model
-
get_name
()[source]¶ Make a timestamped name for a data-schema migration config file
- Returns
the name
- Return type
str
- Raises
MigratorError – if either the data or the schema git repo are not initialized
-
static
get_name_static
(data_repo_name, schema_repo_name)[source]¶ Make a timestamped name for a data-schema migration config file
- Parameters
data_repo_name (
str
) – name of the data reposchema_repo_name (
str
) – name of the schema repo
- Returns
the name
- Return type
str
-
get_schema_package
()[source]¶ Obtain the name of the schema package from the schema file
- Returns
the package name
- Return type
str
-
import_custom_IO_classes
(io_classes_file_basename=None)[source]¶ If the schema repo has an IO classes file, import custom IO classes for accessing data files
- Parameters
io_classes_file_basename (
str
, optional) – custom basename for the custom IO classes file, which overrides CUSTOM_IO_CLASSES_FILE- Returns
- or
None
: map from ‘reader’ and/or ‘writer’ to IO classes, orNone
: if a custom IO classes file doesn’t exist
- or
- Return type
dict
- Raises
MigratorError – if the custom IO classes file cannot be imported, or if schema_repo_dir is deleted from sys.path while importing it, or neither Reader or Writer are defined in it
-
static
load_config_file
(data_schema_migration_conf_file)[source]¶ Load a data-schema migration config file
- Parameters
data_schema_migration_conf_file (
str
) – path to the data-schema migration config file- Returns
the data in the data-schema migration config file
- Return type
dict
- Raises
MigratorError – if the data-schema migration config file cannot be found, or is not proper YAML, or does not have the right format, or does not contain any data
-
static
make_data_schema_migration_conf_file_cmd
(data_repo_dir, schema_file_url, files_to_migrate, add_to_repo=True)[source]¶ Make a data-schema migration configuration file from CLI input
- Parameters
data_repo_dir (
str
) – directory of the data reposchema_file_url (
str
) – URL for schema’s Python filefiles_to_migrate (
list
ofstr
) – data files to migrateadd_to_repo (
bool
, optional) – if set, add the migration config file to the data repo; default =True
:
- Returns
pathname of the schema changes file that was written
- Return type
str
- Raises
MigratorError – if data_repo_dir isn’t the directory of a repo, or schema_file_url isn’t the URL of a schema file, or files_to_migrate aren’t files
-
static
make_migration_config_file
(data_git_repo, schema_repo_name, add_to_repo=True, **kwargs)[source]¶ Create a data-schema migration config file
- Parameters
data_git_repo (
GitRepo
) – the data git repo that contains the data files to migrateschema_repo_name (
str
) – name of the schema repoadd_to_repo (
bool
, optional) – if set, add the migration config file to the data repo; default =True
:kwargs (
dict
) – optional initial values for data-schema migration config file
- Returns
the pathname to the data-schema migration config file that was written
- Return type
str
- Raises
MigratorError – if the data-schema migration configuration file already exists
-
static
migrate_files
(schema_url, local_dir, data_files)[source]¶ Migrate some data files specified by command line input
Migrate data files in place in a local repository.
- Parameters
schema_url (
str
) – URL of the schema’s Python filelocal_dir (
str
) – directory in a local data repo that contains the data filesdata_files (
list
ofstr
) – data files to migrate
- Returns
list of pathnames of migrated files
- Return type
list
ofstr
- Raises
MigratorError – if schema_url isn’t in the right form, or local_dir isn’t a directory, or any of the data files cannot be found, or the migration fails
-
prepare
()[source]¶ Prepare for migration
Validate this DataSchemaMigration
Clone each schema version specified by a schema change
Generate and prepare
MigrationSpec
: instances for the migration, one for each file
- Raises
MigratorError – if the DataSchemaMigration doesn’t validate
-
record_git_repo
(git_repo)[source]¶ Record a new
GitRepo
: so that its temp dir can be deleted later- Parameters
git_repo (
GitRepo
) – a git repo
-
schema_changes_for_data_file
(data_file)[source]¶ Generate a sequence of SchemaChanges for migrating a data file
- Parameters
data_file (
str
) – a data file in the data git repo- Returns
a sequence of SchemaChanges for migrating a data file
- Return type
list
-
test_migration
()[source]¶ Test a migration
Check …
The trickiest part of a migration is importing the schema. Unfortunately, imports that use the Python import command may fail with migration’s import, which uses the library call importlib.import_module. This should be called whenever a schema that may be migrated is changed.
This method reports:
any validation errors in automatic config files
any validation errors in schema changes files
any errors in transformations
any failures to import schemas
It does not alter any files.
- Parameters
data_repo_location (
str
) – directory or URL of the data repo
-
validate
()[source]¶ Validate files to migrate, and load all schema changes files
- Raises
MigratorError – if any files to migrate do not exist, or all schema changes files cannot be loaded
-
class
obj_tables.migrate.
GitRepo
(repo_location=None, repo_url=None, branch='master', search_parent_directories=False)[source]¶ Bases:
object
Methods for processing a git repo and its commit history
Initialize a
GitRepo
from an existing Git repoIf
repo_location
is a directory then use the Git repo in the directory. Otherwise it must be an URL and the repo is cloned into a temporary directory.- Parameters
repo_location (
str
, optional) – the location of the repo, either its directory or its URLrepo_url (
str
, optional) – the repo’s original URL, which will be set in all copies of a repo that was initially cloned byclone_repo_from_url
branch (
str
, optional) – branch to clone ifrepo_location
is an URL; default ismaster
search_parent_directories (
bool
, optional) –search_parent_directories
option togit.Repo
; if set andrepo_location
is a directory, then all of its parent directories will be searched for a valid repo; default=False
- Returns
root directory for the repo (which contains the .git directory)
- Return type
str
-
__str__
()[source]¶ Provide a string representation
- Returns
a string representation of this GitRepo
- Return type
str
-
add_file
(filename)[source]¶ Add a file to the index
- Parameters
filename (
str
) – path to new file- Raises
MigratorError – if the file cannot be added
-
checkout_commit
(commit_identifier)[source]¶ Checkout a commit for this repo
Use checkout_commit carefully. If it checks out a new commit, then other operations on the repo may behave differently.
- Parameters
commit_identifier (
git.objects.commit.Commit
orstr
) – a commit or a commit’s hash- Raises
MigratorError – if the commit cannot be checked out
-
clone_repo_from_url
(url, branch='master', directory=None)[source]¶ Clone a repo from an URL
- Parameters
url (
str
) – URL for the repobranch (
str
, optional) – branch to clone; default ismaster
directory (
str
, optional) – directory to hold the repo; if not provided, the repo is stored in a new temporary dir
- Returns
(
git.Repo
,str
): the repo cloned, and its root directory- Return type
tuple
- Raises
MigratorError – if repo cannot be cloned from
url
-
commit_changes
(message)[source]¶ Commit the changes in this repo
- Parameters
message (
str
) – the commit message- Raises
MigratorError – if the changes cannot be commited
-
commits_as_graph
()[source]¶ Make a DAG for this repo’s commit dependencies - edges point from dependent commit to parent commit
The DAG contains all commits in the repo on which the latest commit depends. Also creates git_hash_map, a map from all git hashes to their commits.
- Returns
a DAG representing the repo commit history
- Return type
nx.classes.digraph.DiGraph
-
commits_in_dependency_consistent_seq
(commits)[source]¶ Order some git commits into a sequence that’s consistent with the repo’s dependencies
Generate a topological sort of the commits. If the dependency relationship contains branching, then the sequence found is not deterministic, because concurrent nodes can appear in any relative order in the sort. E.g., in a commit DAG with the paths a -> b -> c and a -> d -> c, nodes b and d can appear in either order in the sequence.
- Parameters
commits (
list
ofgit.objects.commit.Commit
) – commits to include in the returned sequence- Returns
- the elements of commits, in a sequence
that’s consistent with git commit dependencies in the repository, ordered from from antecedent to dependent
- Return type
list
ofgit.objects.commit.Commit
-
copy
(tmp_dir=None)[source]¶ Copy this
GitRepo
into a new directoryFor better performance use
copy()
instead ofGitRepo()
orclone_repo_from_url()
if you need multiple copies of a repo, such as multiple instances checked out to different commits. This is an optimization because copying is faster than cloning over the network. To avoidbytecode is stale
errors, doesn’t copy__pycache__
directories.
-
del_temp_dirs
()[source]¶ Delete the temp dirs created by
get_temp_dir
- Returns
the pathname to a temporary directory
- Return type
str
-
fixtures_dir
()[source]¶ Get the repo’s fixtures directory
- Returns
the repo’s fixtures directory
- Return type
str
-
get_commit
(commit_or_hash)[source]¶ Obtain a commit from its hash
Also, if
commit_or_hash
is a commit, simply return it.- Parameters
commit_or_hash (
str
orgit.objects.commit.Commit
) – the hash of a commit or a commit in the repo- Returns
a commit
- Return type
git.objects.commit.Commit
- Raises
MigratorError – if
commit_or_hash
is not a commit and cannot be converted into one
-
get_commits
(commits_or_hashes)[source]¶ Get the commits with the given commits or hashes
- Parameters
commits_or_hashes (
list
ofstr
) – an iterator over commits or commit hashes- Returns
- list of the repo’s commits with the
commits or hashes in
commits_or_hashes
- Return type
list
ofgit.objects.commit.Commit
- Raises
MigratorError – if any commit or hash in commits_or_hashes doesn’t identify a commit in this repo
-
get_dependents
(commit_or_hash)[source]¶ Get all commits that depend on a commit, including transitively
- Parameters
commit_or_hash (
str
orgit.objects.commit.Commit
) – the hash of a commit or a commit in the repo- Returns
all commits that depend on commit_or_hash
- Return type
set
ofgit.objects.commit.Commit
-
static
get_hash
(commit)[source]¶ Get a commit’s hash
- Parameters
commit (
git.objects.commit.Commit
) – a commit- Returns
the commit’s SHA1 hash
- Return type
str
-
get_metadata
(metadata_type)[source]¶ Create a metadata model that describes the current state of this repo
- Parameters
metadata_type (
class
) – a subclass of RepoMetadata, either DataRepoMetadata or SchemaRepoMetadata- Returns
a RepoMetadata object
- Return type
RepoMetadata
- Raises
MigratorError – if the metadata isn’t available in this repo
-
get_temp_dir
()[source]¶ Get a temporary directory, which must eventually be deleted by calling
del_temp_dirs
- Returns
the pathname to a temporary directory
- Return type
str
-
static
hash_prefix
(hash)[source]¶ Get a commit hash’s prefix
- Parameters
hash (
str
) – git commit hash- Returns
hash’s prefix
- Return type
str
-
head_commit
()[source]¶ Get the repo’s head commit
- Returns
the repo’s latest commit
- Return type
git.objects.commit.Commit
-
latest_hash
()[source]¶ Get the hash of the repo’s head commit
- Returns
the head commit’s SHA1 hash
- Return type
str
-
class
obj_tables.migrate.
MigrationController
[source]¶ Bases:
object
Manage migrations
Manage migrations on several dimensions:
Migrate a single model file through a sequence of schemas
Perform migrations parameterized by a configuration file
-
static
migrate_from_config
(migrations_spec_config_file)[source]¶ Perform the migrations specified in a migrations spec config file
- Parameters
migrations_spec_config_file (
str
) – a migrations spec configuration file, written in YAML- Returns
list of (
MigrationSpec
, migrated filenames) pairs- Return type
list
oftuple
-
static
migrate_from_spec
(migration_spec)[source]¶ Perform the migration specified in a
MigrationSpec
- Parameters
migration_spec (
MigrationSpec
) – a migration specification- Returns
of
str
: migrated filenames- Return type
list
-
static
migrate_over_schema_sequence
(migration_spec)[source]¶ Migrate some model files over a sequence of schemas
- Parameters
migration_spec (
MigrationSpec
) – a migration specification- Returns
- for each migration, its sequence of models and
its migrated filename
- Return type
tuple
oflist
,list
- Raises
MigratorError – if schema_files, renamed_models, and seq_of_renamed_attributes are not consistent with each other;
-
class
obj_tables.migrate.
MigrationSpec
(name, **kwargs)[source]¶ Bases:
object
Specification of a sequence of migrations for a list of existing files
-
_REQUIRED_ATTRS
[source]¶ required attributes in a
MigrationSpec
- Type
list
ofstr
-
_ALLOWED_ATTRS
[source]¶ attributes allowed in a
MigrationSpec
- Type
list
ofstr
-
name
[source]¶ name for this
MigrationSpec
- Type
str
-
existing_files (:obj:`list`
of
str
, optional): existing files to migrate
-
schema_files
[source]¶ list of Python files containing model definitions for each state in a sequence of migrations
- Type
list
ofstr
, optional
-
seq_of_renamed_models
[source]¶ list of renamed models for use by a
Migrator
for each migration in a sequence of migrations- Type
list
oflist
, optional
-
seq_of_renamed_attributes
[source]¶ list of renamed attributes for use by a
Migrator
for each migration in a sequence- Type
list
oflist
, optional
-
seq_of_transformations
[source]¶ list of transformations for use by a
Migrator
for each migration in a sequence- Type
list
ofMigrationWrapper
, optional
-
migrated_files (:obj:`list`
of
str
, optional): migration destination files in 1-to-1 correspondence withexisting_files
; if not provided, migrated files use a suffix or are migrated in place
-
io_classes
[source]¶ reader and/or writer for I/O of existing and migrated files, respectively; IMPORTANT NOTE: io_classes are imported and loaded from current schema repositories by
DataSchemaMigration
- Type
dict
oftype
, optional
-
migrate_suffix
[source]¶ suffix added to destination file name, before the file type suffix
- Type
str
, optional
-
migrations_config_file
[source]¶ if created from a configuration file, the file’s path
- Type
str
, optional
-
final_schema_git_metadata
[source]¶ a git metadata model for the repo containing the last schema in the migration; may initialized directly, or constructed from the other
final_schema_*
attributes- Type
SchemaRepoMetadata
, optional
-
_prepared
[source]¶ whether this
MigrationSpec
has been prepared- Type
bool
, optional
-
__str__
()[source]¶ Get str representation
- Returns
string representation of all allowed attributes in a
MigrationSpec
- Return type
str
-
expected_migrated_files
()[source]¶ Provide names of migrated files that migration of this
MigrationSpec
would produce- Returns
- the names of the migrated files that a successful migration of this
MigrationSpec
will produce
- Return type
list
ofstr
-
static
get_migrations_config
(migrations_config_file)[source]¶ Create a list of
MigrationSpec
s from a migrations configuration file- Parameters
migrations_config_file (
str
) – pathname of migrations configuration in YAML file- Returns
migration specifications
- Return type
dict
ofMigrationSpec
- Raises
MigratorError – if
migrations_config_file
cannot be read
-
is_prepared
()[source]¶ Check that this
MigrationSpec
has been prepared- Raises
MigratorError – if this
MigrationSpec
has not been prepared
-
classmethod
load
(migrations_config_file)[source]¶ Create a list of validated and standardized
MigrationSpec
s from a migrations configuration file- Parameters
migrations_config_file (
str
) – pathname of migrations configuration in YAML file- Returns
migration specifications
- Return type
dict
ofMigrationSpec
- Raises
MigratorError – if
migrations_config_file
cannot be read, or the migration specifications inmigrations_config_file
are not valid
-
prepare
()[source]¶ Validate and standardize this
MigrationSpec
- Raises
MigratorError – if migrations_config_file cannot be read, or the migration specifications in migrations_config_file are not valid
-
standardize
()[source]¶ Standardize the attributes of a
MigrationSpec
In particular, standardize a
MigrationSpec
that has been read from a YAML config file
-
-
class
obj_tables.migrate.
MigrationWrapper
[source]¶ Bases:
abc.ABC
Interface for classes that define a pair of methods that can modify
obj_tables.Model
s being migratedMigrationWrapper
defines the interface used by transformations. If it’s defined, transformations usesprepare_existing_models
to modify existing models just before they are migrated and usesmodify_migrated_models
to modify migrated models just after they are migrated.A transformations is associated with each
SchemaChanges
, and obtained from the transformations file configured in a schema changes file.-
static
import_migration_wrapper
(migration_wrapper_file)[source]¶ Import the migration wrapper instances defined in a Python file
:param migration_wrapper_file (
str
: ) name of a file that defines a migration wrappers- Returns
- the
MigrationWrapper
instances inmigration_wrapper_file
, keyed by their attribute names
- the
- Return type
dict
- Raises
MigratorError – if
migration_wrapper_file
cannot be imported
-
static
-
class
obj_tables.migrate.
Migrator
(existing_defs_file=None, migrated_defs_file=None, renamed_models=None, renamed_attributes=None, io_classes=None, transformations=None)[source]¶ Bases:
object
Support schema migration
-
deleted_models
[source]¶ model types defined in the existing models but not the migrated models
- Type
set
-
renamed_models
[source]¶ model types renamed from the existing to the migrated schema
- Type
list
oftuple
-
renamed_attributes
[source]¶ attribute names renamed from the existing to the migrated schema
- Type
list
oftuple
-
renamed_attributes_map
[source]¶ map of attribute names renamed from the existing to the migrated schema
- Type
dict
-
_migrated_copy_attr_name
[source]¶ attribute name used to point existing models to corresponding migrated models; not used in any existing schema
- Type
str
-
io_classes
[source]¶ reader and writer for I/O of existing and migrated files, respectively; defaults provided in DEFAULT_IO_CLASSES
- Type
dict
oftype
-
transformations
[source]¶ optional transformations which modify models before and/or after migration
- Type
Construct a Migrator
If it’s defined, transformations uses its prepare_existing_models method to modify existing models just before they are migrated and uses its modify_migrated_models method to modify migrated models just after they are migrated. A different transformations is associated with each granular step in a sequence of migrations (see
SchemaChanges
below) so that different transformations can wrap migration in each step.- Parameters
existing_defs_file (
str
, optional) – path of a file containing existing Model definitionsmigrated_defs_file (
str
, optional) – path of a file containing migrated Model definitions; filenames optional so that Migrator can use models defined in memoryrenamed_models (
list
oftuple
, optional) – model types renamed from the existing to the migrated schema; has the form [(‘Existing_1’, ‘Migrated_1’), …, (‘Existing_n’, ‘Migrated_n’)], where (‘Existing_i’, ‘Migrated_i’) indicates that existing model Existing_i is being renamed into migrated model Migrated_i.renamed_attributes (
list
oftuple
, optional) – attribute names renamed from the existing to the migrated schema; a list of tuples of the form ((‘Existing_Model_i’, ‘Existing_Attr_x’), (‘Migrated_Model_j’, ‘Migrated_Attr_y’)), which indicates that Existing_Model_i.Existing_Attr_x will migrate to Migrated_Model_j.Migrated_Attr_yio_classes (
dict
oftype
, optional) – reader and/or writer for I/O of existing and migrated files, respectively; if provided, overrides defaults provided in DEFAULT_IO_CLASSEStransformations (
MigrationWrapper
, optional) – transformations which modify models before and/or after migration
-
COLLECTIONS_ATTRS
= ['existing_defs', 'migrated_defs', 'renamed_models', 'models_map', 'renamed_attributes', 'renamed_attributes_map'][source]¶
-