3.1. obj_model package

3.1.1. Submodules

3.1.2. obj_model.abstract module

Support for abstract model classes

Author:Jonathan Karr <karr@mssm.edu>
Date:2017-05-23
Copyright:2016, Karr Lab
License:MIT
class obj_model.abstract.AbstractModel(**kwargs)[source]

Bases: obj_model.core.Model

Abstract model base class

class Meta[source]

Bases: obj_model.core.Meta

attribute_order = ()[source]
attributes = {}[source]
children = {}[source]
frozen_columns = 1[source]
help = ''[source]
indexed_attrs_tuples = ()[source]
inheritance = (<class 'obj_model.abstract.AbstractModel'>,)[source]
local_attributes = {}[source]
merge = 1[source]
ordering = ()[source]
primary_attribute = None[source]
related_attributes = {}[source]
tabular_orientation = 1[source]
unique_together = ()[source]
verbose_name = 'Abstract model'[source]
verbose_name_plural = 'Abstract models'[source]
objects = <obj_model.core.Manager object>[source]
class obj_model.abstract.AbstractModelMeta[source]

Bases: obj_model.core.ModelMeta, abc.ABCMeta

Abstract model metaclass

3.1.3. obj_model.bio module

Biological attributes

Author:Jonathan Karr <karr@mssm.edu>
Date:2017-05-10
Copyright:2017, Karr Lab
License:MIT
class obj_model.bio.BioDnaSeqAttribute(min_length=0, max_length=inf, default=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.bio.BioSeqAttribute

Bio.Seq.Seq attribute with Bio.Alphabet.DNAAlphabet

class obj_model.bio.BioProteinSeqAttribute(min_length=0, max_length=inf, default=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.bio.BioSeqAttribute

Bio.Seq.Seq attribute with Bio.Alphabet.ProteinAlphabet

class obj_model.bio.BioRnaSeqAttribute(min_length=0, max_length=inf, default=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.bio.BioSeqAttribute

Bio.Seq.Seq attribute with Bio.Alphabet.RNAAlphabet

class obj_model.bio.BioSeqAttribute(min_length=0, max_length=inf, default=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.LiteralAttribute

Bio.Seq.Seq attribute

_alphabet[source]

alphabet

Type:Bio.Alphabet.Alphabet
min_length[source]

minimum length

Type:int
max_length[source]

maximum length

Type:int
default[source]

default value

Type:Bio.Seq.Seq
deserialize(value)[source]

Deserialize value

Parameters:value (str) – semantically equivalent representation
Returns:tuple of cleaned value and cleaning error
Return type:tuple of Bio.Seq.Seq, core.InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (dict) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:Bio.Seq.Seq
serialize(value)[source]

Serialize string

Parameters:value (Bio.Seq.Seq) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (Bio.Seq.Seq) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:dict
validate(obj, value)[source]

Determine if value is a valid value

Parameters:
  • obj (Model) – class being validated
  • value (Bio.Seq.Seq) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

validate_unique(objects, values)[source]

Determine if the attribute values are unique

Parameters:
  • objects (list of Model) – list of Model objects
  • values (list of Bio.Seq.Seq) – list of values
Returns:

None if values are unique, otherwise return a list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

class obj_model.bio.FeatureLocationAttribute(default=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.LiteralAttribute

Bio.SeqFeature.FeatureLocation attribute

default[source]

default value

Type:Bio.SeqFeature.FeatureLocation
deserialize(value)[source]

Deserialize value

Parameters:value (str) – semantically equivalent representation
Returns:tuple of cleaned value and cleaning error
Return type:tuple of numpy.array, core.InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (dict) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:Bio.SeqFeature.FeatureLocation
serialize(value)[source]

Serialize string

Parameters:value (numpy.array) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (Bio.SeqFeature.FeatureLocation) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:dict
validate(obj, value)[source]

Determine if value is a valid value

Parameters:
  • obj (Model) – class being validated
  • value (numpy.array) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

validate_unique(objects, values)[source]

Determine if the attribute values are unique

Parameters:
  • objects (list of Model) – list of Model objects
  • values (list of Bio.SeqFeature.FeatureLocation) – list of values
Returns:

None if values are unique, otherwise return a list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

class obj_model.bio.FrequencyPositionMatrixAttribute(verbose_name='', help='')[source]

Bases: obj_model.core.LiteralAttribute

Bio.motif.matrix.FrequencyPositionMatrix attribute

deserialize(value)[source]

Deserialize value

Parameters:value (str) – string representation
Returns:tuple of cleaned value and cleaning error
Return type:tuple of Bio.motifs.matrix.FrequencyPositionMatrix, core.InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (dict) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:Bio.motifs.matrix.FrequencyPositionMatrix
serialize(value)[source]

Serialize value to a string

Parameters:value (Bio.motifs.matrix.FrequencyPositionMatrix) – Python representation
Returns:string representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (Bio.motifs.matrix.FrequencyPositionMatrix) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:dict
validate(obj, value)[source]

Determine if value is a valid value

Parameters:
  • obj (Model) – class being validated
  • value (Bio.motifs.matrix.FrequencyPositionMatrix) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

3.1.4. obj_model.chem module

Chemistry attributes

Author:Jonathan Karr <karr@mssm.edu>
Date:2017-05-10
Copyright:2017, Karr Lab
License:MIT
class obj_model.chem.ChemicalStructureAttribute(default=None, none_value=None, verbose_name='', help='The SMILES- or BpForms-encoded structure of a chemical compound.nnExamples:n Small molecules (SMILES): C([N+])C([O-])=On DNA (BpForms/dna): A{m2C}GTn RNA (BpForms/rna): AC{02G}Un Protein (BpForms/protein): RNC{AA0037}E', primary=False, unique=False)[source]

Bases: obj_model.core.LongStringAttribute

Attribute for the structures of chemical compounds

deserialize(value)[source]

Deserialize value

Parameters:value (str) – string representation of structure
Returns:cleaned value core.InvalidAttribute: cleaning error
Return type:str
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (dict) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:chem.EmpiricalFormula
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize string

Parameters:value (chem.EmpiricalFormula) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (chem.EmpiricalFormula) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:dict
validate(obj, value)[source]

Determine if value is a valid value

Parameters:
  • obj (Model) – class being validated
  • value (chem.EmpiricalFormula) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

validate_unique(objects, values)[source]

Determine if the attribute values are unique

Parameters:
  • objects (list of Model) – list of Model objects
  • values (list of chem.EmpiricalFormula) – list of values
Returns:

None if values are unique, otherwise return a list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

class obj_model.chem.EmpiricalFormulaAttribute(default=None, none_value=None, verbose_name='', help="An empirical formula (e.g. 'H2O', 'CO2', or 'NaCl')", primary=False, unique=False)[source]

Bases: obj_model.core.LiteralAttribute

Empirical formula attribute

deserialize(value)[source]

Deserialize value

Parameters:value (str) – semantically equivalent representation
Returns:cleaned value core.InvalidAttribute: cleaning error
Return type:chem.EmpiricalFormula
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (dict) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:chem.EmpiricalFormula
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize string

Parameters:value (chem.EmpiricalFormula) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (chem.EmpiricalFormula) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:dict
validate(obj, value)[source]

Determine if value is a valid value

Parameters:
  • obj (Model) – class being validated
  • value (chem.EmpiricalFormula) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

validate_unique(objects, values)[source]

Determine if the attribute values are unique

Parameters:
  • objects (list of Model) – list of Model objects
  • values (list of chem.EmpiricalFormula) – list of values
Returns:

None if values are unique, otherwise return a list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

3.1.5. obj_model.core module

Database-independent Django-like object model

Many classes contain the methods serialize() and deserialize(), which invert each other. serialize() converts a python object instance into a string representation, whereas deserialize() parses an object’s string representation – as would be stored in a file or spreadsheet representation of a biochemical model – into a python object instance. deserialize() returns an error when the string representation cannot be parsed into the python object. Deserialization methods for related attributes (subclasses of RelatedAttribute) do not get called until all other attributes have been deserialized. In particular, they’re called by obj_model.io.WorkbookReader.link_model. Therefore, they get passed all objects that are not inline, which can then be referenced to deserialize the related attribute.

Author:Jonathan Karr <karr@mssm.edu>
Author:Arthur Goldberg <Arthur.Goldberg@mssm.edu>
Date:2016-12-12
Copyright:2016, Karr Lab
License:MIT
class obj_model.core.Attribute(init_value=None, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: object

Model attribute

name[source]

name

Type:str
init_value[source]

initial value

Type:object
default[source]

default value

Type:object
default_cleaned_value[source]

value to replace None values with during cleaning, or function which computes the value to replace None values

Type:object
none_value[source]

none value

Type:object
verbose_name[source]

verbose name

Type:str
help[source]

help string

Type:str
primary[source]

indicate if attribute is primary attribute

Type:bool
unique[source]

indicate if attribute value must be unique

Type:bool
unique_case_insensitive[source]

if true, conduct case-insensitive test of uniqueness

Type:bool
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:tuple of cleaned value and cleaning error
Return type:tuple of object, InvalidAttribute or None
copy_value(value, objects_and_copies)[source]

Copy value

Parameters:
  • value (object) – value
  • objects_and_copies (dict) – dictionary that maps objects to their copies
Returns:

copy of value

Return type:

object

deserialize(value)[source]

Deserialize value

Parameters:value (object) – semantically equivalent representation
Returns:tuple of cleaned value and cleaning error
Return type:tuple of object, InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (object) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:object
get_default()[source]

Get default value for attribute

Returns:initial value
Return type:object
get_default_cleaned_value()[source]

Get value to replace None values with during cleaning

Returns:initial value
Return type:object
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
get_init_value(obj)[source]

Get initial value for attribute

Parameters:obj (Model) – object whose attribute is being initialized
Returns:initial value
Return type:object
get_none_value()[source]

Get none value

Returns:none value
Return type:object
merge(left, right, right_objs_in_left, left_objs_in_right)[source]

Merge an attribute of elements of two models

Parameters:
  • left (Model) – an element in a model to merge
  • right (Model) – an element in a second model to merge
  • right_objs_in_left (dict) – mapping from objects in right model to objects in left model
  • left_objs_in_right (dict) – mapping from objects in left model to objects in right model
serialize(value)[source]

Serialize value

Parameters:value (object) – Python representation
Returns:simple Python representation
Return type:bool, float, str, or None
set_value(obj, new_value)[source]

Set value of attribute of object

Parameters:
  • obj (Model) – object
  • new_value (object) – new attribute value
Returns:

attribute value

Return type:

object

to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (object) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:object
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, otherwise return a list

of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

validate_unique(objects, values)[source]

Determine if the attribute values are unique

Parameters:
  • objects (list of Model) – list of Model objects
  • values (list) – list of values
Returns:

None if values are unique, otherwise return a list of

errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

value_equal(val1, val2, tol=0.0)[source]

Determine if attribute values are equal

Parameters:
  • val1 (object) – first value
  • val2 (object) – second value
  • tol (float, optional) – equality tolerance
Returns:

True if attribute values are equal

Return type:

bool

class obj_model.core.BooleanAttribute(default=False, default_cleaned_value=None, none_value=None, verbose_name='', help='Enter a Boolean value')[source]

Bases: obj_model.core.LiteralAttribute

Boolean attribute

default[source]

default value

Type:bool
default_cleaned_value[source]

value to replace None values with during cleaning

Type:bool
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:tuple of cleaned value and cleaning error
Return type:tuple of bool, InvalidAttribute or None
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.DateAttribute(none=True, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.LiteralAttribute

Date attribute

none[source]

if False, the attribute is invalid if its value is None

Type:bool
default[source]

default date

Type:date
default_cleaned_value[source]

value to replace None values with during cleaning, or function which computes the value to replace None values

Type:date
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:(date, None), or (None, InvalidAttribute) reporting error
Return type:tuple
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (str) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:date
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize string

Parameters:value (date) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (date) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:str
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (date) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.DateTimeAttribute(none=True, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.LiteralAttribute

Datetime attribute

none[source]

if False, the attribute is invalid if its value is None

Type:bool
default[source]

default datetime

Type:datetime
default_cleaned_value[source]

value to replace None values with during cleaning, or function which computes the value to replace None values

Type:datetime
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:tuple of cleaned value and cleaning error
Return type:tuple of datetime, InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (str) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:datetime
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize string

Parameters:value (datetime) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (datetime) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:str
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (datetime) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.EmailAttribute(verbose_name='', help='Enter a valid email address', primary=False, unique=False)[source]

Bases: obj_model.core.StringAttribute

Attribute for email addresses

validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (date) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.EnumAttribute(enum_class, none=False, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.core.LiteralAttribute

Enumeration attribute

enum_class[source]

subclass of Enum

Type:type
none[source]

if False, the attribute is invalid if its value is None

Type:bool
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:tuple of cleaned value and cleaning error
Return type:tuple of Enum, InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (str) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:Enum
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize enumeration

Parameters:value (Enum) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (Enum) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:str
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.FloatAttribute(min=nan, max=nan, nan=True, default=nan, default_cleaned_value=nan, none_value=nan, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.NumericAttribute

Float attribute

default[source]

default value

Type:float
default_cleaned_value[source]

value to replace None values with during cleaning

Type:float
min[source]

minimum value

Type:float
max[source]

maximum value

Type:float
nan[source]

if true, allow nan values

Type:bool
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:tuple of cleaned value and cleaning error
Return type:tuple of float, InvalidAttribute or None
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
merge(left, right, right_objs_in_left, left_objs_in_right)[source]

Merge an attribute of elements of two models

Parameters:
  • left (Model) – an element in a model to merge
  • right (Model) – an element in a second model to merge
  • right_objs_in_left (dict) – mapping from objects in right model to objects in left model
  • left_objs_in_right (dict) – mapping from objects in left model to objects in right model
Raises:

ValueError – if the attributes of the elements of the models are different

serialize(value)[source]

Serialize float

Parameters:value (float) – Python representation
Returns:simple Python representation
Return type:float
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

value_equal(val1, val2, tol=0.0)[source]

Determine if attribute values are equal, optionally, up to a tolerance

Parameters:
  • val1 (object) – first value
  • val2 (object) – second value
  • tol (float, optional) – equality tolerance
Returns:

True if attribute values are equal

Return type:

bool

class obj_model.core.IntegerAttribute(min=None, max=None, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.NumericAttribute

Integer attribute

default[source]

default value

Type:int
default_cleaned_value[source]

value to replace None values with during cleaning

Type:int
min[source]

minimum value

Type:int
max[source]

maximum value

Type:int
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:tuple of cleaned value and cleaning error
Return type:tuple of int, InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (float) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:int
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize integer

Parameters:value (int) – Python representation
Returns:simple Python representation
Return type:float
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (int) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:float
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, otherwise return list of

errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.InvalidAttribute(attribute, messages, related=False, location=None, value=None)[source]

Bases: object

Represents an invalid attribute and its errors

attribute[source]

invalid attribute

Type:Attribute
messages[source]

list of error messages

Type:list of str
related[source]

indicates if error is about value or related value

Type:bool
location[source]

a string representation of the attribute’s location in an input file

Type:str, optional
value[source]

invalid input value

Type:str, optional
__str__()[source]

Get string representation of errors

Returns:string representation of errors
Return type:str
set_location_and_value(location, value)[source]

Set the location and value of the attribute

Parameters:
  • location (str) – a string representation of the attribute’s location in an input file
  • value (str) – the invalid input value
class obj_model.core.InvalidModel(model, attributes)[source]

Bases: object

Represents an invalid model, such as a model with an attribute that fails to meet specified constraints

model[source]

Model class

Type:class
attributes[source]

list of invalid attributes and their errors

Type:list of InvalidAttribute
__str__()[source]

Get string representation of errors

Returns:string representation of errors
Return type:str
class obj_model.core.InvalidObject(object, attributes)[source]

Bases: object

Represents an invalid object and its errors

object[source]

invalid object

Type:object
attributes[source]

list of invalid attributes and their errors

Type:list of InvalidAttribute
__str__()[source]

Get string representation of errors

Returns:string representation of errors
Return type:str
class obj_model.core.InvalidObjectSet(invalid_objects, invalid_models)[source]

Bases: object

Represents a list of invalid objects and invalid models

invalid_objects[source]

list of invalid objects

Type:list of InvalidObject
invalid_models[source]

list of invalid models

Type:list of InvalidModel
__str__()[source]

Get string representation of errors

Returns:string representation of errors
Return type:str
get_model_errors_by_model()[source]

Get model errors grouped by models

Returns:InvalidModel: dictionary of model errors, grouped by model
Return type:dict of Model
get_object_errors_by_model()[source]

Get object errors grouped by model

Returns:list of InvalidObject: dictionary of object errors, grouped by model
Return type:dict of Model
class obj_model.core.LiteralAttribute(init_value=None, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.core.Attribute

Base class for literal attributes (Boolean, enumeration, float, integer, string, etc.)

copy_value(value, objects_and_copies)[source]

Copy value

Parameters:
  • value (object) – value
  • objects_and_copies (dict) – dictionary that maps objects to their copies
Returns:

copy of value

Return type:

object

deserialize(value)[source]

Deserialize value

Parameters:value (object) – semantically equivalent representation
Returns:tuple of cleaned value and cleaning error
Return type:tuple of object, InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (object) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:object
merge(left, right, right_objs_in_left, left_objs_in_right)[source]

Merge an attribute of elements of two models

Parameters:
  • left (Model) – an element in a model to merge
  • right (Model) – an element in a second model to merge
  • right_objs_in_left (dict) – mapping from objects in right model to objects in left model
  • left_objs_in_right (dict) – mapping from objects in left model to objects in right model
Raises:

ValueError – if the attributes of the elements of the models are different

serialize(value)[source]

Serialize value

Parameters:value (object) – Python representation
Returns:simple Python representation
Return type:bool, float, str, or None
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (object) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:object
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, otherwise return a

list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.LocalAttribute(attr, primary_class, is_primary=True)[source]

Bases: object

Meta data about a local attribute in a class

attr[source]

attribute

Type:Attribute
cls[source]

class which owns this attribute

Type:type
name[source]

name of the attr in cls

Type:str
related_class[source]

other class which is related to this attribute

Type:type
related_name[source]

name of this attribute in related_cls

Type:str
primary_class[source]

class in which this attribute was defined

Type:type
primary_name[source]

name of this attribute in primary_cls

Type:str
secondary_class[source]

related class to primary_cls

Type:type
secondary_name[source]

name of this attribute in secondary_cls

Type:str
is_primary[source]

True if this attr was defined in cls (cls`=:obj:`primary_cls)

Type:bool

True if this attribute is an instance of RelatedAttribute

Type:bool
is_related_to_many (obj

bool): True if the value of this attribute is a list (*-to-many relationship)

class obj_model.core.LongStringAttribute(min_length=0, max_length=4294967295, default='', default_cleaned_value='', none_value='', verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.core.StringAttribute

Long string attribute

class obj_model.core.Manager(cls)[source]

Bases: object

Enable O(1) dictionary-based searching of a Model’s instances

This class is inspired by Django’s Manager class. An instance of Manger is associated with each Model and accessed as the class attribute objects (as in Django). The tuples of attributes to index are specified by the indexed_attrs_tuples attribute of core.Model.Meta, which contains a tuple of tuples of attributes to index. Model`s with empty `indexed_attrs_tuples attributes incur no overhead from Manager.

Manager maintains a dictionary for each indexed attribute tuple, and a reverse index from each Model instance to its indexed attribute tuple keys.

These data structures support * O(1) get operations for Model instances indexed by a indexed attribute tuple * O(1) Model instance insert and update operations

cls[source]

the Model class which is being managed

Type:class
_new_instances[source]

set of all new instances of cls that have not been indexed, stored as weakrefs, so Model’s that are otherwise unused can be garbage collected

Type:WeakSet
_index_dicts[source]

indices that enable lookup of Model instances from their Meta.indexed_attrs_tuples mapping: <attr names tuple> -> <attr values tuple> -> WeakSet(<model_obj instances>)

Type:dict mapping tuple to WeakSet
_reverse_index[source]

a reverse index that provides all of each Model’s indexed attribute tuple keys mapping: <model_obj instances> -> <attr names tuple> -> <attr values tuple>

Type:WeakKeyDictionary mapping Model instance to dict
num_ops_since_gc[source]

number of operations since the last gc of weaksets

Type:int
GC_PERIOD = 1000[source]
all()[source]

Provide all instances of the Model managed by this Manager

Returns:a list of all instances of the managed Model or None if the Model is not indexed
Return type:list of Model
clear_new_instances()[source]

Clear the set of new instances that have not been inserted

get(**kwargs)[source]

Get the Model instance(s) that match the attribute name,value pair(s) in kwargs

The keys in kwargs must correspond to an entry in the Model’s indexed_attrs_tuples. Warning: this method is non-deterministic. To obtain Manager’s O(1) performance, Model instances in the index are stored in WeakSet’s. Therefore, the order of elements in the list returned is not reproducible. Applications that need reproducibility must deterministically order elements in lists returned by this method.

Parameters:

kwargs (dict) – keyword args mapping from attribute name(s) to value(s)

Returns:

a list of Model instances whose indexed attribute tuples have the values in kwargs; otherwise None, indicating no match

Return type:

list of Model

Raises:
  • ValueError – if no arguments are provided, or the attribute name(s) in kwargs.keys()
  • do not correspond to an indexed attribute tuple of the Model
get_one(**kwargs)[source]

Get one Model instance that matches the attribute name,value pair(s) in kwargs

Uses get.

Parameters:kwargs (dict) – keyword args mapping from attribute name(s) to value(s)
Returns:a Model instance whose indexed attribute tuples have the values in kwargs, or None if no Model satisfies the query
Return type:Model
Raises:ValueError – if get raises an exception, or if multiple instances match.
insert_all_new()[source]

Insert all new instances of this Manager’s Model’s into the search indices

reset()[source]

Reset this Manager

Empty Manager’s indices. Since Manager globally indexes all instances of a Model, this method is useful when multiple models are loaded sequentially.

upsert(model_obj)[source]

Update the indices for model_obj that are used to search on indexed attribute tuples

Upsert means update or insert. Update the indices if model_obj is already stored, otherwise insert model_obj. Users of Manager are responsible for calling this method if model_obj changes.

Costs O(I) where I is the number of indexed attribute tuples for the Model.

Parameters:model_obj (Model) – a Model instance
upsert_all()[source]

Upsert the indices for all of this Manager’s Model’s

class obj_model.core.ManyToManyAttribute(related_class, related_name='', default=[], default_cleaned_value=[], related_default=[], none_value=<class 'list'>, min_related=0, max_related=inf, min_related_rev=0, max_related_rev=inf, verbose_name='', verbose_related_name='', help='', related_manager=<class 'obj_model.core.ManyToManyRelatedManager'>)[source]

Bases: obj_model.core.RelatedAttribute

Represents a many-to-many relationship between two types of objects.

related_manager[source]

related manager

Type:type
copy_value(value, objects_and_copies)[source]

Copy value

Parameters:
  • value (list of Model) – value
  • objects_and_copies (dict) – dictionary that maps objects to their copies
Returns:

copy of value

Return type:

list of Model

deserialize(values, objects, decoded=None)[source]

Deserialize value

Parameters:
  • values (object) – String representation
  • objects (dict) – dictionary of objects, grouped by model
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

tuple of cleaned value and cleaning error

Return type:

tuple of object, InvalidAttribute or None

get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
get_init_value(obj)[source]

Get initial value for attribute

Parameters:obj (Model) – object whose attribute is being initialized
Returns:initial value
Return type:object

Get initial related value for attribute

Parameters:obj (object) – object whose attribute is being initialized
Returns:initial value
Return type:value (object)
Raises:ValueError – if related property is not defined
merge(left, right, right_objs_in_left, left_objs_in_right)[source]

Merge an attribute of elements of two models

Parameters:
  • left (Model) – an element in a model to merge
  • right (Model) – an element in a second model to merge
  • right_objs_in_left (dict) – mapping from objects in right model to objects in left model
  • left_objs_in_right (dict) – mapping from objects in left model to objects in right model
Raises:

ValueError – if the attributes of the elements of the models are different

related_validate(obj, value)[source]

Determine if value is a valid value of the related attribute

Parameters:
  • obj (Model) – object being validated
  • value (list of Model) – value to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

serialize(value, encoded=None)[source]

Serialize related object

Parameters:
  • value (list of Model) – Python representation
  • encoded (dict, optional) – dictionary of objects that have already been encoded
Returns:

simple Python representation

Return type:

str

Update the values of the related attributes of the attribute

Parameters:
  • obj (object) – object whose attribute should be set
  • new_values (list) – value of the attribute
Returns:

value of the attribute

Return type:

list

Raises:

ValueError – if related property is not defined

set_value(obj, new_values)[source]

Get value of attribute of object

Parameters:
  • obj (Model) – object
  • new_values (list) – new attribute value
Returns:

new attribute value

Return type:

list

validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (list of Model) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.ManyToManyRelatedManager(object, attribute, related=True)[source]

Bases: obj_model.core.RelatedManager

Represent values and related values of related attributes

append(value, propagate=True)[source]

Add value to list

Parameters:
  • value (object) – value
  • propagate (bool, optional) – propagate change to related attribute
Returns:

self

Return type:

RelatedManager

cut(kind=None)[source]

Cut values and their children of kind kind into separate graphs.

If kind is None, children are defined to be the values of the related attributes defined in each class.

Parameters:kind (str, optional) – kind of children to include
Returns:cut values and their children
Return type:list of Model
remove(value, update_list=True, propagate=True)[source]

Remove value from list

Parameters:
  • value (object) – value
  • update_list (bool, optional) – update list
  • propagate (bool, optional) – propagate change to related attribute
Returns:

self

Return type:

RelatedManager

class obj_model.core.ManyToOneAttribute(related_class, related_name='', default=None, default_cleaned_value=None, related_default=[], none_value=None, min_related=0, min_related_rev=0, max_related_rev=inf, verbose_name='', verbose_related_name='', help='', related_manager=<class 'obj_model.core.ManyToOneRelatedManager'>)[source]

Bases: obj_model.core.RelatedAttribute

Represents a many-to-one relationship between two types of objects. This is analagous to a foreign key relationship in a database.

related_manager[source]

related manager

Type:type
copy_value(value, objects_and_copies)[source]

Copy value

Parameters:
  • value (Model) – value
  • objects_and_copies (dict) – dictionary that maps objects to their copies
Returns:

copy of value

Return type:

Model

deserialize(value, objects, decoded=None)[source]

Deserialize value

Parameters:
  • value (str) – String representation
  • objects (dict) – dictionary of objects, grouped by model
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

tuple of cleaned value and cleaning error

Return type:

tuple of object, InvalidAttribute or None

get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation

Get initial related value for attribute

Parameters:obj (object) – object whose attribute is being initialized
Returns:initial value
Return type:value (object)
Raises:ValueError – if related property is not defined
merge(left, right, right_objs_in_left, left_objs_in_right)[source]

Merge an attribute of elements of two models

Parameters:
  • left (Model) – an element in a model to merge
  • right (Model) – an element in a second model to merge
  • right_objs_in_left (dict) – mapping from objects in right model to objects in left model
  • left_objs_in_right (dict) – mapping from objects in left model to objects in right model
Raises:

ValueError – if the attributes of the elements of the models are different

related_validate(obj, value)[source]

Determine if value is a valid value of the related attribute

Parameters:
  • obj (Model) – object being validated
  • value (list of Model) – value to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

serialize(value, encoded=None)[source]

Serialize related object

Parameters:
  • value (Model) – Python representation
  • encoded (dict, optional) – dictionary of objects that have already been encoded
Returns:

simple Python representation

Return type:

str

Update the values of the related attributes of the attribute

Parameters:
  • obj (object) – object whose attribute should be set
  • new_values (list) – value of the attribute
Returns:

value of the attribute

Return type:

list

Raises:

ValueError – if related property is not defined

set_value(obj, new_value)[source]

Update the values of the related attributes of the attribute

Parameters:
  • obj (object) – object whose attribute should be set
  • new_value (Model) – new attribute value
Returns:

new attribute value

Return type:

Model

validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (Model) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.ManyToOneRelatedManager(object, attribute)[source]

Bases: obj_model.core.RelatedManager

Represent values of related attributes

append(value, propagate=True)[source]

Add value to list

Parameters:
  • value (object) – value
  • propagate (bool, optional) – propagate change to related attribute
Returns:

self

Return type:

RelatedManager

cut(kind=None)[source]

Cut values and their children of kind kind into separate graphs.

If kind is None, children are defined to be the values of the related attributes defined in each class.

Parameters:kind (str, optional) – kind of children to include
Returns:cut values and their children
Return type:list of Model
remove(value, update_list=True, propagate=True)[source]

Remove value from list

Parameters:
  • value (object) – value
  • propagate (bool, optional) – propagate change to related attribute
Returns:

self

Return type:

RelatedManager

class obj_model.core.Model(**kwargs)[source]

Bases: object

Base object model

_source[source]

file location, worksheet, column, and row where the object was defined

Type:ModelSource
objects[source]

a Manager that supports searching for Model instances

Type:Manager
DEFAULT_INDENT = 3[source]
DEFAULT_MAX_DEPTH = 2[source]
class Meta[source]

Bases: object

Meta data for Model

attributes[source]

attributes

Type:collections.OrderedDict of str, Attribute
related_attributes[source]

attributes declared in related objects

Type:collections.OrderedDict of str, `Attribute
local_attributes[source]

dictionary that maps the names of all local attributes to their instances, including attributes defined in this class and attributes defined in related classes

Type:collections.OrderedDict of str, Attribute
primary_attribute[source]

attribute with primary = True

Type:Attribute
unique_together[source]

controls what tuples of attribute values must be unique

Type:tuple of tuple’s of attribute names
indexed_attrs_tuples[source]

tuples of attributes on which instances of this Model will be indexed by the Model’s Manager

Type:tuple of tuple’s of attribute names
attribute_order[source]

tuple of attribute names, in the order in which they should be displayed

Type:tuple of str
verbose_name[source]

verbose name to refer to an instance of the model

Type:str
verbose_name_plural[source]

plural verbose name for multiple instances of the model

Type:str
help[source]

description of the model (e.g., to print in the table of contents in Excel)

Type:str
tabular_orientation[source]

orientation of model objects in table (e.g. Excel)

Type:TabularOrientation
frozen_columns[source]

number of Excel columns to freeze

Type:int
inheritance[source]

tuple of all superclasses

Type:tuple of class
ordering[source]

controls the order in which objects should be printed when serialized

Type:tuple of attribute names
children[source]

dictionary that maps types of children to names of attributes which compose each type of children

Type:dict that maps str to tuple of str
merge[source]

type of merging operation

Type:ModelMerge
attribute_order = ()[source]
attributes = None[source]
children = {}[source]
frozen_columns = 1[source]
help = ''[source]
indexed_attrs_tuples = ()[source]
inheritance = None[source]
merge = 1[source]
ordering = None[source]
primary_attribute = None[source]
related_attributes = None[source]
tabular_orientation = 1[source]
unique_together = ()[source]
verbose_name = ''[source]
verbose_name_plural = ''[source]
__enter__()[source]

Enter context

__exit__(type, value, traceback)[source]

Exit context

__setattr__(attr_name, value, propagate=True)[source]

Set attribute and validate any unique attribute constraints

Parameters:
  • attr_name (str) – attribute name
  • value (object) – value
  • propagate (bool, optional) – propagate change through attribute set_value and set_related_value
__str__()[source]

Get the string representation of an object

Returns:string representation of object
Return type:str
classmethod are_attr_paths_equal(attr_path, other_attr_path)[source]

Determine if two attribute paths are semantically equal

Parameters:
  • attr_path (list of list of object) – the path to an attribute or nested attribute of a model
  • other_attr_path (list of list of object) – the path to another attribute or nested attribute of a model
Returns:

True if the paths are semantically equal

Return type:

bool

Determine if the immediate related attributes of the class can be serialized

Returns:True if the related attributes can be serialized
Return type:bool
clean()[source]

Clean all of this Model’s attributes

Returns:
None if the object is valid,
otherwise return a list of errors as an instance of InvalidObject
Return type:InvalidObject or None
copy()[source]

Create a copy

Returns:model copy
Return type:Model
cut(kind=None)[source]

Cut the object and its children from the rest of the object graph.

If kind is None, children are defined to be the values of the related attributes defined in each class.

Parameters:kind (str, optional) – kind of children to get
Returns:same object, but cut from the rest of the object graph
Return type:Model
cut_relations(objs)[source]

Cut relations to objects not in objs.

Parameters:objs (set of Model) – objects to retain relations to
classmethod deserialize(value, objects)[source]

Deserialize value

Parameters:
  • value (str) – String representation
  • objects (dict) – dictionary of objects, grouped by model
Returns:

tuple of cleaned value

and cleaning error

Return type:

tuple of object, InvalidAttribute or None

difference(other, tol=0.0)[source]

Get the semantic difference between two models

Parameters:
  • other (Model) – other Model
  • tol (float, optional) – equality tolerance
Returns:

difference message

Return type:

str

classmethod from_dict(json, decode_primary_objects=True, primary_objects=None, decoded=None)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of an object that is compatible with JSON and YAML, including references to objects through __id keys.

Parameters:
  • json (dict) – simple Python representation of the object
  • decode_primary_objects (bool, optional) – if True, decode primary classes otherwise just look up objects by their IDs
  • primary_objects (list, optional) – list of instances of primary classes (i.e. non-line classes)
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

decoded object

Return type:

Model

gen_merge_map(other)[source]

Create a dictionary that maps instances of objects in another model to objects in a model

Parameters:other (Model) – other model
Returns:
dictionary that maps instances of objects in another model to objects
in a model
list: list of instances of objects in another model which have no parallel
in the model
Return type:dict
gen_serialized_val_obj_map()[source]

Generate mappings from serialized values to objects

Returns:dictionary which maps types of models to dictionaries which serialized values to objects
Return type:dict
Raises:ValueError – if serialized values are not unique within each type

Optimally obtain all objects related to objects in objs

The set of all Model`s can be viewed as a graph whose nodes are :obj:`Model instances and whose edges are related connections. Because related edges are bi-directional, this graph is a set of strongly connected components and no edges connect the components.

The algorithm here finds all Model`s that are reachable from a set of instances in `O(n), where n is the size of the reachable set. This algorithm is optimal. It achieves this performance because obj.get_related() takes O(n(c)) where n(c) is the number of nodes in the component containing obj, and each component is only explored once because all of a component’s nodes are stored in found_objs when the component is first explored.

In addition, this method is deterministic because ordered dictionaries preserve insertion order.

Parameters:
  • objs (iterator of Model, optional) – some objects
  • forward (bool, optional) – if True, get all forward related objects
  • reverse (bool, optional) – if True, get all reverse related objects
Returns:

all objects in objs and all objects related to them, without any duplicates

Return type:

list of Model

classmethod get_attr_index(attr)[source]

Get the index of an attribute within Meta.attribute_order

Parameters:attr (Attribute) – attribute
Returns:index of attribute within Meta.attribute_order
Return type:int
classmethod get_attrs(type=None, forward=True, reverse=True)[source]

Get attributes of a type, optionally including attributes from related classes. By default, return all attributes.

Parameters:
  • type (type or tuple of type, optional) – type of attributes to get
  • forward (bool, optional) – if True, include attributes from class
  • reverse (bool, optional) – if True, include attributes from related classes
Returns:

dictionary of the names and instances

of matching attributes

Return type:

dict of str, Attribute

get_attrs_by_val(type=None, reverse=True, include=None, exclude=None)[source]

Get attributes whose type is type and values are in include and not exclude, optionally including attributes from related classes. By default, get all attributes.

Parameters:
  • type (type or tuple of type, optional) – type of attributes to get
  • reverse (bool, optional) – if True, include attributes from related classes
  • include (list, optional) – list of values to filter for
  • exclude (list, optional) – list of values to filter out
Returns:

dictionary of the names and instances

of matching attributes

Return type:

dict of str, Attribute

get_children(kind=None, _Model__type=None, recursive=True, **kwargs)[source]

Get a kind of children.

If kind is None, children are defined to be the values of the related attributes defined in each class.

Parameters:
  • kind (str, optional) – kind of children to get
  • __type (types.TypeType or tuple of types.TypeType) – subclass(es) of Model
  • recursive (bool, optional) – if True, get children recursively

:param kwargs (dict of str: object): dictionary of attribute name/value pairs

Returns:children
Return type:list of Model
get_empty_literal_attrs()[source]

Get empty (None, ‘’, or NaN) literal attributes

Returns:
dictionary of the names and instances
of empty literal attributes
Return type:dict of str, Attribute

Get empty (None or []) related attributes

Parameters:reverse (bool, optional) – if True, include attributes from related classes
Returns:
dictionary of the names and instances
of empty related attributes
Return type:dict of str, Attribute
get_immediate_children(kind=None, _Model__type=None, **kwargs)[source]

Get a kind of immediate children

If kind is None, children are defined to be the values of the related attributes defined in each class.

Parameters:
  • kind (str, optional) – kind of children to get
  • __type (types.TypeType or tuple of types.TypeType) – subclass(es) of Model

:param kwargs (dict of str: object): dictionary of attribute name/value pairs

Returns:immediate children
Return type:list of Model
classmethod get_literal_attrs()[source]

Get literal attributes

Returns:
dictionary of the names and instances
of literal attributes
Return type:dict of str, Attribute
classmethod get_manager()[source]

Get the manager for the model

Returns:manager
Return type:Manager
classmethod get_nested_attr(attr_path)[source]

Get the value of an attribute or a nested attribute of a model

Parameters:attr_path (list of list of str) – the path to an attribute or nested attribute of a model
Returns:nested attribute
Return type:Attribute
get_nested_attr_val(attr_path)[source]

Get the value of an attribute or a nested attribute of a model

Parameters:attr_path (list of list of object) – the path to an attribute or nested attribute of a model
Returns:value of the attribute or nested attribute
Return type:Object
get_non_empty_literal_attrs()[source]

Get non-empty (None, ‘’, or NaN) literal attributes

Returns:
dictionary of the names and instances
of non-empty literal attributes
Return type:dict of str, Attribute

Get non-empty (None or []) related attributes

Parameters:reverse (bool, optional) – if True, include attributes from related classes
Returns:
dictionary of the names and instances
of non-empty related attributes
Return type:dict of str, Attribute
get_primary_attribute()[source]

Get value of primary attribute

Returns:value of primary attribute
Return type:object

Get all related objects reachable from self

Parameters:
  • forward (bool, optional) – if True, get all forward related objects
  • reverse (bool, optional) – if True, get all reverse related objects
Returns:

related objects, without any duplicates

Return type:

list of Model

Get related attributes

Parameters:reverse (bool, optional) – if True, include attributes from related classes
Returns:
dictionary of the names and instances
of related attributes
Return type:dict of str, Attribute
classmethod get_sort_key(object, attr_name)[source]

Get sort key for Model instance object based on cls.Meta.ordering

Parameters:
  • object (Model) – Model instance
  • attr_name (str) – attribute name
Returns:

sort key for object

Return type:

object

get_source(attr_name)[source]

Get file location of attribute with name attr_name

Provide the type, filename, worksheet, row, and column of attr_name. Row and column use 1-based counting. Column is provided in Excel format if the file was a spreadsheet.

Parameters:attr_name (str) – attribute name
Returns:tuple of (type, basename, worksheet, row, column)
Raises:ValueError if the location of attr_name is unknown
has_attr_vals(_Model__type=None, **kwargs)[source]

Check if the type and values of the attributes of an object match a set of conditions

Parameters:__type (types.TypeType or tuple of types.TypeType) – subclass(es) of Model
:param **kwargs (dict of str: object): dictionary of attribute name/value pairs to find matching
object or create new object
Returns:
True if the object is an instance of __type and the
the values of the attributes of the object match kwargs
Return type:bool
is_equal(other, tol=0.0)[source]

Determine whether two models are semantically equal

Parameters:
  • other (Model) – object to compare
  • tol (float, optional) – equality tolerance
Returns:

True if objects are semantically equal, else False

Return type:

bool

classmethod is_serializable()[source]

Determine if the class (and its related classes) can be serialized

Raises:boolTrue if the class can be serialized
merge(other, normalize=True, validate=True)[source]

Merge another model into a model

Parameters:
  • other (Model) – other model
  • normalize (bool, optional) – if True, normalize models and merged model
  • validate (bool, optional) – if True, validate models and merged model
merge_attrs(other, other_objs_in_self, self_objs_in_other)[source]

Merge attributes of two objects

Parameters:
  • other (Model) – other model
  • other_objs_in_self (dict) – dictionary that maps instances of objects in another model to objects in a model
  • self_objs_in_other (dict) – dictionary that maps instances of objects in a model to objects in another model
normalize()[source]

Normalize an object into a canonical form. Specifically, this method sorts the RelatedManagers into a canonical order because their order has no semantic meaning. Importantly, this canonical form is reproducible. Thus, this canonical form facilitates reproducible computations on top of Model objects.

pformat(max_depth=2, indent=3)[source]

Return a human-readable string representation of this Model.

Follows the graph of related Model’s up to a depth of max_depth. Model’s at depth max_depth+1 are represented by ‘<class name>: …’, while deeper Model’s are not traversed or printed. Re-encountered Model’s do not get printed, and are indicated by ‘<attribute name>: –’. Attributes that are related or iterable are indented.

For example, we have:

Model1_classname:       # Each model starts with its classname, followed by a list of
    attr1: value1           # attribute names & values.
    attr2: value2
    attr3:                  # Reference attributes can point to other Models; we indent these under the attribute name
        Model2_classname:   # Reference attribute attr3 contains Model2;
            ...                 # its attributes follow.
    attr4:
        Model3_classname:   # An iteration over reference attributes is a list at constant indentation:
            ...
    attr5:
        Model2_classname: --    # Traversing the Model network may re-encounter a Model; they're listed with '--'
    attr6:
        Model5_classname:
            attr7:
                Model5_classname: ...   # The size of the output is controlled with max_depth;
                                        # models encountered at depth = max_depth+1 are shown with '...'
Parameters:
  • max_depth (int, optional) – the maximum depth to which related Model’s should be printed
  • indent (int, optional) – number of spaces to indent
Returns:

obj:str: readable string representation of this Model

pprint(stream=None, max_depth=2, indent=3)[source]
serialize()[source]

Get value of primary attribute

Returns:value of primary attribute
Return type:str
set_nested_attr_val(attr_path, value)[source]

Set the value of an attribute or a nested attribute of a model

Parameters:
  • attr_path (list of list of object) – the path to an attribute or nested attribute of a model
  • value (object) – new value
Returns:

the same model with the value of an attribute

modified

Return type:

Model

set_source(path_name, sheet_name, attribute_seq, row)[source]

Set metadata about source of the file, worksheet, columns, and row where the object was defined

Parameters:
  • path_name (str) – pathname of source file for object
  • sheet_name (str) – name of spreadsheet containing source data for object
  • attribute_seq (list) – sequence of attribute names in source file; blank values indicate attributes that were ignored
  • row (int) – row number of object in its source file
classmethod sort(objects)[source]

Sort list of Model objects

Parameters:objects (list of Model) – list of objects
Returns:sorted list of objects
Return type:list of Model
to_dict(encode_primary_objects=True, max_depth=inf, encoded=None)[source]

Encode an object using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML. Use __id keys to avoid infinite recursion by encoding each object once and referring to objects by their __id for each repeated reference.

Parameters:
  • encode_primary_objects (bool, optional) – if True, encode primary classes otherwise just encode their IDs
  • max_depth (int, optional) – maximum depth to serialize
  • encoded (dict, optional) – objects that have already been encoded and their assigned JSON identifiers
Returns:

simple Python representation of the object

Return type:

dict

validate()[source]

Determine if the object is valid

Returns:
None if the object is valid,
otherwise return a list of errors as an instance of InvalidObject
Return type:InvalidObject or None

Validate attribute values

Raises:ValueError – if related attributes are not valid (e.g. if a class that is the subject of a relationship does not have a primary attribute)
classmethod validate_unique(objects)[source]

Validate attribute uniqueness

Parameters:objects (list of Model) – list of objects
Returns:list of invalid attributes and their errors
Return type:InvalidModel or None
class obj_model.core.ModelMerge[source]

Bases: int, enum.Enum

Types of model merging operations

append = 2[source]
join = 1[source]
class obj_model.core.ModelMeta[source]

Bases: type

static __new__(metacls, name, bases, namespace)[source]
Parameters:
  • metacls (Model) – Model, or a subclass of Model
  • name (str) – Model class name
  • bases (tuple) – tuple of superclasses
  • namespace (dict) – namespace of Model class definition
Returns:

a new instance of Model, or a subclass of Model

Return type:

Model

create_model_manager()[source]

Create a Manager for this Model

The Manager is accessed via a Model’s objects attribute

cls[source]

the Model class which is being managed

Type:type
init_attribute_order()[source]

Initialize the order in which the attributes should be printed across Excel columns

init_attributes()[source]

Initialize attributes

init_inheritance()[source]

Create tuple of this model and superclasses which are subclasses of Model

init_ordering()[source]

Initialize how to sort objects

init_primary_attribute()[source]

Initialize the primary attribute of a model

Initialize related attributes

init_verbose_names()[source]

Initialize the singular and plural verbose names of a model

normalize_attr_tuples(attribute)[source]

Normalize a tuple of tuples of attribute names

Parameters:attribute (str) – the name of the attribute to validate and normalize
static normalize_tuple_of_tuples_of_attribute_names(tuple_of_tuples_of_attribute_names)[source]

Normalize a tuple of tuples of attribute names by sorting each member tuple

Enables simple indexing and searching of tuples

Parameters:tuple_of_tuples_of_attribute_names (tuple) – a tuple of tuples of attribute names
Returns:a tuple of sorted tuples of attribute names
Return type:tuple
classmethod validate_attr_tuples(name, bases, namespace, meta_attribute_name)[source]

Validate a tuple of tuples of attribute names

Parameters:meta_attribute_name (str) – the name of the attribute to validate and normalize
Raises:ValueError – if attributes are not valid
classmethod validate_attribute_inheritance(name, bases, namespace)[source]

Check attribute inheritance

Raises:ValueError – if subclass overrides a superclass attribute (instance of Attribute) with an incompatible attribute (i.e. an attribute that is not a subclass of the class of the super class’ attribute)
classmethod validate_attributes(name, bases, namespace)[source]

Validate attribute values

Raises:ValueError – if attributes are not valid
classmethod validate_meta(name, bases, namespace)[source]
classmethod validate_primary_attribute(name, bases, namespace)[source]

Check the attributes

Raises:ValueError – if there are multiple primary attributes

Check the related attributes

Raises:ValueError – if an OneToManyAttribute or ManyToOneAttribute has a related_name equal to its name
class obj_model.core.ModelSource(path_name, sheet_name, attribute_seq, row)[source]

Bases: object

Represents the file, sheet, columns, and row where a Model instance was defined

path_name[source]

pathname of source file for object

Type:str
sheet_name[source]

name of spreadsheet containing source data for object

Type:str
attribute_seq[source]

sequence of attribute names in source file; blank values indicate attributes that were ignored

Type:list
row[source]

row number of object in its source file

Type:int
class obj_model.core.NumericAttribute(init_value=None, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.core.LiteralAttribute

Base class for numeric literal attributes (float, integer)

exception obj_model.core.ObjModelWarning[source]

Bases: UserWarning

obj_model warning

class obj_model.core.OneToManyAttribute(related_class, related_name='', default=[], default_cleaned_value=[], related_default=None, none_value=<class 'list'>, min_related=0, max_related=inf, min_related_rev=0, verbose_name='', verbose_related_name='', help='', related_manager=<class 'obj_model.core.OneToManyRelatedManager'>)[source]

Bases: obj_model.core.RelatedAttribute

Represents a one-to-many relationship between two types of objects. This is analagous to a foreign key relationship in a database.

related_manager[source]

related manager

Type:type
copy_value(value, objects_and_copies)[source]

Copy value

Parameters:
  • value (list of Model) – value
  • objects_and_copies (dict) – dictionary that maps objects to their copies
Returns:

copy of value

Return type:

list of Model

deserialize(values, objects, decoded=None)[source]

Deserialize value

Parameters:
  • values (object) – String representation
  • objects (dict) – dictionary of objects, grouped by model
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

tuple of cleaned value and cleaning error

Return type:

tuple of object, InvalidAttribute or None

get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
get_init_value(obj)[source]

Get initial value for attribute

Parameters:obj (Model) – object whose attribute is being initialized
Returns:initial value
Return type:object
merge(left, right, right_objs_in_left, left_objs_in_right)[source]

Merge an attribute of elements of two models

Parameters:
  • left (Model) – an element in a model to merge
  • right (Model) – an element in a second model to merge
  • right_objs_in_left (dict) – mapping from objects in right model to objects in left model
  • left_objs_in_right (dict) – mapping from objects in left model to objects in right model
Raises:

ValueError – if the attributes of the elements of the models are different

related_validate(obj, value)[source]

Determine if value is a valid value of the related attribute

Parameters:
  • obj (Model) – object being validated
  • value (Model) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

serialize(value, encoded=None)[source]

Serialize related object

Parameters:
  • value (list of Model) – Python representation
  • encoded (dict, optional) – dictionary of objects that have already been encoded
Returns:

simple Python representation

Return type:

str

Update the values of the related attributes of the attribute

Parameters:
  • obj (object) – object whose attribute should be set
  • new_value (Model) – new attribute value
Returns:

new attribute value

Return type:

Model

Raises:

ValueError – if related property is not defined

set_value(obj, new_values)[source]

Update the values of the related attributes of the attribute

Parameters:
  • obj (object) – object whose attribute should be set
  • new_values (list) – value of the attribute
Returns:

value of the attribute

Return type:

list

validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (list of Model) – value to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.OneToManyRelatedManager(object, attribute)[source]

Bases: obj_model.core.RelatedManager

Represent values of related attributes

append(value, propagate=True)[source]

Add value to list

Parameters:
  • value (object) – value
  • propagate (bool, optional) – propagate change to related attribute
Returns:

self

Return type:

RelatedManager

cut(kind=None)[source]

Cut values and their children of kind kind into separate graphs.

If kind is None, children are defined to be the values of the related attributes defined in each class.

Parameters:kind (str, optional) – kind of children to include
Returns:cut values and their children
Return type:list of Model
remove(value, update_list=True, propagate=True)[source]

Remove value from list

Parameters:
  • value (object) – value
  • propagate (bool, optional) – propagate change to related attribute
Returns:

self

Return type:

RelatedManager

class obj_model.core.OneToOneAttribute(related_class, related_name='', default=None, default_cleaned_value=None, related_default=None, none_value=None, min_related=0, min_related_rev=0, verbose_name='', verbose_related_name='', help='')[source]

Bases: obj_model.core.RelatedAttribute

Represents a one-to-one relationship between two types of objects.

copy_value(value, objects_and_copies)[source]

Copy value

Parameters:
  • value (Model) – value
  • objects_and_copies (dict) – dictionary that maps objects to their copies
Returns:

copy of value

Return type:

Model

deserialize(value, objects, decoded=None)[source]

Deserialize value

Parameters:
  • value (str) – String representation
  • objects (dict) – dictionary of objects, grouped by model
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

tuple of cleaned value and cleaning error

Return type:

tuple of object, InvalidAttribute or None

get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
merge(left, right, right_objs_in_left, left_objs_in_right)[source]

Merge an attribute of elements of two models

Parameters:
  • left (Model) – an element in a model to merge
  • right (Model) – an element in a second model to merge
  • right_objs_in_left (dict) – mapping from objects in right model to objects in left model
  • left_objs_in_right (dict) – mapping from objects in left model to objects in right model
Raises:

ValueError – if the attributes of the elements of the models are different

related_validate(obj, value)[source]

Determine if value is a valid value of the related attribute

Parameters:
  • obj (Model) – object being validated
  • value (list of Model) – value to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

serialize(value, encoded=None)[source]

Serialize related object

Parameters:
  • value (Model) – Python representation
  • encoded (dict, optional) – dictionary of objects that have already been encoded
Returns:

simple Python representation

Return type:

str

Update the values of the related attributes of the attribute

Parameters:
  • obj (object) – object whose attribute should be set
  • new_value (Model) – value of the attribute
Returns:

value of the attribute

Return type:

Model

Raises:

ValueError – if related property is not defined or the attribute of new_value is not None

set_value(obj, new_value)[source]

Update the values of the related attributes of the attribute

Parameters:
  • obj (object) – object whose attribute should be set
  • new_value (Model) – new attribute value
Returns:

new attribute value

Return type:

Model

Raises:

ValueError – if related attribute of new_value is not None

validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (Model) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.PositiveFloatAttribute(max=nan, nan=True, default=nan, default_cleaned_value=nan, none_value=nan, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.FloatAttribute

Positive float attribute

get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.PositiveIntegerAttribute(max=None, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.IntegerAttribute

Positive integer attribute

get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.RegexAttribute(pattern, flags=0, min_length=0, max_length=None, default='', default_cleaned_value='', none_value='', verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.StringAttribute

Regular expression attribute

pattern[source]

regular expression pattern

Type:str
flags[source]

regular expression flags

Type:int
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.RelatedAttribute(related_class, related_name='', init_value=None, default=None, default_cleaned_value=None, none_value=None, related_init_value=None, related_default=None, min_related=0, max_related=inf, min_related_rev=0, max_related_rev=inf, verbose_name='', verbose_related_name='', help='')[source]

Bases: obj_model.core.Attribute

Attribute which represents a relationship with other `Model`(s)

primary_class[source]

the type of the class that this related attribute references

Type:class
related_class[source]

the type of the class that contains a related attribute

Type:class
related_name[source]

name of related attribute on related_class

Type:str

verbose related name

Type:str
related_init_value[source]

initial value of related attribute

Type:object
related_default[source]

default value of related attribute

Type:object

minimum number of related objects in the forward direction

Type:int

maximum number of related objects in the forward direction

Type:int

minimum number of related objects in the reverse direction

Type:int

maximum number of related objects in the reverse direction

Type:int
deserialize(value, objects, decoded=None)[source]

Deserialize value

Parameters:
  • values (object) – String representation
  • objects (dict) – dictionary of objects, grouped by model
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

tuple of cleaned value and cleaning error

Return type:

tuple of object, InvalidAttribute or None

from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (object) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:object

Get default related value for attribute

Parameters:obj (Model) – object whose attribute is being initialized
Returns:initial value
Return type:object
Raises:ValueError – if related property is not defined

Get initial related value for attribute

Parameters:obj (object) – object whose attribute is being initialized
Returns:initial value
Return type:value (object)
Raises:ValueError – if related property is not defined
related_validate(obj, value)[source]

Determine if value is a valid value of the related attribute

Parameters:
  • obj (Model) – object to validate
  • value (list) – value to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

serialize(value, encoded=None)[source]

Serialize related object

Parameters:
  • value (Model) – Python representation
  • encoded (dict, optional) – dictionary of objects that have already been encoded
Returns:

simple Python representation

Return type:

str

Update the values of the related attributes of the attribute

Parameters:
  • obj (object) – object whose attribute should be set
  • new_values (object) – value of the attribute
Returns:

value of the attribute

Return type:

object

to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (object) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:object
class obj_model.core.RelatedManager(object, attribute, related=True)[source]

Bases: list

Represent values and related values of related attributes

object[source]

model instance

Type:Model
attribute[source]

attribute

Type:Attribute
related[source]

is related attribute

Type:bool
add(value, **kwargs)[source]

Add value to list

Parameters:value (object) – value
Returns:self
Return type:RelatedManager
append(value, **kwargs)[source]

Add value to list

Parameters:value (object) – value
Returns:self
Return type:RelatedManager
clear()[source]

Remove all elements from list

Returns:self
Return type:RelatedManager
create(_RelatedManager__type=None, **kwargs)[source]

Create instance of primary class and add to list

Parameters:__type (types.TypeType or tuple of types.TypeType) – subclass(es) of Model

:param kwargs (dict of str: object): dictionary of attribute name/value pairs

Returns:created object
Return type:Model
Raises:ValueError – if keyword argument is not an attribute of the class
difference_update(values)[source]

Retain only values of list not in values

Parameters:values (list) – values to difference with list
Returns:self
Return type:RelatedManager
discard(value)[source]

Remove value from list if value in list

Parameters:value (object) – value
Returns:self
Return type:RelatedManager
extend(values)[source]

Add values to list

Parameters:values (list) – values to add to list
Returns:self
Return type:RelatedManager
get(_RelatedManager__type=None, **kwargs)[source]

Get related objects by attribute/value pairs and, optionally, only return matches that are also instances of Model subclass __type.

Parameters:__type (types.TypeType or tuple of types.TypeType) – subclass(es) of Model
:param **kwargs (dict of str: object): dictionary of attribute name/value pairs to find matching
objects
Returns:matching instances of Model
Return type:list of Model
get_one(_RelatedManager__type=None, **kwargs)[source]

Get a related object by attribute/value pairs; report an error if multiple objects match and, optionally, only return matches that are also instances of Model subclass __type.

Parameters:__type (types.TypeType or tuple of types.TypeType) – subclass(es) of Model
:param **kwargs (dict of str: object): dictionary of attribute name/value pairs to find matching
objects
Returns:matching instance of Model, or None if no matching instance
Return type:Model or None
Raises:ValueError – if multiple matching objects
get_or_create(_RelatedManager__type=None, **kwargs)[source]

Get or create a related object by attribute/value pairs. Optionally, only get or create instances of Model subclass __type.

Parameters:__type (types.TypeType or tuple of types.TypeType) – subclass(es) of Model
:param **kwargs (dict of str: object): dictionary of attribute name/value pairs to find matching
object or create new object
Returns:existing or new object
Return type:Model
index(*args, **kwargs)[source]

Get related object index by attribute/value pairs

Parameters:
  • *args (list of Model) – object to find
  • **kwargs (dict of str, object) – dictionary of attribute name/value pairs to find matching objects
Returns:

index of matching object

Return type:

int

Raises:

ValueError – if no argument or keyword argument is provided, if argument and keyword arguments are both provided, if multiple arguments are provided, if the keyword attribute/value pairs match no object, or if the keyword attribute/value pairs match multiple objects

intersection_update(values)[source]

Retain only intersection of list and values

Parameters:values (list) – values to intersect with list
Returns:self
Return type:RelatedManager
pop(i=-1)[source]

Remove an arbitrary element from the list

Parameters:i (int, optional) – index of element to remove
Returns:removed element
Return type:object
symmetric_difference_update(values)[source]

Retain values in only one of list and values

Parameters:values (list) – values to difference with list
Returns:self
Return type:RelatedManager
update(values)[source]

Add values to list

Parameters:values (list) – values to add to list
Returns:self
Return type:RelatedManager
exception obj_model.core.SchemaWarning[source]

Bases: obj_model.core.ObjModelWarning

Schema warning

class obj_model.core.SlugAttribute(verbose_name='', help=None, primary=True, unique=True)[source]

Bases: obj_model.core.RegexAttribute

Slug attribute to be used for string IDs

class obj_model.core.StringAttribute(min_length=0, max_length=255, default='', default_cleaned_value='', none_value='', verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.core.LiteralAttribute

String attribute

default[source]

default value

Type:str
default_cleaned_value[source]

value to replace None values with during cleaning

Type:str
min_length[source]

minimum length

Type:int
max_length[source]

maximum length

Type:int
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:tuple of cleaned value and cleaning error
Return type:tuple of str, InvalidAttribute or None
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize string

Parameters:value (str) – Python representation
Returns:simple Python representation
Return type:str
validate(obj, value)[source]

Determine if value is a valid value for this StringAttribute

Parameters:
  • obj (Model) – class being validated
  • value (object) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.TabularOrientation[source]

Bases: enum.Enum

Describes a table’s orientation

  • row: the first row contains attribute names; subsequents rows store objects
  • column: the first column contains attribute names; subsequents columns store objects
  • cell: a cell contains a table, as a comma-separated list for example
  • multiple_cells: multiple cells within a row or column
cell = 3[source]
column = 2[source]
multiple_cells = 4[source]
row = 1[source]
class obj_model.core.TimeAttribute(none=True, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.LiteralAttribute

Time attribute

none[source]

if False, the attribute is invalid if its value is None

Type:bool
default[source]

default time

Type:time
default_cleaned_value[source]

value to replace None values with during cleaning, or function which computes the value to replace None values

Type:time
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:tuple of cleaned value and cleaning error
Return type:tuple of time, InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (str) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:time
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize string

Parameters:value (time) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (time) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:str
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (time) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of InvalidAttribute

Return type:

InvalidAttribute or None

class obj_model.core.UrlAttribute(verbose_name='', help='Enter a valid URL', primary=False, unique=False)[source]

Bases: obj_model.core.RegexAttribute

URL attribute to be used for URLs

class obj_model.core.Validator[source]

Bases: object

Engine to validate sets of objects

clean(objects)[source]

Clean a list of objects and return their errors

Parameters:object (list of Model) – list of objects
Returns:list of invalid objects/models and their errors
Return type:InvalidObjectSet or None
run(objects, get_related=False)[source]

Validate a list of objects and return their errors

Parameters:
  • objects (Model or list of Model) – object or list of objects
  • get_related (bool, optional) – if true, get all related objects
Returns:

list of invalid objects/models and their errors

Return type:

InvalidObjectSet or None

validate(objects)[source]

Validate a list of objects and return their errors

Parameters:object (list of Model) – list of Model instances
Returns:list of invalid objects/models and their errors
Return type:InvalidObjectSet or None
obj_model.core.excel_col_name(col)[source]

Convert column number to an Excel-style string.

From http://stackoverflow.com/a/19169180/509882

Parameters:col (int) – column number (positive integer)
Returns:alphabetic column name
Return type:str
Raises:ValueError – if col is not positive
obj_model.core.get_model(name, module=None)[source]

Get first Model with name name

Parameters:
  • name (str) – name
  • module (Module, optional) – module
Returns:

model class

Return type:

class

obj_model.core.get_models(module=None, inline=True)[source]

Get models

Parameters:
  • module (module, optional) – module
  • inline (bool, optional) – if true, return inline models
Returns:

list of model classes

Return type:

list of class

3.1.6. obj_model.expression module

Utilities for processing mathematical expressions used by obj_model models

Author:Arthur Goldberg <Arthur.Goldberg@mssm.edu>
Author:Jonathan Karr <karr@mssm.edu>
Date:2018-12-19
Copyright:2016-2018, Karr Lab
License:MIT
class obj_model.expression.Expression[source]

Bases: object

Generic methods for mathematical expressions

_parsed_expression[source]

parsed expression

Type:ParsedExpression
class Meta[source]

Bases: object

Meta data for subclasses of Expression

expression_term_models[source]

names of classes which can appear as terms in the expression

Type:tuple of str
expression_valid_functions[source]

Python functions which can appear in the expression

Type:tuple of types.FunctionType
expression_is_linear[source]

if True, validate that the expression is linear

Type:bool
expression_type[source]

type of the expression

Type:type
expression_is_linear = False[source]
expression_term_models = ()[source]
expression_type = None[source]
expression_unit_registry = None[source]
expression_valid_functions = (<class 'float'>, <built-in function fabs>, <built-in function ceil>, <built-in function floor>, <built-in function round>, <built-in function exp>, <built-in function expm1>, <built-in function pow>, <built-in function sqrt>, <built-in function log>, <built-in function log1p>, <built-in function log10>, <built-in function log2>, <built-in function factorial>, <built-in function sin>, <built-in function cos>, <built-in function tan>, <built-in function acos>, <built-in function asin>, <built-in function atan>, <built-in function atan2>, <built-in function hypot>, <built-in function degrees>, <built-in function radians>, <built-in function min>, <built-in function max>)[source]
classmethod deserialize(model_cls, value, objects)[source]

Deserialize expression

Parameters:
  • model_cls (type) – expression class
  • value (str) – string representation of the mathematical expression, in a Python expression
  • objects (dict) – dictionary of objects which can be used in expression, grouped by model
Returns:

on error return (None, InvalidAttribute),

otherwise return (object in this class with instantiated _parsed_expression, None)

Return type:

tuple

static make_expression_obj(model_type, expression, objs)[source]

Make an expression object

Parameters:
  • model_type (type) – an Model that uses a mathemetical expression, like Function and Observable
  • expression (str) – the expression used by the model_type being created
  • objs (dict of dict) – all objects that are referenced in expression
Returns:

if successful, (Model, None) containing a new instance of

model_type’s expression helper class; otherwise, (None, InvalidAttribute) reporting the error

Return type:

tuple

classmethod make_obj(model, model_type, primary_attr, expression, objs, allow_invalid_objects=False)[source]

Make a model that contains an expression by using its expression helper class

For example, this uses FunctionExpression to make a Function.

Parameters:
  • model (Model) – an instance of Model which is the root model
  • model_type (type) – a subclass of Model that uses a mathemetical expression, like Function and Observable
  • primary_attr (object) – the primary attribute of the model_type being created
  • expression (str) – the expression used by the model_type being created
  • objs (dict of dict) – all objects that are referenced in expression
  • allow_invalid_objects (bool, optional) – if set, return object - not error - if the expression object does not validate
Returns:

a new instance of model_type, or,

if an error occurs, an InvalidAttribute reporting the error

Return type:

Model or InvalidAttribute

merge_attrs(other, other_objs_in_self, self_objs_in_other)[source]

Merge attributes of two objects

Parameters:
  • other (Model) – other model
  • other_objs_in_self (dict) – dictionary that maps instances of objects in another model to objects in a model
  • self_objs_in_other (dict) – dictionary that maps instances of objects in a model to objects in another model
serialize()[source]

Generate string representation

Returns:value of primary attribute
Return type:str
classmethod set_lin_coeffs(obj)[source]

Set the linear coefficients for the related objects

Parameters:obj (Model) – expression object
classmethod validate(model_obj, parent_obj)[source]

Determine whether an expression model is valid by eval’ing its deserialized expression

Parameters:
  • model_obj (Expression) – expression object
  • parent_obj (Model) – parent of expression object
Returns:

None if the object is valid,

otherwise return a list of errors in an InvalidObject instance

Return type:

InvalidObject or None

class obj_model.expression.ExpressionDynamicTermMeta[source]

Bases: obj_model.expression.ExpressionTermMeta

Meta data for subclasses with dynamic values that can appear in expressions

class obj_model.expression.ExpressionExpressionTermMeta[source]

Bases: obj_model.expression.ExpressionTermMeta

Meta data for subclasses with expressions that can appear in expressions

expression_term_model[source]

name of attribute which encodes the expression for the term

Type:str
expression_term_model = None[source]
class obj_model.expression.ExpressionManyToOneAttribute(related_class, related_name='', default=None, default_cleaned_value=None, related_default=[], none_value=None, min_related=0, min_related_rev=0, max_related_rev=inf, verbose_name='', verbose_related_name='', help='', related_manager=<class 'obj_model.core.ManyToOneRelatedManager'>)[source]

Bases: obj_model.core.ManyToOneAttribute

Expresion many-to-one attribute

deserialize(value, objects, decoded=None)[source]

Deserialize value

Parameters:
  • value (str) – String representation
  • objects (dict) – dictionary of objects, grouped by model
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

tuple of cleaned value and cleaning error

Return type:

tuple of object, InvalidAttribute or None

get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(expression, encoded=None)[source]

Serialize related object

Parameters:
  • expression (Expression) – the related Expression
  • encoded (dict, optional) – dictionary of objects that have already been encoded
Returns:

simple Python representation of the rate law expression

Return type:

str

class obj_model.expression.ExpressionOneToOneAttribute(related_class, related_name='', default=None, default_cleaned_value=None, related_default=None, none_value=None, min_related=0, min_related_rev=0, verbose_name='', verbose_related_name='', help='')[source]

Bases: obj_model.core.OneToOneAttribute

Expression one-to-one attribute

deserialize(value, objects, decoded=None)[source]

Deserialize value

Parameters:
  • value (str) – String representation
  • objects (dict) – dictionary of objects, grouped by model
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

tuple of cleaned value and cleaning error

Return type:

tuple of object, InvalidAttribute or None

get_excel_validation(sheet_models=None)[source]

Get Excel validation

Parameters:sheet_models (list of Model, optional) – models encoded as separate sheets
Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(expression, encoded=None)[source]

Serialize related object

Parameters:
  • expression (obj_model.Model) – the referenced Expression
  • encoded (dict, optional) – dictionary of objects that have already been encoded
Returns:

simple Python representation

Return type:

str

class obj_model.expression.ExpressionStaticTermMeta[source]

Bases: obj_model.expression.ExpressionTermMeta

Meta data for subclasses with static values that can appear in expressions

expression_term_value[source]

name of attribute which encodes the value of the term

Type:str
expression_term_value = 'value'[source]
class obj_model.expression.ExpressionTermMeta[source]

Bases: object

Meta data for subclasses that can appear in expressions

expression_term_token_pattern[source]

token pattern for the name of the term in expression

Type:tuple
expression_term_units[source]

name of attribute which describes the units of the expression term

Type:str
expression_term_token_pattern = (1,)[source]
expression_term_units = 'units'[source]
class obj_model.expression.IdMatch[source]

Bases: tuple

IdMatch(model_type, token_pattern, match_string): Matched token pattern used by tokenize

__getnewargs__()[source]

Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, model_type, token_pattern, match_string)[source]

Create new instance of IdMatch(model_type, token_pattern, match_string)

__repr__()[source]

Return a nicely formatted representation string

match_string[source]

The matched string

model_type[source]

The type of Model matched

token_pattern[source]

The token pattern used by the match

class obj_model.expression.LexMatch[source]

Bases: tuple

LexMatch(obj_model_tokens, num_py_tokens): result returned by a lexer method that matches a obj_model expression element

__getnewargs__()[source]

Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, obj_model_tokens, num_py_tokens)[source]

Create new instance of LexMatch(obj_model_tokens, num_py_tokens)

__repr__()[source]

Return a nicely formatted representation string

num_py_tokens[source]

Number of Python tokens consumed

obj_model_tokens[source]

List of ObjModelToken’s created

class obj_model.expression.LinearParsedExpressionValidator[source]

Bases: obj_model.expression.ParsedExpressionValidator

Verify whether a sequence of tokens (`ObjModelToken`s) describes a linear function of identifiers

In particular, a valid linear expression must have the structure:
  • (identifier | number ‘*’ identifier) ((‘+’ | ‘-‘) (identifier | number ‘*’ identifier))*
TRANSITIONS = [('need number or id', (<ObjModelTokenCodes.number: 3>, None), 'need * id'), ('need * id', (<ObjModelTokenCodes.op: 4>, '*'), 'need id'), ('need id', (<ObjModelTokenCodes.obj_id: 1>, None), 'need + | - | end'), ('need number or id', (<ObjModelTokenCodes.obj_id: 1>, None), 'need + | - | end'), ('need + | - | end', (<ObjModelTokenCodes.op: 4>, '+'), 'need number or id'), ('need + | - | end', (<ObjModelTokenCodes.op: 4>, '-'), 'need number or id'), ('need + | - | end', (None, None), 'end')][source]
class obj_model.expression.ObjModelToken[source]

Bases: tuple

ObjModelToken(code, token_string, model_type, model_id, model): ObjModelToken in a parsed obj_model expression

__getnewargs__()[source]

Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, code, token_string, model_type=None, model_id=None, model=None)[source]

Create new instance of ObjModelToken(code, token_string, model_type, model_id, model)

__repr__()[source]

Return a nicely formatted representation string

code[source]

ObjModelTokenCodes encoding

model[source]

When code is obj_id, the obj_model obj

model_id[source]

When code is obj_id, the obj_model obj’s id

model_type[source]

When code is obj_id, the obj_model obj’s type

token_string[source]

The token’s string

class obj_model.expression.ObjModelTokenCodes[source]

Bases: int, enum.Enum

ObjModelToken codes used in parsed expressions

math_func_id = 2[source]
number = 3[source]
obj_id = 1[source]
op = 4[source]
other = 5[source]
class obj_model.expression.ParsedExpression(model_cls, attr, expression, objs)[source]

Bases: object

An expression in a obj_model Model

These expressions are limited Python expressions with specific semantics:

  • They must be syntactically correct Python.
  • No Python keywords, strings, or tokens that do not belong in expressions are allowed.
  • All Python identifiers must be the primary attribute of an obj_model object or the name of a
    function in the math package. Objects in the model are provided in _objs, and the allowed subset of functions in math must be provided in an iterator in the expression_valid_functions attribute of the Meta class of a model whose whose expression is being processed.
  • Currently (July, 2018), identifiers may refer to `Species`s, `Parameter`s, `Observable`s, `Reaction`s,
    Observable’s and `DfbaObjReaction`s.
  • Cycles of references are illegal.
  • An identifier must unambiguously refer to exactly one related Model in a model.
  • Each Model that can be used in an expression must have an ID that is a simple Python identifier,
    or define expression_term_token_pattern as an attribute that describes the Model’s syntactic Python structure. See Species for an example.
  • Every expression must be computable at any time during a simulation. The evaluation of an expression
    always occurs at a precise simulation time, which is implied by the expression but not explicitly represented. E.g., a reference to a Species means its concentration at the time the expression is eval`ed. These are the meanings of references: * `Species: its current concentration * Parameter: its value, which is static * Observable: its current value, whose units depend on its definition * Reaction: its current flux * DfbaObjReaction: its current flux
  • The modeller is responsible for ensuring that units in expressions are internally consistent and appropriate
    for the expression’s use
Attributes:

model_cls (type): the Model which has an expression attr (str): the attribute name of the expression in model_cls expression (str): the expression defined in the obj_model Model _py_tokens (list of collections.namedtuple): a list of Python tokens generated by tokenize.tokenize() _objs (dict): dict of obj_model Models that might be referenced in expression; maps

model type to a dict mapping ids to Model instances
expression_term_models (set): obj_model Models that model_cls objects are allowed to use,
as indicated in model_cls.Meta.expression_term_models, intersected with _objs.keys() might be referenced in expression; maps

valid_functions (set): the union of all valid_functions attributes for _objs unit_registry (pint.UnitRegistry): unit registry related_objects (dict): models that are referenced in expression; maps model type to

dict that maps model id to model instance
lin_coeffs (dict): linear coefficients of models that are referenced in expression;
maps model type to dict that maps models to coefficients

errors (list of str): errors found when parsing an expression fails _obj_model_tokens (list of ObjModelToken): tokens obtained when an expression is successfully

tokenize`d; if empty, then this `ParsedExpression cannot use eval()

_compiled_expression (str): compiled expression that can be evaluated by eval _compiled_expression_with_units (str): compiled expression with units that can be evaluated by eval _compiled_namespace (dict): compiled namespace for evaluation by eval _compiled_namespace_with_units (dict): compiled namespace with units for evaluation by eval

FUNC_PATTERN = (1, 7)[source]
LEGAL_TOKENS = {1, 2, 7, 8, 9, 10, 12, 14, 15, 16, 17, 20, 21, 22, 23, 28, 29, 30, 35}[source]
LEGAL_TOKENS_NAMES = ('NUMBER', 'NAME', 'LSQB', 'RSQB', 'DOT', 'COMMA', 'DOUBLESTAR', 'MINUS', 'PLUS', 'SLASH', 'STAR', 'LPAR', 'RPAR', 'EQUAL', 'GREATER', 'GREATEREQUAL', 'LESS', 'LESSEQUAL', 'NOTEQUAL')[source]
MODEL_TYPE_DISAMBIG_PATTERN = (1, 23, 1)[source]
eval(values, with_units=False)[source]

Evaluate the expression

Approach:

  1. Compile the expression
  2. Replace references to used models in self._obj_model_tokens with values
  3. Join the elements of self._obj_model_tokens into a Python expression
  4. eval the Python expression
Parameters:
  • values (dict) – dictionary that maps model types to dictionaries that map model ids to values
  • with_units (bool, optional) – if True, include units
Returns:

the value of the expression

Return type:

float, int, or bool

Raises:

ParsedExpressionError – if the expression has not been compiled or the evaluation fails

get_str(obj_model_token_to_str, with_units=False, number_units=' * __dimensionless__')[source]

Generate string representation of expression, e.g. for evaluation by eval method

Parameters:
  • obj_model_token_to_str (callable) – method to get string representation of a token that represents an instance of Model.
  • with_units (bool, optional) – if True, include units
  • number_units (str, optional) – default units for numbers
Returns:

string representation of expression

Return type:

str

Raises:

ParsedExpressionError – if the expression is invalid

legal_token = 28[source]
legal_token_name = 'NOTEQUAL'[source]
recreate_whitespace(expr)[source]

Insert the whitespace in this object’s expression into an expression with the same token count

Used to migrate an expression to a different set of model type names.

Parameters:expr (str) – a syntactically correct Python expression
Returns:
expr with the whitespace in this instance’s expression inserted between
its Python tokens
Return type:str
Raises:ParsedExpressionError – if tokenizing expr raises an exception, or if expr doesn’t have the same number of Python tokens as self.expression
test_eval(values=1.0, with_units=False)[source]

Test evaluate the expression with the value of all used models equal to test_val.

This is used to validate this ParsedExpression, as well as for testing.

Parameters:
  • values (float or dict, optional) – value, dictionary that maps model types to values, or dictionary that maps model types to dictionaries that map model ids to values
  • with_units (bool, optional) – if True, evaluate units
Returns:

the value of the expression

Return type:

float, int, or bool

Raises:

ParsedExpressionError – if the expression evaluation fails

tokenize(case_fold_match=False)[source]

Tokenize a Python expression in self.expression

Parameters:case_fold_match (bool, optional) – if set, casefold identifiers before matching; identifier keys in self._objs must already be casefold’ed; default = False
Returns:of ObjModelToken`s * :obj:`dict: dict of Model instances used by this list, grouped by Model type * list of str: list of errors
Return type:
  • list
Raises:ParsedExpressionError – if model_cls does not have a Meta attribute
exception obj_model.expression.ParsedExpressionError(message=None)[source]

Bases: Exception

Exception raised for errors in ParsedExpression

message[source]

the exception’s message

Type:str
class obj_model.expression.ParsedExpressionValidator(start_state, accepting_state, transitions, empty_is_valid=False)[source]

Bases: object

Verify whether a sequence of ObjModelToken tokens

An ParsedExpressionValidator consists of two parts:

  • An optional method _validate_tokens that examines the content of individual tokens and returns (True, True) if they are all valid, or (False, error) otherwise. It can be overridden by subclasses.
  • A DFSMAcceptor which determines whether the tokens describe a particular pattern validate() combines these parts.
dfsm_acceptor[source]

the DFSM acceptor

Type:DFSMAcceptor
empty_is_valid[source]

if set, then an empty sequence of tokens is valid

Type:bool
validate(expression)[source]

Indicate whether tokens is valid

Parameters:expression (ParsedExpression) – parsed expression
Returns:(False, error) if tokens is valid, or (True, None) if it is
Return type:tuple

3.1.7. obj_model.io module

Reading/writing schema objects to/from files

  • Comma separated values (.csv)
  • Excel (.xlsx)
  • JavaScript Object Notation (.json)
  • Tab separated values (.tsv)
  • Yet Another Markup Language (.yaml, .yml)
Author:Jonathan Karr <karr@mssm.edu>
Author:Arthur Goldberg <Arthur.Goldberg@mssm.edu>
Date:2016-11-23
Copyright:2016, Karr Lab
License:MIT
exception obj_model.io.IoWarning[source]

Bases: obj_model.core.ObjModelWarning

IO warning

class obj_model.io.JsonReader[source]

Bases: obj_model.io.ReaderBase

Read model objects from a JSON or YAML file

run(path, models=None, ignore_missing_sheets=False, ignore_extra_sheets=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, group_objects_by_model=False, validate=True)[source]

Read model objects from file(s) and, optionally, validate them

Parameters:
  • path (str) – path to file(s)
  • models (types.TypeType or list of types.TypeType, optional) – type or list of type of objects to read
  • ignore_missing_sheets (bool, optional) – if False, report an error if a worksheet/ file is missing for one or more models
  • ignore_extra_sheets (bool, optional) – if True and all models are found, ignore other worksheets or files
  • ignore_sheet_order (bool, optional) – if True, do not require the sheets to be provided in the canonical order
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • ignore_missing_attributes (bool, optional) – if False, report an error if a worksheet/file doesn’t contain all of attributes in a model in models
  • ignore_extra_attributes (bool, optional) – if True, do not report errors if attributes in the data are not in the model
  • ignore_attribute_order (bool) – if True, do not require the attributes to be provided in the canonical order
  • group_objects_by_model (bool, optional) – if True, group decoded objects by their types
  • validate (bool, optional) – if True, validate the data
Returns:

model objects grouped by Model class

Return type:

dict

Raises:

ValueError – if the input format is not supported, model names are not unique, or the data is invalid

class obj_model.io.JsonWriter[source]

Bases: obj_model.io.WriterBase

Write model objects to a JSON or YAML file

run(path, objects, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, toc=False, extra_entries=0, data_repo_metadata=False, schema_package=None)[source]

Write a list of model classes to a JSON or YAML file

Parameters:
  • path (str) – path to write file(s)
  • objects (Model or list of Model) – object or list of objects
  • models (list of Model, optional) – models
  • get_related (bool, optional) – if True, write object and all related objects
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • validate (bool, optional) – if True, validate the data
  • title (str, optional) – title
  • description (str, optional) – description
  • keywords (str, optional) – keywords
  • version (str, optional) – version
  • language (str, optional) – language
  • creator (str, optional) – creator
  • toc (bool, optional) – if True, include additional worksheet with table of contents
  • extra_entries (int, optional) – additional entries to display
  • data_repo_metadata (bool, optional) – if True, try to write metadata information about the file’s Git repo; the repo must be current with origin, except for the file
  • schema_package (str, optional) – the package which defines the obj_model schema used by the file; if not None, try to write metadata information about the the schema’s Git repository: the repo must be current with origin
Raises:

ValueError – if model names are not unique or output format is not supported

class obj_model.io.Reader[source]

Bases: obj_model.io.ReaderBase

static get_reader(path)[source]

Get the IO class whose run() method can read the file(s) at path

Parameters:path (str) – path to write file(s)
Returns:reader class
Return type:type
Raises:ValueError – if extension is not supported
run(path, models=None, ignore_missing_sheets=False, ignore_extra_sheets=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, group_objects_by_model=False, validate=True)[source]

Read a list of model objects from file(s) and, optionally, validate them

Parameters:
  • path (str) – path to file(s)
  • models (types.TypeType or list of types.TypeType, optional) – type of object to read or list of types of objects to read
  • ignore_missing_sheets (bool, optional) – if False, report an error if a worksheet/ file is missing for one or more models
  • ignore_extra_sheets (bool, optional) – if True and all models are found, ignore other worksheets or files
  • ignore_sheet_order (bool, optional) – if True, do not require the sheets to be provided in the canonical order
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • ignore_missing_attributes (bool, optional) – if False, report an error if a worksheet/file doesn’t contain all of attributes in a model in models
  • ignore_extra_attributes (bool, optional) – if True, do not report errors if attributes in the data are not in the model
  • ignore_attribute_order (bool) – if True, do not require the attributes to be provided in the canonical order
  • group_objects_by_model (bool, optional) – if True, group decoded objects by their types
  • validate (bool, optional) – if True, validate the data
Returns:

if group_objects_by_model is set returns dict: model objects grouped

by Model class, otherwise returns list: of model objects

Return type:

obj

class obj_model.io.ReaderBase[source]

Bases: object

Interface for classes which write model objects to file(s)

MODELS[source]

default types of models to export and the order in which to export them

Type:tuple of type
MODELS = ()[source]
run(path, models=None, ignore_missing_sheets=False, ignore_extra_sheets=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, group_objects_by_model=False, validate=True)[source]

Read a list of model objects from file(s) and, optionally, validate them

Parameters:
  • path (str) – path to file(s)
  • models (types.TypeType or list of types.TypeType, optional) – type of object to read or list of types of objects to read
  • ignore_missing_sheets (bool, optional) – if False, report an error if a worksheet/ file is missing for one or more models
  • ignore_extra_sheets (bool, optional) – if True and all models are found, ignore other worksheets or files
  • ignore_sheet_order (bool, optional) – if True, do not require the sheets to be provided in the canonical order
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • ignore_missing_attributes (bool, optional) – if False, report an error if a worksheet/file doesn’t contain all of attributes in a model in models
  • ignore_extra_attributes (bool, optional) – if True, do not report errors if attributes in the data are not in the model
  • ignore_attribute_order (bool) – if True, do not require the attributes to be provided in the canonical order
  • group_objects_by_model (bool, optional) – if True, group decoded objects by their types
  • validate (bool, optional) – if True, validate the data
Returns:

model objects grouped by Model class

Return type:

dict

class obj_model.io.WorkbookReader[source]

Bases: obj_model.io.ReaderBase

Read model objects from an Excel file or CSV and TSV files

classmethod get_ambiguous_sheet_names(sheet_names, models)[source]

Get names of sheets that cannot be unambiguously mapped to models (sheet names that map to multiple models).

Parameters:
  • sheet_names (list of str) – names of the sheets in the workbook/files
  • models (list of Model) – list of models
Returns:

dictionary of ambiguous sheet names and their matching models

Return type:

dict of str, list of Model

classmethod get_model_sheet_name(sheet_names, model)[source]

Get the name of the worksheet/file which corresponds to a model

Parameters:
  • sheet_names (list of str) – names of the sheets in the workbook/files
  • model (Model) – model
Returns:

name of sheet corresponding to the model or None if there is no sheet for the model

Return type:

str

Raises:

ValueError – if the model matches more than one sheet

classmethod get_possible_model_sheet_names(model)[source]

Return set of possible sheet names for a model

Parameters:model (Model) – Model
Returns:set of possible sheet names for a model
Return type:set
classmethod header_row_col_names(index, file_ext, tabular_orientation)[source]

Determine row and column names for header entries.

Parameters:
  • index (int) – index in header sequence
  • file_ext (str) – extension for model file
  • orientation (TabularOrientation) – orientation of the stored table
Returns:

tuple of row, column, header_entries

Construct object graph

Parameters:
  • model (Model) – an obj_model.core.Model
  • attributes (list of Attribute) – attribute order of data
  • data (list of list of object) – nested list of object data
  • objects (list) – list of model objects in order of data
  • objects_by_primary_attribute (dict) – dictionary of model objects grouped by model
  • decoded (dict, optional) – dictionary of objects that have already been decoded
Returns:

list of parsing errors

Return type:

list of str

read_model(reader, model, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, validate=True)[source]

Instantiate a list of objects from data in a table in a file

Parameters:
  • reader (wc_utils.workbook.io.Reader) – reader
  • model (type) – the model describing the objects’ schema
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • ignore_missing_attributes (bool, optional) – if False, report an error if the worksheet/files don’t have all of attributes in the model
  • ignore_extra_attributes (bool, optional) – if True, do not report errors if attributes in the data are not in the model
  • ignore_attribute_order (bool) – if True, do not require the attributes to be provided in the canonical order
  • validate (bool, optional) – if True, validate the data
Returns:

tuple of

list of Attribute, list of list of object, list of str, list of Model: tuple of * attribute order of data * a two-dimensional nested list of object data * a list of parsing errors * constructed model objects

read_sheet(reader, sheet_name, num_row_heading_columns=0, num_column_heading_rows=0)[source]

Read worksheet or file into a two-dimensional list

Parameters:
  • reader (wc_utils.workbook.io.Reader) – reader
  • sheet_name (str) – worksheet name
  • num_row_heading_columns (int, optional) – number of columns of row headings
  • num_column_heading_rows (int, optional) – number of rows of column headings
Returns:

  • list of list: two-dimensional list of table values
  • list of list: row headings
  • list of list: column_headings

Return type:

tuple

Raises:

ValueError – if worksheet doesn’t have header rows or columns

run(path, models=None, ignore_missing_sheets=False, ignore_extra_sheets=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False, group_objects_by_model=True, validate=True)[source]

Read a list of model objects from file(s) and, optionally, validate them

File(s) may be a single Excel workbook with multiple worksheets or a set of delimeter separated files encoded by a single path with a glob pattern.

Parameters:
  • path (str) – path to file(s)
  • models (types.TypeType or list of types.TypeType, optional) – type or list of type of objects to read
  • ignore_missing_sheets (bool, optional) – if False, report an error if a worksheet/ file is missing for one or more models
  • ignore_extra_sheets (bool, optional) – if True and all models are found, ignore other worksheets or files
  • ignore_sheet_order (bool, optional) – if True, do not require the sheets to be provided in the canonical order
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • ignore_missing_attributes (bool, optional) – if False, report an error if a worksheet/file doesn’t contain all of attributes in a model in models
  • ignore_extra_attributes (bool, optional) – if True, do not report errors if attributes in the data are not in the model
  • ignore_attribute_order (bool) – if True, do not require the attributes to be provided in the canonical order
  • group_objects_by_model (bool, optional) – if True, group decoded objects by their types
  • validate (bool, optional) – if True, validate the data
Returns:

if group_objects_by_model set returns dict: of model objects grouped by Model class;

else returns list: of all model objects

Return type:

obj

Raises:

ValueError – if

  • Sheets cannot be unambiguously mapped to models
  • The file(s) indicated by path is missing a sheet for a model and ignore_missing_sheets is False
  • The file(s) indicated by path contains extra sheets that don’t correspond to one of models and ignore_extra_sheets is False
  • The worksheets are file(s) indicated by path are not in the canonical order and ignore_sheet_order is False
  • Some models are not serializable
  • The data contains parsing errors found by read_model
class obj_model.io.WorkbookWriter[source]

Bases: obj_model.io.WriterBase

Write model objects to an Excel file or CSV or TSV file(s)

static create_worksheet_style(model, extra_entries=0)[source]

Create worksheet style for model

Parameters:
  • model (type) – model class
  • extra_entries (int, optional) – additional entries to display
Returns:

worksheet style

Return type:

WorksheetStyle

run(path, objects, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, toc=True, extra_entries=0, data_repo_metadata=False, schema_package=None)[source]
Write a list of model instances to an Excel file, with one worksheet for each model class,
or to a set of .csv or .tsv files, with one file for each model class
Parameters:
  • path (str) – path to write file(s)
  • objects (Model or list of Model) – model instance or list of model instances
  • models (list of Model, optional) – models in the order that they should appear as worksheets; all models which are not in models will follow in alphabetical order
  • get_related (bool, optional) – if True, write objects and all their related objects
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • validate (bool, optional) – if True, validate the data
  • title (str, optional) – title
  • description (str, optional) – description
  • keywords (str, optional) – keywords
  • version (str, optional) – version
  • language (str, optional) – language
  • creator (str, optional) – creator
  • toc (bool, optional) – if True, include additional worksheet with table of contents
  • extra_entries (int, optional) – additional entries to display
  • data_repo_metadata (bool, optional) – if True, try to write metadata information about the file’s Git repo; the repo must be current with origin, except for the file
  • schema_package (str, optional) – the package which defines the obj_model schema used by the file; if not None, try to write metadata information about the the schema’s Git repository: the repo must be current with origin
Raises:

ValueError – if no model is provided or a class cannot be serialized

write_model(writer, model, objects, sheet_models, include_all_attributes=True, encoded=None, extra_entries=0)[source]

Write a list of model objects to a file

Parameters:
  • writer (wc_utils.workbook.io.Writer) – io writer
  • model (type) – model
  • objects (list of Model) – list of instances of model
  • sheet_models (list of Model) – models encoded as separate sheets
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • encoded (dict, optional) – objects that have already been encoded and their assigned JSON identifiers
  • extra_entries (int, optional) – additional entries to display
write_sheet(writer, model, data, headings, validation, extra_entries=0, merge_ranges=None)[source]

Write data to sheet

Parameters:
  • writer (wc_utils.workbook.io.Writer) – io writer
  • model (type) – model
  • data (list of list of object) – list of list of cell values
  • headings (list of list of str) – list of list of row headingsvalidations
  • validation (WorksheetValidation) – validation
  • extra_entries (int, optional) – additional entries to display
  • merge_ranges (list of tuple) – list of ranges of cells to merge
write_toc(writer, models, grouped_objects)[source]

Write a worksheet with a table of contents

Parameters:
  • writer (wc_utils.workbook.io.Writer) – io writer
  • models (list of Model, optional) – models in the order that they should appear in the table of contents
class obj_model.io.Writer[source]

Bases: obj_model.io.WriterBase

Write a list of model objects to file(s)

static get_writer(path)[source]

Get writer

Parameters:path (str) – path to write file(s)
Returns:writer class
Return type:type
Raises:ValueError – if extension is not supported
run(path, objects, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, toc=True, extra_entries=0, data_repo_metadata=False, schema_package=None)[source]
Write a list of model classes to an Excel file, with one worksheet for each model, or to
a set of .csv or .tsv files, with one file for each model.
Parameters:
  • path (str) – path to write file(s)
  • objects (Model or list of Model) – object or list of objects
  • models (list of Model, optional) – models in the order that they should appear as worksheets; all models which are not in models will follow in alphabetical order
  • get_related (bool, optional) – if True, write objects and all related objects
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • validate (bool, optional) – if True, validate the data
  • title (str, optional) – title
  • description (str, optional) – description
  • keywords (str, optional) – keywords
  • version (str, optional) – version
  • language (str, optional) – language
  • creator (str, optional) – creator
  • toc (bool, optional) – if True, include additional worksheet with table of contents
  • extra_entries (int, optional) – additional entries to display
  • data_repo_metadata (bool, optional) – if True, try to write metadata information about the file’s Git repo; the repo must be current with origin, except for the file
  • schema_package (str, optional) – the package which defines the obj_model schema used by the file; if not None, try to write metadata information about the the schema’s Git repository: the repo must be current with origin
class obj_model.io.WriterBase[source]

Bases: object

Interface for classes which write model objects to file(s)

MODELS[source]

default types of models to export and the order in which to export them

Type:tuple of type
MODELS = ()[source]
make_metadata_objects(data_repo_metadata, path, schema_package)[source]

Make models that store Git repository metadata

Metadata models can only be created from suitable Git repos. Failures to obtain metadata are reported as warnings that do not interfeer with writing data files.

Parameters:
  • data_repo_metadata (bool) – if True, try to obtain metadata information about the Git repo containing path; the repo must be current with origin, except for the file at path
  • path (str) – path of the file(s) that will be written
  • schema_package (str, optional) – the package which defines the obj_model schema used by the file; if not None, try to obtain metadata information about the the schema’s Git repository from a package on sys.path: the repo must be current with its origin
Returns:

metadata objects(s) created

Return type:

list of Model

run(path, objects, models=None, get_related=True, include_all_attributes=True, validate=True, title=None, description=None, keywords=None, version=None, language=None, creator=None, toc=True, extra_entries=0, data_repo_metadata=False, schema_package=None)[source]
Write a list of model classes to an Excel file, with one worksheet for each model, or to
a set of .csv or .tsv files, with one file for each model.
Parameters:
  • path (str) – path to write file(s)
  • objects (Model or list of Model) – object or list of objects
  • models (list of Model, optional) – models
  • get_related (bool, optional) – if True, write object and all related objects
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • validate (bool, optional) – if True, validate the data
  • title (str, optional) – title
  • description (str, optional) – description
  • keywords (str, optional) – keywords
  • version (str, optional) – version
  • language (str, optional) – language
  • creator (str, optional) – creator
  • toc (bool, optional) – if True, include additional worksheet with table of contents
  • extra_entries (int, optional) – additional entries to display
  • data_repo_metadata (bool, optional) – if True, try to write metadata information about the file’s Git repo; the repo must be current with origin, except for the file
  • schema_package (str, optional) – the package which defines the obj_model schema used by the file; if not None, try to write metadata information about the the schema’s Git repository: the repo must be current with origin
obj_model.io.convert(source, destination, models, ignore_missing_sheets=False, ignore_extra_sheets=False, ignore_sheet_order=False, include_all_attributes=True, ignore_missing_attributes=False, ignore_extra_attributes=False, ignore_attribute_order=False)[source]

Convert among comma-separated (.csv), Excel (.xlsx), JavaScript Object Notation (.json), tab-separated (.tsv), and Yet Another Markup Language (.yaml, .yml) formats

Parameters:
  • source (str) – path to source file
  • destination (str) – path to save converted file
  • models (list of type) – list of models
  • ignore_missing_sheets (bool, optional) – if False, report an error if a worksheet/ file is missing for one or more models
  • ignore_extra_sheets (bool, optional) – if True and all models are found, ignore other worksheets or files
  • ignore_sheet_order (bool, optional) – if True, do not require the sheets to be provided in the canonical order
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • ignore_missing_attributes (bool, optional) – if False, report an error if a worksheet/file doesn’t contain all of attributes in a model in models
  • ignore_extra_attributes (bool, optional) – if True, do not report errors if attributes in the data are not in the model
  • ignore_attribute_order (bool) – if True, do not require the attributes to be provided in the canonical order
obj_model.io.create_template(path, models, title=None, description=None, keywords=None, version=None, language=None, creator=None, toc=True, extra_entries=10)[source]

Create a template for a model

Parameters:
  • path (str) – path to write file(s)
  • models (list) – list of model, in the order that they should appear as worksheets; all models which are not in models will follow in alphabetical order
  • title (str, optional) – title
  • description (str, optional) – description
  • keywords (str, optional) – keywords
  • version (str, optional) – version
  • language (str, optional) – language
  • creator (str, optional) – creator
  • toc (bool, optional) – if True, include additional worksheet with table of contents
  • extra_entries (int, optional) – additional entries to display
obj_model.io.get_fields(cls, include_all_attributes=True, sheet_models=None)[source]

Get the attributes, headings, and validation for a worksheet

Parameters:
  • cls (type) – Model type (subclass of Model)
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
  • sheet_models (list of Model, optional) – list of models encoded as separate worksheets; used to setup Excel validation for related attributes
Returns:

  • list of Attribute: Attributes of cls in the order they should be encoded as one or more columns in a worksheet. Attributes which define *-to-one relationships to other classes which are encoded as multiple cells (TabularOrientation.multiple_cells) will be encoded as multiple columns. All other attributes will be encoded as a single column.

    This represents a nested tree of attributes. For classes which have *-to-one relationships to other classes which are encoded as multiple cells, the tree has two levels. For all other classes, the tree only has a single level.

  • list of tuple of Attribute: Flattened representation of the first return value. This is a list of attributes of cls and attributes of classes related to cls by *-to-one relationships that are encoded as multiple cells (TabularOrientation.multiple_cells), in the order they are encoded as columns in a worksheet.

    Each element of the list is a tuple.

    1. For attributes of cls that represent *-to-one relationships to classes encoded as multiple cells, the first element will be the attribute. This will be used to populate a merged cell in Row 1 of the worksheet which represents the heading for the multiple columns that encode the attributes of the related class. For all other attributes, the first element will be None, and no value will be printed in Row 1.
    2. The second element will be the attribute that should be encoded in the column. For attributes that represent *-to-one relationships to related classes encoded as multiple cells, this will be an attribute of the related class. For all other attributes, this will be an attribute of cls. This will be used to populate the columns headings for the worksheet. For classes that have *-to-one relationships with classes encoded as multiple columns, the column headings will appear in Row 2 (and the group headings specified by the first element of the tuple will be in Row 1). For all other classes, the column headings will appear in Row 1.
  • list: field headings

  • list: list of field headings to merge

  • list: list of field validations

Return type:

tuple

obj_model.io.get_ordered_attributes(cls, include_all_attributes=True)[source]

Get the attributes for a class in the order that they should be printed

Parameters:
  • cls (type) – Model type (subclass of Model)
  • include_all_attributes (bool, optional) – if True, export all attributes including those not explictly included in Model.Meta.attribute_order
Returns:

attributes in the order they should be printed

Return type:

list of Attribute

3.1.8. obj_model.migrate module

Support schema migration

Author:Arthur Goldberg <Arthur.Goldberg@mssm.edu>
Date:2018-11-18
Copyright:2018, Karr Lab
License:MIT
class obj_model.migrate.CementControllers[source]

Bases: object

Cement Controllers for cement CLIs in data and schema repos involved with migrating files

Because these controllers are used by multiple schema and data repos, they’re defined here and imported into __main__.py modules in schema repos that use obj_model to define data schemas and into __main__.py modules in data repos that contain data files to migrate. wc_lang is an example schema repo. wc_sim is an example data repo that contains data files whose schema is defined in wc_lang.

class DataSchemaMigrationConfigController(*args, **kw)[source]

Bases: cement.ext.ext_argparse.ArgparseController

Create a data-schema migration configuration file

This controller is used by data repos.

class Meta[source]

Bases: object

arguments = [(['--data_repo_dir'], {'type': <class 'str'>, 'help': 'path of the directory of the repository storing the data file(s) to migrate; defaults to the current directory', 'default': '.'}), (['schema_url'], {'type': <class 'str'>, 'help': 'URL of the schema in its git repository, including the branch'}), (['file_to_migrate'], {'action': 'store', 'type': <class 'str'>, 'nargs': '+', 'help': 'a file to migrate'})][source]
description = 'Create a data-schema migration configuration file'[source]
help = 'Create a data-schema migration configuration file'[source]
label = 'make-data-schema-migration-config-file'[source]
stacked_on = 'base'[source]
stacked_type = 'nested'[source]
class MigrateController(*args, **kw)[source]

Bases: cement.ext.ext_argparse.ArgparseController

Perform a migration configured by a data-schema migration config file

This controller is used by data repos.

class Meta[source]

Bases: object

arguments = [(['migration_config_file'], {'type': <class 'str'>, 'help': 'name of the data-schema migration configuration file to use'})][source]
description = 'Migrate data file(s) as configured in a data-schema migration configuration file'[source]
help = 'Migrate data file(s) as configured in a data-schema migration configuration file'[source]
label = 'do-configured-migration'[source]
stacked_on = 'base'[source]
stacked_type = 'nested'[source]
class MigrateFileController(*args, **kw)[source]

Bases: cement.ext.ext_argparse.ArgparseController

Migrate specified data file(s)

This controller is used by data repos.

class Meta[source]

Bases: object

arguments = [(['--data_repo_dir'], {'type': <class 'str'>, 'help': 'path of the directory of the repository storing the data file(s) to migrate; defaults to the current directory', 'default': '.'}), (['schema_url'], {'type': <class 'str'>, 'help': 'URL of the schema in its git repository, including the branch'}), (['file_to_migrate'], {'action': 'store', 'type': <class 'str'>, 'nargs': '+', 'help': 'a file to migrate'})][source]
description = 'Migrate specified data file(s)'[source]
help = 'Migrate specified data file(s)'[source]
label = 'migrate-data'[source]
stacked_on = 'base'[source]
stacked_type = 'nested'[source]
class SchemaChangesTemplateController(*args, **kw)[source]

Bases: cement.ext.ext_argparse.ArgparseController

Create a template schema changes file

This controller is used by schema repos.

class Meta[source]

Bases: object

arguments = [(['--schema_repo_dir'], {'type': <class 'str'>, 'help': "path of the directory of the schema's repository; defaults to the current directory", 'default': '.'}), (['--commit'], {'type': <class 'str'>, 'help': 'hash of a commit containing the changes; default is most recent commit'})][source]
description = 'Create a template schema changes file'[source]
help = 'Create a template schema changes file'[source]
label = 'make-changes-template'[source]
stacked_on = 'base'[source]
stacked_type = 'nested'[source]
class obj_model.migrate.DataSchemaMigration(**kwargs)[source]

Bases: object

Automate the migration of the data files in a repo

A data repo stores the data files that need to be migrated. A schema repo contains the schemas that provide the data models for these data files. The data and schema repos may be the same repo or two different repos.

DataSchemaMigration uses configuration information in the data and schema repos to migrate data files in the data repo to the latest version of the schema repo.

The data repo must contain a migrations directory that has:

  • Data-schema migration configuration files, written in YAML

A data-schema migration configuration file contains the attributes described in DataSchemaMigration._CONFIG_ATTRIBUTES:

  • files_to_migrate: a list of files to be migrated
  • schema_repo_url: the URL of the schema repo
  • branch: the branch of the schema repo
  • schema_file: the relative path of the schema file in the schema repo
  • migrator: the type of migrator to use

The schema repo contains a migrations directory that contains schema changes files, which may refer to associated transformations files. Hashes in the changes files must refer to commits in the schema repo. These files are managed by SchemaChanges objects. Migration will not work if changes to the schema are not documented in schema changes files.

data_repo_location[source]

directory or URL of the data repo

Type:str
data_git_repo[source]

a GitRepo for a git clone of the data repo

Type:GitRepo
schema_git_repo[source]

a GitRepo for a clone of the schema repo

Type:GitRepo
data_config_file_basename[source]

the basename of the YAML configuration file for the migration, which is stored in the data repo’s migrations directory

Type:str
migration_config_data[source]

the data in the data-schema migration config file

Type:dict
loaded_schema_changes[source]

all validated schema change files

Type:list
migration_specs[source]

the migration’s specification

Type:MigrationSpec
git_repos[source]

all GitRepo`s create by this `DataSchemaMigration

Type:list of GitRepo
__str__()[source]

Provide a string representation

Returns:a string representation of this DataSchemaMigration
Return type:str
automated_migrate(tmp_dir=None)[source]

Migrate the data repo’s data files

Migrate to the current version of the schema repo, and migrate data files in place. If the data repo passed to DataSchemaMigration was a directory, then the migrated data files will be stored in that directory.

Parameters:tmp_dir (str, optional) – if the data repo passed to DataSchemaMigration was an URL, then the migrated files will be returned in a temporary directory. If tmp_dir is provided then it will contain the migrated files; if not, then a temporary directory is created to hold them, and the caller is responsible for deleting it.
Returns:the migrated files, and the value of tmp_dir
Return type:tuple of list, str
clean_up()[source]

Delete the temp dirs used by this DataSchemaMigration’s git repos

generate_migration_spec(data_file, schema_changes)[source]

Generate a MigrationSpec from a sequence of schema changes

The migration will migrate data_file in place.

Parameters:
  • data_file (str) – the existing data file that will be migrated
  • schema_changes (list of SchemaChanges) – a sequence of schema changes instances
Returns:

a MigrationSpec that specifies a migration of the file through

the sequence of schema changes

Return type:

MigrationSpec

Raises:

MigratorError – if the MigrationSpec that’s created doesn’t validate

get_data_file_git_commit_hash(data_file)[source]

Get the git commit hash of the schema repo that describes a data file

Parameters:data_file (str) – pathname of a data file
Returns:the hash
Return type:str
Raises:MigratorError – if data_file does not contain a schema repo metadata model
get_name()[source]

Make a timestamped name for a data-schema migration config file

Returns:the name
Return type:str
Raises:MigratorError – if either the data or the schema git repo are not initialized
static get_name_static(data_repo_name, schema_repo_name)[source]

Make a timestamped name for a data-schema migration config file

Parameters:
  • data_repo_name (str) – name of the data repo
  • schema_repo_name (str) – name of the schema repo
Returns:

the name

Return type:

str

static load_config_file(data_schema_migration_conf_file)[source]

Load a data-schema migration config file

Parameters:data_schema_migration_conf_file (str) – path to the data-schema migration config file
Returns:the data in the data-schema migration config file
Return type:dict
Raises:MigratorError – if the data-schema migration config file cannot be found, or is not proper YAML, or does not have the right format, or does not contain any data
static make_data_schema_migration_conf_file_cmd(data_repo_dir, schema_file_url, files_to_migrate, add_to_repo=True)[source]

Make a data-schema migration configuration file from CLI input

Parameters:
  • data_repo_dir (str) – directory of the data repo
  • schema_file_url (str) – URL for schema’s Python file
  • files_to_migrate (list of str) – data files to migrate
  • add_to_repo (bool, optional) – if set, add the migration config file to the data repo; default = True:
Returns:

pathname of the schema changes file that was written

Return type:

str

Raises:

MigratorError – if data_repo_dir isn’t the directory of a repo, or schema_file_url isn’t the URL of a schema file, or files_to_migrate aren’t files

static make_migration_config_file(data_git_repo, schema_repo_name, add_to_repo=True, **kwargs)[source]

Create a data-schema migration config file

Parameters:
  • data_git_repo (GitRepo) – the data git repo that contains the data files to migrate
  • schema_repo_name (str) – name of the schema repo
  • add_to_repo (bool, optional) – if set, add the migration config file to the data repo; default = True:
  • kwargs (dict) – optional initial values for data-schema migration config file
Returns:

the pathname to the data-schema migration config file that was written

Return type:

str

Raises:

MigratorError – if the data-schema migration configuration file already exists

static migrate_files(schema_url, local_dir, data_files)[source]

Migrate some data files specified by command line input

Migrate data files in place in a local repository.

Parameters:
  • schema_url (str) – URL of the schema’s Python file
  • local_dir (str) – directory in a local data repo that contains the data files
  • data_files (list of str) – data files to migrate
Returns:

list of pathnames of migrated files

Return type:

list of str

Raises:

MigratorError – if schema_url isn’t in the right form, or local_dir isn’t a directory, or any of the data files cannot be found, or the migration fails

prepare()[source]

Prepare for migration

  • Validate this DataSchemaMigration
  • Clone each schema version specified by a schema change
  • Generate and prepare MigrationSpec: instances for the migration, one for each file
Raises:MigratorError – if the DataSchemaMigration doesn’t validate
record_git_repo(git_repo)[source]

Record a new GitRepo: so that its temp dir can be deleted later

Parameters:git_repo (GitRepo) – a git repo
schema_changes_for_data_file(data_file)[source]

Generate a sequence of SchemaChanges for migrating a data file

Parameters:data_file (str) – a data file in the data git repo
Returns:a sequence of SchemaChanges for migrating a data file
Return type:list
test_migration()[source]

Test a migration

Check …

The trickiest part of a migration is importing the schema. Unfortunately, imports that use the Python import command may fail with migration’s import, which uses the library call importlib.import_module. This should be called whenever a schema that may be migrated is changed.

This method reports:

  • any validation errors in automatic config files
  • any validation errors in schema changes files
  • any errors in transformations
  • any failures to import schemas

It does not alter any files.

Parameters:data_repo_location (str) – directory or URL of the data repo
validate()[source]

Validate files to migrate, and load all schema changes files

Raises:MigratorError – if any files to migrate do not exist, or all schema changes files cannot be loaded
verify_schemas()[source]

Verify that each schema can be independently imported

It can be difficult to import a schema via importlib.import_module() in import_module_for_migration(). This method tests that proactively.

Returns:all errors obtained
Return type:list of str
class obj_model.migrate.GitRepo(repo_location=None, branch='master', search_parent_directories=False, original_location=None)[source]

Bases: object

Methods for processing a git repo and its commit history

repo_dir[source]

the repo’s root directory

Type:str
repo_url[source]

the repo’s URL, if known

Type:str
branch[source]

the repo’s branch, if it was cloned

Type:str
original_location[source]

the repo’s original root directory or URL, used for debugging

Type:str
repo[source]

the GitPython repo

Type:git.Repo
commit_DAG[source]

NetworkX DAG of the repo’s commit history

Type:nx.classes.digraph.DiGraph
git_hash_map[source]

map from each git hash in the repo to its commit

Type:dict
temp_dirs[source]

temporary directories that hold repo clones

Type:list of str
EMPTY_SUBDIR = 'empty_subdir'[source]
__str__()[source]

Provide a string representation

Returns:a string representation of this GitRepo
Return type:str
add_file(filename)[source]

Add a file to the index

Parameters:filename (str) – path to new file
Raises:MigratorError – if the file cannot be added
checkout_commit(commit_identifier)[source]

Checkout a commit for this repo

Use checkout_commit carefully. If it checks out a new commit, then other operations on the repo may behave differently.

Parameters:commit_identifier (git.objects.commit.Commit or str) – a commit or a commit’s hash
Raises:MigratorError – if the commit cannot be checked out
clone_repo_from_url(url, branch='master', directory=None)[source]

Clone a repo from an URL

Parameters:
  • url (str) – URL for the repo
  • branch (str, optional) – branch to clone; default is master
  • directory (str, optional) – directory to hold the repo; if not provided, the repo is stored in a new temporary dir
Returns:

(git.Repo, str): the repo cloned, and its root directory

Return type:

tuple

Raises:

MigratorError – if repo cannot be cloned from url

commit_changes(message)[source]

Commit the changes in this repo

Parameters:message (str) – the commit message
Raises:MigratorError – if the changes cannot be commited
commits_as_graph()[source]

Make a DAG for this repo’s commit dependencies - edges point from dependent commit to parent commit

The DAG contains all commits in the repo on which the latest commit depends. Also creates git_hash_map, a map from all git hashes to their commits.

Returns:a DAG representing the repo commit history
Return type:nx.classes.digraph.DiGraph
commits_in_dependency_consistent_seq(commits)[source]

Order some git commits into a sequence that’s consistent with the repo’s dependencies

Note that the sequence found is not deterministic, because nodes without dependency relationships can appear in any order. E.g., in a commit DAG with the paths a -> b -> c and a -> d -> c, nodes b and d can appear in either order in the sequece.

Parameters:commits (list of git.objects.commit.Commit) – commits to include in the returned sequence
Returns:
the elements of commits, in a sequence
that’s consistent with git commit dependencies in the repository, ordered from from antecedent to dependent
Return type:list of git.objects.commit.Commit
copy(tmp_dir=None)[source]

Copy this GitRepo into a new directory

For better performance use copy() instead of GitRepo() or clone_repo_from_url() if you need multiple copies of a repo, such as multiple instances checked out to different commits. This is an optimization because copying is faster than cloning over the network. To avoid bytecode is stale errors, doesn’t copy __pycache__ directories.

Parameters:tmp_dir (str, optional) – directory to hold the repo; if not provided, the repo will be stored in a new temporary dir
Returns:a new GitRepo that’s a copy of self in a new temporary directory
Return type:GitRepo
del_temp_dirs()[source]

Delete the temp dirs created by get_temp_dir

Returns:the pathname to a temporary directory
Return type:str
fixtures_dir()[source]

Get the repo’s fixtures directory

Returns:the repo’s fixtures directory
Return type:str
get_commit(commit_or_hash)[source]

Obtain a commit from its hash

Also, if commit_or_hash is a commit, simply return it.

Parameters:commit_or_hash (str or git.objects.commit.Commit) – the hash of a commit or a commit in the repo
Returns:a commit
Return type:git.objects.commit.Commit
Raises:MigratorError – if commit_or_hash is not a commit and cannot be converted into one
get_commits(commits_or_hashes)[source]

Get the commits with the given commits or hashes

Parameters:commits_or_hashes (list of str) – an iterator over commits or commit hashes
Returns:
list of the repo’s commits with the
commits or hashes in commits_or_hashes
Return type:list of git.objects.commit.Commit
Raises:MigratorError – if any of the commits or hashes don’t identify a commit in the repo
get_dependents(commit_or_hash)[source]

Get all commits that depend on a commit, including transitively

Parameters:commit_or_hash (str or git.objects.commit.Commit) – the hash of a commit or a commit in the repo
Returns:all commits that depend on commit_or_hash
Return type:set of git.objects.commit.Commit
static get_hash(commit)[source]

Get a commit’s hash

Parameters:commit (git.objects.commit.Commit) – a commit
Returns:the commit’s SHA1 hash
Return type:str
get_temp_dir()[source]

Get a temporary directory, which must eventually be deleted by calling del_temp_dirs

Returns:the pathname to a temporary directory
Return type:str
static hash_prefix(hash)[source]

Get a commit hash’s prefix

Parameters:hash (str) – git commit hash
Returns:hash’s prefix
Return type:str
head_commit()[source]

Get the repo’s head commit

Returns:the repo’s latest commit
Return type:git.objects.commit.Commit
latest_hash()[source]

Get the hash of the repo’s latest commit

Returns:the latest commit’s SHA1 hash
Return type:str
migrations_dir()[source]

Get the repo’s migrations directory

Returns:the repo’s migrations directory
Return type:str
repo_name()[source]

Get the repo’s name

Returns:the repo’s name
Return type:str
exception obj_model.migrate.MigrateWarning[source]

Bases: UserWarning

Migrate warning

class obj_model.migrate.MigrationController[source]

Bases: object

Manage migrations

Manage migrations on several dimensions:

  • Migrate a single model file through a sequence of schemas
  • Perform migrations parameterized by a configuration file
static migrate_from_config(migrations_config_file)[source]

Perform the migrations specified in a config file

Parameters:migrations_config_file (str) – pathname of migrations configuration in YAML file
Returns:list of (MigrationSpec, migrated filenames) pairs
Return type:list of tuple
static migrate_from_spec(migration_spec)[source]

Perform the migration specified in a MigrationSpec

Parameters:migration_spec (MigrationSpec) – a migration specification
Returns:of str: migrated filenames
Return type:list
static migrate_over_schema_sequence(migration_spec)[source]

Migrate some model files over a sequence of schemas

Parameters:migration_spec (MigrationSpec) – a migration specification
Returns:
for each migration, its sequence of models and
its migrated filename
Return type:tuple of list, list
Raises:MigratorError – if schema_files, renamed_models, and seq_of_renamed_attributes are not consistent with each other;
class obj_model.migrate.MigrationSpec(name, migrator='standard_migrator', existing_files=None, schema_files=None, git_hashes=None, seq_of_renamed_models=None, seq_of_renamed_attributes=None, seq_of_transformations=None, migrated_files=None, migrate_suffix=None, migrate_in_place=False, migrations_config_file=None)[source]

Bases: object

Specification of a sequence of migrations for a list of existing files

_REQUIRED_ATTRS[source]

required attributes in a MigrationSpec

Type:list of str
_CHANGES_LISTS[source]

lists of changes in a migration

Type:list of str
_ALLOWED_ATTRS[source]

attributes allowed in a MigrationSpec

Type:list of str
name[source]

name for this MigrationSpec

Type:str
migrator[source]

the name of a Migrator to use for migrations, which must be a key in self.MIGRATOR_CREATOR_MAP; default = standard_migrator, which maps to Migrator

Type:str
existing_files (:obj:`list`

of str, optional): existing files to migrate

schema_files[source]

list of Python files containing model definitions for each state in a sequence of migrations

Type:list of str, optional
git_hashes[source]

list of the git hashes of the git versions that contain the schemas

Type:list of str, optional
seq_of_renamed_models[source]

list of renamed models for use by a Migrator for each migration in a sequence of migrations

Type:list of list, optional
seq_of_renamed_attributes[source]

list of renamed attributes for use by a Migrator for each migration in a sequence of migrations

Type:list of list, optional
seq_of_transformations[source]

list of transformations for use by a Migrator for each migration in a sequence of migrations

Type:list of dict, optional
migrated_files (:obj:`list`

of str, optional): migration destination files in 1-to-1 correspondence with existing_files; if not provided, migrated files use a suffix or are migrated in place

migrate_suffix[source]

suffix added to destination file name, before the file type suffix

Type:str, optional
migrate_in_place[source]

whether to migrate in place

Type:bool, optional
migrations_config_file[source]

if created from a configuration file, the file’s path

Type:str, optional
_prepared[source]

whether this MigrationSpec has been prepared

Type:bool, optional
DEFAULT_MIGRATOR = 'standard_migrator'[source]
MIGRATOR_CREATOR_MAP = {'standard_migrator': <class 'obj_model.migrate.Migrator'>, 'wc_lang': <function Migrator.generate_wc_lang_migrator>}[source]
__str__()[source]

Get str representation

Returns:string representation of all allowed attributes in a MigrationSpec
Return type:str
expected_migrated_files()[source]

Provide names of migrated files that migration of this MigrationSpec would produce

Returns:
the names of the migrated files that a successful migration of this
MigrationSpec will produce
Return type:list of str
static get_migrations_config(migrations_config_file)[source]

Create a list of `MigrationSpec`s from a migrations configuration file

Parameters:migrations_config_file (str) – pathname of migrations configuration in YAML file
Returns:migration specifications
Return type:dict of MigrationSpec
Raises:MigratorError – if migrations_config_file cannot be read
get_migrator()[source]

Obtain callable that creates Migrator`s for this `MigrationSpec

Returns:a callable that creates Migrator`s for this `MigrationSpec
Return type:callable
is_prepared()[source]

Check that this MigrationSpec has been prepared

Raises:MigratorError – if this MigrationSpec has not been prepared
classmethod load(migrations_config_file)[source]

Create a list of validated and standardized `MigrationSpec`s from a migrations configuration file

Parameters:migrations_config_file (str) – pathname of migrations configuration in YAML file
Returns:migration specifications
Return type:dict of MigrationSpec
Raises:MigratorError – if migrations_config_file cannot be read, or the migration specifications in migrations_config_file are not valid
prepare()[source]

Validate and standardize this MigrationSpec

Raises:MigratorError – if migrations_config_file cannot be read, or the migration specifications in migrations_config_file are not valid
standardize()[source]

Standardize the attributes of a MigrationSpec

In particular, standardize a MigrationSpec read from a YAML config file

validate()[source]

Validate the attributes of a migration specification

Returns:list of errors found
Return type:list of str
class obj_model.migrate.Migrator(existing_defs_file=None, migrated_defs_file=None, renamed_models=None, renamed_attributes=None, transformations=None)[source]

Bases: object

Support schema migration

existing_schema[source]

the existing schema, and its properties

Type:SchemaModule
migrated_schema[source]

the migrated schema, and its properties

Type:SchemaModule
existing_defs[source]

obj_model.Model definitions of the existing models, keyed by name

Type:dict
migrated_defs[source]

obj_model.Model definitions of the migrated models, keyed by name

Type:dict
deleted_models[source]

model types defined in the existing models but not the migrated models

Type:set
renamed_models[source]

model types renamed from the existing to the migrated schema

Type:list of tuple
models_map[source]

map from existing model names to migrated model names

Type:dict
renamed_attributes[source]

attribute names renamed from the existing to the migrated schema

Type:list of tuple
renamed_attributes_map[source]

map of attribute names renamed from the existing to the migrated schema

Type:dict
_migrated_copy_attr_name[source]

attribute name used to point existing models to corresponding migrated models; not used in any existing schema

Type:str
transformations[source]

map of transformation types in SUPPORTED_TRANSFORMATIONS to callables

Type:dict
COLLECTIONS_ATTRS = ['existing_defs', 'migrated_defs', 'renamed_models', 'models_map', 'renamed_attributes', 'renamed_attributes_map'][source]
MIGRATED_COPY_ATTR_PREFIX = '__migrated_copy'[source]
MIGRATE_SUFFIX = '_migrated'[source]
MODIFY_MIGRATED_MODELS = 'MODIFY_MIGRATED_MODELS'[source]
PARSED_EXPR = '_parsed_expression'[source]
PREPARE_EXISTING_MODELS = 'PREPARE_EXISTING_MODELS'[source]
SCALAR_ATTRS = ['deleted_models', '_migrated_copy_attr_name'][source]
SUPPORTED_TRANSFORMATIONS = ['PREPARE_EXISTING_MODELS', 'MODIFY_MIGRATED_MODELS'][source]
__str__()[source]

Get str representation

Returns:
string representation of a Migrator; collections attributes are rendered
by pformat
Return type:str
full_migrate(existing_file, migrated_file=None, migrate_suffix=None, migrate_in_place=False)[source]

Migrate data from an existing file to a migrated file

Parameters:
  • existing_file (str) – pathname of file to migrate
  • migrated_file (str, optional) – pathname of migrated file; if not provided, save migrated file with migrated suffix in same directory as existing file
  • migrate_suffix (str, optional) – suffix of automatically created migrated filename; default is Migrator.MIGRATE_SUFFIX
  • migrate_in_place (bool, optional) – if set, overwrite existing_file with the migrated file and ignore migrated_file and migrate_suffix
Returns:

name of migrated file

Return type:

str

Raises:

MigratorError – if migrate_in_place is False and writing the migrated file would overwrite an existing file

static generate_wc_lang_migrator(**kwargs)[source]

Generate a Migrator for a WC-Lang (wc_lang) model

WC-Lang model files must contain exactly one Model instance, and as a convenience for users, they allow uninitialized model attributes which reference the Model instance. Like wc_lang.io.Reader, the Migrator generated here initializes these attributes. This ensures that model files written by the Migrator contain initialized Model references, and enables round-trip migrations of WC-Lang model files.

Parameters:kwargs (dict) arguments for `Migrator() –
Returns:
a new Migrator, with a transformation that initializes model
attributes which reference the Model instance
Return type:Migrator
Raises:MigratorError – if kwargs contains transformations
migrate(existing_models)[source]

Migrate existing model instances to the migrated schema

:param existing_models (list of obj_model.Model: ) the models being migrated

Returns:the migrated models
Return type:list of obj_model.Model
modules = {}[source]
static path_of_migrated_file(existing_file, migrate_suffix=None, migrate_in_place=False)[source]

Determine the pathname of the migrated file

Parameters:
  • existing_file (str) – pathname of file being migrated
  • migrate_suffix (str, optional) – suffix of automatically created migrated filename; default is Migrator.MIGRATE_SUFFIX
  • migrate_in_place (bool, optional) – if set, migrated file is existing_file, which will be overwritten
Returns:

name of migrated file

Return type:

str

prepare()[source]

Prepare for migration

Raises:MigratorError – if renamings are not valid, or inconsistencies exist between corresponding existing and migrated classes
read_existing_model(existing_file)[source]

Read models from existing file

Does not perform validation – data in existing model file must be already validated with the existing schema.

Parameters:existing_file (str) – pathname of file to migrate
Returns:the models in existing_file
Return type:list of obj_model.Model
run(files)[source]

Migrate some files

Parameters:files (list) – names of model files to migrate
write_migrated_file(migrated_models, model_order, existing_file, migrated_file=None, migrate_suffix=None, migrate_in_place=False)[source]

Write migrated models to an external representation

Does not perform validation – validation must be performed independently.

:param migrated_models (list of obj_model.Model: ) the migrated models :param model_order (list of obj_model.core.ModelMeta: ) migrated models in the order

they should appear in a workbook
Parameters:
  • existing_file (str) – pathname of file that is being migrated
  • migrated_file (str, optional) – pathname of migrated file; if not provided, save migrated file with migrated suffix in same directory as source file
  • migrate_suffix (str, optional) – suffix of automatically created migrated filename; default is Migrator.MIGRATE_SUFFIX
  • migrate_in_place (bool, optional) – if set, overwrite existing_file with the migrated file and ignore migrated_file and migrate_suffix
Returns:

name of migrated file

Return type:

str

Raises:

MigratorError – if migrate_in_place is False and writing the migrated file would overwrite an existing file

exception obj_model.migrate.MigratorError(message=None)[source]

Bases: Exception

Exception raised for errors in obj_model.migrate

message[source]

the exception’s message

Type:str
class obj_model.migrate.SchemaChanges(schema_repo=None, schema_changes_file=None, commit_hash=None, renamed_models=None, renamed_attributes=None, transformations_file=None)[source]

Bases: object

Specification of the changes to a schema in a git commit

More generally, a SchemaChanges should encode the set of changes to a schema over the sequence of git commits since the previous SchemaChanges.

_CHANGES_FILE_ATTRS[source]

required attributes in a schema changes file

Type:list of str
_ATTRIBUTES[source]

attributes in a SchemaChanges instance

Type:list of str
schema_repo[source]

a Git repo that defines the data model (schema) of the data being migrated

Type:GitRepo
schema_changes_file[source]

the schema changes file

Type:str
commit_hash[source]

hash from a schema changes file

Type:str
renamed_models[source]

list of renamed models in the commit

Type:list, optional
renamed_attributes[source]

list of renamed attributes in the commit

Type:list, optional
transformations_file[source]

the name of a Python file containing transformations

Type:str, optional
transformations[source]

the transformations for a migration to the schema, in a dictionary of callables

Type:dict, optional
__str__()[source]

Provide a string representation

Returns:a string representation of this SchemaChanges
Return type:str
static all_schema_changes_files(migrations_directory)[source]

Find all schema changes files in a git repo

Parameters:migrations_directory (str) – path to the migrations directory in a git repo
Returns:of str: pathnames of the schema changes files
Return type:list
Raises:MigratorError – if no schema changes files are found
static all_schema_changes_with_commits(schema_repo)[source]

Instantiate all schema changes in a git repo

Obtain all validated schema change files.

Parameters:schema_repo (GitRepo) – an initialized repo for the schema
Returns:list of errors, list all validated schema change files
Return type:tuple
static find_file(schema_repo, commit_hash)[source]

Find a schema changes file in a git repo

Parameters:
  • schema_repo (GitRepo) – an initialized repo for the schema
  • commit_hash (str) – a git commit hash
Returns:

the pathname of the file found

Return type:

str

Raises:

MigratorError – if a file with the hash cannot be found, or multiple files have the hash

generate_filename(commit_hash)[source]

Generate a filename for a template schema changes file

Returns:the filename
Return type:str
static generate_instance(schema_changes_file)[source]

Generate a SchemaChanges instance from a schema changes file

Parameters:schema_changes_file (str) – path to the schema changes file
Returns:the SchemaChanges instance
Return type:SchemaChanges
static get_date_timestamp()[source]

Get a current date timestamp, with second resolution

Returns:the timestamp
Return type:str
get_hash()[source]

Get the repo’s current commit hash

Returns:the hash
Return type:str
static hash_prefix_from_sc_file(schema_changes_file)[source]

Get the hash prefix from a schema changes filename

Parameters:schema_changes_file (str) – the schema changes file
Returns:the hash prefix in a schema changes filename
Return type:str
import_transformations()[source]

Import the transformation functions referenced in a schema changes file

Returns:the transformations for a migration to the schema, in a dictionary of callables
Return type:dict
Raises:MigratorError – if the transformations file cannot be imported, or it does not have a ‘transformations’ attribute, or ‘transformations’ isn’t a dict of callables as specified by Migrator.SUPPORTED_TRANSFORMATIONS
static load(schema_changes_file)[source]

Read a schema changes file

Parameters:schema_changes_file (str) – path to the schema changes file
Returns:the data in the schema changes file
Return type:dict
Raises:MigratorError – if the schema changes file cannot be found, or is not proper YAML, or does not have the right format, or does not contain any changes
make_template(schema_url=None, commit_hash=None)[source]

Make a template schema changes file

The template includes the repo hash which it describes and empty values for SchemaChanges attributes.

Parameters:
  • schema_url (str, optional) – URL of the schema repo; if not provided, self.schema_repo must be already initialized
  • commit_hash (str, optional) – hash of the schema repo commit which the template schema changes file describes; default is the most recent commit
Returns:

pathname of the schema changes file that was written

Return type:

str

Raises:

MigratorError – if a repo cannot be cloned from schema_url, or checked out from commit_hash, or the schema changes file already exists

static make_template_command(schema_dir, commit_hash=None)[source]

Make a template schema changes file with CLI input

Parameters:
  • schema_dir (str) – directory of the schema repo
  • commit_hash (str, optional) – hash of the schema repo commit which the template schema changes file describes; default is the most recent commit
Returns:

pathname of the schema changes file that was written

Return type:

str

Raises:

MigratorError – if a repo cannot be cloned from schema_url, or checked out from commit_hash, or the schema changes file already exists

static validate(schema_changes_kwargs)[source]

Check that the attributes of the arguments to SchemaChanges have the right structure

Parameters:schema_changes_kwargs (dict) – kwargs arguments to SchemaChanges generated by loading a schema changes file
Returns:errors in schema_changes_kwargs
Return type:list
class obj_model.migrate.SchemaModule(module_path, dir=None, annotation=None)[source]

Bases: object

Represent and import a schema module

module_path[source]

path to the module

Type:str
abs_module_path[source]

absolute path to the module

Type:str
directory[source]

if the module is in a package, the path to the package’s directory; otherwise the directory containing the module

Type:str
package_name[source]

if the module is in a package, the name of the package containing the module; otherwise None

Type:str
module_name[source]

the module’s module name

Type:str
annotation[source]

an optional annotation, often the original path of a module that hass been copied; used for debugging

Type:str
MODULES = {}[source]
MODULE_ANNOTATIONS = {}[source]
MUNGED_MODEL_NAME_SUFFIX = '_MUNGED WITH SPACES'[source]
get_path()[source]
import_module_for_migration(validate=True, required_attrs=None, debug=False, mod_patterns=None, print_code=False)[source]

Import a schema from a Python module in a file, which may be in a package

Parameters:
  • validate (bool, optional) – whether to validate the module; default is True
  • required_attrs (list of str, optional) – list of attributes that must be present in the imported module
  • debug (bool, optional) – if true, print debugging output; default is False
  • mod_patterns (list of str, optional) – RE patterns used to search for modules in sys.modules; modules whose names match a pattern are output when debug is true
  • print_code (bool, optional) – if true, while debugging print code being imported; default is False
Returns:

the Module loaded from self.module_path

Return type:

Module

Raises:

MigratorError – if the schema at self.module_path cannot be imported, or if validate is True and any related attribute in any model references a model

not in the module

or if the module is missing a required attribute

in_package()[source]

Is the schema in a package

Returns:whether the schema is in a package
Return type:bool
static parse_module_path(module_path)[source]

Parse the path to a module

If the module is not in a package, provide its directory and module name. If the module is in a package, provide its directory, package name and module name. The directory can be used as a sys.path entry.

Parameters:module_path (str) – path of a Python module file
Returns:a triple containing directory, package name and module name, as described above. If the module is not in a package, then package name is None.
Return type:tuple
Raises:MigratorError – if module_path is not the name of a Python file, or is not a file
run()[source]

Import a schema and provide its `obj_model.Model`s

Returns:the imported Models
Return type:dict
Raises:MigratorError – if self.module_path cannot be loaded
class obj_model.migrate.Utils[source]

Bases: object

Utilities for migration

static find_schema_modules()[source]

Find the modules used by a schema

Useful for creating schema changes files for a schema repo

Returns:???
Return type:argparse.Namespace
class obj_model.migrate.VirtualEnvUtil(name, dir=None)[source]

Bases: object

Support creation, use and distruction of virtual environments for Python packages

Will be used to allow different schema versions depend on different package versions

name[source]

name of the VirtualEnvUtil

Type:str
activate()[source]

Use this VirtualEnvUtil

deactivate()[source]

Stop using this VirtualEnvUtil

destroy()[source]

Destroy this VirtualEnvUtil

Distruction deletes the directory storing the VirtualEnvUtil

destroyed()[source]

Test whether this VirtualEnvUtil has been destroyed

install_from_pip_spec(pip_spec)[source]

Install a package from a pip specification

Parameters:pip_spec (str) – a pip specification for a package to load
Raises:ValueError – if the package described by pip_spec cannot be installed
is_installed(pip_spec)[source]

3.1.9. obj_model.obj_math module

Math attributes

Author:Jonathan Karr <karr@mssm.edu>
Date:2017-05-10
Copyright:2017, Karr Lab
License:MIT
class obj_model.obj_math.NumpyArrayAttribute(min_length=0, max_length=inf, default=None, none_value=None, verbose_name='', help='', primary=False, unique=False)[source]

Bases: obj_model.core.LiteralAttribute

numpy.array attribute

min_length[source]

minimum length

Type:int
max_length[source]

maximum length

Type:int
default[source]

default value

Type:numpy.ndarray
deserialize(value)[source]

Deserialize value

Parameters:value (str) – semantically equivalent representation
Returns:tuple of cleaned value and cleaning error
Return type:tuple of numpy.array, core.InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (list) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:numpy.array
serialize(value)[source]

Serialize string

Parameters:value (numpy.array) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (numpy.array) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:list
validate(obj, value)[source]

Determine if value is a valid value

Parameters:
  • obj (Model) – class being validated
  • value (numpy.array) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

validate_unique(objects, values)[source]

Determine if the attribute values are unique

Parameters:
  • objects (list of Model) – list of Model objects
  • values (list of numpy.array) – list of values
Returns:

None if values are unique, otherwise return a list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

class obj_model.obj_math.SympyBasicAttribute(type=<class 'sympy.core.basic.Basic'>, default=None, none_value=None, verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.core.LiteralAttribute

Base class for SymPy expression, symbol attributes

type[source]

attribute type (e.g. sympy.Basic, sympy.Expr, sympy.Symbol)

Type:sympy.core.assumptions.ManagedProperties
default[source]

default value

Type:sympy.Basic
deserialize(value)[source]

Deserialize value

Parameters:value (str) – semantically equivalent representation
Returns:tuple of cleaned value and cleaning error
Return type:tuple of sympy.Basic, core.InvalidAttribute or None
from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (list) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:sympy.Basic
serialize(value)[source]

Serialize string

Parameters:value (sympy.Basic) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (sympy.Basic) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:str
validate(obj, value)[source]

Determine if value is a valid value

Parameters:
  • obj (Model) – class being validated
  • value (sympy.Basic) – value of attribute to validate
Returns:

None if attribute is valid, other return list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

validate_unique(objects, values)[source]

Determine if the attribute values are unique

Parameters:
  • objects (list of Model) – list of Model objects
  • values (list of sympy.Basic) – list of values
Returns:

None if values are unique, otherwise return a list of errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

class obj_model.obj_math.SympyExprAttribute(default=None, none_value=None, verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.obj_math.SympyBasicAttribute

SymPy expression attribute

default[source]

default value

Type:sympy.Expr
serialize(value)[source]

Serialize string

Parameters:value (sympy.Expr) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (sympy.Expr) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:str
class obj_model.obj_math.SympySymbolAttribute(default=None, none_value=None, verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.obj_math.SympyBasicAttribute

SymPy symbol attribute

default[source]

default value

Type:sympy.Symbol
serialize(value)[source]

Serialize string

Parameters:value (sympy.Symbol) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (sympy.Symbol) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:str

3.1.10. obj_model.ontology module

Ontology attributes

Author:Jonathan Karr <karr@mssm.edu>
Date:2019-01-14
Copyright:2019, Karr Lab
License:MIT
class obj_model.ontology.OntologyAttribute(ontology, namespace=None, terms=None, none=True, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help='', primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.core.LiteralAttribute

Ontology attribute

ontology (obj

pronto.Ontology): ontology

namespace[source]

prefix in term ids

Type:str
terms[source]

list of allowed terms. If None, all terms are allowed.

Type:list of pronto.term.Term
none[source]

if False, the attribute is invalid if its value is None

Type:bool
clean(value)[source]

Convert attribute value into the appropriate type

Parameters:value (object) – value of attribute to clean
Returns:cleaned value core.InvalidAttribute or None: cleaning error
Return type:pronto.term.Term or None
copy_value(value, objects_and_copies)[source]

Copy value

Parameters:
  • value (object) – value
  • objects_and_copies (dict) – dictionary that maps objects to their copies
Returns:

copy of value

Return type:

object

from_builtin(json)[source]

Decode a simple Python representation (dict, list, str, float, bool, None) of a value of the attribute that is compatible with JSON and YAML

Parameters:json (str) – simple Python representation of a value of the attribute
Returns:decoded value of the attribute
Return type:pronto.term.Term
get_default()[source]

Get default value for attribute

Returns:initial value
Return type:object
get_default_cleaned_value()[source]

Get value to replace None values with during cleaning

Returns:initial value
Return type:object
get_excel_validation(sheet_models=None)[source]

Get Excel validation

Returns:validation
Return type:wc_utils.workbook.io.FieldValidation
serialize(value)[source]

Serialize ontology instance

Parameters:value (pronto.term.Term) – Python representation
Returns:simple Python representation
Return type:str
to_builtin(value)[source]

Encode a value of the attribute using a simple Python representation (dict, list, str, float, bool, None) that is compatible with JSON and YAML

Parameters:value (pronto.term.Term) – value of the attribute
Returns:simple Python representation of a value of the attribute
Return type:str
validate(obj, value)[source]

Determine if value is a valid value of the attribute

Parameters:
  • obj (Model) – object being validated
  • value (pronto.term.Term) – value of attribute to validate
Returns:

None if attribute is valid, other return list of

errors as an instance of core.InvalidAttribute

Return type:

core.InvalidAttribute or None

value_equal(val1, val2, tol=0.0)[source]

Determine if attribute values are equal

Parameters:
  • val1 (pronto.Term) – first value
  • val2 (pronto.Term) – second value
  • tol (float, optional) – equality tolerance
Returns:

True if attribute values are equal

Return type:

bool

3.1.11. obj_model.units module

Unit attribute

Author:Jonathan Karr <karr@mssm.edu>
Date:2019-01-20
Copyright:2019, Karr Lab
License:MIT
class obj_model.units.UnitAttribute(registry, choices=None, none=True, default=None, default_cleaned_value=None, none_value=None, verbose_name='', help="Units (e.g. 'second', 'meter', or 'gram')", primary=False, unique=False, unique_case_insensitive=False)[source]

Bases: obj_model.core.LiteralAttribute

Unit attribute

registry[source]

unit registry

Type:pint.UnitRegistry
choices[source]

allowed values

Type:tuple of pint.unit._Unit
none[source]

if False, the attribute is invalid if its value is None

Type:bool