storage Package

This is just a shortcut, in order to make all DAO functions accessible from a single point, in an uniform manner, without supplementary name space. We want DAO functions to be separated in multiple files, because they are too many to easily follow.

class tvb.core.entities.storage.DAO[source]

Bases: DatatypeDAO, OperationDAO, CaseDAO, BurstDAO, WorkflowDAO

Empty class, build only for inheriting from all DAO classes.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

burst_dao

class tvb.core.entities.storage.burst_dao.BurstDAO[source]

Bases: RootDAO

DAO layer for Burst entities.

count_bursts_with_name(burst_name, project_id)[source]

Return the number of burst already named ‘custom_b%’ and NOT ‘custom_b%_%’ in current project.

get_burst_by_id(burst_id)[source]

Get the BurstConfiguration entity with the given id

get_burst_for_direct_operation_id(operation_id, is_group=False)[source]
get_burst_for_operation_id(operation_id, is_group=False)[source]
get_bursts_for_project(project_id, page_start=0, page_size=200, count=False)[source]

Get latest 50 BurstConfiguration entities for the current project

get_max_burst_id()[source]

Return the maximum of the currently stored burst IDs to be used as the new burst name. This is not a thread-safe value, but we use it just for a label.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

datatype_dao

DAO operations related to generic DataTypes are defined here.

class tvb.core.entities.storage.datatype_dao.DatatypeDAO[source]

Bases: RootDAO

DATATYPE and DATA_TYPES_GROUPS RELATED METHODS

compute_bursts_disk_size(burst_gids)[source]

SUM the disk_size of all data types generated by each requested burst Do not count DataType Groups as those already include the size of the entities inside the group. :returns a map from burst id to disk size

count_all_datatypes()[source]

Gives you the count of all the datatypes currently stored by TVB. Is used by the file storage update manager to upgrade from version to the next.

count_datatypes(project_id, datatype_class)[source]
count_datatypes_in_burst(burst_gid)[source]

Returns the number of DataTypes from the specified BurstConfiguration ID.

count_datatypes_in_group(datatype_group_id)[source]

Returns the number of DataTypes from the specified DataTypeGroup ID.

get_all_datatypes(page_start=0, page_size=200)[source]

Return a list with all of the datatypes currently available in TVB. Is used by the file storage update manager to upgrade from version to the next.

Parameters:
  • page_start – the index from which to start adding datatypes to the result list

  • page_size – maximum number of entities to retrieve

get_data_in_project(project_id, visibility_filter=None, filter_value=None)[source]

Get all the DataTypes for a given project, including Linked Entities and DataType Groups.

Parameters:
  • project_id – ID for the current project to filter for

  • visibility_filter – when not None, will filter by DataTye fields

  • filter_value – when not None, will filter with ilike multiple DataType string attributes

get_datatype_by_gid(gid, load_lazy=True, load_lazy_extra_info=False)[source]

Retrieve a DataType DB reference by a global identifier.

get_datatype_details(datatype_gid)[source]

Returns the details for the dataType with the given GID.

get_datatype_extra_info(datatype_gid)[source]
get_datatype_group_by_gid(datatype_group_gid)[source]

Returns the DataTypeGroup with the specified gid.

get_datatype_group_disk_size(dt_group_id)[source]

Return the size of all the DataTypes from this datatype group.

get_datatype_in_group(datatype_group_id=None, operation_group_id=None)[source]

Return a list of id-s of the DataTypes in the given dt group.

get_datatype_measure_group_from_ts_from_pse(ts_gid, datatype_measure_class)[source]

Having a Time Series that was part of a pse, this method should return the datatype group of a datatype measure that was part of the same pse

get_datatypegroup_by_op_group_id(operation_group_id)[source]

Returns the DataTypeGroup corresponding to a certain OperationGroup.

get_datatypegroup_for_project(project_id)[source]

Returns all DataTypeGroup entities from a project.

get_datatypes_for_range(op_group_id, range_json)[source]

Retrieve from DB, DataTypes resulted after executing a specific range operation.

get_datatypes_from_datatype_group(datatype_group_id)[source]

Retrieve all datatype which are part from the given datatype group.

get_datatypes_in_project(project_id, only_visible=False)[source]

Get all the DataTypes for a given project with no other filter apart from the projectId

get_disk_size_for_operation(operation_id)[source]

Return the disk size for the operation by summing over the disk space of the resulting DataTypes.

get_last_data_with_uid(uid, datatype_class=<class 'tvb.core.entities.model.model_datatype.DataType'>)[source]

Retrieve the last dataType ID witch has UDI field as the passed parameter, or None if nothing found.

get_linked_datatypes_in_project(project_id)[source]

Return a list of datatypes linked into this project :param project_id: the id of the project

Get the links to a specific datatype

get_number_of_bursts(project_id)[source]
get_selections_for_project(project_id, datatype_gid, filter_ui_name=None)[source]

Get available selections for a given project and data type.

get_stored_pse_filters(datatype_group_gid, filter_ui_name=None)[source]
Returns:

Stored PSE filters for a given DatTypeGroup or None

get_summary_for_group(datatype_group_id)[source]

:return (disk_size SUM, subject)

get_values_of_datatype(project_id, datatype_class, filters=None, page_size=50)[source]

Retrieve a list of dataTypes matching a filter inside a project. :returns: (results, total_count) maximum page_end rows are returned, to avoid endless time when loading a page

is_datatype_group(datatype_gid)[source]

Used to check if the DataType with the specified GID is a DataTypeGroup.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>
set_datatype_visibility(datatype_gid, is_visible)[source]

Sets the dataType visibility. If the given dataType is a dataTypeGroup or it is part of a dataType group than this method will set the visibility for each dataType from this group.

try_load_last_entity_of_type(project_id, datatype_class)[source]

exceptions

Created on Jan 15, 2013

exception tvb.core.entities.storage.exceptions.BaseStorageException(message)[source]

Bases: Exception

Base class for all TVB storage exceptions.

exception tvb.core.entities.storage.exceptions.InvalidTransactionAccess(message)[source]

Bases: BaseStorageException

Exception raised in case you have any faulty access to a transaction.

exception tvb.core.entities.storage.exceptions.NestedTransactionUnsupported(message)[source]

Bases: BaseStorageException

Nested transactions are not supported unless in testing.

operation_dao

DAO operations related to Algorithms and User Operations are defined here.

class tvb.core.entities.storage.operation_dao.OperationDAO[source]

Bases: RootDAO

OPERATION RELATED METHODS

compute_disk_size_for_started_ops(user_id)[source]

Get all the disk space that should be reserved for the started operations of this user.

count_resulted_datatypes(operation_id)[source]

Returns the number of resulted datatypes from the specified operation.

filter_category(displayname, rawinput, display, launchable, order_nr)[source]

Retrieve category with given id

get_adapters_from_categories(categories)[source]

Retrieve a list of stored adapters in the given categories.

get_algorithm_by_id(algorithm_id)[source]
get_algorithm_by_module(module_name, class_name)[source]
get_algorithm_categories()[source]

Retrieve all existent categories of Algorithms.

get_all_algorithms()[source]
get_all_operations_for_uploaders(project_id)[source]

Returns all finished upload operations.

get_applicable_adapters(compatible_class_names, launch_categ)[source]

Retrieve a list of algorithms in a given list of categories with a given dataType classes as required input.

get_category_by_id(categ_id)[source]

Retrieve category with given id

get_figure_count(project_id, user_id)[source]

Used to generate sequential image names.

get_filtered_operations(project_id, filter_chain, page_start=0, page_size=200, is_count=False)[source]
Parameters:
  • project_id – current project ID

  • filter_chain – instance of FilterChain

  • is_count – when True, return a number, otherwise the list of operation entities

:return a list of filtered operation in current project, page by page, or the total count for them.

get_launchable_categories(elimin_viewers=False)[source]

Retrieve algorithm categories which can be launched on right-click (optionally filter visualizers)

get_operation_by_gid(operation_gid)[source]

Retrieve OPERATION entity for a given gid.

get_operation_by_id(operation_id)[source]

Retrieve OPERATION entity for a given Identifier.

get_operation_lazy_by_gid(operation_gid)[source]

Retrieve OPERATION entity for a given gid.

get_operation_numbers(proj_id)[source]

Count total number of operations started for current project.

get_operation_process_for_operation(operation_id)[source]

Get the OperationProcessIdentifier for this operation id.

get_operationgroup_by_gid(gid)[source]

Retrieve by GID

get_operationgroup_by_id(op_group_id)[source]

Retrieve by ID

get_operations(status=None, algorithm_classname='SimulatorAdapter')[source]
get_operations_for_hpc_job()[source]
get_operations_in_group(operation_group_id, is_count=False, only_first_operation=False, only_gids=False)[source]

Retrieve OPERATION entities for a given group.

get_operations_with_error_in_project(project_id)[source]

Retrieve OPERATION with errors entities for a given project.

get_previews(project_id, user_id=None, selected_session_name='all_sessions')[source]

This method returns a tuple of 2 elements. The first element represents a dictionary of form {‘$session_name’: [list_of_figures]}. This dictionary contains data only for the selected self.session. If the selected session is ‘all_sessions’ than it will contain data for all the sessions. The second element of the returned tuple is a dictionary of form {‘$session_name’: $no_of_figures_in_this_session, …}. This dictionary contains information about all the sessions.

selected_session_name - represents the name of the session for which you

want to obtain the stored figures.

get_raw_categories()[source]

Retrieve categories with raw_input = true

get_results_for_operation(operation_id)[source]

Retrieve DataTypes entities, resulted after executing an operation.

get_uploader_categories()[source]

Retrieve categories with raw_input = true

get_visualisers_categories()[source]

Retrieve categories with display = true

is_upload_operation(operation_gid)[source]

Returns True only if the operation with the given gid is an upload operation.

load_figure(figure_id)[source]

Load a figure with all it’s lazy load fields to have all required info available.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>
set_operation_and_group_visibility(entity_gid, is_visible, is_operation_group=False)[source]

Sets the operation visibility.

If ‘is_operation_group’ is True than this method will change the visibility for all the operation from the OperationGroup with the GID field equal to ‘entity_gid’.

try_get_operation_by_id(operation_id)[source]

Try to call self.get_operation_by_id, but when operation was not found, instead of failing, return None. This could be called from situations like: stopping & removing op. A check for None is compulsory after this call!

project_dao

DAO operation related to Users and Projects are defined here.

class tvb.core.entities.storage.project_dao.CaseDAO[source]

Bases: RootDAO

USER and PROJECT RELATED OPERATIONS

add_members_to_project(proj_id, selected_user_ids)[source]

Add link between Users and Project.

compute_user_generated_disk_size(user_id)[source]

Do a SUM on DATA_TYPES table column DISK_SIZE, for the current user. :returns 0 when no DT are found, or SUM from DB.

count_projects_for_name(name, different_id)[source]

Retrieve the number of projects with a given name currently in DB.

count_users_for_name(name)[source]

Retrieve the number of users in DB for a given name.

delete_members_for_project(project_id, members)[source]

Remove all linked user to current project.

delete_project(project_id)[source]

Remove PROJECT entity by ID.

get_administrators()[source]

Retrieve all users with Admin role

get_all_projects(page_start=0, page_size=200, is_count=False)[source]

Retrieve all Project entities currently in the system. WARNING: use this wisely, as it might easily overflow the system.

get_all_users(different_names=None, page_start=0, page_size=200, is_count=False)[source]

Retrieve all USERS in DB, except given users and system user.

:return link between a given DT and a given project id

get_linkable_projects_for_user(user_id, data_id)[source]

Return all projects a given user can link some data given by a data_id to.

:return all links referring to a given project_id

get_members_of_project(proj_id)[source]

Retrieve USER entities with rights on current project.

get_project_by_gid(project_gid)[source]

Retrieve PROJECT entity for a given identifier. THROW SqlException when not found.

get_project_by_id(project_id)[source]

Retrieve PROJECT entity for a given identifier. THROW SqlException when not found.

get_project_by_name(project_name)[source]

Retrieve PROJECT entity for a given name. THROW SQLException when not found.

get_project_disk_size(project_id)[source]

Do a SUM on DATA_TYPES table column DISK_SIZE, for the current project. :returns 0 when no DT are found, or SUM from DB.

get_project_for_operation(operation_id)[source]

Find parent project for current operation. THROW SqlException when not found.

get_project_lazy_by_gid(project_gid)[source]

Retrieve PROJECT entity for a given identifier. THROW SqlException when not found.

get_projects_for_user(user_id, page_start=0, page_size=200, is_count=False)[source]

Return all projects a given user can access (administrator or not).

get_system_user()[source]

Retrieve System user from DB.

get_user_by_email(email, name_hint='')[source]

Find a user by email address and name.

Parameters:
  • email – Valid email address string, to search for its exact match in DB

  • name_hint – string for a user’s name; to search with like in DB

Returns:

None if none or more than one users matches the criteria.

get_user_by_gid(gid)[source]

Retrieve USER entity by gid.

get_user_by_id(user_id)[source]

Retrieve USER entity by ID.

get_user_by_name(name)[source]

Retrieve USER entity by name.

get_user_for_datatype(dt_id)[source]

Get the user who created a DT

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

root_dao

Base DAO behavior.

class tvb.core.entities.storage.root_dao.RootDAO[source]

Bases: object

GLOBAL METHODS

EXCEPTION_DATATYPE_GROUP = 'DataTypeGroup'
EXCEPTION_DATATYPE_SIMULATION = 'SimulationState'
get_datatype_by_id(data_id)[source]

Retrieve DataType entity by ID.

get_generic_entity(entity_type, filter_value, select_field='id')[source]

Retrieve an entity of entity_type, filtered by select_field = filter_value.

get_time_series_by_gid(data_gid)[source]
logger = <Logger tvb.core.entities.storage.root_dao (INFO)>
remove_datatype(gid)[source]

When removing dataType, load fully so that sql-alchemy removes from all tables referenced.

remove_entity(entity_class, entity_id)[source]

Find entity by Id and Type, end then remove it. Return True, when entity was removed successfully, of False when exception.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>
store_entities(entities_list)[source]

Store in DB a list of generic entities.

store_entity(entity, merge=False)[source]

Store in DB one generic entity.

session_maker

tvb.core.entities.storage.session_maker.MetaClassFactory(decorator_functions, new_attributes)[source]

A meta-class factory that creates a meta-class which makes sure a list of decorators are applied to all it’s classes and also adds a dictionary of attributes.

Parameters:
  • decorator_functions – a list of functions. These will be applied as decorators to all methods from the class that uses the returned meta-class.

  • new_attributes – a dictionary of attribute_name & attribute_value pairs that will be added to the class that uses the returned meta-class

tvb.core.entities.storage.session_maker.SESSION_META_CLASS

alias of MetaClass

tvb.core.entities.storage.session_maker.SessionMaker(*args, **kwargs)[source]

Called when a new instance is about to be created.

class tvb.core.entities.storage.session_maker.SessionsStack[source]

Bases: object

Helper class that holds a stack of SqlAlchemy’s session object and a counter that keeps track of how many transactions are opened.

close_session()[source]

Method called by all ‘add_session decorated methods. First check if there are any changes that needed to be committed but weren’t. Then either close the session if it’s not part of a transaction, or just expunge all objects otherwise.

close_transaction()[source]

Close a transaction. Make sure to commit beforehand so all changes are written to database. Then depending on if we are top level or not either close or expunge session.

property current_session

Property just for ease of access. Current session will always be top of stack.

open_session()[source]

Create a new session. If we are part of a transaction we bind it to the parent session, otherwise just create a new session.

rollback_transaction()[source]

RollBack a transaction. If we are part of nested transaction - rollback everything up to top parent transaction.

start_transaction()[source]

Start a new transaction. If this is top level transaction just created new session. Otherwise depending if we support nested or not, either raise exception or create a session bound to parent one.

tvb.core.entities.storage.session_maker.add_session(func)[source]

Decorator that handles session related precautions before/after method call. Before each new method a session is created that will later on be closed/rolled back as necessary. This is intended to be used on all DAO methods

tvb.core.entities.storage.session_maker.build_db_engine()[source]
tvb.core.entities.storage.session_maker.singleton(cls)[source]

Class decorator that makes sure only one instance of that class is ever returned.

tvb.core.entities.storage.session_maker.transactional(func)[source]

Decorator that makes sure that all DAO calls that will result from the decorated method will be encapsulated in a transaction that will be rolled back if any unexpected exceptions appear. This is intended to be used on service layer methods.

workflow_dao

DAO layer for WorkFlow and Burst entities.

class tvb.core.entities.storage.workflow_dao.WorkflowDAO[source]

Bases: RootDAO

DAO layer for WorkFlow and Burst entities.

get_dynamic(dyn_id)[source]
get_dynamic_by_name(name)[source]
get_dynamics_for_user(user_id)[source]
get_non_validated_entities(reference_time)[source]

Get a list of all categories and algorithm groups that were not found valid since the reference_time. Used in initializer on each start to filter out any entities that for some reason became invalid. :return tuple (list of entities to get invalidated) (list of entities to be removed)

session = <tvb.core.entities.storage.session_maker.SessionMaker object>