fireworks.utilities package

Submodules

fireworks.utilities.dagflow module

fireworks.utilities.dict_mods module

This module allows you to modify a dict (a spec) using another dict (an instruction). The main method of interest is apply_dictmod().

This code is based heavily on the Ansible class of custodian <https://pypi.python.org/pypi/custodian>, but simplifies it considerably for the limited use cases required by FireWorks.

fireworks.utilities.dict_mods.apply_mod(modification, obj)

Note that modify makes actual in-place modifications. It does not return a copy.

Parameters

  • modification – Modification must be {action_keyword : settings}, where action_keyword is a supported DictMod

  • obj – A dict to be modified

fireworks.utilities.dict_mods.arrow_to_dot(input_dict)

Converts arrows (‘->’) in dict keys to dots ‘.’ recursively. Allows for storing MongoDB neseted document queries in MongoDB.

Parameters

input_dict (dict) –

Returns

dict

fireworks.utilities.dict_mods.get_nested_dict(input_dict, key)

fireworks.utilities.filepad module

This module defines the FilePad class. FilePad is a convenient api to mongodb/gridfs to add/delete/update any file of any size.

class fireworks.utilities.filepad.FilePad(host='localhost', port=27017, database='fireworks', username=None, password=None, authsource=None, uri_mode=False, mongoclient_kwargs=None, filepad_coll_name='filepad', gridfs_coll_name='filepad_gfs', logdir=None, strm_lvl=None, text_mode=False)

Bases: monty.json.MSONable

__init__(host='localhost', port=27017, database='fireworks', username=None, password=None, authsource=None, uri_mode=False, mongoclient_kwargs=None, filepad_coll_name='filepad', gridfs_coll_name='filepad_gfs', logdir=None, strm_lvl=None, text_mode=False)
Parameters

  • host (str) – hostname

  • port (int) – port number

  • database (str) – database name

  • username (str) –

  • password (str) –

  • authsource (str) – authSource parameter for MongoDB authentication; defaults to “name” (i.e., db name) if not set

  • uri_mode (bool) – if set True, all Mongo connection parameters occur through a MongoDB URI string (set as the host).

  • filepad_coll_name (str) – filepad collection name

  • gridfs_coll_name (str) – gridfs collection name

  • logdir (str) – path to the log directory

  • strm_lvl (str) – the logger stream level

  • text_mode (bool) – whether to use text_mode for file read/write (instead of binary). Might be useful if working only with text files between Windows and Unix systems

add_file(path, identifier=None, compress=True, metadata=None)

Insert the file specified by the path into gridfs. The gridfs id and identifier are returned. Note: identifier must be unique, i.e, no insertion if the identifier already exists in the db.

Parameters

  • path (str) – path to the file

  • identifier (str) – file identifier. If identifier = None then the identifier is set to the object id returned by gridfs insertion.

  • compress (bool) – compress or not

  • metadata (dict) – file metadata

Returns

the id returned by gridfs, identifier

Return type

(str, str)

classmethod auto_load()

Returns FilePad object

build_indexes(indexes=None, background=True)

Build the indexes.

Parameters

  • indexes (list) – list of single field indexes to be built.

  • background (bool) – Run in the background or not.

count(filter=None, **kwargs)

Get the number of documents in filepad.

Parameters

  • filter (dict) –

  • kwargs (dict) – see pymongo.Collection.count for the supported keyword arguments.

Returns

the count

Return type

int

delete_file(identifier)

Delete the document with the matching identifier. The contents in the gridfs as well as the associated document in the filepad are deleted.

Parameters

identifier (str) – the file identifier

delete_file_by_id(gfs_id)
Parameters

gfs_id (str) – the file id

delete_file_by_query(query)
Parameters

query (dict) – pymongo query dict

classmethod from_db_file(db_file, admin=True)
Parameters

db_file (str) – path to the filepad cred file

Returns

FilePad object

get_file(identifier)

Get file by identifier

Parameters

identifier (str) – the file identifier

Returns

the file content as a string, document dictionary

Return type

(str, dict)

get_file_by_id(gfs_id)
Parameters

gfs_id (str) – the gridfs id for the file.

Returns

the file content as a string, document dictionary

Return type

(str, dict)

get_file_by_query(query, sort_key=None, sort_direction=- 1)
Parameters

  • query (dict) – pymongo query dict

  • sort_key (str, optional) – sort key, default None

  • sort_direction (int, optional) – default pymongo.DESCENDING

Returns

list of all (file content as a string, document dictionary)

Return type

list

reset()

Reset filepad and the gridfs collections

update_file(identifier, path, compress=True)

Update the filecontents in the gridfs, update the gfs_id in the document and retain the rest.

Parameters

  • identifier (str) – the unique file identifier

  • path (str) – path to the new file whose contents will replace the existing one.

  • compress (bool) – whether or not to compress the contents before inserting to gridfs

Returns

old file id , new file id

Return type

(str, str)

update_file_by_id(gfs_id, path, compress=True)

Update the file in the gridfs with the given id and retain the rest of the document.

Parameters

  • gfs_id (str) – the gridfs id for the file.

  • path (str) – path to the new file whose contents will replace the existing one.

  • compress (bool) – whether or not to compress the contents before inserting to gridfs

Returns

old file id , new file id

Return type

(str, str)

fireworks.utilities.fw_serializers module

This module aids in serializing and deserializing objects.

To serialize a FW object, refer to the documentation for the FWSerializable class. To de-serialize an object, refer to the documentation for the FWSerializable class and load_object() method.

  • you can de-serialize explicitly, e.g. FWObject.from_dict() to enforce a FWObject instance as the result

  • you can de-serialize implicitly, e.g. load_object() to search for the correct Class dynamically

The implicit method is especially useful if you don’t know in advance which subclass of a base class your serialization might point to. e.g. if you require some type of Quadrilateral, a serialization from your collaborator might point to a Square, Rhombus, or Rectangle, and you might not know which one in advance…

Some advantages:
  • Robust with regard to code refactorings even in implicit loading given certain reasonable

    guidelines on fw_name.

  • Simple to allow a superclass to define all the serializations for its subclasses, removing

    code repetition(in particular, note that from_dict is a class method rather than a static method, allowing use of self)

  • Decorators aid in some of the routine parts of the serialization, such as adding the _fw_name key

  • Both JSON and YAML file import/export are naturally and concisely supported within the framework.

  • Auto-detect and proper loading of JSON and YAML files

  • Proper JSON handling of datetime (both encoding and decoding) and UTF-8 strings

  • In some cases, objects can be serialized/deserialized extremely concisely, by use of only

    their fw_name (if no parameters are needed to describe the object)

fireworks.utilities.fw_serializers.DATETIME_HANDLER(obj)
class fireworks.utilities.fw_serializers.FWSerializable

Bases: object

To create a serializable object within FireWorks, you should subclass this class and implement the to_dict() and from_dict() methods.

If you want the load_object() implicit de-serialization to work, you must also:

  • Use the @serialize_fw decorator on to_dict()

  • Override the _fw_name parameter with a unique key.

See documentation of load_object() for more details.

The use of @classmethod for from_dict allows you to define a superclass that implements the to_dict() and from_dict() for all its subclasses.

For an example of serialization, see the class QueueAdapterBase.

as_dict()
abstract classmethod from_dict(m_dict)
classmethod from_file(filename, f_format=None)

Load a serialization of this object from a file.

Parameters

  • filename (str) – filename to read

  • f_format (str) – serialization format, default checks the filename extension

Returns

FWSerializable

classmethod from_format(f_str, f_format='json')

convert from a String representation to its Object.

Parameters

  • f_str (str) – the String representation

  • f_format (str) – serialization format of the String (default json)

Returns

FWSerializable

property fw_name
to_db_dict()
abstract to_dict()
to_file(filename, f_format=None, **kwargs)

Write a serialization of this object to a file.

Parameters

  • filename (str) – filename to write to

  • f_format (str) – serialization format, default checks the filename extension

to_format(f_format='json', **kwargs)

returns a String representation in the given format

Parameters

f_format (str) – the format to output to (default json)

fireworks.utilities.fw_serializers.get_default_serialization(cls)
fireworks.utilities.fw_serializers.load_object(obj_dict)

Creates an instantiation of a class based on a dictionary representation. We implicitly determine the Class through introspection along with information in the dictionary.

We search for a class with the _fw_name property equal to obj_dict[‘_fw_name’] If the @module key is set, that module is checked first for a matching class to improve speed of lookup. Afterwards, the modules in the USER_PACKAGES global parameter are checked.

Refactoring class names, module names, etc. will not break object loading as long as:

  1. the _fw_name property is maintained the same AND

  2. the refactored module is kept within USER_PACKAGES

You can get around these limitations if you really want: i) If you want to change the fw_name of an object you can set the FW_NAME_UPDATES key ii) If you want to put a refactored module in a new place add an entry to USER_PACKAGES

Parameters

obj_dict (dict) – the dict representation of the class

fireworks.utilities.fw_serializers.load_object_from_file(filename, f_format=None)

Implicitly load an object from a file. just a friendly wrapper to load_object()

Parameters

  • filename (str) – the filename to load an object from

  • f_format (str) – the serialization format (default is auto-detect based on filename extension)

fireworks.utilities.fw_serializers.reconstitute_dates(obj_dict)
fireworks.utilities.fw_serializers.recursive_deserialize(func)

a decorator to add FW serializations keys see documentation of FWSerializable for more details

fireworks.utilities.fw_serializers.recursive_dict(obj, preserve_unicode=True)
fireworks.utilities.fw_serializers.recursive_serialize(func)

a decorator to add FW serializations keys see documentation of FWSerializable for more details

fireworks.utilities.fw_serializers.serialize_fw(func)

a decorator to add FW serializations keys see documentation of FWSerializable for more details

fireworks.utilities.fw_utilities module

class fireworks.utilities.fw_utilities.DataServer(address=None, authkey=None, serializer='pickle', ctx=None)

Bases: multiprocessing.managers.BaseManager

Provide a server that can host shared objects between multiprocessing Processes (that normally can’t share data). For example, a common LaunchPad is shared between processes and pinging launches is coordinated to limit DB hits.

classmethod setup(launchpad)
Parameters

launchpad (LaunchPad) –

Returns

DataServer

class fireworks.utilities.fw_utilities.NestedClassGetter

Bases: object

Used to help pickle inner classes, e.g. see Workflow.Links When called with the containing class as the first argument, and the name of the nested class as the second argument, returns an instance of the nested class.

fireworks.utilities.fw_utilities.create_datestamp_dir(root_dir, l_logger, prefix='block_')

Internal method to create a new block or launcher directory. The dir name is based on the time and the FW_BLOCK_FORMAT

Parameters

  • root_dir – directory to create the new dir in

  • l_logger – the logger to use

  • prefix – the prefix for the new dir, default=”block_”

fireworks.utilities.fw_utilities.explicit_serialize(o)
fireworks.utilities.fw_utilities.get_fw_logger(name: str, l_dir: None = None, file_levels: Tuple[str, str] = ('DEBUG', 'ERROR'), stream_level: str = 'DEBUG', formatter: logging.Formatter = <logging.Formatter object>, clear_logs: bool = False) → logging.Logger

Convenience method to return a logger.

Parameters

  • name – name of the logger that sets the groups, e.g. ‘group1.set2’

  • l_dir – the directory to put the log file

  • file_levels – iterable describing level(s) to log to file(s). default: (‘DEBUG’, ‘ERROR’)

  • stream_level – level to log to standard output. default: ‘DEBUG’

  • formatter – logging format. default: FW_LOGGING_FORMATTER

  • clear_logs – whether to clear the logger with the same name

fireworks.utilities.fw_utilities.get_my_host()
fireworks.utilities.fw_utilities.get_my_ip()
fireworks.utilities.fw_utilities.get_slug(m_str)
fireworks.utilities.fw_utilities.log_exception(m_logger, msgs)

A shortcut wrapper around log_fancy for exceptions

Parameters

  • m_logger (logger) – The logger object

  • msgs ([str]) – String or iterable of Strings, will be joined by newlines

fireworks.utilities.fw_utilities.log_fancy(m_logger, msgs, log_lvl='info', add_traceback=False)

A wrapper around the logger messages useful for multi-line logs. Helps to group log messages by adding a fancy border around it, which enhances readability of log lines meant to be read as a unit.

Parameters

  • m_logger (logger) – The logger object

  • log_lvl (str) – The level to log at

  • msgs ([str]) – a String or iterable of Strings

  • add_traceback (bool) – add traceback text, useful when logging exceptions (default False)

fireworks.utilities.fw_utilities.log_multi(m_logger, msg, log_lvl='info')
Parameters

  • m_logger (logger) – The logger object

  • msg (str) – a String to log

  • log_lvl (str) – The level to log at

fireworks.utilities.fw_utilities.redirect_local()

temporarily redirect stdout or stderr to fws.error and fws.out

fireworks.utilities.update_collection module

fireworks.utilities.update_collection.update_launchpad_data(lp, replacements, **kwargs)

If you want to update a text string in your entire FireWorks database with a replacement, use this method. For example, you might want to update a directory name preamble like “/scratch/user1” to “/project/user2”. The algorithm does a text replacement over the entire BSON document. The original collection is backed up within the database with extension “_xiv_{Date}”.

Parameters

  • (LaunchPad) (lp) – a FireWorks LaunchPad object

  • (dict) (replacements) – e.g. {“old_path1”: “new_path1”, “scratch/”:”project/”}

  • kwargs – Additional arguments accepted by the update_path_in_collection method

fireworks.utilities.update_collection.update_path_in_collection(db, collection_name, replacements, query=None, dry_run=False, force_clear=False)

updates the text specified in replacements for the documents in a MongoDB collection. This can be used to mass-update an outdated value (e.g., a directory path or tag) in that collection. The algorithm does a text replacement over the entire BSON document. The original collection is backed up within the database with extension “_xiv_{Date}”.

Parameters

  • db (Database) – MongoDB db object

  • collection_name (str) – name of a MongoDB collection to update

  • replacements (dict) – e.g. {“old_path1”: “new_path1”, “scratch/”:”project/”}

  • query (dict) – criteria for query, default None if you want all documents to be updated

  • dry_run (bool) – if True, only a new collection with new paths is created and original “collection” is not replaced

  • force_clear (bool) – careful! If True, the collection f”{collection}_temp_refactor” is removed!

Returns

None, but if dry_run==False it replaces the collection with the updated one

fireworks.utilities.visualize module

fireworks.utilities.visualize.plot_wf(wf, depth_factor=1.0, breadth_factor=2.0, labels_on=True, numerical_label=False, text_loc_factor=1.0, save_as=None, style='rD--', markersize=10, markerfacecolor='blue', fontsize=12)

Generate a visual representation of the workflow. Useful for checking whether the firework connections are in order before launching the workflow.

Parameters

  • wf (Workflow) – workflow object.

  • depth_factor (float) – adjust this to stretch the plot in y direction.

  • breadth_factor (float) – adjust this to stretch the plot in x direction.

  • labels_on (bool) – whether to label the nodes or not. The default is to label the nodes using the firework names.

  • numerical_label (bool) – set this to label the nodes using the firework ids.

  • text_loc_factor (float) – adjust the label location.

  • save_as (str) – save the figure to the given name.

  • style (str) – marker style.

  • markersize (int) – marker size.

  • markerfacecolor (str) – marker face color.

  • fontsize (int) – font size for the node label.

fireworks.utilities.visualize.wf_to_graph(wf: fireworks.core.firework.Workflow, dag_kwargs: Dict[str, Any] = {}, wf_show_tasks: bool = True) → None

Renders a graph representation of a workflow or firework. Workflows are rendered as the control flow of the firework, while Fireworks are rendered as a sequence of Firetasks.

Copied from https://git.io/JO6L8.

Parameters

  • workflow (Workflow | Firework) – Workflow or Firework to be rendered.

  • dag_kwargs (dict[str, Any]) – Arguments passed to Digraph.attr(). Defaults to {}.

  • wf_show_tasks (bool) – When rendering a Workflow, whether to show each Firetask in the graph. Defaults to False.

Returns

The Workflow or Firework directed acyclic graph.

Return type

Digraph

Module contents