fireworks.user_objects.firetasks package

Submodules

fireworks.user_objects.firetasks.dataflow_tasks module

This module includes dataflow firetask tasks

class fireworks.user_objects.firetasks.dataflow_tasks.CommandLineTask(*args, **kwargs)

Bases: fireworks.core.firework.FireTaskBase

A Firetask to execute external commands in a shell

Required params:
  • command_spec (dict): a dictionary specification of the command (see below for details)
Optional params:
  • inputs ([str]): list of labels, one for each input argument
  • outputs ([str]): list of labels, one for each output argument
  • chunk_number (int): the serial number of the firetask when it is part of a series generated by a ForeachTask
command_spec = {
‘command’: [str], # mandatory label_1: dict_1, # optional label_2: dict_2, # optional …

} The ‘command’ is a representation of the command as to be used with subprocess package. The optional keys label_1, label_2, etc. are the actual labels used in the inputs and outputs. The dictionaries dict_1, dict_2, etc. have the following schema: {

‘binding’: {
prefix: str or None, separator: str or None

}, ‘source’: {

‘type’: ‘path’ or ‘data’ or ‘identifier’
or ‘stdin’ or ‘stdout’ or ‘stderr’ or None,

‘value’: str or int or float

}, ‘target’: {

‘type’: ‘path’ or ‘data’ or ‘identifier’
or ‘stdin’ or ‘stdout’ or ‘stderr’ or None,

‘value’: str

}

}

  • If the ‘type’ in the ‘source’ field is ‘data’ the ‘value’ can be of

types ‘str’, ‘int’ and ‘float’.

  • When a str is found instead of dict for some ‘source’, for example

‘source’: ‘string’, ‘string’ is replaced with spec[‘string’] which must be available and of the schema of the ‘source’.

  • When a str is found instead of dict for some label, for example

label: ‘string’, ‘string’ is replaced with spec[‘string’] which can be a dictionary with this schema or a list of such dictionaries.

static command_line_tool(inputs=None, outputs=None)

This function composes and executes a command from provided specifications.

Required parameters:
  • command ([str]): the command as to be passed to subprocess.Popen
Optional parameters:
  • inputs ([dict, [dict]]): list of the specifications for inputs; multiple inputs may be passed in one list of dictionaries
  • outputs ([dict]): list of the specifications for outputs
Returns:
‘target’: {
‘type’: ‘path’ or ‘data’ or ‘identifier’
or ‘stdin’ or ‘stdout’ or ‘stderr’ or None

’value’: str

}

If outputs is None then an empty list is returned.

Return type:
  • list of target dictionaries for each output
optional_params = ['inputs', 'outputs', 'chunk_number']
required_params = ['command_spec']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.dataflow_tasks.ForeachTask(*args, **kwargs)

Bases: fireworks.core.firework.FireTaskBase

This firetask branches the workflow creating parallel fireworks using FWAction: one firework for each element or each chunk from the split list. Each firework in this generated list contains the firetask specified in the task dictionary. If the number of chunks is specified the split list will be divided into this number of chunks and each chunk will be processed by one of the generated child fireworks.

Required params:
  • task (dict): a dictionary version of the firetask
  • split (str): a label of an input list; it must be available both in the inputs list of the specified task and in the spec.
Optional params:
  • number of chunks (int): if provided the split input list will be divided into this number of sublists and each will be processed by a separate child firework
optional_params = ['number of chunks']
required_params = ['task', 'split']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.dataflow_tasks.ImportDataTask(*args, **kwargs)

Bases: fireworks.core.firework.FireTaskBase

Update the spec with data from file in a nested dictionary at a position specified by a mapstring = maplist[0]/maplist[1]/… i.e. spec[maplist[0]][maplist[1]]… = data

optional_params = []
required_params = ['filename', 'mapstring']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.dataflow_tasks.JoinDictTask(*args, **kwargs)

Bases: fireworks.core.firework.FireTaskBase

combines specified spec fields into a dictionary

optional_params = ['rename']
required_params = ['inputs', 'output']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.dataflow_tasks.JoinListTask(*args, **kwargs)

Bases: fireworks.core.firework.FireTaskBase

combines specified spec fields into a list.

required_params = ['inputs', 'output']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)

fireworks.user_objects.firetasks.fileio_tasks module

class fireworks.user_objects.firetasks.fileio_tasks.ArchiveDirTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

Wrapper around shutil.make_archive to make tar archives.

Parameters:
  • base_name (str) – Name of the file to create, including the path, minus any format-specific extension.
  • format (str) – Optional. one of “zip”, “tar”, “bztar” or “gztar”. Defaults to gztar.
optional_params = ['format']
required_params = ['base_name']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.fileio_tasks.CompressDirTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

Compress all files in a directory.

Parameters:
  • dest (str) – Optional. Path to compress.
  • compression (str) – Optional. Can only be gz or bz2. Defaults to gz.
  • ignore_errors (bool) – Optional. Whether to ignore errors. Defaults to False.
optional_params = ['compression', 'dest', 'ignore_errors']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.fileio_tasks.DecompressDirTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

Decompress all files in a directory. Autodetects gz, bz2 and z file extensions.

Parameters:
  • dest (str) – Optional. Path to decompress.
  • ignore_errors (bool) – Optional. Whether to ignore errors. Defaults to False.
optional_params = ['dest', 'ignore_errors']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.fileio_tasks.FileDeleteTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

A Firetask to delete files:

Required params:
  • files_to_delete: ([str]) Filenames to delete
Optional params:
  • dest: (str) Shared path for files
  • ignore_errors (bool): Whether to ignore errors. Defaults to True.
required_params = ['files_to_delete']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.fileio_tasks.FileTransferTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

A Firetask to Transfer files. Note that

Required params:
  • mode: (str) - move, mv, copy, cp, copy2, copytree, copyfile, rtransfer
  • files: ([str]) or ([(str, str)]) - list of source files, or dictionary containing
    ‘src’ and ‘dest’ keys
  • dest: (str) destination directory, if not specified within files parameter (else optional)
Optional params:
  • server: (str) server host for remote transfer
  • user: (str) user to authenticate with on remote server
  • key_filename: (str) optional SSH key location for remote transfer
  • max_retry: (int) number of times to retry failed transfers; defaults to 0 (no retries)
  • retry_delay: (int) number of seconds to wait between retries; defaults to 10
fn_list = {'copy': <function copy at 0x108db0f28>, 'copy2': <function copy2 at 0x108db6048>, 'copyfile': <function copyfile at 0x108db0d08>, 'copytree': <function copytree at 0x108db6158>, 'cp': <function copy at 0x108db0f28>, 'move': <function move at 0x108db6400>, 'mv': <function move at 0x108db6400>}
optional_params = ['server', 'user', 'key_filename', 'max_retry', 'retry_delay']
required_params = ['mode', 'files']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.fileio_tasks.FileWriteTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

A Firetask to write files:

Required params:
  • files_to_write: ([{filename:(str), contents:(str)}]) List of dicts with filenames
    and contents
Optional params:
  • dest: (str) Shared path for files
required_params = ['files_to_write']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)

fireworks.user_objects.firetasks.filepad_tasks module

class fireworks.user_objects.firetasks.filepad_tasks.AddFilesTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

A Firetask to add files to the filepad.

Required params:
  • paths (list/str): either list of paths or a glob pattern string.
Optional params:
  • identifiers ([str]): list of identifiers, one for each file in the paths list
  • directory (str): path to directory where the pattern matching is to be done.
  • filepad_file (str): path to the filepad db config file
  • compress (bool): whether or not to compress the file before inserting to gridfs
  • metadata (dict): metadata to store along with the file, stored in ‘metadata’ key
optional_params = ['identifiers', 'directory', 'filepad_file', 'compress', 'metadata']
required_params = ['paths']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.filepad_tasks.DeleteFilesTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

A Firetask to delete files from the filepad

Required params:
  • identifiers ([str]): identifiers of files to delete
Optional params:
  • filepad_file (str): path to the filepad db config file
optional_params = ['filepad_file']
required_params = ['identifiers']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.filepad_tasks.GetFilesTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

A Firetask to fetch files from the filepad and write it to specified directory(current working directory if not specified)

Required params:
  • identifiers ([str]): identifiers of files to fetch
Optional params:
  • filepad_file (str): path to the filepad db config file
  • dest_dir (str): destination directory, default is the current working directory
  • new_file_names ([str]): if provided, the retrieved files will be renamed
optional_params = ['filepad_file', 'dest_dir', 'new_file_names']
required_params = ['identifiers']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
fireworks.user_objects.firetasks.filepad_tasks.get_fpad(fpad_file)

fireworks.user_objects.firetasks.script_task module

This module includes tasks to integrate scripts and python functions

class fireworks.user_objects.firetasks.script_task.PyTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

Runs any python function! Extremely powerful, which allows you to essentially run any accessible method on the system. The optional inputs and outputs lists may contain spec keys to add to args list and to make the function output available in the curent and in children fireworks.

Required parameters:
  • func (str): Fully qualified python method. E.g., json.dump, or shutil
    .copy, or some other function that is not part of the standard library!
Optional parameters:
  • args (list): List of args. Default is empty.
  • kwargs (dict): Dictionary of keyword args. Default is empty.
  • auto_kwargs (bool): If True, all other params not starting with ‘_’ are supplied as keyword args
  • stored_data_varname (str): Whether to store the output in FWAction. If this is a string that does not evaluate to False, the output of the function will be stored as FWAction(stored_data={stored_data_varname: output}). The name is deliberately long to avoid potential name conflicts.
  • inputs ([str]): a list of keys in spec which will be used as inputs; the generated arguments list will be appended to args
  • outputs ([str]): a list of spec keys that will be used to pass the function’s outputs to child fireworks
  • chunk_number (int): a serial number of the Firetask within a group of Firetasks generated by a ForeachTask
optional_params = ['args', 'kwargs', 'auto_kwargs', 'stored_data_varname', 'inputs', 'outputs', 'chunk_number']
required_params = ['func']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)
class fireworks.user_objects.firetasks.script_task.ScriptTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

Runs a user-defined script

classmethod from_str(shell_cmd, parameters=None)
required_params = ['script']
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)

fireworks.user_objects.firetasks.templatewriter_task module

class fireworks.user_objects.firetasks.templatewriter_task.TemplateWriterTask(*args, **kwargs)

Bases: fireworks.core.firework.FiretaskBase

Task to write templated files via Jinja2 library Required parameters:

  • template_file: (str) - path to template file
  • context: (dict) - variable replacements for the template file
  • output_file: (str) - output file
Optional parameters:
  • append: (bool) - append to output file (instead of overwrite)
  • template_dir: (str) - directory in which to find the template file
run_task(fw_spec)

This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources.
Returns:(FWAction)

fireworks.user_objects.firetasks.unittest_tasks module

class fireworks.user_objects.firetasks.unittest_tasks.ExportTestSerializer(a)

Bases: fireworks.utilities.fw_serializers.FWSerializable

__init__(a)

Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(m_dict)
to_dict()
class fireworks.user_objects.firetasks.unittest_tasks.TestSerializer(a, m_date)

Bases: fireworks.utilities.fw_serializers.FWSerializable

__init__(a, m_date)

Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(m_dict)
to_dict(*args, **kwargs)

Module contents