fireworks.queue package

Submodules

fireworks.queue.queue_adapter module

This module contains contracts for defining adapters to various queueing systems, e.g. PBS/SLURM/SGE.

class fireworks.queue.queue_adapter.Command(command)

Bases: object

Helper class - run subprocess commands in a different thread with TIMEOUT option. From https://gist.github.com/kirpit/1306188 Based on jcollado’s solution: http://stackoverflow.com/questions/1191374/subprocess-with-timeout/4825933#4825933

__init__(command)

initialize the object.

Parameters

command – command to run

command = None
error = ''
output = ''
process = None
run(timeout=None, **kwargs)

Run the command.

Parameters

  • timeout (float) – timeout

  • kwargs (dict) –

Returns

(status, output, error)

status = None
class fireworks.queue.queue_adapter.QScriptTemplate(template)

Bases: string.Template

delimiter = '$$'
pattern = re.compile('\n \\$\\$(?:\n (?P<escaped>\\$\\$) | # Escape sequence of two delimiters\n (?P<named>(?a:[_a-z][_a-z0-9]*)) | # delimiter and a Python identifier\n {(?P<braced>(?a:[_a-z][_, re.IGNORECASE|re.VERBOSE)
class fireworks.queue.queue_adapter.QueueAdapterBase

Bases: collections.defaultdict, fireworks.utilities.fw_serializers.FWSerializable

The QueueAdapter is responsible for all interactions with a specific queue management system. This includes handling all details of queue script format as well as queue submission and

management.

A user should extend this class with implementations that work on specific queue systems. Examples and implementations are in: fireworks/user_objects/queue_adapters.

Documentation on implementing queue adapters can be found on FireWorks home page, https://materialsproject.github.io/fireworks

defaults = {}
classmethod from_dict(m_dict)
abstract get_njobs_in_queue(username=None)

Returns the number of jobs currently in the queue for the user.

Parameters

username (str) – the username of the jobs to count (default is to autodetect)

Returns

(int) number of jobs in the queue

get_qlogger(name)
get_script_str(launch_dir)

returns a (multi-line) String representing the queue script, e.g. PBS script. Uses the template_file along with internal parameters to create the script.

Parameters

launch_dir (str) – The directory the job will be launched in

Returns

(str) the queue script

q_name = 'OVERRIDE_ME'
submit_cmd = 'OVERRIDE_ME'
abstract submit_to_queue(script_file)

Submits the job to the queue and returns the job id.

Parameters

script_file – (str) name of the script file to use (String)

Returns

(int) job_id

template_file = 'OVERRIDE_ME'
to_dict(*args, **kwargs)

fireworks.queue.queue_launcher module

This module is used to submit jobs to a queue on a cluster. It can submit a single job, or if used in “rapid-fire” mode, can submit multiple jobs within a directory structure. The details of job submission and queue communication are handled using Queueadapter, which specifies a QueueAdapter as well as desired properties of the submit script.

fireworks.queue.queue_launcher.launch_rocket_to_queue(launchpad, fworker, qadapter, launcher_dir='.', reserve=False, strm_lvl='INFO', create_launcher_dir=False, fill_mode=False, fw_id=None)

Submit a single job to the queue.

Parameters

  • launchpad (LaunchPad) –

  • fworker (FWorker) –

  • qadapter (QueueAdapterBase) –

  • launcher_dir (str) – The directory where to submit the job

  • reserve (bool) – Whether to queue in reservation mode

  • strm_lvl (str) – level at which to stream log messages

  • create_launcher_dir (bool) – Whether to create a subfolder launcher+timestamp, if needed

  • fill_mode (bool) – whether to submit jobs even when there is nothing to run (only in non-reservation mode)

  • fw_id (int) – specific fw_id to reserve (reservation mode only)

fireworks.queue.queue_launcher.rapidfire(launchpad, fworker, qadapter, launch_dir='.', block_dir=None, nlaunches=0, njobs_queue=0, njobs_block=500, sleep_time=None, reserve=False, strm_lvl='INFO', timeout=None, fill_mode=False)

Submit many jobs to the queue.

Parameters

  • launchpad (LaunchPad) –

  • fworker (FWorker) –

  • qadapter (QueueAdapterBase) –

  • launch_dir (str) – directory where we want to write the blocks

  • block_dir (str) – directory to use as block dir. Can be a new or existing block. Dirname must start with ‘block_’.

  • nlaunches (int) – total number of launches desired; “infinite” for loop, 0 for one round

  • njobs_queue (int) – stops submitting jobs when njobs_queue jobs are in the queue, 0 for no limit. If 0 skips the check on the number of jobs in the queue.

  • njobs_block (int) – automatically write a new block when njobs_block jobs are in a single block

  • sleep_time (int) – secs to sleep between rapidfire loop iterations

  • reserve (bool) – Whether to queue in reservation mode

  • strm_lvl (str) – level at which to stream log messages

  • timeout (int) – # of seconds after which to stop the rapidfire process

  • fill_mode (bool) – whether to submit jobs even when there is nothing to run (only in non-reservation mode)

fireworks.queue.queue_launcher.setup_offline_job(launchpad, fw, launch_id)

Module contents