Jobflow is a free, open-source library for writing and executing workflows. Complex workflows can be defined using simple python functions and executed locally or on arbitrary computing resources using the FireWorks workflow manager.
Some features that distinguish jobflow are dynamic workflows, easy compositing and connecting of workflows, and the ability to store workflow outputs across multiple databases.
Is jobflow for me#
jobflow is intended to be a friendly workflow software that is easy to get started with, but flexible enough to handle complicated use cases.
Some of its features include:
A clean and flexible Python API.
A powerful approach to compositing and connecting workflows — information passing between jobs is a key goal of jobflow. Workflows can be nested allowing for a natural way to build complex workflows.
Integration with multiple databases (MongoDB, S3, GridFS, and more) through the Maggma package.
Support for the FireWorks workflow management system, allowing workflow execution on multicore machines or through a queue, on a single machine or multiple machines.
Support for dynamic workflows — workflows that modify themselves or create new ones based on what happens during execution.
Workflows in jobflows are made up of two main components:
Jobis an atomic computing job. Essentially any python function can be
Job, provided its input and return values can be serialized to json. Anything returned by the job is considered an “output” and is stored in the jobflow database.
Flowis a collection of
Jobobjects or other
Flowobjects. The connectivity between jobs is determined automatically from the job inputs. The execution order of jobs is automatically determined based on their connectivity.
Python functions can be easily converted in to
Job objects using the
In the example below, we define a job to add two numbers.
from jobflow import job, Flow @job def add(a, b): return a + b add_first = add(1, 5) add_second = add(add_first.output, 5) flow = Flow([add_first, add_second]) flow.draw_graph().show()
The output of the job is accessed using the
output attribute. As the job has not
yet been run,
output contains be a reference to a future output. Outputs can be used
as inputs to other jobs and will be automatically “resolved” before the job is
Finally, we created a flow using the two
Job objects. The connectivity between
the jobs is determined automatically and can be visualised using the flow graph.
The jobflow is a Python 3.7+ library and can be installed using pip.
pip install jobflow
Quickstart and tutorials#
To get a first glimpse of jobflow, we suggest that you follow our quickstart tutorial. Later tutorials delve into the advanced features of jobflow.
Track changes to jobflow through the changelog.
jobflow is released under a modified BSD license; the full text can be found here.
Jobflow was designed by Alex Ganose, Anubhav Jain, Gian-Marco Rignanese, David Waroquiers, and Guido Petretto. Alex Ganose implemented the first version of the package. Later versions have benefited from the contributions of several research groups. A full list of contributors is available here.