This page is available as a Jupyter notebook: tutorials/6-makers.ipynb.

Makers

In this tutorial, you will:

• Learn about Makers.

• Understand how to use a Maker to update the parameters of jobs in a flow.

What is a Maker?

A Maker is class that makes it convenient to update parameters on-the-fly in a workflow. It is best explained by example.

Let’s start by defining a simple Maker that either adds or multiplies two numbers together, which we will do twice to make a flow. Note that all classes inheriting from the Maker base class must have a name variable and a make method.

from dataclasses import dataclass

from jobflow import Flow, Maker, job
from jobflow.managers.local import run_locally


@dataclass
class AddMaker(Maker):
    name: str = "Add Maker"
    operation: str = "add"

    @job
    def make(self, a, b):
        if self.operation == "add":
            return a + b
        if self.operation == "mult":
            return a * b
        raise ValueError(f"Unknown operation: {self.operation}")


job1 = AddMaker().make(a=2, b=3)
job2 = AddMaker().make(a=job1.output, b=4)

flow = Flow([job1, job2])
responses = run_locally(flow)

2023-06-08 09:57:41,477 INFO Started executing jobs locally
2023-06-08 09:57:41,569 INFO Starting job - Add Maker (9d9db153-0d05-46ac-af3d-e4ac82b21134)
2023-06-08 09:57:41,570 INFO Finished job - Add Maker (9d9db153-0d05-46ac-af3d-e4ac82b21134)
2023-06-08 09:57:41,570 INFO Starting job - Add Maker (8473bc55-dc4d-4eb3-ab33-8eb4a57a9b75)
2023-06-08 09:57:41,571 INFO Finished job - Add Maker (8473bc55-dc4d-4eb3-ab33-8eb4a57a9b75)
2023-06-08 09:57:41,571 INFO Finished executing jobs locally

for uuid, response in responses.items():
    print(f"{uuid} -> {response}")

9d9db153-0d05-46ac-af3d-e4ac82b21134 -> {1: Response(output=5, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False)}
8473bc55-dc4d-4eb3-ab33-8eb4a57a9b75 -> {1: Response(output=9, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False)}

Right now, nothing particularly special has happened here. But what if you had a much more complex workflow with many steps and you wanted to change the AddMaker keyword arguments, but only for a few individual jobs in the flow? That’s where the Maker comes in handy. Let’s see how it works.

from dataclasses import dataclass

from jobflow import Flow, Maker, job
from jobflow.managers.local import run_locally


@dataclass
class AddMaker(Maker):
    name: str = "Add Maker"
    operation: str = "add"

    @job
    def make(self, a, b):
        if self.operation == "add":
            return a + b
        if self.operation == "mult":
            return a * b
        raise ValueError(f"Unknown operation: {self.operation}")


@dataclass
class SubtractMaker(Maker):
    name: str = "Subtract Maker"

    @job
    def make(self, a, b):
        return b - a


job1 = AddMaker().make(a=2, b=3)
job2 = SubtractMaker().make(a=job1.output, b=4)
flow = Flow([job1, job2])

flow.update_maker_kwargs({"operation": "mult"}, name_filter="Add Maker")
responses = run_locally(flow)

2023-06-08 09:57:41,583 INFO Started executing jobs locally
2023-06-08 09:57:41,584 INFO Starting job - Add Maker (3fa685ae-8221-4a14-be87-afd5bc00121b)
2023-06-08 09:57:41,584 INFO Finished job - Add Maker (3fa685ae-8221-4a14-be87-afd5bc00121b)
2023-06-08 09:57:41,585 INFO Starting job - Subtract Maker (2cd6bf8d-74b1-4485-98f6-b5455e0432fb)
2023-06-08 09:57:41,585 INFO Finished job - Subtract Maker (2cd6bf8d-74b1-4485-98f6-b5455e0432fb)
2023-06-08 09:57:41,586 INFO Finished executing jobs locally

for uuid, response in responses.items():
    print(f"{uuid} -> {response}")

3fa685ae-8221-4a14-be87-afd5bc00121b -> {1: Response(output=6, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False)}
2cd6bf8d-74b1-4485-98f6-b5455e0432fb -> {1: Response(output=-2, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False)}

In this example, we have updated the keyword arguments (“kwargs”) of jobs in a flow using a name_filter and the update_maker_kwargs function, which functions because the classes in the flow are themselves Maker objects.

Of course, we could have simply done job1 = AddMaker(operation="mult").make(a=2, b=3) to begin with, but in practice if you were to have instead impotred this flow from some external Python package, you might not be able to modify the AddMaker class directly. In this case, the Maker class provides a convenient way to update the parameters of jobs in a flow without having to redefine the flow itself.