We’re now going to express this DAG in a Makefile ready for Make.
Make is a software build tool first released in 1977. It uses files called Makefiles to specify how to build the target program. A Makefile lets you specify tasks that contribute to the build, and express dependencies between these tasks, forming a directed acyclic graph.
Because the tasks in a Makefile are just shell commands, we can use
make to orchestrate a batch processing pipeline, as long as each individual step in the pipeline is invokable from a shell.
Here’s the Makefile for our job,
example-dag.makefile with a simple
echo to represent each task, plus some
sleeps to make it easier to see what is going on when we run it:
By default Make will attempt to build the first rule found in the supplied Makefile - I like to call the first rule
done and make it dependent on the last task in our DAG. You can see that the rest of our rules consist of:
To learn a lot more about the Makefile syntax, check out the GNU make manual.
Let’s visualize this Makefile, using the makefile2dot Python script:
Here is the generated diagram:
The DAG flows bottom-to-top, which is a little quirky - it reflects the fact that Makefiles normally create a target which is built on top of multiple intermediate files.
Now that we have our Makefile, we need to run it! Here is the command we’ll use, together with the output:
In reverse order, our command line arguments to
make are as follows:
-fspecifies the Makefile to run (otherwise the default is
Makefilein the current directory)
-jlets Make run with as much parallelism as is needed - so that our Huskimo and Snowplow tasks can run at the same time
-kmeans “keep going” through the DAG as far as possible - so for example, even if Huskimo fails, we can still complete the Snowplow tasks before failing the overall job
make has given us the DAG component of our orchestration problem - what about the scheduling aspect? For this, we can simply use Cron.
Edit your crontab:
And add an entry to run our Makefile every morning at 3am UTC:
And that’s it! We now have our DAG scheduled to run nightly.
Our job will fail if one or more of the steps fails - meaning that a shell command in the step returns a non-zero code.
Let’s simulate a failure with the following
failing-dag.makefile - note the
exit 1 under the StorageLoader rule:
Let’s run this:
Make has faithfully reported our failure! So how do we recover from this? Typically we will:
Some of the orchestration tools in this post’s introduction make this recovery process quite straightforward - things are a little more complex with our
The first thing we need to remember is that we are running with the
-k flag, meaning that processing kept going post-failure, continuing to run steps on any branches of the DAG which were not (yet) dependent on the failing step.
This behavior makes it much easier to reason about our job failure. We don’t have to worry about what was running at the exact time of step failure; instead we just review the DAG to see which steps cannot have run:
When troubleshooting a job failure, always review the Make error output carefully to make sure that there weren’t in fact failures in multiple branches of the DAG!
With this done, we can now produce a scratch Makefile just for the job resumption,
Note how we removed all the completed steps, and removed dangling references to the completed steps in the dependencies of the outstanding steps.
A quick visualization of this Makefile:
Here is the now much simpler DAG:
Finally let’s run this:
Great - we’ve now completed our recovery from the job failure.
This blog post has shown how you can use simple tools,
cron, to orchestrate complex batch processing jobs.
The Makefile-based approach is simple, some might say crude - it certainly doesn’t have all the bells and whistles of a tool like Chronos or Airflow. However, this approach has many fewer failure states and it can be much easier to reason about and resolve job failures.
Even if you plan on implementing a full-blown distributed orchestration tool, it can be worth prototyping new jobs using something as simple as Make - give it a go and let us know your thoughts in the comments!