We are pleased to announce version 0.5.0 of SQL Runner. This release adds some powerful new features, including local and Consul-based remote locking to ensure that SQL Runner runs your playbooks as singletons.
This release adds the ability to lock your run - this ensures that you cannot accidentally start another job whilst one is already running. This feature is aimed primarily at automation, whereby you schedule your jobs on a cron or some other scheduler and you need to ensure that each job is a singleton.
We have also introduced two types of locks that you can use:
Hard lock: This lock will be set and removed only if a job completes successfully; if the job fails then this lock will persist and will subsequently prevent this job from being run again until the lock is removed. The default lock type for SQL Runner
Soft lock: This lock will be set and removed regardless of if the job completes successfully; it simply prevents concurrent runs from occurring
For those scheduling jobs randomly over a cluster we have also added the ability to set locks using Consul. This allows you to set and delete a lock which is then automatically available to all members of the Consul cluster.
To use Consul instead of the local filesystem simply append the -consul argument.
To set a hard lock:
Here’s the run failing because of an existing lockfile:
To set a soft lock:
2. Checking and deleting locks
On top of being able to set locks we have also added functions to check and delete these locks. This enables you to automate maintenance of locks based on your own logic.
To check for a lock’s existence:
To delete a lock:
3. Running a single query
This release also adds the ability to run a single query of a step in a playbook. This has two main uses:
Allowing you to represent your playbook as a DAG of individual queries, making resumption from a failure point easier to reason about
Allowing you to debug individual queries without having to first make new playbooks that contain only the queries in question
To run an individual query:
Note the usage of a double colon between the elements of the runQuery value. For this function to work you must ensure that you do not include a double colon anywhere in either your step or query names.
Sample output from our integration test suite:
4. Other changes
The playbook & SQL loading interfaces in our Golang code have been abstracted to allow for easy extensibility (#62). Big thanks to community member Lincoln Ritter for contributing this functionality!
Under the hood, this release also bumps our Postgres library dependency to pg.v5 (#73), and adds tcpKeepAlive=true to help with long-running queries via NAT (#57).
SQL Runner 0.5.0 is available as a standalone binary for 64-bit Linux, Windows and macOS on Bintray. Download them as follows: