Command line interface (CLI) for running jobs/tasks
kiba-extend
uses Thor to provide a command line interface for working with your ETL project.
I chose Thor over Rake because it is awkward to pass options/parameters in Rake, and because automated testing of Rake tasks is convoluted. (ref)
Help on the CLI
The following command will list all available tasks.
thor -T
This lets you search for only tasks beginning with “reg”:
thor list reg
Some of the task descriptions may be truncated in the display, though. This also doesn’t tell you what parameters/options you can pass in.
To get more details on a given task:
thor --help TASKNAME
For example: thor --help reg:list
or thor --help jobs:tagged
Conventions in the help
Plain parameters
When you see:
Usage:
thor jobs:tagged TAG
The all caps word is a placeholder for a parameter that gets passed in without an option flag. For example, the following returns a list of jobs tagged with “report”:
thor jobs:tagged report
Boolean options
Boolean options are presented a bit oddly in the help. For example:
Options:
r, [--run], [--no-run] # Whether to run the matching jobs
Any of the following will work, according to your preference:
To find the jobs, list, and run them:
thor jobs:tagged report -r true
thor jobs:tagged cspace --run
thor jobs:tagged -r true cspace
thor jobs:tagged --run true cspace
thor jobs:tagged --run cspace
The find and list the jobs without running them:
thor jobs:tagged report -r false
thor jobs:tagged cspace --no-run
thor jobs:tagged -r false cspace
thor jobs:tagged --run false cspace
thor jobs:tagged --no-run cspace
However the following does run the jobs, so use one of the more straightforward options above:
thor jobs:tagged cspace --no-run true
Other options
Usage:
v, [--verbosity=VERBOSITY] # Only relevant if run=true. How much info to print to screen
# Default: normal
# Possible values: minimal, normal, verbose
In this case, replace the all-caps word with one of the possible values (if listed), or your uncontrolled string.
To use the full option name:
thor jobs:tagged cspace --run --verbosity=verbose
To use the alias:
thor jobs:tagged cspace --run -v verbose
Architecture/design
Thor tasks are defined in kiba-extend/lib/tasks
.
There is a Thorfile in the kiba-extend
base directory that autoloads those tasks and runs the CLI when you type thor commands. (How this works is some kinda ruby/thor library magic I haven’t dug into fully).
Your ETL project base directory (if following the repo template/FWM example), will also have a Thorfile in its base directory, which will call in all of kiba-extend
’s tasks, as well as any you create in your own repo’s /lib/tasks
directory.