Plugin system

Some of the operation might not be common sense to other user, so these actions are moved to plugins and trigger via the hook.

How to use it

To enable a plugin, we could setup via Configure or override it with corresponding argument. The parent function/tool pass the Namespace object to the deligated function, and use the returned one in the rest part of functions.

The deligated function could do what ever it want, including overwrite the arguments to impact the behavior, except logging behavior, which is already configured before the hook.

Take pre-submit hook for example, it could be set in pre_submit, or use --pre-submit in Submit. A PreSubmitArguments would be given to the plugin function and the it could run actions and override the parameters.

Create your own plugin

Plugins are importted and triggered in runtime. Therefore you could place your functions everywhere, you only need to ensure it is under your PYTHONPATH.

A plugin function must in following signature:

def action(source: str, args: argparse.Namespace) -> argparse.Namespace:
    ...

It takes arguments source and args, and returns Namespace object back to parent function.

Input argument source the hook name, for preventing user trigger this function in wrong hook. It is hardcoded string and always in uppercase, see table below for details. Argument args an Namespace object, the plugin function could do what they want and modify its value.

Expected input value and typed namespace instances are:

name

source string

Typed namespace reference

pre-submit

PRE-SUBMIT

PreSubmitArguments

task-success

TASK-SUCCESS

PreSubmitArguments

task-failed

TASK-FAILED

PreSubmitArguments

task-ended

TASK-ENDED

TaskEndedArguments

Builtin plugin

Upload script to S3

livy.cli.plugin.upload_s3(source, args)

Pre-submit action to upload local file to AWS S3, for Hadoop to read the files. It could be useful if you are using this tool with EMR.

This plugin requires configurations section plugin:upload_s3 set in user config file. Following key are read:

  • bucket: Required key. S3 bucket name

  • folder_format: Required key. S3 prefix name format to store the files; it would be expanded inside the plugin with variables time (currenttime), uuid (random genersted uuid) or script_name (base name part of main application script).

  • expire_days: Optional key. Would set object expire date if given.

Example configs:

{
    "pre-submit:upload_s3": {
        "bucket": "example-bucket",
        "folder_format": "livy-{time:%Y%m%d}-{script_name}-{uuid}",
        "expire_days": 3
    }
}
Parameters
Return type

livy.cli.submit.PreSubmitArguments

Configure API Reference