Configure

config is a command to get or set the configuration. This command is underlying on livy.utils.ConfigBase, it saves the values on local directory and let the corresponding command filled with the pre-configured settings.

Usage

List all configurable keys

usage: livy config list [-h] [section]

positional arguments:
  section     Only show specific section

optional arguments:
  -h, --help  show this help message and exit

It provides a command to view the list, but is without description. For the details, see below.

Get value

usage: livy config get [-h] name

positional arguments:
  name        Config name to be retrieved.

optional arguments:
  -h, --help  show this help message and exit

Set value

usage: livy config set [-h] name value

positional arguments:
  name        Name of config to be updated.
  value       Value to be set.

optional arguments:
  -h, --help  show this help message and exit

Not all of used values could be set via this command. For example, values used Plugin system should be inputted manually.

All configurable keys

A valid config name for the CLI tool should be in prefix.name format, the prefix would be placed in class description and name is the attribute name in each of config class. For example, root.api_url would be a valid key for CLI tool.

The value shows after the attribute is default value when it is not override by user. Some of them could be None.

class livy.cli.config.RootSection

Prefix root. Basic settings that might be applied to all actions.

api_url: str = None

Base-URL for Livy API server.

class livy.cli.config.LocalLoggingSection

Prefix logs. Logging behavior on local.

format: str = '%(levelcolor)s%(asctime)s [%(levelname)s] %(name)s:%(reset)s %(message)s'

Log message format. %(levelcolor)s and %(reset)s are two special keys only take effect on console output in this tool. They are and used to colorize the output. For all the other avaliable keys, see log attributes.

date_format: str = '%Y-%m-%d %H:%M:%S %z'

Date format in log message. It uses strftime format.

output_file: bool = False

Output logs into file by default. A log file with random name would be created on the working directory when it set to True.

with_progressbar: bool = True

Convert Spark TaskSetManager’s logs (Finished task X in stage Y on example.com (1/10)) into progress bar. This feature requires tqdm.

highlight_loggers: List[str] = []

Highlight logs from these loggers. This option only take effects when colorama is installed.

hide_loggers: List[str] = []

Hide logs from these loggers. This option does not affect progress bar displaying.

logfile_level: livy.cli.config.LocalLoggingSection.LogLevel = 10

Default log level on output to log file.

class livy.cli.config.ReadLogSection

Prefix read-log. For Read log tool.

keep_watch: bool = True

Keep watching for batch activity until it is finished.

class livy.cli.config.SubmitSection

Prefix submit. For Submit tool.

pre_submit: List[str] = []

Enabled Pre-submit plugin list.

driver_memory: str = None

Amount of memory to use for the driver process. Need to specific unit, e.g. 12gb or 34mb.

driver_cores: int = None

Number of cores to use for the driver process.

executor_memory: str = None

Amount of memory to use per executor process. Need to specific unit, e.g. 12gb or 34mb.

executor_cores: int = None

Number of cores to use for each executor.

num_executors: int = None

Number of executors to launch for this batch.

spark_conf: List[Tuple[str, str]] = []

Key value pairs to override spark configuration properties.

watch_log: bool = True

Watching for logs after the task is submitted. This option shares the same behavior to keep_watch, only different is the scope it take effects.

task_success: List[str] = []

Plugins to be trigger when task is finished and success.

task_fail: List[str] = []

Plugins to be trigger when task is ended and failed.

task_ended: List[str] = []

Plugins to be trigger when task is ended, regardless to its state.

Kill Plugin system