The embedded Lua interpreter

As seen for tasks and conditions, an embedded Lua interpreter can be used to run scripts to respectively execute structured tasks and verify complex conditions without the need to execute external commands. This is generally quicker for small operations, because the interpreter is already available while whenever is running, it only needs to be initialized and provided the script that, in turn, has been loaded at startup. Tasks (or checks) that would last even seconds, can be performed in a small fraction of the time.

The embedded interpreter makes the standard Lua library available for scripts, which includes:

• string manipulation

• mathematics

• io and file operations

• access to OS information and facilities.

The Lua manual is the primary source for information about the capabilities of the standard library.

Moreover, the embedded interpreter provides an isolation level that is comparable to the one that can be achieved by running external commands: every time that a script has to be run, a new interpreter is initialized, that normally works without interaction with other scripts that may have run before, or may be running aside.

While the Lua language is supported in its entirety, the embedded interpreter has both some limitations and some enhancements, that come from its awareness of being part of whenever.

Tip

While many of the features and configuration options described here may not be very useful for casual scripts configured directly by editing the configuration file, they are intended to be helpful when implementing specific tasks or conditions in a frontend.

Configuration

Both tasks and conditions based on Lua scripts can configure the interpreter with common parameters. Apart from the script parameter, that should contain the script to be run as a plain string, the following configuration parameters are available:

• init_script_path, the optional path to an initialization script, that can be used to fine-tune the interpreter, for example by setting the module search path and other specific options;

• variables_to_set, an optional map that associates identifiers to values that the interpreter will find already set when it starts running: the identifiers are plain TOML keys and as such they will be available in the script as usual Lua identifiers, and the values can only be of boolean, numeric, and string type;

• expected_results, a map that associates identifiers (plain TOML keys here as well) to simple values that can be of boolean, numeric, and string type: although not mandatory, at least an expected result is necessary for Lua script based conditions to perform a test that can either succeed or fail.

Tip

The script has to be provided as a literal using the script parameter (TOML literal strings, especially in the multiline flavor, come in handy for this purpose), however, in case the execution of a script is needed, that is available as a separate file, it is always possible to use the require and dofile() Lua functions to achieve this.

Both in tasks and conditions, the dictionary of espected result is checked when the script finishes running. Depending on the value of the expect_all parameter (which is false by default) the script is considered to have succeeded as follows:

• when expect_all is true and all the identifiers specified in the map hold exactly the specified values,

• when expect_all is false and at least one of the identifiers holds the specified value.

Because of the limitations in the types of values that can be provided in the configuration file, the tests that whenever can perform after the script has finished running are actually quite simple. However, of course, nothing forbids to perform more articulated tests in the script itself and assign, for instance, a value to a boolean variable that depends on the outcome of such tests.

Note

Since an independent interpreter is initialized for every Lua based item, the above mentioned parameters are set specifically for every item at each run; at the moment there is no way to change the default values for these parameters.

Even though every script gets its own, isolated Lua interpreter, whenever offers a way for possibly different scripts to interoperate by sharing simple values, and to store intermediate results for subsequent runs of the script associated to an item. Details on this capability can be found below in the Enhancements section.

The Lua interpreter is initialized at each run by

• setting the additional variables, including the ones implicitly provided by whenever,

• implementing the extra functionalities, and

• executing the startup script,

exactly in this order.

Initialization script

For more fine-tuned initialization, that can even be common to several instances of the interpreter, an initialization script in Lua can be specified in each item configuration using the init_script_path parameter: the value should be set to the full path of the script itself. It can be used to configure the Lua interpreter, for example by setting the package.path[1] to a directory where pure Lua modules can be found, or by preloading modules.

Limitations

The prominent limitation is that the Lua interpreter in whenever can not load binary modules.[2] This means that many popular libraries, mostly used in proper applications, are not (and cannot be made) available to the scripts that this application can execute: trying to require such libraries, even in case they can be found in the library search path, just results in an error.

Enhancements

The non-standard features that come with this version of the interpreter cover the following topics:

• knowing the reason why a script is run

• sending messages to the whenever log

• performing basic network operations, namely simple HTTP requests

• sharing data with other scripts within the same whenever session.

All of these features are accessed each through specific module-like interfaces, variables, or Lua tables.

The reason for running a script can be accessed through two predefined variables:

• whenever_task which is a string reporting the name of the task which defines the script: it is obviously available in Lua based tasks only,

• whenever_condition, is another string that in the case of Lua based condition reports the name of the condition being checked, and for tasks reports the condition that triggered the tsk itself.

The log module exposes commands that allow to forward messages to the whenever log: this can be useful for debugging, of course, but also for frontends that need to communicate with specially crafted Lua based items. Messages can be sent at every supported log level (error, warn, info, debug, trace). The library functions are the following:

• log.error(message)

• log.warn(message)

• log.info(message)

• log.debug(message)

• log.trace(message)

and take a single string (message) as their argument. Each of these functions issues a log message at the respective log level.

Warning

The message is not issued on its own, but prefixed by the standard log information (such as the date, the application and the log level) plus extra information that identify the condition and/or the task to which the script that issued the message belongs.

There is no need to require the log module, as it is available to the interpreter by default.

An example for both the log module and the whenever_... variables can be found in the TRACE task shown in the README file:

[[task]]
type = "lua"
name = "TRACE"
script = '''log.warn("Trace: *** VERIFIED CONDITION *** `" .. whenever_condition .. "`")'''

that is, a task that can be used to debug conditions.

Tip

Triple single quotes are used in TOML to denote literal multiline strings: literal means that everything betwen the two occurrences of the triple single quotes is interpreted as it is, including quotes of any type and backslashes. This is particularly useful in this case, allowing a short script to be provided in the configuration without losing readability.

The remaining additional features are more complex, and deserve dedicated sub-paragraphs.

HTTP queries (optional)

The embedded Lua interpreter offers the possibility to query network services for responses, using the HTTP protocol, both encrypted (HTTPS) and unencrypted. This feature is provided via a built in module named http. The requests can be performed using both the GET and the POST methods, respectively using

• http.get(url [, headers]), and

• http.post(url [, body [, headers]])

where the optional parameters are expected to be in the url argument for GET queries and in the body argument (with appropriate headers if necessary) for POST queries.

These functions return the response body as it has been received if successful, or an error if the query could not be performed. Along with the response body, the interpreter receives the response status code as a Lua integer number.

Warning

A request error, as every other error, causes always the script to fail: if this is not the intended behavior, it can be useful to enclose the code that sends the request in a pcall() statement in order to catch the error.

In a Lua script, the two utilities can be exploited as follows to retrieve information from an HTTP server:

-- script parameters
success = true
url = "https://httpbin.org/get?param1=42&param2=some"
check1 = '"param1": "42"'
check2 = '"param2": "some"'

-- retrieve data and check it: `httpbin.org` answers in the JSON
-- format by default, and the checks are substrings of the response
resp = http.get(url)
success = success and string.find(resp, check1, 1, true)
success = success and string.find(resp, check2, 1, true)

and the answer that httpbin.org provides is similar to the following:

{
  "args": {
    "param1": "42",
    "param2": "some"
  },
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "X-Amzn-Trace-Id": "Root=1-69c6bb80-08f03a13145c13042443a168"
  },
  "origin": "1.2.3.4",
  "url": "https://httpbin.org/get?param1=42&param2=some"
}

so, if the expected_results parameter in the item configuration had been set to { success = true }, the script execution is considered successful – as long as the server was reachable and could respond.

As said before, a similar script can be used, that uses the POST method:

-- script parameters
success = true
url = "https://httpbin.org/post"

-- set the headers: here we use mixed case only for aesthetical reasons
headers = {}
headers["Content-Type"] = "application/x-www-form-urlencoded"
body = "param1=42&param2=some"
check1 = '"param1": "42"'
check2 = '"param2": "some"'

-- perform the request and check the results
txt = http.post(url, body, headers)
success = success and string.find(txt, check1, 1, true)
success = success and string.find(txt, check2, 1, true)

by constructing the appropriate headers and body, and providing everything to the server.

Caution

While the URL and the request body are simple strings, the headers parameter must be provided as a Lua table, with suitable strings for both keys (the header names, or fields) and values. The table can be built in Lua in the usual way, as in the provided example.

While the examples above only capture the body of the HTTP response, both http.get() and http.post() have three return values, in the following order:

• the response body as a string,

• the HTTP status code as an integer, and

• the response headers as a table that maps strings to strings.

Therefore a complete HTTP request has the following form:

body, status, headers = http.get("https://httpbin.org/get?param1=42&param2=some")

where, for instance, headers["host"] == "httpbin.org" holds true. Please note that all header names are converted to lowercase: per RFC 2616 header names are case insensitive, and just using lowercase is a way to standardize access when treated as structured data.

Of course, Lua allows to capture just one or two, or all of the three return values.

These utilities cover a great range of what can be useful for network access in a Lua script from within whenever: for more complex needs, external commands can be used anyway.

Note

The http module is only available when the lua_httpreq feature is enabled; however, both the standard configurations and the provided binaries come with the feature enabled.

Private and shared states (optional)

As said at the beginning of this chapter, even though the Lua interpreter is freshly reinitialized at each run, there is the possibility to “remember” intermediate results across subsequent sessions of the same script, or even to exchange data with other scripts. This is made possible through private and shared states.

A private state is simply a Lua table that is available to every script. This table, named state, is available to every script, and cannot be accessed by other scripts. Values in the table can be set as follows:

state.message = "Have a nice day!"
state.answer = 42
state.been_here = true

and a subsequent run of the same script in the same task or condition, is able to retrieve those values:

success = state.answer == 42

thus becoming aware of results that may have been obtained in previous runs. The state is saved when the script finishes running, even if the outcome is unsuccessful for any reason.

Warning

The entries of the state table can only contain values of type boolean, numeric, or string. Any other type of value, including nil, will cause an error at the end of the script and the state will not be saved.

Since the private state is only accessible by the item it is associated to, it is normally safe to access and modify it freely, without any type of synchronization: there is only one case in which what happens is hardly predictable, that is when more than one condition activate the same Lua based task, almost at the same time – or, at least, an instance of the script is started while another instance of the same script, in the same task is already running: no matter whether or not the first instance already stored something into the private state, the second instance will access the same values as the first instance, unmodified. Moreover, the values that the instance that ends first saves to the private state, will be simply overwritten by the one that the instance that finishes next saves in turn: attempting to use the synchronization utilities does not help, since the state is read at once before the script runs, and saved after it finished running.

Shared states, on the other side, are accessible to all Lua scripts, and behave differently: they need to be explicitly loaded and saved by a script, respectively using the sharedstate.load(name) and sharedstate.save(name, table) functions.

When a shared state is saved, it becomes immediately available for access and modification to other scripts, or to the same script in a second moment: it can be loaded using the name that it has been saved with. Shared state names, although strings, must have the form of an identifier, that is, start with an underscore or a letter followed by alphanumeric characters and underscores. And, just like identifiers, state names are case sensitive. The tables that can be saved can only contain values of the types allowed in private states: booleans, numbers, and strings.

Reading and writing shared states are protected from concurrency, that is, it is impossible for a script to access a shared state that is accessed by another script: therefore it is safe to avoid synchronization when loading or saving a shared state. Nevertheless, an example of shared state that uses synchronization is provided below, in the paragraph dedicated to the synchronization utilities.

A method is provided also to remove an existing shared state, sharedstate.remove(name), where name is the name of the state to delete.

Note

The state table and the sharedstate module are only available when the lua_sync feature is enabled; however, both the standard configurations and the provided binaries come with the feature enabled.

Synchronization utilities (optional)

The synchronization utilities are provided in order to avoid that Lua scripts, that run at the same time, concurrently access a resource (for instance, a file), possibly causing an avoidable script error. The possibility to activate a lock, and to wait for a lock to be released, synchronizes the access to resources in a safe way.

The utilities to acquire and release a lock are, respectively, sync.lock(name [, timeout]) and sync.release(name) where name is a string which has the form of an identifier, and the optional timeout parameter is the amount of time, in seconds, that the calling script should wait for the lock to be released before giving up: in case the lock is not acquired the return value is false. When the timeout is omitted, lock() will wait forever. The timeout parameter can also be a fractional number, and the precision is in milliseconds.

Although not mandatory, the script that acquires a lock should explicitly release it as soon as possible: these synchronization utilities are made available to simplify exclusive access in the most flexible way, so that the lock() function just puts a lock on a resource, and release() just releases it, independently from which script calls the former or the latter. In other words, lock() can be called from a script and release() from another, although it is not encouraged.

A sync.sleep(seconds) function is also provided, that stops the script for the provided amount of time in seconds. The seconds parameter is a number, can be fractional, and the precision is, in this case as well, in milliseconds.

In the following example, two scripts contend a shared state: it can be useful to use locks for shared states too, if the scripts that contend it should wait for each other to release it.

First script:

sleep_seconds = math.random() * 10
log.warn("LOCK:A waiting to lock shared state")
l = sync.lock("SharedState1_LOCK")
if l then
    log.warn("LOCK:A shared state LOCKED")
    sync.sleep(sleep_seconds)
    sst = sharedstate.load("SharedState1")
    v = sst.v
    if v == nil then
        v = 0
    end
    v = v + 1
    sst.v = v
    sharedstate.save("SharedState1", sst)
    sync.release("SharedState1_LOCK")
    log.warn("LOCK:A shared state UNLOCKED: now v = " .. tostring(v))
else
    log.error("LOCK:A shared state NOT LOCKED")
end

and second:

sleep_seconds = math.random() * 10
log.warn("LOCK:B waiting to lock shared state")
l = sync.lock("SharedState1_LOCK", 2)
if l then
    log.warn("LOCK:B shared state LOCKED")
    sync.sleep(sleep_seconds)
    sst = sharedstate.load("SharedState1")
    v = sst.v
    if v == nil then
        v = 0
    end
    v = v * 2
    sst.v = v
    sharedstate.save("SharedState1", sst)
    sync.release("SharedState1_LOCK")
    log.warn("LOCK:B shared state UNLOCKED: now v = " .. tostring(v))
else
    log.error("LOCK:B shared state NOT LOCKED")
end

The examples use the logging system to notify the outcome of the synchronization operations. Only the second script forces the lock() utility to wait at most two seconds to acquire the lock, after which it will just log an error. Both scripts access a shared state both to read and to write a single entry, v, which is a number that each script manipulates. The random sleep time has been introduced in order to randomly cause contentions.

Caution

Locks are never destroyed: they are intended to be shared among scripts, so once a lock is created it remains in memory until whenever exits. Even though locks do not require a significant amount of resources, it is adviced not to instance them with dynamic names, that is, with names that are built up from variable elements, because indefinite proliferation of locks might end up in unwanted resource occupation.

Note

The sync module is only available when the lua_sync feature is enabled. In fact, the feature is active by default in the standard configurations and in the provided binaries.

Conclusion

The embedded Lua interpreter can be intended, especially in the case of scripts manually created by the user, as a quick replacement for external scripts and commands: the interpreter initialization is very fast, while calling an external command is slower and resource hungry. Lua is a very expressive language, quite common as a scripting engine for applications for many reasons, one of which is that it has been expressedly created for this purpose, and another is that it is very compact.

In whenever, the internal Lua interpreter can also be used as a building block for complex workflows: thanks to the recent additions, it acts as a stateful engine where scripts activated as both tasks and conditions can cooperate. Its usefulness becomes even more evident in the case of frontends that create specific tasks and conditions that should be aware of each other.

[1]

The package.cpath setting can be modified as well, however it generally has no effect due to the fact that loading binary modules is normally disabled.

[2]

There is actually the possibility to recompile whenever with support for binary Lua modules: this is disabled by default and in the standard configurations for the supported platforms. The tests on this option are still unsatisfactory, and in any case it comes with the risk of corrupting the application state.