.. _65-lua:

The embedded *Lua* interpreter
==============================

As seen for :ref:`tasks <40-tasks-lua>` and :ref:`conditions <50-conditions-lua>`, an embedded
*Lua* interpreter can be used to run scripts to respectively execute structured tasks and
verify complex conditions without the need to execute external commands. This is generally
quicker for small operations, because the interpreter is already available while **whenever**
is running, it only needs to be initialized and provided the script that, in turn, has been
loaded at startup. Tasks (or checks) that would last even seconds, can be performed in a small
fraction of the time.

The embedded interpreter makes the standard *Lua* library available for scripts, which includes:

* string manipulation
* mathematics
* io and file operations
* access to OS information and facilities.

The `Lua manual <https://www.lua.org/manual/>`__ is the primary source for information about the
capabilities of the standard library.

Moreover, the embedded interpreter provides an isolation level that is comparable to the one
that can be achieved by running external commands: every time that a script has to be run, a new
interpreter is initialized, that normally works without interaction with other scripts that may
have run before, or may be running aside.

While the *Lua* language is supported in its entirety, the embedded interpreter has both some
limitations and some enhancements, that come from its awareness of being part of **whenever**.

.. tip::
  While many of the features and configuration options described here may not be very useful
  for casual scripts configured directly by editing the configuration file, they are intended
  to be helpful when implementing specific tasks or conditions in a frontend.


.. _65-lua-configuration:

Configuration
-------------

Both *tasks* and *conditions* based on *Lua* scripts can configure the interpreter with
common parameters. Apart from the ``script`` parameter, that should contain the script to
be run as a plain string, the following configuration parameters are available:

* ``init_script_path``, the optional path to an initialization script, that can be used
  to fine-tune the interpreter, for example by setting the module search path and other
  specific options;
* ``variables_to_set``, an optional map that associates identifiers to values that the
  interpreter will find already set when it starts running: the identifiers are plain
  TOML keys and as such they will be available in the script as usual *Lua* identifiers,
  and the values can only be of *boolean*, *numeric*, and *string* type;
* ``expected_results``, a map that associates identifiers (plain TOML keys here as well)
  to simple values that can be of *boolean*, *numeric*, and *string* type: although not
  mandatory, at least an expected result is necessary for *Lua* script based conditions
  to perform a test that can either succeed or fail.

.. tip::
  The script has to be provided as a literal using the ``script`` parameter (TOML
  `literal strings <https://toml.io/en/v1.1.0#string>`__, especially in the *multiline*
  flavor, come in handy for this purpose), however, in case the execution of a script is
  needed, that is available as a separate file, it is always possible to use the
  ``require`` and ``dofile()`` *Lua* functions to achieve this.

Both in *tasks* and *conditions*, the dictionary of espected result is checked when the
script finishes running. Depending on the value of the ``expect_all`` parameter (which is
*false* by default) the script is considered to have succeeded as follows:

* when ``expect_all`` is *true* and all the identifiers specified in the map hold exactly
  the specified values,
* when ``expect_all`` is *false* and at least one of the identifiers holds the specified
  value.

Because of the limitations in the types of values that can be provided in the configuration
file, the tests that **whenever** can perform after the script has finished running are
actually quite simple. However, of course, nothing forbids to perform more articulated tests
in the script itself and assign, for instance, a value to a boolean variable that depends
on the outcome of such tests.

.. note::
  Since an independent interpreter is initialized for every *Lua* based item, the above
  mentioned parameters are set specifically for every item at each run; at the moment there
  is no way to change the default values for these parameters.

Even though every script gets its own, isolated *Lua* interpreter, **whenever** offers a
way for possibly different scripts to interoperate by sharing simple values, and to store
intermediate results for subsequent runs of the script associated to an item. Details on
this capability can be found below in the :ref:`Enhancements <65-lua-enhancements>`
section.

The Lua interpreter is initialized at each run by

* setting the additional variables, including the ones implicitly provided by **whenever**,
* implementing the extra functionalities, and
* executing the startup script,

exactly in this order.


.. _65-lua-initscript:

Initialization script
---------------------

For more fine-tuned initialization, that can even be common to several instances of the
interpreter, an initialization script in *Lua* can be specified in each item configuration
using the ``init_script_path`` parameter: the value should be set to the full path of the
script itself. It can be used to configure the *Lua* interpreter, for example by setting
the ``package.path``\ [#fn-1]_ to a directory where pure *Lua* modules can be found, or by
preloading modules.


.. _65-lua-limitations:

Limitations
-----------

The prominent limitation is that the *Lua* interpreter in **whenever** *can not* load binary
modules.\ [#fn-2]_ This means that many popular libraries, mostly used in proper applications,
are not (and cannot be made) available to the scripts that this application can execute: trying
to ``require`` such libraries, even in case they can be found in the library search path, just
results in an error.


.. _65-lua-enhancements:

Enhancements
------------

The non-standard features that come with this version of the interpreter cover the following
topics:

* knowing the reason why a script is run
* sending messages to the **whenever** log
* performing basic network operations, namely simple HTTP requests
* sharing data with other scripts within the same **whenever** session.

All of these features are accessed each through specific module-like interfaces, variables, or
*Lua tables*.

The *reason* for running a script can be accessed through two predefined variables:

* ``whenever_task`` which is a string reporting the name of the task which defines the script:
  it is obviously available in *Lua* based tasks only,
* ``whenever_condition``, is another string that in the case of *Lua* based condition reports
  the name of the condition being checked, and for tasks reports the condition that triggered
  the tsk itself.

The ``log`` module exposes commands that allow to forward messages to the **whenever** log:
this can be useful for debugging, of course, but also for frontends that need to communicate
with specially crafted *Lua* based items. Messages can be sent at every supported log level
(\ *error*, *warn*, *info*, *debug*, *trace*\ ). The library functions are the following:

* ``log.error(message)``
* ``log.warn(message)``
* ``log.info(message)``
* ``log.debug(message)``
* ``log.trace(message)``

and take a single string (``message``) as their argument. Each of these functions issues a
log message at the respective log level.

.. warning::
  The message is not issued on its own, but prefixed by the standard log information (such
  as the date, the application and the log level) plus extra information that identify the
  condition and/or the task to which the script that issued the message belongs.

There is no need to ``require`` the ``log`` module, as it is available to the interpreter
by default.

An example for both the ``log`` module and the ``whenever_...`` variables can be found in the
*TRACE* task shown in the README file:

.. code-block:: toml

  [[task]]
  type = "lua"
  name = "TRACE"
  script = '''log.warn("Trace: *** VERIFIED CONDITION *** `" .. whenever_condition .. "`")'''

that is, a task that can be used to debug conditions.

.. tip::
  Triple single quotes are used in TOML to denote *literal multiline strings*: *literal* means
  that everything betwen the two occurrences of the triple single quotes is interpreted as it
  is, including quotes of any type and backslashes. This is particularly useful in this case,
  allowing a short script to be provided in the configuration without losing readability.

The remaining additional features are more complex, and deserve dedicated sub-paragraphs.


.. _65-lua-enhancements-http:

HTTP queries (optional)
~~~~~~~~~~~~~~~~~~~~~~~

The embedded *Lua* interpreter offers the possibility to query network services for
responses, using the HTTP protocol, both encrypted (HTTPS) and unencrypted. This feature
is provided via a built in module named ``http``. The requests can be performed using
both the *GET* and the *POST* methods, respectively using

* ``http.get(url [, headers])``, and
* ``http.post(url [, body [, headers]])``

where the optional parameters are expected to be in the ``url`` argument for *GET* queries
and in the ``body`` argument (with appropriate headers if necessary) for *POST* queries.

These functions return the response body as it has been received if successful, or an
error if the query could not be performed. Along with the response body, the interpreter
receives the response *status code* as a *Lua* integer number.

.. warning::
  A request error, as every other error, causes always the script to **fail**: if this
  is not the intended behavior, it can be useful to enclose the code that sends the
  request in a ``pcall()`` statement in order to catch the error.

In a *Lua* script, the two utilities can be exploited as follows to retrieve information
from an HTTP server:

.. code-block:: lua

  -- script parameters
  success = true
  url = "https://httpbin.org/get?param1=42&param2=some"
  check1 = '"param1": "42"'
  check2 = '"param2": "some"'

  -- retrieve data and check it: `httpbin.org` answers in the JSON
  -- format by default, and the checks are substrings of the response
  resp = http.get(url)
  success = success and string.find(resp, check1, 1, true)
  success = success and string.find(resp, check2, 1, true)

and the answer that *httpbin.org* provides is similar to the following:

.. code-block:: json

  {
    "args": {
      "param1": "42",
      "param2": "some"
    },
    "headers": {
      "Accept": "*/*",
      "Host": "httpbin.org",
      "X-Amzn-Trace-Id": "Root=1-69c6bb80-08f03a13145c13042443a168"
    },
    "origin": "1.2.3.4",
    "url": "https://httpbin.org/get?param1=42&param2=some"
  }

so, if the ``expected_results`` parameter in the item configuration had been set to
``{ success = true }``, the script execution is considered successful -- as long as the server
was reachable and could respond.

As said before, a similar script can be used, that uses the *POST* method:

.. code-block:: lua

  -- script parameters
  success = true
  url = "https://httpbin.org/post"

  -- set the headers: here we use mixed case only for aesthetical reasons
  headers = {}
  headers["Content-Type"] = "application/x-www-form-urlencoded"
  body = "param1=42&param2=some"
  check1 = '"param1": "42"'
  check2 = '"param2": "some"'

  -- perform the request and check the results
  txt = http.post(url, body, headers)
  success = success and string.find(txt, check1, 1, true)
  success = success and string.find(txt, check2, 1, true)

by constructing the appropriate headers and body, and providing everything to the server.

.. caution::
  While the URL and the request body are simple strings, the ``headers`` parameter must
  be provided as a *Lua table*, with suitable strings for both keys (the header *names*,
  or *fields*) and values. The *table* can be built in *Lua* in the usual way, as in the
  provided example.

While the examples above only capture the *body* of the HTTP response, both ``http.get()``
and ``http.post()`` have three return values, in the following order:

* the response *body* as a string,
* the HTTP *status code* as an integer, and
* the response *headers* as a table that maps strings to strings.

Therefore a complete HTTP request has the following form:

.. code-block:: lua

  body, status, headers = http.get("https://httpbin.org/get?param1=42&param2=some")

where, for instance, ``headers["host"] == "httpbin.org"`` holds *true*. Please note that *all
header names are converted to lowercase*: per RFC 2616 header names are case insensitive, and
just using lowercase is a way to standardize access when treated as structured data.

Of course, *Lua* allows to capture just one or two, or all of the three return values.

These utilities cover a great range of what can be useful for network access in a *Lua*
script from within **whenever**: for more complex needs, external commands can be used
anyway.

.. note::
  The *http* module is only available when the ``lua_httpreq`` feature is enabled; however, 
  both the standard configurations and the provided binaries come with the feature enabled.


.. _65-lua-persistence:

Private and shared states (optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As said at the beginning of this chapter, even though the *Lua* interpreter is freshly
reinitialized at each run, there is the possibility to "remember" intermediate results across
subsequent sessions of the same script, or even to exchange data with other scripts. This is
made possible through *private* and *shared states*.

A *private state* is simply a *Lua table* that is available to every script. This table, named
``state``, is available to every script, and cannot be accessed by other scripts. Values in the
table can be set as follows:

.. code-block:: lua

  state.message = "Have a nice day!"
  state.answer = 42
  state.been_here = true

and a subsequent run of the *same* script in the *same* task or condition, is able to retrieve
those values:

.. code-block:: lua

  success = state.answer == 42

thus becoming aware of results that may have been obtained in previous runs. The state is saved
when the script finishes running, even if the outcome is unsuccessful for any reason.

.. warning::
  The entries of the ``state`` table can only contain values of type *boolean*, *numeric*, or
  *string*. Any other type of value, including *nil*, will cause an error at the end of the
  script and the state will *not* be saved.

Since the private state is only accessible by the item it is associated to, it is normally safe
to access and modify it freely, without any type of synchronization: there is only one case in
which what happens is hardly predictable, that is when more than one condition activate the same
*Lua* based task, almost at the same time -- or, at least, an instance of the script is started
while another instance of the *same* script, in the *same* task is already running: no matter
whether or not the first instance already stored something into the private state, the second
instance will access the same values as the first instance, unmodified. Moreover, the values
that the instance that ends first saves to the private state, will be simply overwritten by the
one that the instance that finishes next saves in turn: attempting to use the synchronization
utilities does not help, since the state is read at once *before* the script runs, and saved
*after* it finished running.

*Shared states*, on the other side, are accessible to *all* *Lua* scripts, and behave
differently: they need to be explicitly loaded and saved by a script, respectively using the
``sharedstate.load(name)`` and ``sharedstate.save(name, table)`` functions.

When a shared state is saved, it becomes immediately available for access and modification to
other scripts, or to the same script in a second moment: it can be loaded using the name that
it has been saved with. Shared state names, although strings, must have the form of an
identifier, that is, start with an underscore or a letter followed by alphanumeric characters
and underscores. And, just like identifiers, state names are case sensitive. The tables that
can be saved can only contain values of the types allowed in private states: *booleans*,
*numbers*, and *strings*.

Reading and writing shared states are protected from concurrency, that is, it is impossible
for a script to access a shared state that is accessed by another script: therefore it is
safe to avoid synchronization when loading or saving a shared state. Nevertheless, an example
of shared state that uses synchronization is provided below, in the paragraph dedicated to
the synchronization utilities.

A method is provided also to remove an existing shared state, ``sharedstate.remove(name)``,
where ``name`` is the name of the state to delete.

.. note::
  The *state* table and the *sharedstate* module are only available when the ``lua_sync``
  feature is enabled; however, both the standard configurations and the provided binaries
  come with the feature enabled.


.. _65-lua-synchronization:

Synchronization utilities (optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The synchronization utilities are provided in order to avoid that *Lua* scripts, that run at
the same time, concurrently access a resource (for instance, a file), possibly causing an
avoidable script error. The possibility to activate a lock, and to wait for a lock to be
released, synchronizes the access to resources in a safe way.

The utilities to acquire and release a lock are, respectively, ``sync.lock(name [, timeout])``
and ``sync.release(name)`` where ``name`` is a string which has the form of an identifier, and
the optional ``timeout`` parameter is the amount of time, in seconds, that the calling script
should wait for the lock to be released before giving up: in case the lock is not acquired the
return value is *false*. When the timeout is omitted, ``lock()`` will wait forever. The
``timeout`` parameter can also be a fractional number, and the precision is in milliseconds.

Although not mandatory, the script that acquires a lock should *explicitly* release it as soon
as possible: these synchronization utilities are made available to simplify exclusive access
in the most flexible way, so that the ``lock()`` function just puts a lock on a resource, and
``release()`` just releases it, independently from which script calls the former or the latter.
In other words, ``lock()`` can be called from a script and ``release()`` from another, although
it is not encouraged.

A ``sync.sleep(seconds)`` function is also provided, that stops the script for the provided
amount of time in seconds. The ``seconds`` parameter is a number, can be fractional, and the
precision is, in this case as well, in milliseconds.

In the following example, two scripts contend a shared state: it can be useful to use locks
for shared states too, if the scripts that contend it should wait for each other to release it.

First script:

.. code-block:: lua

  sleep_seconds = math.random() * 10
  log.warn("LOCK:A waiting to lock shared state")
  l = sync.lock("SharedState1_LOCK")
  if l then
      log.warn("LOCK:A shared state LOCKED")
      sync.sleep(sleep_seconds)
      sst = sharedstate.load("SharedState1")
      v = sst.v
      if v == nil then
          v = 0
      end
      v = v + 1
      sst.v = v
      sharedstate.save("SharedState1", sst)
      sync.release("SharedState1_LOCK")
      log.warn("LOCK:A shared state UNLOCKED: now v = " .. tostring(v))
  else
      log.error("LOCK:A shared state NOT LOCKED")
  end

and second:

.. code-block:: lua

  sleep_seconds = math.random() * 10
  log.warn("LOCK:B waiting to lock shared state")
  l = sync.lock("SharedState1_LOCK", 2)
  if l then
      log.warn("LOCK:B shared state LOCKED")
      sync.sleep(sleep_seconds)
      sst = sharedstate.load("SharedState1")
      v = sst.v
      if v == nil then
          v = 0
      end
      v = v * 2
      sst.v = v
      sharedstate.save("SharedState1", sst)
      sync.release("SharedState1_LOCK")
      log.warn("LOCK:B shared state UNLOCKED: now v = " .. tostring(v))
  else
      log.error("LOCK:B shared state NOT LOCKED")
  end

The examples use the logging system to notify the outcome of the synchronization operations.
Only the second script forces the ``lock()`` utility to wait at most two seconds to acquire
the lock, after which it will just log an error. Both scripts access a shared state both to
read and to write a single entry, ``v``, which is a number that each script manipulates. The
random sleep time has been introduced in order to randomly cause contentions.

.. caution::
  Locks are never destroyed: they are intended to be shared among scripts, so once a lock is
  created it remains in memory until **whenever** exits. Even though locks do not require a
  significant amount of resources, it is adviced not to instance them with dynamic names, that
  is, with names that are built up from variable elements, because indefinite proliferation of
  locks might end up in unwanted resource occupation.

.. note::
  The *sync* module is only available when the ``lua_sync`` feature is enabled. In fact, the
  feature is active by default in the standard configurations and in the provided binaries.


.. _65-lua-conclusion:

Conclusion
----------

The embedded *Lua* interpreter can be intended, especially in the case of scripts manually
created by the user, as a quick replacement for external scripts and commands: the interpreter
initialization is very fast, while calling an external command is slower and resource hungry.
*Lua* is a very expressive language, quite common as a scripting engine for applications for
many reasons, one of which is that it has been *expressedly* created for this purpose, and
another is that it is very compact.

In **whenever**, the internal *Lua* interpreter can also be used as a building block for
complex workflows: thanks to the recent additions, it acts as a stateful engine where scripts
activated as both *tasks* and *conditions* can cooperate. Its usefulness becomes even more
evident in the case of frontends that create specific tasks and conditions that should be aware
of each other.


.. [#fn-1] The ``package.cpath`` setting can be modified as well, however it generally has no
           effect due to the fact that loading binary modules is normally disabled.

.. [#fn-2] There is actually the possibility to recompile **whenever** with support for binary
           *Lua* modules: this is disabled by default and in the standard configurations for the
           supported platforms. The tests on this option are still unsatisfactory, and in any
           case it comes with the risk of corrupting the application state.