From 43998eae625e97fe382d814373ec766fb46e132d Mon Sep 17 00:00:00 2001 From: Xavier Delaruelle Date: Tue, 10 Nov 2026 07:20:10 +0100 Subject: [PATCH 1/2] doc: add initial lua-modulefile-support design doc Signed-off-by: Xavier Delaruelle --- .hunspell.en.dic | 3 + doc/source/design/lua-modulefile-support.rst | 171 +++++++++++++++++++ 2 files changed, 174 insertions(+) create mode 100644 doc/source/design/lua-modulefile-support.rst diff --git a/.hunspell.en.dic b/.hunspell.en.dic index 15e354f84..b315436f3 100644 --- a/.hunspell.en.dic +++ b/.hunspell.en.dic @@ -1674,3 +1674,6 @@ TMOD getPathElementWeight withdrawPriorityDemands getPathWeightList +ENVI +ENDEVAL +EXITEVAL diff --git a/doc/source/design/lua-modulefile-support.rst b/doc/source/design/lua-modulefile-support.rst new file mode 100644 index 000000000..07dd0d759 --- /dev/null +++ b/doc/source/design/lua-modulefile-support.rst @@ -0,0 +1,171 @@ +.. _lua-modulefile-support: + +Lua modulefile support +====================== + +The objective is to be able to evaluate Lua modulefiles as Lmod does. Same +syntax should be supported. Also the same precedence mechanisms than Lmod +should be supported regarding mixing Lua and Tcl modulefiles in same +modulepath should. + +Evaluating Lua modulefile should have similar performances than Tcl modulefile +evaluation. Of course, it will cost a bit more as a Lua interpreter has to be +run in addition to the Tcl interpreter. But user must not notice the extra +processing time. + +Modulefile evaluation +--------------------- + +For the sake of efficiency as less processes as possible should be involved to +evaluate Lua modulefiles and Lua modulercs. + +The idea is to start a persistent Lua script and interact with it to make it +evaluate the successive modulefiles without restart. One Lua script will be +made to evaluate Lua modulefiles, another Lua script will be made to evaluate +Lua modulercs. + +The general architecture is to start a Tcl interpreter that starts the +persistent Lua script and interact with it. + +The Lua scripts will support all Lua Module functions supported by Lmod. They +will translate these functions into Tcl modulefile commands that will be +sent back to the Tcl interpreter for evaluation. + +Lua script also have to update their local environment to get the correct +value for environment variable if these value are accessed later-on by +modulefile. + +Environment changes resulting from executed Tcl modulefile commands should be +forwarded to the Lua script to make it update itself. + +If executed Tcl modulefile command ends evaluation, this information should be +sent to Lua interpreter to end evaluation on its side. + +Error or other kind of messages are sent to the Tcl interpreter as command to +evaluate. + +Between two modulefile or modulerc evaluations, Lua script should reset to an +initial state (variable and function reset as done for Tcl interpreter). + +Tcl-Lua interpreter bridge implementation +----------------------------------------- + +Goal is to have a persistent Lua interpreter connected to a Tcl interpreter: + +* Tcl interpreter starts the Lua interpreter as a piped command +* It sends it information on its *stdin* channel + + * Evaluation properties, aka *DATA* + * Environment variable changes, aka *ENVI* + * Lua code, aka *CODE* + * Exit current evaluation, aka *EXIT* + +* Lua interpreter sends back information via its *stdout* channel + + * Tcl code to interpret (including error or other messages to raise) + * End of evaluation signal (via a ``__MODULES_ITRP_ENDEVAL__`` marker) + +* Tcl interpreter close pipe to mean end of work +* When *stdin* is not working anymore, Lua interpreter exits + +Information exchange from Tcl to Lua +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Information exchange from Tcl interpreter to Lua interpreter is done via the +channel opened when Lua interpreter is executed as a piped command. + +Thus Lua interpreter gets the information through its stdin channel. Stdin is +read line by line, within a co-routine, not to block the interpreter execution +with an I/O wait. + +Each line read from stdin is analyzed to determine if another line should get +read immediately (before yielding). The ``__MODULES_ITRP_EXCHANGE__`` marker +is used to indicate multiple lines should be read, until reaching another +`__MODULES_ITRP_EXCHANGE__`` marker line. + +An empty modulefile content or an empty list of environment change have to be +sent by the Tcl interpreter to release the Lua interpreter from its stdin read +wait. + +Reading things line by line may have an impact on performances, for instance +if modulefiles have numerous code lines. It should be evaluated. + +Some leads to improve the Tcl-Lua performances: + +* Improve asynchronous I/O co-routine mechanism to be able to read full content + of modulefile in one read call (is this possible?) +* Reduce the number of information to exchange between the 2 interpreter + + * Compute environment variable changes directly within Lua interpreter, but + it will need to handle any protected variables or other kind of + mechanisms (but these information may be sent with other properties via + *DATA*). + * Send modulefile path instead of content to read it directly from Lua + interpreter. But doing so, it will not benefit from module cache. + +Evaluation end from Tcl or Lua +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Evaluation may end on the Lua side: + +* Everything goes well: end of evaluation is signaled by Lua interp to Tcl +* Code does not compile: report module error Tcl code is sent to Tcl +* Code evaluation fails: report module error Tcl code is sent to Tcl unless an + *EXIT* order was already received from Tcl (a Lua internal marker, + ``__MODULES_ITRP_EXITEVAL__``, is used to convey this *EXIT* order) + +Evaluation may end on the Tcl side: + +* Tcl code is evaluated to handle an error raised on the Lua side (either the + code does not compile or its evaluation fails): in this case, nothing has to + be sent to the Lua interp, as evaluation is already finished there +* Tcl code evaluation raises an error: an *EXIT* order is sent to the Lua + interp to kill current evaluation. + +Sandboxing evaluation in Lua interpreter +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Lua provides a convenient ``load`` function to evaluate Lua code. A restricted +environment can be defined for this evaluation. With that feature, the list of +functions reachable by evaluated code can be controlled. + +With ``load``, the variables or functions defined by some evaluated code are +accessible from the next ``load`` evaluation. Some cleanup should be done thus +between evaluations, like performed on Tcl side. Such feature may not be +necessary on day 0 as Lmod is not providing such cleaning between evals. + +``load`` command first evaluate code to produce a function. During this step +it catches any syntax error. Then this function can be evaluated. + +``os.exit`` command is changed into a call to the Tcl ``exit`` modulefile +command. ``os.exit`` would otherwise ends current modulefile evaluation but +also the whole Lua interpreter script. + +Handling evaluation modes +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The various evaluation modes (``load``, ``unload``, etc) are handled by the +Tcl interpreter which is initialized to perform a mode-specific evaluation of +the modulefile commands. As the Lua interpreter sends to the Tcl interpreter +a conversion of the Lua commands in Tcl commands, it does not need to handle +the mode particularities. + +The ``help`` Lua command is translated into new ``module-help`` Tcl command +outputting its string argument to the stderr channel. + +There is no equivalent of ``ModulesDisplay`` and ``ModulesTest`` in Lmod. Also +Lmod has no ``test`` evaluation mode. Running ``module spider`` evaluates +modules with ``spider`` mode on Lmod whereas ``scan`` mode is used on Modules. + +Differences with Lmod +--------------------- + +This section lists the differences spotted between a Lua modulefile evaluated +with Lmod and Modules. Most differences comes from the translation of the Lua +modulefile commands into Tcl modulefile commands: the former may have +different behavior than the latter. + +* ``os.exit`` is evaluated as the Tcl ``exit`` command that does not return + an exit code value equal to its argument. + +.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: From 4f970b73be60ff1fa05ceedfbac9b696a290e2f7 Mon Sep 17 00:00:00 2001 From: Xavier Delaruelle Date: Mon, 16 Nov 2026 15:32:49 -0600 Subject: [PATCH 2/2] doc: complete lua-modulefile-support design Extend the initial design note on Lua modulefile support into a full specification: * Lua modulefile support is always enabled. Rules to locate Lua modulefiles and modulercs, their precedence over Tcl files and their handling in module cache are defined after Lmod behavior. * Tcl-Lua bridge is detailed: start of the persistent Lua script, supported Lua versions, message format, protection of the communication channel, nested evaluations, evaluation end, error reporting, debugging and sandboxing. * Only the names of changed environment variables are sent to Lua interpreter, which fetches their value when a modulefile reads them or runs a command. Sending values with each message makes the cost of the bridge grow with the number of loaded modules. * Lmod modulefile and modulerc functions are reviewed: mapping to Tcl modulefile commands, functions handled by the Lua script, behaviors specific to an evaluation mode and way to identify the module tool. * Site customization is covered: hooks, site-specific functions and properties file are ported to the Modules counterparts. * Implementation stages, installation options, packaging, documentation, linting of Lua files, testsuite and performance benchmark are described. * Differences with Lmod are listed. Performance figures come from a throw-away prototype of the bridge run on the modulefiles of the load2 benchmark. Assisted-by: Claude:claude-fable-5-2 Signed-off-by: Xavier Delaruelle --- .codespellrc | 3 +- .hunspell.en.dic | 97 +- doc/source/design/lua-modulefile-support.rst | 2271 +++++++++++++++++- 3 files changed, 2305 insertions(+), 66 deletions(-) diff --git a/.codespellrc b/.codespellrc index 87aa2795c..6f5504cb4 100644 --- a/.codespellrc +++ b/.codespellrc @@ -6,7 +6,8 @@ # ignore command names used in code: uncomplete # ignore hyphenated wording already used throughout Tab completion option # descriptions: By-pass -ignore-words-list = ans,ot,ue,fo,haa,varius,ure,tthe,tbe,nin,desig,matchin,compre,invers,compiletime,seh,lod,nd,uncomplete,by-pass +# ignore option names of Lmod functions: userA +ignore-words-list = ans,ot,ue,fo,haa,varius,ure,tthe,tbe,nin,desig,matchin,compre,invers,compiletime,seh,lod,nd,uncomplete,by-pass,usera # skip git repository internal content # skip generated content: ChangeLog.gz, doc/build, modulecmd*.tcl, modules.sum # modules.log, lib/config.sub, lib/config.guess, lib/config.status, lib/configure, diff --git a/.hunspell.en.dic b/.hunspell.en.dic index b315436f3..60f3de54d 100644 --- a/.hunspell.en.dic +++ b/.hunspell.en.dic @@ -1674,6 +1674,97 @@ TMOD getPathElementWeight withdrawPriorityDemands getPathWeightList -ENVI -ENDEVAL -EXITEVAL +coroutine +fileevent +isAvail +isDir +isFile +lf +metatable +myFileName +myShellName +pathJoin +stdlib +vwait +getinfo +lfs +LmodBreak +modeA +pcall +popen +mf +mfinterp +hierarchyA +luacheck +luacov +LuaJIT +splitFileName +CLOEXEC +const +CPATH +lua53 +setfenv +loadstring +anyA +displayName +fn +forbiddenT +fullName +loadOrder +mgrload +moduleKindT +origUserName +sn +stackDepth +userName +errWarnMsgHook +groupName +isForbiddenHook +isVisibleHook +listHook +msgHook +packagebasename +reverseMapPathFilter +SiteName +updateFn +writeCache +collectgarbage +setpause +setstepmul +restoreSettings +abspath +barefilename +dofile +extname +forbidRegex +groupA +hideRegex +hostid +isDefined +isNotDefined +isPending +LmodMsgRaw +LmodSystemError +LmodVersion +LmodWarning +loadfile +moduleStackTraceBack +myConfig +myModuleFullNameAndAlias +myModuleUsrAndAliasName +myShellType +notGroupA +notUserA +realpath +removeExt +requireFullName +syslog +userA +userInGroups +UUIDString +cpath +luacheckdatadir +elseif +SUBMINOR +gpu +validT diff --git a/doc/source/design/lua-modulefile-support.rst b/doc/source/design/lua-modulefile-support.rst index 07dd0d759..451fb8e07 100644 --- a/doc/source/design/lua-modulefile-support.rst +++ b/doc/source/design/lua-modulefile-support.rst @@ -6,13 +6,264 @@ Lua modulefile support The objective is to be able to evaluate Lua modulefiles as Lmod does. Same syntax should be supported. Also the same precedence mechanisms than Lmod should be supported regarding mixing Lua and Tcl modulefiles in same -modulepath should. +modulepath. Evaluating Lua modulefile should have similar performances than Tcl modulefile evaluation. Of course, it will cost a bit more as a Lua interpreter has to be run in addition to the Tcl interpreter. But user must not notice the extra processing time. +Enabling Lua modulefile support +------------------------------- + +Lua modulefile support is always enabled. There is no option to turn it on or +off: sites control it by adding or not Lua modulefiles and modulercs in their +modulepaths. + +Such option has been considered to protect existing modulepaths, as Lua +modulefile support changes the modulefiles found in modulepaths that contain +files with a ``.lua`` extension. It is not retained: + +* Without Lua modulefile support, ``.lua`` files are Tcl modulefiles that are + invalid as they do not start with the Modules magic cookie. No existing + setup relies on them. +* Behavior only changes in a directory containing both a Tcl modulefile and a + Lua modulefile of the same name, where the Lua modulefile is now selected. + This is what Lmod does on the same directory, thus what is expected by a + site that shares modulepaths between the two tools. +* No cost is added for sites that do not use Lua modulefiles, as Lua + interpreter is only started when a Lua file has to be evaluated. +* An option means a second code path wherever ``.lua`` files are located, + including in module cache, and a testsuite to run in both states. + +A configuration option designates the Lua interpreter program to run. Its +default value is set at installation time. When this program cannot be run, +Lua modulefiles and modulercs are still located and take precedence as +described below. An error is raised when one of them has to be evaluated. + +Locating Lua modulefiles +------------------------ + +Lua modulefiles and modulercs are located as follows: + +* Any file with a ``.lua`` extension found in a modulepath is a Lua + modulefile. Modules magic cookie is not checked on these files. +* Module name of a Lua modulefile is its file path relative to the modulepath + without the ``.lua`` extension. Directories whose name ends with ``.lua`` + keep their full name. +* A file named :file:`.lua` is ignored, as it has no module name. +* A Lua modulefile whose module name is also the name of a directory located + beside it (:file:`foo/1.0.lua` and :file:`foo/1.0/`) is ignored: this name + designates the directory and the modulefiles it contains. Such clash cannot + occur between Tcl modulefiles. +* Tcl modulercs designate Lua modulefiles with their module name (without + ``.lua`` extension). +* Path ignore patterns (:mfcmd:`modulepath-ignore`) are matched against file + names, thus with the ``.lua`` extension for Lua modulefiles. These patterns + describe the content of directories, not module names: a pattern that ends + with the name of a Tcl modulefile does not match the Lua modulefile of the + same module name. +* A ``default`` symbolic link that targets a Lua modulefile is a Lua + modulefile. +* A :file:`.modulerc.lua` file is a Lua modulerc, as well as any global rc + file whose name ends with ``.lua``. + +Precedence between Lua and Tcl files +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a Lua file and a Tcl file compete for the same role, the same file is +selected than on Lmod. The general rule on Lmod is that Lua files always take +priority over non-Lua files. The behavior described below corresponds to +Lmod 9.4. + +Modulefiles: + +* A Lua modulefile takes precedence over a Tcl modulefile of the same name in + the same directory (:file:`foo/1.0.lua` over :file:`foo/1.0`), whatever the + order these files are found when reading the directory. +* Content of the Lua modulefile is not looked at to apply this precedence: it + applies even if the Lua modulefile is erroneous. The Tcl modulefile is never + silently evaluated in place of a broken Lua modulefile. +* Only a Lua modulefile that cannot be read by the current user leaves its + place to the Tcl modulefile. +* Across modulepaths, language of modulefiles plays no role: the modulepath + order applies the same way than between Tcl modulefiles. + +Modulercs located in modulepath directories: + +* Lmod evaluates one single file per directory, the first one found in the + following order: :file:`.modulerc.lua`, :file:`.modulerc` then + :file:`.version`. A :file:`.modulerc.lua` thus supersedes the + :file:`.modulerc` and :file:`.version` files of the same directory, which + are not evaluated. This is the behavior adopted, including for the modulerc + located at the root of a modulepath. +* Like for modulefiles, :file:`.modulerc.lua` supersedes the Tcl files even if + it is erroneous. An evaluation error is reported and the Tcl files stay + ignored. +* When no :file:`.modulerc.lua` exists, both :file:`.modulerc` and + :file:`.version` are evaluated as currently done by Modules, whereas Lmod + only evaluates :file:`.modulerc`. Precedence between Tcl files is not + changed by Lua modulefile support. +* On Lmod a ``default`` symbolic link comes first in the above order: when + such link exists, none of the rc files of the directory is evaluated. This + is not adopted: an rc file may define more than the default version + (aliases, hidden modules, etc) and Modules currently evaluates the Tcl rc + files of a directory that contains a ``default`` symbolic link. + +Global rc files: + +* User rc file: :file:`$HOME/.modulerc.lua` supersedes :file:`$HOME/.modulerc` + which is not evaluated. +* Rc file of the installation: :file:`rc.lua` in the configuration directory + supersedes the :file:`rc` file of this directory. +* Rc files designated by configuration (:envvar:`MODULERCFILE`) are all + evaluated. Each one is a Lua modulerc if its name ends with ``.lua``, a Tcl + modulerc otherwise. When a directory is designated, Modules looks for a + :file:`modulerc` file in it: by analogy a :file:`modulerc.lua` file found + there supersedes it. This last rule has no Lmod equivalent, as Lmod + evaluates all the files of such directory. +* Evaluation order between these three kinds of global rc file is unchanged. + +These rules have several consequences on the code that locates modulefiles: + +* As the file name of a Lua modulefile differs from its module name, the path + of a modulefile cannot be computed from its modulepath and module name + anymore. It is always fetched from the search result structure. +* Module cache records the file name of Lua modulefiles rather than their + module name. Module name is computed when cache file is evaluated. The same + precedence rules apply whether modulepath content is fetched from a + directory walk or from a cache file (see `Module cache`_). +* The reverse deductions also take the extension into account: getting the + modulepath from the path and the name of a modulefile, and telling that a + module is virtual because its name is not the end of the path of its + modulefile. +* Code that rebuilds the path of a modulefile from the modulepath and an entry + of the search result (to evaluate modulefiles in ``whatis`` mode or to match + a module against a rule set on a full path name) is changed to use the path + recorded in this entry. +* The optimization applied when an exact module name and version is searched, + which only keeps the entries of the directory needed to tell if this version + is the implicit default, stays applied on directories containing ``.lua`` + files. Entries are sorted and compared there on their module name, thus + without extension, as the extension changes how file names sort. It is worth + keeping: on the load of a module with more than a hundred dependencies, + walking the full directories adds as much time as half of the whole bridge. +* A ``.lua`` entry is known to be a Lua modulefile once the existing test + that tells whether an entry is a directory is done. No system call is added + to recognize Lua modulefiles. + +Module cache +------------ + +Lua modulefiles and modulercs are recorded in module cache files, like their +Tcl counterparts (see :ref:`module-cache`). Same result should be obtained +whether a cache file is used or not. + +Cache commands for Lua files +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Two cache commands are added to record Lua files: + +* ``lua-modulefile-content`` records a Lua modulefile +* ``lua-modulerc-content`` records a Lua modulerc + +``lua-modulefile-content`` accepts the following arguments, in this order: + +* relative file path, including the ``.lua`` extension +* file last modification date (as a Unix epoch time) +* file body content + +``lua-modulerc-content`` accepts the following arguments, in this order: + +* relative file path +* file body content + +For instance: + +* ``lua-modulefile-content foo/1.0.lua 1234567890 {body}`` +* ``lua-modulerc-content foo/.modulerc.lua {body}`` + +These commands mirror their Tcl counterpart without the file module header +argument, as there is no magic cookie in Lua files. Like for +``modulerc-content``, modification time is not needed for modulercs. + +Existing ``modulefile-content`` and ``modulerc-content`` commands are not +reused for Lua files. A Modules version that knows nothing about Lua +modulefiles would otherwise take these entries for valid Tcl modulefiles and +evaluate their Lua code as Tcl code. + +There is no Lua counterpart to the ``modulefile-invalid`` command: validity of +a Lua modulefile is not checked when it is located, as there is no magic +cookie to look for. + +Lua files or directories with limited access are recorded with the existing +``limited-access-file`` and ``limited-access-directory`` commands. These +commands only record a name. When such entry is tested, the file is handled +according to its name like during a directory walk. + +All Lua and Tcl files found in a modulepath are recorded, including the Tcl +files that are superseded by a Lua file (like :file:`foo/1.0` when +:file:`foo/1.0.lua` exists). Precedence is applied when cache file is used, +not when it is built. + +Cache evaluation +^^^^^^^^^^^^^^^^ + +``lua-modulefile-content`` populates the same memory structures than +``modulefile-content``: + +* file content, modification time and validity are recorded with the full + path of the file (with ``.lua`` extension) as key +* the entry added to the content list of the modulepath is named after the + module (without ``.lua`` extension) and designates the full path of the + file + +``lua-modulerc-content`` records the file content and adds a modulerc entry to +the content list of the modulepath. + +Content of a Lua file obtained from cache is transmitted to the Lua +interpreter in the evaluation request. Lua interpreter does not read the file +itself. + +Precedence between Lua and Tcl files is applied on the content list of the +modulepath: + +* A Lua modulefile entry replaces the Tcl modulefile entry of the same name, + whatever the order these entries appear in cache file. +* When a :file:`.modulerc.lua` entry exists for a directory, the + :file:`.modulerc` and :file:`.version` entries of this directory are not + evaluated. +* Entries with limited access are tested first, as a Lua modulefile that + cannot be read leaves its place to the Tcl modulefile of the same name. + +Cache validity +^^^^^^^^^^^^^^ + +Version number set in cache file header is the minimal Modules version able to +use this cache file. As adding cache commands increases this minimal version: + +* A cache file that contains at least one Lua cache command gets the version + that introduces Lua modulefile support as minimal version. Older Modules + versions silently ignore such cache file and walk the modulepath directory + instead, where they do not consider Lua files. +* A cache file that contains no Lua cache command keeps its current minimal + version. Modulepaths without Lua files are this way not impacted. + +Cache files built by an older version +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A cache file built by a Modules version that does not support Lua modulefiles +describes the ``.lua`` files as invalid modulefiles (``modulefile-invalid`` +command), without their content. As Lua files are never recorded this way +when Lua cache commands exist, such entry indicates an outdated cache file. + +A ``modulefile-invalid`` entry whose name ends with ``.lua`` is handled like a +file with limited access: file is read from the modulepath directory when it +matches the search. Lua +modulefiles are this way found even if cache file has not been rebuilt. +Rebuilding the cache file is only needed to benefit from cache for these +files. + Modulefile evaluation --------------------- @@ -20,31 +271,47 @@ For the sake of efficiency as less processes as possible should be involved to evaluate Lua modulefiles and Lua modulercs. The idea is to start a persistent Lua script and interact with it to make it -evaluate the successive modulefiles without restart. One Lua script will be -made to evaluate Lua modulefiles, another Lua script will be made to evaluate -Lua modulercs. +evaluate the successive modulefiles without restart. A single Lua script, thus +a single Lua process, evaluates both Lua modulefiles and Lua modulercs: each +kind of file is evaluated against its own set of reachable functions. -The general architecture is to start a Tcl interpreter that starts the -persistent Lua script and interact with it. +The general architecture is that :file:`modulecmd.tcl` starts the persistent +Lua script and interacts with it. The Lua script is started when the first Lua +modulefile or modulerc has to be evaluated, not to penalize situations where +only Tcl modulefiles are involved. -The Lua scripts will support all Lua Module functions supported by Lmod. They +The Lua script will support all Lua Module functions supported by Lmod. It will translate these functions into Tcl modulefile commands that will be -sent back to the Tcl interpreter for evaluation. +sent back to the Tcl interpreter for evaluation. This way there is only one +implementation of the modulefile commands. These Tcl commands are evaluated +in the regular modulefile Tcl interpreter that corresponds to the current +evaluation mode and depth level. + +Lua evaluation is triggered from inside the modulefile Tcl interpreter: where +the content of a Tcl modulefile is evaluated, a dedicated command is called +instead to evaluate the Lua modulefile through the bridge. This way the whole +evaluation process that surrounds the evaluation of the modulefile content is +shared between Tcl and Lua modulefiles (variant checks, mode-specific +procedures, error rendering, etc). -Lua script also have to update their local environment to get the correct +Everything that could be achieved locally by the Lua script is performed +there without involving the Tcl interpreter (see `Work split between Lua and +Tcl`_). + +Lua script also has to update its local environment to get the correct value for environment variable if these value are accessed later-on by -modulefile. +modulefile or by a command it executes. -Environment changes resulting from executed Tcl modulefile commands should be -forwarded to the Lua script to make it update itself. +Environment variables changed by executed Tcl modulefile commands are notified +to the Lua script, which fetches their value when it is needed. -If executed Tcl modulefile command ends evaluation, this information should be +If executed Tcl modulefile command ends evaluation, this information is sent to Lua interpreter to end evaluation on its side. Error or other kind of messages are sent to the Tcl interpreter as command to evaluate. -Between two modulefile or modulerc evaluations, Lua script should reset to an +Between two modulefile or modulerc evaluations, Lua script resets to an initial state (variable and function reset as done for Tcl interpreter). Tcl-Lua interpreter bridge implementation @@ -55,53 +322,497 @@ Goal is to have a persistent Lua interpreter connected to a Tcl interpreter: * Tcl interpreter starts the Lua interpreter as a piped command * It sends it information on its *stdin* channel - * Evaluation properties, aka *DATA* - * Environment variable changes, aka *ENVI* - * Lua code, aka *CODE* - * Exit current evaluation, aka *EXIT* + * Evaluation request: evaluation properties and Lua code to evaluate + * Command reply: status and result obtained from the evaluation of a Tcl + command sent by the Lua interpreter and name of the environment variables + it has changed * Lua interpreter sends back information via its *stdout* channel - * Tcl code to interpret (including error or other messages to raise) - * End of evaluation signal (via a ``__MODULES_ITRP_ENDEVAL__`` marker) + * Tcl command to evaluate (including error or other messages to raise) + * End of evaluation signal * Tcl interpreter close pipe to mean end of work * When *stdin* is not working anymore, Lua interpreter exits -Information exchange from Tcl to Lua +Exchanges are synchronous. Once an evaluation request is sent, Tcl interpreter +reads the pipe with blocking calls in a loop: it evaluates each command +received and sends its reply, until the end of evaluation signal is received. +No event loop (``fileevent``, ``vwait``) is involved on Tcl side. On Lua side, +a blocking read is performed after each command sent. No coroutine is needed +as the Lua interpreter has nothing else to do while waiting for the reply. + +Starting the Lua interpreter +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Lua script is named :file:`mfinterp.lua`. This name follows the vocabulary +used for the Tcl code: ``mf`` stands for modulefile and ``interp`` designates +the interpreters that evaluate modulefiles and modulercs. A name made of the +full ``modulefile`` word is avoided as the script also evaluates modulercs and +as it could be taken for an actual modulefile. + +Source code of the script is stored in a :file:`lua` directory at the root of +the source repository, like the :file:`tcl` directory holds the Tcl code. If +the script grows, its code can be split there in several files (bridge, +sandbox, modulefile functions, modulerc functions) that are concatenated at +build time, as done for :file:`modulecmd.tcl`. + +:file:`mfinterp.lua` is generated by ``make`` at the root of the source +repository and is installed in the same directory than :file:`modulecmd.tcl` +(``libexecdir``). :file:`modulecmd.tcl` this way locates the script next to +itself, whether it is run from the source repository (by the testsuite for +instance) or from an installation. The script is installed as a data file, not +as a program: it is never executed directly but always passed as argument to +the Lua interpreter program set in configuration. It thus needs no shebang. + +Embedding the code of the script in :file:`modulecmd.tcl` to send it to the +Lua interpreter is not retained: this code would be parsed by the Tcl +interpreter at each :file:`modulecmd.tcl` run, even when no Lua modulefile is +evaluated. + +``LUA_INIT`` environment variable and its version-specific variants (like +``LUA_INIT_5_4``) designate code that Lua interpreter executes when it starts. +Such code may write on *stdout* before the bridge is ready. These variables +are thus withdrawn from the environment the Lua interpreter is started with +and restored afterward. + +Lua interpreter is started in the environment of the user, which loaded +modules may have changed. ``LUA_PATH`` and ``LUA_CPATH`` variables set by a +module providing another Lua version would make Lua script load libraries +that do not match its interpreter. To be independent of these variables: + +* ``configure`` script records the library search paths of the designated + Lua interpreter (``package.path`` and ``package.cpath`` values). A site + whose Lua libraries are installed in a specific location sets ``LUA_PATH`` + and ``LUA_CPATH`` when running ``configure`` to get this location recorded. +* :file:`mfinterp.lua` sets ``package.path`` and ``package.cpath`` to the + recorded values when it starts, before loading any library. These values + are set in the script when it is generated by ``make``. +* These search paths also apply to the ``require`` calls made by modulefiles, + as on Lmod. + +The ``-E`` option of the Lua interpreter, which ignores all these variables, +is not used: the Lua interpreter program set in configuration may be a wrapper +that relies on them to load a tool before the Lua script (see `Coverage`_). + +Variables put in quarantine (:mconfig:`quarantine_support`), like +``LD_LIBRARY_PATH``, get the same treatment as when a Tcl interpreter is +created: Lua interpreter is started with the value defined for the run of +:file:`modulecmd.tcl`. Their actual value is then restored on Tcl side and +recorded as an environment change, so Lua process environment gets it before +a modulefile reads it or runs a command. Commands executed by modulefiles +this way run +with the environment of the user, whereas the Lua interpreter and its +libraries have been loaded with the protected one. + +Lua script relies on the ``posix`` Lua library to update its process +environment and to set up its file descriptors. This library is a requirement +of Lmod, thus it is expected to be available on sites using Lua modulefiles. +Functions of the ``posix`` and ``lfs`` libraries are also made available to +modulefiles, as done by Lmod. They are loaded only when a modulefile accesses +them (see `Sandboxing evaluation in Lua interpreter`_). + +Supported Lua versions +^^^^^^^^^^^^^^^^^^^^^^ + +Lua modulefile support is expected to work on Enterprise Linux 8 and newer +systems with the Lua interpreter they provide. The oldest of these systems +sets the minimal versions to support: + +* Lua 5.3 (Enterprise Linux 8 provides Lua 5.3.4, newer releases provide + Lua 5.4) +* ``posix`` library 33.3 (version 33.3.1 on Enterprise Linux 8) +* ``lfs`` library 1.6 (version 1.6.3 on Enterprise Linux 8) + +:file:`mfinterp.lua` is thus written to run the same way on Lua 5.3 and +Lua 5.4: + +* It only uses the syntax and the standard library functions that exist in + both versions. Features introduced by Lua 5.4 are not used (variable + attributes like ```` or ````, ``warn`` function, etc). +* It only uses the functions and constants of the ``posix`` library that + exist in its minimal supported version, through the sub-modules of this + library. For instance the ``FD_CLOEXEC`` constant is not defined by version + 33.3.1, so its value is used instead when the constant is missing. +* Every new Lua version is added to the versions tested when it gets + available. +* Version check made by :file:`configure` script and by :file:`mfinterp.lua` + accepts any version equal to or newer than Lua 5.3. It is computed from the + version number rather than compared to a list of known versions. + +Lua 5.5 is not provided yet by the systems to support, but sites may build it +themselves. :file:`mfinterp.lua` runs on it as long as the following points +are respected: + +* Control variable of a ``for`` loop is not assigned in the loop body. It is + read-only on Lua 5.5 and such assignment is a syntax error. +* Error object caught from evaluated code may be ``nil`` on Lua 5.3 and Lua + 5.4 (``error`` function called without argument), whereas Lua 5.5 replaces + it by a message string. Both cases are handled. +* ``setpause`` and ``setstepmul`` options of ``collectgarbage`` function are + not used, as they are removed from Lua 5.5. +* ``global`` word is not used as a variable name. It is the keyword of the + global declarations introduced by Lua 5.5. It is still accepted as a name + by default but Lua may be built with this word reserved. + +luacheck does not know Lua 5.5, so it does not catch these points: they are +covered by running the integration tests with this version. ``lfs`` library +1.9 and ``posix`` library 39 build and work on Lua 5.5. + +Global declarations of Lua 5.5 do not change how evaluation is sandboxed. They +are checked when code is compiled, and the global variables they declare are +still stored in the environment table of the evaluation. + +Lua 5.1, Lua 5.2 and LuaJIT are not supported, whereas Lmod runs on them. +Supporting them requires a limited amount of alternative code (``setfenv`` +function instead of the environment argument of ``load`` to set the evaluation +environment of modulefiles, standard functions that were renamed). But it +restricts the Lua code to the features of Lua 5.1 and requires to run the +tests with this version, for which the systems to support do not provide the +``posix`` and ``lfs`` libraries. It is not retained as these systems all +provide Lua 5.3 or newer. + +As a consequence, a site whose Lmod runs on Lua 5.1 or LuaJIT has to use a +newer Lua interpreter with Modules. Its modulefiles may behave differently +there, whatever the module tool: number to string conversion has changed with +the integer type introduced by Lua 5.3 and some standard functions have been +removed. The ``unpack`` and ``loadstring`` functions of Lua 5.1 stay +available to modulefiles, as on Lmod (see `Sandboxing evaluation in Lua +interpreter`_). + +Lua 5.5 brings similar changes for modulefiles: assigning the control +variable of a ``for`` loop does not compile anymore, and float numbers are +converted to string with more digits (``0.1 + 0.2`` gives +``0.30000000000000004`` instead of ``0.3``). A modulefile relying on the +global declarations of Lua 5.5 is reported as erroneous by :command:`module +lint`, as luacheck does not know this syntax. + +``configure`` script checks that the designated Lua interpreter is of a +supported version. :file:`mfinterp.lua` also checks it when it starts, as the +Lua interpreter may be changed afterward through configuration: an error is +reported if the version is too old. + +This constraint applies to the Lua code of Modules. Lua modulefiles and +modulercs can use any feature of the Lua interpreter that evaluates them. + +Message format +^^^^^^^^^^^^^^ + +Every message is made of one single line. Each interpreter writes its +messages in the native syntax of the interpreter that receives them. This way +messages are parsed by the native parser of each language, which is faster and +safer than a parser written in script. + +* Message sent by Lua interpreter is a valid Tcl list. First element is the + message kind: a modulefile command to evaluate, a helper procedure to call + or the end of evaluation signal. For the two first kinds, next elements are + the Tcl command name and its arguments. Tcl interpreter evaluates the + command and its arguments as a list, without any additional substitution. +* Message sent by Tcl interpreter is a Lua ``return`` statement, that Lua + interpreter evaluates with ``load``. First returned value is the message + kind, either an evaluation request or a command reply. Evaluation + properties are expressed as a Lua table and the names of the environment + variables changed as a Lua list. + +Modulefile commands are evaluated in the modulefile Tcl interpreter. Helper +procedures answer the queries that do not correspond to a modulefile command +(like getting the loaded modules with their modulefile). They are called in +the main Tcl interpreter and are restricted to a fixed list of procedures. + +Result of a modulefile command is a Tcl value, which has no type. It is sent +as a Lua string, or omitted when empty, and the Lua function that has sent the +command converts it into what it has to return (a boolean for ``isloaded`` for +instance). Helper procedures are written for the bridge: they express their +result in Lua syntax, so its type is preserved. Lua script distinguishes this +way a boolean from a string, and an undefined value (``nil``) from an empty +string. + +Evaluation properties that only depend on the evaluation mode (list of the +commands that are no-operation, shell name and kind) are sent with the first +evaluation request made for this mode. Lua script keeps them for the next +evaluations in the same mode, whose request only carries what is specific to +the modulefile. + +Lua script has a dedicated function to quote a string as a Tcl list element: +special characters are escaped with a backslash and newline characters are +translated into their ``\n`` escape sequence. Enclosing strings in braces is +not enough: it fails on strings containing unbalanced braces or ending with a +backslash and it would let a value inject Tcl code. + +Tcl side has the equivalent procedure to quote a string as a Lua string +literal: backslash, double quote, newline, carriage return and null +characters are escaped. + +As newline characters are always escaped, a message never spans over multiple +lines. No multi-line marker or length prefix is needed. Modulefile content is +transmitted this way, as a quoted Lua string in the evaluation request. It is +preferred over sending the path of the modulefile to let the Lua interpreter +read it: one single read is needed to get the content and the evaluation +benefits from module cache. + +Both ends of the pipe are fully buffered and explicitly flushed once a message +is written, to get one write system call per message. Pipe channel is +configured with ``lf`` translation on Tcl side and keeps the system encoding, +to transmit environment variable values unaltered. + +Protecting the communication channel ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Information exchange from Tcl interpreter to Lua interpreter is done via the -channel opened when Lua interpreter is executed as a piped command. +*stdin* and *stdout* of the Lua process are the two ends of the bridge. If +left in place, a ``print`` call in a modulefile corrupts the message flow and +a command executed by the modulefile may consume the messages sent by the Tcl +interpreter. + +When it starts, Lua script duplicates these two file descriptors onto other +descriptor numbers, flagged *close-on-exec*, and uses them for the bridge. +Then *stdout* is redirected onto *stderr* and *stdin* onto :file:`/dev/null`. +Output produced by the commands that modulefiles execute is this way sent to +*stderr*. + +Output written by the modulefile code itself does not reach these file +descriptors. Messages written by a Tcl modulefile with ``puts stderr`` go +through the report code of Modules: they are placed in the message block of +the module being evaluated and follow verbosity and pager settings. For Lua +modulefiles to behave the same, the output functions of the evaluation +environment send their text to Tcl side as a ``puts stderr`` command: + +* ``print`` function +* ``io.write`` function +* ``write`` method of ``io.stdout`` and ``io.stderr``, which are replaced in + the evaluation environment by objects that only provide the methods of an + output file + +Text written on *stdout* is reported on *stderr* like the rest. It is not +sent to the shell for evaluation: output of a modulefile that is not shell +code would break the evaluation of the whole :command:`module` command +result. + +Environment variable changes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Environment variable changes are tracked where they are made on Tcl side: +procedures that update the ``env`` array when evaluating modulefile commands +(``set-env``, ``unset-env`` and ``reset-to-unset-env``) record the name of +the variable they change as soon as a Lua interpreter is running. Names of +the variables changed since the previous message are added to the command +reply or to the evaluation request, then record is cleared. + +Only names are sent, not values. Lua script flags the variables it is notified +of as outdated in its process environment and fetches their value from Tcl +side, through a helper procedure, when this value is needed: + +* When the modulefile reads an outdated variable with ``os.getenv``: this + variable is fetched. +* Before the modulefile runs an external command (``capture``, ``subprocess``, + ``os.execute`` and ``io.popen`` functions): all outdated variables are + fetched in one exchange, for the command to run in an accurate environment. + +Helper procedure returns a Lua table that associates each requested variable +to its value, or to ``false`` if the variable is not defined. Lua script +applies it to its own process environment (``setenv`` from the ``posix`` +library) and clears the outdated flag of these variables. + +``os.getenv``, ``os.execute`` and ``io.popen`` functions of the evaluation +environment are thus wrappers of the standard ones. The same wrappers replace +these functions in the global environment of the Lua script, for the code +loaded with ``require`` or ``loadfile`` to behave the same. + +Sending the values with each message, which is simpler, has been implemented +first in a prototype and discarded. Variables that get a new element at each +module load make the size of messages grow with the number of loaded modules: + +* Variables holding the state of Modules (``LOADEDMODULES``, ``_LMFILES_``, + ``__MODULES_LMPREREQ``, etc) are changed by every module load. Loading a + module with 136 dependencies whose modulefiles only declare their + requirements transmits 2.5 megabytes of such values to Lua interpreter and + takes more than 20% longer than with Tcl modulefiles. +* Path-like variables (``PATH``, ``LD_LIBRARY_PATH``, etc) are changed by most + modulefiles. With 10 path commands per modulefile on the same load, 3 + megabytes are transmitted and time spent on the bridge is more than doubled. + +Modulefiles seldom read the variables they or their dependencies set, and even +less the state of Modules. Fetching values on request makes the size of the +messages independent of the size of the environment: on the above loads, 0.1 +and 0.2 megabyte are sent to Lua interpreter instead. + +These procedures are used whatever the language of the modulefile that +changes the environment. Changes made by a Tcl modulefile are thus +transmitted the same way: + +* A Lua modulefile loads a module whose modulefile is written in Tcl: the + variables set by this Tcl modulefile are recorded during its evaluation and + their names are sent with the reply to the ``load`` function. The Lua + modulefile gets their value with ``os.getenv`` right after this call. +* A Tcl modulefile is evaluated between two Lua modulefiles that do not + depend on each other: names of the variables it sets are sent with the + evaluation request of the second Lua modulefile. This is why changes are + recorded even when no Lua evaluation is ongoing. + +A Tcl modulefile that directly writes to the ``env`` array instead of using +modulefile commands bypasses these procedures. Such change is not transmitted +to the Lua interpreter. + +An environment variable that is unset by a modulefile is kept defined on +Tcl side, set to an empty value and flagged as cleared, not to break its later +reference by a Tcl modulefile. Lua language distinguishes an undefined value +from an empty string, thus such variable is reported as not defined to the +Lua script. + +Value of a variable is read from the ``env`` array when Lua script asks for +it. A variable changed several times before being read is this way transmitted +once, with its last value. + +Environment is also changed on Tcl side when a module evaluation fails: the +whole evaluation context saved before this evaluation is restored +(``restoreSettings`` procedure), including the content of the ``env`` array. +This restoration does not go through the procedures mentioned above. It +records itself the variables it sets back to their previous value or unsets, +so Lua interpreter is notified of them with the next message. For instance, +when the load of a module requested by a Lua modulefile fails, the environment +changes made by this module before its failure are undone on Tcl side, and the +reply to the ``load`` function carries the names of the restored variables in +addition to the error. The requesting modulefile, if it continues or catches +the error, this way reads the same values as the ones known on Tcl side. This +applies whether the failed modulefile is written in Lua or in Tcl. + +Any other code that changes the ``env`` array while a Lua interpreter is +running has to record the variables it changes the same way. -Thus Lua interpreter gets the information through its stdin channel. Stdin is -read line by line, within a co-routine, not to block the interpreter execution -with an I/O wait. +Saving the whole environment before each command evaluation to compare it +afterward is not an option: fetching the full content of the ``env`` array is +by far the most expensive operation of an exchange (see `Performance +considerations`_). -Each line read from stdin is analyzed to determine if another line should get -read immediately (before yielding). The ``__MODULES_ITRP_EXCHANGE__`` marker -is used to indicate multiple lines should be read, until reaching another -`__MODULES_ITRP_EXCHANGE__`` marker line. +Nested evaluations +^^^^^^^^^^^^^^^^^^ -An empty modulefile content or an empty list of environment change have to be -sent by the Tcl interpreter to release the Lua interpreter from its stdin read -wait. +Evaluating a Tcl command sent by Lua interpreter may trigger the evaluation +of another Lua modulefile or modulerc: a ``load`` Lua function for instance +that leads to the load of a module whose modulefile is written in Lua, which +may itself load its own dependencies and so on. When this happens, Lua +interpreter is waiting for the reply to its command and receives an evaluation +request instead. -Reading things line by line may have an impact on performances, for instance -if modulefiles have numerous code lines. It should be evaluated. +After sending a command, Lua interpreter thus reads messages in a loop: an +evaluation request received is evaluated right away, then loop continues +until getting the command reply. On Tcl side, the nested evaluation runs its +own read loop until it gets its end of evaluation signal. -Some leads to improve the Tcl-Lua performances: +Evaluation levels on Tcl side +""""""""""""""""""""""""""""" -* Improve asynchronous I/O co-routine mechanism to be able to read full content - of modulefile in one read call (is this possible?) -* Reduce the number of information to exchange between the 2 interpreter +On Tcl side, each level of evaluation has its own interpreter. Modulefile +interpreter is named after the evaluation mode and the depth of the +evaluation (number of modulefiles currently evaluating), and modulerc +interpreter after this depth. A modulefile evaluated while another one is +evaluating thus never shares the interpreter of this other modulefile. These +interpreters are kept to be reused by the next evaluation occurring at the +same level, after their state has been reset. - * Compute environment variable changes directly within Lua interpreter, but - it will need to handle any protected variables or other kind of - mechanisms (but these information may be sent with other properties via - *DATA*). - * Send modulefile path instead of content to read it directly from Lua - interpreter. But doing so, it will not benefit from module cache. +This organization is unchanged for Lua modulefiles. The Tcl commands a Lua +modulefile is translated into are evaluated in the modulefile interpreter of +its level, like the commands of a Tcl modulefile evaluated at this level would +be. Tcl side remains the reference for the evaluation depth, which counts Tcl +and Lua modulefiles alike: a Lua modulefile that loads a Tcl modulefile that +loads a Lua modulefile leads to evaluations at depth 1, 2 and 3, whereas Lua +interpreter only sees the first and the last one. + +Evaluation levels on Lua side +""""""""""""""""""""""""""""" + +There is one single Lua interpreter, thus one single Lua state, whatever the +number of nested evaluations. A nested evaluation is a nested function call: +the outer evaluation is suspended in the function that waits for the reply to +its command, and the inner evaluation runs from this function, deeper in the +call stack of the Lua interpreter. When the inner evaluation ends, the wait +loop of the outer one resumes. + +No Lua counterpart of the per-level Tcl interpreters is needed to isolate +evaluations from each other. Everything that belongs to an evaluation is held +in an evaluation context, created when the evaluation request is received and +dropped when the evaluation ends. These contexts are stacked: the context at +the top of the stack is the one of the evaluation currently running. An +evaluation context holds: + +* The evaluation properties received in the request: modulefile path, module + name, evaluation mode, list of Tcl commands that are no-operation in this + mode, etc. Nested evaluations may use different modes: a module loading in + ``load`` mode may trigger the unload of another one, evaluated in ``unload`` + mode. +* The evaluation environment: table holding the variables and functions the + modulefile defines (see `Sandboxing evaluation in Lua interpreter`_). A + nested modulefile does not see what the modulefile that triggers its + evaluation has defined, and cannot change it. +* The evaluation status: whether a stop of this evaluation has been requested. + +Lmod functions find the context they apply to at the top of the stack. They +are shared by all evaluations and hold no evaluation-specific information. + +As an evaluation context is cheap to create, contexts are not kept to be +reused like Tcl interpreters are. Each evaluation starts from a new one, so +there is no state to reset. + +Pairing messages with evaluations +""""""""""""""""""""""""""""""""" + +Evaluations are strictly nested on both sides: an evaluation always ends +before the one that has triggered it resumes. As a consequence: + +* A command sent by Lua interpreter always belongs to the innermost Lua + evaluation. On Tcl side, the read loop that receives it is the one of this + evaluation, which knows the Tcl interpreter to evaluate the command in. +* A command reply sent by Tcl interpreter always answers the last command sent + by the innermost Lua evaluation. +* An evaluation request received by Lua interpreter always starts a new + evaluation, nested in the current one if any. + +Messages thus do not carry any evaluation identifier. + +What is shared between evaluation levels +"""""""""""""""""""""""""""""""""""""""" + +* Process environment of the Lua interpreter and the record of its outdated + variables. An environment variable changed by a nested evaluation is seen by + the outer evaluation once it resumes, as the changes are notified with the + command reply. It is the same on Tcl side, where all interpreters share the + same environment. +* Lua modules loaded with ``require``. A module loaded by a modulefile stays + loaded for the evaluations that follow, as on Lmod. Global variables defined + by code loaded with ``require`` or ``loadfile`` are shared the same way. + +Errors and evaluation stop +"""""""""""""""""""""""""" + +Each evaluation runs in its own protected call. An error raised in a nested +evaluation, or a request to stop it, ends this evaluation only: Lua +interpreter signals its end to Tcl, which determines the result of the command +that has triggered it. The outer evaluation then receives the reply to this +command like for any other command: a successful result, an error it may +catch, or a request to stop in turn. + +A stop request is recorded in the evaluation context, not globally: stopping a +nested evaluation does not stop the outer ones. + +Nothing has to be undone on Lua side when a nested evaluation fails. Global +variables and functions it has defined are dropped with its environment +table. Changes it has made through modulefile commands are held on Tcl side, +which restores its state and transmits the environment variables to set back +(see `Environment variable changes`_). + +If the Lua interpreter ends unexpectedly, every evaluation in progress in +this interpreter is lost. Tcl side reports an error for each of them, as each +of their read loops gets the end of the pipe, and starts a new Lua interpreter +at the next Lua evaluation. + +Nesting limit +""""""""""""" + +Lua interpreter limits the depth of nested calls made through its C functions, +and each nested evaluation consumes a part of it. About 190 Lua evaluations +can be nested, on Lua 5.3, Lua 5.4 and Lua 5.5 alike. This is far above the +depth of actual module dependency chains. If this limit is reached, the +innermost evaluation fails with an error, reported like any other Lua +evaluation error. Evaluation end from Tcl or Lua ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -109,18 +820,94 @@ Evaluation end from Tcl or Lua Evaluation may end on the Lua side: * Everything goes well: end of evaluation is signaled by Lua interp to Tcl -* Code does not compile: report module error Tcl code is sent to Tcl -* Code evaluation fails: report module error Tcl code is sent to Tcl unless an - *EXIT* order was already received from Tcl (a Lua internal marker, - ``__MODULES_ITRP_EXITEVAL__``, is used to convey this *EXIT* order) +* Code does not compile or its evaluation fails: end of evaluation is + signaled with the error message and an error code + +The error is not reported by a command sent from Lua side. It is carried by +the end of evaluation signal, and the command of the modulefile Tcl +interpreter that has requested the Lua evaluation raises it there, with its +error code. A failed Lua evaluation this way goes through the code that +handles a failed Tcl evaluation: same error counting and return code, same +downgrade into a warning when a module is unloaded in force mode. A dedicated +error code designates the errors coming from Lua, for their message to be +reported with the path of the modulefile instead of a Tcl stack trace. + +Evaluation may end on the Tcl side, when the evaluation of a Tcl command +requests to stop the evaluation (like ``exit`` or ``break``). In this case, +the status of the command reply indicates that evaluation should stop. Lua +interpreter then raises an error whose value is an internal object rather than +a message. Once this specific error is caught, Lua interpreter knows that +there is nothing to report and just signals the end of evaluation. Tcl side +has kept the return code of the command that has requested the stop: once the +end of evaluation signal is received, this code is returned in the modulefile +Tcl interpreter, as if the command was called by a Tcl modulefile. -Evaluation may end on the Tcl side: +A modulefile may call a function through ``pcall`` and catch this way the +error raised to stop the evaluation. Stop request is thus recorded in the +context of the evaluation on Lua side: any function translated into a Tcl +command that is called afterward raises the stop error again without any +exchange, and the end of evaluation signal is sent whatever the way the +modulefile code ends. -* Tcl code is evaluated to handle an error raised on the Lua side (either the - code does not compile or its evaluation fails): in this case, nothing has to - be sent to the Lua interp, as evaluation is already finished there -* Tcl code evaluation raises an error: an *EXIT* order is sent to the Lua - interp to kill current evaluation. +Errors raised by Tcl commands +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When the evaluation of a Tcl command raises an error, the command reply +conveys the error message and the Tcl error code. A regular Lua error is then +raised from the place where the Lua function is called in the modulefile. Lua +modulefile may catch it with ``pcall``, like a Tcl modulefile catches the +errors of modulefile commands with ``catch``. + +If this error is not caught, the Tcl error code is transmitted back with the +end of evaluation signal. Errors that have already been reported (like the +load failure of a requirement) or that should be reported as-is are this way +rendered like they are for Tcl modulefiles. To achieve this, Lua script +records in the evaluation context the last error received from Tcl side: the +error that ends the evaluation gets the Tcl error code if it is this recorded +error. + +Message of an error that carries an error code of Modules is raised unchanged +on Lua side. Other errors of Tcl commands, like a wrong number of arguments, +are internal to the translation: their message is prefixed by the location of +the function call in the modulefile and they are reported as Lua errors. + +Location of errors +^^^^^^^^^^^^^^^^^^ + +Modulefile content is loaded with its file path as chunk name. Lua error +messages this way designate the modulefile and the line number. Lua shortens +long chunk names in its error messages: the full modulefile path, known from +the evaluation properties, is restored in the message before reporting it. + +For errors raised by Tcl commands, the location in the Lua modulefile is +computed only when the error occurs (with ``debug.getinfo``), by walking up +the call stack until reaching a function that belongs to the modulefile. +Nothing has to be computed or transmitted for the commands that succeed. + +Lua error messages are reported as-is. Lua script does not interpret them nor +complement them with a guess of what the modulefile intended. For instance a +modulefile calling a function that is not provided gets the regular Lua error +raised for the call of an undefined variable. Like every Lua evaluation +error, it is transmitted to Tcl side which reports it the way an evaluation +error of a Tcl modulefile is reported. + +Lua interpreter ignores the first line of a file it loads if this line starts +with a ``#`` character. This is not the case when code is loaded from a +string, like done here. This first line is thus blanked before loading the +modulefile content, which preserves the line numbering. + +Debugging the bridge +^^^^^^^^^^^^^^^^^^^^ + +Exchanges are traced on Tcl side, where every message goes through, with the +existing debugging verbosity levels: + +* Start and end of the Lua interpreter, with its command line, and start and + end of each Lua evaluation are reported at the ``debug`` level. +* Every message sent to or received from the Lua interpreter is reported at + the ``debug2`` level. + +Lua script writes no debugging message by itself. Sandboxing evaluation in Lua interpreter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -129,18 +916,236 @@ Lua provides a convenient ``load`` function to evaluate Lua code. A restricted environment can be defined for this evaluation. With that feature, the list of functions reachable by evaluated code can be controlled. -With ``load``, the variables or functions defined by some evaluated code are -accessible from the next ``load`` evaluation. Some cleanup should be done thus -between evaluations, like performed on Tcl side. Such feature may not be -necessary on day 0 as Lmod is not providing such cleaning between evals. - ``load`` command first evaluate code to produce a function. During this step it catches any syntax error. Then this function can be evaluated. +A base environment table is built once when Lua script starts. It contains +the Lua functions and libraries made available and the Lmod functions. Library +tables (``os``, ``io``, ``string``, etc) set in this base environment are +copies of the original ones: changing a function in the sandbox, like +``os.exit``, has this way no effect on the function used by the Lua script +itself. + +A modulefile may also change a function of a library table. For such change +not to be seen by the other evaluations, whether nested or following, an +evaluation does not use the library tables of the base environment directly: +it gets its own copy of a library table the first time it accesses it. + +Two functions of this base environment are defined as done by Lmod, whatever +the version of the Lua interpreter: + +* ``loadstring`` compiles a string of Lua code. It is the ``load`` function of + the Lua standard library, which is not reachable under its own name as + ``load`` designates the Lmod function that loads modules. ``loadstring`` is + the name of this function on Lua 5.1 and the one Lmod documents for + modulefiles. +* ``unpack`` is the ``table.unpack`` function, which was a global function on + Lua 5.1. + +These two functions are part of what modulefiles can rely on. They are +described in the documentation of the Lua modulefile functions. + +Functions called on a string with the method syntax (``s:upper()``) are not +looked up in the ``string`` table of the evaluation environment but in the +original ``string`` table, through the metatable shared by all strings. Two +consequences follow: + +* A modulefile that changes a function in its ``string`` table does not + change what the method syntax calls, for itself and for the others. +* String functions added by Lmod (``split``, ``escape``, ``trim``, etc) are + used by modulefiles with the method syntax (``s:split(":")``). They are + thus defined in the original ``string`` table when Lua script starts, in + addition to the ``string`` table of the base environment. + +Lmod exposes to modulefiles a subset of the standard library tables and a +few functions of the ``lfs`` and ``posix`` libraries (``posix.uname``, +``posix.setenv``, ``posix.stat``, ``posix.open``, ``posix.hostid`` and the +syslog functions). The base environment provides at least this set. As the +Lua script only loads the sub-modules of the ``posix`` library it requires, +the ``posix`` table of the base environment is built from these functions, +each one loaded from its sub-module the first time it is accessed. +``posix.setenv`` called by a modulefile only changes the environment of the +Lua process, like on Lmod: the change is not known by Tcl side and is not +applied to the shell. + +``dofile`` function raises an error stating it is not supported, as on Lmod. + +Code loaded by a modulefile with ``require`` or ``loadfile`` is not evaluated +in the evaluation environment of this modulefile but in the global environment +of the Lua script. Sites use this to share helper functions between +modulefiles, and such code calls Lmod functions like ``pathJoin`` or +``prepend_path``. For this to work: + +* Lmod functions are also defined as global variables of the Lua script, + under the name Lmod gives them. ``load`` is the exception: it stays there + the function of the Lua standard library, and the function that loads + modules is named ``load_module``, as on Lmod. +* These functions apply to the evaluation in progress, whose context is at + the top of the stack, wherever they are called from. +* Lua script keeps its own local references to the standard functions it + uses, so code loaded this way cannot break the bridge by redefining a + global function. + +Global variables defined by such code belong to the Lua script and last as +long as the Lua process. They are not reset between evaluations. + +Each evaluation gets a new empty table as environment, whose metatable +redirects unknown names to the base environment. Variables and functions +defined by a modulefile are stored in this table that is discarded once +evaluation is over. State is this way reset between evaluations, and nested +evaluations are isolated from each other, at almost no cost. + ``os.exit`` command is changed into a call to the Tcl ``exit`` modulefile command. ``os.exit`` would otherwise ends current modulefile evaluation but also the whole Lua interpreter script. +Work split between Lua and Tcl +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Lua functions fall in three categories: + +* Functions that are fully handled by the Lua script, without any exchange. + It concerns functions relying on evaluation properties (``myFileName``, + ``myModuleName``, ``myShellName``, ``mode``, etc), on file system + (``pathJoin``, ``isFile``, ``isDir``, etc), on environment (``os.getenv``) + or on external commands (``capture``, ``subprocess``, etc). +* Functions that change the state handled by Tcl interpreter (``setenv``, + ``prepend_path``, ``load``, ``whatis``, etc). They are translated into a Tcl + modulefile command and wait for its reply. +* Functions that query the state handled by Tcl interpreter (``isloaded``, + ``isAvail``, etc). They are translated into a Tcl modulefile command and + return the result obtained in reply. + +Depending on the evaluation mode, some Tcl modulefile commands are +no-operation (most commands on ``whatis`` mode for instance). The list of +these commands is known by Tcl interpreter and is sent to the Lua script among +the evaluation properties of the mode (see `Message format`_). Lua script does +not send the commands that are part of this list. + +Commands that set environment variables are not part of this list, whatever +the mode. On ``whatis`` mode for instance, ``setenv``, ``pushenv``, +``unsetenv`` and the path commands still update the environment on Tcl side, +for the rest of the modulefile to read the variables they set. They are sent +to Tcl interpreter like on ``load`` mode. + +Performance considerations +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following orders of magnitude have been measured with Tcl 9.0 and Lua 5.4 +on a process environment made of about 100 variables: + +* An exchange over the pipe alone (message sent then reply received) costs 5 + to 10 microseconds, depending on the load of the machine. Within an actual + evaluation, 15 to 25 microseconds elapse on Tcl side between a message sent + and the next message received, which covers the Lua code run in between. + Building and parsing messages cost about 5 more microseconds on Tcl side. A + Lua function translated into a Tcl command thus costs 20 to 30 microseconds + more than the Tcl command evaluated alone. +* Fetching the full content of the ``env`` array on Tcl side costs about 400 + microseconds. This is why environment changes are tracked rather than + computed. +* Starting the Lua interpreter costs a bit more than 1 millisecond. Loading + the whole ``posix`` library adds 3 milliseconds, whereas loading one of its + sub-modules (like ``posix.stdlib``) adds a fraction of millisecond. Lua + script thus only loads the sub-modules it requires. + +The cost of one exchange per command is small compared to the evaluation of +the command itself. Gathering several commands in one message has been +considered: it saves the exchange, about 20 microseconds per command, which +represents a fraction of millisecond for a typical modulefile. In return, +errors raised by Tcl commands are detected after the Lua code has moved on, +and pending commands have to be flushed before anything that may observe +their effect. Commands are thus sent one by one. As a message from Lua +interpreter is a Tcl list, batching could be introduced later by sending a +list of commands without changing the message format. + +Starting a new Lua process for each evaluation has also been considered. It +is the simplest architecture: nested evaluations and state reset between +evaluations come for free. Two prototypes built this way have been measured: +both add about 3 milliseconds per Lua modulefile compared to the equivalent +Tcl modulefile (30 modulefiles of 15 commands loaded in about 300 milliseconds +instead of 200). With a persistent Lua process, the start cost is paid once +and each modulefile then costs a few exchanges, 0.2 to 0.4 millisecond for +such modulefiles. + +The overhead of Lua modulefiles on a large load has been measured with a +prototype of the bridge, on the 137 modulefiles loaded by the ``load2`` test +of :file:`script/mb` and on their translation in Lua. These modulefiles mostly +declare requirements. Path and variable commands have been added to them to +get larger modulefiles. Time spent on the bridge is measured on Tcl side: it +covers the messages built and parsed and the wait for Lua interpreter, start +of the Lua process included. + ++--------------+------------+------------------+---------------+----------+ +| Commands per | Messages | Load time of Tcl | Time spent on | Overhead | +| modulefile | sent | modulefiles | the bridge | | ++==============+============+==================+===============+==========+ +| 5 | 755 | 0.45 s | 20 ms | 4.4 % | ++--------------+------------+------------------+---------------+----------+ +| 15 | 2125 | 1.4 s | 70 ms | 5.0 % | ++--------------+------------+------------------+---------------+----------+ +| 35 | 4865 | 4.7 s | 115 ms | 2.4 % | ++--------------+------------+------------------+---------------+----------+ + +Run time of the whole command confirms these values: :file:`script/mb` reports +about 475 milliseconds for the ``load2`` test and 500 milliseconds for its +``load2_lua`` counterpart. This difference is of the same order as the +variation observed between two runs of the same test on a machine that is not +idle. + +The cost of a command on the bridge is constant, whereas the cost of running +this command on Tcl side grows with the number of loaded modules: from 0.4 +millisecond when a couple of modules are loaded to about 1 millisecond with a +hundred of modules. The larger the modulefiles, the smaller the relative +overhead of Lua modulefiles. The same applies when unloading or purging these +modules. Listing loaded modules evaluates no modulefile, thus no Lua +interpreter is started. + +This prototype has shown that two points weigh more than the bridge itself if +they are overlooked: + +* Sending the value of the changed environment variables with each message + multiplies the time spent on the bridge by about 4 on the first of these + loads and by more than 6 on the last one (see `Environment variable + changes`_). +* Walking the full content of the directories containing ``.lua`` files when + an exact module version is searched adds 10 milliseconds on the first load + (see `Precedence between Lua and Tcl files`_). + +Global rc files are evaluated at every run of the :command:`module` command, +whatever the sub-command. When one of them is written in Lua, the Lua +interpreter is started at every run, including for the sub-commands that +evaluate no modulefile like :subcmd:`list`. The start cost, less than 2 +milliseconds, is then paid by every command. A site that wants to avoid it +writes its global rc files in Tcl. Lua rc files located in modulepath +directories only start the Lua interpreter when these directories are +searched. + +The overhead is the most visible on a command evaluating a single small Lua +file: starting the Lua process then represents a few percent of the run time +of the :command:`module` command. + +For this overhead to stay constant per command, the following points have to +be respected by the implementation: + +* No work proportional to the size of the environment or to the number of + loaded modules is done per command on the bridge: only the names of the + variables changed by the command are sent back to Lua interpreter. +* Reply to a command that changes nothing in environment carries no + environment change structure. +* Evaluation properties that do not depend on the modulefile are not sent + with each evaluation request. +* Commands that are no-op in the current evaluation mode are handled on Lua + side, without any exchange. A ``whatis`` evaluation over a whole modulepath + this way only exchanges for the ``whatis`` function calls and for the + functions that set environment variables. +* Lua interpreter is started only when a Lua file has to be evaluated. + +Linking Lua interpreter into the Tcl process through a C extension would +save the pipe exchanges and the process start. It is not retained as it +requires Lua development files at build time and makes Lua modulefile support +dependent of the Modules Tcl extension library, which is optional. + Handling evaluation modes ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -150,13 +1155,1076 @@ the modulefile commands. As the Lua interpreter sends to the Tcl interpreter a conversion of the Lua commands in Tcl commands, it does not need to handle the mode particularities. -The ``help`` Lua command is translated into new ``module-help`` Tcl command -outputting its string argument to the stderr channel. +The ``help`` Lua function is translated into the ``module-help`` Tcl +command. Its arguments are concatenated into a single string. There is no equivalent of ``ModulesDisplay`` and ``ModulesTest`` in Lmod. Also Lmod has no ``test`` evaluation mode. Running ``module spider`` evaluates modules with ``spider`` mode on Lmod whereas ``scan`` mode is used on Modules. +The ``mode`` Lua function returns the name Lmod gives to the current +evaluation mode, as modulefiles written for Lmod compare its result to these +names: + +* ``load``, ``unload``, ``help`` and ``whatis`` modes have the same name on + both tools +* ``show`` is returned for ``display`` mode +* ``spider`` is returned for ``scan`` mode +* ``load`` is returned for ``refresh`` mode, as Lmod does when it refreshes + the loaded modules +* ``test`` is returned for ``test`` mode, which has no Lmod equivalent + +Name of the Modules evaluation mode is still part of the evaluation +properties, for the needs of the Lua script. + +A few Lmod behaviors depend on the evaluation mode and cannot be obtained +from the sole translation into a Tcl command. They are handled by the Lua +script, which knows the evaluation mode from the evaluation properties: + +* ``modeA`` table option of Lmod functions: the function is skipped if the + current evaluation mode is not part of the listed modes. When unloading, a + function enabled for the ``unload`` mode is applied as-is instead of being + reversed. To achieve this, the load mode variant of the Tcl command is + called through a helper procedure. As on Lmod, the function is always + applied on ``display`` mode, where it is reported, and it is applied on + ``scan`` mode if ``load`` is part of the listed modes. +* ``modeA`` only accepts the ``load``, ``unload`` and ``normal`` values. + ``normal`` designates the regular behavior of the function, the same as + when no ``modeA`` option is set. Another value is an error, like any + unknown option name in the table passed to a function. +* ``LmodError`` is reported as a warning when unloading. It does nothing on + ``whatis``, ``help`` and ``scan`` modes, where the evaluation continues, so + information of a module can be queried even if it cannot be loaded. + ``LmodSystemError`` raises an error whatever the mode. +* ``LmodMessage`` and ``LmodMsgRaw`` do nothing on ``whatis``, ``help``, + ``scan`` and ``refresh`` modes. +* ``LmodBreak`` only stops the evaluation when loading. +* ``unload`` unloads the designated modules when loading and when unloading + on Lmod. Tcl :mfcmd:`module unload` command does nothing when + unloading, which is kept. + +Mapping Lmod functions to Tcl commands +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Most Lmod functions have a direct Tcl modulefile command equivalent. Some +mappings require attention: + +* ``prereq`` and ``prereq_any`` only check that the requirement is loaded, + whereas ``depends_on`` and ``depends_on_any`` load it automatically. They + respectively map to the ``prereq-all``, ``prereq-any``, ``depends-on`` and + ``depends-on-any`` commands. Whether the requirement is automatically + loaded depends on Modules side on the automated module handling mode rather + than on the command used. +* ``isloaded`` returns true for a module that is loaded or currently loading. + ``is-loaded`` Tcl command only returns true for a loaded module: its result + is combined with the query of the modules currently loading that + ``isPending`` relies on. +* Version modifier functions (``between``, ``latest``, etc) return an object + that is translated into an advanced version specifier when passed to a + function that expects a module specification (``between("foo", "1.2", + "2.0")`` gives ``foo@1.2:2.0``). Advanced version specifiers have to be + enabled. +* An exclusive upper bound (``<`` character before version) is translated + into an abbreviated upper bound, which includes all versions that extend + it. Trailing elements of the version equal to ``0`` are dropped, then its + last element is decreased by one: ``between("foo", "2.7", "<3.0")`` gives + ``foo@2.7:2`` and ``between("foo", "2.7", "<3.2")`` gives ``foo@2.7:3.1``. + An error is raised if the last element is not a number or if no element + remains. +* An exclusive lower bound (``<`` character after version) cannot be + expressed with advanced version specifiers. An error is raised when such + bound is used. +* ``priority`` option of ``prepend_path`` and ``append_path`` maps to the + ``--priority`` option of the corresponding Tcl commands. +* Third argument of ``prepend_path``, ``append_path`` and ``remove_path``, or + their ``delim`` option, maps to the ``--delim`` option of the corresponding + Tcl commands. +* ``remove_path`` maps to ``remove-path`` with ``--remove-on-unload`` option. +* ``setenv`` called with a true third argument only sets the variable if it + is not defined yet. It maps to ``setenv`` with ``--set-if-undef`` option. +* ``set_shell_function`` receives the function body for *sh* shells and the + one for *csh* shells. The body that corresponds to the current shell is + passed to ``set-function``. As *csh* shells have no function, the second + body is defined there as an alias with ``set-alias``, as done by Lmod. + ``unset_shell_function`` follows the same rule. +* ``complete`` and ``uncomplete`` receive a shell name as first argument. + ``uncomplete`` Tcl command does not have this argument: Lua script skips + the call if the shell name does not match the current shell. +* ``source_sh`` receives the script and its arguments as a single string, + which is split into words to call ``source-sh``. +* ``extensions`` receives a single string of comma-separated names, which is + split to call the ``extensions`` Tcl command with one argument per name. +* ``add_property`` maps to ``add-property``, which sets its value argument as + a tag. ``remove_property`` is a no-operation. Properties are not described + in a properties file like on Lmod (see `Properties file`_). +* ``family``, ``requireFullName``, ``always_load``, ``try_load``, + ``load_any``, ``purge`` and ``unsetenv`` map to the Tcl command or + :mfcmd:`module` sub-command of the same name. ``try_add`` and + ``always_unload`` are other names of ``try_load`` and ``unload``. +* ``LmodMessage`` and ``LmodMsgRaw`` map to ``puts stderr``, the latter with + ``-nonewline`` option. ``LmodWarning`` maps to ``reportWarning`` and + ``LmodError`` to the Tcl ``error`` command. Their arguments are + concatenated into a single string. +* ``isPending`` and ``moduleStackTraceBack`` query the modules currently + loading through a helper procedure. +* ``export_shell_function`` marks on Lmod a shell function as exported to + the sub-shells. ``set-function`` Tcl command always exports the function + it defines on the shells that support it. ``export_shell_function`` is + thus a no-operation. +* ``pushenv`` with a ``false`` value maps to a ``pushenv`` of an empty value. + The variable is then set to an empty string, whereas it is unset on Lmod. +* ``execute`` maps to a ``puts stdout`` of the command when current mode is + part of its ``modeA`` option. +* ``inherit`` looks for the modulefile of the same name in the modulepaths + that come after the one of the current modulefile. A Lua modulefile found is + evaluated in the current Lua evaluation environment. The content of a Tcl + modulefile found is evaluated in the current modulefile Tcl interpreter. +* ``capture``, ``subprocess`` and commands run with ``os.execute`` or + ``io.popen`` get the environment as currently changed by the modulefile, + since Lua process environment is updated before running them. + +Functions handled by the Lua script +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Functions that need no exchange with Tcl side are written in Lua. Their +result has to be the same as on Lmod for the same arguments, which is checked +by tests whose expected results are obtained from Lmod. The following ones +require attention: + +* ``LmodVersion`` returns the version of Lmod whose modulefile functions are + implemented by Lua script, not the version of Modules. Modulefiles compare + this value with ``convertToCanonical`` to know if a function is available. + Returning the Modules version would make them disable features that are + supported. This reference version is currently Lmod 9.4. It is raised when + all the modulefile functions of a newer Lmod version are handled. Functions + added by new Lmod versions are reviewed for each feature release of + Modules. +* ``convertToCanonical`` returns a string built so that the comparison of two + such strings orders the versions they come from. Version ordering has to be + the one of Lmod. +* File functions (``pathJoin``, ``dirname``, ``extname``, ``removeExt``, + ``barefilename``, ``splitFileName``, ``abspath``, ``realpath``, + ``path_regularize``, ``mkdir_recursive``, ``isDir``, ``isFile``) follow the + rules of Lmod. For instance ``pathJoin`` removes the trailing slash and + ``isFile`` is true for anything that exists and is not a directory. +* ``myShellType`` returns ``sh`` for the *sh*, *bash*, *ksh* and *zsh* + shells, ``csh`` for the *csh* and *tcsh* shells and the name of the shell + otherwise. ``myShellName`` returns the shell name as known by Modules. +* ``myModuleName`` and ``myModuleVersion`` split the full name of the module + at its last ``/`` character: what precedes is the name and what follows is + the version. A module whose full name contains no ``/`` character has an + empty version. Lmod determines this split from the layout of the modulepath + directory. Results only differ for modules with a version spanning over + several directory levels (for :file:`foo/1/2.lua` Lmod may return ``foo`` + and ``1/2`` where ``foo/1`` and ``2`` are returned here). +* ``myModuleUsrAndAliasName`` and ``myModuleFullNameAndAlias`` return two + values on Lmod, which differ when a module in a dot-hidden directory is + designated without the dot. Modules does not have this mechanism: the same + value is returned twice. +* ``userInGroups`` is true if current user is *root* or is member of one of + the groups passed. User and groups are obtained with the ``posix`` library. +* ``myConfig`` returns a configuration value of Lmod. ``username`` and + ``usergroups`` keys are supported. Other keys designate configuration that + has no counterpart: ``nil`` is returned. +* ``protected_getenv``, ``isDefined``, ``isNotDefined``, ``UUIDString``, + ``hierarchyA`` and ``is_spider`` only rely on the evaluation properties and + on the evaluation environment. +* ``colorize`` returns its text argument with color codes only if Modules is + configured to output colors. ``color_banner`` outputs a separator line on + *stderr*. +* ``dbg`` functions and ``haveDynamicMPATH`` are no-operation. +* ``capture`` runs on Lmod its command with the ``LD_LIBRARY_PATH`` and + ``LD_PRELOAD`` values recorded at installation, unlike ``subprocess``. Both + functions use the current environment on Modules. + +Lmod checks the type of the arguments passed to its functions: most functions +only accept strings. The same checks are applied by the Lua script, so an +erroneous call is reported with the name of the function and the location of +the call in the modulefile rather than as an error of a Tcl command. + +Identifying the module tool +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A Lua modulefile shared between Lmod and Modules needs to know which tool +evaluates it, and which version, to adapt to their differences or to use a +function that only one of them provides. Two distinct pieces of information +are available for this, on both tools: + +* ``LmodVersion`` function tells the level of the Lmod modulefile functions + that is supported. On Modules it returns the reference Lmod version, as + described above. It never tells anything about Modules itself: its result + does not change when functions specific to Modules are added. Lmod also + provides this version through environment variables of its process, that + some modulefiles read instead of calling the function: ``LMOD_VERSION``, + and ``LMOD_VERSION_MAJOR``, ``LMOD_VERSION_MINOR`` and + ``LMOD_VERSION_SUBMINOR`` for each of its elements. Lua script defines + these four variables in the environment of the Lua process when it starts, + from the reference Lmod version. An element this version does not have is + set to ``0``, as on Lmod. +* ``ModuleTool`` and ``ModuleToolVersion`` environment variables tell the + tool and its own version. Lmod defines them in the environment of its + process, set to ``Lmod`` and to its version, so modulefiles read them with + ``os.getenv``. Lua script defines them the same way in the environment of + the Lua process when it starts, set to ``Modules`` and to the version of + Modules. These are the values of the :mfvar:`ModuleTool` and + :mfvar:`ModuleToolVersion` variables of Tcl modulefiles. Like on Lmod, + these variables are not exported to the shell of the user. + +These two variables are not defined as Lua global variables, as Lmod does +not define such variables: a test written this way would not work on Lmod. + +All these variables only exist in the environment of the Lua process. They +are not known by Tcl side and not applied to the shell of the user. + +A modulefile then adapts to the tool as follows: + +.. code-block:: lua + + local tool = os.getenv("ModuleTool") + local version = convertToCanonical(os.getenv("ModuleToolVersion")) + + if tool == "Modules" and version >= convertToCanonical("5.9") then + -- use a function specific to Modules + elseif tool == "Lmod" then + -- Lmod way to achieve the same + end + + if convertToCanonical(LmodVersion()) >= convertToCanonical("8.6") then + -- use an Lmod function supported by both tools from this level + end + +Functions specific to Modules that are added at the last implementation +stage are thus tested with ``ModuleTool`` and ``ModuleToolVersion``, never +with ``LmodVersion``. + +Modulerc functions +^^^^^^^^^^^^^^^^^^ + +Lua rc files are evaluated in a more restricted environment than modulefiles. +Like on Lmod, it only contains the modulerc functions, ``myConfig``, +``pathJoin``, ``isDir``, ``isFile``, ``require``, ``loadfile`` and a subset of +the ``os`` and ``io`` tables. Modulefile functions are not available. + +Modulerc functions map to Tcl modulerc commands: + +* ``module_version`` and ``module_alias`` map to ``module-version`` and + ``module-alias``. +* ``hide_version`` and ``hide_modulefile`` map to the Tcl commands of the same + name. +* ``hide`` maps to ``module-hide`` and ``forbid`` to ``module-forbid``. Each + key of the table they receive has an option counterpart: + + +---------------------------+-------------------------------------------+ + | Key | Option | + +===========================+===========================================+ + | ``name`` | Modulefile arguments (one per name) | + +---------------------------+-------------------------------------------+ + | ``kind`` (``hide``) | ``--soft`` for ``soft``, ``--hard`` for | + | | ``hard``, no option for ``hidden`` | + +---------------------------+-------------------------------------------+ + | ``hidden_loaded`` | ``--hidden-loaded`` | + | (``hide``) | | + +---------------------------+-------------------------------------------+ + | ``before``, ``after`` | ``--before``, ``--after`` | + +---------------------------+-------------------------------------------+ + | ``userA``, ``groupA`` | ``--user``, ``--group`` | + +---------------------------+-------------------------------------------+ + | ``notUserA``, | ``--not-user``, ``--not-group`` | + | ``notGroupA`` | | + +---------------------------+-------------------------------------------+ + | ``message``, | ``--message``, ``--nearly-message`` | + | ``nearly_message`` | | + | (``forbid``) | | + +---------------------------+-------------------------------------------+ + + A name may be a module name, a module name and version or the path of a + modulefile. All three are accepted by the Tcl commands. An unknown key or a + value of the wrong type is an error. +* ``module_virtual`` maps to ``module-virtual``. Its second argument + designates on Lmod the module that implements the virtual name, by its + name and version, whereas the Tcl command expects the path of a modulefile. + Lua script converts this name into a file path: the name is appended to the + modulepath directory the rc file belongs to, with the ``.lua`` extension if + such file exists. For a global rc file, each enabled modulepath is tried in + turn. This lookup is made by the Lua script on the file system, as querying + the module search on Tcl side during the evaluation of an rc file would + start this search again. Lmod also hides the implementing module, which + is not done on Modules: this module stays visible unless a hiding rule is + defined for it. +* ``hideRegex`` and ``forbidRegex`` map to ``module-hide-regex`` and + ``module-forbid-regex``. They receive the same table than ``hide`` and + ``forbid``, whose keys are converted into the same options, except that + each name is a pattern rather than a module designation. Like on Lmod, a + pattern is searched in the name and version of modules and is not anchored. + + Patterns are Lua patterns in a Lua rc file, whereas rules are matched on + Tcl side with Tcl regular expressions. Evaluating the Lua patterns in the + Lua process is not an option: rules are tested against every module looked + at, long after the evaluation of the rc file, and each test would require + an exchange between the two processes. Lua script converts instead each + pattern into a Tcl regular expression when the function is called, so rc + files written for Lmod are used as is: + + +---------------------------+-------------------------------------------+ + | Lua pattern item | Tcl regular expression | + +===========================+===========================================+ + | ``%`` followed by a | ``\`` followed by this character | + | non-alphanumeric | | + | character | | + +---------------------------+-------------------------------------------+ + | ``%a``, ``%c``, ``%d``, | Corresponding bracket expression | + | ``%g``, ``%l``, ``%p``, | (``[[:alpha:]]``, ``[[:digit:]]``, | + | ``%s``, ``%u``, ``%w``, | ``[[:space:]]``, etc) | + | ``%x`` | | + +---------------------------+-------------------------------------------+ + | Upper case version of | Negated bracket expression | + | these classes | (``[^[:alpha:]]``, etc) | + +---------------------------+-------------------------------------------+ + | ``.``, ``*``, ``+``, | Kept as is | + | ``?``, sets, captures | | + +---------------------------+-------------------------------------------+ + | ``-`` quantifier | ``*`` | + +---------------------------+-------------------------------------------+ + | ``%1`` to ``%9`` | ``\1`` to ``\9`` | + +---------------------------+-------------------------------------------+ + | Characters that are | Escaped with ``\`` | + | literal in Lua and | | + | special in Tcl (``\``, | | + | ``|``, ``{``, ``}``) | | + +---------------------------+-------------------------------------------+ + + The following points are specific to Lua patterns and are respected by the + conversion: + + * ``-`` is the non-greedy version of ``*``. Only the fact that a pattern + matches is used, not the extent of the match, so both quantifiers are + equivalent here. + * A quantifier applies to a single character, class or set. Elsewhere (at + the beginning of the pattern, after a capture) these characters are + literal and are escaped. A pattern like ``-debug$`` thus designates the + names ending with ``-debug``. + * ``^`` is an anchor only at the beginning of the pattern and ``$`` only at + its end. They are escaped elsewhere. + * Within a set, ``%`` escapes and character classes are converted the same + way (``[%d_]`` gives ``[[:digit:]_]``). + + Some items have no counterpart in Tcl regular expressions: ``%b`` + (balanced match), ``%f`` (frontier), position captures and upper case + classes within a set. A pattern using one of them raises an error that + names the unsupported item. As on Tcl side, the resulting regular + expression is matched without case sensitivity when the ``icase`` + configuration option enables it, whereas Lmod match is always case + sensitive. + +Modulerc commands that are specific to Modules (``module-tag``, +``module-warn``, ``modulepath-label``, etc) have no Lua function in the +initial implementation (see `Implementation stages`_). As a Lua rc file +supersedes the Tcl rc file of the same directory, a site that needs these +commands for a directory writes its rc file in Tcl. + +Loading a module that is already loaded +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Lmod and Modules behave differently when a modulefile asks to load a module +that is already loaded. + +On Lmod, ``load`` unloads the loaded module that has the same name than the +module to load, then loads the requested one. Requested module is thus +reloaded if it is already loaded, and replaces another version of this module +if such version is loaded. As they rely on the same code, this also applies +to ``try_load``, ``load_any`` and ``always_load``. For ``load_any`` it applies +to the first module of its list that can be loaded. It does not apply to +``depends_on`` and ``depends_on_any``, which leave an already loaded module +untouched and count one more reference to it. + +On Modules, :subcmd:`load` sub-command does nothing for a module that is +already loaded. Another version of the same module is loaded in addition to +the one already loaded, unless the configuration requires that a single +module of a given name is loaded. + +Lmod functions are translated into the corresponding Modules sub-command or +modulefile command without trying to reproduce the reload: + +* An already loaded module is not reloaded by ``load``, ``try_load``, + ``load_any`` and ``always_load``. +* A loaded module of the same name but of a different version is handled the + way Modules is configured to. + +Reproducing Lmod behavior would make Lua modulefiles behave differently than +Tcl modulefiles on the same installation, for a mechanism that Modules handles +with its own dependency tracking and conflict resolution. + +``mgrload`` function +"""""""""""""""""""" + +On Lmod, ``mgrload`` loads one module described by an object obtained from +``loaded_modules``. It is designed for a modulefile that unloads all loaded +modules, changes something then loads them again. Unlike ``load``, the +modulefile loaded this way gets its own module load requests ignored, and the +module keeps the properties recorded in the object: how many modules depend +on it and whether it was loaded by the user or by another module. + +On Modules, ``mgrload`` is translated like ``load``: the module named in the +object is loaded if it is not already loaded, and nothing is reported if it +cannot be loaded when the first argument of the function is false. Function +returns whether the module is loaded. As ``load`` does not reload modules on +Modules, both functions are nearly equivalent there. Two differences remain +with Lmod: + +* Module load requests of the modulefile loaded this way are evaluated. When + modules are loaded again in their initial order, as done in the above use + case, these requests find their target already loaded. +* A module loaded this way is recorded as loaded by the modulefile that calls + ``mgrload``, even if it was initially loaded by the user. + +``loaded_modules`` function +""""""""""""""""""""""""""" + +On Lmod, ``loaded_modules`` returns an array with one table per loaded +module, in load order. On Modules, list of loaded modules is obtained from Tcl +side through a helper procedure and the tables are built by the Lua script. +Only the simple properties are set to an actual value: + +* ``fullName``, ``name`` and ``displayName``: name and version of the module +* ``sn``: name of the module without its version +* ``userName``: same value than ``fullName``, as the name initially used to + designate a loaded module is not recorded by Modules +* ``fn``: path of the modulefile +* ``loadOrder``: position of the module among the loaded modules + +The other properties that Lmod sets describe information that Modules does not +track or tracks differently. They are defined with an empty value so +modulefile code that reads them does not fail: + +* ``stackDepth`` and ``ref_count``: set to 0 +* ``origUserName``: set to ``false`` +* ``depends_on_anyA``, ``moduleKindT`` and ``forbiddenT``: empty table + +Such table can be passed to ``mgrload``, which only uses its ``userName`` +property. + +Software hierarchy +^^^^^^^^^^^^^^^^^^ + +Lmod sites often organize their modules as a hierarchy: a compiler module +adds to :envvar:`MODULEPATH` the directory of the modules built with this +compiler, an MPI module does the same for the modules built with this MPI +library. Lua modulefiles written for such setup work on Modules, as changing +:envvar:`MODULEPATH` with ``prepend_path`` enables the corresponding +modulepath. But the two tools handle differently the modules loaded from a +modulepath once this modulepath is removed: + +* On Lmod, unloading the compiler module makes the modules loaded from its + directory inactive, and switching to another compiler reloads them from the + directory of the new compiler. +* On Modules, a module only depends on the modules it declares as + requirement. Modules loaded from the directory of the compiler stay loaded, + as they were, when this compiler is unloaded or switched. If they declare + the compiler as a requirement (``depends_on`` or ``prereq``), the automated + module handling mode unloads them with the compiler and reloads them from + the new directory on a switch. + +This difference does not come from the language of modulefiles and is not +changed by Lua modulefile support. + +Hierarchical modulefiles also read environment variables that Lmod defines: + +* ``family`` Tcl command sets the ``LMOD_FAMILY_`` variable and the + ``LMOD_FAMILY__VERSION`` variable, the latter holding the version of + the module, which is commonly used to build the path of the directory to + enable. These variables are set by the Tcl command, so they are defined + whatever the language of the modulefile. For a module without version, + ``LMOD_FAMILY__VERSION`` is set to an empty string whereas Lmod sets + it to the ``false`` string. +* Other variables (``MODULEPATH_ROOT``, ``LMOD_SYSTEM_NAME``, etc) are set by + the initialization of Lmod or by the site. They are not defined by Modules: + a site whose modulefiles rely on them defines them in its initialization + setup. + +Site customization +------------------ + +Lmod sites adapt its behavior with a :file:`SitePackage.lua` file. This file +has two distinct roles: it registers hook functions called by Lmod at given +points of its execution, and it defines site-specific functions made +available to modulefiles. Sites also describe the properties modulefiles may +set in a properties file. + +Hooks +^^^^^ + +Lmod hook functions are Lua code running within Lmod itself. They rely on the +internal objects of Lmod to query the loaded modules or the evaluation stack, +and some hooks receive such objects as argument. These hooks are not +modulefile code: evaluating them would require to emulate the internals of +Lmod. + +Hook functions registered in a :file:`SitePackage.lua` file are not run by +Modules. Sites migrate their hook code as Tcl procedures defined in +:file:`siteconfig.tcl` and registered with the ``add-hook`` command (see +:ref:`hook-api`). This way hooks have a single implementation, written in +Tcl, that applies to both Tcl and Lua modulefiles. No Lua interpreter has to +be started for hooks when no Lua file is evaluated. + +Following table describes how the hooks of Lmod map to hook events of +Modules. + ++--------------------------+------------------------------------------------+ +| Lmod hook | Modules counterpart | ++==========================+================================================+ +| ``load`` | ``after-modulefile-eval`` event, with ``load`` | +| | evaluation mode | ++--------------------------+------------------------------------------------+ +| ``unload`` | ``after-modulefile-eval`` event, with | +| | ``unload`` evaluation mode | ++--------------------------+------------------------------------------------+ +| ``load_spider`` | ``after-modulefile-eval`` event, with ``scan`` | +| | evaluation mode | ++--------------------------+------------------------------------------------+ +| ``isVisibleHook``, | No hook event. Rules are expressed with | +| ``isForbiddenHook`` | ``module-hide`` and ``module-forbid`` commands | ++--------------------------+------------------------------------------------+ +| ``avail`` | No hook event. Labels are set with | +| | ``modulepath-label`` command | ++--------------------------+------------------------------------------------+ +| ``startup``, | No hook event currently | +| ``finalize``, | | +| ``restore``, | | +| ``decorate_module``, | | +| ``msgHook``, | | +| ``errWarnMsgHook``, | | +| ``listHook``, | | +| ``colorize_fullName``, | | +| ``SiteName`` | | ++--------------------------+------------------------------------------------+ +| ``parse_updateFn``, | Not applicable. These hooks relate to features | +| ``writeCache``, | specific to Lmod: spider cache, reverse map | +| ``groupName``, | and category | +| ``packagebasename``, | | +| ``reverseMapPathFilter``,| | +| ``spider_decoration``, | | +| ``category`` | | ++--------------------------+------------------------------------------------+ + +Hook events of Modules are added when a need is expressed. Sites whose hook +has no counterpart are invited to request the corresponding hook event to the +Modules project. Several of the missing hooks return a value used by Lmod +(text to output, content to add to a modulefile). Hook procedures of Modules +currently have their return value ignored, so the hook API has to be extended +for such events. + +Site-specific functions +^^^^^^^^^^^^^^^^^^^^^^^ + +:file:`SitePackage.lua` also defines functions and makes them callable from +modulefiles with ``sandbox_registration``. A modulefile calling such function +fails to evaluate on Modules as the function is unknown. + +Support for these site-specific functions is not part of this initial work. +It may be added later: a site Lua file loaded once when the Lua interpreter +starts, with the functions it registers through ``sandbox_registration`` +added to the base environment table of the sandbox. + +Properties file +^^^^^^^^^^^^^^^ + +On Lmod, the properties a modulefile may set with ``add_property`` are +described in a properties file: :file:`lmodrc.lua` in the installation +directory of Lmod, completed by the files of the same name found in system +locations, by :file:`$HOME/.lmodrc.lua` and by the file designated by +``LMOD_RC``. For each property name, this file defines the valid values and +how each value is displayed (short string, color and description). + +These files are not read by Modules. ``add_property`` sets its value +argument as a tag on the module, and tags are configured with the options of +Modules. Sites port the content of their properties file as follows: + ++----------------------------+--------------------------------------------+ +| Properties file | Modules counterpart | ++============================+============================================+ +| Valid names and values | None. Any value is accepted and set as | +| (``validT``) | tag, property name is ignored | ++----------------------------+--------------------------------------------+ +| Short string of a value | Abbreviation of the tag | +| (``short``) | (:mconfig:`tag_abbrev`) | ++----------------------------+--------------------------------------------+ +| Color of a value | Color of the tag (:mconfig:`colors`) | +| (``color``) | | ++----------------------------+--------------------------------------------+ +| Color applied on module | Tag whose name is colored rather than the | +| name or on short string | module name (:mconfig:`tag_color_name`) | +| (``full_color``) | | ++----------------------------+--------------------------------------------+ +| Description of a value | None. Tag name is the description reported | +| (``doc``) | in the output key | ++----------------------------+--------------------------------------------+ + +Some behaviors differ from Lmod as a consequence: + +* A property has a single value on Lmod: setting it again replaces the + previous value. Tags add up on Modules: a modulefile that sets the same + property twice gets both values as tags. +* A value made of several elements separated by a colon character + (``gpu:mic``) sets one tag per element. Lmod may define a specific display + for such combination, whereas each tag is reported on its own. +* An unknown property name or value is not an error. +* ``sticky`` value of the ``lmod`` property corresponds to the ``sticky`` tag + of Modules, which has the same effect. + +The properties file also designates the spider cache directories of Lmod. +This has no counterpart: module cache files of Modules are stored in the +modulepath directories. + +Implementation stages +--------------------- + +Lua modulefile support is introduced in stages: + +1. The initial implementation provides the Lua functions that Lmod supports, + for modulefiles and for modulercs, and nothing more. It is released in + Modules 5.8. +2. Feedback from sites using it on their existing Lua modulefiles is expected + to reveal cases that were not identified during the initial + implementation. The feature is stabilized against them. +3. Once stabilized, new Lua functions are added to cover the Tcl modulefile + and modulerc commands that have no Lmod equivalent (``variant``, + ``module-tag``, ``module-warn``, ``modulepath-label``, etc). + +Until the last stage, a feature of Modules that has no Lmod function is not +reachable from a Lua file: a modulefile or an rc file that needs it is +written in Tcl. Functions added at the last stage are not known by Lmod, so +a file that uses them cannot be shared with it anymore. + +Installation +------------ + +Options +^^^^^^^ + +The following options are added. Their name follows the options that exist +for the Tcl linter: + ++------------------------+-------------------------------+----------------+ +| Purpose | Installation option | Configuration | +| | | option | ++========================+===============================+================+ +| Lua interpreter | ``--with-lua`` | ``lua_interp`` | +| program | | | ++------------------------+-------------------------------+----------------+ +| Lua linter program | ``--with-lua-linter`` | ``lua_linter`` | ++------------------------+-------------------------------+----------------+ +| Lua linter options | ``--with-lua-linter-opts`` | ``lua_linter`` | ++------------------------+-------------------------------+----------------+ +| Installation of the | ``--enable-luacheck-addons`` | | +| luacheck configuration | | | +| files | | | ++------------------------+-------------------------------+----------------+ +| Location of the | ``--luacheckdatadir`` | | +| luacheck configuration | | | +| files | | | ++------------------------+-------------------------------+----------------+ + +Each configuration option is also set through its environment variable +(``MODULES_LUA_INTERP`` and ``MODULES_LUA_LINTER``). + +``configure`` script looks for a ``lua`` program by default. If none is found +or if its version is not supported, a warning is printed and installation +continues: Lua interpreter can be installed or designated afterward. The +``posix`` library is checked the same way. + +Packaging +^^^^^^^^^ + +Lua interpreter and the ``posix`` and ``lfs`` libraries are not required to +install or run Modules. In the RPM package they are declared as weak +dependencies, to get them installed by default without forcing them on sites +that only use Tcl modulefiles. luacheck is not a dependency, like Nagelfar. + +Supported platforms +^^^^^^^^^^^^^^^^^^^ + +Lua modulefile support relies on the ``posix`` library, thus on a Unix-like +system. It is not available on a native Windows installation +(``--enable-windows-support``), where the Lua interpreter is considered as a +program that cannot be run: Lua files are located but an error is raised when +one of them has to be evaluated. + +Documentation +^^^^^^^^^^^^^ + +Lua modulefile functions are documented in a dedicated manual page named +:manpage:`modulefile-lua(5)`, beside the :manpage:`modulefile(5)` page that +describes the Tcl commands. Its name starts with ``modulefile`` to be listed +next to this existing page. It contains: + +* The supported modulefile and modulerc functions, with the Tcl command each + of them corresponds to +* The differences with the Lmod implementation, in two separate lists: what + is not supported and what is done differently +* How to port the site customization of Lmod: hook functions and properties + file +* How to write a Lua modulefile that copes with both implementations: how to + identify the tool and its version, and how to test the level of Lmod + functions supported (see `Identifying the module tool`_) + +The `Differences with Lmod`_ section of this document is the source of these +two lists. + +Linting Lua modulefiles and modulercs +------------------------------------- + +:subcmd:`lint` sub-command checks Lua modulefiles and modulercs like it checks +the Tcl ones (see :ref:`lint-sub-command`). Selection of the files to lint is +unchanged: Lua files are found among the modulefiles and modulercs that the +sub-command already collects, whether it is called with or without arguments. + +Kind of file +^^^^^^^^^^^^ + +Language and kind of each selected file determine how it is linted: + +* A file is a Lua file if its name ends with ``.lua`` or if it is a + ``default`` symbolic link that targets such file. It is a Tcl file + otherwise. +* A Lua file is a global rc file if its path corresponds to a global or user + rc file location. +* It is a modulerc if its name is :file:`.modulerc.lua`. +* It is a modulefile otherwise. + +Module cache files are Tcl files whatever the language of the files they +record. They are still linted with the Tcl linter, whose syntax database for +cache files is extended with the ``lua-modulefile-content`` and +``lua-modulerc-content`` commands. Lua code recorded in a cache file is not +linted through the cache file but through the Lua file it comes from. + +Configuration +^^^^^^^^^^^^^ + +Lua files are checked by an external linter program that performs a static +analysis of their code, as done for Tcl files. Lua files are not evaluated nor +transmitted to the Lua interpreter that evaluates modulefiles. + +A configuration option defines the Lua linter program to use and its options, +like :mconfig:`tcl_linter` does for Tcl files: + +* `luacheck`_ is the default +* Installation options help to change the linter program and its options at + installation time +* ``configure`` script checks that the program exists. A warning is printed + if it is not found, not an error, as the linter can be installed later-on + +Tcl linter and Lua linter are independent. :subcmd:`lint` sub-command +currently raises an error if no Tcl linter is configured. This error is now +only raised when a Tcl file has to be linted. The same way, an error is raised +when a Lua file has to be linted and no Lua linter is configured. + +luacheck linting +^^^^^^^^^^^^^^^^ + +luacheck needs to know the functions that are available to modulefiles and +modulercs. It would otherwise report every Lmod function as an undefined +variable. This is the counterpart of the Nagelfar syntax databases installed +to lint Tcl modulefiles. + +luacheck configuration files are installed for this purpose, one per kind of +Lua file. Each of them defines a set of standard globals describing the +functions and variables the corresponding evaluation environment provides, +and selects it in addition to the standard globals of Lua: + +* Modulefile: Lmod modulefile functions and helper functions +* Modulerc and global rc: Lmod modulerc functions and helper functions + +An installation option controls whether these files are installed, and another +one where they are installed, like for the Nagelfar addons. + +When the configured linter is luacheck, the following is added to its command +line: + +* The configuration file that corresponds to the kind of the linted file. + Designating a configuration file also prevents luacheck from looking for + such file in the current directory, which makes the result independent of + where the :command:`module` command is run. +* Options to get a plain output with warning codes and without colors, to + parse it reliably. + +These configuration files accept the globals of every Lua version, as +modulefiles are written for the Lua version of the site. They do not check +line length, which is a matter of style. + +The list of functions described in these configuration files has to match the +functions that :file:`mfinterp.lua` actually provides. Fixtures of the +testsuite call every Lua modulefile and modulerc function: they are evaluated +without error and linted without any undefined variable reported. + +luacheck only accepts a first line starting with a ``#`` character if it is a +shebang line (``#!``), whereas Lua interpreter ignores any first line starting +with ``#``. For a file starting with such line that is not a shebang, content +is passed to luacheck on its standard input with this first line blanked, and +file name is given with the ``--filename`` option. luacheck would otherwise +report a syntax error on a file that the Lua interpreter accepts. + +Exit code of luacheck tells if warnings (1) or errors (2) have been found. +These exit codes are not an execution failure of the linter, unlike the higher +exit codes it returns on a fatal error (unreadable file, wrong option). + +Report output +^^^^^^^^^^^^^ + +Messages are reported like for Tcl files, under a ``Linting `` +message block and with the `` line : message`` +structure. + +* luacheck messages are parsed to extract line number, code and message. + Messages whose code starts with ``E`` (syntax errors) are reported as + ``ERROR`` and those whose code starts with ``W`` as ``WARNING``. +* Output of another Lua linter is reported as-is. + +Verbosity levels and exit code of the sub-command follow the same rules than +for Tcl files. + +.. _luacheck: https://github.com/lunarmodules/luacheck + +Testsuite +--------- + +Lua modulefile support is tested at two levels: Lua code is linted and the +whole feature is tested through :file:`modulecmd.tcl`. Coverage of the Lua +code is measured over these integration tests. + +Like for Tcl code, there is no test of individual Lua functions. Everything is +tested through :file:`modulecmd.tcl`, which is how the Lua code is used. No +constraint is this way put on :file:`mfinterp.lua` to make its internal +functions reachable by a test tool. + +Tests that need a Lua interpreter or a Lua test tool are skipped when this +program or tool is not available. The testsuite can this way still be run on +systems where Lua is not installed. + +Integration tests +^^^^^^^^^^^^^^^^^ + +Evaluation of Lua modulefiles and modulercs by :file:`modulecmd.tcl` is tested +in the ``modules`` testsuite, like any other feature. These tests are the +only ones that exercise both ends of the bridge together. + +A :file:`testsuite/modules.55-lua` directory is added to this testsuite. It +holds the tests dedicated to the Lua modulefile and modulerc functions, with +test files organized per kind of function: + +* Environment functions (``setenv``, ``prepend_path``, ``pushenv``, etc) +* Shell functions (``set_alias``, ``set_shell_function``, ``complete``, + ``source_sh``, ``execute``) +* Module dependency functions (``load``, ``depends_on``, ``prereq``, + ``conflict``, ``family``, ``inherit``, etc) +* Module information functions (``whatis``, ``help``, ``add_property``, + ``extensions``) +* Query and introspection functions (``isloaded``, ``mode``, + ``myModuleName``, ``hierarchyA``, etc) +* Message and evaluation control functions (``LmodMessage``, ``LmodError``, + ``LmodBreak``, ``os.exit``) +* File and miscellaneous helper functions (``pathJoin``, ``isFile``, + ``capture``, etc) +* Modulerc functions (``module_version``, ``module_alias``, ``hide``, + ``forbid``, etc), including the conversion of each kind of Lua pattern + item by ``hideRegex`` and ``forbidRegex`` +* Bridge behaviors that are not tied to a specific function + +Each function is tested in the evaluation modes where its behavior differs +(load, unload, display, help, whatis, scan, refresh) and with the ``modeA`` +option when it supports it. + +Functions handled by the Lua script alone have to return the same result as +on Lmod. Lmod is not required to run the testsuite: results expected by the +tests are fixed values. A maintainer script evaluates the same fixtures with +an Lmod installation and compares the results with these values. It is run +when such function is added or changed and when the reference Lmod version +is raised. + +Existing sections of the ``modules`` testsuite are also extended, to cover the +existing features when Lua modulefiles or modulercs are found in modulepaths: + +* Module location and name resolution (:file:`modules.20-locate`): precedence + between Lua and Tcl files, default version, symbolic versions and aliases + defined by Lua or Tcl modulercs on Lua modulefiles, modulepath and global + rc files written in Lua, hide and forbid mechanisms applied to Lua + modulefiles, Lua files with limited access +* Module cache (:file:`modules.30-cache`): build and use of cache files + containing Lua files, same result obtained with and without cache, cache + file built by an older version +* Modulefile commands and module handling (:file:`modules.50-cmds`): + dependency handling, tags, variants and other mechanisms involving several + modules when some of them are Lua modulefiles +* Scan evaluation and extra match search (:file:`modules.51-scan`) +* Collections (:file:`modules.61-coll`): save and restore of collections + containing Lua modulefiles +* Sub-commands and configuration (:file:`modules.70-maint`): sub-commands + that locate or evaluate modulefiles applied to Lua modulefiles + (``display``, ``help``, ``test``, ``whatis``, ``apropos``, ``source``, + ``path``, ``paths``, ``is-avail``, ``aliases``, etc), ``lint`` sub-command + applied to Lua modulefiles and modulercs, and + configuration options that designate the Lua interpreter and the Lua linter +* Deep modulefile directories (:file:`modules.80-deep`) +* Listing sub-commands (:file:`modules.90-avail`, :file:`modules.91-sort` and + :file:`modules.92-spider`): Lua modulefiles reported under their module + name by ``avail`` and ``spider`` in every output format + +Lua fixtures are stored in dedicated modulepath directories of the testsuite. +They are not added to the existing fixture directories, as many existing tests +enumerate the whole content of these directories. + +The following situations require specific attention in these tests: + +* Precedence between Lua and Tcl files, for modulefiles and for each kind of + modulerc (in modulepath directory, at modulepath root and global ones) +* Values that challenge message quoting in both directions: empty string, + braces, backslash, newline, double quote, dollar and bracket characters, + multi-byte characters +* Output produced by a modulefile or by a command it executes (``print``, + ``io.write``, ``os.execute``), which must not disturb the bridge. Output of + the modulefile code is reported in the message block of its module and + follows the verbosity level. +* Lua library loaded with ``require`` that calls Lmod functions +* ``LUA_PATH`` and ``LUA_CPATH`` set to locations holding libraries that + cannot be loaded, and variables put in quarantine +* Commands executed by a modulefile that rely on environment changes made + earlier in this modulefile +* Nested evaluations mixing languages: Lua modulefile loading a Lua + modulefile, loading a Tcl modulefile, and the other way round +* Errors: Lua syntax error, Lua runtime error, error raised by a Tcl command, + whether caught or not with ``pcall``, error located in a modulefile with a + long path +* Evaluation stop (``LmodBreak``, ``os.exit``) including when called through + ``pcall`` +* State isolation: a variable or a function defined by a modulefile is not + visible from the next one evaluated +* Lua interpreter that cannot be run, that ends during an evaluation, that + sends an unexpected message or that is started with ``LUA_INIT`` set. Fake + Lua interpreter programs are used to produce the situations a regular Lua + interpreter does not lead to. +* File starting with a ``#`` line + +A basic evaluation of a Lua modulefile is also added to the ``install`` +testsuite to check that the installed :file:`mfinterp.lua` is found. + +Coverage +^^^^^^^^ + +Coverage of the Lua code is measured with `luacov`_ when the testsuite is run +in coverage mode (``COVERAGE=y``), over the integration tests. + +luacov is enabled by loading it when the Lua interpreter starts (``-l`` option +of the Lua interpreter). Nothing is added in :file:`mfinterp.lua` for this: +the configuration option that designates the Lua interpreter is set to a +wrapper script of the testsuite. This wrapper runs the Lua interpreter with +luacov loaded. ``LUA_INIT`` variable cannot be used for this purpose as it is +withdrawn when the Lua interpreter is started. + +luacov saves the statistics it has collected when the Lua interpreter exits. +Lua script thus ends normally when its *stdin* is closed. Statistics are lost +if the Lua process is ended by a signal. + +A luacov configuration file restricts the collection to the Lua code of +Modules and sets where statistics are saved. As the Lua interpreter is started +by many :file:`modulecmd.tcl` runs, possibly at the same time when the +testsuite is run in parallel, each Lua process saves its statistics in a file +of its own, named after its process identifier. These files are merged at the +end of the testsuite run, like done for the Tcl coverage logs. luacov does not +provide a command for this, so a small script based on its statistics module +sums the hit counts of all files. + +luacov measures line coverage: a line is covered as soon as one statement of +this line is executed. Lua code is thus written with one statement per line +for the coverage result to be meaningful. + +Once statistics are merged, luacov report is generated. It shows the Lua code +with the hit count of each line, which is the counterpart of the +:file:`tcl/*.tcl_m` files to spot the lines that are not covered. Statistics +are also converted into a format accepted by the coverage service used by the +continuous integration, as done for Tcl coverage results. Reporters bundled +with luacov do not produce such format, so a conversion script or an +additional reporter is needed. + +.. _luacov: https://github.com/lunarmodules/luacov + +Lint +^^^^ + +The ``lint`` testsuite is extended to check the files of the :file:`lua` +directory with `luacheck`_, like shell scripts are checked with ShellCheck. +No warning is expected. + +A luacheck configuration file is added at the root of the repository: + +* Standard globals are limited to those of the oldest supported Lua version + (``lua53`` standard). Use of a function that only exists in a newer version + is this way reported. +* Maximal line length is set to 78 characters, like for Tcl code. +* Global variables or functions set on purpose are declared. + +luacheck has its own parser, which accepts the syntax of every Lua version. A +syntax introduced after Lua 5.3 (variable attributes for instance) is thus not +reported. Running the integration tests with the oldest supported Lua version +covers this. + +The :file:`lua` directory is added to the directories the ``lint`` testsuite +scans to verify that every script file is either linted or explicitly +skipped. + +Spell checks run by the git hooks of the repository apply to the Lua code. +Names of some Lmod functions it defines, like the version modifier functions, +are taken for misspellings by codespell: they are added to the words this tool +ignores. + +Continuous integration +^^^^^^^^^^^^^^^^^^^^^^ + +Lua interpreter, ``posix`` and ``lfs`` libraries, luacov and luacheck are +installed in the continuous integration jobs. Integration tests are run with +the Lua interpreter of the system, which is Lua 5.4 on recent systems. They +are also run with the oldest supported versions of Lua and of its libraries, +in a job based on an Enterprise Linux 8 system. + +Performance benchmark +^^^^^^^^^^^^^^^^^^^^^ + +:file:`script/mb` benchmark tool is extended to measure the evaluation of Lua +modulefiles and compare it against the evaluation of Tcl modulefiles. New +tests are added, each of them mimicking an existing test with the same +modulefiles written in Lua: + +* ``load2_lua``: like ``load2``, load a module with more than a hundred + dependencies +* ``list2_lua``: like ``list2``, list these modules once loaded +* ``purge_lua``: like ``purge``, unload all these modules +* ``unload_lua``: like ``unload``, unload a module and its dependencies + +Lua modulefiles used by these tests are the counterpart of the Tcl +modulefiles of the existing tests. They are stored in dedicated modulepath +directories: if stored along the Tcl modulefiles they would take precedence +over them and change what the existing tests measure. Loaded environment of +``list2_lua``, ``purge_lua`` and ``unload_lua`` tests is recorded in +:file:`testsuite/mb` directory like for the other tests. + +:file:`script/mb` builds the :file:`modulecmd.tcl` of each version it +compares. It also builds :file:`mfinterp.lua` for the versions that provide +it, as this script is located next to the :file:`modulecmd.tcl` that is run. + +These tests are skipped for the versions of Modules that do not support Lua +modulefiles. ``load2_lua``, ``list2_lua`` and ``purge_lua`` are part of the +tests that can be compared with Lmod: it evaluates there Lua modulefiles +natively, whereas it has to translate the Tcl modulefiles of the ``load2``, +``list2`` and ``purge`` tests. + +Comparing the result of each Lua test with the test it mimics gives the +overhead of Lua modulefiles. This comparison is checked when the bridge is +changed, to catch a cost that grows with the number of loaded modules. + +Modulefiles of these tests mostly declare requirements and set few +environment variables. A cost that grows with the size of the environment is +not visible on them. It is caught by the integration tests instead: one of +them checks, from the messages reported at the ``debug2`` level, that a reply +never carries the value of an environment variable. + Differences with Lmod --------------------- @@ -167,5 +2235,84 @@ different behavior than the latter. * ``os.exit`` is evaluated as the Tcl ``exit`` command that does not return an exit code value equal to its argument. +* When unloading, the value set by ``setenv`` stays visible through + ``os.getenv`` in the rest of the modulefile, as done for Tcl modulefiles. + Lmod unsets the variable right away. +* Displaying a Lua modulefile reports the Tcl modulefile commands it has been + translated into, not its Lua function calls. +* Exclusive lower bound of version modifier functions (``<`` character after + version) is not supported. Exclusive upper bound (``<`` character before + version) is supported when the last element of the version is a number. +* Inclusive upper bound of version modifier functions includes the versions + that extend it: ``between("foo", "2.7", "3.0")`` matches ``foo/3.0.1`` + whereas Lmod stops at ``foo/3.0``. +* Automatic load of requirements depends on the automated module handling + mode, not on the use of ``depends_on`` rather than ``prereq``. +* ``mode`` function returns ``spider`` in ``scan`` evaluation mode. This mode + is close to the ``spider`` mode of Lmod but not identical: Lmod uses it to + build its spider cache, whereas Modules uses it when a search needs + information defined in modulefiles. ``mode`` function returns ``test`` in + ``test`` evaluation mode, that does not exist on Lmod. +* ``load``, ``try_load``, ``load_any`` and ``always_load`` do not reload a + module that is already loaded, and do not replace a loaded module of the + same name unless Modules is configured to (see `Loading a module that is + already loaded`_). +* ``mgrload`` evaluates the module load requests of the modulefile it loads + and does not preserve how the module was initially loaded. +* Tables returned by ``loaded_modules`` only hold the name, version, + modulefile and load order of loaded modules. Other properties are empty. +* ``prepend_path`` and ``append_path`` do not move a path entry that is + already part of the variable, they count one more reference to it. Lmod + moves such entry at the beginning or at the end of the variable, unless + configured to follow this rule. +* Loading a module that is member of the same ``family`` than a loaded module + is an error, unless Modules is configured to unload conflicting modules. + Lmod swaps the two modules by default. +* ``unload`` does nothing when the modulefile is unloaded. +* Modules loaded from a modulepath enabled by another module stay loaded when + this other module is unloaded or switched, unless they declare it as a + requirement (see `Software hierarchy`_). +* Variables set by the initialization of Lmod (``MODULEPATH_ROOT``, + ``LMOD_SYSTEM_NAME``, etc) are not defined. +* ``LmodVersion`` function and ``LMOD_VERSION`` variables give the version of + Lmod whose modulefile functions are supported, not the version of a + running Lmod. ``ModuleTool`` variable is set to ``Modules``. +* ``pushenv`` called with a ``false`` value sets the variable to an empty + string. Lmod unsets it. +* On ``whatis`` and ``help`` modes, path functions update the environment + variables for the rest of the modulefile, like ``setenv`` does. Lmod only + applies ``setenv`` and ``pushenv`` on these modes. +* ``print`` and ``io.stdout`` output is reported on *stderr* with the other + messages of the module. It is not evaluated by the shell. ``execute`` + function is the way to run a command in the shell. +* Global variables and functions defined by a modulefile are not visible from + the modulefiles evaluated afterward. On Lmod all modulefiles share the same + environment, so a global variable set by one modulefile can be read by the + following ones. Code shared between modulefiles has to be placed in a Lua + library loaded with ``require``. +* ``help`` and ``whatis`` text is output as is, without the markdown + rendering made by Lmod. +* Shell functions defined with ``set_shell_function`` are always exported to + the sub-shells. On Lmod they are exported only if ``export_shell_function`` + is called. +* ``hideRegex`` and ``forbidRegex`` do not support the ``%b`` and ``%f`` + pattern items, position captures and upper case classes within a set. +* ``module_virtual`` does not hide the module that implements the virtual + name. ``hide_version`` has to be called on this module to get it hidden as + on Lmod. +* ``myConfig`` only knows the ``username`` and ``usergroups`` keys. +* When no :file:`.modulerc.lua` file exists in a directory, :file:`.version` + file is evaluated in addition to :file:`.modulerc` file. Lmod only + evaluates the latter. +* Rc files of a directory are evaluated even if this directory contains a + ``default`` symbolic link. Lmod ignores them in this case. +* Hook functions registered in :file:`SitePackage.lua` are not run. Hooks + are defined in :file:`siteconfig.tcl` (see `Hooks`_). +* Site-specific functions registered with ``sandbox_registration`` in + :file:`SitePackage.lua` are not available to modulefiles. +* Properties file (:file:`lmodrc.lua`) is not read. Values set with + ``add_property`` are tags, configured with the tag options of Modules (see + `Properties file`_). They are not validated, and several values set for + the same property add up instead of replacing each other. .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: