A diffusion language model assembled from implicit learners. A language model is usually one network that runs one way. LenticulumLM builds one from parts: each part is a relation over a few token positions, the model is their product, and every query is answered by root finding with the known tokens clamped. Continuation, backward prediction and infilling are the same model asked with different clamps.
It builds on Lenticulum.jl, which provides the implicit learners (diffusion models read as relations), their products, and their beliefs.
Status: research prototype. Julia, not registered, one author. What is here is exact against closed forms; learned windows, a discrete learner and scale are open (roadmap).
Text of
are an
| query | clamped | answered |
|---|---|---|
| continuation | the first tokens | the distribution of each later position |
| backward prediction | the last tokens | the distribution of each earlier position |
| infilling | both ends | the middle |
Every answer of a query is a stable root of the model's field, with a Laplace weight, and the roots read per position are categorical beliefs.
using LenticulumLM
v = Vocabulary(['x', 'y'])
T = [0.9 0.1; 0.4 0.6] # a bigram chain, stationary π = (0.8, 0.2)
π₀ = [0.8, 0.2]
model, ps, st = sequence_relation(v, window_relation(v, π₀ .* T), window_relation(v, π₀), 3)
pos, P, _ = token_marginals(v, model, ps, st, ['x', nothing, 'y']) # fill in the middle
P # (0.6, 0.4) = T(x, ·) .* T(·, y), normalisedOn a three-symbol chain, continuation, backward prediction and infilling all agree with the
closed form to src/sequence.md §3).
The repository mirrors Lenticulum.jl's:
| where | what |
|---|---|
src/ |
the package; an implementation note (.md) beside each .jl file |
test/ |
the test suite, against closed forms |
vault/ |
the Obsidian vault: design, roadmap, bibliography (Start Here) |
docs/ |
Documenter pages, Literate tutorials, and the Quartz build of the vault (docs/site/) |
flake.nix |
the development shells: every tool the experiments need |
experiments/ |
one directory per experiment, each with its own environment (planned) |
LenticulumLM is a workspace for experiments rather than a library to install. The Nix flake sets up everything: Julia (yours if it is on the PATH, else its own 1.12), the docs toolchain, the Hugging Face CLI, and on request Lean, Metamath or Isabelle (plan):
nix develop # Julia, Node, Python with the hf CLI, git-lfs
nix develop .#gpu # the same, with the host's NVIDIA driver visible to CUDA.jl
nix develop .#lean # + elan; also .#metamath, .#isabelle, .#proversData and checkpoints live outside the repository, in $LENTICULUMLM_DATA and
$LENTICULUMLM_MODELS (default ~/.cache/lenticulum/lm/). With direnv, .envrc enters the
default shell on cd.
Neither LenticulumLM nor Lenticulum.jl is registered. Project.toml takes Lenticulum's
packages from GitHub, so a clone instantiates on its own:
using Pkg
Pkg.activate("path/to/LenticulumLM")
Pkg.instantiate()
Pkg.test()To change both repositories at once, develop the local Lenticulum packages into the environment. This overrides the GitHub sources in the (untracked) manifest:
Pkg.develop([PackageSpec(path = "../Lenticulum.jl"),
PackageSpec(path = "../Lenticulum.jl/lib/LenticulumCore.jl"),
PackageSpec(path = "../Lenticulum.jl/lib/Mycelium.jl"),
PackageSpec(path = "../Lenticulum.jl/lib/VariationalDiffusion.jl")])- The vault, on the web at mathstruct.org/LenticulumLM (the API
documentation is under api/dev), or in Obsidian:
open the repository root and start at
vault/Start Here.md. - Lenticulum's vault, for the implicit learners themselves: mathstruct.org/Lenticulum.jl.
- The CT-ML wiki, for the general category theory both build on: mathstruct.org/CategoryTheory-ML-Wiki.
MIT, see LICENSE.