Skip to content

Add portable runtime APIs, platform tooling and structured markup - #38

Merged
bytetroll merged 39 commits into
masterfrom
nil-sockets-entropy-bytes
Nov 13, 2026
Merged

bytetroll merged 39 commits into
masterfrom
nil-sockets-entropy-bytes

Conversation

@bytetroll

@bytetroll bytetroll commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Matrix web applications need binary-safe socket I/O, OS entropy, byte access and stable file modification times. This adds those APIs and verifies the same interpreter behavior on Linux, Windows, Intel macOS and Apple Silicon macOS. Timestamps also run through LLVM JIT/AOT. Socket, byte and entropy APIs remain interpreter-backed on every host until the native text ABI can preserve embedded NUL bytes.

Sockets use opaque VM-owned handles, close on teardown/reset/fault, reject stale handles, preserve Windows SOCKET width and distinguish EOF from timeout/error. Polling is bounded and writes handle the platform's SIGPIPE behavior. Secure randomness uses the host OS provider; timestamps use native filesystem metadata.

Application tests exposed additional platform defects corrected here:

  • Darwin compiler identity hashes loaded images, shared-cache files and admitted linker dependencies. Universal tools and non-linkable SDK resources are modeled explicitly; unsupported dependency profiles remain uncached. Real native reuse and invalidation now pass on both Mac architectures.
  • DAP keeps VM construction, execution, inspection and UI cleanup on process main, with a bounded reader for cooperative pause/termination. Repeated stepping and real desktop sessions exercise the ownership contract.
  • Win32 and AppKit share bounded layout allocation for hidden-child reflow, grid spans, expansion and fill. AppKit also corrects effective sensitivity, focus, resizing and Unicode entry limits. Test-only AppKit input injection stays out of compiler processes.
  • MSVC freestanding zero/copy loops no longer introduce CRT imports. A bounded ELF/COFF/Mach-O archive reader now enforces the import boundary on every host, including malformed-object and mixed-profile rejection.
  • Compiler and linker share bounded Windows output replacement. Real temporary-lock, permanent-lock, read-only and preservation controls catch the former single-attempt linker failure without removing old output or changing permissions.
  • SQL fixture loader paths apply to execution processes, retaining the normal build identity. Editor fixtures respect captured source identity and clean open-tab URIs. Darwin real-project deadlines retain timing and timeout diagnostics; the SDK build script now honors CMAKE_BUILD_PARALLEL_LEVEL.

SDK packaging now keeps compilation private, then copies validated library files into fresh destination entries that inherit the SDK's normal access policy. The former temporary-directory rename preserved private permissions and blocked a second Windows account. Linux umask/rollback controls and actual limited-account Windows reads cover the correction, with private staging still inaccessible.

Validation is recorded by exact source/candidate identity in the notebook:

Surface Recorded result
Linux development SDK 9fa354d All 590 source CTests, 1,184 installed consumer routes, twelve PostgreSQL routes, 146 debugger checks/15 sessions, checkpoints, twenty GTK desktop/input cases and all 88 editor groups pass. SDK files/modes unchanged through relocation.
Windows development SDK 9fa354d Full source: 588 pass, zero fail, two headless display skips subsequently exercised in Nathan's desktop. All 1,184 installed routes, twelve PostgreSQL routes, 146 debugger checks/15 sessions, five checkpoint sessions, twenty Win32 cases, four desktop DAP sessions and all 88 editor groups pass. The old candidate needed a recorded Nathan read/execute grant on nillib; no accepted SDK was changed.
Linux source 59ffa97 Push and PR workflows independently replayed: LLVM ON 590 pass/no skips, LLVM OFF 441 pass plus 143 expected disabled-native skips, extension 162 pass/no skips.
Four-host web runtime The same six interpreter contracts pass on Linux, Windows and both Macs. The latest 59ffa97 four-host workflow also succeeds.
Both Mac current-source application/editor hosts At 59ffa97, all 22 platform/tooling/packaging CTests, four desktop DAP sessions and all 88 editor groups pass per host. VS Code 1.146.0, all nine modules, no exclusions; 620 evidence hashes independently verified.
Relocated Mac SDKs 1efbe66 Both pass all 1,184 installed consumer routes, twelve PostgreSQL routes, 146 debugger checks/15 sessions and checkpoints. Original full source reports retain 583 passes, four fixture failures resolved by later source checks, and one ELF-only execution skip.
Both Mac installed desktop/editor hosts Both pass twenty installed AppKit cases, four desktop DAP sessions and all 88 editor groups/all nine modules/no exclusions. ARM passes run 43146638765; Intel passes diagnostic follow-up 43149252918 with unchanged SDK bytes and deadlines. The earlier Intel 180-second project timeout remains unexplained and preserved; its successful follow-up takes 161.171 seconds in that group.
SDK library publication 3cc2515 Linux: thirteen actual mixed-source/native tests and 46 qualification controls pass, including umasks 022/027/077 and failed-copy rollback. Windows: thirteen tests plus limited-account reads/hashes of all 102 published files pass, while private staging remains inaccessible. Both Macs pass the pinned 13-test native packaging invocation; original JUnit output retains only eight complete result lines, explicitly scoped in replay. Future logs retain a bounded 64 KiB per passing SDK/project test.
Portable editor diagnostics Five fixture-only controls pass on Linux and Windows. Actual Windows reproduces the old helper's two path-separator failures, then passes all five after explicit Darwin path validation. Full Windows extension units: 166 pass, one expected POSIX-only skip, zero failures. No product behavior or deadline changed.

Sources: Mac 22-contract campaign, ARM installed replay and preserved Intel failure, successful Intel installed replay, and the parity notebook.

The supported common development profiles have installed acceptance at the pinned revisions above. Failed/interrupted attempts, the unresolved historical Intel timeout and common socket/byte/entropy native limitations remain explicit. Same-user Mac acceptance does not qualify arbitrary shared deployment.

Ny and Matrix now also support multiline formatted text and structured HTML markup literals (NIL-0027). Literal documents are checked at compile time; scalar holes use context-specific escaping, documents compose without double escaping, and conditions/loops live inside the document. Both frontends share the scanner and document checker. XML, compact rendering, dynamic attribute sets and a runtime DOM remain deferred.

The completion review adds regressions for malformed Ny block openers, frontend-specific hole lexing, browser URL normalization, unsafe attribute contexts, ambiguous URL fragments and malformed directive/document boundaries. The four-host interpreter workflow now runs the markup and formatted-text contracts plus all four positive fixture routes; both Mac native workflows also run the ten language contracts and toolchain fixtures.

The new features require a compiler and standard-library source built from this branch. The currently installed SDK predates them and is not replaced by this PR. Source checks and the historical installed acceptance above are separate qualifications. No stable release or Lich release is activated.

The public web server has since completed HTTP framing, pipelining, conditional/range handling and rooted static-file snapshots in exec-nil-framework-web PR #1, with all four platform gates green at 480ecae. Its rooted-file runtime additions and WebView bridge are separate dependent PRs, not part of this language handoff.

bytetroll and others added 9 commits November 11, 2026 14:51
Three gaps in the runtime, found while writing a web framework in Matrix and
each one a hard stop rather than an inconvenience.

**No networking at all.** Not an unfinished API: the intrinsic table covered
files, paths, directories, databases, the environment, the console, regular
expressions and GPU compute, and nothing that opens a socket. nil.std.net has
described transport-independent HTTP for a while and said in its own comments
that the transport was absent; the socket design under experimental/ had
nothing beneath it. So no NIL or Matrix program could accept a connection.

nil.std.net.socket now has listen, accept, connect, read, write, close,
is_open, set_timeout, set_no_delay and peer_address, over net_support.hpp.
Everything returns a Result, because a network call fails for reasons outside
the program and for a server those are ordinary answers. A program is handed
an opaque token, never a host descriptor, so it cannot touch something it was
not given; a listener and a connection are told apart by name rather than by
EINVAL; and the runtime closes whatever a run leaves open, so a program that
exits or crashes without closing its listener does not leave the port bound
against the next run. Writes use MSG_NOSIGNAL: without it a peer closing
mid-write delivers SIGPIPE and the default disposition ends the process, which
means a server that dies when a browser closes a tab. Binding does not resolve
names -- numeric, empty or localhost -- because a DNS lookup on that path is
both blocking and a way to end up listening somewhere unintended; connecting
does resolve, because a client is handed names.

**No cryptographically secure randomness.** nil.std.math is MINSTD seeded at
12345 and says itself that it is not cryptographic; nil.std.security had
base64 and hex and no random; there were no crypto intrinsics; and
/dev/urandom was not a way round it, since the file intrinsics refuse a
character device by design and the unchecked read returns nothing for one. A
session id, a CSRF token or a password-reset link drawn from MINSTD is one
anyone can compute, so any library needing a credential had to invent
something predictable or drop the feature.

nil.std.security.random now has random_hex, random_bytes and random_below over
entropy_support.hpp -- getrandom(2) with a /dev/urandom fallback on POSIX,
rand_s on Windows. No userspace pool, no seeding, no state, and no fallback to
a weak source: if the kernel cannot answer the result is an error, because
quietly returning something guessable is how this goes wrong unnoticed.
random_below uses rejection sampling rather than a remainder, which would skew
towards the low values. random_hex is the one to reach for, partly because a
credential wants to be text and partly because Matrix cannot index an Ny byte
array.

**No way to read a byte.** nil.std.text measures and slices by byte but had no
byte-to-number conversion, and convert.char_to_i32 -- the one that looked like
it would -- takes a whole Unicode scalar and aborts the process on a partial
one. Not a fault attempt can catch: a fatal runtime error. So a byte inside a
multi-byte character could not be inspected, and such a character could not be
isolated in order to inspect it whole, because isolating it needed the same
test. Percent-encoding anything outside ASCII was unreachable because of this,
and so was reading arbitrary bytes off a socket.

text.byte_at and text.from_byte are deliberately about bytes and say so.
from_byte(195) is the single byte 0xC3 where convert.i32_to_char(195) is the
two bytes encoding U+00C3; joining text joins bytes, so a character can now be
assembled one byte at a time. byte_at answers minus one past the end rather
than raising, because reading past the end is how a byte scan finds its end.

All three are interpreter-backed; the native route does not carry them yet,
and the declarations say so.

Tests: tests/matrix/execution/byte-access and entropy are exact-output
fixtures, the second asserting shape, bounds and that two draws differ -- the
one check that would catch MINSTD standing in for a secure source.
tests/tools/net_socket_contract.py runs a Matrix page as a process and plays
the other end from Python, because what matters about a socket is what the
peer sees. Six cases: bytes cross the wire unchanged including a zero byte and
invalid UTF-8, a handle the program never received is refused, a listener is
not a connection, a leaked listener's port is released for the next run, a
read deadline expires instead of hanging, and a peer that vanishes mid-write
is reported rather than killing the process. Registered as net.socket-contract.
The full Matrix execution suite passes: 37 fixtures.

Docs: docs/standard-library/sockets.md is new, random.md and text-spans.md
grew sections, and index.md no longer claims network transports and secure
random generation are experimental, because they are not any more.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ny.routes.reject-unbacked-io-security asserts that a declaration with no
runtime provider is rejected at compile time rather than accepted and then
failing where it is used. random_bytes was on that list, and the entropy
intrinsic took it off: it now type-checks, so the fixture's expected
diagnostics no longer matched and the test failed in both CI configurations.

Removed from the fixture rather than left failing, because what the test
guards is the gap between a declaration and a provider, and for that call
there is no longer a gap. Compression, hashing and encryption are still
designs under experimental/ with nothing behind them and stay. A comment in
the fixture records where random_bytes went, and
tests/matrix/execution/entropy.matrix is the coverage that replaces it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sockets in the previous commit could serve one connection at a time, which
is a development server and not a web server. This is what it takes to serve
many.

An event loop, not a thread per connection, because a thread is not available
and will not be soon: the interpreter holds one stack, one heap and one set of
globals, and there is not a mutex, an atomic or a thread_local anywhere in it.
nil.std.threading offers sleep. Making the VM thread-safe is a far larger
piece of work than this, and an event loop does not need it.

So: set_blocking, watch, unwatch, watch_count, wait and next_ready, over
poll(2). The watch set lives in the runtime rather than being passed in per
call, which is not tidiness -- a Matrix page cannot index an Ny generic, so an
array-taking interface would be an event loop only half the language could
write. next_ready packs the handle and the flags into one number because an
intrinsic returns one value, and two calls whose meaning depended on being
made in order would be worse than arithmetic the caller can see;
ready_handle, ready_flags, is_readable, is_writable and is_finished are
ordinary Ny functions over the packing.

The readiness flags are deliberately not poll(2)'s numbering, which differs
between platforms: a program that learned the host's values would break when
it moved.

A non-blocking accept with nothing pending now answers zero rather than
reporting an error. Handles start at one so it cannot be mistaken for one, and
for an event loop it is the ordinary case -- poll said the listener was
readable and the connection was withdrawn in between. Reporting it would mean
a log line every time a port scanner connected and left.

set_reuse_port is the other half of using more than one core. A second core
means a second process here, and two processes cannot share a listening socket
without it. Opt-in and distinct from the SO_REUSEADDR that listen always sets:
that one only allows rebinding a port in TIME_WAIT, while this one allows two
servers to share a port, and switched on by accident it would silently split
traffic between a running server and a stray one.

A wait replaces whatever the previous one found, rather than accumulating, so
a program that did not drain the set cannot later be handed a handle that has
since been closed and its number reissued. A socket closed since the last wait
is dropped from the set rather than polled on a descriptor that now means
something else, so a loop that forgets to unwatch a connection it closed is
untidy rather than broken. An interrupted poll is retried, because an
unrelated signal must not look to a server like a broken socket.

Two new contract cases, bringing net.socket-contract to eight. The first
covers the watch set: an empty one does not wait, watching twice replaces
rather than adds, a drained ready set reads zero, unwatching something
unwatched succeeds, a closed socket leaves the set, and the packing round-trips.

The second is the one worth reading. Sixty-four clients connect at once and
each sends its request in two parts with a quarter-second pause between. A
server that serves one connection at a time has to wait out the first client's
pause before it touches the second, so the run costs sixty-four pauses; an
event loop overlaps them and costs about one. The test asserts that difference
and that the server held all sixty-four at once, rather than asserting a
throughput number -- a number would measure the machine, and this measures the
shape. Locally: 200 clients, all answered, 0.31s against the 50s a serial
server would need.

docs/standard-library/sockets.md gains the loop's shape, the three things it
has to get right (drain the listener; leave connections blocking with a
timeout, because EAGAIN and a clean close are both no bytes and only a
blocking read tells them apart; readable and finished can both be true), and
what is still missing. The claim that there was no way to wait on several
sockets is gone, since it was the thing this commit removes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two more things a web server cannot be written without.

net.receive and net.send_some are the non-blocking counterparts of read and
write. write is the dangerous one in an event loop: a large response to a
client that reads slowly fills the socket buffer, write_all blocks, and one
slow reader stops every other connection. send_some takes what the socket will
accept from an offset and reports it, leaving the remainder for when poll says
the socket will take more.

receive is two calls because a non-blocking read has three outcomes and a
Result<text, text> has room for two: bytes arrived, the peer closed, or nothing
is available yet. It reports which, and net.received hands over the bytes.
Conflating the last two is the classic event-loop bug -- a spurious wakeup
looks like a hangup and the connection is dropped in the middle of a request.

file.modified_at closes the other gap. nil.std.io had a size and no timestamp,
which left two ordinary jobs unreachable: a cache could not tell whether what
it held was still current without re-reading the file, which defeats the
cache, and an HTTP server could not send Last-Modified or answer
If-Modified-Since, so every client re-fetched every unchanged file in full,
forever. Milliseconds since the Unix epoch, because a filesystem timestamp is
not reliably finer than that across platforms and a caller comparing two
should not be comparing precision it does not have.

The conversion is the portable one rather than std::chrono::clock_cast, which
says it in a line and is not available on every toolchain this still builds
on: file_clock's epoch is unspecified and need not match the system clock's,
so the two are related by taking a reading from each at the same moment.

tests/matrix/execution/file-time asserts the answer is a plausible epoch time
in milliseconds -- a seconds-based answer fails the lower bound, which is the
mistake worth catching -- that writing again moves it forward, and that an
absent file is reported rather than given a zero a comparison would read as
1970 and treat as very old indeed. 38 execution fixtures now pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bytetroll bytetroll changed the title Sockets, secure random, and byte access to text Add portable web runtime APIs and qualify platform tooling Nov 11, 2026
bytetroll and others added 7 commits November 13, 2026 00:35
The guide is the reference an agent writes Matrix from, and it described
none of NIL-0014's formatted literals or NIL-0016's queries and text
patterns, so generated code still builds text with plus ladders and
hand-rolled loops. Each added example is compiled and run: the formatted
literals join guide-constructs, and the query and pattern sections get a
standard-library fixture of their own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A generator currently assembles a document out of fragments, which puts
the control flow outside the markup, leaves the structure unchecked and
makes escaping manual and context-blind. NIL-0027 stages the fix: a
multiline formatted block first, then a markup literal that holds its own
conditionals and loops and evaluates to a checked tree rather than text.

The record settles the four decisions taken with it: raw-text element
bodies stay literal, the parser carries no dialect, the value is a tree
rendered by nil.std.markup, and the amortized text builder lands first.
It also records that the obvious node-record representation cannot
execute natively, so a markup value is an arena of scalar and text
arrays.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIL-0027 section 2. 'formatted' in front of 'begin text' opts a raw block
into interpolation, so a generated document is written as one unit instead
of a ladder of concatenations. Every raw-block rule still holds: the
terminator is a line whose only content is 'end text', the common leading
indentation is removed and deeper indentation kept, and nothing inside is
escaped. A single hole still stays on one physical line.

Indentation removal applies to literal segments only. The lexer records the
source offset each dedented line starts at and maps every scanned piece back
through it, so a hole's tokens keep their original file coordinates and
NIL-0014's editing contract is unchanged. Because the block emits the token
stream the quoted form already emits, the parser, checker, lowering, VM and
native routes needed no change.

Covered by the formatted-text-matrix parity fixture across the VM, binary
and native routes, malformed blocks rejected on every route, a hole renamed
inside a CRLF block while an identical word in doubled braces is left alone,
and completion inside a block hole. Ny keeps only its single-line form.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIL-0027 section 3. 'begin markup' holds its own conditionals and loops, so
the control flow lives inside the page instead of around it, and the block
is parsed as markup at compile time: an unbalanced element, an unterminated
directive, an arm that opens an element it does not close, and a hole that
would name an element or an attribute are all compile errors.

Escaping belongs to the position, never to the author. A hole learns during
parsing whether it landed in element content, an attribute value, a whole
URL attribute or a component inside one, and lowers to the matching
nil.std.markup function; an executable scheme is neutralised rather than
dropped. '{raw ...}' is the single greppable way to bypass that.

The value is a nil.std.markup document rather than text, which is what lets
a paragraph return a fragment and another block splice it instead of
escaping it twice. NIL-0027 section 3.6 records why the record-graph tree
that section originally specified was replaced by compile-time
serialization: it cannot execute natively, and it bought only the one
capability that was already deferred.

Inside script and style a brace is an ordinary character, so CSS and
JavaScript paste in unaltered; a body that is exactly one hole still
interpolates through '{raw ...}'. 'markup' and 'raw' stay ordinary names.

The literal reuses the block scanner from section 2, so holes keep their
original source coordinates, and emit_markup produces ordinary NIL, so one
emitter covers the VM and native routes. Covered by the markup-matrix
parity fixture across the VM, binary, native and native-toolchain routes,
and by a contract rejecting sixteen malformed documents on every route
while proving hostile input is escaped once in each context.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIL-0027 section 2 for the other surface. '$"""' opens the block Matrix
spells 'formatted begin text', sharing scan_formatted_block, so both
frontends emit the same token stream and the same interpolation AST.

One difference is forced by the surface rather than chosen: Matrix's
'end text' owns its line, while Ny closes on the first line whose content
begins with '"""' and resumes lexing immediately after those quotes,
because Ny is expression oriented and println($"""...""") has to work.

A block is raw in both surfaces. Ny's backslash escapes apply to a quoted
$"..." and not inside $""", which is what makes a path, a pattern or
embedded JSON writable without doubling anything. Indentation removal
applies to literal segments only, so holes keep their original file
coordinates.

An unterminated block now reports its own message rather than only
'Expected expression': the parser surfaces lexer messages for interpolation
and formatted text, and formatted blocks join that list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIL-0027 section 3 for the other surface. The document parsing, the
structural checks and the escape decisions move into markup_structure, which
both frontends drive; each supplies only its own directive words and its own
expression lexing, so the surfaces cannot drift in what a document means.
Ny writes {if}, {elseif}, {else}, {end if}, {for x in xs} and {end for};
Matrix keeps {when} and {repeat for each}. The two parity fixtures render
the same page byte for byte apart from their own indentation.

Ny opens a block with '$markup' on its own line rather than the
'markup { ... }' the design record first guessed. A markup block is full of
braces - holes, and the CSS or JavaScript inside a raw-text element - so a
brace-delimited block would end at the first '}' in a stylesheet. The
terminator has to be a line-anchored keyword, and the '$' is what keeps the
opener unambiguous, since Ny has no 'begin' and a bare word before a newline
could be an ordinary name.

Two things only the Ny side could have found. The sigil skips six characters
after the already-consumed '$', not seven, and getting that wrong swallowed
the first content line into the header. Ny also names an imported type by
the spelling the author wrote, so the document type is now recognised by the
declaration it resolves to rather than by its name - a program's own struct
called 'document' must not be spliced unescaped.

Covered by a markup.ny parity fixture across the VM, binary, native and
native-toolchain routes, and by ten more rejected documents plus an
unterminated block in the markup contract.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bytetroll bytetroll changed the title Add portable web runtime APIs and qualify platform tooling Add portable runtime APIs, platform tooling and structured markup Nov 13, 2026
@bytetroll
bytetroll merged commit 10a5a82 into master Nov 13, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant