Repository navigation
Add portable runtime APIs, platform tooling and structured markup - #38
Merged
Merged
Conversation
Three gaps in the runtime, found while writing a web framework in Matrix and each one a hard stop rather than an inconvenience. **No networking at all.** Not an unfinished API: the intrinsic table covered files, paths, directories, databases, the environment, the console, regular expressions and GPU compute, and nothing that opens a socket. nil.std.net has described transport-independent HTTP for a while and said in its own comments that the transport was absent; the socket design under experimental/ had nothing beneath it. So no NIL or Matrix program could accept a connection. nil.std.net.socket now has listen, accept, connect, read, write, close, is_open, set_timeout, set_no_delay and peer_address, over net_support.hpp. Everything returns a Result, because a network call fails for reasons outside the program and for a server those are ordinary answers. A program is handed an opaque token, never a host descriptor, so it cannot touch something it was not given; a listener and a connection are told apart by name rather than by EINVAL; and the runtime closes whatever a run leaves open, so a program that exits or crashes without closing its listener does not leave the port bound against the next run. Writes use MSG_NOSIGNAL: without it a peer closing mid-write delivers SIGPIPE and the default disposition ends the process, which means a server that dies when a browser closes a tab. Binding does not resolve names -- numeric, empty or localhost -- because a DNS lookup on that path is both blocking and a way to end up listening somewhere unintended; connecting does resolve, because a client is handed names. **No cryptographically secure randomness.** nil.std.math is MINSTD seeded at 12345 and says itself that it is not cryptographic; nil.std.security had base64 and hex and no random; there were no crypto intrinsics; and /dev/urandom was not a way round it, since the file intrinsics refuse a character device by design and the unchecked read returns nothing for one. A session id, a CSRF token or a password-reset link drawn from MINSTD is one anyone can compute, so any library needing a credential had to invent something predictable or drop the feature. nil.std.security.random now has random_hex, random_bytes and random_below over entropy_support.hpp -- getrandom(2) with a /dev/urandom fallback on POSIX, rand_s on Windows. No userspace pool, no seeding, no state, and no fallback to a weak source: if the kernel cannot answer the result is an error, because quietly returning something guessable is how this goes wrong unnoticed. random_below uses rejection sampling rather than a remainder, which would skew towards the low values. random_hex is the one to reach for, partly because a credential wants to be text and partly because Matrix cannot index an Ny byte array. **No way to read a byte.** nil.std.text measures and slices by byte but had no byte-to-number conversion, and convert.char_to_i32 -- the one that looked like it would -- takes a whole Unicode scalar and aborts the process on a partial one. Not a fault attempt can catch: a fatal runtime error. So a byte inside a multi-byte character could not be inspected, and such a character could not be isolated in order to inspect it whole, because isolating it needed the same test. Percent-encoding anything outside ASCII was unreachable because of this, and so was reading arbitrary bytes off a socket. text.byte_at and text.from_byte are deliberately about bytes and say so. from_byte(195) is the single byte 0xC3 where convert.i32_to_char(195) is the two bytes encoding U+00C3; joining text joins bytes, so a character can now be assembled one byte at a time. byte_at answers minus one past the end rather than raising, because reading past the end is how a byte scan finds its end. All three are interpreter-backed; the native route does not carry them yet, and the declarations say so. Tests: tests/matrix/execution/byte-access and entropy are exact-output fixtures, the second asserting shape, bounds and that two draws differ -- the one check that would catch MINSTD standing in for a secure source. tests/tools/net_socket_contract.py runs a Matrix page as a process and plays the other end from Python, because what matters about a socket is what the peer sees. Six cases: bytes cross the wire unchanged including a zero byte and invalid UTF-8, a handle the program never received is refused, a listener is not a connection, a leaked listener's port is released for the next run, a read deadline expires instead of hanging, and a peer that vanishes mid-write is reported rather than killing the process. Registered as net.socket-contract. The full Matrix execution suite passes: 37 fixtures. Docs: docs/standard-library/sockets.md is new, random.md and text-spans.md grew sections, and index.md no longer claims network transports and secure random generation are experimental, because they are not any more. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ny.routes.reject-unbacked-io-security asserts that a declaration with no runtime provider is rejected at compile time rather than accepted and then failing where it is used. random_bytes was on that list, and the entropy intrinsic took it off: it now type-checks, so the fixture's expected diagnostics no longer matched and the test failed in both CI configurations. Removed from the fixture rather than left failing, because what the test guards is the gap between a declaration and a provider, and for that call there is no longer a gap. Compression, hashing and encryption are still designs under experimental/ with nothing behind them and stay. A comment in the fixture records where random_bytes went, and tests/matrix/execution/entropy.matrix is the coverage that replaces it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sockets in the previous commit could serve one connection at a time, which is a development server and not a web server. This is what it takes to serve many. An event loop, not a thread per connection, because a thread is not available and will not be soon: the interpreter holds one stack, one heap and one set of globals, and there is not a mutex, an atomic or a thread_local anywhere in it. nil.std.threading offers sleep. Making the VM thread-safe is a far larger piece of work than this, and an event loop does not need it. So: set_blocking, watch, unwatch, watch_count, wait and next_ready, over poll(2). The watch set lives in the runtime rather than being passed in per call, which is not tidiness -- a Matrix page cannot index an Ny generic, so an array-taking interface would be an event loop only half the language could write. next_ready packs the handle and the flags into one number because an intrinsic returns one value, and two calls whose meaning depended on being made in order would be worse than arithmetic the caller can see; ready_handle, ready_flags, is_readable, is_writable and is_finished are ordinary Ny functions over the packing. The readiness flags are deliberately not poll(2)'s numbering, which differs between platforms: a program that learned the host's values would break when it moved. A non-blocking accept with nothing pending now answers zero rather than reporting an error. Handles start at one so it cannot be mistaken for one, and for an event loop it is the ordinary case -- poll said the listener was readable and the connection was withdrawn in between. Reporting it would mean a log line every time a port scanner connected and left. set_reuse_port is the other half of using more than one core. A second core means a second process here, and two processes cannot share a listening socket without it. Opt-in and distinct from the SO_REUSEADDR that listen always sets: that one only allows rebinding a port in TIME_WAIT, while this one allows two servers to share a port, and switched on by accident it would silently split traffic between a running server and a stray one. A wait replaces whatever the previous one found, rather than accumulating, so a program that did not drain the set cannot later be handed a handle that has since been closed and its number reissued. A socket closed since the last wait is dropped from the set rather than polled on a descriptor that now means something else, so a loop that forgets to unwatch a connection it closed is untidy rather than broken. An interrupted poll is retried, because an unrelated signal must not look to a server like a broken socket. Two new contract cases, bringing net.socket-contract to eight. The first covers the watch set: an empty one does not wait, watching twice replaces rather than adds, a drained ready set reads zero, unwatching something unwatched succeeds, a closed socket leaves the set, and the packing round-trips. The second is the one worth reading. Sixty-four clients connect at once and each sends its request in two parts with a quarter-second pause between. A server that serves one connection at a time has to wait out the first client's pause before it touches the second, so the run costs sixty-four pauses; an event loop overlaps them and costs about one. The test asserts that difference and that the server held all sixty-four at once, rather than asserting a throughput number -- a number would measure the machine, and this measures the shape. Locally: 200 clients, all answered, 0.31s against the 50s a serial server would need. docs/standard-library/sockets.md gains the loop's shape, the three things it has to get right (drain the listener; leave connections blocking with a timeout, because EAGAIN and a clean close are both no bytes and only a blocking read tells them apart; readable and finished can both be true), and what is still missing. The claim that there was no way to wait on several sockets is gone, since it was the thing this commit removes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two more things a web server cannot be written without. net.receive and net.send_some are the non-blocking counterparts of read and write. write is the dangerous one in an event loop: a large response to a client that reads slowly fills the socket buffer, write_all blocks, and one slow reader stops every other connection. send_some takes what the socket will accept from an offset and reports it, leaving the remainder for when poll says the socket will take more. receive is two calls because a non-blocking read has three outcomes and a Result<text, text> has room for two: bytes arrived, the peer closed, or nothing is available yet. It reports which, and net.received hands over the bytes. Conflating the last two is the classic event-loop bug -- a spurious wakeup looks like a hangup and the connection is dropped in the middle of a request. file.modified_at closes the other gap. nil.std.io had a size and no timestamp, which left two ordinary jobs unreachable: a cache could not tell whether what it held was still current without re-reading the file, which defeats the cache, and an HTTP server could not send Last-Modified or answer If-Modified-Since, so every client re-fetched every unchanged file in full, forever. Milliseconds since the Unix epoch, because a filesystem timestamp is not reliably finer than that across platforms and a caller comparing two should not be comparing precision it does not have. The conversion is the portable one rather than std::chrono::clock_cast, which says it in a line and is not available on every toolchain this still builds on: file_clock's epoch is unspecified and need not match the system clock's, so the two are related by taking a reading from each at the same moment. tests/matrix/execution/file-time asserts the answer is a plausible epoch time in milliseconds -- a seconds-based answer fails the lower bound, which is the mistake worth catching -- that writing again moves it forward, and that an absent file is reported rather than given a zero a comparison would read as 1970 and treat as very old indeed. 38 execution fixtures now pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guide is the reference an agent writes Matrix from, and it described none of NIL-0014's formatted literals or NIL-0016's queries and text patterns, so generated code still builds text with plus ladders and hand-rolled loops. Each added example is compiled and run: the formatted literals join guide-constructs, and the query and pattern sections get a standard-library fixture of their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A generator currently assembles a document out of fragments, which puts the control flow outside the markup, leaves the structure unchecked and makes escaping manual and context-blind. NIL-0027 stages the fix: a multiline formatted block first, then a markup literal that holds its own conditionals and loops and evaluates to a checked tree rather than text. The record settles the four decisions taken with it: raw-text element bodies stay literal, the parser carries no dialect, the value is a tree rendered by nil.std.markup, and the amortized text builder lands first. It also records that the obvious node-record representation cannot execute natively, so a markup value is an arena of scalar and text arrays. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIL-0027 section 2. 'formatted' in front of 'begin text' opts a raw block into interpolation, so a generated document is written as one unit instead of a ladder of concatenations. Every raw-block rule still holds: the terminator is a line whose only content is 'end text', the common leading indentation is removed and deeper indentation kept, and nothing inside is escaped. A single hole still stays on one physical line. Indentation removal applies to literal segments only. The lexer records the source offset each dedented line starts at and maps every scanned piece back through it, so a hole's tokens keep their original file coordinates and NIL-0014's editing contract is unchanged. Because the block emits the token stream the quoted form already emits, the parser, checker, lowering, VM and native routes needed no change. Covered by the formatted-text-matrix parity fixture across the VM, binary and native routes, malformed blocks rejected on every route, a hole renamed inside a CRLF block while an identical word in doubled braces is left alone, and completion inside a block hole. Ny keeps only its single-line form. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIL-0027 section 3. 'begin markup' holds its own conditionals and loops, so
the control flow lives inside the page instead of around it, and the block
is parsed as markup at compile time: an unbalanced element, an unterminated
directive, an arm that opens an element it does not close, and a hole that
would name an element or an attribute are all compile errors.
Escaping belongs to the position, never to the author. A hole learns during
parsing whether it landed in element content, an attribute value, a whole
URL attribute or a component inside one, and lowers to the matching
nil.std.markup function; an executable scheme is neutralised rather than
dropped. '{raw ...}' is the single greppable way to bypass that.
The value is a nil.std.markup document rather than text, which is what lets
a paragraph return a fragment and another block splice it instead of
escaping it twice. NIL-0027 section 3.6 records why the record-graph tree
that section originally specified was replaced by compile-time
serialization: it cannot execute natively, and it bought only the one
capability that was already deferred.
Inside script and style a brace is an ordinary character, so CSS and
JavaScript paste in unaltered; a body that is exactly one hole still
interpolates through '{raw ...}'. 'markup' and 'raw' stay ordinary names.
The literal reuses the block scanner from section 2, so holes keep their
original source coordinates, and emit_markup produces ordinary NIL, so one
emitter covers the VM and native routes. Covered by the markup-matrix
parity fixture across the VM, binary, native and native-toolchain routes,
and by a contract rejecting sixteen malformed documents on every route
while proving hostile input is escaped once in each context.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIL-0027 section 2 for the other surface. '$"""' opens the block Matrix spells 'formatted begin text', sharing scan_formatted_block, so both frontends emit the same token stream and the same interpolation AST. One difference is forced by the surface rather than chosen: Matrix's 'end text' owns its line, while Ny closes on the first line whose content begins with '"""' and resumes lexing immediately after those quotes, because Ny is expression oriented and println($"""...""") has to work. A block is raw in both surfaces. Ny's backslash escapes apply to a quoted $"..." and not inside $""", which is what makes a path, a pattern or embedded JSON writable without doubling anything. Indentation removal applies to literal segments only, so holes keep their original file coordinates. An unterminated block now reports its own message rather than only 'Expected expression': the parser surfaces lexer messages for interpolation and formatted text, and formatted blocks join that list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIL-0027 section 3 for the other surface. The document parsing, the
structural checks and the escape decisions move into markup_structure, which
both frontends drive; each supplies only its own directive words and its own
expression lexing, so the surfaces cannot drift in what a document means.
Ny writes {if}, {elseif}, {else}, {end if}, {for x in xs} and {end for};
Matrix keeps {when} and {repeat for each}. The two parity fixtures render
the same page byte for byte apart from their own indentation.
Ny opens a block with '$markup' on its own line rather than the
'markup { ... }' the design record first guessed. A markup block is full of
braces - holes, and the CSS or JavaScript inside a raw-text element - so a
brace-delimited block would end at the first '}' in a stylesheet. The
terminator has to be a line-anchored keyword, and the '$' is what keeps the
opener unambiguous, since Ny has no 'begin' and a bare word before a newline
could be an ordinary name.
Two things only the Ny side could have found. The sigil skips six characters
after the already-consumed '$', not seven, and getting that wrong swallowed
the first content line into the header. Ny also names an imported type by
the spelling the author wrote, so the document type is now recognised by the
declaration it resolves to rather than by its name - a program's own struct
called 'document' must not be spliced unescaped.
Covered by a markup.ny parity fixture across the VM, binary, native and
native-toolchain routes, and by ten more rejected documents plus an
unterminated block in the markup contract.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Matrix web applications need binary-safe socket I/O, OS entropy, byte access and stable file modification times. This adds those APIs and verifies the same interpreter behavior on Linux, Windows, Intel macOS and Apple Silicon macOS. Timestamps also run through LLVM JIT/AOT. Socket, byte and entropy APIs remain interpreter-backed on every host until the native text ABI can preserve embedded NUL bytes.
Sockets use opaque VM-owned handles, close on teardown/reset/fault, reject stale handles, preserve Windows SOCKET width and distinguish EOF from timeout/error. Polling is bounded and writes handle the platform's SIGPIPE behavior. Secure randomness uses the host OS provider; timestamps use native filesystem metadata.
Application tests exposed additional platform defects corrected here:
SDK packaging now keeps compilation private, then copies validated library files into fresh destination entries that inherit the SDK's normal access policy. The former temporary-directory rename preserved private permissions and blocked a second Windows account. Linux umask/rollback controls and actual limited-account Windows reads cover the correction, with private staging still inaccessible.
Validation is recorded by exact source/candidate identity in the notebook:
9fa354d9fa354d59ffa9759ffa97four-host workflow also succeeds.59ffa97, all 22 platform/tooling/packaging CTests, four desktop DAP sessions and all 88 editor groups pass per host. VS Code 1.146.0, all nine modules, no exclusions; 620 evidence hashes independently verified.1efbe663cc2515Sources: Mac 22-contract campaign, ARM installed replay and preserved Intel failure, successful Intel installed replay, and the parity notebook.
The supported common development profiles have installed acceptance at the pinned revisions above. Failed/interrupted attempts, the unresolved historical Intel timeout and common socket/byte/entropy native limitations remain explicit. Same-user Mac acceptance does not qualify arbitrary shared deployment.
Ny and Matrix now also support multiline formatted text and structured HTML markup literals (NIL-0027). Literal documents are checked at compile time; scalar holes use context-specific escaping, documents compose without double escaping, and conditions/loops live inside the document. Both frontends share the scanner and document checker. XML, compact rendering, dynamic attribute sets and a runtime DOM remain deferred.
The completion review adds regressions for malformed Ny block openers, frontend-specific hole lexing, browser URL normalization, unsafe attribute contexts, ambiguous URL fragments and malformed directive/document boundaries. The four-host interpreter workflow now runs the markup and formatted-text contracts plus all four positive fixture routes; both Mac native workflows also run the ten language contracts and toolchain fixtures.
The new features require a compiler and standard-library source built from this branch. The currently installed SDK predates them and is not replaced by this PR. Source checks and the historical installed acceptance above are separate qualifications. No stable release or Lich release is activated.
The public web server has since completed HTTP framing, pipelining, conditional/range handling and rooted static-file snapshots in exec-nil-framework-web PR #1, with all four platform gates green at
480ecae. Its rooted-file runtime additions and WebView bridge are separate dependent PRs, not part of this language handoff.