From 24a034318eaa63fa960b8298634da85bb01611f9 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 14:42:40 -0500 Subject: [PATCH 01/39] Sockets, secure random, and byte access to text Three gaps in the runtime, found while writing a web framework in Matrix and each one a hard stop rather than an inconvenience. **No networking at all.** Not an unfinished API: the intrinsic table covered files, paths, directories, databases, the environment, the console, regular expressions and GPU compute, and nothing that opens a socket. nil.std.net has described transport-independent HTTP for a while and said in its own comments that the transport was absent; the socket design under experimental/ had nothing beneath it. So no NIL or Matrix program could accept a connection. nil.std.net.socket now has listen, accept, connect, read, write, close, is_open, set_timeout, set_no_delay and peer_address, over net_support.hpp. Everything returns a Result, because a network call fails for reasons outside the program and for a server those are ordinary answers. A program is handed an opaque token, never a host descriptor, so it cannot touch something it was not given; a listener and a connection are told apart by name rather than by EINVAL; and the runtime closes whatever a run leaves open, so a program that exits or crashes without closing its listener does not leave the port bound against the next run. Writes use MSG_NOSIGNAL: without it a peer closing mid-write delivers SIGPIPE and the default disposition ends the process, which means a server that dies when a browser closes a tab. Binding does not resolve names -- numeric, empty or localhost -- because a DNS lookup on that path is both blocking and a way to end up listening somewhere unintended; connecting does resolve, because a client is handed names. **No cryptographically secure randomness.** nil.std.math is MINSTD seeded at 12345 and says itself that it is not cryptographic; nil.std.security had base64 and hex and no random; there were no crypto intrinsics; and /dev/urandom was not a way round it, since the file intrinsics refuse a character device by design and the unchecked read returns nothing for one. A session id, a CSRF token or a password-reset link drawn from MINSTD is one anyone can compute, so any library needing a credential had to invent something predictable or drop the feature. nil.std.security.random now has random_hex, random_bytes and random_below over entropy_support.hpp -- getrandom(2) with a /dev/urandom fallback on POSIX, rand_s on Windows. No userspace pool, no seeding, no state, and no fallback to a weak source: if the kernel cannot answer the result is an error, because quietly returning something guessable is how this goes wrong unnoticed. random_below uses rejection sampling rather than a remainder, which would skew towards the low values. random_hex is the one to reach for, partly because a credential wants to be text and partly because Matrix cannot index an Ny byte array. **No way to read a byte.** nil.std.text measures and slices by byte but had no byte-to-number conversion, and convert.char_to_i32 -- the one that looked like it would -- takes a whole Unicode scalar and aborts the process on a partial one. Not a fault attempt can catch: a fatal runtime error. So a byte inside a multi-byte character could not be inspected, and such a character could not be isolated in order to inspect it whole, because isolating it needed the same test. Percent-encoding anything outside ASCII was unreachable because of this, and so was reading arbitrary bytes off a socket. text.byte_at and text.from_byte are deliberately about bytes and say so. from_byte(195) is the single byte 0xC3 where convert.i32_to_char(195) is the two bytes encoding U+00C3; joining text joins bytes, so a character can now be assembled one byte at a time. byte_at answers minus one past the end rather than raising, because reading past the end is how a byte scan finds its end. All three are interpreter-backed; the native route does not carry them yet, and the declarations say so. Tests: tests/matrix/execution/byte-access and entropy are exact-output fixtures, the second asserting shape, bounds and that two draws differ -- the one check that would catch MINSTD standing in for a secure source. tests/tools/net_socket_contract.py runs a Matrix page as a process and plays the other end from Python, because what matters about a socket is what the peer sees. Six cases: bytes cross the wire unchanged including a zero byte and invalid UTF-8, a handle the program never received is refused, a listener is not a connection, a leaked listener's port is released for the next run, a read deadline expires instead of hanging, and a peer that vanishes mid-write is reported rather than killing the process. Registered as net.socket-contract. The full Matrix execution suite passes: 37 fixtures. Docs: docs/standard-library/sockets.md is new, random.md and text-spans.md grew sections, and index.md no longer claims network transports and secure random generation are experimental, because they are not any more. Co-Authored-By: Claude Opus 5 --- CMakeLists.txt | 15 + docs/standard-library/index.md | 22 +- docs/standard-library/random.md | 63 ++ docs/standard-library/sockets.md | 140 +++++ docs/standard-library/text-spans.md | 44 ++ nilcompiler/src/entropy_support.hpp | 222 +++++++ nilcompiler/src/net_support.hpp | 433 +++++++++++++ nilcompiler/src/vm.cpp | 403 ++++++++++++ nilcompiler/src/vm.hpp | 16 + nilstd/src/net/package.ny | 13 +- nilstd/src/net/socket.ny | 133 ++++ nilstd/src/security/package.ny | 10 +- nilstd/src/security/random.ny | 67 ++ nilstd/src/text/string_utils.ny | 36 ++ tests/matrix/execution/byte-access.expected | 13 + tests/matrix/execution/byte-access.matrix | 52 ++ .../matrix/execution/byte-access.needs-stdlib | 0 tests/matrix/execution/entropy.expected | 10 + tests/matrix/execution/entropy.matrix | 112 ++++ tests/matrix/execution/entropy.needs-stdlib | 0 tests/tools/net_socket_contract.py | 573 ++++++++++++++++++ 21 files changed, 2364 insertions(+), 13 deletions(-) create mode 100644 docs/standard-library/sockets.md create mode 100644 nilcompiler/src/entropy_support.hpp create mode 100644 nilcompiler/src/net_support.hpp create mode 100644 nilstd/src/net/socket.ny create mode 100644 nilstd/src/security/random.ny create mode 100644 tests/matrix/execution/byte-access.expected create mode 100644 tests/matrix/execution/byte-access.matrix create mode 100644 tests/matrix/execution/byte-access.needs-stdlib create mode 100644 tests/matrix/execution/entropy.expected create mode 100644 tests/matrix/execution/entropy.matrix create mode 100644 tests/matrix/execution/entropy.needs-stdlib create mode 100644 tests/tools/net_socket_contract.py diff --git a/CMakeLists.txt b/CMakeLists.txt index e69001db..cf8f5627 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1023,6 +1023,21 @@ if(BUILD_TESTING) --interpreter $ --root "${CMAKE_CURRENT_SOURCE_DIR}") set_tests_properties(matrix.samples PROPERTIES LABELS "matrix;routes" TIMEOUT 180) + # The network intrinsics, against a real peer. A socket cannot be + # checked by comparing stdout, because the interesting behaviour is + # what the other end of the connection sees: the test runs a Matrix + # page as a process and plays the other side from Python. It covers + # the cases a simpler fixture passes without exercising -- an opaque + # handle refusing a number it never issued, a leaked listener whose + # port the runtime still has to release, a read deadline expiring, and + # a peer that vanishes mid-write being reported rather than killing + # the process. + add_test(NAME net.socket-contract + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/net_socket_contract.py" + --compiler $ + --interpreter $ + --root "${CMAKE_CURRENT_SOURCE_DIR}") + set_tests_properties(net.socket-contract PROPERTIES LABELS "net;routes;rc" TIMEOUT 300) add_test(NAME nillsp.semantic-protocol COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/lsp_semantic_protocol.py" --lsp $) diff --git a/docs/standard-library/index.md b/docs/standard-library/index.md index 1d10bc3f..a489ec7f 100644 --- a/docs/standard-library/index.md +++ b/docs/standard-library/index.md @@ -20,6 +20,10 @@ pagination, projection and the Matrix-authored collection terminals. The [vector norm reference](math-vectors.md) describes stable two- and three-component lengths, finite-value checks, and checked normalization shared by Ny and Matrix. +The [socket reference](sockets.md) defines TCP listening, accepting, +connecting and transfer, with opaque handles the runtime closes for a program +that does not. + The [random-generator reference](random.md) defines independent mutable streams, reproducible seeds, cloning, and bounded integer/floating sampling. The [checked text-span reference](text-spans.md) defines exact byte ranges, @@ -212,12 +216,18 @@ fixed-width encoding, and CRC, including detection of a one-bit change. Each active `nil.std` family has at least one exact-output fixture through textual and binary NIL, but those fixtures establish selected behavior rather than every declaration in a family. The aggregate `prelude` has strict -compile/package coverage, not its own standalone behavior oracle. Network -transports, JSON, broad concurrency/threading, compression, -cryptographic hashing, encryption, and secure random generation remain experimental. -`http.send` and all network transports remain quarantined because no runtime -dispatcher is shipped, and active `nil.std.system` does not include subprocess -execution or agent messaging. +compile/package coverage, not its own standalone behavior oracle. JSON, broad +concurrency/threading, compression, cryptographic hashing and encryption remain +experimental, and active `nil.std.system` does not include subprocess execution +or agent messaging. + +TCP stream sockets are active as of `nil.std.net.socket`, and secure random +values as of `nil.std.security.random`; both are interpreter-backed and their +contracts are checked by dedicated fixtures. See the +[socket reference](sockets.md) and the secure-random section of the +[random reference](random.md). `http.send` and the higher-level transports — +HTTP client and server, WebSocket, IMAP, SMTP — remain quarantined, now for +want of a built provider rather than for want of a transport to build one on. The [database and networking capability boundary](database-networking.md) records the audited split between active deterministic builders and quarantined diff --git a/docs/standard-library/random.md b/docs/standard-library/random.md index da9925b9..83748494 100644 --- a/docs/standard-library/random.md +++ b/docs/standard-library/random.md @@ -89,3 +89,66 @@ draws, and extreme floating ranges. The negative contract runner exercises inverted and nonfinite bounds for both frontends and both APIs. Route checks cover textual/binary NIL, bundles, LLVM JIT and AOT; installed consumers also exercise source and binary-library imports and both assembler/linker pipelines. + +--- + +# Secure random values + +Import `nil.std.security.random`. These are a different thing from the +generators above and the distinction is the whole point: MINSTD is +deterministic and seeded at a known value, so it is the right generator for a +simulation, a shuffle or a jittered retry, and the wrong one for anything a +user holds. + +A session identifier, a CSRF token, a password-reset link and a nonce are all +bearer credentials. One drawn from MINSTD is one anyone can compute: two draws +reveal the stream. + +| Ny API | Behavior | +| --- | --- | +| `random_hex(byte_count: i64) -> Result` | Random bytes as lower-case hex, two characters per byte. | +| `random_bytes(byte_count: i64) -> Result` | Random bytes as they are, carried as text. | +| `random_below(bound: i64) -> Result` | A uniform whole number in `[0, bound)`. | + +Interpreter-backed. The native route does not carry these yet. + +`random_hex` is the one to reach for. A credential wants to be text anyway, and +hex survives a cookie, a URL, a header and a log line with no further escaping; +it is also the form Matrix can use, because a Matrix page cannot index an Ny +byte array. Sixteen bytes — thirty-two characters — is a reasonable session +identifier. + +`random_bytes` returns arbitrary bytes carried in a text value. It is not valid +UTF-8 in general, so read it with `nil.std.text`'s `byte_at`, never with +`substring` or a character count. + +`random_below` uses rejection sampling rather than a remainder. Taking a draw +modulo the bound skews towards the low values whenever the bound does not +divide the range evenly — a bias small enough never to be noticed and large +enough to matter to anything that was counting on uniformity. + +These ask the operating system and do nothing else: `getrandom(2)` on Linux +falling back to `/dev/urandom`, and the CRT's `rand_s` on Windows. There is no +userspace pool, no seeding and no state. There is also no fallback to a weak +source — if the kernel cannot provide entropy the answer is an error, because +quietly returning something guessable is how this goes wrong in a way nobody +notices until it matters. + +A single draw is limited to 1024 bytes. A key is 32 and a generous token is 64; +asking for a megabyte is a program that wants a stream cipher seeded from here +instead. + +```matrix +read nil.std.core +read nil.std.security.random + +new-session-id returns text begin paragraph + value drawn is nil.std.security::random_hex with byte_count being 16 + consider drawn + when Result.Ok holding token then give token + !!! No fallback. A predictable identifier is worse than no session. + when Result.Error holding why then raise entropy-unavailable with reason being why + otherwise raise entropy-unavailable with reason being "no entropy" + end consider +end paragraph +``` diff --git a/docs/standard-library/sockets.md b/docs/standard-library/sockets.md new file mode 100644 index 00000000..464d6d28 --- /dev/null +++ b/docs/standard-library/sockets.md @@ -0,0 +1,140 @@ +# Stream sockets + +Import `nil.std.net.socket` from Ny or Matrix. This is TCP as the runtime +provides it: the transport `nil.std.net.http` has been describing without one. + +Interpreter-backed. The native route does not carry these yet, and a program +compiled through it will not find them. + +| Ny API | Behavior | +| --- | --- | +| `listen(host: text, port: i32, backlog: i32) -> Result` | Open, bind and listen in one step; gives back a handle. | +| `accept(handle: i64) -> Result` | Block until a connection arrives; gives back a handle to it. | +| `connect(host: text, port: i32) -> Result` | Connect out, resolving a name if it is one. | +| `read(handle: i64, maximum_bytes: i64) -> Result` | Read what has arrived, up to a ceiling. | +| `write(handle: i64, data: text) -> Result` | Send all of it; gives back the byte count. | +| `close(handle: i64) -> Result` | Close and release the handle. | +| `is_open(handle: i64) -> bool` | Whether the runtime still holds this handle. | +| `set_timeout(handle: i64, milliseconds: i64) -> Result` | Deadline for reads and writes; zero removes it. | +| `set_no_delay(handle: i64, enabled: bool) -> Result` | Send small writes immediately. | +| `peer_address(handle: i64) -> Result` | The peer's address and port, as text. | + +Matrix uses the same names; `maximum_bytes` is spelled `maximum_bytes` because +it is an Ny declaration and Matrix keeps a foreign name's spelling. + +## Everything returns a Result + +A network call fails for reasons entirely outside the program — the port is +taken, the peer went away, the name does not resolve. For a server those are +ordinary answers on an ordinary day, so none of these raise. + +## Handles are opaque + +A program is handed a token into a runtime-owned table, never a host +descriptor. It cannot read from or close something it was not given, including +a descriptor the host opened for its own purposes, and a number it never +received is refused with a sentence rather than used. + +A listener and a connection are told apart: accepting on a connection, or +reading from a listener, is a named error and not an `EINVAL`. + +The runtime closes whatever a run leaves open. A program that exits without +closing its listener — or one that crashes — does not leave the port bound +against the next run, which during development is the difference between a +server that restarts and one that fails every second time for a reason +unrelated to the change just made. + +## Binding does not resolve names + +`host` is numeric, empty, or `localhost`. A server binds an address it already +knows, and a DNS lookup on that path is both a blocking call and a way to end +up listening somewhere unintended. An empty host means every interface, which +is right behind a firewall and wrong on a laptop on a shared network; the +caller chooses and the library does not choose for them. + +`connect` does resolve, because a client is handed names. A host with both an +IPv4 and an IPv6 address is tried in turn and the first to connect wins. + +## Reads are short, and a close is not an error + +TCP is a stream. A request may arrive in several pieces, so a read returning +less than asked for is normal. An orderly close by the peer comes back as +empty text rather than an error, so a loop reading to the end of a request +sees one shape; `is_open` distinguishes them when it matters. + +What `read` gives back is bytes, not necessarily valid UTF-8 — it is whatever +the peer sent. Read it with `nil.std.text`'s `byte_at` rather than `substring` +if it may not be text. One call never transfers more than 8 MiB. + +## Writes complete or report how far they got + +A single `send` is permitted to take less than it was given. `write` loops +until everything is gone or the attempt fails, because the number of servers +that have shipped a truncated response from the one-call version is large. On +failure the count says how far it reached. + +A peer that closes mid-write is reported rather than fatal: the runtime +suppresses the signal whose default disposition would end the process. A +server that dies when a browser closes a tab is not a server, and this is the +case that a test which never closes a connection early will not catch. + +## Timeouts are not optional in practice + +A single-threaded server without `set_timeout` is one slow client away from +serving nobody: a connection that opens and then says nothing holds the accept +loop for as long as it likes. Set a deadline on every accepted connection. + +## An echo server + +```matrix +book example.echo + +read nil.std.core +read nil.std.text.string_utils +read nil.std.net.socket + +main returns number begin paragraph + value opened is nil.std.net::listen with host being "127.0.0.1", port being 8080, backlog being 16 + consider opened + when Result.Error holding why then begin block + say "cannot listen: " plus why + give 1 + end block + when Result.Ok holding listener then begin block + repeat forever + value taken is nil.std.net::accept with handle being listener + consider taken + when Result.Ok holding client then begin block + evaluate nil.std.net::set_timeout with handle being client, milliseconds being 5000 + value heard is nil.std.net::read with handle being client, maximum_bytes being 65536 + consider heard + when Result.Ok holding incoming then + evaluate nil.std.net::write with handle being client, data being incoming + otherwise say "read failed" + end consider + evaluate nil.std.net::close with handle being client + end block + when Result.Error holding why then say "accept failed: " plus why + otherwise say "accept gave neither" + end consider + end repeat + give 0 + end block + otherwise give 1 + end consider +end paragraph +``` + +## What is not here + +One connection at a time. There are no threads to hand a connection to and no +async in the language, so a server built on this accepts, serves and closes in +order. That is enough for a development server, a local tool, or anything +behind a reverse proxy that holds the slow clients. + +No TLS. No datagrams. No `select` or `poll`, so a program cannot wait on +several sockets at once — which is the piece that would have to arrive before +concurrent serving is possible. + +The contract these keep is checked by `tests/tools/net_socket_contract.py`, +which runs a Matrix page as a process and plays the other end from Python. diff --git a/docs/standard-library/text-spans.md b/docs/standard-library/text-spans.md index cedb99c8..0b0d2ba1 100644 --- a/docs/standard-library/text-spans.md +++ b/docs/standard-library/text-spans.md @@ -64,3 +64,47 @@ Paired Ny/Matrix fixtures cover exact spans, empty results, invalid bounds, signed extrema, UTF-8 byte positions, aliases, and the compatibility clamp. They are registered for text/binary NIL, bundles, LLVM JIT/AOT, standalone assembler/linker execution, and installed source/binary libraries. + +## Byte values + +`nil.std.text` measures and slices by byte — `length` of `"café"` is 5 — but +until these two existed there was no way to read a byte *as a number* or write +one as text. + +| Function | Result | +| --- | --- | +| `byte_at(s: text, index: i64) -> i64` | The byte at an index, `0..255`, or minus one when the index is outside the text. | +| `from_byte(code: i64) -> text` | One byte as text; empty text for a code outside `0..255`. | + +The conversion that looks like it would do this is +`nil.std.core`'s `convert.char_to_i32`, and it will not: it takes a whole +Unicode scalar and **aborts the process** on a partial one. That is a fatal +runtime error, not a fault `attempt` can catch. So a byte in the middle of a +multi-byte character could not be inspected, and such a character could not be +isolated in order to inspect it whole, because isolating it needed the same +test. Percent-encoding anything outside ASCII was unreachable because of this, +and so was reading arbitrary bytes off a socket. + +These two are about bytes and say so in their names. They know nothing about +UTF-8; assembling characters from bytes is the caller's job. + +`from_byte(195)` is the single byte `0xC3`, where `convert.i32_to_char(195)` is +the *two* bytes that encode U+00C3. Joining text joins bytes, so a multi-byte +character can be built one byte at a time: + +```matrix +read nil.std.text.string_utils + +!!! The two bytes that spell "é", built from their values. +value accented is (nil.std.text::from_byte with code being 195) + plus (nil.std.text::from_byte with code being 169) +``` + +`byte_at` answers rather than raising when the index is out of range, because +reading past the end is how a scan over bytes discovers it has finished, and a +loop that must check the length separately is a loop that will sometimes +forget to. + +`try_char_at` above gives the same byte as a one-byte text wrapped in an +`Option`. Use it when the byte is wanted as text and `byte_at` when it is +wanted as a number. diff --git a/nilcompiler/src/entropy_support.hpp b/nilcompiler/src/entropy_support.hpp new file mode 100644 index 00000000..2edb9db0 --- /dev/null +++ b/nilcompiler/src/entropy_support.hpp @@ -0,0 +1,222 @@ +#ifndef NIL_ENTROPY_SUPPORT_HPP +#define NIL_ENTROPY_SUPPORT_HPP + +// Cryptographically secure random bytes for the NIL runtime. +// +// Before this, nothing in the standard library could produce an unpredictable +// value. `nil.std.math` is MINSTD — a 31-bit linear congruential generator +// whose global stream starts at the seed 12345 — and says in its own comments +// that it is not cryptographic; two draws reveal the whole stream. +// `nil.std.security` had base64 and hex and no random at all. /dev/urandom was +// not a way round it either: the file intrinsics refuse a character device by +// design, and the unchecked read returns nothing for one. +// +// That gap is not an inconvenience, it is a vulnerability generator. Session +// identifiers, CSRF tokens, password-reset links and nonces are all bearer +// credentials, and a credential drawn from MINSTD is a credential anyone can +// compute. Any library that needed one had to either invent something +// predictable or refuse to offer the feature. +// +// This asks the operating system and does nothing clever. There is no +// userspace pool to get wrong, no seeding, no state, and no fallback to a +// weak source: if the kernel cannot provide entropy, the answer is an error, +// because silently returning something guessable is how this goes wrong. + +#include +#include +#include + +#if defined(_WIN32) +// rand_s is the CRT's wrapper over RtlGenRandom. It needs no extra library, +// which keeps this from changing how the project links on Windows. +#define _CRT_RAND_S +#include +#else +#include +#include +#include +#include +#if defined(__linux__) +#include +#endif +#endif + +namespace ny::entropy_support { + +/** + * The most one call may ask for. + * + * A key is 32 bytes and a generous token is 64. A request for a megabyte of + * entropy is a program using the wrong tool — a stream cipher seeded from + * here is the right one — and the ceiling makes that a readable error instead + * of a quiet stall on a machine whose pool is cold. + */ +constexpr std::int64_t max_random_bytes = 1024; + +struct Drawn { + std::string bytes; + std::string error; + bool ok() const { return error.empty(); } +}; + +#if !defined(_WIN32) + +/** + * Fill from the kernel. + * + * getrandom(2) first, because it needs no file descriptor and so cannot fail + * because the process ran out of them or because something unmounted /dev. + * A short return is retried: the call is documented to give fewer bytes than + * asked for when a signal arrives, and treating that as failure would make an + * unrelated SIGWINCH break a token. + */ +inline bool fill_from_kernel(unsigned char *into, std::size_t count, + std::string &error) { +#if defined(__linux__) && defined(SYS_getrandom) + std::size_t filled = 0; + bool syscall_usable = true; + while (filled < count) { + const long taken = ::syscall(SYS_getrandom, into + filled, count - filled, 0); + if (taken > 0) { + filled += static_cast(taken); + continue; + } + if (taken < 0 && errno == EINTR) + continue; + // ENOSYS means a kernel older than the call; anything else here is real, + // but the device is still worth trying before giving up on the machine. + syscall_usable = false; + break; + } + if (syscall_usable && filled == count) + return true; +#endif + + // The device, for a kernel without the syscall. Opened and closed per call: + // this is not a hot path, and holding a descriptor open for the life of the + // process is a descriptor a program did not ask the runtime to keep. + std::FILE *device = std::fopen("/dev/urandom", "rb"); + if (device == nullptr) { + error = "random: no entropy source is available on this system"; + return false; + } + const std::size_t taken = std::fread(into, 1, count, device); + std::fclose(device); + if (taken != count) { + error = "random: the entropy source returned fewer bytes than requested"; + return false; + } + return true; +} + +#else + +inline bool fill_from_kernel(unsigned char *into, std::size_t count, + std::string &error) { + std::size_t filled = 0; + while (filled < count) { + unsigned int drawn = 0; + if (::rand_s(&drawn) != 0) { + error = "random: no entropy source is available on this system"; + return false; + } + for (int byte = 0; byte < 4 && filled < count; ++byte) { + into[filled] = static_cast((drawn >> (byte * 8)) & 0xFF); + filled += 1; + } + } + return true; +} + +#endif + +inline Drawn random_bytes(std::int64_t count) { + Drawn drawn; + if (count < 1) { + drawn.error = "random: ask for at least one byte"; + return drawn; + } + if (count > max_random_bytes) { + drawn.error = "random: a single draw is limited to " + + std::to_string(max_random_bytes) + " bytes"; + return drawn; + } + std::string buffer(static_cast(count), '\0'); + if (!fill_from_kernel(reinterpret_cast(buffer.data()), + buffer.size(), drawn.error)) + return drawn; + drawn.bytes = std::move(buffer); + return drawn; +} + +/** + * The same bytes as lower-case hex. + * + * Here rather than left to the caller because the caller is often a Matrix + * page, and Matrix cannot index the byte array this would otherwise return. + * A session id wants to be text anyway, and hex is the encoding that survives + * a cookie, a URL and a log line without further escaping. + */ +inline Drawn random_hex(std::int64_t count) { + Drawn drawn = random_bytes(count); + if (!drawn.ok()) + return drawn; + static const char digits[] = "0123456789abcdef"; + std::string out; + out.reserve(drawn.bytes.size() * 2); + for (const unsigned char byte : drawn.bytes) { + out.push_back(digits[(byte >> 4) & 0x0F]); + out.push_back(digits[byte & 0x0F]); + } + drawn.bytes = std::move(out); + return drawn; +} + +/** + * A uniform number in [0, bound). + * + * By rejection, not by remainder. Taking a 64-bit draw modulo the bound makes + * the low values more likely whenever the bound does not divide 2^64, and for + * a small bound the bias is small enough to never be noticed and large enough + * to matter to anything counting on uniformity. + */ +struct Chosen { + std::int64_t value = 0; + std::string error; + bool ok() const { return error.empty(); } +}; + +inline Chosen random_below(std::int64_t bound) { + Chosen chosen; + if (bound < 1) { + chosen.error = "random: the bound must be at least one"; + return chosen; + } + const std::uint64_t span = static_cast(bound); + // The largest multiple of the span that fits; draws at or above it are + // thrown away so every remaining draw maps to exactly one result. + const std::uint64_t limit = std::uint64_t(-1) - (std::uint64_t(-1) % span) - 1; + + // Bounded attempts: with the worst-case span just over half the range, the + // chance of thirty-two rejections in a row is below one in four billion, so + // reaching the end of this loop means the entropy source is not random. + for (int attempt = 0; attempt < 32; ++attempt) { + Drawn drawn = random_bytes(8); + if (!drawn.ok()) { + chosen.error = drawn.error; + return chosen; + } + std::uint64_t raw = 0; + for (int byte = 0; byte < 8; ++byte) + raw = (raw << 8) | static_cast(drawn.bytes[static_cast(byte)]); + if (raw <= limit) { + chosen.value = static_cast(raw % span); + return chosen; + } + } + chosen.error = "random: the entropy source did not yield a usable draw"; + return chosen; +} + +} // namespace ny::entropy_support +#endif diff --git a/nilcompiler/src/net_support.hpp b/nilcompiler/src/net_support.hpp new file mode 100644 index 00000000..7ecddcdd --- /dev/null +++ b/nilcompiler/src/net_support.hpp @@ -0,0 +1,433 @@ +#ifndef NIL_NET_SUPPORT_HPP +#define NIL_NET_SUPPORT_HPP + +// Stream sockets for the NIL runtime. +// +// The standard library has carried a transport-independent HTTP module and an +// unbuilt socket design for some time, with nothing underneath either: there +// was no network intrinsic of any kind, so no NIL or Matrix program could +// accept a connection. This is that layer, kept to the smallest surface a +// server and a client actually need. +// +// Everything here returns a value and never raises. A network call fails for +// reasons entirely outside the program — the port is taken, the peer went +// away, the name does not resolve — and those are ordinary answers a server +// has to handle on every request, not exceptional conditions. +// +// Three decisions are worth stating because they are the ones that bite: +// +// * Writes use MSG_NOSIGNAL. Without it, a peer that closes mid-write +// delivers SIGPIPE and the default disposition kills the process. A web +// server that dies when a browser closes a tab is not a web server. +// * A listening socket sets SO_REUSEADDR but never SO_REUSEPORT. The first +// lets a restart rebind a port still in TIME_WAIT; the second would let +// two unrelated servers share one port and silently split the traffic. +// * Binding never resolves a name through DNS. A server binds an address it +// already knows, and a DNS lookup on that path is a blocking call plus a +// way to end up listening somewhere unintended. `localhost` is understood +// directly; everything else must be numeric. Connecting does resolve, +// because a client is given names. + +#include +#include + +#if defined(_WIN32) +#include +#include +#else +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#endif + +namespace ny::net_support { + +/** One read never takes more than this, however much the caller asks for. */ +constexpr std::int64_t max_transfer_bytes = 8LL * 1024 * 1024; + +/** A listen backlog past this is clamped; the kernel clamps anyway. */ +constexpr std::int64_t max_backlog = 1024; + +struct Opened { + int descriptor = -1; + std::string error; + bool ok() const { return descriptor >= 0 && error.empty(); } +}; + +struct Transferred { + std::string data; + std::int64_t count = 0; + /** The peer closed its side in an orderly way. Not an error. */ + bool closed = false; + std::string error; + bool ok() const { return error.empty(); } +}; + +#if defined(_WIN32) + +// Winsock needs initialising once per process, and the counted +// WSAStartup/WSACleanup pair means a library may start it without taking the +// decision away from whatever else in the process also needs it. +inline bool ensure_started(std::string &error) { + static bool started = false; + static std::string start_error; + if (!started && start_error.empty()) { + WSADATA data; + const int result = WSAStartup(MAKEWORD(2, 2), &data); + if (result != 0) + start_error = "net: Winsock could not be started"; + else + started = true; + } + if (!start_error.empty()) { + error = start_error; + return false; + } + return true; +} + +inline std::string last_error_text() { return "net: the socket call failed"; } +inline int close_descriptor(int descriptor) { + return ::closesocket(static_cast(descriptor)); +} +using transfer_size = int; +constexpr int no_signal_flag = 0; + +#else + +inline bool ensure_started(std::string &) { return true; } + +inline std::string last_error_text() { + const int captured = errno; + const char *text = std::strerror(captured); + return std::string("net: ") + (text ? text : "the socket call failed"); +} + +inline int close_descriptor(int descriptor) { return ::close(descriptor); } +using transfer_size = ssize_t; +#if defined(MSG_NOSIGNAL) +constexpr int no_signal_flag = MSG_NOSIGNAL; +#else +constexpr int no_signal_flag = 0; +#endif + +#endif + +/** + * The address a listening socket may bind. + * + * Numeric only, by design — see the note at the top. `localhost` is accepted + * because it is what a developer types and because its meaning here is not in + * doubt; an empty host means every interface, which is what a container or a + * reverse proxy behind a firewall wants and what a laptop on a café network + * very much does not. The caller chooses, and this does not choose for them. + */ +inline bool fill_bind_address(const std::string &host, std::int64_t port, + sockaddr_in &into, std::string &error) { + if (port < 1 || port > 65535) { + error = "net: a port must be between 1 and 65535"; + return false; + } + std::memset(&into, 0, sizeof(into)); + into.sin_family = AF_INET; + into.sin_port = htons(static_cast(port)); + + if (host.empty() || host == "0.0.0.0" || host == "*") { + into.sin_addr.s_addr = htonl(INADDR_ANY); + return true; + } + if (host == "localhost") { + into.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + return true; + } + if (::inet_pton(AF_INET, host.c_str(), &into.sin_addr) == 1) + return true; + + error = "net: a bind address must be numeric, empty, or localhost"; + return false; +} + +/** Open, bind and listen in one step, because no caller wants them apart. */ +inline Opened listen_on(const std::string &host, std::int64_t port, + std::int64_t backlog) { + Opened opened; + if (!ensure_started(opened.error)) + return opened; + + sockaddr_in address{}; + if (!fill_bind_address(host, port, address, opened.error)) + return opened; + + const int descriptor = static_cast(::socket(AF_INET, SOCK_STREAM, 0)); + if (descriptor < 0) { + opened.error = last_error_text(); + return opened; + } + + const int reuse = 1; + ::setsockopt(descriptor, SOL_SOCKET, SO_REUSEADDR, + reinterpret_cast(&reuse), sizeof(reuse)); + + if (::bind(descriptor, reinterpret_cast(&address), + sizeof(address)) != 0) { + opened.error = last_error_text(); + close_descriptor(descriptor); + return opened; + } + + std::int64_t depth = backlog < 1 ? 128 : backlog; + if (depth > max_backlog) + depth = max_backlog; + if (::listen(descriptor, static_cast(depth)) != 0) { + opened.error = last_error_text(); + close_descriptor(descriptor); + return opened; + } + + opened.descriptor = descriptor; + return opened; +} + +/** Wait for a connection. Retries an interruption rather than reporting one. */ +inline Opened accept_on(int listening) { + Opened opened; + if (!ensure_started(opened.error)) + return opened; + for (;;) { + const int descriptor = static_cast(::accept(listening, nullptr, nullptr)); + if (descriptor >= 0) { + opened.descriptor = descriptor; + return opened; + } +#if !defined(_WIN32) + if (errno == EINTR) + continue; +#endif + opened.error = last_error_text(); + return opened; + } +} + +/** + * Connect to a host, resolving a name if it is one. + * + * Unlike binding, this resolves: a client is handed names. The first address + * that connects wins, so a host with both an IPv4 and an IPv6 address works + * without the caller choosing. + */ +inline Opened connect_to(const std::string &host, std::int64_t port) { + Opened opened; + if (!ensure_started(opened.error)) + return opened; + if (port < 1 || port > 65535) { + opened.error = "net: a port must be between 1 and 65535"; + return opened; + } + if (host.empty()) { + opened.error = "net: connecting needs a host"; + return opened; + } + + addrinfo hints{}; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + addrinfo *found = nullptr; + const std::string service = std::to_string(port); + if (::getaddrinfo(host.c_str(), service.c_str(), &hints, &found) != 0 || + found == nullptr) { + opened.error = "net: the host could not be resolved"; + return opened; + } + + std::string attempt_error = "net: no address accepted the connection"; + for (addrinfo *candidate = found; candidate != nullptr; + candidate = candidate->ai_next) { + const int descriptor = static_cast(::socket( + candidate->ai_family, candidate->ai_socktype, candidate->ai_protocol)); + if (descriptor < 0) + continue; + if (::connect(descriptor, candidate->ai_addr, + static_cast(candidate->ai_addrlen)) == 0) { + ::freeaddrinfo(found); + opened.descriptor = descriptor; + return opened; + } + attempt_error = last_error_text(); + close_descriptor(descriptor); + } + ::freeaddrinfo(found); + opened.error = attempt_error; + return opened; +} + +/** + * Read what has arrived, up to a ceiling. + * + * A short read is normal and is not an error: TCP is a stream and a request + * may arrive in pieces. `closed` is how the orderly end of a stream is told + * apart from a failure, because a caller reading until the peer finishes needs + * to know which happened and both return no data. + */ +inline Transferred read_some(int descriptor, std::int64_t wanted) { + Transferred result; + if (!ensure_started(result.error)) + return result; + if (wanted < 1) { + result.error = "net: a read must ask for at least one byte"; + return result; + } + if (wanted > max_transfer_bytes) + wanted = max_transfer_bytes; + + std::string buffer(static_cast(wanted), '\0'); + for (;;) { + const transfer_size taken = + ::recv(descriptor, buffer.data(), static_cast(wanted), 0); + if (taken > 0) { + buffer.resize(static_cast(taken)); + result.data = std::move(buffer); + result.count = static_cast(taken); + return result; + } + if (taken == 0) { + result.closed = true; + return result; + } +#if !defined(_WIN32) + if (errno == EINTR) + continue; +#endif + result.error = last_error_text(); + return result; + } +} + +/** + * Write all of it, or say how far it got. + * + * Looping is the caller's job in C and should not be: a single `send` is + * permitted to take less than it was given, and the number of servers that + * have shipped a truncated response because nobody checked is large. + */ +inline Transferred write_all(int descriptor, const std::string &data) { + Transferred result; + if (!ensure_started(result.error)) + return result; + + std::size_t sent = 0; + while (sent < data.size()) { + const std::size_t remaining = data.size() - sent; + const transfer_size taken = + ::send(descriptor, data.data() + sent, + static_cast(remaining), no_signal_flag); + if (taken > 0) { + sent += static_cast(taken); + continue; + } +#if !defined(_WIN32) + if (taken < 0 && errno == EINTR) + continue; +#endif + result.count = static_cast(sent); + result.error = last_error_text(); + return result; + } + result.count = static_cast(sent); + return result; +} + +/** + * Put a deadline on reads and writes. + * + * A single-threaded server with no timeout is one slow client away from + * serving nobody: a connection that opens and then says nothing holds the + * accept loop forever. Zero removes the deadline again. + */ +inline std::string set_timeout(int descriptor, std::int64_t milliseconds) { + std::string error; + if (!ensure_started(error)) + return error; + if (milliseconds < 0) + return "net: a timeout cannot be negative"; + +#if defined(_WIN32) + const DWORD value = static_cast(milliseconds); + if (::setsockopt(descriptor, SOL_SOCKET, SO_RCVTIMEO, + reinterpret_cast(&value), sizeof(value)) != 0 || + ::setsockopt(descriptor, SOL_SOCKET, SO_SNDTIMEO, + reinterpret_cast(&value), sizeof(value)) != 0) + return last_error_text(); +#else + timeval value{}; + value.tv_sec = static_cast(milliseconds / 1000); + value.tv_usec = static_cast((milliseconds % 1000) * 1000); + if (::setsockopt(descriptor, SOL_SOCKET, SO_RCVTIMEO, &value, sizeof(value)) != 0 || + ::setsockopt(descriptor, SOL_SOCKET, SO_SNDTIMEO, &value, sizeof(value)) != 0) + return last_error_text(); +#endif + return {}; +} + +/** The peer's address as text, for a log line. Never part of a decision. */ +inline Transferred peer_address(int descriptor) { + Transferred result; + if (!ensure_started(result.error)) + return result; + + sockaddr_storage address{}; +#if defined(_WIN32) + int size = sizeof(address); +#else + socklen_t size = sizeof(address); +#endif + if (::getpeername(descriptor, reinterpret_cast(&address), &size) != 0) { + result.error = last_error_text(); + return result; + } + + char host[INET6_ADDRSTRLEN] = {0}; + if (address.ss_family == AF_INET) { + const auto *four = reinterpret_cast(&address); + if (::inet_ntop(AF_INET, &four->sin_addr, host, sizeof(host)) == nullptr) { + result.error = "net: the peer address could not be read"; + return result; + } + result.data = std::string(host) + ":" + std::to_string(ntohs(four->sin_port)); + return result; + } + if (address.ss_family == AF_INET6) { + const auto *six = reinterpret_cast(&address); + if (::inet_ntop(AF_INET6, &six->sin6_addr, host, sizeof(host)) == nullptr) { + result.error = "net: the peer address could not be read"; + return result; + } + result.data = "[" + std::string(host) + "]:" + std::to_string(ntohs(six->sin6_port)); + return result; + } + result.error = "net: the peer uses an address family this build does not read"; + return result; +} + +/** Stop a write-blocked peer from keeping a response in a buffer. */ +inline std::string set_no_delay(int descriptor, bool enabled) { + std::string error; + if (!ensure_started(error)) + return error; + const int value = enabled ? 1 : 0; + if (::setsockopt(descriptor, IPPROTO_TCP, TCP_NODELAY, + reinterpret_cast(&value), sizeof(value)) != 0) + return last_error_text(); + return {}; +} + +} // namespace ny::net_support +#endif diff --git a/nilcompiler/src/vm.cpp b/nilcompiler/src/vm.cpp index a158a019..7772a0c3 100644 --- a/nilcompiler/src/vm.cpp +++ b/nilcompiler/src/vm.cpp @@ -13,6 +13,8 @@ #include "numeric_literal.hpp" #include "scalar_text.hpp" #include "integer_conversion.hpp" +#include "entropy_support.hpp" +#include "net_support.hpp" #include "secure_file_io.hpp" #include "time_support.hpp" #include "../../common/windows_environment.hpp" @@ -746,6 +748,22 @@ std::ostream &VM::outputStream() { return output_stream_ ? *output_stream_ : std::cout; } +/** + * Closes every socket a run still holds. + * + * A NIL program is not required to close its listener before it exits, and a + * program that crashes certainly will not. Leaving the descriptor open would + * leave the port bound against the next run, which during development means + * every second start of a server fails for a reason that has nothing to do + * with the code just changed. + */ +void VM::closeOpenSockets() { + for (auto &entry : socket_handles_) + net_support::close_descriptor(entry.second.descriptor); + socket_handles_.clear(); + next_socket_handle_ = 1; +} + void VM::resetRuntimeState(const std::string &root_function) { // Configuration belongs to the host and survives a restart. Everything that // can be observed or owned by the Ny program belongs to one invocation. @@ -757,6 +775,7 @@ void VM::resetRuntimeState(const std::string &root_function) { databases_.clear(); file_handles_.clear(); next_file_handle_ = 1; + closeOpenSockets(); stack_.clear(); globals_.clear(); gc_marked_structs_.clear(); @@ -1093,6 +1112,8 @@ void VM::validateCheckpointAdmission() const { throw std::runtime_error("VM checkpoints do not support memory profiling"); if (!file_handles_.empty()) throw std::runtime_error("VM checkpoints do not support open file handles"); + if (!socket_handles_.empty()) + throw std::runtime_error("VM checkpoints do not support open sockets"); for (const auto &instruction : program_) { switch (instruction.op) { case Opcode::INTEROP_CALL: @@ -3751,6 +3772,388 @@ void VM::executeInstruction(const Instruction &instr) { } } + // --- Network intrinsics --- + // + // The standard library has described sockets and an HTTP server for a + // while with nothing underneath them. This is that floor. Each call gives + // back a Result, because every one of these fails for reasons outside the + // program and a server has to answer all of them on an ordinary day. + // + // A program never sees a descriptor. It is handed a token into + // socket_handles_, which is what stops it reading from a descriptor the + // host opened and what lets the runtime close whatever is still open when + // the run ends. + if (fn_name.rfind("net.", 0) == 0) { + auto push_ok_text = [&](const std::string &text) { + EnumValue ok; ok.variant_name = "Result::Ok"; ok.payloads.push_back(text); + pushStack(ok); + }; + auto push_ok_number = [&](std::int64_t number) { + EnumValue ok; ok.variant_name = "Result::Ok"; ok.payloads.push_back(number); + pushStack(ok); + }; + auto push_error = [&](const std::string &message) { + EnumValue bad; bad.variant_name = "Result::Error"; bad.payloads.push_back(message); + pushStack(bad); + }; + auto text_at = [&](std::size_t index, std::string &into) -> bool { + if (index >= args.size() || !std::holds_alternative(args[index])) + return false; + into = std::get(args[index]); + return true; + }; + auto number_at = [&](std::size_t index, std::int64_t &into) -> bool { + if (index >= args.size()) return false; + if (const auto *wide = std::get_if(&args[index])) { into = *wide; return true; } + return false; + }; + // Resolves a token to a live socket. Every failure here is the program + // using a token it was never given or has already closed, so the + // message says that rather than reporting a system error it did not + // cause. + auto socket_at = [&](std::size_t index, VM::SocketHandle *&into, + std::string &complaint) -> bool { + std::int64_t token = 0; + if (!number_at(index, token)) { + complaint = fn_name + ": expected a socket handle"; + return false; + } + auto found = socket_handles_.find(token); + if (found == socket_handles_.end()) { + complaint = fn_name + ": that socket is not open"; + return false; + } + into = &found->second; + return true; + }; + + if (fn_name == "net.listen") { + std::string host; + std::int64_t port = 0; + std::int64_t backlog = 0; + if (!text_at(0, host) || !number_at(1, port) || !number_at(2, backlog)) { + push_error("net.listen: expected a host, a port and a backlog"); + return; + } + const auto opened = net_support::listen_on(host, port, backlog); + if (!opened.ok()) { + push_error(opened.error); + return; + } + const std::int64_t token = next_socket_handle_++; + socket_handles_.emplace(token, VM::SocketHandle{opened.descriptor, true}); + push_ok_number(token); + return; + } + + if (fn_name == "net.accept") { + VM::SocketHandle *held = nullptr; + std::string complaint; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + if (!held->listening) { + push_error("net.accept: that socket is a connection, not a listener"); + return; + } + const auto opened = net_support::accept_on(held->descriptor); + if (!opened.ok()) { + push_error(opened.error); + return; + } + const std::int64_t token = next_socket_handle_++; + socket_handles_.emplace(token, VM::SocketHandle{opened.descriptor, false}); + push_ok_number(token); + return; + } + + if (fn_name == "net.connect") { + std::string host; + std::int64_t port = 0; + if (!text_at(0, host) || !number_at(1, port)) { + push_error("net.connect: expected a host and a port"); + return; + } + const auto opened = net_support::connect_to(host, port); + if (!opened.ok()) { + push_error(opened.error); + return; + } + const std::int64_t token = next_socket_handle_++; + socket_handles_.emplace(token, VM::SocketHandle{opened.descriptor, false}); + push_ok_number(token); + return; + } + + // Reads what has arrived, which may be less than asked for. An orderly + // close comes back as empty text and not as an error, so a caller + // reading to the end of a request sees the same shape either way; + // net.is_open tells them which it was when it matters. + if (fn_name == "net.read") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::int64_t wanted = 0; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + if (!number_at(1, wanted)) { + push_error("net.read: expected a socket handle and a byte count"); + return; + } + if (held->listening) { + push_error("net.read: that socket is a listener, not a connection"); + return; + } + auto taken = net_support::read_some(held->descriptor, wanted); + if (!taken.ok()) { + push_error(taken.error); + return; + } + push_ok_text(taken.data); + return; + } + + if (fn_name == "net.write") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::string data; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + if (!text_at(1, data)) { + push_error("net.write: expected a socket handle and text to send"); + return; + } + if (held->listening) { + push_error("net.write: that socket is a listener, not a connection"); + return; + } + auto sent = net_support::write_all(held->descriptor, data); + if (!sent.ok()) { + push_error(sent.error); + return; + } + push_ok_number(sent.count); + return; + } + + if (fn_name == "net.close") { + std::int64_t token = 0; + if (!number_at(0, token)) { + push_error("net.close: expected a socket handle"); + return; + } + auto found = socket_handles_.find(token); + if (found == socket_handles_.end()) { + // Closing something already closed is not worth a failure: it is + // what a cleanup path does when an earlier one already ran. + push_ok_number(0); + return; + } + net_support::close_descriptor(found->second.descriptor); + socket_handles_.erase(found); + push_ok_number(1); + return; + } + + if (fn_name == "net.set_timeout") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::int64_t milliseconds = 0; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + if (!number_at(1, milliseconds)) { + push_error("net.set_timeout: expected a socket handle and milliseconds"); + return; + } + const auto error = net_support::set_timeout(held->descriptor, milliseconds); + if (!error.empty()) { + push_error(error); + return; + } + push_ok_number(milliseconds); + return; + } + + if (fn_name == "net.set_no_delay") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::int64_t enabled = 1; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + number_at(1, enabled); + const auto error = net_support::set_no_delay(held->descriptor, enabled != 0); + if (!error.empty()) { + push_error(error); + return; + } + push_ok_number(enabled != 0 ? 1 : 0); + return; + } + + if (fn_name == "net.peer_address") { + VM::SocketHandle *held = nullptr; + std::string complaint; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + auto found = net_support::peer_address(held->descriptor); + if (!found.ok()) { + push_error(found.error); + return; + } + push_ok_text(found.data); + return; + } + + // Whether the runtime still holds this token. Not whether the peer is + // reachable -- nothing short of a write can tell you that. + if (fn_name == "net.is_open") { + std::int64_t token = 0; + if (!number_at(0, token)) { + pushStack((int64_t)0); + return; + } + pushStack((int64_t)(socket_handles_.count(token) != 0 ? 1 : 0)); + return; + } + + throw std::runtime_error("unsupported network intrinsic: " + fn_name); + } + + // --- Entropy intrinsics --- + // + // The only unpredictable source in the standard library. nil.std.math is + // MINSTD and says itself that it is not cryptographic, so anything + // drawing a credential has to come here. There is no seeding and no + // state: these ask the kernel, and if the kernel cannot answer they fail + // rather than returning something a caller might mistake for a secret. + if (fn_name.rfind("random.", 0) == 0) { + auto push_ok_text = [&](const std::string &text) { + EnumValue ok; ok.variant_name = "Result::Ok"; ok.payloads.push_back(text); + pushStack(ok); + }; + auto push_ok_number = [&](std::int64_t number) { + EnumValue ok; ok.variant_name = "Result::Ok"; ok.payloads.push_back(number); + pushStack(ok); + }; + auto push_error = [&](const std::string &message) { + EnumValue bad; bad.variant_name = "Result::Error"; bad.payloads.push_back(message); + pushStack(bad); + }; + auto number_at = [&](std::size_t index, std::int64_t &into) -> bool { + if (index >= args.size()) return false; + if (const auto *wide = std::get_if(&args[index])) { into = *wide; return true; } + return false; + }; + + if (fn_name == "random.hex") { + std::int64_t count = 0; + if (!number_at(0, count)) { + push_error("random.hex: expected a byte count"); + return; + } + auto drawn = entropy_support::random_hex(count); + if (!drawn.ok()) { + push_error(drawn.error); + return; + } + push_ok_text(drawn.bytes); + return; + } + + if (fn_name == "random.bytes") { + std::int64_t count = 0; + if (!number_at(0, count)) { + push_error("random.bytes: expected a byte count"); + return; + } + auto drawn = entropy_support::random_bytes(count); + if (!drawn.ok()) { + push_error(drawn.error); + return; + } + // As text, so a byte string crosses into Matrix, which cannot index + // an Ny byte array. The bytes are arbitrary and are not valid UTF-8 + // in general: read them with text.byte_at, not with substring. + push_ok_text(drawn.bytes); + return; + } + + if (fn_name == "random.below") { + std::int64_t bound = 0; + if (!number_at(0, bound)) { + push_error("random.below: expected an upper bound"); + return; + } + auto chosen = entropy_support::random_below(bound); + if (!chosen.ok()) { + push_error(chosen.error); + return; + } + push_ok_number(chosen.value); + return; + } + + throw std::runtime_error("unsupported entropy intrinsic: " + fn_name); + } + + // --- Byte access to text --- + // + // nil.std.text measures and slices by byte but had no way to read a byte + // as a number, and the one conversion that looked like it would -- + // convert.char_to_i32 -- takes a whole Unicode scalar and raises on a + // partial one, which no `attempt` can catch. So a byte in the middle of a + // multi-byte character could not be inspected, and the character could + // not be isolated to inspect whole, because isolating it needed the same + // test. Percent-encoding anything non-ASCII was unreachable because of it. + // + // These two are deliberately about bytes and say so in their names. They + // do not know what UTF-8 is: that is the caller's to assemble, and now it + // can be. + if (fn_name == "text.byte_at") { + if (args.size() != 2 || !std::holds_alternative(args[0])) { + pushStack((int64_t)-1); + return; + } + const std::string &subject = std::get(args[0]); + std::int64_t index = -1; + if (const auto *wide = std::get_if(&args[1])) index = *wide; + // Out of range is minus one rather than a raise: reading past the end + // is how a scan over bytes discovers it has finished, and a loop that + // has to check the length first is a loop that will sometimes forget. + if (index < 0 || static_cast(index) >= subject.size()) { + pushStack((int64_t)-1); + return; + } + pushStack((int64_t)static_cast( + subject[static_cast(index)])); + return; + } + + if (fn_name == "text.from_byte") { + std::int64_t code = -1; + if (!args.empty()) { + if (const auto *wide = std::get_if(&args[0])) code = *wide; + } + if (code < 0 || code > 255) { + pushStack(std::string()); + return; + } + // One byte, not one character. 195 gives the single byte 0xC3, where + // convert.i32_to_char would give the two bytes that encode U+00C3. + pushStack(std::string(1, static_cast(static_cast(code)))); + return; + } + // --- Path Intrinsics --- if (fn_name == "path.combine") { if (args.size() == 2) { diff --git a/nilcompiler/src/vm.hpp b/nilcompiler/src/vm.hpp index cf4e2126..31a6cccc 100644 --- a/nilcompiler/src/vm.hpp +++ b/nilcompiler/src/vm.hpp @@ -612,6 +612,22 @@ class VM { std::unordered_map file_handles_; int64_t next_file_handle_ = 1; + // Sockets follow the same rule as streams: a NIL program is given a token, + // never a host descriptor, so it cannot read from or close something it was + // not handed -- including a descriptor the host opened for its own purposes. + // `listening` is kept so that accepting on a connection, or reading from a + // listener, is an error with a sentence in it rather than an EINVAL. + struct SocketHandle { + int descriptor = -1; + bool listening = false; + }; + std::unordered_map socket_handles_; + int64_t next_socket_handle_ = 1; + + // Closes every socket a run still holds. A program that exits without + // closing its listener must not leave the port bound against the next one. + void closeOpenSockets(); + // Type size helper (simplified - all types are 8 bytes for now) size_t get_type_size(const std::string &type_name) const { return 8; // All types treated as 64-bit for simplicity diff --git a/nilstd/src/net/package.ny b/nilstd/src/net/package.ny index ae2ec99e..7a3c9d62 100644 --- a/nilstd/src/net/package.ny +++ b/nilstd/src/net/package.ny @@ -1,9 +1,14 @@ @pragma once -# ny.lang.net currently guarantees transport-independent HTTP protocol values -# and request/response construction. The HTTP client, HTTP server, NyWeb, -# sockets, WebSocket, IMAP, and SMTP designs remain under experimental/ until -# a built provider and route-level integration tests ship with the SDK. +# nil.std.net provides transport-independent HTTP protocol values and +# request/response construction (http.ny), and TCP stream sockets backed by +# runtime intrinsics (socket.ny). Sockets are interpreter-backed; the native +# route does not carry them yet. +# +# The HTTP client, HTTP server, WebSocket, IMAP and SMTP designs remain under +# experimental/ until a built provider and route-level integration tests ship +# with the SDK. They now have a transport to be built on. package nil.std.net import nil.std.net.http +import nil.std.net.socket diff --git a/nilstd/src/net/socket.ny b/nilstd/src/net/socket.ny new file mode 100644 index 00000000..0e4138b7 --- /dev/null +++ b/nilstd/src/net/socket.ny @@ -0,0 +1,133 @@ +@schema +@pragma once + +# ============================================================================= +# nil.std.net — stream sockets +# ============================================================================= +# +# TCP, as the runtime provides it. This is the transport nil.std.net.http has +# been describing without one: until these intrinsics existed, no NIL or Matrix +# program could accept a connection, and the socket design under experimental/ +# had nothing beneath it to build on. +# +# Every call returns a Result. A network call fails for reasons entirely +# outside the program — the port is taken, the peer went away, the name does +# not resolve — and for a server those are ordinary answers on an ordinary +# day, not exceptional conditions. None of these raise. +# +# A program is handed an opaque handle, never a host descriptor. It cannot +# read from or close something it was not given, and the runtime closes +# whatever is still open when the run ends, so a program that exits without +# closing its listener does not leave the port bound against the next one. +# +# Interpreter-backed. The native route does not carry these yet. + +module nil.std.net + +import nil.std.core + +# ----------------------------------------------------------------------------- +# Listening and accepting +# ----------------------------------------------------------------------------- + +# Open a socket, bind it and listen, in one step. +# +# `host` is numeric, empty, or "localhost". Binding never resolves a name +# through DNS: a server binds an address it already knows, and a lookup on +# that path is both a blocking call and a way to end up listening somewhere +# unintended. Empty means every interface — right behind a firewall or a +# reverse proxy, and wrong on a laptop on a shared network. The caller +# chooses; this does not choose for them. +# +# A backlog below one becomes 128. The kernel clamps the ceiling regardless. +@intrinsic("net.listen") +pub fn listen(host: text, port: i32, backlog: i32) -> Result + +# Wait for a connection and return a handle to it. +# +# Blocks until one arrives. An interrupted call is retried inside the runtime +# rather than reported, so an unrelated signal does not look like a failure. +@intrinsic("net.accept") +pub fn accept(handle: i64) -> Result + +# ----------------------------------------------------------------------------- +# Connecting +# ----------------------------------------------------------------------------- + +# Connect to a host, resolving a name if it is one. +# +# Unlike listen, this resolves: a client is handed names. A host with both an +# IPv4 and an IPv6 address is tried in turn and the first to connect wins. +@intrinsic("net.connect") +pub fn connect(host: text, port: i32) -> Result + +# ----------------------------------------------------------------------------- +# Reading and writing +# ----------------------------------------------------------------------------- + +# Read what has arrived, up to `maximum_bytes`. +# +# A short read is normal and is not an error: TCP is a stream, and a request +# may arrive in several pieces. An orderly close by the peer comes back as +# empty text, so a loop reading to the end of a request sees one shape; use +# is_open when the difference matters. +# +# The bytes are returned as text and are not necessarily valid UTF-8 — they +# are whatever the peer sent. Read them with nil.std.text's byte_at rather +# than with substring if they may not be text. +# +# One call never transfers more than 8 MiB however much is asked for. +@intrinsic("net.read") +pub fn read(handle: i64, maximum_bytes: i64) -> Result + +# Send all of it, and report how many bytes went. +# +# A single send is permitted to take less than it was given; this loops until +# everything is gone or the attempt fails, because the number of servers that +# have shipped a truncated response from the one-call version is large. On +# failure the count says how far it got. +# +# A peer that closes mid-write is reported, not fatal: the runtime suppresses +# the signal that would otherwise end the process. +@intrinsic("net.write") +pub fn write(handle: i64, data: text) -> Result + +# ----------------------------------------------------------------------------- +# Lifetime and options +# ----------------------------------------------------------------------------- + +# Close a socket and release its handle. +# +# Closing something already closed succeeds and reports zero, because that is +# what a cleanup path does when an earlier one has already run. +@intrinsic("net.close") +pub fn close(handle: i64) -> Result + +# Whether the runtime still holds this handle. +# +# Not whether the peer is reachable: nothing short of a write can tell you +# that, and anything claiming otherwise is guessing. +@intrinsic("net.is_open") +pub fn is_open(handle: i64) -> bool + +# Put a deadline on reads and writes, in milliseconds. Zero removes it. +# +# A single-threaded server without this is one slow client away from serving +# nobody: a connection that opens and then says nothing holds the accept loop +# for as long as it likes. +@intrinsic("net.set_timeout") +pub fn set_timeout(handle: i64, milliseconds: i64) -> Result + +# Send small writes immediately instead of waiting to coalesce them. +# +# Worth setting on a connection serving a request/response protocol, where +# the last small write of a response is the one the client is waiting for. +@intrinsic("net.set_no_delay") +pub fn set_no_delay(handle: i64, enabled: bool) -> Result + +# The peer's address and port, as text, for a log line. +# +# For logging and diagnostics. Not for an access decision: an address is not +# an identity, and behind any proxy it is the proxy's. +@intrinsic("net.peer_address") +pub fn peer_address(handle: i64) -> Result diff --git a/nilstd/src/security/package.ny b/nilstd/src/security/package.ny index ee5db676..092debcd 100644 --- a/nilstd/src/security/package.ny +++ b/nilstd/src/security/package.ny @@ -7,10 +7,13 @@ # Provides: # - Base64: portable encode and strict decode (base64.ny) # - Hex: portable lowercase encoding (encoding.ny) +# - Random: cryptographically secure bytes, hex and bounded draws from the +# operating system (random.ny). The only unpredictable source in the +# standard library; nil.std.math is MINSTD and is not one. # -# Unbuilt hash, crypto, and secure-random design declarations are preserved -# under experimental/. Their native shims are not ABI-safe Ny providers and do -# not enter nilstd or nillib. +# Unbuilt hash and crypto design declarations are preserved under +# experimental/. Their native shims are not ABI-safe Ny providers and do not +# enter nilstd or nillib. # # Cross-refs: # ny.lang.core — Result, byte/u8 types @@ -21,3 +24,4 @@ package nil.std.security import nil.std.security.base64 import nil.std.security.encoding +import nil.std.security.random diff --git a/nilstd/src/security/random.ny b/nilstd/src/security/random.ny new file mode 100644 index 00000000..c0ad6574 --- /dev/null +++ b/nilstd/src/security/random.ny @@ -0,0 +1,67 @@ +@schema +@pragma once + +# ============================================================================= +# nil.std.security — cryptographically secure random values +# ============================================================================= +# +# The only unpredictable source in the standard library. +# +# nil.std.math is MINSTD, a 31-bit linear congruential generator whose global +# stream starts at the seed 12345, and it says in its own comments that it is +# not cryptographic. Two draws from it reveal the whole stream. That is the +# right generator for a simulation, a shuffle or a jittered retry, and the +# wrong one for anything a user holds: a session identifier, a CSRF token, a +# password-reset link, a nonce. Those are bearer credentials, and one drawn +# from MINSTD is one anyone can compute. +# +# Until these intrinsics existed there was no alternative — no hash, no HMAC, +# no random here, and /dev/urandom unreadable through the file intrinsics, +# which refuse a character device by design. A library that needed a +# credential had to invent something predictable or drop the feature. +# +# These ask the operating system and do nothing else. No userspace pool, no +# seeding, no state to get wrong, and no fallback to a weak source: if the +# kernel cannot provide entropy the answer is an error, because quietly +# returning something guessable is how this goes wrong in a way nobody +# notices until it matters. +# +# A single draw is limited to 1024 bytes. A key is 32 and a generous token is +# 64; asking for a megabyte is a program that wants a stream cipher seeded +# from here instead. +# +# Interpreter-backed. The native route does not carry these yet. + +module nil.std.security + +import nil.std.core + +# Random bytes as lower-case hex, two characters per byte. +# +# The one to reach for. A credential wants to be text anyway, and hex +# survives a cookie, a URL, a header and a log line with no further escaping. +# It is also the form Matrix can use, since a Matrix page cannot index an Ny +# byte array. +# +# 16 bytes (32 characters) is a reasonable session identifier; 32 bytes is +# generous and still short enough to read. +@intrinsic("random.hex") +pub fn random_hex(byte_count: i64) -> Result + +# Random bytes as they are. +# +# Returned as text because that is what crosses into Matrix, but the content +# is arbitrary bytes and is not valid UTF-8 in general. Read it with +# nil.std.text's byte_at, never with substring or length-as-characters. +# Prefer random_hex unless the raw bytes are what is wanted. +@intrinsic("random.bytes") +pub fn random_bytes(byte_count: i64) -> Result + +# A uniform whole number in [0, bound). +# +# By rejection sampling, not by remainder. Taking a draw modulo the bound +# skews towards the low values whenever the bound does not divide the range +# evenly — a bias small enough never to be noticed and large enough to matter +# to anything that was counting on uniformity. +@intrinsic("random.below") +pub fn random_below(bound: i64) -> Result diff --git a/nilstd/src/text/string_utils.ny b/nilstd/src/text/string_utils.ny index 7a08699f..7c59ee50 100644 --- a/nilstd/src/text/string_utils.ny +++ b/nilstd/src/text/string_utils.ny @@ -785,3 +785,39 @@ end fn pub fn int_to_string_i64(n: i64) -> text return _int64_to_string(n) end fn + +# ============================================================================= +# Byte access +# ============================================================================= +# +# This module measures and slices by byte — length of "café" is 5 — but had +# no way to read a byte as a number. The conversion that looked like it would, +# convert.char_to_i32, takes a whole Unicode scalar and raises on a partial +# one, and that raise is a fatal runtime error rather than a fault `attempt` +# can catch. So a byte in the middle of a multi-byte character could not be +# inspected, and such a character could not be isolated in order to inspect it +# whole, because isolating it needed the same test. Percent-encoding anything +# outside ASCII was unreachable because of this, and so was reading the bytes +# off a socket. +# +# These two are about bytes and say so in their names. They know nothing about +# UTF-8; assembling characters from bytes is the caller's job, and now it is a +# job that can be done. + +# The byte at an index, 0..255, or minus one when the index is out of range. +# +# Out of range answers rather than raising, because reading past the end is +# how a scan over bytes discovers it has finished, and a loop that must check +# the length separately is a loop that will sometimes forget to. +@intrinsic("text.byte_at") +pub fn byte_at(s: text, index: i64) -> i64 + +# One byte as text. Not one character. +# +# from_byte(195) is the single byte 0xC3, where convert.i32_to_char(195) is +# the two bytes that encode U+00C3. Joining text joins bytes, so a multi-byte +# character can be built one byte at a time with this. +# +# A code outside 0..255 gives empty text. +@intrinsic("text.from_byte") +pub fn from_byte(code: i64) -> text diff --git a/tests/matrix/execution/byte-access.expected b/tests/matrix/execution/byte-access.expected new file mode 100644 index 00000000..d3972138 --- /dev/null +++ b/tests/matrix/execution/byte-access.expected @@ -0,0 +1,13 @@ +65 +A +-1 +-1 +2 +195 +169 +2 +1 +1 +255 +0 +0 diff --git a/tests/matrix/execution/byte-access.matrix b/tests/matrix/execution/byte-access.matrix new file mode 100644 index 00000000..126e6ab3 --- /dev/null +++ b/tests/matrix/execution/byte-access.matrix @@ -0,0 +1,52 @@ +!!! Byte access to text. +!!! +!!! nil.std.text slices and measures by byte, and until text.byte_at and +!!! text.from_byte existed there was no way to read a byte as a number or +!!! write one as text. The conversion that looked like it would do it, +!!! convert.char_to_i32, takes a whole Unicode scalar and aborts the process +!!! on a partial one, so a byte inside a multi-byte character was unreachable +!!! and so was percent-encoding anything outside ASCII. +!!! +!!! What this page pins down is that these two are about bytes and not about +!!! characters, which is the whole reason they were added. + +book matrix.byte-access + +read nil.std.core +read nil.std.text.string_utils + +main returns number begin paragraph + !!! An ASCII byte reads as its code and writes back as itself. + say nil.std.text::byte_at with s being "A", index being 0 + say nil.std.text::from_byte with code being 65 + + !!! Out of range answers rather than raising, so a scan can discover the + !!! end by reading past it. + say nil.std.text::byte_at with s being "A", index being 1 + say nil.std.text::byte_at with s being "A", index being 0 minus 1 + + !!! Two bytes built one at a time make one character, because joining text + !!! joins bytes. This is the case convert.i32_to_char cannot express: it + !!! would give the two bytes that encode U+00C3 for a request for 195. + value one-character is (nil.std.text::from_byte with code being 195) + plus (nil.std.text::from_byte with code being 169) + say nil.std.text::length with s being one-character + say nil.std.text::byte_at with s being one-character, index being 0 + say nil.std.text::byte_at with s being one-character, index being 1 + + !!! The same character written as a scalar is the same two bytes, which is + !!! what proves the byte route and the scalar route agree. + value as-scalar is nil.std.core::Convert.i32_to_char with code being 233 + say nil.std.text::length with s being as-scalar + say (as-scalar equals one-character) becomes text + + !!! Every byte value is reachable, including zero and the high half that + !!! is not valid UTF-8 on its own. + say nil.std.text::length with s being (nil.std.text::from_byte with code being 0) + say nil.std.text::byte_at with s being (nil.std.text::from_byte with code being 255), index being 0 + + !!! Outside the byte range there is nothing to give. + say nil.std.text::length with s being (nil.std.text::from_byte with code being 256) + say nil.std.text::length with s being (nil.std.text::from_byte with code being 0 minus 1) + give 0 +end paragraph diff --git a/tests/matrix/execution/byte-access.needs-stdlib b/tests/matrix/execution/byte-access.needs-stdlib new file mode 100644 index 00000000..e69de29b diff --git a/tests/matrix/execution/entropy.expected b/tests/matrix/execution/entropy.expected new file mode 100644 index 00000000..7a0531b2 --- /dev/null +++ b/tests/matrix/execution/entropy.expected @@ -0,0 +1,10 @@ +32 +64 +-1 +-1 +1 +1 +8 +1 +0 +refused diff --git a/tests/matrix/execution/entropy.matrix b/tests/matrix/execution/entropy.matrix new file mode 100644 index 00000000..fe525061 --- /dev/null +++ b/tests/matrix/execution/entropy.matrix @@ -0,0 +1,112 @@ +!!! Cryptographically secure random values. +!!! +!!! nil.std.math is MINSTD, seeded at 12345, and says itself that it is not +!!! cryptographic. Before nil.std.security's random intrinsics there was no +!!! alternative anywhere in the standard library, and /dev/urandom was no way +!!! round it: the file intrinsics refuse a character device by design. Any +!!! library needing a session id, a token or a nonce had to either invent +!!! something predictable or drop the feature. +!!! +!!! Nothing here can assert a particular value, which is the point. What it +!!! asserts is the shape, the bounds, and that two draws differ — the last +!!! being the one check that would have caught MINSTD standing in for this. + +book matrix.entropy + +read nil.std.core +read nil.std.text.string_utils +read nil.std.security.random + +!!! Reports a Result as a width, or minus one if it failed, so the +!!! expected output does not depend on the bytes drawn. +hex-width receives byte-count as number returns number begin paragraph + value drawn is nil.std.security::random_hex with byte_count being byte-count + consider drawn + when Result.Ok holding token then give nil.std.text::length with s being token + otherwise give 0 minus 1 + end consider +end paragraph + +hex-of receives byte-count as number returns text begin paragraph + value drawn is nil.std.security::random_hex with byte_count being byte-count + consider drawn + when Result.Ok holding token then give token + otherwise give "" + end consider +end paragraph + +!!! Whether every character is a lower-case hex digit. A generator that +!!! returned raw bytes through the hex door would fail here. +all-hex receives token as text returns boolean begin paragraph + value width is nil.std.text::length with s being token + when width is less than 1 then + give false + end when + variable index as number is 0 + repeat while index is less than width + value code is nil.std.text::byte_at with s being token, index being index + value is-digit is code is at least 48 and code is at most 57 + value is-lower is code is at least 97 and code is at most 102 + when not (is-digit or is-lower) then + give false + end when + set index to index plus 1 + end repeat + give true +end paragraph + +main returns number begin paragraph + !!! Two hex characters per byte, which is what makes the width predictable + !!! even though the content is not. + say hex-width with byte-count being 16 + say hex-width with byte-count being 32 + + !!! The bounds are refused rather than clamped: a caller asking for zero + !!! bytes of entropy has a bug, and so has one asking for a megabyte. + say hex-width with byte-count being 0 + say hex-width with byte-count being 1025 + + value token is hex-of with byte-count being 16 + say (all-hex with token being token) becomes text + + !!! The check that matters. Two draws from a secure source differ; two + !!! draws from a counter or a seeded LCG at a fixed start do not. + say (not ((hex-of with byte-count being 16) equals (hex-of with byte-count being 16))) becomes text + + !!! Raw bytes come back as text and are read by byte, not by character. + value raw is nil.std.security::random_bytes with byte_count being 8 + consider raw + when Result.Ok holding bytes then say nil.std.text::length with s being bytes + otherwise say 0 minus 1 + end consider + + !!! A bounded draw lands inside its bound, every time. + variable inside as boolean is true + variable round as number is 0 + repeat while round is less than 50 + value picked is nil.std.security::random_below with bound being 10 + consider picked + when Result.Ok holding amount then begin block + when amount is less than 0 or amount is at least 10 then + set inside to false + end when + end block + otherwise set inside to false + end consider + set round to round plus 1 + end repeat + say inside becomes text + + !!! A bound of one has exactly one answer, and a bound below one is a bug. + value only is nil.std.security::random_below with bound being 1 + consider only + when Result.Ok holding amount then say amount + otherwise say 0 minus 1 + end consider + value impossible is nil.std.security::random_below with bound being 0 + consider impossible + when Result.Ok holding amount then say amount + otherwise say "refused" + end consider + give 0 +end paragraph diff --git a/tests/matrix/execution/entropy.needs-stdlib b/tests/matrix/execution/entropy.needs-stdlib new file mode 100644 index 00000000..e69de29b diff --git a/tests/tools/net_socket_contract.py b/tests/tools/net_socket_contract.py new file mode 100644 index 00000000..3a852af6 --- /dev/null +++ b/tests/tools/net_socket_contract.py @@ -0,0 +1,573 @@ +#!/usr/bin/env python3 +"""The network intrinsics, against a real peer. + +A socket cannot be checked by comparing stdout: the interesting behaviour is +what the other end of the connection sees. So each case here runs a Matrix +page as a process and plays the other side from Python. + +What is being pinned down is the contract the standard library promises, not +the implementation: + + * a program can listen, accept, read and write, and what arrives on the wire + is byte-for-byte what the page sent; + * a handle is opaque -- a number the program was never given is refused with + a sentence, not used as a file descriptor; + * a listener and a connection are not interchangeable; + * the runtime closes what a program leaves open, so a port does not stay + bound against the next run; + * a read deadline expires instead of hanging the process forever; and + * a peer that vanishes mid-write is an error the program can see, rather + than a signal that kills it. + +That last one is the reason this file exists rather than a simpler fixture. A +server that dies when a browser closes a tab passes every test that does not +close a connection early. +""" + +from __future__ import annotations + +import argparse +import os +import socket +import subprocess +import sys +import tempfile +import time +from pathlib import Path + + +def free_port() -> int: + """A port the kernel says is free, released before the page binds it. + + There is a race here in principle. In practice the kernel does not hand + the same ephemeral port out twice in the time between these two lines, and + the alternative -- a fixed port -- fails whenever anything else on the + machine happens to be using it, which is worse because it looks like a + code failure. + """ + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe: + probe.bind(("127.0.0.1", 0)) + return int(probe.getsockname()[1]) + + +def build(compiler: Path, stdlib: Path, source: str, workdir: Path, name: str) -> Path: + page = workdir / f"{name}.matrix" + page.write_text(source) + artifact = workdir / f"{name}.nil" + built = subprocess.run( + [str(compiler), "--no-bundle", "-I", str(stdlib), + "--output", str(artifact), str(page)], + capture_output=True, text=True) + if built.returncode != 0: + raise AssertionError( + f"{name} did not compile:\n{built.stdout}{built.stderr}") + return artifact + + +def wait_for_line(process: subprocess.Popen, marker: str, timeout: float = 20.0) -> None: + """Read the page's stdout until it says it is ready.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + line = process.stdout.readline() + if line == "": + raise AssertionError( + f"the page exited before printing {marker!r}") + if marker in line: + return + raise AssertionError(f"the page never printed {marker!r}") + + +ECHO_PAGE = """ +book net.contract.echo + +read nil.std.core +read nil.std.text.string_utils +read nil.std.net.socket + +main returns number begin paragraph + value opened is nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 8 + consider opened + when Result.Error holding why then begin block + say "listen failed: " plus why + give 1 + end block + when Result.Ok holding listener then begin block + say "ready" + value taken is nil.std.net::accept with handle being listener + consider taken + when Result.Ok holding client then begin block + value heard is nil.std.net::read with handle being client, maximum_bytes being 4096 + consider heard + when Result.Ok holding incoming then begin block + evaluate nil.std.net::write with handle being client, data being incoming + say "echoed " plus ((nil.std.text::length with s being incoming) becomes text) + end block + when Result.Error holding why then say "read failed: " plus why + otherwise say "read gave neither" + end consider + evaluate nil.std.net::close with handle being client + end block + when Result.Error holding why then say "accept failed: " plus why + otherwise say "accept gave neither" + end consider + evaluate nil.std.net::close with handle being listener + say "done" + give 0 + end block + otherwise give 1 + end consider +end paragraph +""" + + +HANDLE_PAGE = """ +book net.contract.handles + +read nil.std.core +read nil.std.text.string_utils +read nil.std.net.socket + +!!! Reports a Result as its error text, or "ok" when it succeeded, so the +!!! refusals can be asserted without depending on a system message. Two of +!!! them, because read gives back text and the rest give back a count, and +!!! Matrix has no overload sets. +reason-for receives outcome as Result of number, text returns text begin paragraph + consider outcome + when Result.Ok holding amount then give "ok" + when Result.Error holding why then give why + otherwise give "neither" + end consider +end paragraph + +reason-for-read receives outcome as Result of text, text returns text begin paragraph + consider outcome + when Result.Ok holding incoming then give "ok" + when Result.Error holding why then give why + otherwise give "neither" + end consider +end paragraph + +main returns number begin paragraph + !!! A handle the program was never given. If these were descriptors, 1 + !!! would be this process's standard output. + say reason-for-read with outcome being (nil.std.net::read with handle being 1, maximum_bytes being 10) + say reason-for with outcome being (nil.std.net::write with handle being 1, data being "x") + say (nil.std.net::is_open with handle being 1) becomes text + + !!! Closing something never opened is not a failure: it is what a cleanup + !!! path does when an earlier one already ran. + say reason-for with outcome being (nil.std.net::close with handle being 404) + + value opened is nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 4 + consider opened + when Result.Ok holding listener then begin block + say (nil.std.net::is_open with handle being listener) becomes text + !!! A listener is not a connection and will not pretend to be one. + say reason-for-read with outcome being (nil.std.net::read with handle being listener, maximum_bytes being 4) + say reason-for with outcome being (nil.std.net::write with handle being listener, data being "x") + evaluate nil.std.net::close with handle being listener + say (nil.std.net::is_open with handle being listener) becomes text + !!! And it stays closed. + say reason-for-read with outcome being (nil.std.net::read with handle being listener, maximum_bytes being 4) + end block + when Result.Error holding why then say "listen failed: " plus why + otherwise say "listen gave neither" + end consider + + !!! A bind address is numeric, empty or localhost; a name is refused + !!! rather than resolved, because binding must not do DNS. + say reason-for with outcome being + (nil.std.net::listen with host being "example.com", port being PORT, backlog being 4) + say reason-for with outcome being + (nil.std.net::listen with host being "127.0.0.1", port being 0, backlog being 4) + say reason-for with outcome being + (nil.std.net::listen with host being "127.0.0.1", port being 70000, backlog being 4) + give 0 +end paragraph +""" + + +LEAK_PAGE = """ +book net.contract.leak + +read nil.std.core +read nil.std.net.socket + +!!! Binds and then exits without closing anything. The runtime has to clean +!!! up after it, or the next run of this same page cannot bind the port. +main returns number begin paragraph + value opened is nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 4 + consider opened + when Result.Ok holding listener then say "bound" + when Result.Error holding why then say "listen failed: " plus why + otherwise say "neither" + end consider + give 0 +end paragraph +""" + + +TIMEOUT_PAGE = """ +book net.contract.timeout + +read nil.std.core +read nil.std.net.socket + +!!! A connection that opens and says nothing must not hold the process. With +!!! no deadline this page would wait as long as the client cared to. +main returns number begin paragraph + value opened is nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 4 + consider opened + when Result.Ok holding listener then begin block + say "ready" + value taken is nil.std.net::accept with handle being listener + consider taken + when Result.Ok holding client then begin block + evaluate nil.std.net::set_timeout with handle being client, milliseconds being 300 + value heard is nil.std.net::read with handle being client, maximum_bytes being 16 + consider heard + when Result.Ok holding incoming then say "read returned" + when Result.Error holding why then say "timed out" + otherwise say "neither" + end consider + evaluate nil.std.net::close with handle being client + end block + otherwise say "accept failed" + end consider + evaluate nil.std.net::close with handle being listener + say "done" + give 0 + end block + otherwise give 1 + end consider +end paragraph +""" + + +HANGUP_PAGE = """ +book net.contract.hangup + +read nil.std.core +read nil.std.net.socket + +!!! Writes a great deal to a peer that has already gone. Without the runtime +!!! suppressing SIGPIPE this page does not reach its last line: the default +!!! disposition ends the process, and a server built on it dies whenever a +!!! browser closes a tab. +main returns number begin paragraph + value opened is nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 4 + consider opened + when Result.Ok holding listener then begin block + say "ready" + value taken is nil.std.net::accept with handle being listener + consider taken + when Result.Ok holding client then begin block + variable payload as text is "0123456789" + variable doubling as number is 0 + repeat while doubling is less than 16 + set payload to payload plus payload + set doubling to doubling plus 1 + end repeat + variable attempt-count as number is 0 + variable reported as boolean is false + repeat while attempt-count is less than 20 and not reported + value sent is nil.std.net::write with handle being client, data being payload + consider sent + when Result.Error holding why then set reported to true + otherwise set attempt-count to attempt-count plus 1 + end consider + end repeat + say "write reported: " plus (reported becomes text) + evaluate nil.std.net::close with handle being client + end block + otherwise say "accept failed" + end consider + evaluate nil.std.net::close with handle being listener + say "survived" + give 0 + end block + otherwise give 1 + end consider +end paragraph +""" + + +CLIENT_PAGE = """ +book net.contract.client + +read nil.std.core +read nil.std.text.string_utils +read nil.std.net.socket + +!!! The other direction: connecting out, and resolving a name while doing it. +main returns number begin paragraph + value opened is nil.std.net::connect with host being "localhost", port being PORT + consider opened + when Result.Error holding why then begin block + say "connect failed: " plus why + give 1 + end block + when Result.Ok holding server then begin block + evaluate nil.std.net::set_no_delay with handle being server, enabled being true + value peer is nil.std.net::peer_address with handle being server + consider peer + when Result.Ok holding address then say "peer seen: " + plus ((nil.std.text::contains with s being address, needle being ":") becomes text) + otherwise say "peer unavailable" + end consider + evaluate nil.std.net::write with handle being server, data being "ping" + value heard is nil.std.net::read with handle being server, maximum_bytes being 64 + consider heard + when Result.Ok holding incoming then say "heard " plus incoming + when Result.Error holding why then say "read failed: " plus why + otherwise say "neither" + end consider + !!! The orderly close of a peer is empty text, not an error, which + !!! is how a read loop knows the request is finished. + value after-close is nil.std.net::read with handle being server, maximum_bytes being 64 + consider after-close + when Result.Ok holding incoming then say "closed reads empty: " + plus ((incoming equals "") becomes text) + when Result.Error holding why then say "read after close failed: " plus why + otherwise say "neither" + end consider + evaluate nil.std.net::close with handle being server + say "done" + give 0 + end block + otherwise give 1 + end consider +end paragraph +""" + + +def run_page(interpreter: Path, artifact: Path) -> subprocess.Popen: + return subprocess.Popen( + [str(interpreter), str(artifact)], + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, + bufsize=1) + + +def case_echo(compiler, interpreter, stdlib, workdir) -> list[str]: + """What the page writes is what the wire carries, bytes included.""" + problems: list[str] = [] + port = free_port() + artifact = build(compiler, stdlib, + ECHO_PAGE.replace("PORT", str(port)), workdir, "echo") + page = run_page(interpreter, artifact) + try: + wait_for_line(page, "ready") + # Deliberately not valid UTF-8 and containing a zero byte: a socket + # carries bytes, and text that has been through an encoder somewhere + # would come back changed. + payload = b"ping\x00\xc3\xa9\xff\xfe end" + with socket.create_connection(("127.0.0.1", port), timeout=10) as client: + client.sendall(payload) + client.shutdown(socket.SHUT_WR) + received = b"" + while len(received) < len(payload): + chunk = client.recv(4096) + if not chunk: + break + received += chunk + if received != payload: + problems.append( + f"echo: the wire carried {received!r}, not {payload!r}") + output = page.stdout.read() + if f"echoed {len(payload)}" not in output: + problems.append( + f"echo: the page did not report {len(payload)} bytes read; it said {output!r}") + if "done" not in output: + problems.append(f"echo: the page did not finish; it said {output!r}") + finally: + page.kill() + page.wait(timeout=10) + return problems + + +def case_handles(compiler, interpreter, stdlib, workdir) -> list[str]: + """A handle is a token into the runtime, not a descriptor.""" + port = free_port() + artifact = build(compiler, stdlib, + HANDLE_PAGE.replace("PORT", str(port)), workdir, "handles") + finished = subprocess.run([str(interpreter), str(artifact)], + capture_output=True, text=True, timeout=60) + lines = finished.stdout.splitlines() + expected = [ + ("read on an unknown handle", lambda v: "not open" in v), + ("write on an unknown handle", lambda v: "not open" in v), + ("is_open on an unknown handle", lambda v: v == "0"), + ("close on an unknown handle", lambda v: v == "ok"), + ("is_open on a listener", lambda v: v == "1"), + ("read on a listener", lambda v: "listener" in v), + ("write on a listener", lambda v: "listener" in v), + ("is_open after close", lambda v: v == "0"), + ("read after close", lambda v: "not open" in v), + ("listen on a name", lambda v: "numeric" in v), + ("listen on port zero", lambda v: "between 1 and 65535" in v), + ("listen above the port range", lambda v: "between 1 and 65535" in v), + ] + problems: list[str] = [] + if len(lines) != len(expected): + return [f"handles: expected {len(expected)} lines, got {len(lines)}: " + f"{finished.stdout!r}{finished.stderr!r}"] + for (label, holds), value in zip(expected, lines): + if not holds(value): + problems.append(f"handles: {label} gave {value!r}") + return problems + + +def case_port_released(compiler, interpreter, stdlib, workdir) -> list[str]: + """A page that leaves a listener open must not keep the port.""" + port = free_port() + artifact = build(compiler, stdlib, + LEAK_PAGE.replace("PORT", str(port)), workdir, "leak") + problems: list[str] = [] + for attempt in (1, 2, 3): + finished = subprocess.run([str(interpreter), str(artifact)], + capture_output=True, text=True, timeout=60) + if "bound" not in finished.stdout: + problems.append( + f"port-released: run {attempt} could not bind; it said " + f"{finished.stdout!r}{finished.stderr!r}") + break + return problems + + +def case_timeout(compiler, interpreter, stdlib, workdir) -> list[str]: + """A read deadline expires rather than holding the process.""" + problems: list[str] = [] + port = free_port() + artifact = build(compiler, stdlib, + TIMEOUT_PAGE.replace("PORT", str(port)), workdir, "timeout") + page = run_page(interpreter, artifact) + try: + wait_for_line(page, "ready") + with socket.create_connection(("127.0.0.1", port), timeout=10): + # Connect and say nothing at all. The page's own deadline is what + # has to end this, so hold the connection open across it. + time.sleep(1.2) + page.wait(timeout=20) + output = page.stdout.read() + if "timed out" not in output: + problems.append( + f"timeout: the read did not expire; the page said {output!r}") + except subprocess.TimeoutExpired: + problems.append("timeout: the page never finished, so the deadline did nothing") + finally: + page.kill() + page.wait(timeout=10) + return problems + + +def case_peer_hangup(compiler, interpreter, stdlib, workdir) -> list[str]: + """A peer that vanishes mid-write is reported, not fatal.""" + problems: list[str] = [] + port = free_port() + artifact = build(compiler, stdlib, + HANGUP_PAGE.replace("PORT", str(port)), workdir, "hangup") + page = run_page(interpreter, artifact) + try: + wait_for_line(page, "ready") + client = socket.create_connection(("127.0.0.1", port), timeout=10) + # Reset rather than a graceful close, so the next write fails outright + # instead of being accepted into a buffer nobody will drain. + client.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, + b"\x01\x00\x00\x00\x00\x00\x00\x00") + client.close() + page.wait(timeout=30) + output = page.stdout.read() + if "survived" not in output: + problems.append( + "peer-hangup: the page did not reach its last line, so a lost " + f"peer ended the process; it said {output!r}") + if page.returncode != 0: + problems.append( + f"peer-hangup: the page exited {page.returncode}, which is how " + "an unsuppressed SIGPIPE looks") + except subprocess.TimeoutExpired: + problems.append("peer-hangup: the page never finished") + finally: + page.kill() + page.wait(timeout=10) + return problems + + +def case_client(compiler, interpreter, stdlib, workdir) -> list[str]: + """Connecting out, resolving a name, and reading a closed peer.""" + problems: list[str] = [] + listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + listener.bind(("127.0.0.1", 0)) + listener.listen(4) + port = int(listener.getsockname()[1]) + artifact = build(compiler, stdlib, + CLIENT_PAGE.replace("PORT", str(port)), workdir, "client") + page = run_page(interpreter, artifact) + try: + listener.settimeout(20) + connection, _ = listener.accept() + with connection: + greeting = connection.recv(64) + if greeting != b"ping": + problems.append(f"client: the page sent {greeting!r}, not b'ping'") + connection.sendall(b"pong") + page.wait(timeout=30) + output = page.stdout.read() + for marker in ("peer seen: 1", "heard pong", "closed reads empty: 1", "done"): + if marker not in output: + problems.append(f"client: expected {marker!r} in {output!r}") + except (socket.timeout, subprocess.TimeoutExpired): + problems.append("client: the page did not complete its exchange") + finally: + listener.close() + page.kill() + page.wait(timeout=10) + return problems + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--compiler", type=Path, required=True) + parser.add_argument("--interpreter", type=Path, required=True) + parser.add_argument("--root", type=Path, required=True) + args = parser.parse_args() + + compiler = args.compiler.resolve(strict=True) + interpreter = args.interpreter.resolve(strict=True) + stdlib = (args.root / "nilstd" / "src").resolve(strict=True) + + cases = ( + ("echo", case_echo), + ("handles", case_handles), + ("port-released", case_port_released), + ("timeout", case_timeout), + ("peer-hangup", case_peer_hangup), + ("client", case_client), + ) + + problems: list[str] = [] + with tempfile.TemporaryDirectory() as raw: + workdir = Path(raw) + for name, run in cases: + try: + problems.extend(run(compiler, interpreter, stdlib, workdir)) + except Exception as failure: # noqa: BLE001 - report, never abort the run + problems.append(f"{name}: raised {failure!r}") + + if problems: + print("Network intrinsic contract failures:") + for problem in problems: + print(f" - {problem}") + return 1 + + print(f"Network intrinsics: {len(cases)} contract case(s) held — " + "listen/accept/read/write carry bytes unchanged, handles are opaque, " + "a listener is not a connection, the runtime releases a leaked port, " + "a read deadline expires, a lost peer is reported rather than fatal, " + "and connecting out resolves a name.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 15cb0465efaf08c9f7b1c1b1a777258152f27dbf Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 15:02:00 -0500 Subject: [PATCH 02/39] random_bytes is no longer an unbacked declaration ny.routes.reject-unbacked-io-security asserts that a declaration with no runtime provider is rejected at compile time rather than accepted and then failing where it is used. random_bytes was on that list, and the entropy intrinsic took it off: it now type-checks, so the fixture's expected diagnostics no longer matched and the test failed in both CI configurations. Removed from the fixture rather than left failing, because what the test guards is the gap between a declaration and a provider, and for that call there is no longer a gap. Compression, hashing and encryption are still designs under experimental/ with nothing behind them and stay. A comment in the fixture records where random_bytes went, and tests/matrix/execution/entropy.matrix is the coverage that replaces it. Co-Authored-By: Claude Opus 5 --- tests/language/reject-unbacked-io-security.ny | 11 ++++++++++- tests/language/reject-unbacked-io-security.stderr | 7 +++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/tests/language/reject-unbacked-io-security.ny b/tests/language/reject-unbacked-io-security.ny index 4cf2d6d6..0a02becb 100644 --- a/tests/language/reject-unbacked-io-security.ny +++ b/tests/language/reject-unbacked-io-security.ny @@ -3,11 +3,20 @@ module ny.tests.reject_unbacked_io_security import ny.lang.io.* import ny.lang.security.* +# A declaration with no runtime provider must be rejected at compile time +# rather than accepted and then failing at the point of use. Compression, +# hashing and encryption are still designs under experimental/ with no +# intrinsic behind them, so each of these is undefined. +# +# `random_bytes` was on this list until nil.std.security.random gained an +# entropy intrinsic. It is now backed, and the fixture that covers it is +# tests/matrix/execution/entropy.matrix. It was removed from here rather than +# left to fail, because the thing this test guards is the gap between a +# declaration and a provider, and for that call there is no longer a gap. fn main() -> i32 let bytes: vec = vec.new() gzip_compress(bytes) sha256("data") - random_bytes(16) aes_encrypt(bytes, bytes) return 0 end fn diff --git a/tests/language/reject-unbacked-io-security.stderr b/tests/language/reject-unbacked-io-security.stderr index 938258ce..fb6a16d1 100644 --- a/tests/language/reject-unbacked-io-security.stderr +++ b/tests/language/reject-unbacked-io-security.stderr @@ -1,5 +1,4 @@ Type checking failed: - {{SOURCE}}:8: Undefined identifier 'gzip_compress' - {{SOURCE}}:9: Undefined identifier 'sha256' - {{SOURCE}}:10: Undefined identifier 'random_bytes' - {{SOURCE}}:11: Undefined identifier 'aes_encrypt' + {{SOURCE}}:18: Undefined identifier 'gzip_compress' + {{SOURCE}}:19: Undefined identifier 'sha256' + {{SOURCE}}:20: Undefined identifier 'aes_encrypt' From 0f64cc896a84ec469038319277416b29e5eee456 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 15:08:18 -0500 Subject: [PATCH 03/39] Non-blocking sockets, poll, and SO_REUSEPORT The sockets in the previous commit could serve one connection at a time, which is a development server and not a web server. This is what it takes to serve many. An event loop, not a thread per connection, because a thread is not available and will not be soon: the interpreter holds one stack, one heap and one set of globals, and there is not a mutex, an atomic or a thread_local anywhere in it. nil.std.threading offers sleep. Making the VM thread-safe is a far larger piece of work than this, and an event loop does not need it. So: set_blocking, watch, unwatch, watch_count, wait and next_ready, over poll(2). The watch set lives in the runtime rather than being passed in per call, which is not tidiness -- a Matrix page cannot index an Ny generic, so an array-taking interface would be an event loop only half the language could write. next_ready packs the handle and the flags into one number because an intrinsic returns one value, and two calls whose meaning depended on being made in order would be worse than arithmetic the caller can see; ready_handle, ready_flags, is_readable, is_writable and is_finished are ordinary Ny functions over the packing. The readiness flags are deliberately not poll(2)'s numbering, which differs between platforms: a program that learned the host's values would break when it moved. A non-blocking accept with nothing pending now answers zero rather than reporting an error. Handles start at one so it cannot be mistaken for one, and for an event loop it is the ordinary case -- poll said the listener was readable and the connection was withdrawn in between. Reporting it would mean a log line every time a port scanner connected and left. set_reuse_port is the other half of using more than one core. A second core means a second process here, and two processes cannot share a listening socket without it. Opt-in and distinct from the SO_REUSEADDR that listen always sets: that one only allows rebinding a port in TIME_WAIT, while this one allows two servers to share a port, and switched on by accident it would silently split traffic between a running server and a stray one. A wait replaces whatever the previous one found, rather than accumulating, so a program that did not drain the set cannot later be handed a handle that has since been closed and its number reissued. A socket closed since the last wait is dropped from the set rather than polled on a descriptor that now means something else, so a loop that forgets to unwatch a connection it closed is untidy rather than broken. An interrupted poll is retried, because an unrelated signal must not look to a server like a broken socket. Two new contract cases, bringing net.socket-contract to eight. The first covers the watch set: an empty one does not wait, watching twice replaces rather than adds, a drained ready set reads zero, unwatching something unwatched succeeds, a closed socket leaves the set, and the packing round-trips. The second is the one worth reading. Sixty-four clients connect at once and each sends its request in two parts with a quarter-second pause between. A server that serves one connection at a time has to wait out the first client's pause before it touches the second, so the run costs sixty-four pauses; an event loop overlaps them and costs about one. The test asserts that difference and that the server held all sixty-four at once, rather than asserting a throughput number -- a number would measure the machine, and this measures the shape. Locally: 200 clients, all answered, 0.31s against the 50s a serial server would need. docs/standard-library/sockets.md gains the loop's shape, the three things it has to get right (drain the listener; leave connections blocking with a timeout, because EAGAIN and a clean close are both no bytes and only a blocking read tells them apart; readable and finished can both be true), and what is still missing. The claim that there was no way to wait on several sockets is gone, since it was the thing this commit removes. Co-Authored-By: Claude Opus 5 --- docs/standard-library/sockets.md | 91 +++++++- nilcompiler/src/net_support.hpp | 194 +++++++++++++++++ nilcompiler/src/vm.cpp | 162 +++++++++++++++ nilcompiler/src/vm.hpp | 15 ++ nilstd/src/net/socket.ny | 120 +++++++++++ tests/tools/net_socket_contract.py | 323 ++++++++++++++++++++++++++++- 6 files changed, 897 insertions(+), 8 deletions(-) diff --git a/docs/standard-library/sockets.md b/docs/standard-library/sockets.md index 464d6d28..ef984d8b 100644 --- a/docs/standard-library/sockets.md +++ b/docs/standard-library/sockets.md @@ -18,6 +18,14 @@ compiled through it will not find them. | `set_timeout(handle: i64, milliseconds: i64) -> Result` | Deadline for reads and writes; zero removes it. | | `set_no_delay(handle: i64, enabled: bool) -> Result` | Send small writes immediately. | | `peer_address(handle: i64) -> Result` | The peer's address and port, as text. | +| `set_blocking(handle: i64, blocking: bool) -> Result` | Whether a call waits when there is nothing to do. | +| `set_reuse_port(handle: i64, enabled: bool) -> Result` | Let other processes bind this same port. | +| `watch(handle: i64, flags: i64) -> Result` | Watch a socket for readability, writability, or both. | +| `unwatch(handle: i64) -> Result` | Stop watching. | +| `watch_count() -> i64` | How many sockets are watched. | +| `wait(timeout_milliseconds: i64) -> Result` | Wait for readiness; gives back how many are ready. | +| `next_ready() -> Result` | One ready socket per call, packed; zero when drained. | +| `ready_handle` / `ready_flags` / `is_readable` / `is_writable` / `is_finished` | Unpack what `next_ready` gave. | Matrix uses the same names; `maximum_bytes` is spelled `maximum_bytes` because it is an Ny declaration and Matrix keeps a foreign name's spelling. @@ -125,16 +133,85 @@ main returns number begin paragraph end paragraph ``` +## Serving many connections at once + +An event loop, not a thread per connection. This runtime is single-threaded by +construction — the interpreter holds one stack, one heap and one set of +globals, with no lock anywhere near them, and `nil.std.threading` offers only +`sleep` — so concurrency here is `poll` telling you which sockets can be acted +on, and one pass round a loop acting on each in turn. + +That is nginx's shape rather than Apache's prefork, and it is the one this +runtime can keep. It is also the one that scales: 64 simultaneous connections, +each sending its request in two parts with a quarter-second pause between, are +served in about one pause rather than sixty-four of them. The contract test +asserts exactly that difference, because a throughput figure would measure the +machine while this measures the shape. + +The watch set lives in the runtime rather than being passed in per call, so no +array crosses the intrinsic boundary. That is not tidiness: a Matrix page +cannot index an Ny generic, and an event loop that could only be written in Ny +would be one that half the language could not use. + +### The shape of the loop + +```matrix +evaluate nil.std.net::set_blocking with handle being listener, blocking being false +evaluate nil.std.net::watch with handle being listener, flags being nil.std.net::ready_readable + +repeat forever + evaluate nil.std.net::wait with timeout_milliseconds being 10000 + variable draining as boolean is true + repeat while draining + value packed is nil.std.net::next_ready with nothing + !!! zero means the ready set is drained; otherwise unpack and act + end repeat +end repeat +``` + +`next_ready` packs the handle and the flags into one number — `handle * 8 + +flags` — because an intrinsic returns one value, and two calls whose meaning +depended on being made in the right order would be worse than arithmetic the +caller can see. `ready_handle`, `ready_flags`, `is_readable`, `is_writable` and +`is_finished` are ordinary Ny functions over it, so nobody has to write the +division. + +### Three things the loop has to get right + +**Drain the listener.** `poll` reports a listener readable once however many +connections are waiting, so accept until it answers zero. A loop that accepts +one per wait falls behind under load and never catches up. + +**Leave accepted connections blocking, and give them a timeout.** `poll` has +said the connection is readable, so a read returns at once — and an empty read +then means the peer closed rather than "nothing yet", which is a distinction +an event loop has to be able to make. A non-blocking read cannot make it: +EAGAIN and a clean close both produce no bytes. + +**`is_readable` and `is_finished` can both be true.** A peer that hung up may +still have sent something, and that something has to be read before the +connection is dropped, or the last request of a session is lost. + +### More than one core + +A second core means a second process, because a second thread would need an +interpreter that could survive one. `set_reuse_port` lets several processes +bind the same port and the kernel load-balances accepts between them. + +It is opt-in and distinct from the `SO_REUSEADDR` that `listen` always sets: +that one only allows rebinding a port left in `TIME_WAIT`, while this one +deliberately allows two servers to share a port. Switched on by accident it +would silently split traffic between a running server and a stray one. + ## What is not here -One connection at a time. There are no threads to hand a connection to and no -async in the language, so a server built on this accepts, serves and closes in -order. That is enough for a development server, a local tool, or anything -behind a reverse proxy that holds the slow clients. +No TLS. No datagrams. No `sendfile`, so a file goes out through the program +rather than straight from the page cache. -No TLS. No datagrams. No `select` or `poll`, so a program cannot wait on -several sockets at once — which is the piece that would have to arrive before -concurrent serving is possible. +`poll` is O(n) in the watch set per call, which starts to matter in the +thousands. `epoll` and `kqueue` do not; the interface here takes the whole set +per call precisely so that either can replace `poll` underneath without a +caller noticing. The contract these keep is checked by `tests/tools/net_socket_contract.py`, which runs a Matrix page as a process and plays the other end from Python. diff --git a/nilcompiler/src/net_support.hpp b/nilcompiler/src/net_support.hpp index 7ecddcdd..b54a280c 100644 --- a/nilcompiler/src/net_support.hpp +++ b/nilcompiler/src/net_support.hpp @@ -30,6 +30,7 @@ #include #include +#include #if defined(_WIN32) #include @@ -43,6 +44,7 @@ #include #include #include +#include #include #include #include @@ -60,6 +62,15 @@ constexpr std::int64_t max_backlog = 1024; struct Opened { int descriptor = -1; std::string error; + /** + * Nothing to do right now, on a socket that was asked not to block. + * + * Told apart from an error because for an event loop it is the ordinary + * case: poll said the listener was readable, and by the time accept ran the + * connection had been withdrawn. Reporting that as a failure would make a + * server log a line every time a port scanner connected and left. + */ + bool would_block = false; bool ok() const { return descriptor >= 0 && error.empty(); } }; @@ -122,6 +133,15 @@ constexpr int no_signal_flag = 0; #endif +/** Whether the last call failed only because it would have blocked. */ +inline bool last_call_would_block() { +#if defined(_WIN32) + return ::WSAGetLastError() == WSAEWOULDBLOCK; +#else + return errno == EAGAIN || errno == EWOULDBLOCK; +#endif +} + /** * The address a listening socket may bind. * @@ -212,6 +232,10 @@ inline Opened accept_on(int listening) { if (errno == EINTR) continue; #endif + if (last_call_would_block()) { + opened.would_block = true; + return opened; + } opened.error = last_error_text(); return opened; } @@ -417,6 +441,176 @@ inline Transferred peer_address(int descriptor) { return result; } +/** + * Readiness flags, as a NIL program sees them. + * + * Deliberately not poll(2)'s numbering: POLLIN and friends differ between + * platforms, and a program that learned the host's values would be a program + * that breaks when it moves. These three are the ones a server acts on. + */ +constexpr std::int64_t ready_readable = 1; +constexpr std::int64_t ready_writable = 2; +/** The peer hung up, or the socket is in error. Either way it is finished. */ +constexpr std::int64_t ready_finished = 4; + +/** One watched socket and what it is being watched for. */ +struct Watched { + std::int64_t token = 0; + int descriptor = -1; + std::int64_t wanted = ready_readable; +}; + +/** One socket that is ready, and what it is ready for. */ +struct Ready { + std::int64_t token = 0; + std::int64_t flags = 0; +}; + +/** + * Whether a socket blocks when there is nothing to do. + * + * A blocking accept is what a one-connection-at-a-time server uses and is + * perfectly correct for it. An event loop needs the opposite: it has already + * been told by poll that something is ready, and a blocking call that then + * finds nothing — a spurious wakeup, a connection withdrawn between the poll + * and the accept — would stop the whole loop on behalf of one socket. + */ +inline std::string set_blocking(int descriptor, bool blocking) { + std::string error; + if (!ensure_started(error)) + return error; +#if defined(_WIN32) + u_long mode = blocking ? 0 : 1; + if (::ioctlsocket(static_cast(descriptor), FIONBIO, &mode) != 0) + return last_error_text(); +#else + const int flags = ::fcntl(descriptor, F_GETFL, 0); + if (flags < 0) + return last_error_text(); + const int wanted = blocking ? (flags & ~O_NONBLOCK) : (flags | O_NONBLOCK); + if (::fcntl(descriptor, F_SETFL, wanted) < 0) + return last_error_text(); +#endif + return {}; +} + +/** + * Let several processes bind the same port. + * + * The missing half of running on more than one core. This runtime is + * single-threaded by construction — the interpreter holds one stack, one heap + * and one set of globals, with no lock anywhere near them — so a second core + * means a second process, and two processes cannot share a listening socket + * without this. The kernel then load-balances accepts between them. + * + * Distinct from SO_REUSEADDR, which `listen_on` always sets and which only + * allows rebinding a port in TIME_WAIT. This one deliberately allows two + * servers to share a port, so it is opt-in: switched on by accident it would + * silently split traffic between a running server and a stray one. + */ +inline std::string set_reuse_port(int descriptor, bool enabled) { + std::string error; + if (!ensure_started(error)) + return error; +#if defined(SO_REUSEPORT) + const int value = enabled ? 1 : 0; + if (::setsockopt(descriptor, SOL_SOCKET, SO_REUSEPORT, + reinterpret_cast(&value), sizeof(value)) != 0) + return last_error_text(); + return {}; +#else + (void)descriptor; + if (!enabled) + return {}; + return "net: this platform has no SO_REUSEPORT, so a port cannot be shared"; +#endif +} + +/** + * Wait until one of the watched sockets can be acted on. + * + * poll(2) rather than epoll or kqueue. poll is portable, needs no per-platform + * registration, and is O(n) in the watch set — which starts to matter in the + * thousands and is not where this runtime's limits are. The interface here + * takes the whole set per call precisely so that a later epoll or kqueue + * implementation can keep it without the caller noticing. + * + * A negative timeout waits indefinitely, which is what an idle server does + * between requests; zero polls and returns at once. + */ +inline std::string poll_wait(const std::vector &watching, + std::int64_t timeout_ms, + std::vector &into) { + into.clear(); + std::string error; + if (!ensure_started(error)) + return error; + if (watching.empty()) + return {}; + +#if defined(_WIN32) + std::vector entries; +#else + std::vector entries; +#endif + entries.reserve(watching.size()); + for (const auto &one : watching) { +#if defined(_WIN32) + WSAPOLLFD entry{}; + entry.fd = static_cast(one.descriptor); +#else + pollfd entry{}; + entry.fd = one.descriptor; +#endif + entry.events = 0; + if ((one.wanted & ready_readable) != 0) + entry.events = static_cast(entry.events | POLLIN); + if ((one.wanted & ready_writable) != 0) + entry.events = static_cast(entry.events | POLLOUT); + entries.push_back(entry); + } + + int signalled = 0; + for (;;) { +#if defined(_WIN32) + signalled = ::WSAPoll(entries.data(), static_cast(entries.size()), + static_cast(timeout_ms)); +#else + signalled = ::poll(entries.data(), static_cast(entries.size()), + static_cast(timeout_ms)); +#endif + if (signalled >= 0) + break; +#if !defined(_WIN32) + // An interrupted wait is not a failure. Reporting one would make an + // unrelated signal look to the server like a broken socket. + if (errno == EINTR) + continue; +#endif + return last_error_text(); + } + + for (std::size_t index = 0; index < entries.size(); ++index) { + const short revents = entries[index].revents; + if (revents == 0) + continue; + std::int64_t flags = 0; + if ((revents & POLLIN) != 0) + flags |= ready_readable; + if ((revents & POLLOUT) != 0) + flags |= ready_writable; + // POLLHUP with POLLIN still has buffered data to read first, so both are + // reported and the caller drains before closing. POLLNVAL means the + // descriptor is not open, which for a token-based API is a bug here + // rather than in the program, but it must not loop forever on it. + if ((revents & (POLLERR | POLLHUP | POLLNVAL)) != 0) + flags |= ready_finished; + if (flags != 0) + into.push_back(Ready{watching[index].token, flags}); + } + return {}; +} + /** Stop a write-blocked peer from keeping a response in a buffer. */ inline std::string set_no_delay(int descriptor, bool enabled) { std::string error; diff --git a/nilcompiler/src/vm.cpp b/nilcompiler/src/vm.cpp index 7772a0c3..79a8f2ab 100644 --- a/nilcompiler/src/vm.cpp +++ b/nilcompiler/src/vm.cpp @@ -776,6 +776,10 @@ void VM::resetRuntimeState(const std::string &root_function) { file_handles_.clear(); next_file_handle_ = 1; closeOpenSockets(); + watch_order_.clear(); + watch_wanted_.clear(); + ready_.clear(); + ready_cursor_ = 0; stack_.clear(); globals_.clear(); gc_marked_structs_.clear(); @@ -3858,6 +3862,12 @@ void VM::executeInstruction(const Instruction &instr) { return; } const auto opened = net_support::accept_on(held->descriptor); + // Zero means "nothing pending" on a listener that was asked not to + // block. Handles start at one, so it cannot be mistaken for one. + if (opened.would_block) { + push_ok_number(0); + return; + } if (!opened.ok()) { push_error(opened.error); return; @@ -3998,6 +4008,158 @@ void VM::executeInstruction(const Instruction &instr) { return; } + if (fn_name == "net.set_blocking") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::int64_t blocking = 1; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + number_at(1, blocking); + const auto error = net_support::set_blocking(held->descriptor, blocking != 0); + if (!error.empty()) { + push_error(error); + return; + } + push_ok_number(blocking != 0 ? 1 : 0); + return; + } + + if (fn_name == "net.set_reuse_port") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::int64_t enabled = 1; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + number_at(1, enabled); + const auto error = net_support::set_reuse_port(held->descriptor, enabled != 0); + if (!error.empty()) { + push_error(error); + return; + } + push_ok_number(enabled != 0 ? 1 : 0); + return; + } + + // Watching, waiting and draining. The set lives in the VM so that no + // array crosses this boundary; see the fields on VM for why. + if (fn_name == "net.watch") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::int64_t wanted = net_support::ready_readable; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + std::int64_t token = 0; + number_at(0, token); + if (args.size() > 1) + number_at(1, wanted); + if (wanted == 0) + wanted = net_support::ready_readable; + if (watch_wanted_.find(token) == watch_wanted_.end()) + watch_order_.push_back(token); + watch_wanted_[token] = wanted; + push_ok_number(static_cast(watch_order_.size())); + return; + } + + if (fn_name == "net.unwatch") { + std::int64_t token = 0; + if (!number_at(0, token)) { + push_error("net.unwatch: expected a socket handle"); + return; + } + // Unwatching something not watched is not a failure: it is what a + // close path does when it does not track whether it registered. + if (watch_wanted_.erase(token) != 0) + watch_order_.erase(std::remove(watch_order_.begin(), watch_order_.end(), token), + watch_order_.end()); + push_ok_number(static_cast(watch_order_.size())); + return; + } + + if (fn_name == "net.watch_count") { + pushStack((int64_t)static_cast(watch_order_.size())); + return; + } + + // Waits, then replaces the ready set. A wait discards whatever the + // previous one found: a program that did not drain it has moved on, and + // keeping stale readiness would hand it a handle that may since have + // been closed and reissued. + if (fn_name == "net.wait") { + std::int64_t timeout = 0; + if (!number_at(0, timeout)) { + push_error("net.wait: expected a timeout in milliseconds"); + return; + } + ready_.clear(); + ready_cursor_ = 0; + + std::vector watching; + watching.reserve(watch_order_.size()); + // Rebuilt per wait from the live handle table, so a socket closed + // since the last wait is simply not watched rather than polled on a + // descriptor number that now means something else. + std::vector still_open; + still_open.reserve(watch_order_.size()); + for (const auto token : watch_order_) { + auto found = socket_handles_.find(token); + if (found == socket_handles_.end()) { + watch_wanted_.erase(token); + continue; + } + still_open.push_back(token); + watching.push_back(net_support::Watched{token, found->second.descriptor, + watch_wanted_[token]}); + } + watch_order_ = std::move(still_open); + + if (watching.empty()) { + // Nothing to wait for. Sleeping for the timeout would be the honest + // reading, but a loop that watched nothing by mistake would then be + // an invisible stall; zero ready and an immediate return lets the + // caller notice. + push_ok_number(0); + return; + } + + std::vector found; + const auto error = net_support::poll_wait(watching, timeout, found); + if (!error.empty()) { + push_error(error); + return; + } + ready_.reserve(found.size()); + for (const auto &one : found) + ready_.emplace_back(one.token, one.flags); + push_ok_number(static_cast(ready_.size())); + return; + } + + // One ready socket per call, packed as handle * 8 + flags, and zero when + // the set is drained. + // + // Packed rather than returned as a pair because an intrinsic gives back + // one value, and two calls whose meaning depended on being made in + // order would be worse than arithmetic the caller can see. The Ny + // declarations carry ready_handle and ready_flags so no caller has to + // write the division. + if (fn_name == "net.next_ready") { + if (ready_cursor_ >= ready_.size()) { + push_ok_number(0); + return; + } + const auto &one = ready_[ready_cursor_]; + ready_cursor_ += 1; + push_ok_number(one.first * 8 + one.second); + return; + } + if (fn_name == "net.peer_address") { VM::SocketHandle *held = nullptr; std::string complaint; diff --git a/nilcompiler/src/vm.hpp b/nilcompiler/src/vm.hpp index 31a6cccc..e0059ac6 100644 --- a/nilcompiler/src/vm.hpp +++ b/nilcompiler/src/vm.hpp @@ -624,6 +624,21 @@ class VM { std::unordered_map socket_handles_; int64_t next_socket_handle_ = 1; + // The event loop's watch set and the result of its last wait. + // + // Held by the runtime rather than passed in per call, so no array has to + // cross the intrinsic boundary -- which matters because a Matrix page cannot + // index an Ny generic, and an event loop that could only be written in Ny + // would be an event loop half the language cannot use. + // + // `ready_cursor_` walks ready_ once. A program drains it with repeated calls + // and gets zero when it is empty, which is the same shape as reading a + // stream to its end and needs no second concept. + std::vector watch_order_; + std::unordered_map watch_wanted_; + std::vector> ready_; + std::size_t ready_cursor_ = 0; + // Closes every socket a run still holds. A program that exits without // closing its listener must not leave the port bound against the next one. void closeOpenSockets(); diff --git a/nilstd/src/net/socket.ny b/nilstd/src/net/socket.ny index 0e4138b7..193c8fe3 100644 --- a/nilstd/src/net/socket.ny +++ b/nilstd/src/net/socket.ny @@ -131,3 +131,123 @@ pub fn set_no_delay(handle: i64, enabled: bool) -> Result # an identity, and behind any proxy it is the proxy's. @intrinsic("net.peer_address") pub fn peer_address(handle: i64) -> Result + +# ----------------------------------------------------------------------------- +# Serving more than one connection at a time +# ----------------------------------------------------------------------------- +# +# This runtime is single-threaded by construction: the interpreter holds one +# stack, one heap and one set of globals, with no lock anywhere near them, and +# nil.std.threading offers only sleep. So concurrency here is an event loop, +# not a thread per connection — poll says which sockets can be acted on, and +# one pass round the loop acts on each in turn. +# +# For more than one core, run more than one process and let them share the +# port with set_reuse_port. The kernel load-balances accepts between them. +# That is nginx's shape rather than Apache's prefork, and it is the one this +# runtime can actually keep. +# +# The watch set lives in the runtime rather than being passed in per call, so +# no array has to cross the intrinsic boundary. That is not tidiness: a Matrix +# page cannot index an Ny generic, and an event loop that could only be +# written in Ny would be one half the language could not use. + +# Readiness flags. Deliberately not poll(2)'s numbering, which differs between +# platforms — a program that learned the host's values would break on a move. +pub const ready_readable: i64 = 1 +pub const ready_writable: i64 = 2 +# The peer hung up, or the socket is in error. Either way it is finished. +pub const ready_finished: i64 = 4 + +# Whether a socket blocks when there is nothing to do. +# +# An event loop wants its listener non-blocking: poll has already said a +# connection is pending, and a blocking accept that then finds nothing — +# because the peer withdrew in between — would stop the whole loop on behalf +# of one socket. +# +# Leave accepted connections blocking and give them a timeout instead. Poll +# has said they are readable, so a read returns at once, and an empty read +# then means the peer closed rather than "nothing yet", which is a +# distinction an event loop has to be able to make. +@intrinsic("net.set_blocking") +pub fn set_blocking(handle: i64, blocking: bool) -> Result + +# Let other processes bind this same port. +# +# Opt-in, and distinct from the SO_REUSEADDR that listen always sets. That one +# only allows rebinding a port left in TIME_WAIT; this one deliberately allows +# two servers to share a port, which switched on by accident would silently +# split traffic between a running server and a stray one. +@intrinsic("net.set_reuse_port") +pub fn set_reuse_port(handle: i64, enabled: bool) -> Result + +# Watch a socket, for the given flags. Watching it again replaces the flags. +# +# A listener is watched for ready_readable, which means a connection is +# pending. A connection is watched for ready_readable while a request is being +# read, and for ready_writable while a response is going out to a peer that is +# not taking it fast enough. +@intrinsic("net.watch") +pub fn watch(handle: i64, flags: i64) -> Result + +# Stop watching. Unwatching something not watched succeeds, because that is +# what a close path does when it does not track whether it registered. +@intrinsic("net.unwatch") +pub fn unwatch(handle: i64) -> Result + +# How many sockets are watched. +@intrinsic("net.watch_count") +pub fn watch_count() -> i64 + +# Wait until a watched socket can be acted on; gives back how many are ready. +# +# A negative timeout waits indefinitely, which is what an idle server does +# between requests. Zero polls and returns at once. +# +# Each wait replaces whatever the last one found. A program that did not drain +# the previous set has moved on, and stale readiness would hand it a handle +# that may since have been closed and its number reissued. +# +# A socket closed since the last wait is dropped from the watch set rather +# than polled, so a loop that forgets to unwatch a connection it closed is +# untidy rather than broken. +@intrinsic("net.wait") +pub fn wait(timeout_milliseconds: i64) -> Result + +# One ready socket per call, packed; zero when the set is drained. +# +# Packed because an intrinsic returns one value, and two calls whose meaning +# depended on being made in the right order would be worse than arithmetic the +# caller can see. Use ready_handle and ready_flags rather than writing it out. +@intrinsic("net.next_ready") +pub fn next_ready() -> Result + +# The socket from a packed next_ready value. +pub fn ready_handle(packed: i64) -> i64 + return packed / 8 +end fn + +# The readiness flags from a packed next_ready value. +pub fn ready_flags(packed: i64) -> i64 + return packed % 8 +end fn + +# Whether a packed value says the socket has something to read. +pub fn is_readable(packed: i64) -> bool + return (ready_flags(packed) & ready_readable) != 0 +end fn + +# Whether a packed value says the socket will take a write. +pub fn is_writable(packed: i64) -> bool + return (ready_flags(packed) & ready_writable) != 0 +end fn + +# Whether a packed value says the socket is finished. +# +# Both this and is_readable can be true at once: a peer that hung up may still +# have sent something, and that something has to be read before the connection +# is dropped or the last request of a session is lost. +pub fn is_finished(packed: i64) -> bool + return (ready_flags(packed) & ready_finished) != 0 +end fn diff --git a/tests/tools/net_socket_contract.py b/tests/tools/net_socket_contract.py index 3a852af6..3ba1e108 100644 --- a/tests/tools/net_socket_contract.py +++ b/tests/tools/net_socket_contract.py @@ -341,6 +341,210 @@ def wait_for_line(process: subprocess.Popen, marker: str, timeout: float = 20.0) """ + +POLL_PAGE = """ +book net.contract.poll + +read nil.std.core +read nil.std.net.socket + +reason-for receives outcome as Result of number, text returns text begin paragraph + consider outcome + when Result.Ok holding amount then give "ok" + when Result.Error holding why then give why + otherwise give "neither" + end consider +end paragraph + +amount-of receives outcome as Result of number, text returns number begin paragraph + consider outcome + when Result.Ok holding amount then give amount + otherwise give 0 minus 1 + end consider +end paragraph + +main returns number begin paragraph + value opened is nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 8 + consider opened + when Result.Ok holding listener then begin block + !!! An empty watch set does not wait: a loop that watched nothing + !!! by mistake would otherwise be an invisible stall. + say amount-of with outcome being (nil.std.net::wait with timeout_milliseconds being 10000) + say nil.std.net::watch_count with nothing + + say reason-for with outcome being (nil.std.net::set_blocking with handle being listener, blocking being false) + evaluate nil.std.net::watch with handle being listener, flags being nil.std.net::ready_readable + say nil.std.net::watch_count with nothing + !!! Watching again replaces the flags rather than adding a second entry. + evaluate nil.std.net::watch with handle being listener, flags being nil.std.net::ready_readable + say nil.std.net::watch_count with nothing + + !!! Nothing is connecting, so nothing is ready, and the drained + !!! ready set reads as zero. + say amount-of with outcome being (nil.std.net::wait with timeout_milliseconds being 50) + say amount-of with outcome being (nil.std.net::next_ready with nothing) + + !!! A non-blocking accept with nothing pending is zero, not an error. + say amount-of with outcome being (nil.std.net::accept with handle being listener) + + !!! Unwatching something not watched succeeds. + say reason-for with outcome being (nil.std.net::unwatch with handle being 991) + say reason-for with outcome being (nil.std.net::unwatch with handle being listener) + say nil.std.net::watch_count with nothing + + !!! A closed socket is dropped from the watch set by the next wait + !!! rather than polled on a descriptor that may have been reissued. + evaluate nil.std.net::watch with handle being listener, flags being nil.std.net::ready_readable + evaluate nil.std.net::close with handle being listener + say amount-of with outcome being (nil.std.net::wait with timeout_milliseconds being 10) + say nil.std.net::watch_count with nothing + + !!! Packing: handle and flags travel as one number. + say nil.std.net::ready_handle with packed being 41 + say nil.std.net::ready_flags with packed being 41 + say (nil.std.net::is_readable with packed being 41) becomes text + say (nil.std.net::is_finished with packed being 41) becomes text + say (nil.std.net::is_writable with packed being 41) becomes text + give 0 + end block + when Result.Error holding why then begin block + say "listen failed: " plus why + give 1 + end block + otherwise give 1 + end consider +end paragraph +""" + + +EVENT_LOOP_PAGE = """ +book net.contract.event-loop + +read nil.std.core +read nil.std.text.string_utils +read nil.std.net.socket + +pending begin record + field handle as number + field gathered as text + field live as boolean +end record + +find-slot receives slots as array of pending, handle as number returns number begin paragraph + variable index as number is 0 + repeat while index is less than the length of slots + value one is slots at index + when one.live and one.handle equals handle then + give index + end when + set index to index plus 1 + end repeat + give 0 minus 1 +end paragraph + +main returns number begin paragraph + value opened is nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 128 + consider opened + when Result.Error holding why then begin block + say "listen failed: " plus why + give 1 + end block + when Result.Ok holding listener then begin block + evaluate nil.std.net::set_blocking with handle being listener, blocking being false + evaluate nil.std.net::watch with handle being listener, flags being nil.std.net::ready_readable + say "ready" + + variable slots as array of pending is [] + variable answered as number is 0 + variable live-now as number is 0 + variable most-at-once as number is 0 + + repeat while answered is less than EXPECTED + evaluate nil.std.net::wait with timeout_milliseconds being 10000 + variable draining as boolean is true + repeat while draining + value packed is nil.std.net::next_ready with nothing + consider packed + when Result.Ok holding value-held then begin block + when value-held equals 0 then + set draining to false + end when + value handle is nil.std.net::ready_handle with packed being value-held + when value-held is greater than 0 and handle equals listener then + variable accepting as boolean is true + repeat while accepting + value taken is nil.std.net::accept with handle being listener + consider taken + when Result.Ok holding client then begin block + when client equals 0 then + set accepting to false + end when + when client is greater than 0 then + evaluate nil.std.net::set_timeout with handle being client, milliseconds being 10000 + evaluate nil.std.net::watch with handle being client, flags being nil.std.net::ready_readable + append build pending with handle being client, gathered being "", live being true to slots + set live-now to live-now plus 1 + when live-now is greater than most-at-once then + set most-at-once to live-now + end when + end when + end block + otherwise set accepting to false + end consider + end repeat + end when + when value-held is greater than 0 and not (handle equals listener) then + value slot is find-slot with slots being slots, handle being handle + when slot is at least 0 then + value heard is nil.std.net::read with handle being handle, maximum_bytes being 4096 + consider heard + when Result.Ok holding piece then begin block + value held is slots at slot + value whole is held.gathered plus piece + value done is nil.std.text::contains with s being whole, needle being "END" + value gone is piece equals "" + when not done and not gone then + set slots at slot to build pending with handle being handle, + gathered being whole, live being true + end when + when done then + evaluate nil.std.net::write with handle being handle, + data being "ok:" plus ((nil.std.text::length with s being whole) becomes text) + end when + when done or gone then + evaluate nil.std.net::unwatch with handle being handle + evaluate nil.std.net::close with handle being handle + set slots at slot to build pending with handle being 0, gathered being "", live being false + set live-now to live-now minus 1 + set answered to answered plus 1 + end when + end block + otherwise begin block + evaluate nil.std.net::unwatch with handle being handle + evaluate nil.std.net::close with handle being handle + set slots at slot to build pending with handle being 0, gathered being "", live being false + set live-now to live-now minus 1 + set answered to answered plus 1 + end block + end consider + end when + end when + end block + otherwise set draining to false + end consider + end repeat + end repeat + + evaluate nil.std.net::close with handle being listener + say "answered " plus (answered becomes text) + say "most-at-once " plus (most-at-once becomes text) + give 0 + end block + otherwise give 1 + end consider +end paragraph +""" + def run_page(interpreter: Path, artifact: Path) -> subprocess.Popen: return subprocess.Popen( [str(interpreter), str(artifact)], @@ -526,6 +730,119 @@ def case_client(compiler, interpreter, stdlib, workdir) -> list[str]: return problems + +def case_poll(compiler, interpreter, stdlib, workdir) -> list[str]: + """The watch set, the ready set, and the packing between them.""" + port = free_port() + artifact = build(compiler, stdlib, + POLL_PAGE.replace("PORT", str(port)), workdir, "poll") + finished = subprocess.run([str(interpreter), str(artifact)], + capture_output=True, text=True, timeout=120) + lines = finished.stdout.splitlines() + expected = [ + ("an empty watch set does not wait", lambda v: v == "0"), + ("nothing is watched yet", lambda v: v == "0"), + ("a listener can be made non-blocking", lambda v: v == "ok"), + ("watching counts one", lambda v: v == "1"), + ("watching again replaces rather than adds", lambda v: v == "1"), + ("nothing connecting means nothing ready", lambda v: v == "0"), + ("a drained ready set reads zero", lambda v: v == "0"), + ("a non-blocking accept with nothing pending is zero", lambda v: v == "0"), + ("unwatching something unwatched succeeds", lambda v: v == "ok"), + ("unwatching a watched socket succeeds", lambda v: v == "ok"), + ("the watch set is empty again", lambda v: v == "0"), + ("a closed socket is not polled", lambda v: v == "0"), + ("a closed socket leaves the watch set", lambda v: v == "0"), + ("the packed handle is recovered", lambda v: v == "5"), + ("the packed flags are recovered", lambda v: v == "1"), + ("packed readability is read", lambda v: v == "1"), + ("packed finishedness is read", lambda v: v == "0"), + ("packed writability is read", lambda v: v == "0"), + ] + if len(lines) != len(expected): + return [f"poll: expected {len(expected)} lines, got {len(lines)}: " + f"{finished.stdout!r}{finished.stderr!r}"] + problems: list[str] = [] + for (label, holds), value in zip(expected, lines): + if not holds(value): + problems.append(f"poll: {label} gave {value!r}") + return problems + + +def case_event_loop(compiler, interpreter, stdlib, workdir) -> list[str]: + """Many connections at once, and the timing that proves it. + + Each client sends its request in two parts with a gap between. A server + that serves one connection at a time must wait out the first client's gap + before it touches the second, so the run costs clients x gap. An event + loop overlaps them and costs about one gap. The assertion is on that + difference rather than on a throughput number, because a number would + measure the machine and this measures the shape. + """ + clients = 64 + gap = 0.25 + serial_cost = clients * gap + problems: list[str] = [] + port = free_port() + artifact = build(compiler, stdlib, + EVENT_LOOP_PAGE.replace("PORT", str(port)) + .replace("EXPECTED", str(clients)), + workdir, "event_loop") + page = run_page(interpreter, artifact) + answered: list[bytes] = [] + failures: list[str] = [] + guard = __import__("threading").Lock() + + def one_client() -> None: + try: + with socket.create_connection(("127.0.0.1", port), timeout=30) as client: + client.sendall(b"part-one ") + time.sleep(gap) + client.sendall(b"END") + reply = client.recv(64) + with guard: + (answered if reply.startswith(b"ok:") else failures).append(reply) + except Exception as failure: # noqa: BLE001 + with guard: + failures.append(repr(failure)) + + try: + wait_for_line(page, "ready") + threading = __import__("threading") + started = time.monotonic() + workers = [threading.Thread(target=one_client) for _ in range(clients)] + for worker in workers: + worker.start() + for worker in workers: + worker.join(timeout=60) + elapsed = time.monotonic() - started + + if len(answered) != clients: + problems.append( + f"event-loop: {len(answered)} of {clients} clients were answered; " + f"first failures {failures[:2]}") + if elapsed > serial_cost / 4: + problems.append( + f"event-loop: {clients} clients took {elapsed:.2f}s, which is not " + f"meaningfully better than the {serial_cost:.0f}s a one-at-a-time " + "server would need, so the connections were not overlapped") + + page.wait(timeout=30) + output = page.stdout.read() + if f"answered {clients}" not in output: + problems.append(f"event-loop: the page said {output!r}") + # The point of the whole exercise: they were open simultaneously. + if f"most-at-once {clients}" not in output: + problems.append( + "event-loop: the server never held all the connections at once, " + f"so it was draining them in batches; it said {output!r}") + except (AssertionError, subprocess.TimeoutExpired) as failure: + problems.append(f"event-loop: {failure!r}") + finally: + page.kill() + page.wait(timeout=10) + return problems + def main() -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--compiler", type=Path, required=True) @@ -544,6 +861,8 @@ def main() -> int: ("timeout", case_timeout), ("peer-hangup", case_peer_hangup), ("client", case_client), + ("poll", case_poll), + ("event-loop", case_event_loop), ) problems: list[str] = [] @@ -565,7 +884,9 @@ def main() -> int: "listen/accept/read/write carry bytes unchanged, handles are opaque, " "a listener is not a connection, the runtime releases a leaked port, " "a read deadline expires, a lost peer is reported rather than fatal, " - "and connecting out resolves a name.") + "connecting out resolves a name, the watch set behaves, and an event " + "loop holds many connections at once instead of draining them in " + "turn.") return 0 From dfb7ad7087f7ed5f71b64b3a99aec4a6489529df Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 15:17:26 -0500 Subject: [PATCH 04/39] Non-blocking transfer, and when a file was last written Two more things a web server cannot be written without. net.receive and net.send_some are the non-blocking counterparts of read and write. write is the dangerous one in an event loop: a large response to a client that reads slowly fills the socket buffer, write_all blocks, and one slow reader stops every other connection. send_some takes what the socket will accept from an offset and reports it, leaving the remainder for when poll says the socket will take more. receive is two calls because a non-blocking read has three outcomes and a Result has room for two: bytes arrived, the peer closed, or nothing is available yet. It reports which, and net.received hands over the bytes. Conflating the last two is the classic event-loop bug -- a spurious wakeup looks like a hangup and the connection is dropped in the middle of a request. file.modified_at closes the other gap. nil.std.io had a size and no timestamp, which left two ordinary jobs unreachable: a cache could not tell whether what it held was still current without re-reading the file, which defeats the cache, and an HTTP server could not send Last-Modified or answer If-Modified-Since, so every client re-fetched every unchanged file in full, forever. Milliseconds since the Unix epoch, because a filesystem timestamp is not reliably finer than that across platforms and a caller comparing two should not be comparing precision it does not have. The conversion is the portable one rather than std::chrono::clock_cast, which says it in a line and is not available on every toolchain this still builds on: file_clock's epoch is unspecified and need not match the system clock's, so the two are related by taking a reading from each at the same moment. tests/matrix/execution/file-time asserts the answer is a plausible epoch time in milliseconds -- a seconds-based answer fails the lower bound, which is the mistake worth catching -- that writing again moves it forward, and that an absent file is reported rather than given a zero a comparison would read as 1970 and treat as very old indeed. 38 execution fixtures now pass. Co-Authored-By: Claude Opus 5 --- docs/standard-library/index.md | 3 +- docs/standard-library/sockets.md | 19 ++- nilcompiler/src/net_support.hpp | 44 ++++++ nilcompiler/src/vm.cpp | 143 ++++++++++++++++++ nilcompiler/src/vm.hpp | 6 + nilstd/src/io/file.ny | 16 ++ nilstd/src/net/socket.ny | 38 +++++ tests/matrix/execution/file-time.expected | 5 + tests/matrix/execution/file-time.matrix | 53 +++++++ tests/matrix/execution/file-time.needs-stdlib | 0 10 files changed, 321 insertions(+), 6 deletions(-) create mode 100644 tests/matrix/execution/file-time.expected create mode 100644 tests/matrix/execution/file-time.matrix create mode 100644 tests/matrix/execution/file-time.needs-stdlib diff --git a/docs/standard-library/index.md b/docs/standard-library/index.md index a489ec7f..f05b6c70 100644 --- a/docs/standard-library/index.md +++ b/docs/standard-library/index.md @@ -22,7 +22,8 @@ three-component lengths, finite-value checks, and checked normalization shared by Ny and Matrix. The [socket reference](sockets.md) defines TCP listening, accepting, connecting and transfer, with opaque handles the runtime closes for a program -that does not. +that does not, and the poll-based event loop that lets one process serve many +connections at once. The [random-generator reference](random.md) defines independent mutable streams, reproducible seeds, cloning, and bounded integer/floating sampling. diff --git a/docs/standard-library/sockets.md b/docs/standard-library/sockets.md index ef984d8b..b2d9e350 100644 --- a/docs/standard-library/sockets.md +++ b/docs/standard-library/sockets.md @@ -26,6 +26,9 @@ compiled through it will not find them. | `wait(timeout_milliseconds: i64) -> Result` | Wait for readiness; gives back how many are ready. | | `next_ready() -> Result` | One ready socket per call, packed; zero when drained. | | `ready_handle` / `ready_flags` / `is_readable` / `is_writable` / `is_finished` | Unpack what `next_ready` gave. | +| `receive(handle: i64, maximum_bytes: i64) -> Result` | Take what arrived: count, 0 for closed, -1 for nothing yet. | +| `received() -> Result` | The bytes from the last `receive`. | +| `send_some(handle: i64, data: text, offset: i64) -> Result` | Send what the socket takes now; -1 when it would block. | Matrix uses the same names; `maximum_bytes` is spelled `maximum_bytes` because it is an Ny declaration and Matrix keeps a foreign name's spelling. @@ -182,11 +185,17 @@ division. connections are waiting, so accept until it answers zero. A loop that accepts one per wait falls behind under load and never catches up. -**Leave accepted connections blocking, and give them a timeout.** `poll` has -said the connection is readable, so a read returns at once — and an empty read -then means the peer closed rather than "nothing yet", which is a distinction -an event loop has to be able to make. A non-blocking read cannot make it: -EAGAIN and a clean close both produce no bytes. +**Use `receive` and `send_some`, not `read` and `write`, on a non-blocking +connection.** `read` and `write` suit a blocking socket: read waits for +something, write loops until everything has gone. In an event loop `write` is +the dangerous one — a large response to a client that reads slowly fills the +socket buffer, the call blocks, and one slow reader stops everybody. + +`receive` exists because a non-blocking read has three outcomes and a +`Result` has room for two. It reports the byte count, `0` for a +peer that closed, or `-1` for nothing yet, and `received` hands over the bytes. +Conflating the last two is the classic event-loop bug: a spurious wakeup then +looks like a hangup and the connection is dropped mid-request. **`is_readable` and `is_finished` can both be true.** A peer that hung up may still have sent something, and that something has to be read before the diff --git a/nilcompiler/src/net_support.hpp b/nilcompiler/src/net_support.hpp index b54a280c..4ce03cdb 100644 --- a/nilcompiler/src/net_support.hpp +++ b/nilcompiler/src/net_support.hpp @@ -79,6 +79,8 @@ struct Transferred { std::int64_t count = 0; /** The peer closed its side in an orderly way. Not an error. */ bool closed = false; + /** Nothing available right now, on a socket asked not to block. */ + bool would_block = false; std::string error; bool ok() const { return error.empty(); } }; @@ -330,6 +332,48 @@ inline Transferred read_some(int descriptor, std::int64_t wanted) { if (errno == EINTR) continue; #endif + if (last_call_would_block()) { + result.would_block = true; + return result; + } + result.error = last_error_text(); + return result; + } +} + +/** + * Write what the socket will take right now, and say how much that was. + * + * The non-blocking counterpart of write_all. A server sending a large + * response to a client that reads it slowly cannot afford write_all: the + * socket buffer fills, the call blocks, and one slow reader stops the whole + * event loop. This takes what it can, reports it, and leaves the rest to the + * caller to send when poll says the socket will accept more. + */ +inline Transferred write_some(int descriptor, const std::string &data, + std::size_t offset) { + Transferred result; + if (!ensure_started(result.error)) + return result; + if (offset >= data.size()) + return result; + + for (;;) { + const transfer_size taken = + ::send(descriptor, data.data() + offset, + static_cast(data.size() - offset), no_signal_flag); + if (taken > 0) { + result.count = static_cast(taken); + return result; + } +#if !defined(_WIN32) + if (taken < 0 && errno == EINTR) + continue; +#endif + if (last_call_would_block()) { + result.would_block = true; + return result; + } result.error = last_error_text(); return result; } diff --git a/nilcompiler/src/vm.cpp b/nilcompiler/src/vm.cpp index 79a8f2ab..459dfe28 100644 --- a/nilcompiler/src/vm.cpp +++ b/nilcompiler/src/vm.cpp @@ -780,6 +780,7 @@ void VM::resetRuntimeState(const std::string &root_function) { watch_wanted_.clear(); ready_.clear(); ready_cursor_ = 0; + last_received_.clear(); stack_.clear(); globals_.clear(); gc_marked_structs_.clear(); @@ -4008,6 +4009,95 @@ void VM::executeInstruction(const Instruction &instr) { return; } + // Reading and writing without blocking. + // + // Two calls rather than one, because a single Result has no + // room for the three answers a non-blocking read has to give: bytes + // arrived, the peer closed, or nothing is available yet. Conflating the + // last two is the classic event-loop bug -- a spurious wakeup then + // looks like a hangup and the connection is dropped mid-request. So + // `receive` reports which happened and `received` hands over the bytes. + if (fn_name == "net.receive") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::int64_t wanted = 0; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + if (!number_at(1, wanted)) { + push_error("net.receive: expected a socket handle and a byte count"); + return; + } + if (held->listening) { + push_error("net.receive: that socket is a listener, not a connection"); + return; + } + auto taken = net_support::read_some(held->descriptor, wanted); + if (!taken.ok()) { + last_received_.clear(); + push_error(taken.error); + return; + } + if (taken.would_block) { + last_received_.clear(); + push_ok_number(-1); + return; + } + if (taken.closed) { + last_received_.clear(); + push_ok_number(0); + return; + } + last_received_ = std::move(taken.data); + push_ok_number(static_cast(last_received_.size())); + return; + } + + if (fn_name == "net.received") { + push_ok_text(last_received_); + return; + } + + // Sends what the socket will take now, from an offset into the data. + // Minus one means it would block: the buffer is full and poll will say + // when it is not. Zero bytes sent with no block is possible for an + // empty remainder and is not an error. + if (fn_name == "net.send_some") { + VM::SocketHandle *held = nullptr; + std::string complaint; + std::string data; + std::int64_t offset = 0; + if (!socket_at(0, held, complaint)) { + push_error(complaint); + return; + } + if (!text_at(1, data) || !number_at(2, offset)) { + push_error("net.send_some: expected a socket handle, text, and an offset"); + return; + } + if (held->listening) { + push_error("net.send_some: that socket is a listener, not a connection"); + return; + } + if (offset < 0) { + push_error("net.send_some: an offset cannot be negative"); + return; + } + auto sent = net_support::write_some(held->descriptor, data, + static_cast(offset)); + if (!sent.ok()) { + push_error(sent.error); + return; + } + if (sent.would_block) { + push_ok_number(-1); + return; + } + push_ok_number(sent.count); + return; + } + if (fn_name == "net.set_blocking") { VM::SocketHandle *held = nullptr; std::string complaint; @@ -4832,6 +4922,59 @@ void VM::executeInstruction(const Instruction &instr) { return; } + // file.modified_at(path) -> Result, in milliseconds since the + // epoch. + // + // There was a size and no time, which left two ordinary jobs unreachable. + // A cache cannot tell whether what it holds is still current without + // re-reading the file, which defeats the cache; and an HTTP server cannot + // send Last-Modified or answer If-Modified-Since, so every client fetches + // every unchanged file in full, forever. + // + // Milliseconds rather than the host's native resolution: a filesystem + // timestamp is not reliably finer than that across platforms, and a + // caller comparing two of them should not be comparing precision it does + // not have. + if (fn_name == "file.modified_at") { + auto push_ok_number = [&](std::int64_t number) { + EnumValue ok; ok.variant_name = "Result::Ok"; ok.payloads.push_back(number); + pushStack(ok); + }; + auto push_error = [&](const std::string &message) { + EnumValue bad; bad.variant_name = "Result::Error"; bad.payloads.push_back(message); + pushStack(bad); + }; + if (args.size() != 1 || !std::holds_alternative(args[0])) { + push_error("file.modified_at: expected one text path"); + return; + } + const std::string &path = std::get(args[0]); + std::error_code ec; + const auto written = fs::last_write_time(path, ec); + if (ec) { + // The path and the reason are both left out: a caller that needs to + // tell "absent" from "unreadable" asks file.exists, and an error + // carrying a path puts it into whatever log the caller keeps. + push_error("file.modified_at: the file has no readable timestamp"); + return; + } + // file_clock's epoch is unspecified and need not match the system + // clock's, so the two are related by taking a reading from each at the + // same moment and carrying the difference across. std::chrono::clock_cast + // would say this in one line and is not available on every toolchain + // this still builds on. + const auto file_now = fs::file_time_type::clock::now(); + const auto system_now = std::chrono::system_clock::now(); + const auto since_now = + std::chrono::duration_cast( + written - file_now); + const auto as_system = std::chrono::time_point_cast( + system_now + since_now); + push_ok_number(static_cast( + as_system.time_since_epoch().count())); + return; + } + if (fn_name == "file.exists") { if (args.size() == 1 && std::holds_alternative(args[0])) { diff --git a/nilcompiler/src/vm.hpp b/nilcompiler/src/vm.hpp index e0059ac6..46c19cdf 100644 --- a/nilcompiler/src/vm.hpp +++ b/nilcompiler/src/vm.hpp @@ -639,6 +639,12 @@ class VM { std::vector> ready_; std::size_t ready_cursor_ = 0; + // The bytes from the last net.receive. Held here because a non-blocking + // read has three outcomes -- bytes, a closed peer, or nothing yet -- and a + // single Result has room for two. net.receive reports which + // happened and net.received hands the bytes over. + std::string last_received_; + // Closes every socket a run still holds. A program that exits without // closing its listener must not leave the port bound against the next one. void closeOpenSockets(); diff --git a/nilstd/src/io/file.ny b/nilstd/src/io/file.ny index 266b499c..749000c9 100644 --- a/nilstd/src/io/file.ny +++ b/nilstd/src/io/file.ny @@ -81,6 +81,22 @@ pub fn copy(source: text, dest: text) -> Result pub fn move(source: text, dest: text) -> Result # Get file size in bytes +# When the file was last written, in milliseconds since the Unix epoch. +# +# Two ordinary jobs need this and neither can be done without it. A cache +# cannot tell whether what it holds is still current without re-reading the +# file, which defeats the cache. An HTTP server cannot send Last-Modified or +# answer If-Modified-Since, so every client re-fetches every unchanged file in +# full, forever. +# +# Milliseconds rather than the host's native resolution: a filesystem +# timestamp is not reliably finer than that across platforms, and a caller +# comparing two should not be comparing precision it does not have. The error +# carries neither the path nor the reason; ask `exists` to tell an absent file +# from an unreadable one. +@intrinsic("file.modified_at") +pub fn modified_at(path: text) -> Result + @intrinsic("file.size") pub fn size(path: text) -> Result diff --git a/nilstd/src/net/socket.ny b/nilstd/src/net/socket.ny index 193c8fe3..7707c5fc 100644 --- a/nilstd/src/net/socket.ny +++ b/nilstd/src/net/socket.ny @@ -251,3 +251,41 @@ end fn pub fn is_finished(packed: i64) -> bool return (ready_flags(packed) & ready_finished) != 0 end fn + +# ----------------------------------------------------------------------------- +# Reading and writing without blocking +# ----------------------------------------------------------------------------- +# +# `read` and `write` above suit a blocking socket: read waits for something, +# write loops until everything has gone. An event loop cannot use write that +# way — a large response to a client that reads slowly fills the socket buffer, +# the call blocks, and one slow reader stops the whole loop. These are the +# non-blocking pair. + +# Take what has arrived, and say which of three things happened. +# +# Returns the byte count on success, 0 when the peer closed its side, and -1 +# when nothing is available yet. Call `received` for the bytes themselves. +# +# Two calls rather than one because a single Result has room for +# two answers and a non-blocking read has three. Conflating "nothing yet" with +# "the peer closed" is the classic event-loop bug: a spurious wakeup then +# looks like a hangup and the connection is dropped in the middle of a request. +@intrinsic("net.receive") +pub fn receive(handle: i64, maximum_bytes: i64) -> Result + +# The bytes from the last successful `receive` on any socket. +# +# Valid immediately after a `receive` that returned a positive count; empty +# after one that reported a close or a would-block. Read it before the next +# receive, which replaces it. +@intrinsic("net.received") +pub fn received() -> Result + +# Send what the socket will take right now, starting at an offset. +# +# Returns the number of bytes sent, or -1 when the socket would block. The +# caller keeps the remainder and sends it when `wait` reports the socket +# writable, which is what stops a slow reader from holding up everyone else. +@intrinsic("net.send_some") +pub fn send_some(handle: i64, data: text, offset: i64) -> Result diff --git a/tests/matrix/execution/file-time.expected b/tests/matrix/execution/file-time.expected new file mode 100644 index 00000000..bf318ba7 --- /dev/null +++ b/tests/matrix/execution/file-time.expected @@ -0,0 +1,5 @@ +1 +1 +1 +17 +-1 diff --git a/tests/matrix/execution/file-time.matrix b/tests/matrix/execution/file-time.matrix new file mode 100644 index 00000000..11c409f3 --- /dev/null +++ b/tests/matrix/execution/file-time.matrix @@ -0,0 +1,53 @@ +!!! When a file was last written. +!!! +!!! nil.std.io had a size and no timestamp, which left two ordinary jobs +!!! unreachable: a cache could not tell whether what it held was still +!!! current without re-reading the file, and an HTTP server could not send +!!! Last-Modified or answer If-Modified-Since. +!!! +!!! Nothing here can assert a particular instant. What it asserts is that the +!!! answer is a plausible epoch time in milliseconds, that writing again +!!! moves it forward rather than back, and that an absent file is reported +!!! rather than given a zero that a comparison would read as 1970. + +book matrix.file-time + +read nil.std.core +read nil.std.io.file + +stamp-of receives path as text returns number begin paragraph + value found is nil.std.io::modified_at with path being path + consider found + when Result.Ok holding stamped then give stamped + otherwise give 0 minus 1 + end consider +end paragraph + +main returns number begin paragraph + value path is "matrix-file-time-fixture.txt" + evaluate nil.std.io::write_text with path being path, content being "first" + value first-stamp is stamp-of with path being path + + !!! Milliseconds since the Unix epoch, so any real file is well past the + !!! start of 2020 and well short of the year 2200. A seconds-based answer + !!! would fail the lower bound, which is the mistake worth catching. + say (first-stamp is greater than 1577836800000) becomes text + say (first-stamp is less than 7258118400000) becomes text + + evaluate nil.std.io::write_text with path being path, content being "second and longer" + value second-stamp is stamp-of with path being path + say (second-stamp is at least first-stamp) becomes text + + !!! The size moved too, which is what a validator pairs with the time. + value measured is nil.std.io::size with path being path + consider measured + when Result.Ok holding width then say width + otherwise say 0 minus 1 + end consider + + evaluate nil.std.io::delete with path being path + !!! An absent file is reported, not given a zero that a comparison would + !!! read as the epoch and treat as very old indeed. + say stamp-of with path being path + give 0 +end paragraph diff --git a/tests/matrix/execution/file-time.needs-stdlib b/tests/matrix/execution/file-time.needs-stdlib new file mode 100644 index 00000000..e69de29b From 45f5569a764a8ebbb4a1ef41ea171de643e7b4bf Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 17:42:07 -0500 Subject: [PATCH 05/39] Fix web runtime portability and verify socket ownership across hosts --- .github/workflows/macos-appkit-provider.yml | 57 +++- .github/workflows/macos-bringup.yml | 4 +- .github/workflows/web-runtime-parity.yml | 142 ++++++++++ CMakeLists.txt | 31 +++ README.md | 8 +- docs/platform-support.md | 37 +-- docs/standard-library/random.md | 6 +- docs/standard-library/sockets.md | 54 ++-- docs/standard-library/ui.md | 20 +- nilcompiler/CMakeLists.txt | 4 +- nilcompiler/src/entropy_support.hpp | 54 +++- nilcompiler/src/file_time_support.hpp | 89 ++++++ nilcompiler/src/native_runtime.cpp | 8 + nilcompiler/src/net_support.hpp | 240 ++++++++++------ nilcompiler/src/ordinary_io_intrinsic.hpp | 3 +- nilcompiler/src/ui_runtime.hpp | 14 +- nilcompiler/src/vm.cpp | 86 +++--- nilcompiler/src/vm.hpp | 7 +- nilstd/src/io/file.ny | 3 + nilstd/src/net/socket.ny | 38 ++- notebook/2026-11-11-web-runtime-parity.md | 105 +++++++ tests/file_time_support_tests.cpp | 104 +++++++ tests/matrix/execution/entropy.matrix | 4 +- tests/net_lifecycle_tests.cpp | 157 +++++++++++ tests/net_support_tests.cpp | 230 ++++++++++++++++ tests/tools/file_modified_at_contract.py | 180 ++++++++++++ tests/tools/net_socket_contract.py | 289 +++++++++++++++++--- tests/tools/ui_display_contract.py | 20 +- 28 files changed, 1723 insertions(+), 271 deletions(-) create mode 100644 .github/workflows/web-runtime-parity.yml create mode 100644 nilcompiler/src/file_time_support.hpp create mode 100644 notebook/2026-11-11-web-runtime-parity.md create mode 100644 tests/file_time_support_tests.cpp create mode 100644 tests/net_lifecycle_tests.cpp create mode 100644 tests/net_support_tests.cpp create mode 100644 tests/tools/file_modified_at_contract.py diff --git a/.github/workflows/macos-appkit-provider.yml b/.github/workflows/macos-appkit-provider.yml index 83c1d85b..ea6d641e 100644 --- a/.github/workflows/macos-appkit-provider.yml +++ b/.github/workflows/macos-appkit-provider.yml @@ -13,6 +13,11 @@ name: macOS AppKit provider checks type: choice options: [provider, display] default: provider + llvm: + description: Enable native application routes in the display campaign + type: choice + options: ["OFF", "ON"] + default: "OFF" permissions: contents: read @@ -21,9 +26,8 @@ concurrency: group: macos-appkit-provider-${{ github.ref }}-${{ inputs.architecture }} cancel-in-progress: false -# Build and exercise the AppKit desktop provider only. LLVM is off and no SDK -# is configured, qualified or published, so a cycle here costs minutes rather -# than a full source campaign. +# Provider-only checks keep LLVM off. The display campaign can enable native +# application routes. Neither scope qualifies or publishes an SDK. jobs: provider: if: ${{ inputs.scope == 'provider' }} @@ -80,7 +84,7 @@ jobs: display: if: ${{ inputs.scope == 'display' }} - name: macOS ${{ matrix.host.arch }} AppKit display contract + name: macOS ${{ matrix.host.arch }} AppKit display / LLVM ${{ inputs.llvm }} runs-on: ${{ matrix.host.runner }} timeout-minutes: 60 strategy: @@ -91,6 +95,9 @@ jobs: defaults: run: shell: bash + env: + NIL_CI_LLVM: ${{ inputs.llvm }} + HOMEBREW_NO_AUTO_UPDATE: '1' steps: - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 with: @@ -105,18 +112,48 @@ jobs: print('CGMainDisplayID:', core.CGMainDisplayID()) " # The language-level contract needs the compiler, interpreter and test - # runner as well as the provider, so this is a whole build; LLVM stays - # off to keep it to one route rather than a full source campaign. - - name: Configure without LLVM + # runner as well as the provider. LLVM ON adds actual native cases. + - name: Configure the selected application routes timeout-minutes: 8 run: | + nil_llvm_args=() + if [ "$NIL_CI_LLVM" = ON ]; then + nil_llvm_prefix="$(brew --prefix llvm@18)" + test -x "$nil_llvm_prefix/bin/llvm-config" + nil_llvm_args+=("-DLLVM_DIR=$nil_llvm_prefix/lib/cmake/llvm") + fi cmake -S . -B build/ci \ -DCMAKE_BUILD_TYPE=Release \ -DBUILD_TESTING=ON \ - -DNIL_ENABLE_LLVM=OFF + -DNIL_ENABLE_LLVM="$NIL_CI_LLVM" "${nil_llvm_args[@]}" - name: Build timeout-minutes: 40 - run: cmake --build build/ci --parallel 3 + run: >- + cmake --build build/ci --parallel 2 --target nilcompiler nilinterpreter + niltest nil_ui_runtime_appkit_tests nil_ui_lifecycle_tests - name: Language-level desktop contract timeout-minutes: 15 - run: ctest --test-dir build/ci -R 'nil.ui' --output-on-failure + run: | + # Anchor the product namespace: 'nil.ui' also matches nilbuild.*. + ctest --test-dir build/ci -R '^nil[.]ui([.-]|$)' \ + --no-tests=error --output-on-failure --output-junit appkit-display-results.xml + python3 - <<'PY' + import xml.etree.ElementTree as ET + cases = list(ET.parse('build/ci/appkit-display-results.xml').getroot().iter('testcase')) + expected = {'nil.ui-runtime', 'nil.ui-windows-display-harness', 'nil.ui-display', + 'nil.ui-intrinsic-contract', 'nil.ui-models'} + assert len(cases) == len(expected) and {case.get('name') for case in cases} == expected + assert all(case.get('status') == 'run' and case.find('skipped') is None and + case.find('failure') is None and case.find('error') is None for case in cases) + PY + - name: Preserve complete or partial application evidence + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: appkit-display-${{ matrix.host.arch }}-llvm-${{ inputs.llvm }}-${{ github.sha }}-${{ github.run_attempt }} + path: | + build/ci/appkit-display-results.xml + build/ci/Testing/Temporary/LastTest.log + build/ci/CMakeCache.txt + if-no-files-found: error + retention-days: 14 diff --git a/.github/workflows/macos-bringup.yml b/.github/workflows/macos-bringup.yml index ccff96ea..75cda122 100644 --- a/.github/workflows/macos-bringup.yml +++ b/.github/workflows/macos-bringup.yml @@ -94,7 +94,7 @@ jobs: echo "NIL_CI_PQ_INCLUDE=OFF" >> "$GITHUB_ENV" echo "PostgreSQL: coherent postgresql@16 distribution absent; provider unavailable in this image profile" >> build/macos-evidence/providers.txt fi - echo "GTK: macOS backend not implemented; complete CTest inventory retains unsupported results" >> build/macos-evidence/providers.txt + echo "Desktop: native AppKit provider; display tests require a window server session" >> build/macos-evidence/providers.txt if command -v pandoc > /dev/null; then pandoc --version >> build/macos-evidence/providers.txt else @@ -167,7 +167,7 @@ jobs: # Even a partial build supplies useful NOT_RUN/failed test inventory. # An earlier build failure continues to fail the job. if: ${{ !cancelled() && steps.configure.outcome == 'success' }} - timeout-minutes: 20 + timeout-minutes: 35 run: | set -o pipefail ctest --test-dir build/ci --output-on-failure --parallel 2 \ diff --git a/.github/workflows/web-runtime-parity.yml b/.github/workflows/web-runtime-parity.yml new file mode 100644 index 00000000..ae4dba20 --- /dev/null +++ b/.github/workflows/web-runtime-parity.yml @@ -0,0 +1,142 @@ +name: Web runtime platform checks + +"on": + pull_request: + paths: + - '.github/workflows/web-runtime-parity.yml' + - 'CMakeLists.txt' + - 'nilcompiler/**' + - 'nilstd/**' + - 'tests/net_*' + - 'tests/file_time_support_tests.cpp' + - 'tests/tools/net_socket_contract.py' + - 'tests/tools/file_modified_at_contract.py' + - 'tests/matrix/**' + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: web-runtime-parity-${{ github.ref }} + cancel-in-progress: true + +# These are real host/interpreter checks for the new web-runtime APIs. LLVM is +# disabled; success does not qualify native socket support or an installed SDK. +jobs: + runtime: + name: ${{ matrix.host.name }} interpreter runtime + runs-on: ${{ matrix.host.runner }} + timeout-minutes: 60 + strategy: + fail-fast: false + max-parallel: 4 + matrix: + host: + - {name: Linux x86_64, runner: ubuntu-24.04, python: python3} + - {name: Windows x86_64, runner: windows-2022, python: python} + - {name: macOS x86_64, runner: macos-15-intel, python: python3} + - {name: macOS arm64, runner: macos-15, python: python3} + defaults: + run: + shell: bash + env: + PYTHONDONTWRITEBYTECODE: '1' + PYTHONUTF8: '1' + HOMEBREW_NO_AUTO_UPDATE: '1' + NIL_CI_PYTHON: ${{ matrix.host.python }} + steps: + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 + with: + persist-credentials: false + - name: Record source and host + timeout-minutes: 2 + run: | + "$NIL_CI_PYTHON" - <<'PY' + import json, os, platform, subprocess, sys + from pathlib import Path + root = Path('build/web-parity-evidence') + root.mkdir(parents=True) + revision = subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True, timeout=15).strip() + assert revision == os.environ['GITHUB_SHA'] + report = dict(revision=revision, system=platform.system(), machine=platform.machine(), + python=sys.executable, python_version=sys.version, + llvm=False, scope='host interpreter runtime; no SDK or native socket qualification') + (root/'identity.json').write_text(json.dumps(report, indent=2)+'\n') + print(json.dumps(report)) + PY + - name: Configure focused Release build + timeout-minutes: 8 + run: | + set -o pipefail + nil_python_executable="$("$NIL_CI_PYTHON" -c 'import sys; print(sys.executable)')" + cmake -S . -B build/web-parity \ + -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DNIL_ENABLE_LLVM=OFF \ + "-DPython3_EXECUTABLE=$nil_python_executable" \ + -DNILCOMPILER_SQLITE_INCLUDE_DIR=OFF -DNILCOMPILER_POSTGRESQL_INCLUDE_DIR=OFF \ + 2>&1 | tee build/web-parity-evidence/configure.log + - name: Build compiler and focused fixtures + timeout-minutes: 40 + run: | + set -o pipefail + cmake --build build/web-parity --config Release --parallel 2 \ + --target nilcompiler nilinterpreter nil_net_support_tests \ + nil_net_lifecycle_tests nil_file_time_support_tests \ + 2>&1 | tee build/web-parity-evidence/build.log + - name: Run real peer and language contracts + timeout-minutes: 8 + run: | + "$NIL_CI_PYTHON" - <<'PY' + import json, subprocess + from pathlib import Path + pattern = r'^(net[.](support-contract|lifecycle-contract|socket-contract)|nil[.](file-time-support|file-modified-at-contract)|matrix[.]execution)$' + base = ['ctest', '--test-dir', 'build/web-parity', '-C', 'Release', '-R', pattern] + root = Path('build/web-parity-evidence').resolve() + inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) + (root/'inventory.json').write_bytes(inventory) + expected = {'net.support-contract', 'net.lifecycle-contract', 'net.socket-contract', + 'nil.file-time-support', 'nil.file-modified-at-contract', 'matrix.execution'} + assert {test['name'] for test in json.loads(inventory)['tests']} == expected + with (root/'ctest.log').open('wb') as output: + result = subprocess.run(base+['--no-tests=error', '--output-on-failure', + '--test-output-size-passed', '16384', '--test-output-size-failed', '65536', + '--output-junit', str(root/'results.xml')], stdout=output, stderr=subprocess.STDOUT, timeout=450) + print((root/'ctest.log').read_text(errors='replace')) + result.check_returncode() + import xml.etree.ElementTree as ET + cases = list(ET.parse(root/'results.xml').getroot().iter('testcase')) + assert len(cases) == len(expected) and {case.get('name') for case in cases} == expected + assert all(case.get('status') == 'run' and case.find('skipped') is None and case.find('failure') is None and + case.find('error') is None for case in cases), 'Unexpected failure or skipped contract' + PY + - name: Preserve source identity and evidence + if: always() + timeout-minutes: 2 + run: | + "$NIL_CI_PYTHON" - <<'PY' + import hashlib, json, os, shutil, subprocess + from pathlib import Path + root = Path('build/web-parity-evidence') + root.mkdir(parents=True, exist_ok=True) + contracts = Path('build/web-parity/test-evidence') + if contracts.is_dir(): + shutil.copytree(contracts, root/'contracts') + dirty = subprocess.check_output(['git', 'status', '--porcelain', '--untracked-files=no'], text=True, timeout=15) + revision = subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True, timeout=15).strip() + files = {} + for path in sorted(root.rglob('*')): + if path.is_file() and not path.is_symlink(): + data = path.read_bytes() + files[path.relative_to(root).as_posix()] = dict(bytes=len(data), sha256=hashlib.sha256(data).hexdigest()) + (root/'evidence.json').write_text(json.dumps(dict(revision=revision, + source_unchanged=not dirty, files=files), indent=2)+'\n') + assert not dirty and revision == os.environ['GITHUB_SHA'] + PY + - name: Upload complete or partial results + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: web-runtime-${{ matrix.host.runner }}-${{ github.sha }}-${{ github.run_attempt }} + path: build/web-parity-evidence/ + if-no-files-found: error + retention-days: 14 diff --git a/CMakeLists.txt b/CMakeLists.txt index cf8f5627..f4d082d4 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -237,6 +237,29 @@ if(BUILD_TESTING) set_tests_properties(nil.owned-contract PROPERTIES LABELS "nil;ownership;rc" TIMEOUT 30) find_package(Threads REQUIRED) + add_executable(nil_net_support_tests tests/net_support_tests.cpp) + target_compile_features(nil_net_support_tests PRIVATE cxx_std_17) + target_include_directories(nil_net_support_tests PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/nilcompiler/src") + target_link_libraries(nil_net_support_tests PRIVATE Threads::Threads) + if(WIN32) + target_link_libraries(nil_net_support_tests PRIVATE ws2_32) + endif() + add_test(NAME net.support-contract COMMAND $) + set_tests_properties(net.support-contract PROPERTIES LABELS "net;runtime;rc" TIMEOUT 60) + + add_executable(nil_net_lifecycle_tests tests/net_lifecycle_tests.cpp) + target_compile_features(nil_net_lifecycle_tests PRIVATE cxx_std_17) + target_include_directories(nil_net_lifecycle_tests PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/nilcompiler/src") + target_link_libraries(nil_net_lifecycle_tests PRIVATE nilcompiler_core) + add_test(NAME net.lifecycle-contract COMMAND $) + set_tests_properties(net.lifecycle-contract PROPERTIES LABELS "net;runtime;vm;rc" TIMEOUT 60) + + add_executable(nil_file_time_support_tests tests/file_time_support_tests.cpp) + target_compile_features(nil_file_time_support_tests PRIVATE cxx_std_17) + target_include_directories(nil_file_time_support_tests PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/nilcompiler/src") + add_test(NAME nil.file-time-support COMMAND $) + set_tests_properties(nil.file-time-support PROPERTIES LABELS "io;runtime;rc" TIMEOUT 30) + add_executable(ny_database_runtime_tests tests/database_runtime_tests.cpp) target_link_libraries(ny_database_runtime_tests PRIVATE nilruntime) target_include_directories(ny_database_runtime_tests PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/nilcompiler/src") @@ -1038,6 +1061,14 @@ if(BUILD_TESTING) --interpreter $ --root "${CMAKE_CURRENT_SOURCE_DIR}") set_tests_properties(net.socket-contract PROPERTIES LABELS "net;routes;rc" TIMEOUT 300) + add_test(NAME nil.file-modified-at-contract + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/file_modified_at_contract.py" + --compiler $ + --interpreter $ + --nilstd "${CMAKE_CURRENT_SOURCE_DIR}/nilstd/src" + --llvm "${NIL_ENABLE_LLVM}" + --output-parent "${CMAKE_BINARY_DIR}/test-evidence") + set_tests_properties(nil.file-modified-at-contract PROPERTIES LABELS "io;routes;rc" TIMEOUT 180) add_test(NAME nillsp.semantic-protocol COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/lsp_semantic_protocol.py" --lsp $) diff --git a/README.md b/README.md index a0cf6f16..fc81e8f8 100644 --- a/README.md +++ b/README.md @@ -255,8 +255,12 @@ assembly and executable-context embedding, inspection and dead-strip retention checks. Both macOS development SDKs also pass the full installed standard-library campaign, isolated PostgreSQL application routes, installed debugger checks and execution checkpoints. Both hosts now pass the underlying -AppKit window/event checks, including directed keyboard input and native closure -of a verified child window. The GTK port and desktop acceptance remain in progress. +AppKit provider and interpreter language-level desktop checks. Windows uses its +native Win32 provider; Linux uses GTK4. macOS editor integration and a complete +current SDK desktop campaign remain open. The newer socket, entropy and text-byte +intrinsics are interpreter-backed; their cross-platform acceptance is being +brought up to date. File timestamps now share an implementation across interpreter +and native routes, with Linux text/binary NIL, JIT and AOT checks passing. Incremental reuse and native memory-limit tests also remain open, so full platform parity is not yet established. The [platform support matrix](docs/platform-support.md) separates verified diff --git a/docs/platform-support.md b/docs/platform-support.md index 45707c72..64e53e13 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -1,7 +1,7 @@ # Platform support and validation Implementation, development acceptance and release qualification are separate. -This page summarizes the recorded development baselines as of 2026-11-05; +This page summarizes the recorded development baselines as of 2026-11-11; the [v1 release profile](release/v1-profile.md) defines the reviewed binary release boundary. A newer source checkout does not update an installed SDK. @@ -164,8 +164,8 @@ the complete admission rules. and pass the installed standard-library and debugger/checkpoint campaigns. The optional isolated PostgreSQL 16 profile now passes its coherent client/server and installed source/binary routes on both architectures; - the scoped-address validation correction passes on both hosts. The complete - corrected Intel candidate remains pending after a source-campaign timeout. + the scoped-address validation correction passes on both hosts. The corrected + Intel retry at `0c88d2f` completes all installed campaigns, as recorded above. - The two native lifetime stress tests still lack a demonstrated Darwin memory bound. Native diagnostics show that the fixed 96 MiB virtual-memory cap is rejected at process setup; larger diagnostic profiles do not replace it. @@ -179,19 +179,13 @@ the complete admission rules. real window server session on both hosted architectures, 93 checks each, and `nil.ui-display` now passes there too: the provider fixture, the lifecycle fixture and the Matrix desktop suite. Real keyboard input is - covered in process, because macOS requires accessibility trust to post - events to another process and a hosted runner does not grant it; that is a - limit of cross-process injection, not of the provider, so the interaction - suite is Linux and Windows only. GTK on macOS is retired, because a GTK - process there never runs the AppKit event loop and so has no accessibility - hierarchy at all. Both hosted Mac architectures - now pass the prerequisite GLib, Cairo-Quartz and HarfBuzz/CoreText compile, - link and runtime checks, plus exact own-process AppKit key dispatch, view - delivery and window closure. Directed keyboard input to a verified child and - native accessibility closure also pass on both hosts. Both have built and - installed the private GTK dependency stack and passed actual widget input, - CoreText/font layout, PNG decoding and window teardown checks. Integration - with the product runtime and language-level desktop acceptance remain open. + covered in process by the AppKit provider fixture. The current AppKit + campaign does not qualify cross-process keyboard injection or accessibility; + the language interaction suite currently runs on Linux and Windows. Earlier + private GTK and AppKit prerequisite probes are retained as historical evidence, + but GTK is no longer the macOS product provider. The language-level AppKit + display campaign uses LLVM OFF and therefore establishes the interpreter route, + not native/JIT application or editor acceptance. - Editor desktop integration, untested provider configurations and release-package acceptance remain separate from the passing installed campaigns. Shipping a Mach-O tool alone does not qualify these surfaces. @@ -199,3 +193,14 @@ the complete admission rules. The [macOS workflow](../.github/workflows/macos-bringup.yml) records the complete inventory, including failures and skips. The [release gates](release/v1-profile.md#qualification-and-publication) still apply before publishing a macOS SDK or declaring parity. + +## New web-runtime APIs + +The socket, secure-random and text-byte additions in the open web-runtime branch +are currently interpreter-backed. File timestamps share a stable native-metadata +implementation across interpreter and native routes: Linux text NIL, binary NIL, +JIT and AOT all pass unchanged-file and changed-metadata checks. Linux results +alone do not establish Windows or macOS support, and declarations do not establish +native/JIT execution. Cross-platform validation of these additions is in progress. +Their evidence must join the older desktop, tooling and SDK baselines before +declaring current-source parity. diff --git a/docs/standard-library/random.md b/docs/standard-library/random.md index 83748494..11db0eda 100644 --- a/docs/standard-library/random.md +++ b/docs/standard-library/random.md @@ -127,8 +127,10 @@ modulo the bound skews towards the low values whenever the bound does not divide the range evenly — a bias small enough never to be noticed and large enough to matter to anything that was counting on uniformity. -These ask the operating system and do nothing else: `getrandom(2)` on Linux -falling back to `/dev/urandom`, and the CRT's `rand_s` on Windows. There is no +These use the operating system: `getrandom(2)` on Linux with `/dev/urandom` +fallback, `getentropy` on macOS, and `BCryptGenRandom` with the system-preferred +generator on Windows. The Windows implementation resolves the OS copy of +`bcrypt.dll` explicitly. There is no userspace pool, no seeding and no state. There is also no fallback to a weak source — if the kernel cannot provide entropy the answer is an error, because quietly returning something guessable is how this goes wrong in a way nobody diff --git a/docs/standard-library/sockets.md b/docs/standard-library/sockets.md index b2d9e350..f0e9fe93 100644 --- a/docs/standard-library/sockets.md +++ b/docs/standard-library/sockets.md @@ -9,6 +9,7 @@ compiled through it will not find them. | Ny API | Behavior | | --- | --- | | `listen(host: text, port: i32, backlog: i32) -> Result` | Open, bind and listen in one step; gives back a handle. | +| `listen_reuse_port(host: text, port: i32, backlog: i32) -> Result` | Opt into POSIX port sharing before binding; unsupported on Windows. | | `accept(handle: i64) -> Result` | Block until a connection arrives; gives back a handle to it. | | `connect(host: text, port: i32) -> Result` | Connect out, resolving a name if it is one. | | `read(handle: i64, maximum_bytes: i64) -> Result` | Read what has arrived, up to a ceiling. | @@ -19,7 +20,7 @@ compiled through it will not find them. | `set_no_delay(handle: i64, enabled: bool) -> Result` | Send small writes immediately. | | `peer_address(handle: i64) -> Result` | The peer's address and port, as text. | | `set_blocking(handle: i64, blocking: bool) -> Result` | Whether a call waits when there is nothing to do. | -| `set_reuse_port(handle: i64, enabled: bool) -> Result` | Let other processes bind this same port. | +| `set_reuse_port(handle: i64, enabled: bool) -> Result` | Set the platform option on an existing socket; use `listen_reuse_port` when opening shared listeners. | | `watch(handle: i64, flags: i64) -> Result` | Watch a socket for readability, writability, or both. | | `unwatch(handle: i64) -> Result` | Stop watching. | | `watch_count() -> i64` | How many sockets are watched. | @@ -49,7 +50,10 @@ received is refused with a sentence rather than used. A listener and a connection are told apart: accepting on a connection, or reading from a listener, is a named error and not an `EINVAL`. -The runtime closes whatever a run leaves open. A program that exits without +The runtime closes whatever a run leaves open at normal completion, an uncaught +runtime error, reset, program replacement or VM destruction. Tokens are not +reissued during a VM's lifetime, and its watch set, readiness queue and receive +buffer are cleared with the invocation. A program that exits without closing its listener — or one that crashes — does not leave the port bound against the next run, which during development is the difference between a server that restarts and one that fails every second time for a reason @@ -70,19 +74,23 @@ IPv4 and an IPv6 address is tried in turn and the first to connect wins. TCP is a stream. A request may arrive in several pieces, so a read returning less than asked for is normal. An orderly close by the peer comes back as -empty text rather than an error, so a loop reading to the end of a request -sees one shape; `is_open` distinguishes them when it matters. +empty text rather than an error. A timeout or a non-blocking `read` with no data +returns an error. `is_open` reports ownership of the local handle; it does not +detect whether the peer has closed. Use `receive` to distinguish would-block +from EOF in an event loop. What `read` gives back is bytes, not necessarily valid UTF-8 — it is whatever the peer sent. Read it with `nil.std.text`'s `byte_at` rather than `substring` if it may not be text. One call never transfers more than 8 MiB. -## Writes complete or report how far they got +## Writes complete or report an error A single `send` is permitted to take less than it was given. `write` loops until everything is gone or the attempt fails, because the number of servers that have shipped a truncated response from the one-call version is large. On -failure the count says how far it reached. +failure, `write` returns an error and may already have sent part of the data; +its public error does not contain that partial count. Use `send_some` and retain +the offset when the caller needs to track progress. A peer that closes mid-write is reported rather than fatal: the runtime suppresses the signal whose default disposition would end the process. A @@ -93,7 +101,11 @@ case that a test which never closes a connection early will not catch. A single-threaded server without `set_timeout` is one slow client away from serving nobody: a connection that opens and then says nothing holds the accept -loop for as long as it likes. Set a deadline on every accepted connection. +loop for as long as it likes. Set a timeout on every accepted blocking connection. +`set_timeout` accepts 0 through 2,147,483,647 milliseconds and applies to individual +socket operations; it is not an end-to-end HTTP request deadline. Finite `wait` +values have the same maximum and retain their deadline across interruptions. +Negative `wait` values wait indefinitely. An empty watch set returns immediately. ## An echo server @@ -144,12 +156,10 @@ globals, with no lock anywhere near them, and `nil.std.threading` offers only `sleep` — so concurrency here is `poll` telling you which sockets can be acted on, and one pass round a loop acting on each in turn. -That is nginx's shape rather than Apache's prefork, and it is the one this -runtime can keep. It is also the one that scales: 64 simultaneous connections, -each sending its request in two parts with a quarter-second pause between, are -served in about one pause rather than sixty-four of them. The contract test -asserts exactly that difference, because a throughput figure would measure the -machine while this measures the shape. +The contract test opens 64 simultaneous connections and sends the first part of +each request. It releases the remaining parts only after the server confirms it +has processed every first part. This barrier proves that one incomplete request +does not prevent the others from making progress; it is not a throughput claim. The watch set lives in the runtime rather than being passed in per call, so no array crosses the intrinsic boundary. That is not tidiness: a Matrix page @@ -204,13 +214,17 @@ connection is dropped, or the last request of a session is lost. ### More than one core A second core means a second process, because a second thread would need an -interpreter that could survive one. `set_reuse_port` lets several processes -bind the same port and the kernel load-balances accepts between them. - -It is opt-in and distinct from the `SO_REUSEADDR` that `listen` always sets: -that one only allows rebinding a port left in `TIME_WAIT`, while this one -deliberately allows two servers to share a port. Switched on by accident it -would silently split traffic between a running server and a stray one. +interpreter that could survive one. On POSIX hosts, every participating listener +must use `listen_reuse_port`, which sets `SO_REUSEPORT` before binding. Setting +the option after `listen` cannot make a failed second bind succeed. Connection +distribution follows the host's socket implementation; this API does not promise +a particular scheduling policy. + +Sharing is opt-in. Ordinary POSIX listeners use `SO_REUSEADDR` for restart +behavior; ordinary Windows listeners use `SO_EXCLUSIVEADDRUSE` to reject a +competing bind. Windows does not implement this port-sharing constructor and +returns an explicit error. It still supports the single-process event loop and +multiple simultaneous connections. ## What is not here diff --git a/docs/standard-library/ui.md b/docs/standard-library/ui.md index a02c4155..d835a284 100644 --- a/docs/standard-library/ui.md +++ b/docs/standard-library/ui.md @@ -5,10 +5,10 @@ events and explicit model bindings. Ny programs can import the same module. The VM and native compiler use one desktop provider per platform, each behind the same checked boundary: a GTK4 provider on Linux and a Win32 provider on Windows. Windows remains a development target, not a qualified release platform. -The macOS provider is being integrated; AppKit replaces the earlier GTK-on-macOS -work, because a GTK process on macOS never runs the AppKit event loop and is -therefore invisible to the accessibility APIs that assistive technology and -directed input depend on. See [platform support](../platform-support.md). +The macOS provider is AppKit. Its provider, lifecycle and interpreter +language-level desktop contracts pass on Intel and Apple Silicon; current SDK, +native/JIT application and editor acceptance remain separate. AppKit replaces the +earlier private GTK-on-macOS experiments. See [platform support](../platform-support.md). Examples display the framework generation as **NIL²**, with **NIL^2** as the plain-text spelling. This presentation does not change numeric release or artifact identities in `VERSION`. @@ -65,16 +65,14 @@ Because the provider calls Cocoa and CoreGraphics directly, every linked program carries those frameworks, which is why they appear in the Darwin link lines alongside the runtime. -GTK is no longer a macOS provider. A GTK process on macOS never runs -`-[NSApplication run]`, and AppKit answers accessibility requests from the main -run loop, so such a process has no accessibility hierarchy at all: every -accessibility query against it fails, including the application element's own -role, which makes it invisible to assistive technology and to directed input. +GTK is no longer a macOS provider. The earlier private GTK profile failed its +cross-process accessibility probes; those observations do not qualify every +possible GTK configuration. AppKit is the supported source implementation. On macOS, creation and polling must run on the process main thread, which is where AppKit requires its windows to be used. Ordinary interpreter, JIT and -native entry routes use the main thread; UI behaviour through these routes -still needs the desktop acceptance campaign. DAP currently executes the +native entry routes use the main thread. The interpreter display campaign passes; +native/JIT application acceptance still needs its own campaign. DAP currently executes the debuggee on a worker, so DAP with macOS UI is outside this port, and pausing a text or JSON debug session does not automatically pump window events. AppKit supplies no box or grid container, so, as on Win32, the provider owns the diff --git a/nilcompiler/CMakeLists.txt b/nilcompiler/CMakeLists.txt index 73733a5e..85605901 100644 --- a/nilcompiler/CMakeLists.txt +++ b/nilcompiler/CMakeLists.txt @@ -45,8 +45,8 @@ set(NILCOMPILER_CORE_SOURCES src/executable_context.cpp ) -# One desktop provider per platform: GTK is the Linux (and macOS) provider, -# Win32 is the Windows provider. Both implement the same ui.v1 contract and +# One desktop provider per platform: GTK on Linux, Win32 on Windows, and +# AppKit on macOS. All implement the same ui.v1 contract and # share its toolkit-independent half through src/ui_runtime_shared.hpp, so # exactly one of them is ever compiled into the runtime. if(WIN32) diff --git a/nilcompiler/src/entropy_support.hpp b/nilcompiler/src/entropy_support.hpp index 2edb9db0..af708092 100644 --- a/nilcompiler/src/entropy_support.hpp +++ b/nilcompiler/src/entropy_support.hpp @@ -24,13 +24,18 @@ #include #include +#include #include #if defined(_WIN32) -// rand_s is the CRT's wrapper over RtlGenRandom. It needs no extra library, -// which keeps this from changing how the project links on Windows. -#define _CRT_RAND_S -#include +#ifndef NOMINMAX +#define NOMINMAX +#endif +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include +#include #else #include #include @@ -90,6 +95,21 @@ inline bool fill_from_kernel(unsigned char *into, std::size_t count, } if (syscall_usable && filled == count) return true; +#elif defined(__APPLE__) + // Darwin supplies getentropy without opening a device. Each call accepts at + // most 256 bytes; a failed OS request never falls back to a userspace PRNG. + std::size_t filled = 0; + while (filled < count) { + const std::size_t chunk = count - filled > 256 ? 256 : count - filled; + if (::getentropy(into + filled, chunk) == 0) { + filled += chunk; + continue; + } + if (errno == EINTR) continue; + error = "random: no entropy source is available on this system"; + return false; + } + return true; #endif // The device, for a kernel without the syscall. Opened and closed per call: @@ -113,17 +133,21 @@ inline bool fill_from_kernel(unsigned char *into, std::size_t count, inline bool fill_from_kernel(unsigned char *into, std::size_t count, std::string &error) { - std::size_t filled = 0; - while (filled < count) { - unsigned int drawn = 0; - if (::rand_s(&drawn) != 0) { - error = "random: no entropy source is available on this system"; - return false; - } - for (int byte = 0; byte < 4 && filled < count; ++byte) { - into[filled] = static_cast((drawn >> (byte * 8)) & 0xFF); - filled += 1; - } + // Resolve the system DLL explicitly. This avoids a CRT include-order + // dependency and does not add a loader search through the working directory. + using Generate = NTSTATUS (WINAPI *)(BCRYPT_ALG_HANDLE, PUCHAR, ULONG, ULONG); + static const Generate generate = []() -> Generate { + const HMODULE library = ::LoadLibraryExW(L"bcrypt.dll", nullptr, LOAD_LIBRARY_SEARCH_SYSTEM32); + if (!library) return nullptr; + const auto result = reinterpret_cast(::GetProcAddress(library, "BCryptGenRandom")); + if (!result) ::FreeLibrary(library); + // Keep a successful module loaded for the lifetime of the function pointer. + return result; + }(); + if (count > (std::numeric_limits::max)() || !generate || + generate(nullptr, into, static_cast(count), BCRYPT_USE_SYSTEM_PREFERRED_RNG) < 0) { + error = "random: no entropy source is available on this system"; + return false; } return true; } diff --git a/nilcompiler/src/file_time_support.hpp b/nilcompiler/src/file_time_support.hpp new file mode 100644 index 00000000..0d36d8e0 --- /dev/null +++ b/nilcompiler/src/file_time_support.hpp @@ -0,0 +1,89 @@ +#pragma once + +#include "../../common/strict_json.hpp" +#include +#include +#include +#include + +#ifdef _WIN32 +#include "../../common/windows_environment.hpp" +#else +#include +#endif + +namespace ny::file_time_support { + +// Floor to Unix milliseconds, including times before 1970. No clock readings +// participate in this conversion: unchanged metadata has an unchanged result. +inline std::optional from_unix_parts(std::int64_t seconds, + std::int64_t nanoseconds) { + if (nanoseconds < 0 || nanoseconds >= 1000000000) return std::nullopt; + const auto fraction = static_cast(nanoseconds / 1000000); + constexpr auto maximum = static_cast( + (std::numeric_limits::max)()); + if (seconds >= 0) { + const auto whole = static_cast(seconds); + if (whole > maximum / 1000) return std::nullopt; + const auto base = whole * 1000; + if (fraction > maximum - base) return std::nullopt; + return static_cast(base + fraction); + } + const auto magnitude = static_cast(-(seconds + 1)) + 1; + constexpr auto minimum_magnitude = maximum + 1; + if (magnitude > (minimum_magnitude + 999) / 1000) return std::nullopt; + const auto result = magnitude * 1000 - fraction; + if (result > minimum_magnitude) return std::nullopt; + if (result == minimum_magnitude) return (std::numeric_limits::min)(); + return -static_cast(result); +} + +// FILETIME is an unsigned count of 100 ns ticks since 1601. The epoch offset +// is exactly divisible by a millisecond, so division before subtraction also +// floors negative Unix timestamps correctly and cannot overflow an i64. +inline std::int64_t from_windows_ticks(std::uint64_t ticks) { + constexpr std::int64_t epoch_milliseconds = 11644473600000LL; + return static_cast(ticks / 10000) - epoch_milliseconds; +} + +// Follow symlinks as last_write_time did, and retain directory timestamps. +// Text paths are strict UTF-8 on every host; never pass an embedded NUL to the +// OS. This API intentionally returns no path-bearing host diagnostic. +inline std::optional modified_at(const std::string &path) { + if (path.empty() || path.find('\0') != std::string::npos) return std::nullopt; + try { + nil::detail::valid_utf8(path); +#ifdef _WIN32 + const auto wide = nil::detail::environment_utf16(path); + if (!wide) return std::nullopt; + const HANDLE file = CreateFileW(wide->c_str(), FILE_READ_ATTRIBUTES, + FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, nullptr, + OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, nullptr); + if (file == INVALID_HANDLE_VALUE) return std::nullopt; + FILETIME written{}; + const bool obtained = GetFileTime(file, nullptr, nullptr, &written) != FALSE; + const bool closed = CloseHandle(file) != FALSE; + if (!obtained || !closed) return std::nullopt; + const auto ticks = (static_cast(written.dwHighDateTime) << 32) | + written.dwLowDateTime; + return from_windows_ticks(ticks); +#else + struct stat state{}; + if (::stat(path.c_str(), &state) != 0) return std::nullopt; +#ifdef __APPLE__ + const auto written = state.st_mtimespec; +#else + const auto written = state.st_mtim; +#endif + static_assert(std::numeric_limits::is_signed && + std::numeric_limits::digits <= 63, + "file timestamps require a signed at-most-64-bit time_t"); + return from_unix_parts(static_cast(written.tv_sec), + static_cast(written.tv_nsec)); +#endif + } catch (...) { + return std::nullopt; + } +} + +} // namespace ny::file_time_support diff --git a/nilcompiler/src/native_runtime.cpp b/nilcompiler/src/native_runtime.cpp index 974913c2..2a1a0d5c 100644 --- a/nilcompiler/src/native_runtime.cpp +++ b/nilcompiler/src/native_runtime.cpp @@ -14,6 +14,7 @@ #include "secure_file_io.hpp" #include "time_support.hpp" +#include "file_time_support.hpp" #include "parallel_support.hpp" #include "owned_runtime.hpp" @@ -1875,6 +1876,13 @@ std::int32_t nyrt_io_v1(const char *operation, std::uint64_t first, } } + if (name == "file.modified_at") { + const auto written = ny::file_time_support::modified_at(text(first)); + if (!written) return fail("the file has no readable timestamp"); + *value = static_cast(*written); + return 1; + } + const auto path = host_path(first); std::error_code code; if (name == "file.exists") { diff --git a/nilcompiler/src/net_support.hpp b/nilcompiler/src/net_support.hpp index 4ce03cdb..6657a415 100644 --- a/nilcompiler/src/net_support.hpp +++ b/nilcompiler/src/net_support.hpp @@ -16,23 +16,30 @@ // // Three decisions are worth stating because they are the ones that bite: // -// * Writes use MSG_NOSIGNAL. Without it, a peer that closes mid-write -// delivers SIGPIPE and the default disposition kills the process. A web -// server that dies when a browser closes a tab is not a web server. -// * A listening socket sets SO_REUSEADDR but never SO_REUSEPORT. The first -// lets a restart rebind a port still in TIME_WAIT; the second would let -// two unrelated servers share one port and silently split the traffic. +// * Writes suppress SIGPIPE per socket on Darwin and per send on Linux. +// * Ordinary listeners are exclusive on Windows and use SO_REUSEADDR on +// POSIX. Sharing a port requires the explicit pre-bind constructor. // * Binding never resolves a name through DNS. A server binds an address it // already knows, and a DNS lookup on that path is a blocking call plus a // way to end up listening somewhere unintended. `localhost` is understood // directly; everything else must be numeric. Connecting does resolve, // because a client is given names. +#include +#include #include +#include +#include #include #include #if defined(_WIN32) +#ifndef NOMINMAX +#define NOMINMAX +#endif +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif #include #include #else @@ -53,6 +60,19 @@ namespace ny::net_support { +#if defined(_WIN32) +using socket_descriptor = SOCKET; +constexpr socket_descriptor invalid_socket = INVALID_SOCKET; +static_assert(sizeof(socket_descriptor) == sizeof(std::uintptr_t), + "Winsock handles require pointer-width storage"); +#else +using socket_descriptor = int; +constexpr socket_descriptor invalid_socket = -1; +#endif + +constexpr std::int64_t max_timeout_ms = 2147483647; + + /** One read never takes more than this, however much the caller asks for. */ constexpr std::int64_t max_transfer_bytes = 8LL * 1024 * 1024; @@ -60,7 +80,7 @@ constexpr std::int64_t max_transfer_bytes = 8LL * 1024 * 1024; constexpr std::int64_t max_backlog = 1024; struct Opened { - int descriptor = -1; + socket_descriptor descriptor = invalid_socket; std::string error; /** * Nothing to do right now, on a socket that was asked not to block. @@ -71,7 +91,7 @@ struct Opened { * server log a line every time a port scanner connected and left. */ bool would_block = false; - bool ok() const { return descriptor >= 0 && error.empty(); } + bool ok() const { return descriptor != invalid_socket && error.empty(); } }; struct Transferred { @@ -91,26 +111,25 @@ struct Transferred { // WSAStartup/WSACleanup pair means a library may start it without taking the // decision away from whatever else in the process also needs it. inline bool ensure_started(std::string &error) { - static bool started = false; - static std::string start_error; - if (!started && start_error.empty()) { - WSADATA data; - const int result = WSAStartup(MAKEWORD(2, 2), &data); - if (result != 0) - start_error = "net: Winsock could not be started"; - else - started = true; - } - if (!start_error.empty()) { - error = start_error; - return false; - } - return true; + struct Startup { + int result; + Startup() : result(0) { + WSADATA data{}; + result = WSAStartup(MAKEWORD(2, 2), &data); + } + ~Startup() { if (result == 0) WSACleanup(); } + }; + static const Startup startup; + if (startup.result == 0) return true; + error = "net: Winsock could not be started (" + std::to_string(startup.result) + ")"; + return false; } -inline std::string last_error_text() { return "net: the socket call failed"; } -inline int close_descriptor(int descriptor) { - return ::closesocket(static_cast(descriptor)); +inline std::string last_error_text() { + return "net: the socket call failed (" + std::to_string(::WSAGetLastError()) + ")"; +} +inline int close_descriptor(socket_descriptor descriptor) { + return ::closesocket(descriptor); } using transfer_size = int; constexpr int no_signal_flag = 0; @@ -125,7 +144,7 @@ inline std::string last_error_text() { return std::string("net: ") + (text ? text : "the socket call failed"); } -inline int close_descriptor(int descriptor) { return ::close(descriptor); } +inline int close_descriptor(socket_descriptor descriptor) { return ::close(descriptor); } using transfer_size = ssize_t; #if defined(MSG_NOSIGNAL) constexpr int no_signal_flag = MSG_NOSIGNAL; @@ -135,6 +154,25 @@ constexpr int no_signal_flag = 0; #endif +/** Apply process-local socket safety before exposing the descriptor. */ +inline std::string prepare_socket(socket_descriptor descriptor) { +#if defined(SO_NOSIGPIPE) + const int enabled = 1; + if (::setsockopt(descriptor, SOL_SOCKET, SO_NOSIGPIPE, + reinterpret_cast(&enabled), sizeof(enabled)) != 0) + return last_error_text(); +#endif +#if defined(_WIN32) + if (!::SetHandleInformation(reinterpret_cast(descriptor), HANDLE_FLAG_INHERIT, 0)) + return "net: the socket could not be made non-inheritable"; +#else + const int flags = ::fcntl(descriptor, F_GETFD); + if (flags < 0 || ::fcntl(descriptor, F_SETFD, flags | FD_CLOEXEC) < 0) + return last_error_text(); +#endif + return {}; +} + /** Whether the last call failed only because it would have blocked. */ inline bool last_call_would_block() { #if defined(_WIN32) @@ -155,6 +193,10 @@ inline bool last_call_would_block() { */ inline bool fill_bind_address(const std::string &host, std::int64_t port, sockaddr_in &into, std::string &error) { + if (host.find('\0') != std::string::npos) { + error = "net: a host must not contain a NUL byte"; + return false; + } if (port < 1 || port > 65535) { error = "net: a port must be between 1 and 65535"; return false; @@ -180,7 +222,7 @@ inline bool fill_bind_address(const std::string &host, std::int64_t port, /** Open, bind and listen in one step, because no caller wants them apart. */ inline Opened listen_on(const std::string &host, std::int64_t port, - std::int64_t backlog) { + std::int64_t backlog, bool share_port = false) { Opened opened; if (!ensure_started(opened.error)) return opened; @@ -189,15 +231,39 @@ inline Opened listen_on(const std::string &host, std::int64_t port, if (!fill_bind_address(host, port, address, opened.error)) return opened; - const int descriptor = static_cast(::socket(AF_INET, SOCK_STREAM, 0)); - if (descriptor < 0) { +#if !defined(SO_REUSEPORT) || defined(_WIN32) + if (share_port) { + opened.error = "net: this platform has no shared-listener SO_REUSEPORT support"; + return opened; + } +#endif + const socket_descriptor descriptor = ::socket(AF_INET, SOCK_STREAM, 0); + if (descriptor == invalid_socket) { opened.error = last_error_text(); return opened; } + opened.error = prepare_socket(descriptor); const int reuse = 1; - ::setsockopt(descriptor, SOL_SOCKET, SO_REUSEADDR, - reinterpret_cast(&reuse), sizeof(reuse)); +#if defined(_WIN32) + constexpr int reuse_option = SO_EXCLUSIVEADDRUSE; +#else + constexpr int reuse_option = SO_REUSEADDR; +#endif + if (opened.error.empty() && + ::setsockopt(descriptor, SOL_SOCKET, reuse_option, + reinterpret_cast(&reuse), sizeof(reuse)) != 0) + opened.error = last_error_text(); +#if defined(SO_REUSEPORT) && !defined(_WIN32) + if (opened.error.empty() && share_port && + ::setsockopt(descriptor, SOL_SOCKET, SO_REUSEPORT, + reinterpret_cast(&reuse), sizeof(reuse)) != 0) + opened.error = last_error_text(); +#endif + if (!opened.error.empty()) { + close_descriptor(descriptor); + return opened; + } if (::bind(descriptor, reinterpret_cast(&address), sizeof(address)) != 0) { @@ -220,14 +286,16 @@ inline Opened listen_on(const std::string &host, std::int64_t port, } /** Wait for a connection. Retries an interruption rather than reporting one. */ -inline Opened accept_on(int listening) { +inline Opened accept_on(socket_descriptor listening) { Opened opened; if (!ensure_started(opened.error)) return opened; for (;;) { - const int descriptor = static_cast(::accept(listening, nullptr, nullptr)); - if (descriptor >= 0) { - opened.descriptor = descriptor; + const socket_descriptor descriptor = ::accept(listening, nullptr, nullptr); + if (descriptor != invalid_socket) { + opened.error = prepare_socket(descriptor); + if (opened.error.empty()) opened.descriptor = descriptor; + else close_descriptor(descriptor); return opened; } #if !defined(_WIN32) @@ -254,6 +322,10 @@ inline Opened connect_to(const std::string &host, std::int64_t port) { Opened opened; if (!ensure_started(opened.error)) return opened; + if (host.find('\0') != std::string::npos) { + opened.error = "net: a host must not contain a NUL byte"; + return opened; + } if (port < 1 || port > 65535) { opened.error = "net: a port must be between 1 and 65535"; return opened; @@ -277,10 +349,15 @@ inline Opened connect_to(const std::string &host, std::int64_t port) { std::string attempt_error = "net: no address accepted the connection"; for (addrinfo *candidate = found; candidate != nullptr; candidate = candidate->ai_next) { - const int descriptor = static_cast(::socket( - candidate->ai_family, candidate->ai_socktype, candidate->ai_protocol)); - if (descriptor < 0) + const socket_descriptor descriptor = ::socket( + candidate->ai_family, candidate->ai_socktype, candidate->ai_protocol); + if (descriptor == invalid_socket) + continue; + attempt_error = prepare_socket(descriptor); + if (!attempt_error.empty()) { + close_descriptor(descriptor); continue; + } if (::connect(descriptor, candidate->ai_addr, static_cast(candidate->ai_addrlen)) == 0) { ::freeaddrinfo(found); @@ -303,7 +380,7 @@ inline Opened connect_to(const std::string &host, std::int64_t port) { * apart from a failure, because a caller reading until the peer finishes needs * to know which happened and both return no data. */ -inline Transferred read_some(int descriptor, std::int64_t wanted) { +inline Transferred read_some(socket_descriptor descriptor, std::int64_t wanted) { Transferred result; if (!ensure_started(result.error)) return result; @@ -350,7 +427,7 @@ inline Transferred read_some(int descriptor, std::int64_t wanted) { * event loop. This takes what it can, reports it, and leaves the rest to the * caller to send when poll says the socket will accept more. */ -inline Transferred write_some(int descriptor, const std::string &data, +inline Transferred write_some(socket_descriptor descriptor, const std::string &data, std::size_t offset) { Transferred result; if (!ensure_started(result.error)) @@ -361,7 +438,8 @@ inline Transferred write_some(int descriptor, const std::string &data, for (;;) { const transfer_size taken = ::send(descriptor, data.data() + offset, - static_cast(data.size() - offset), no_signal_flag); + static_cast(std::min(data.size() - offset, + static_cast(max_transfer_bytes))), no_signal_flag); if (taken > 0) { result.count = static_cast(taken); return result; @@ -386,7 +464,7 @@ inline Transferred write_some(int descriptor, const std::string &data, * permitted to take less than it was given, and the number of servers that * have shipped a truncated response because nobody checked is large. */ -inline Transferred write_all(int descriptor, const std::string &data) { +inline Transferred write_all(socket_descriptor descriptor, const std::string &data) { Transferred result; if (!ensure_started(result.error)) return result; @@ -396,7 +474,8 @@ inline Transferred write_all(int descriptor, const std::string &data) { const std::size_t remaining = data.size() - sent; const transfer_size taken = ::send(descriptor, data.data() + sent, - static_cast(remaining), no_signal_flag); + static_cast(std::min(remaining, + static_cast(max_transfer_bytes))), no_signal_flag); if (taken > 0) { sent += static_cast(taken); continue; @@ -420,12 +499,12 @@ inline Transferred write_all(int descriptor, const std::string &data) { * serving nobody: a connection that opens and then says nothing holds the * accept loop forever. Zero removes the deadline again. */ -inline std::string set_timeout(int descriptor, std::int64_t milliseconds) { +inline std::string set_timeout(socket_descriptor descriptor, std::int64_t milliseconds) { std::string error; if (!ensure_started(error)) return error; - if (milliseconds < 0) - return "net: a timeout cannot be negative"; + if (milliseconds < 0 || milliseconds > max_timeout_ms) + return "net: a timeout must be between 0 and 2147483647 milliseconds"; #if defined(_WIN32) const DWORD value = static_cast(milliseconds); @@ -446,7 +525,7 @@ inline std::string set_timeout(int descriptor, std::int64_t milliseconds) { } /** The peer's address as text, for a log line. Never part of a decision. */ -inline Transferred peer_address(int descriptor) { +inline Transferred peer_address(socket_descriptor descriptor) { Transferred result; if (!ensure_started(result.error)) return result; @@ -500,7 +579,7 @@ constexpr std::int64_t ready_finished = 4; /** One watched socket and what it is being watched for. */ struct Watched { std::int64_t token = 0; - int descriptor = -1; + socket_descriptor descriptor = invalid_socket; std::int64_t wanted = ready_readable; }; @@ -519,13 +598,13 @@ struct Ready { * finds nothing — a spurious wakeup, a connection withdrawn between the poll * and the accept — would stop the whole loop on behalf of one socket. */ -inline std::string set_blocking(int descriptor, bool blocking) { +inline std::string set_blocking(socket_descriptor descriptor, bool blocking) { std::string error; if (!ensure_started(error)) return error; #if defined(_WIN32) u_long mode = blocking ? 0 : 1; - if (::ioctlsocket(static_cast(descriptor), FIONBIO, &mode) != 0) + if (::ioctlsocket(descriptor, FIONBIO, &mode) != 0) return last_error_text(); #else const int flags = ::fcntl(descriptor, F_GETFL, 0); @@ -539,24 +618,16 @@ inline std::string set_blocking(int descriptor, bool blocking) { } /** - * Let several processes bind the same port. - * - * The missing half of running on more than one core. This runtime is - * single-threaded by construction — the interpreter holds one stack, one heap - * and one set of globals, with no lock anywhere near them — so a second core - * means a second process, and two processes cannot share a listening socket - * without this. The kernel then load-balances accepts between them. - * - * Distinct from SO_REUSEADDR, which `listen_on` always sets and which only - * allows rebinding a port in TIME_WAIT. This one deliberately allows two - * servers to share a port, so it is opt-in: switched on by accident it would - * silently split traffic between a running server and a stray one. + * Change SO_REUSEPORT on an existing socket where the host supports it. + * This cannot configure a second listener before bind: shared listeners must + * instead use listen_on(..., true) on every participating process. Windows + * rejects enabling this option rather than emulating it with SO_REUSEADDR. */ -inline std::string set_reuse_port(int descriptor, bool enabled) { +inline std::string set_reuse_port(socket_descriptor descriptor, bool enabled) { std::string error; if (!ensure_started(error)) return error; -#if defined(SO_REUSEPORT) +#if defined(SO_REUSEPORT) && !defined(_WIN32) const int value = enabled ? 1 : 0; if (::setsockopt(descriptor, SOL_SOCKET, SO_REUSEPORT, reinterpret_cast(&value), sizeof(value)) != 0) @@ -589,8 +660,12 @@ inline std::string poll_wait(const std::vector &watching, std::string error; if (!ensure_started(error)) return error; + if (timeout_ms > max_timeout_ms) + return "net: a finite wait cannot exceed 2147483647 milliseconds"; if (watching.empty()) return {}; + if (watching.size() > static_cast((std::numeric_limits::max)())) + return "net: too many watched sockets"; #if defined(_WIN32) std::vector entries; @@ -599,9 +674,11 @@ inline std::string poll_wait(const std::vector &watching, #endif entries.reserve(watching.size()); for (const auto &one : watching) { + if (one.wanted <= 0 || (one.wanted & ~(ready_readable | ready_writable)) != 0) + return "net: watch flags must select readability, writability, or both"; #if defined(_WIN32) WSAPOLLFD entry{}; - entry.fd = static_cast(one.descriptor); + entry.fd = one.descriptor; #else pollfd entry{}; entry.fd = one.descriptor; @@ -614,24 +691,27 @@ inline std::string poll_wait(const std::vector &watching, entries.push_back(entry); } - int signalled = 0; + using Clock = std::chrono::steady_clock; + const auto deadline = Clock::now() + std::chrono::milliseconds(timeout_ms < 0 ? 0 : timeout_ms); + int remaining = timeout_ms < 0 ? -1 : static_cast(timeout_ms); for (;;) { #if defined(_WIN32) - signalled = ::WSAPoll(entries.data(), static_cast(entries.size()), - static_cast(timeout_ms)); + const int signalled = ::WSAPoll(entries.data(), static_cast(entries.size()), remaining); #else - signalled = ::poll(entries.data(), static_cast(entries.size()), - static_cast(timeout_ms)); + const int signalled = ::poll(entries.data(), static_cast(entries.size()), remaining); #endif - if (signalled >= 0) - break; -#if !defined(_WIN32) - // An interrupted wait is not a failure. Reporting one would make an - // unrelated signal look to the server like a broken socket. - if (errno == EINTR) - continue; + if (signalled >= 0) break; +#if defined(_WIN32) + const bool interrupted = ::WSAGetLastError() == WSAEINTR; +#else + const bool interrupted = errno == EINTR; #endif - return last_error_text(); + if (!interrupted) return last_error_text(); + if (timeout_ms >= 0) { + const auto left = deadline - Clock::now(); + if (left <= Clock::duration::zero()) return {}; + remaining = static_cast(std::chrono::ceil(left).count()); + } } for (std::size_t index = 0; index < entries.size(); ++index) { @@ -656,7 +736,7 @@ inline std::string poll_wait(const std::vector &watching, } /** Stop a write-blocked peer from keeping a response in a buffer. */ -inline std::string set_no_delay(int descriptor, bool enabled) { +inline std::string set_no_delay(socket_descriptor descriptor, bool enabled) { std::string error; if (!ensure_started(error)) return error; diff --git a/nilcompiler/src/ordinary_io_intrinsic.hpp b/nilcompiler/src/ordinary_io_intrinsic.hpp index 1fd115d0..1689f5b6 100644 --- a/nilcompiler/src/ordinary_io_intrinsic.hpp +++ b/nilcompiler/src/ordinary_io_intrinsic.hpp @@ -32,7 +32,8 @@ inline std::optional intrinsic(const std::string &name) { return Contract{{"text"}, "bool"}; if (name == "file.read_text") return Contract{{"text"}, "text", Output::result}; if (name == "file.read_bytes") return Contract{{"text"}, "array", Output::result}; - if (name == "file.size") return Contract{{"text"}, "i64", Output::result}; + if (name == "file.size" || name == "file.modified_at") + return Contract{{"text"}, "i64", Output::result}; if (name == "file.delete" || name == "environment.unset" || name == "environment.set_current_directory") return Contract{{"text"}, "bool", Output::result}; diff --git a/nilcompiler/src/ui_runtime.hpp b/nilcompiler/src/ui_runtime.hpp index e8f15e9d..2e46e158 100644 --- a/nilcompiler/src/ui_runtime.hpp +++ b/nilcompiler/src/ui_runtime.hpp @@ -15,16 +15,14 @@ struct Result { explicit operator bool() const noexcept { return error.empty(); } }; -// Checks the optional GTK 4 provider's symbols, without opening a display. -// macOS requires NIL_UI_GTK_DIRECTORY: an absolute trusted directory containing -// libgtk-4.1.dylib, libgobject-2.0.0.dylib and libglib-2.0.0.dylib. Curated aliases -// may point to canonical external prerequisites; all resolved symbol origins -// and GTK/GObject dependency lookups must match the selected files. +// Checks provider availability without opening a display. Linux loads GTK 4; +// Windows and macOS use their linked Win32 and AppKit providers. Availability +// alone does not establish that an interactive desktop session is usable. bool available() noexcept; // Handles are nonzero, process-unique, and owned by one Context. No toolkit // pointers cross this boundary. UI calls run on the context's creation thread -// and, once GTK initializes, the single process UI thread. +// and obey the selected provider's process-level UI thread requirements. // On macOS that thread must always be the process main thread, including first // initialization. This requirement also applies to embedded/VM/debugger hosts. class Context { @@ -59,9 +57,9 @@ class Context { // kind codes: window=0,column=1,row=2,grid=3,label=4,button=5,entry=6,checkbox=7. Result set_number(std::int64_t handle, const std::string &property, std::int64_t value); Result read_number(std::int64_t handle, const std::string &property); - // Button/checkbox activation uses GTK's actual widget activation path. + // Button/checkbox activation uses the platform's actual control activation path. Result activate(std::int64_t handle); - // Nonblocking: pumps a bounded number of GLib iterations, then snapshots one + // Nonblocking: pumps a bounded number of platform events, then snapshots one // event for this window. number==0 means no event, solely in this private ABI. Result poll(std::int64_t window); // Immutable snapshot fields: kind, target, window, checked. Event kinds: diff --git a/nilcompiler/src/vm.cpp b/nilcompiler/src/vm.cpp index 459dfe28..5231c1f9 100644 --- a/nilcompiler/src/vm.cpp +++ b/nilcompiler/src/vm.cpp @@ -17,6 +17,7 @@ #include "net_support.hpp" #include "secure_file_io.hpp" #include "time_support.hpp" +#include "file_time_support.hpp" #include "../../common/windows_environment.hpp" #include #include @@ -722,6 +723,8 @@ int64_t invoke_native(void *fn_ptr, const std::vector &args) { VM::VM() : pc_(0) {} +VM::~VM() noexcept { closeOpenSockets(); } + void VM::beginMemoryProfile() { if (memory_profile_started_ || !memory_profile_allowed_) return; @@ -734,6 +737,7 @@ bool VM::finishMemoryProfile(const char *status, bool completed) noexcept { // This hook runs at every invocation terminal, even without profiling. // Paused debugger steps retain their UI until completion/reset. ui_.clear(); + closeOpenSockets(); if (!memory_profile_started_ || memory_profile_finished_) return true; memory_profile_finished_ = true; @@ -757,11 +761,17 @@ std::ostream &VM::outputStream() { * every second start of a server fails for a reason that has nothing to do * with the code just changed. */ -void VM::closeOpenSockets() { +void VM::closeOpenSockets() noexcept { for (auto &entry : socket_handles_) net_support::close_descriptor(entry.second.descriptor); socket_handles_.clear(); - next_socket_handle_ = 1; + // Tokens remain monotonic across resets so an old program value cannot name + // a new socket. Readiness and receive buffers belong to this invocation. + watch_order_.clear(); + watch_wanted_.clear(); + ready_.clear(); + ready_cursor_ = 0; + last_received_.clear(); } void VM::resetRuntimeState(const std::string &root_function) { @@ -776,11 +786,6 @@ void VM::resetRuntimeState(const std::string &root_function) { file_handles_.clear(); next_file_handle_ = 1; closeOpenSockets(); - watch_order_.clear(); - watch_wanted_.clear(); - ready_.clear(); - ready_cursor_ = 0; - last_received_.clear(); stack_.clear(); globals_.clear(); gc_marked_structs_.clear(); @@ -3832,7 +3837,24 @@ void VM::executeInstruction(const Instruction &instr) { return true; }; - if (fn_name == "net.listen") { + auto retain_socket = [&](net_support::socket_descriptor descriptor, bool listening) { + if (next_socket_handle_ > (std::numeric_limits::max)() / 8) { + net_support::close_descriptor(descriptor); + push_error("net: socket handle space is exhausted"); + return; + } + const auto token = next_socket_handle_; + try { + socket_handles_.emplace(token, VM::SocketHandle{descriptor, listening}); + } catch (...) { + net_support::close_descriptor(descriptor); + throw; + } + ++next_socket_handle_; + push_ok_number(token); + }; + + if (fn_name == "net.listen" || fn_name == "net.listen_reuse_port") { std::string host; std::int64_t port = 0; std::int64_t backlog = 0; @@ -3840,14 +3862,12 @@ void VM::executeInstruction(const Instruction &instr) { push_error("net.listen: expected a host, a port and a backlog"); return; } - const auto opened = net_support::listen_on(host, port, backlog); + const auto opened = net_support::listen_on(host, port, backlog, fn_name == "net.listen_reuse_port"); if (!opened.ok()) { push_error(opened.error); return; } - const std::int64_t token = next_socket_handle_++; - socket_handles_.emplace(token, VM::SocketHandle{opened.descriptor, true}); - push_ok_number(token); + retain_socket(opened.descriptor, true); return; } @@ -3873,9 +3893,7 @@ void VM::executeInstruction(const Instruction &instr) { push_error(opened.error); return; } - const std::int64_t token = next_socket_handle_++; - socket_handles_.emplace(token, VM::SocketHandle{opened.descriptor, false}); - push_ok_number(token); + retain_socket(opened.descriptor, false); return; } @@ -3891,9 +3909,7 @@ void VM::executeInstruction(const Instruction &instr) { push_error(opened.error); return; } - const std::int64_t token = next_socket_handle_++; - socket_handles_.emplace(token, VM::SocketHandle{opened.descriptor, false}); - push_ok_number(token); + retain_socket(opened.descriptor, false); return; } @@ -3922,6 +3938,10 @@ void VM::executeInstruction(const Instruction &instr) { push_error(taken.error); return; } + if (taken.would_block) { + push_error("net.read: no bytes arrived before the socket deadline or the socket would block"); + return; + } push_ok_text(taken.data); return; } @@ -4240,13 +4260,13 @@ void VM::executeInstruction(const Instruction &instr) { // declarations carry ready_handle and ready_flags so no caller has to // write the division. if (fn_name == "net.next_ready") { - if (ready_cursor_ >= ready_.size()) { - push_ok_number(0); + while (ready_cursor_ < ready_.size()) { + const auto &one = ready_[ready_cursor_++]; + if (socket_handles_.count(one.first) == 0) continue; + push_ok_number(one.first * 8 + one.second); return; } - const auto &one = ready_[ready_cursor_]; - ready_cursor_ += 1; - push_ok_number(one.first * 8 + one.second); + push_ok_number(0); return; } @@ -4949,29 +4969,15 @@ void VM::executeInstruction(const Instruction &instr) { return; } const std::string &path = std::get(args[0]); - std::error_code ec; - const auto written = fs::last_write_time(path, ec); - if (ec) { + const auto written = file_time_support::modified_at(path); + if (!written) { // The path and the reason are both left out: a caller that needs to // tell "absent" from "unreadable" asks file.exists, and an error // carrying a path puts it into whatever log the caller keeps. push_error("file.modified_at: the file has no readable timestamp"); return; } - // file_clock's epoch is unspecified and need not match the system - // clock's, so the two are related by taking a reading from each at the - // same moment and carrying the difference across. std::chrono::clock_cast - // would say this in one line and is not available on every toolchain - // this still builds on. - const auto file_now = fs::file_time_type::clock::now(); - const auto system_now = std::chrono::system_clock::now(); - const auto since_now = - std::chrono::duration_cast( - written - file_now); - const auto as_system = std::chrono::time_point_cast( - system_now + since_now); - push_ok_number(static_cast( - as_system.time_since_epoch().count())); + push_ok_number(*written); return; } diff --git a/nilcompiler/src/vm.hpp b/nilcompiler/src/vm.hpp index 46c19cdf..2f7db71c 100644 --- a/nilcompiler/src/vm.hpp +++ b/nilcompiler/src/vm.hpp @@ -312,6 +312,7 @@ class VM { struct CheckpointState; public: VM(); + ~VM() noexcept; void loadProgram(const std::vector &program, const std::unordered_map &constants); void execute(bool reset = true); @@ -618,7 +619,11 @@ class VM { // `listening` is kept so that accepting on a connection, or reading from a // listener, is an error with a sentence in it rather than an EINVAL. struct SocketHandle { +#ifdef _WIN32 + std::uintptr_t descriptor = ~std::uintptr_t{0}; +#else int descriptor = -1; +#endif bool listening = false; }; std::unordered_map socket_handles_; @@ -647,7 +652,7 @@ class VM { // Closes every socket a run still holds. A program that exits without // closing its listener must not leave the port bound against the next one. - void closeOpenSockets(); + void closeOpenSockets() noexcept; // Type size helper (simplified - all types are 8 bytes for now) size_t get_type_size(const std::string &type_name) const { diff --git a/nilstd/src/io/file.ny b/nilstd/src/io/file.ny index 749000c9..c1058272 100644 --- a/nilstd/src/io/file.ny +++ b/nilstd/src/io/file.ny @@ -94,6 +94,9 @@ pub fn move(source: text, dest: text) -> Result # comparing two should not be comparing precision it does not have. The error # carries neither the path nor the reason; ask `exists` to tell an absent file # from an unreadable one. +# Shared interpreter/native implementation: floor the host timestamp to Unix +# milliseconds without sampling the current clock. Unchanged metadata is stable +# across repeated reads. Paths are UTF-8 and cannot contain NUL. @intrinsic("file.modified_at") pub fn modified_at(path: text) -> Result diff --git a/nilstd/src/net/socket.ny b/nilstd/src/net/socket.ny index 7707c5fc..60413998 100644 --- a/nilstd/src/net/socket.ny +++ b/nilstd/src/net/socket.ny @@ -10,7 +10,7 @@ # program could accept a connection, and the socket design under experimental/ # had nothing beneath it to build on. # -# Every call returns a Result. A network call fails for reasons entirely +# Operations that can fail return a Result. A network call fails for reasons entirely # outside the program — the port is taken, the peer went away, the name does # not resolve — and for a server those are ordinary answers on an ordinary # day, not exceptional conditions. None of these raise. @@ -43,6 +43,12 @@ import nil.std.core @intrinsic("net.listen") pub fn listen(host: text, port: i32, backlog: i32) -> Result +# Bind an explicitly shared listener. Every listener sharing the endpoint must +# opt in before binding; setting reuse-port after listen cannot do this. +# Uses SO_REUSEPORT on Linux/macOS; unsupported hosts return Result.Error. +@intrinsic("net.listen_reuse_port") +pub fn listen_reuse_port(host: text, port: i32, backlog: i32) -> Result + # Wait for a connection and return a handle to it. # # Blocks until one arrives. An interrupted call is retried inside the runtime @@ -69,8 +75,9 @@ pub fn connect(host: text, port: i32) -> Result # # A short read is normal and is not an error: TCP is a stream, and a request # may arrive in several pieces. An orderly close by the peer comes back as -# empty text, so a loop reading to the end of a request sees one shape; use -# is_open when the difference matters. +# empty text. A timeout or would-block result is an error for this blocking +# interface. is_open only reports whether this VM still owns the handle; +# receive distinguishes bytes, EOF and would-block for nonblocking callers. # # The bytes are returned as text and are not necessarily valid UTF-8 — they # are whatever the peer sent. Read them with nil.std.text's byte_at rather @@ -84,8 +91,9 @@ pub fn read(handle: i64, maximum_bytes: i64) -> Result # # A single send is permitted to take less than it was given; this loops until # everything is gone or the attempt fails, because the number of servers that -# have shipped a truncated response from the one-call version is large. On -# failure the count says how far it got. +# have shipped a truncated response from the one-call version is large. +# Failure returns error text without a partial byte count. An event loop +# should use send_some and retain its own progress offset. # # A peer that closes mid-write is reported, not fatal: the runtime suppresses # the signal that would otherwise end the process. @@ -110,7 +118,8 @@ pub fn close(handle: i64) -> Result @intrinsic("net.is_open") pub fn is_open(handle: i64) -> bool -# Put a deadline on reads and writes, in milliseconds. Zero removes it. +# Set each blocking receive/send timeout, in milliseconds. Zero removes it. +# Values must be between 0 and 2147483647 on every supported host. # # A single-threaded server without this is one slow client away from serving # nobody: a connection that opens and then says nothing holds the accept loop @@ -166,19 +175,18 @@ pub const ready_finished: i64 = 4 # because the peer withdrew in between — would stop the whole loop on behalf # of one socket. # -# Leave accepted connections blocking and give them a timeout instead. Poll -# has said they are readable, so a read returns at once, and an empty read -# then means the peer closed rather than "nothing yet", which is a -# distinction an event loop has to be able to make. +# A fully nonblocking loop also sets this on accepted connections, then uses +# receive and send_some to distinguish no progress from an orderly close. +# Readiness can change before the transfer; it is not a promise that a +# blocking read or write will return immediately. @intrinsic("net.set_blocking") pub fn set_blocking(handle: i64, blocking: bool) -> Result -# Let other processes bind this same port. +# Change SO_REUSEPORT on an existing socket where supported. # -# Opt-in, and distinct from the SO_REUSEADDR that listen always sets. That one -# only allows rebinding a port left in TIME_WAIT; this one deliberately allows -# two servers to share a port, which switched on by accident would silently -# split traffic between a running server and a stray one. +# This runs after binding and cannot make a second ordinary listener opt in +# before its bind. Use listen_reuse_port for every shared listener instead. +# Windows returns an error when enabling this option. @intrinsic("net.set_reuse_port") pub fn set_reuse_port(handle: i64, enabled: bool) -> Result diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md new file mode 100644 index 00000000..6c41f7a6 --- /dev/null +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -0,0 +1,105 @@ +# Web runtime and platform parity — 2026-11-11 + +The web handoff resumes at framework `dfb7ad7` on `nil-sockets-entropy-bytes` +([PR 38](https://github.com/exec-software/exec-nil-framework/pull/38)) and web +`594579a` plus preserved local changes. The archived originals are under +`atoma/archived-exec-ny-web` and `atoma/archived-exec-ny-webview`. Platform work +comes first; the web changes are not discarded, reset or overwritten. + +## Current baseline and remaining boundaries + +Windows now uses Win32 and macOS uses AppKit; Linux uses GTK4. November 9 native +AppKit provider and interpreter display results supersede the earlier private +GTK experiments. The AppKit display workflow accidentally selected `nilbuild.*` +with the regex `nil.ui`; its selector is now anchored to the `nil.ui` namespace. +The original failing cache contract remains registered in the full suite. +The full Mac source workflow receives the measured 35-minute aggregate CTest +allowance already used by the successful Intel SDK retry. Individual test +deadlines are unchanged. + +The new socket, entropy, byte-text and timestamp APIs were initially +interpreter-only. Earlier passing installed SDK campaigns do not validate these +new APIs. Darwin incremental cache identity, its native memory-bound tests and +current native application/editor acceptance remain open. The platform/readme +documentation now separates current source implementation from the recorded SDK +and route baselines. + +The latest source validation push run +[43095105713](https://github.com/exec-software/exec-nil-framework/actions/runs/43095105713) +failed before any runner executed: GitHub reports an internal server error, +correlation ID `0fbc4710-ecca-4198-a05c-f35c1d8d4bef`. It supplies no product +test result. The corresponding PR run was still queued during recovery. + +## Portability findings being corrected + +- Windows `SOCKET` values were narrowed into `int`, and its `SO_REUSEADDR` + semantics did not enforce the intended exclusive listener contract. +- Darwin writes lacked per-socket SIGPIPE suppression. +- Finite readiness waits could restart their complete timeout after interruption. +- Windows `rand_s` declarations depended on defining a macro before earlier + headers had already included the CRT. +- Socket ownership was not released at every VM terminal/destructor path. + A process-exit test only observes operating-system cleanup; a new same-process + lifecycle test must observe the VM releasing the listener itself. +- A post-bind reuse-port setter cannot enable a second listener before that + second bind occurs. Sharing requires an explicit pre-bind constructor and + accurate platform support, not silently weakening ordinary listener isolation. +- File timestamps were converted using two separate clock readings. Repeated + reads of an unchanged file could differ by a millisecond. Native metadata + provides a stable epoch value; the scalar API can share that implementation + between the interpreter and native runtime. + +The focused web-runtime workflow builds actual host code and runs an exact six-test +inventory on Linux x86_64, Windows x86_64, Intel macOS and Apple Silicon macOS. +LLVM is disabled there; the report states that scope. The local Linux build and +private Windows VM build additionally exercise available native timestamp routes. +No installed SDK is replaced by these checks. + +The timestamp change now passes eight exact Linux observations: textual NIL, +binary NIL, LLVM JIT and AOT, each before and after a metadata change. Each +observation repeats the unchanged timestamp 100 times; two additional VM cases +reject NUL paths. Fifteen owned commands and thirty raw streams were independently +replayed. The report SHA is +`135a70a9ed8b8a14cda4122abf0b5b3bdbc08378ca376888907c6e5ba9ceff1f`. +The old SDK fails the pre-epoch fractional timestamp control (`-1` versus the +correct floor of `-2` milliseconds). Native Windows adapter checks also pass +116 assertions under strict MSVC warnings; subsequent NUL-host controls and the +full integrated VM build still require their own verification. + +The corrected Linux integration runs now pass nine selected contracts: adapter, +same-process socket lifecycle, ten socket language cases, timestamp helper and +route checks, 38 Matrix execution fixtures, execution checkpoints, VM faults and +parallel runtime. The initial fixture failures are retained alongside the final +results under `parity-web-20261111/linux-integrated-v1` in the VM artifact tree. +These are results across the corrected runs, not a claim that the first run was +green. The macOS display harness now requires all three known desktop cases on +every requested route and prints the full language report into retained CTest +output. Positive and five malformed-inventory controls pass locally; actual +LLVM-enabled AppKit execution still needs both hosted architectures. + +Native socket/byte/entropy parity needs a binary-safe value boundary. The new +VM APIs deliberately carry embedded NUL and arbitrary bytes in `text`; the +current native text ABI and several LLVM operations use null-terminated strings +and `strlen`. Merely registering these intrinsics with that ABI would silently +truncate legitimate network data and entropy. The scalar timestamp operation can +use the existing ordinary-I/O ABI; the byte APIs require an explicit length-aware +design and cross-route tests before native support is claimed. + +## Preserved web handoff + +The interrupted web tree contains the listener-to-engine replacement, template +cache changes, static-file changes, status/main/test edits and four new template +fixtures. Its existing 330 Matrix checks pass. Private, bounded loopback checks +against copies of that source exposed gaps the current unit suite misses: + +- Invalid Content-Length and unsupported Transfer-Encoding can be accepted. +- Pipelined requests lose the second request when surplus input is discarded. +- Range processing precedes conditional checks, ignores If-Range and applies + to HEAD; the observed responses were incorrect 206 results. +- A symlink within a copied public directory could serve an owned marker outside + that directory. Root-constrained file access needs an actual filesystem boundary. + +All diagnostic servers were owned, stopped and reaped. The web source was not +changed. Its README still describes 272 checks and the deleted serial listener; +CI audits source paths but does not execute its runtime suite. These are the next +web tasks after platform parity work, not evidence of IIS/Apache equivalence. diff --git a/tests/file_time_support_tests.cpp b/tests/file_time_support_tests.cpp new file mode 100644 index 00000000..dff66430 --- /dev/null +++ b/tests/file_time_support_tests.cpp @@ -0,0 +1,104 @@ +#include "file_time_support.hpp" + +#include +#include +#include +#include +#include +#ifndef _WIN32 +#include +#endif + +namespace { +int checks = 0; +void require(bool condition, const char *message) { + ++checks; + if (!condition) throw std::runtime_error(message); +} +struct PrivateDirectory { + std::filesystem::path path; + PrivateDirectory() { + const auto seed = std::chrono::steady_clock::now().time_since_epoch().count(); + for (int attempt = 0; attempt < 128; ++attempt) { + auto candidate = std::filesystem::temp_directory_path() / + ("nil-file-time-" + std::to_string(seed) + "-" + std::to_string(attempt)); + std::error_code error; + if (std::filesystem::create_directory(candidate, error)) { + path = std::move(candidate); + return; + } + if (error && error != std::errc::file_exists) + throw std::runtime_error("cannot create private timestamp fixture"); + } + throw std::runtime_error("cannot allocate private timestamp fixture"); + } + ~PrivateDirectory() { + std::error_code ignored; + std::filesystem::remove_all(path, ignored); + } +}; +} + +int main() { + try { + using ny::file_time_support::from_unix_parts; + using ny::file_time_support::from_windows_ticks; + using ny::file_time_support::modified_at; + constexpr auto maximum = (std::numeric_limits::max)(); + constexpr auto minimum = (std::numeric_limits::min)(); + require(from_unix_parts(0, 0) == 0, "Unix epoch"); + require(from_unix_parts(0, 999999) == 0, "submillisecond floor"); + require(from_unix_parts(0, 1000000) == 1, "positive millisecond"); + require(from_unix_parts(-1, 999999999) == -1, "negative submillisecond floor"); + require(from_unix_parts(-1, 999000000) == -1, "negative exact millisecond"); + require(from_unix_parts(-1, 0) == -1000, "negative second"); + require(from_unix_parts(1600000000, 123456789) == 1600000000123LL, "known Unix instant"); + require(!from_unix_parts(0, -1), "negative nanoseconds rejected"); + require(!from_unix_parts(0, 1000000000), "unnormalized nanoseconds rejected"); + require(from_unix_parts(maximum / 1000, 807999999) == maximum, "maximum exact result"); + require(!from_unix_parts(maximum / 1000, 808000000), "upper overflow rejected"); + require(from_unix_parts(minimum / 1000 - 1, 192000000) == minimum, "minimum exact result"); + require(!from_unix_parts(minimum / 1000 - 1, 191999999), "lower overflow rejected"); + require(!from_unix_parts(maximum, 0), "large positive second rejected"); + require(!from_unix_parts(minimum, 0), "large negative second rejected"); + constexpr std::uint64_t epoch_ticks = 116444736000000000ULL; + require(from_windows_ticks(0) == -11644473600000LL, "Windows epoch"); + require(from_windows_ticks(epoch_ticks) == 0, "Windows Unix epoch"); + require(from_windows_ticks(epoch_ticks - 1) == -1, "Windows negative submillisecond floor"); + require(from_windows_ticks(epoch_ticks + 9999) == 0, "Windows positive submillisecond floor"); + require(from_windows_ticks(epoch_ticks + 10000) == 1, "Windows millisecond"); + require(from_windows_ticks(UINT64_MAX) == 1833029933770955LL, "Windows full unsigned range"); + require(!modified_at(""), "empty path rejected"); + require(!modified_at(std::string("\xff", 1)), "invalid UTF-8 rejected"); + + PrivateDirectory private_directory; + const auto path = private_directory.path / + std::filesystem::u8path(u8"time-café-日本語-🦊.txt"); + { std::ofstream output(path, std::ios::binary); output << "timestamp fixture\n"; + require(static_cast(output), "fixture written"); } +#ifdef _WIN32 + const auto file = CreateFileW(path.c_str(), FILE_WRITE_ATTRIBUTES, + FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, nullptr, + OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); + require(file != INVALID_HANDLE_VALUE, "fixture attribute handle"); + constexpr std::uint64_t ticks = epoch_ticks + 16000000001234567ULL; + const FILETIME stamp{static_cast(ticks), static_cast(ticks >> 32)}; + const bool changed = SetFileTime(file, nullptr, nullptr, &stamp) != FALSE; + const bool closed = CloseHandle(file) != FALSE; + require(changed && closed, "exact Windows timestamp installed"); +#else + const timespec stamps[2]{{1600000000, 123456700}, {1600000000, 123456700}}; + require(utimensat(AT_FDCWD, path.c_str(), stamps, 0) == 0, "exact POSIX timestamp installed"); +#endif + for (int i = 0; i < 100; ++i) + require(modified_at(path.u8string()) == 1600000000123LL, "stable exact Unicode-file timestamp"); + require(!modified_at(path.u8string() + ".absent"), "missing file rejected"); + require(!modified_at(path.u8string() + std::string("\0ignored", 8)), "NUL suffix cannot select prefix file"); + require(modified_at(private_directory.path.u8string()).has_value(), "directory metadata retained"); + std::cout << "file time support: " << checks << " checks passed\n"; + return 0; + } catch (const std::exception &error) { + std::cerr << "file time support: " << error.what() << '\n'; + return 1; + } +} diff --git a/tests/matrix/execution/entropy.matrix b/tests/matrix/execution/entropy.matrix index fe525061..6b81168b 100644 --- a/tests/matrix/execution/entropy.matrix +++ b/tests/matrix/execution/entropy.matrix @@ -8,8 +8,8 @@ !!! something predictable or drop the feature. !!! !!! Nothing here can assert a particular value, which is the point. What it -!!! asserts is the shape, the bounds, and that two draws differ — the last -!!! being the one check that would have caught MINSTD standing in for this. +!!! asserts is the shape, the bounds, and that two draws differ. The last is +!!! only a constant-output sanity check; it does not prove cryptographic quality. book matrix.entropy diff --git a/tests/net_lifecycle_tests.cpp b/tests/net_lifecycle_tests.cpp new file mode 100644 index 00000000..5e792e48 --- /dev/null +++ b/tests/net_lifecycle_tests.cpp @@ -0,0 +1,157 @@ +// The VM must release sockets while its host process is still alive. +#include "net_support.hpp" +#include "codegen.hpp" +#include "nil_loader.hpp" +#include "vm.hpp" +#include +#include +#include +#include + +namespace { +namespace net = ny::net_support; +int checks = 0; +void require(bool value, const std::string &message) { + ++checks; + if (!value) throw std::runtime_error(message); +} +struct Socket { + net::socket_descriptor value; + explicit Socket(net::socket_descriptor descriptor) : value(descriptor) {} + ~Socket() { if (value != net::invalid_socket) net::close_descriptor(value); } + Socket(const Socket &) = delete; + Socket &operator=(const Socket &) = delete; +}; +int unused_port() { + std::string error; + require(net::ensure_started(error), error); + Socket probe(::socket(AF_INET, SOCK_STREAM, 0)); + require(probe.value != net::invalid_socket, "open loopback port probe"); + sockaddr_in address{}; address.sin_family = AF_INET; address.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + require(::bind(probe.value, reinterpret_cast(&address), sizeof(address)) == 0, "bind port probe"); +#ifdef _WIN32 + int size = sizeof(address); +#else + socklen_t size = sizeof(address); +#endif + require(::getsockname(probe.value, reinterpret_cast(&address), &size) == 0, "read port probe"); + return ntohs(address.sin_port); +} +void endpoint(int port, bool held, const std::string &phase) { + const auto probe = net::listen_on("127.0.0.1", port, 8); + Socket owned(probe.descriptor); + require(probe.ok() != held, phase + (held ? " must retain the actual listening socket" : " must release the actual listening socket")); +} +const std::string result_type = "nil::std::core::Result"; +ny::nil_function function(std::string name, std::vector parameters, + std::string result, std::vector body) { + ny::nil_function value; + value.name = std::move(name); value.params = std::move(parameters); + value.return_type = std::move(result); value.body = std::move(body); + return value; +} +struct Program { + std::vector code; + std::unordered_map constants; +}; +Program fixture(int port, bool binary, bool fault) { + ny::nil_module module; + module.module_name = "net_lifecycle"; + module.enums = {{result_type, {{"Ok", "i64"}, {"Error", "text"}}}}; + module.string_constants = {{"host", "127.0.0.1"}}; + std::vector body{ + {"PUSH_CONST_STR", {"host"}}, {"PUSH_CONST", {std::to_string(port)}}, {"PUSH_CONST", {"8"}}, + {"INTEROP_CALL", {"intrinsic", "net.listen", "3", result_type}}, + {"UNWRAP_VARIANT", {"1", result_type, "i64"}}, {"STORE_GLOBAL", {"opened", "i64"}}, + {"LOAD_GLOBAL", {"opened", "i64"}}, {"PUSH_CONST", {"1"}}, + {"INTEROP_CALL", {"intrinsic", "net.watch", "2", result_type}}, + {"UNWRAP_VARIANT", {"1", result_type, "i64"}}, {"STORE_GLOBAL", {"watched", "i64"}}}; + if (fault) { + body.push_back({"PUSH_CONST", {"1"}}); body.push_back({"PUSH_CONST", {"0"}}); + body.push_back({"DIV", {"i64"}}); + } else body.push_back({"PUSH_CONST", {"0"}}); + body.push_back({"RET", {}}); + module.functions.push_back(function("main", {}, "i64", std::move(body))); + module.functions.push_back(function("watch_count", {}, "i64", + {{"INTEROP_CALL", {"intrinsic", "net.watch_count", "0", "i64"}}, {"RET", {}}})); + module.functions.push_back(function("is_open", {"handle:i64"}, "bool", + {{"STORE_LOCAL", {"handle"}}, {"LOAD_LOCAL", {"handle"}}, + {"INTEROP_CALL", {"intrinsic", "net.is_open", "1", "bool"}}, {"RET", {}}})); + std::ostringstream bytes(std::ios::out | std::ios::binary); + if (binary) { + ny::codegen_options options; ny::nil_binary_writer writer(bytes, options); writer.emit_binary_module(module); + } else { ny::nil_writer writer(bytes); writer.emit_text_module(module); } + std::istringstream input(bytes.str(), std::ios::in | std::ios::binary); + Program program; + program.code = binary ? ny::parseBinaryProgram(input, program.constants) : ny::parseProgram(input, program.constants); + return program; +} +std::int64_t pause(ny::VM &vm, const Program &program, int port) { + vm.loadProgram(program.code, program.constants); + vm.resetExecution("main"); + for (int i = 0; i < 50 && !vm.hasGlobal("watched"); ++i) + require(vm.stepInstruction(), "program must reach its watched listener"); + require(vm.hasGlobal("watched") && std::get(vm.getGlobal("watched")) == 1, + "exactly one listener is watched before pause"); + endpoint(port, true, "paused invocation"); + const auto token = std::get(vm.getGlobal("opened")); + require(token > 0, "listener token is positive"); + return token; +} +void cleared(ny::VM &vm, std::int64_t token) { + vm.pushValue(token); vm.callFunction("is_open", 1); + require(std::get(vm.getStack().back()) == 0, "terminal socket token is stale"); + vm.callFunction("watch_count", 0); + require(std::get(vm.getStack().back()) == 0, "terminal clears the watch set"); +} +void lifecycle(bool binary) { + const int port = unused_port(); + const auto normal = fixture(port, binary, false); + const auto faulty = fixture(port, binary, true); + ny::VM vm; + const auto first = pause(vm, normal, port); + vm.execute(false); + endpoint(port, false, "normal completion"); + cleared(vm, first); + + const auto second = pause(vm, normal, port); + require(second > first, "completed invocation tokens are never reissued"); + vm.resetExecution("main"); + endpoint(port, false, "debugger reset"); + // resetExecution leaves a prepared main frame, so invoking a separate host + // function here would escape that frame without RET. The bind probe above + // observes actual release without changing or executing the prepared frame. + + const auto third = pause(vm, normal, port); + require(third > second, "reset invocation tokens are never reissued"); + vm.loadProgram(normal.code, normal.constants); + endpoint(port, false, "program replacement"); + cleared(vm, third); + + const auto fourth = pause(vm, faulty, port); + bool failed = false; + try { vm.execute(false); } + catch (const std::runtime_error &error) { failed = std::string(error.what()).find("Division by zero") != std::string::npos; } + require(failed, "fault fixture must actually execute its division error"); + endpoint(port, false, "uncaught runtime error"); + require(fourth > third, "faulting invocation also receives a fresh token"); + + { + auto paused = std::make_unique(); + pause(*paused, normal, port); + paused.reset(); + } + endpoint(port, false, "paused VM destruction"); +} +} + +int main() { + try { + lifecycle(false); lifecycle(true); + std::cout << "Socket VM lifecycle: " << checks << " checks passed\n"; + return 0; + } catch (const std::exception &error) { + std::cerr << "Socket lifecycle failure: " << error.what() << '\n'; + return 1; + } +} diff --git a/tests/net_support_tests.cpp b/tests/net_support_tests.cpp new file mode 100644 index 00000000..954870be --- /dev/null +++ b/tests/net_support_tests.cpp @@ -0,0 +1,230 @@ +// Exercise the OS socket/entropy adapters without an interpreter or public port. +// Include the CRT first: entropy must not depend on an earlier feature macro. +#include +#include "net_support.hpp" +#include "entropy_support.hpp" +#include +#include +#include +#include +#include +#if !defined(_WIN32) +#include +#endif + +namespace { +namespace net = ny::net_support; +int checks = 0; +void require(bool value, const std::string &message) { + ++checks; + if (!value) throw std::runtime_error(message); +} +struct Socket { + net::socket_descriptor value = net::invalid_socket; + explicit Socket(net::socket_descriptor opened = net::invalid_socket) : value(opened) {} + ~Socket() { if (value != net::invalid_socket) net::close_descriptor(value); } + Socket(const Socket &) = delete; + Socket &operator=(const Socket &) = delete; +}; +int unused_port() { + std::string error; + require(net::ensure_started(error), error); + Socket probe(::socket(AF_INET, SOCK_STREAM, 0)); + require(probe.value != net::invalid_socket, "open port probe"); + sockaddr_in address{}; + address.sin_family = AF_INET; + address.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + require(::bind(probe.value, reinterpret_cast(&address), sizeof(address)) == 0, "bind port probe"); +#if defined(_WIN32) + int size = sizeof(address); +#else + socklen_t size = sizeof(address); +#endif + require(::getsockname(probe.value, reinterpret_cast(&address), &size) == 0, "read port probe"); + return ntohs(address.sin_port); +} +void binding() { + const std::string truncated("127.0.0.1\0suffix", 16); + const auto invalid_bind = net::listen_on(truncated, 31337, 8); + require(!invalid_bind.ok() && invalid_bind.error.find("NUL") != std::string::npos, "bind rejects embedded NUL"); + const auto invalid_connect = net::connect_to(truncated, 31337); + require(!invalid_connect.ok() && invalid_connect.error.find("NUL") != std::string::npos, "connect rejects NUL before resolution"); + const int port = unused_port(); + const auto first = net::listen_on("127.0.0.1", port, 8); + require(first.ok(), first.error); + Socket one(first.descriptor); + const auto second = net::listen_on("127.0.0.1", port, 8); + Socket two(second.descriptor); + require(!second.ok(), "ordinary listeners must not share an endpoint"); + require(!net::set_timeout(one.value, -1).empty(), "negative timeout rejected"); + require(!net::set_timeout(one.value, net::max_timeout_ms + 1).empty(), "overflowing timeout rejected"); + std::vector ready; + require(!net::poll_wait({{1, one.value, net::ready_readable}}, net::max_timeout_ms + 1, ready).empty(), + "overflowing poll timeout rejected"); + require(!net::poll_wait({{1, one.value, 8}}, 0, ready).empty(), "unknown watch flags rejected"); + + const int shared_port = unused_port(); + const auto shared_first = net::listen_on("127.0.0.1", shared_port, 8, true); + Socket shared_one(shared_first.descriptor); +#if defined(SO_REUSEPORT) && !defined(_WIN32) + require(shared_first.ok(), shared_first.error); + const auto shared_second = net::listen_on("127.0.0.1", shared_port, 8, true); + Socket shared_two(shared_second.descriptor); + require(shared_second.ok(), shared_second.error); + const auto ordinary = net::listen_on("127.0.0.1", shared_port, 8); + Socket accidental(ordinary.descriptor); + require(!ordinary.ok(), "ordinary listener cannot join an explicitly shared endpoint"); +#else + require(!shared_first.ok() && shared_first.error.find("no shared-listener") != std::string::npos, + "unsupported shared listening is an explicit error"); +#endif +} + +void stream() { + const int port = unused_port(); + const auto opened = net::listen_on("127.0.0.1", port, 8); + require(opened.ok(), opened.error); + Socket listener(opened.descriptor); + require(net::set_blocking(listener.value, false).empty(), "nonblocking listener"); + require(net::accept_on(listener.value).would_block, "idle accept is would-block"); + const auto connected = net::connect_to("127.0.0.1", port); + require(connected.ok(), connected.error); + Socket client(connected.descriptor); + std::vector ready; + require(net::poll_wait({{37, listener.value, net::ready_readable}}, 1000, ready).empty() && + ready.size() == 1 && ready[0].token == 37 && (ready[0].flags & net::ready_readable), "listener readiness"); + const auto accepted = net::accept_on(listener.value); + require(accepted.ok(), accepted.error); + Socket server(accepted.descriptor); + require(net::set_timeout(server.value, 1000).empty(), "server timeout"); + require(net::set_timeout(client.value, 1000).empty(), "client timeout"); +#if defined(SO_NOSIGPIPE) + for (const auto descriptor : {client.value, server.value}) { + int enabled = 0; socklen_t size = sizeof(enabled); + require(::getsockopt(descriptor, SOL_SOCKET, SO_NOSIGPIPE, &enabled, &size) == 0 && enabled == 1, + "both connecting and accepted sockets suppress SIGPIPE"); + } +#endif + require(net::set_blocking(server.value, false).empty(), "nonblocking receive"); + const auto idle = net::read_some(server.value, 32); + require(idle.ok() && idle.would_block && !idle.closed && idle.data.empty(), "would-block differs from EOF"); + const std::string bytes("prefix\0\xfftail", 12); + const auto sent = net::write_some(client.value, bytes, 6); + require(sent.ok() && sent.count == 6 && !sent.would_block, "send from exact byte offset"); + require(net::poll_wait({{9, server.value, net::ready_readable}}, 1000, ready).empty() && !ready.empty(), "data readiness"); + const auto data = net::read_some(server.value, 32); + require(data.ok() && data.data == bytes.substr(6) && data.count == 6 && !data.closed && !data.would_block, + "nonblocking bytes survive zero and invalid UTF-8"); + require(net::write_some(client.value, bytes, bytes.size()).count == 0, "empty remainder succeeds"); + + // A peer that does not read eventually applies backpressure. Bound traffic + // and memory instead of depending on a host's default socket-buffer size. + const int buffer = 4096; + require(::setsockopt(client.value, SOL_SOCKET, SO_SNDBUF, + reinterpret_cast(&buffer), sizeof(buffer)) == 0, "small send buffer"); + require(net::set_blocking(client.value, false).empty(), "nonblocking send"); + const std::string payload(256 * 1024, 'x'); + bool blocked = false; + std::int64_t total = 0; + for (int attempt = 0; attempt < 1024 && total < 32 * 1024 * 1024; ++attempt) { + const auto part = net::write_some(client.value, payload, 0); + require(part.ok(), part.error); + if (part.would_block) { blocked = true; break; } + require(part.count > 0 && part.count <= static_cast(payload.size()), "bounded positive partial send"); + total += part.count; + } + require(blocked && total > 0, "slow peer produces would-block without blocking the process"); +} + +void peer_close() { + const int port = unused_port(); + const auto opened = net::listen_on("127.0.0.1", port, 8); + require(opened.ok(), opened.error); + Socket listener(opened.descriptor); + const auto connected = net::connect_to("127.0.0.1", port); + require(connected.ok(), connected.error); + Socket client(connected.descriptor); + const auto accepted = net::accept_on(listener.value); + require(accepted.ok(), accepted.error); + Socket server(accepted.descriptor); + require(net::set_timeout(client.value, 1000).empty(), "closed-peer client timeout"); + linger reset{}; reset.l_onoff = 1; reset.l_linger = 0; + require(::setsockopt(server.value, SOL_SOCKET, SO_LINGER, + reinterpret_cast(&reset), sizeof(reset)) == 0, "peer reset configuration"); + net::close_descriptor(server.value); server.value = net::invalid_socket; + std::vector ready; + require(net::poll_wait({{1, client.value, net::ready_readable}}, 1000, ready).empty() && !ready.empty(), "peer-close observed"); + // The first failure can be ECONNRESET; a subsequent write is what exposes + // an unsuppressed EPIPE. The process uses SIGPIPE's default disposition. + int failed = 0; + for (int attempt = 0; attempt < 4; ++attempt) + if (!net::write_all(client.value, "closed peer").ok()) ++failed; + require(failed > 0, "closed-peer writes report failure without terminating"); +} + +#if !defined(_WIN32) +volatile std::sig_atomic_t interruptions = 0; +void interrupted(int) { ++interruptions; } +void interrupted_wait() { + const auto opened = net::listen_on("127.0.0.1", unused_port(), 8); + require(opened.ok(), opened.error); + Socket listener(opened.descriptor); + struct sigaction action{}, before{}; + action.sa_handler = interrupted; + sigemptyset(&action.sa_mask); + require(sigaction(SIGUSR1, &action, &before) == 0, "install owned-thread signal handler"); + const pthread_t target = pthread_self(); + std::thread signaler([target] { + for (int i = 0; i < 20; ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + pthread_kill(target, SIGUSR1); + } + }); + const auto start = std::chrono::steady_clock::now(); + std::vector ready; + const auto error = net::poll_wait({{1, listener.value, net::ready_readable}}, 100, ready); + const auto elapsed = std::chrono::steady_clock::now() - start; + signaler.join(); + require(sigaction(SIGUSR1, &before, nullptr) == 0, "restore signal disposition"); + require(error.empty() && ready.empty() && interruptions > 0, "signals do not turn a wait into an error"); + require(elapsed >= std::chrono::milliseconds(70) && elapsed < std::chrono::milliseconds(300), + "finite deadline is not restarted by repeated interruptions"); +} +#endif + +void entropy() { + namespace random = ny::entropy_support; + require(!random::random_bytes(0).ok() && !random::random_bytes(1025).ok(), "entropy byte bounds"); + const auto bytes = random::random_bytes(257); + require(bytes.ok() && bytes.bytes.size() == 257, "OS entropy fills the complete buffer"); + const auto text = random::random_hex(32); + require(text.ok() && text.bytes.size() == 64 && text.bytes.find_first_not_of("0123456789abcdef") == std::string::npos, + "OS entropy hex encoding"); + for (const auto bound : {std::int64_t{1}, std::int64_t{3}, (std::numeric_limits::max)()}) { + for (int i = 0; i < 20; ++i) { + const auto value = random::random_below(bound); + require(value.ok() && value.value >= 0 && value.value < bound, "OS entropy bounded value"); + } + } + require(!random::random_below(0).ok(), "entropy invalid bound"); +} +} + +int main() { + try { +#if defined(_WIN32) + static_assert(std::is_same::value, "socket storage must not narrow a Winsock handle"); +#else + std::signal(SIGPIPE, SIG_DFL); +#endif + binding(); stream(); peer_close(); entropy(); +#if !defined(_WIN32) + interrupted_wait(); +#endif + std::cout << "Socket/entropy support: " << checks << " checks passed\n"; + return 0; + } catch (const std::exception &error) { + std::cerr << error.what() << '\n'; + return 1; + } +} diff --git a/tests/tools/file_modified_at_contract.py b/tests/tools/file_modified_at_contract.py new file mode 100644 index 00000000..f2a9145f --- /dev/null +++ b/tests/tools/file_modified_at_contract.py @@ -0,0 +1,180 @@ +#!/usr/bin/env python3 +"""Compare real file metadata with the public VM/JIT/AOT timestamp operation.""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +from pathlib import Path +import sys +import tempfile +import time + +import ordinary_io_parity_contract as support + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / 'tests')) +from native_output_staging import native_output_path + +ERROR = 'file.modified_at: the file has no readable timestamp' + + +def literal(value: str) -> str: + return '"' + value.replace('\\', '\\\\').replace('"', '\\"') + '"' + + +def source_for(paths: list[Path], nul_case: bool = False) -> str: + calls = '\n'.join(' emit(' + literal(path.name) + ')' for path in paths) + if nul_case: + calls = ' emit(' + literal(paths[0].name)[:-1] + '\\0ignored")' + return '''import nil.std.core +import nil.std.io.file +fn emit(path: text) + match nil::std::io::modified_at(path) + Result.Ok(value) -> println(value) + Result.Error(message) -> println(message) + end match +end fn +fn main() -> i32 +''' + ('' if nul_case else ' for index in range(100)\n emit(' + literal(paths[0].name) + ')\n end for\n') + calls + ''' + return 0 +end fn +''' + + +def signature(paths: list[Path]) -> dict: + result = {} + for path in paths: + if not path.exists(): + result[path.name] = {'missing': True} + continue + info = path.stat() + result[path.name] = {'modified_ns': info.st_mtime_ns, + 'milliseconds': info.st_mtime_ns // 1_000_000, + 'size': info.st_size, + 'directory': path.is_dir()} + if path.is_file(): + result[path.name]['sha256'] = hashlib.sha256(path.read_bytes()).hexdigest() + return result + + +def expected(paths: list[Path]) -> bytes: + values = [str(path.stat().st_mtime_ns // 1_000_000) if path.exists() else ERROR + for path in paths] + return ('\n'.join([values[0]] * 100 + values) + '\n').encode('utf-8') + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--compiler', type=Path, required=True) + parser.add_argument('--interpreter', type=Path, required=True) + parser.add_argument('--nilstd', type=Path, required=True) + parser.add_argument('--llvm', choices=('ON', 'OFF'), required=True) + destination = parser.add_mutually_exclusive_group() + destination.add_argument('--output', type=Path, + help='Fresh exact evidence directory') + destination.add_argument('--output-parent', type=Path, + help='Keep each run in a fresh child of this directory') + args = parser.parse_args() + compiler = args.compiler.resolve(strict=True) + interpreter = args.interpreter.resolve(strict=True) + nilstd = args.nilstd.resolve(strict=True) + if args.output_parent: + parent = args.output_parent.resolve() + parent.mkdir(parents=True, exist_ok=True) + output = Path(tempfile.mkdtemp(prefix='nil-file-time-contract-', dir=parent)) + elif args.output: + output = args.output.resolve() + output.mkdir(parents=True, exist_ok=False) + else: + output = Path(tempfile.mkdtemp(prefix='nil-file-time-contract-')) + work = output / 'work' + work.mkdir() + logs = output / 'logs' + logs.mkdir() + report = {'schema': 'nil.file-modified-at.v1', 'status': 'FAIL', + 'llvm_enabled': args.llvm == 'ON', 'commands': [], 'routes': [], + 'native_routes': 'enabled' if args.llvm == 'ON' else 'disabled by build configuration', + 'native_text_scope': 'NUL-free; embedded NUL checked on interpreter routes', + 'epoch_rule': 'floor native last-write timestamp to Unix milliseconds', + 'whole_seconds': 160} + deadline = time.monotonic() + 160 + ownership = support.load_ownership(ROOT) + environment = {key: value for key, value in os.environ.items() + if not key.startswith(('NIL_', 'NY_'))} + + def run(name, command, wanted=b'', limit=30): + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AssertionError('whole timestamp contract deadline exhausted') + row = support.run_process(ownership, [str(value) for value in command], work, + environment, logs, name, min(limit, remaining)) + report['commands'].append(row) + actual = (logs / row['stdout']).read_bytes() + error = (logs / row['stderr']).read_bytes() + if row['status'] != 'PASS' or not row['cleanup_verified'] or actual != wanted or error: + raise AssertionError(name + ': exact timestamp/exit/stderr/cleanup contract failed') + + try: + paths = [work / 'fraction-café-日本語-🦊.txt', work / 'epoch.txt', + work / 'boundary.txt', work / 'before-epoch.txt', + work / 'directory-café', work / 'absent.txt'] + requested = [1600000000123456700, 0, 1600000000999999900, -1000100] + for path, nanoseconds in zip(paths[:4], requested): + path.write_bytes(b'unchanged metadata fixture\n') + os.utime(path, ns=(nanoseconds, nanoseconds)) + paths[4].mkdir() + os.utime(paths[4], ns=(requested[0], requested[0])) + report['requested_file_times_ns'] = dict(zip((p.name for p in paths[:4]), requested)) + report['before'] = signature(paths) + source = work / 'metadata.ny' + source.write_text(source_for(paths), encoding='utf-8', newline='\n') + base = [compiler, '--quiet', '--no-stdlib', '--sdk-path', nilstd] + commands = [] + for encoding in ('text', 'binary'): + nil = work / (encoding + '.nil') + flags = [] if encoding == 'text' else ['--emit-binary-nil'] + run('compile-' + encoding, [*base, '--no-bundle', *flags, '-o', nil, source]) + if nil.read_bytes().startswith(b'NIL\0') != (encoding == 'binary'): + raise AssertionError('compiled NIL encoding differs') + commands.append((encoding + '-interpreter', [interpreter, '--no-ai', nil])) + if args.llvm == 'ON': + native = work / 'native' + run('compile-native', [*base, '--native', '-o', native, source], limit=60) + commands.append(('llvm-aot', [native_output_path(native)])) + commands.append(('llvm-jit', [*base, '--native', '--jit', '-o', work / 'jit', source])) + for stage in ('initial', 'updated'): + if stage == 'updated': + old = paths[0].stat().st_mtime_ns + os.utime(paths[0], ns=(old + 2345678900, old + 2345678900)) + if paths[0].stat().st_mtime_ns // 1_000_000 == old // 1_000_000: + raise AssertionError('host did not change the fixture timestamp') + before = signature(paths) + for name, command in commands: + run(stage + '-' + name, command, expected(paths), limit=60) + report['routes'].append({'stage': stage, 'route': name, 'status': 'PASS', + 'stable_repetitions': 100}) + if signature(paths) != before: + raise AssertionError('timestamp reads changed the fixtures') + nul = work / 'nul.ny' + nul.write_text(source_for(paths, nul_case=True), encoding='utf-8', newline='\n') + for encoding in ('text', 'binary'): + output_nil = work / ('nul-' + encoding + '.nil') + flags = [] if encoding == 'text' else ['--emit-binary-nil'] + run('compile-nul-' + encoding, [*base, '--no-bundle', *flags, '-o', output_nil, nul]) + run('nul-' + encoding, [interpreter, '--no-ai', output_nil], (ERROR + '\n').encode()) + report['after'] = signature(paths) + report['status'] = 'PASS' + except Exception as error: + report['error'] = str(error) + report['cleanup_verified'] = all(row.get('cleanup_verified') is True for row in report['commands']) + (output / 'report.json').write_text(json.dumps(report, indent=2) + '\n', encoding='utf-8') + print(report['status'] + ': ' + str(output / 'report.json')) + if report['status'] != 'PASS': + print(report.get('error', 'timestamp contract failed'), file=sys.stderr) + return 0 if report['status'] == 'PASS' else 1 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/tests/tools/net_socket_contract.py b/tests/tools/net_socket_contract.py index 3ba1e108..3c03d985 100644 --- a/tests/tools/net_socket_contract.py +++ b/tests/tools/net_socket_contract.py @@ -28,6 +28,9 @@ import argparse import os +import queue +import struct +import threading import socket import subprocess import sys @@ -57,22 +60,94 @@ def build(compiler: Path, stdlib: Path, source: str, workdir: Path, name: str) - built = subprocess.run( [str(compiler), "--no-bundle", "-I", str(stdlib), "--output", str(artifact), str(page)], - capture_output=True, text=True) + capture_output=True, text=True, timeout=60) if built.returncode != 0: raise AssertionError( f"{name} did not compile:\n{built.stdout}{built.stderr}") return artifact +class PageOutput: + """Bounded background drain: readiness deadlines cannot block in readline.""" + def __init__(self, process, stream): + self.process, self.stream = process, stream + self.events = queue.Queue() + self.buffer = b"" + self.ended = False + self.reader = threading.Thread(target=self._drain, name="socket-test-output") + self.reader.start() + + def _drain(self): + total = 0 + try: + while True: + chunk = os.read(self.stream.fileno(), 4096) + if not chunk: + break + total += len(chunk) + if total > 1024 * 1024: + self.events.put(AssertionError("page output exceeded 1 MiB")) + self.process.kill() + break + self.events.put(chunk) + except Exception as error: + self.events.put(error) + finally: + self.events.put(None) + + def _take(self, deadline): + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AssertionError("page output deadline expired") + try: + value = self.events.get(timeout=remaining) + except queue.Empty as error: + raise AssertionError("page output deadline expired") from error + if isinstance(value, Exception): + raise value + if value is None: + self.ended = True + else: + self.buffer += value + + def readline(self, timeout=20.0): + deadline = time.monotonic() + timeout + while b"\n" not in self.buffer and not self.ended: + self._take(deadline) + if b"\n" in self.buffer: + line, self.buffer = self.buffer.split(b"\n", 1) + return (line + b"\n").decode("utf-8", errors="replace") + line, self.buffer = self.buffer, b"" + return line.decode("utf-8", errors="replace") + + def read(self, timeout=20.0): + deadline = time.monotonic() + timeout + while not self.ended: + self._take(deadline) + value, self.buffer = self.buffer, b"" + return value.decode("utf-8", errors="replace") + + def close(self): + self.reader.join(timeout=5) + if self.reader.is_alive(): + raise AssertionError("page output reader did not finish after owned process cleanup") + self.stream.close() + + +def cleanup_page(page): + if page.poll() is None: + page.kill() + page.wait(timeout=10) + page.stdout.close() + + def wait_for_line(process: subprocess.Popen, marker: str, timeout: float = 20.0) -> None: - """Read the page's stdout until it says it is ready.""" deadline = time.monotonic() + timeout while time.monotonic() < deadline: - line = process.stdout.readline() + line = process.stdout.readline(timeout=max(0.001, deadline - time.monotonic())) if line == "": - raise AssertionError( - f"the page exited before printing {marker!r}") - if marker in line: + raise AssertionError(f"the page exited before printing {marker!r}") + if line.strip() == marker: return raise AssertionError(f"the page never printed {marker!r}") @@ -458,6 +533,7 @@ def wait_for_line(process: subprocess.Popen, marker: str, timeout: float = 20.0) variable answered as number is 0 variable live-now as number is 0 variable most-at-once as number is 0 + variable partial-count as number is 0 repeat while answered is less than EXPECTED evaluate nil.std.net::wait with timeout_milliseconds being 10000 @@ -504,6 +580,12 @@ def wait_for_line(process: subprocess.Popen, marker: str, timeout: float = 20.0) value done is nil.std.text::contains with s being whole, needle being "END" value gone is piece equals "" when not done and not gone then + when held.gathered equals "" then + set partial-count to partial-count plus 1 + when partial-count equals EXPECTED then + say "all-partial" + end when + end when set slots at slot to build pending with handle being handle, gathered being whole, live being true end when @@ -545,11 +627,99 @@ def wait_for_line(process: subprocess.Popen, marker: str, timeout: float = 20.0) end paragraph """ +NONBLOCK_PAGE = """ +book net.contract.nonblocking +read nil.std.core +read nil.std.text.string_utils +read nil.std.net.socket + +amount receives result as Result of number, text returns number begin paragraph + consider result + when Result.Ok holding amount-held then give amount-held + when Result.Error holding why then begin block + say "error " plus why + give 0 minus 2 + end block + otherwise give 0 minus 3 + end consider +end paragraph + +main returns number begin paragraph + value listener is amount with result being (nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 8) + say "ready" + value client is amount with result being (nil.std.net::accept with handle being listener) + evaluate nil.std.net::set_blocking with handle being client, blocking being false + evaluate nil.std.net::watch with handle being client, flags being nil.std.net::ready_readable + value idle is amount with result being (nil.std.net::receive with handle being client, maximum_bytes being 64) + say "idle " plus (idle becomes text) + evaluate nil.std.net::wait with timeout_milliseconds being 10000 + value taken is amount with result being (nil.std.net::receive with handle being client, maximum_bytes being 64) + say "received " plus (taken becomes text) + consider nil.std.net::received with nothing + when Result.Ok holding incoming then begin block + value sent is amount with result being (nil.std.net::send_some with handle being client, + data being "xx" plus incoming, offset being 2) + say "sent " plus (sent becomes text) + end block + otherwise give 1 + end consider + evaluate nil.std.net::wait with timeout_milliseconds being 10000 + value closed is amount with result being (nil.std.net::receive with handle being client, maximum_bytes being 64) + say "closed " plus (closed becomes text) + consider nil.std.net::received with nothing + when Result.Ok holding incoming then say "cleared " plus ((nil.std.text::length with s being incoming) becomes text) + otherwise give 1 + end consider + evaluate nil.std.net::close with handle being client + value stale is amount with result being (nil.std.net::next_ready with nothing) + say "stale-ready " plus (stale becomes text) + evaluate nil.std.net::close with handle being listener + say "done" + give 0 +end paragraph +""" + +SHARED_PAGE = """ +book net.contract.shared +read nil.std.core +read nil.std.net.socket + +main returns number begin paragraph + consider nil.std.net::listen_reuse_port with host being "127.0.0.1", port being PORT, backlog being 8 + when Result.Error holding why then say "unsupported " plus why + when Result.Ok holding first then begin block + consider nil.std.net::listen_reuse_port with host being "127.0.0.1", port being PORT, backlog being 8 + when Result.Ok holding second then begin block + say "both shared" + evaluate nil.std.net::close with handle being second + end block + when Result.Error holding why then say "second failed " plus why + otherwise give 1 + end consider + consider nil.std.net::listen with host being "127.0.0.1", port being PORT, backlog being 8 + when Result.Error holding why then say "ordinary refused" + when Result.Ok holding accidental then begin block + say "ordinary admitted" + evaluate nil.std.net::close with handle being accidental + end block + otherwise give 1 + end consider + evaluate nil.std.net::close with handle being first + end block + otherwise give 1 + end consider + give 0 +end paragraph +""" + + def run_page(interpreter: Path, artifact: Path) -> subprocess.Popen: - return subprocess.Popen( + process = subprocess.Popen( [str(interpreter), str(artifact)], - stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, - bufsize=1) + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + bufsize=0) + process.stdout = PageOutput(process, process.stdout) + return process def case_echo(compiler, interpreter, stdlib, workdir) -> list[str]: @@ -584,8 +754,7 @@ def case_echo(compiler, interpreter, stdlib, workdir) -> list[str]: if "done" not in output: problems.append(f"echo: the page did not finish; it said {output!r}") finally: - page.kill() - page.wait(timeout=10) + cleanup_page(page) return problems @@ -659,8 +828,7 @@ def case_timeout(compiler, interpreter, stdlib, workdir) -> list[str]: except subprocess.TimeoutExpired: problems.append("timeout: the page never finished, so the deadline did nothing") finally: - page.kill() - page.wait(timeout=10) + cleanup_page(page) return problems @@ -677,7 +845,7 @@ def case_peer_hangup(compiler, interpreter, stdlib, workdir) -> list[str]: # Reset rather than a graceful close, so the next write fails outright # instead of being accepted into a buffer nobody will drain. client.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, - b"\x01\x00\x00\x00\x00\x00\x00\x00") + struct.pack("HH" if os.name == "nt" else "ii", 1, 0)) client.close() page.wait(timeout=30) output = page.stdout.read() @@ -692,8 +860,7 @@ def case_peer_hangup(compiler, interpreter, stdlib, workdir) -> list[str]: except subprocess.TimeoutExpired: problems.append("peer-hangup: the page never finished") finally: - page.kill() - page.wait(timeout=10) + cleanup_page(page) return problems @@ -725,8 +892,7 @@ def case_client(compiler, interpreter, stdlib, workdir) -> list[str]: problems.append("client: the page did not complete its exchange") finally: listener.close() - page.kill() - page.wait(timeout=10) + cleanup_page(page) return problems @@ -770,18 +936,13 @@ def case_poll(compiler, interpreter, stdlib, workdir) -> list[str]: def case_event_loop(compiler, interpreter, stdlib, workdir) -> list[str]: - """Many connections at once, and the timing that proves it. - - Each client sends its request in two parts with a gap between. A server - that serves one connection at a time must wait out the first client's gap - before it touches the second, so the run costs clients x gap. An event - loop overlaps them and costs about one gap. The assertion is on that - difference rather than on a throughput number, because a number would - measure the machine and this measures the shape. + """Every first fragment must be processed before any client sends its tail. + + A serial read-until-complete server cannot reach this barrier. This proves + overlapping request processing without deriving it from a throughput number. """ clients = 64 - gap = 0.25 - serial_cost = clients * gap + release_tails = threading.Event() problems: list[str] = [] port = free_port() artifact = build(compiler, stdlib, @@ -791,13 +952,14 @@ def case_event_loop(compiler, interpreter, stdlib, workdir) -> list[str]: page = run_page(interpreter, artifact) answered: list[bytes] = [] failures: list[str] = [] - guard = __import__("threading").Lock() + guard = threading.Lock() def one_client() -> None: try: with socket.create_connection(("127.0.0.1", port), timeout=30) as client: client.sendall(b"part-one ") - time.sleep(gap) + if not release_tails.wait(timeout=25): + raise AssertionError("server did not process every partial request") client.sendall(b"END") reply = client.recv(64) with guard: @@ -806,27 +968,24 @@ def one_client() -> None: with guard: failures.append(repr(failure)) + workers = [] try: wait_for_line(page, "ready") - threading = __import__("threading") - started = time.monotonic() workers = [threading.Thread(target=one_client) for _ in range(clients)] for worker in workers: worker.start() + wait_for_line(page, "all-partial") + release_tails.set() + deadline = time.monotonic() + 35 for worker in workers: - worker.join(timeout=60) - elapsed = time.monotonic() - started + worker.join(timeout=max(0, deadline - time.monotonic())) + if any(worker.is_alive() for worker in workers): + raise AssertionError("client threads exceeded their shared deadline") if len(answered) != clients: problems.append( f"event-loop: {len(answered)} of {clients} clients were answered; " f"first failures {failures[:2]}") - if elapsed > serial_cost / 4: - problems.append( - f"event-loop: {clients} clients took {elapsed:.2f}s, which is not " - f"meaningfully better than the {serial_cost:.0f}s a one-at-a-time " - "server would need, so the connections were not overlapped") - page.wait(timeout=30) output = page.stdout.read() if f"answered {clients}" not in output: @@ -839,10 +998,54 @@ def one_client() -> None: except (AssertionError, subprocess.TimeoutExpired) as failure: problems.append(f"event-loop: {failure!r}") finally: - page.kill() - page.wait(timeout=10) + release_tails.set() + cleanup_page(page) + deadline = time.monotonic() + 35 + for worker in workers: + worker.join(timeout=max(0, deadline - time.monotonic())) + if any(worker.is_alive() for worker in workers): + raise AssertionError("client thread cleanup exceeded its shared deadline") return problems +def case_nonblocking(compiler, interpreter, stdlib, workdir) -> list[str]: + port = free_port() + artifact = build(compiler, stdlib, NONBLOCK_PAGE.replace("PORT", str(port)), workdir, "nonblocking") + page = run_page(interpreter, artifact) + try: + wait_for_line(page, "ready") + with socket.create_connection(("127.0.0.1", port), timeout=10) as client: + wait_for_line(page, "idle -1") + payload = b"abc\0\xff" + client.sendall(payload) + received = b"" + while len(received) < len(payload): + piece = client.recv(64) + if not piece: break + received += piece + if received != payload: + return [f"nonblocking: actual bytes {received!r} differ from {payload!r}"] + client.shutdown(socket.SHUT_WR) + page.wait(timeout=20) + output = page.stdout.read() + expected = ["received 5", "sent 5", "closed 0", "cleared 0", "stale-ready 0", "done"] + if page.returncode != 0 or output.splitlines() != expected: + return [f"nonblocking: exit {page.returncode}, output {output!r}"] + return [] + finally: + cleanup_page(page) + + +def case_shared_listener(compiler, interpreter, stdlib, workdir) -> list[str]: + artifact = build(compiler, stdlib, SHARED_PAGE.replace("PORT", str(free_port())), workdir, "shared_listener") + result = subprocess.run([str(interpreter), str(artifact)], stdin=subprocess.DEVNULL, + capture_output=True, text=True, timeout=20) + expected = (["unsupported net: this platform has no shared-listener SO_REUSEPORT support"] + if os.name == "nt" else ["both shared", "ordinary refused"]) + if result.returncode != 0 or result.stderr or result.stdout.splitlines() != expected: + return [f"shared listener: exit {result.returncode}, output {result.stdout!r}, error {result.stderr!r}"] + return [] + + def main() -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--compiler", type=Path, required=True) @@ -863,6 +1066,8 @@ def main() -> int: ("client", case_client), ("poll", case_poll), ("event-loop", case_event_loop), + ("nonblocking", case_nonblocking), + ("shared-listener", case_shared_listener), ) problems: list[str] = [] diff --git a/tests/tools/ui_display_contract.py b/tests/tools/ui_display_contract.py index f64bdc1e..9a85dec9 100644 --- a/tests/tools/ui_display_contract.py +++ b/tests/tools/ui_display_contract.py @@ -245,6 +245,20 @@ def checked(command, env, timeout=180): if process.stderr: process.stderr.close() +def validate_desktop_report(report, llvm): + names = {'widgets_layout_and_model_binding', 'window_ownership_and_size_errors', + 'dynamic_subtrees_and_effective_state'} + routes = {'vm', 'native'} if llvm == 'ON' else {'vm'} + expected = {(name, 'nil.std.ui.gtk.desktop', route) for name in names for route in routes} + tests = report['tests'] + assert report['schema'] == 'nil.test.results.v1' and not report['errors'], report + assert len(tests) == len(expected), report + assert {(test['name'], test['suite'], test['route']) for test in tests} == expected, report + assert all(test['status'] == 'passed' and test['exitCode'] == 77 and + not test['signal'] and not test['timedOut'] and not test['cancelled'] + for test in tests), report + + def darwin(args): """The AppKit contract. macOS has no detachable display to create, so the runner's own window server session is used; without one there is nothing @@ -269,8 +283,10 @@ def darwin(args): desktop = checked([*base, '--target', 'all' if args.llvm == 'ON' else 'vm', '--filter', 'nil.std.ui.gtk.desktop', args.source.resolve()], env) report = json.loads(desktop.stdout) - assert report['tests'] and not report['errors'], report - assert all(test['status'] == 'passed' for test in report['tests']), report + # Preserve each route's actual results in CTest's retained output, and reject + # missing, duplicated or skipped cases rather than accepting any green subset. + print(json.dumps(report, indent=2), flush=True) + validate_desktop_report(report, args.llvm) # The interaction suite is deliberately not run here. It needs keystrokes # delivered to another process, and on macOS posting events to a different # process requires accessibility trust that a hosted runner does not grant. From 965f159d947b6193b707c4a90221a85520657872 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 17:52:05 -0500 Subject: [PATCH 06/39] Fix Windows backend header collision and hosted platform fixtures --- CMakeLists.txt | 9 ++++++++ nilassembler/main.cpp | 4 ++-- nilcompiler/src/entropy_support.hpp | 2 ++ nilcompiler/src/llvm_backend.cpp | 8 +++---- nilcompiler/src/llvm_backend.hpp | 4 +++- notebook/2026-11-11-web-runtime-parity.md | 22 ++++++++++++++++++ tests/matrix/run_execution_tests.py | 6 +++-- tests/native_windows_header_contract.cpp | 27 +++++++++++++++++++++++ tests/tools/file_modified_at_contract.py | 3 ++- 9 files changed, 75 insertions(+), 10 deletions(-) create mode 100644 tests/native_windows_header_contract.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index f4d082d4..156d5b7b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -101,6 +101,15 @@ install( install(FILES LICENSE DESTINATION ".") if(BUILD_TESTING) + if(WIN32) + # Keep the public backend declarations usable after real Windows RPC + # headers, even in an LLVM-OFF build that normally hides this interface. + add_library(nil_native_windows_header_contract OBJECT tests/native_windows_header_contract.cpp) + target_compile_features(nil_native_windows_header_contract PRIVATE cxx_std_17) + target_compile_definitions(nil_native_windows_header_contract PRIVATE NY_HAS_LLVM=1) + target_include_directories(nil_native_windows_header_contract PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/nilcompiler/src") + add_dependencies(nilcompiler nil_native_windows_header_contract) + endif() add_executable(nil_compiler_module_fingerprint_tests tests/compiler_module_fingerprint_tests.cpp) target_compile_features(nil_compiler_module_fingerprint_tests PRIVATE cxx_std_17) if(WIN32) diff --git a/nilassembler/main.cpp b/nilassembler/main.cpp index efb7b0fd..9cbfc922 100644 --- a/nilassembler/main.cpp +++ b/nilassembler/main.cpp @@ -473,9 +473,9 @@ int run_assembler(const command_line &arguments) { native.setFreestandingEntry(arguments.freestanding); native.setDisableRedZone(arguments.no_red_zone); if (arguments.code_model == "small") { - native.setCodeModel(ny::NativeCodegen::CodeModel::small); + native.setCodeModel(ny::NativeCodegen::CodeModel::Small); } else if (arguments.code_model == "large") { - native.setCodeModel(ny::NativeCodegen::CodeModel::large); + native.setCodeModel(ny::NativeCodegen::CodeModel::Large); } native.setEmbedContext(arguments.embed_context); if (!native.initialize()) { diff --git a/nilcompiler/src/entropy_support.hpp b/nilcompiler/src/entropy_support.hpp index af708092..7e670287 100644 --- a/nilcompiler/src/entropy_support.hpp +++ b/nilcompiler/src/entropy_support.hpp @@ -43,6 +43,8 @@ #include #if defined(__linux__) #include +#elif defined(__APPLE__) +#include #endif #endif diff --git a/nilcompiler/src/llvm_backend.cpp b/nilcompiler/src/llvm_backend.cpp index 49ffdec1..268c779b 100644 --- a/nilcompiler/src/llvm_backend.cpp +++ b/nilcompiler/src/llvm_backend.cpp @@ -1197,7 +1197,7 @@ struct NativeCodegen::Impl { bool embed_context = false; bool freestanding_entry = false; bool disable_red_zone = false; - NativeCodegen::CodeModel code_model = NativeCodegen::CodeModel::host; + NativeCodegen::CodeModel code_model = NativeCodegen::CodeModel::Host; llvm::Function *current_function = nullptr; // State matching generateModule usage @@ -1429,13 +1429,13 @@ bool NativeCodegen::initialize() { // whose kernel places user code high asks for `large` instead. std::optional code_model; switch (impl_->code_model) { - case CodeModel::small: + case CodeModel::Small: code_model = llvm::CodeModel::Small; break; - case CodeModel::large: + case CodeModel::Large: code_model = llvm::CodeModel::Large; break; - case CodeModel::host: + case CodeModel::Host: break; } auto target_machine = std::unique_ptr( diff --git a/nilcompiler/src/llvm_backend.hpp b/nilcompiler/src/llvm_backend.hpp index 48cbe3d8..543a4174 100644 --- a/nilcompiler/src/llvm_backend.hpp +++ b/nilcompiler/src/llvm_backend.hpp @@ -67,7 +67,9 @@ class NativeCodegen { // A freestanding consumer that places an object above 4 GiB needs `large`, // because the small model's R_X86_64_32 relocations cannot reach there and // fail at link time rather than at run time. - enum class CodeModel { host, small, large }; + // Windows RPC headers define lowercase `small` as a type macro. Keep these + // identifiers usable regardless of which platform headers a caller included. + enum class CodeModel { Host, Small, Large }; void setCodeModel(CodeModel model); // Write the generated LLVM IR, so what the backend produced can be diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 6c41f7a6..2c0f2fb4 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -77,6 +77,20 @@ every requested route and prints the full language report into retained CTest output. Positive and five malformed-inventory controls pass locally; actual LLVM-enabled AppKit execution still needs both hosted architectures. +The first hosted campaign at `45f5569` retained actual failures: Linux passed all +six contracts; both Macs failed to compile because Darwin declares `getentropy` +in ``. Windows LLVM-OFF passed socket adapters, same-process cleanup, +timestamp adapters and all ten loopback cases. Its timestamp fixture used the +reserved DOS device name `nul.ny`, and its source-map test compared an unresolved +temporary path with the compiler's canonical path. The fixture now uses a normal +filename and resolves the source path before compilation. The separate Windows +LLVM-ON build exposed the RPC header's `small` macro colliding with the new +backend code-model enum. Capitalized enum members preserve the CLI spellings; +a real Windows-header-first compilation protects that interface. Its MSVC check +passes. The Linux rebuild and Matrix, timestamp and freestanding contracts pass. +These corrections require fresh hosted execution; the original failures remain +under `parity-web-20261111/github-web-43113972394` and the private Windows evidence. + Native socket/byte/entropy parity needs a binary-safe value boundary. The new VM APIs deliberately carry embedded NUL and arbitrary bytes in `text`; the current native text ABI and several LLVM operations use null-terminated strings @@ -85,6 +99,14 @@ truncate legitimate network data and entropy. The scalar timestamp operation can use the existing ordinary-I/O ABI; the byte APIs require an explicit length-aware design and cross-route tests before native support is claimed. +This native limitation also exists on Linux. Host parity for these APIs means +equivalent interpreter behavior and consistent native rejection. A subsequent +counted-text ABI should retain one machine-word language values by pointing to +an immutable length/data descriptor, migrate language literals and runtime text +consumers together, adapt C-string FFI boundaries explicitly, and reject mixed +old/new object and runtime ABIs. A hidden pointer-length registry would retain +the same migration burden while adding lifetime and registration ambiguity. + ## Preserved web handoff The interrupted web tree contains the listener-to-engine replacement, template diff --git a/tests/matrix/run_execution_tests.py b/tests/matrix/run_execution_tests.py index bb0828da..0bce6ed1 100755 --- a/tests/matrix/run_execution_tests.py +++ b/tests/matrix/run_execution_tests.py @@ -478,7 +478,9 @@ def carries(prefix, name, marker): def check_debug_metadata(compiler, interpreter, workdir): """Matrix paragraphs and methods need source maps for editor breakpoints.""" - page = pathlib.Path(workdir) / "source-debug.matrix" + # Source maps use canonical paths. Resolve Windows short-name TEMP aliases + # (and POSIX temporary-directory symlinks) before checking serialized bytes. + page = pathlib.Path(workdir).resolve() / "source-debug.matrix" page.write_text( "book debug-check\n\n" "Reading begin record\n field level as number\nend record\n" @@ -508,7 +510,7 @@ def check_debug_metadata(compiler, interpreter, workdir): # same string and this reduces to the original check. spelling = str(page) if binary else str(page).replace("\\", "\\\\") if (spelling.encode() in data) != debug: - failures.append(f"source debug: path presence incorrect for debug={debug}, binary={binary}") + failures.append(f"source debug: path presence incorrect for debug={debug}, binary={binary}, expected={spelling!r}") if debug and not binary: functions = [part for part in data.decode().split("\nfunction ")[1:]] if len(functions) != 3 or any("# source_debug " not in part.split("end function")[0] for part in functions): diff --git a/tests/native_windows_header_contract.cpp b/tests/native_windows_header_contract.cpp new file mode 100644 index 00000000..abdbfa10 --- /dev/null +++ b/tests/native_windows_header_contract.cpp @@ -0,0 +1,27 @@ +// Compile the native-codegen interface after the real Windows RPC headers. +// These headers define `small` as `char`; the compiler's declarations must +// coexist with that caller environment without undefining its macros. +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#include + +#ifndef small +#error Windows RPC small macro is required for this regression +#endif + +#include "llvm_backend.hpp" +#include + +static_assert(std::is_same::value, + "Including the compiler interface must preserve Windows macros"); +static_assert(std::is_enum::value, + "Native code-model declarations must remain valid after Windows headers"); + +void native_windows_header_contract(ny::NativeCodegen &codegen) { + using Model = ny::NativeCodegen::CodeModel; + codegen.setCodeModel(Model::Host); + codegen.setCodeModel(Model::Small); + codegen.setCodeModel(Model::Large); +} diff --git a/tests/tools/file_modified_at_contract.py b/tests/tools/file_modified_at_contract.py index f2a9145f..269ad408 100644 --- a/tests/tools/file_modified_at_contract.py +++ b/tests/tools/file_modified_at_contract.py @@ -157,7 +157,8 @@ def run(name, command, wanted=b'', limit=30): 'stable_repetitions': 100}) if signature(paths) != before: raise AssertionError('timestamp reads changed the fixtures') - nul = work / 'nul.ny' + # NUL is a reserved DOS device name, even with a file extension. + nul = work / 'embedded-zero-path.ny' nul.write_text(source_for(paths, nul_case=True), encoding='utf-8', newline='\n') for encoding in ('text', 'binary'): output_nil = work / ('nul-' + encoding + '.nil') From 19bd5f1c72c76e83e75fc74b93ea0ce12ff8d759 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 17:58:13 -0500 Subject: [PATCH 07/39] Exercise macOS native allocation retention and full editor integration --- .../workflows/macos-application-parity.yml | 223 ++++++++++++ CMakeLists.txt | 20 ++ docs/platform-support.md | 7 + notebook/2026-11-11-web-runtime-parity.md | 32 ++ tests/darwin_native_heap_budget.c | 322 ++++++++++++++++++ tests/darwin_native_heap_budget.h | 26 ++ tests/darwin_native_heap_budget_control.c | 139 ++++++++ tests/tools/darwin_native_heap_budget.py | 217 ++++++++++++ tests/tools/native_managed_lifetime.py | 37 +- tests/tools/native_owned_lifetime.py | 36 +- 10 files changed, 1035 insertions(+), 24 deletions(-) create mode 100644 .github/workflows/macos-application-parity.yml create mode 100644 tests/darwin_native_heap_budget.c create mode 100644 tests/darwin_native_heap_budget.h create mode 100644 tests/darwin_native_heap_budget_control.c create mode 100644 tests/tools/darwin_native_heap_budget.py diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml new file mode 100644 index 00000000..0d9ea3f7 --- /dev/null +++ b/.github/workflows/macos-application-parity.yml @@ -0,0 +1,223 @@ +name: macOS application platform checks + +"on": + pull_request: + paths: + - '.github/workflows/macos-application-parity.yml' + - 'CMakeLists.txt' + - 'nilcompiler/**' + - 'nilinterpreter/**' + - 'nilstd/**' + - 'nilbuild/**' + - 'niltest/**' + - 'nildebugger/**' + - 'nillsp/**' + - 'aperture/**' + - 'common/**' + - 'nillinker/process_runner*' + - 'validation/ny-3d/scripts/debugger_process.py' + - 'tests/darwin_native_heap_budget*' + - 'tests/tools/darwin_native_heap_budget.py' + - 'tests/tools/native_managed_lifetime.py' + - 'tests/tools/native_owned_lifetime.py' + - 'tests/tools/file_modified_at_contract.py' + - 'vscode/**' + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: macos-application-parity-${{ github.ref }} + cancel-in-progress: true + +# Real LLVM native consumers and an isolated VS Code development-extension host. +# No installed SDK, signed release, personal profile or extension is replaced. +jobs: + applications: + name: macOS ${{ matrix.host.arch }} native applications and editor + runs-on: ${{ matrix.host.runner }} + timeout-minutes: 100 + strategy: + fail-fast: false + matrix: + host: + - runner: macos-15-intel + arch: x86_64 + archive: VSCode-darwin.zip + sha256: 74b1deef1e33cc144d913d60c0e98cef6b1e29c1f2993d905c3b58576fa12202 + - runner: macos-15 + arch: arm64 + archive: VSCode-darwin-arm64.zip + sha256: 6e8d27ee0c5a8f2aa3c43e373407eecd11c270c3fe7f46100ef682aac14bb648 + defaults: + run: + shell: bash + env: + HOMEBREW_NO_AUTO_UPDATE: '1' + PYTHONDONTWRITEBYTECODE: '1' + NIL_EDITOR_ARCHIVE: ${{ matrix.host.archive }} + NIL_EDITOR_SHA256: ${{ matrix.host.sha256 }} + steps: + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 + with: + persist-credentials: false + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 + with: + node-version: '22' + cache: npm + cache-dependency-path: vscode/package-lock.json + - name: Configure LLVM applications + timeout-minutes: 8 + run: | + mkdir -p build/application-evidence + git rev-parse HEAD > build/application-evidence/revision.txt + uname -a > build/application-evidence/host.txt + sw_vers >> build/application-evidence/host.txt + set -o pipefail + nil_llvm_prefix="$(brew --prefix llvm@18)" + test -x "$nil_llvm_prefix/bin/llvm-config" + cmake -S . -B build/application -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_TESTING=ON -DNIL_ENABLE_LLVM=ON \ + "-DLLVM_DIR=$nil_llvm_prefix/lib/cmake/llvm" \ + -DNILCOMPILER_SQLITE_INCLUDE_DIR=OFF -DNILCOMPILER_POSTGRESQL_INCLUDE_DIR=OFF \ + 2>&1 | tee build/application-evidence/configure.log + - name: Build application tools + id: tools + timeout-minutes: 40 + run: | + set -o pipefail + cmake --build build/application --parallel 2 --target \ + nilcompiler nilinterpreter nilassembler nillsp nildebugger nilbuild niltest aperture \ + nil_vscode_host_runner nil_file_time_support_tests \ + 2>&1 | tee build/application-evidence/build.log + - name: Build independent allocation controls + id: heap + timeout-minutes: 3 + run: | + set -o pipefail + cmake --build build/application --parallel 2 --target \ + nil_darwin_native_heap_budget nil_darwin_native_heap_budget_control \ + 2>&1 | tee build/application-evidence/heap-build.log + - name: Verify actual native lifetime and timestamp contracts + timeout-minutes: 15 + run: | + python3 - <<'PYTHON' + import json, subprocess, xml.etree.ElementTree as ET + from pathlib import Path + expected = {'ny.native-managed-lifetime', 'ny.native-owned-lifetime', + 'nil.file-time-support', 'nil.file-modified-at-contract'} + pattern = r'^(ny[.]native-(managed|owned)-lifetime|nil[.]file-(time-support|modified-at-contract))$' + base = ['ctest', '--test-dir', 'build/application', '-R', pattern] + root = Path('build/application-evidence').resolve() + inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) + (root/'inventory.json').write_bytes(inventory) + assert {case['name'] for case in json.loads(inventory)['tests']} == expected + with (root/'ctest.log').open('wb') as output: + run = subprocess.run(base+['--no-tests=error', '--output-on-failure', + '--output-junit', str(root/'results.xml')], stdout=output, + stderr=subprocess.STDOUT, timeout=850) + print((root/'ctest.log').read_text(errors='replace')) + run.check_returncode() + cases = list(ET.parse(root/'results.xml').getroot().iter('testcase')) + assert len(cases) == len(expected) and {case.get('name') for case in cases} == expected + assert all(case.get('status') == 'run' and all(case.find(kind) is None + for kind in ['failure','error','skipped']) for case in cases) + PYTHON + - name: Prepare pinned VS Code in this runner's private build directory + id: editor + if: ${{ !cancelled() && steps.tools.outcome == 'success' }} + timeout-minutes: 6 + run: | + # Version 1.146.0; architecture-specific SHA256 values from Microsoft's + # update API on 2026-11-11. No auto-update or system installation. + mkdir -p build/editor-download build/editor-tmp + curl --fail --location --max-time 180 --max-filesize 800000000 \ + "https://vscode.download.prss.microsoft.com/dbazure/download/stable/69047191fc427f2f92828279aae1d90d8eeda7ab/$NIL_EDITOR_ARCHIVE" \ + -o build/editor-download/editor.zip + python3 - <<'PYTHON' + import hashlib, json, os + from pathlib import Path + path = Path('build/editor-download/editor.zip') + digest_state = hashlib.sha256() + with path.open('rb') as stream: + for chunk in iter(lambda: stream.read(1024*1024), b''): digest_state.update(chunk) + digest = digest_state.hexdigest() + assert digest == os.environ['NIL_EDITOR_SHA256'] + Path('build/application-evidence/editor.json').write_text(json.dumps(dict( + version='1.146.0', revision='69047191fc427f2f92828279aae1d90d8eeda7ab', + archive=os.environ['NIL_EDITOR_ARCHIVE'], sha256=digest, bytes=path.stat().st_size), indent=2)+'\n') + PYTHON + ditto -x -k build/editor-download/editor.zip build/editor-download + - name: Build and check the extension + id: extension + if: ${{ !cancelled() && steps.editor.outcome == 'success' }} + timeout-minutes: 5 + working-directory: vscode + run: | + npm ci + npm test + npm run build + - name: Exercise the real editor host + if: ${{ !cancelled() && steps.extension.outcome == 'success' }} + timeout-minutes: 17 + env: + NIL_VSCODE_HOST_TIMEOUT_MS: '900000' + run: | + set -o pipefail + export VSCODE_EXECUTABLE="$GITHUB_WORKSPACE/build/editor-download/Visual Studio Code.app/Contents/MacOS/Electron" + export NIL_VSCODE_TOOL_DIRECTORY="$GITHUB_WORKSPACE/build/application/bin" + export NIL_VSCODE_HOST_RUNNER="$NIL_VSCODE_TOOL_DIRECTORY/nil_vscode_host_runner" + export NIL_VSCODE_SDK_PATH="$GITHUB_WORKSPACE" + export TMPDIR="$GITHUB_WORKSPACE/build/editor-tmp" + for nil_subset in NIL_VSCODE_NILDATA_ONLY NIL_VSCODE_LANGUAGE_VALUES_ONLY \ + NIL_VSCODE_MEMBER_SIGNATURE_ONLY NIL_VSCODE_COMPLETION_ONLY NIL_VSCODE_SEMANTIC_ONLY \ + NIL_VSCODE_WORKSPACE_ONLY NIL_VSCODE_ANALYSIS_ONLY NIL_VSCODE_ANALYSIS_PROBE_ONLY; do + unset "$nil_subset" + done + node vscode/test/run-host.cjs 2>&1 | tee build/application-evidence/editor-host.log + python3 - <<'PYTHON' + import json + from pathlib import Path + report = json.loads(Path('build/vscode-host-smoke/latest.json').read_text()) + assert report['platform'] == 'darwin' and report['status'] == 'PASS' + assert report['vscode'] == '1.146.0' and not report['skippedChecks'] + assert len(report['checks']) >= 100 and all(row['status'] == 'PASS' for row in report['checks']) + PYTHON + - name: Preserve complete or partial evidence + if: always() + timeout-minutes: 2 + run: | + python3 - <<'PYTHON' + import hashlib, json, os, shutil, subprocess + from pathlib import Path + root = Path('build/application-evidence'); root.mkdir(parents=True, exist_ok=True) + for source, name in [('build/application/test-evidence','contracts'), + ('build/vscode-host-smoke','editor-reports')]: + if Path(source).is_dir(): shutil.copytree(source, root/name) + dirty = subprocess.check_output(['git','status','--porcelain','--untracked-files=no'], text=True, timeout=15) + revision = subprocess.check_output(['git','rev-parse','HEAD'], text=True, timeout=15).strip() + files = {} + for path in sorted(root.rglob('*')): + if path.is_file() and not path.is_symlink(): + digest_state = hashlib.sha256() + with path.open('rb') as stream: + for chunk in iter(lambda: stream.read(1024*1024), b''): digest_state.update(chunk) + digest = digest_state.hexdigest() + files[path.relative_to(root).as_posix()] = dict(bytes=path.stat().st_size,sha256=digest) + (root/'evidence.json').write_text(json.dumps(dict(revision=revision, + source_unchanged=not dirty,files=files),indent=2)+'\n') + assert not dirty and revision == os.environ['GITHUB_SHA'] + PYTHON + - name: Upload application results + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: macos-applications-${{ matrix.host.arch }}-${{ github.sha }}-${{ github.run_attempt }} + path: | + build/application-evidence/ + build/application/Testing/Temporary/LastTest.log + build/application/CMakeCache.txt + build/editor-tmp/**/profile/logs/** + if-no-files-found: error + retention-days: 14 diff --git a/CMakeLists.txt b/CMakeLists.txt index 156d5b7b..43c3485e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -975,6 +975,20 @@ if(BUILD_TESTING) set(NIL_MEMORY_LIMIT_ARGUMENTS --memory-launcher $) add_test(NAME nil.native-memory-limit-contract COMMAND nil_native_memory_limiter --self-test) set_tests_properties(nil.native-memory-limit-contract PROPERTIES LABELS "native;memory;rc" TIMEOUT 60) + elseif(APPLE) + # A requested-allocation retention oracle for these exact native + # fixtures. This is distinct from Linux AS and Windows commit caps. + add_library(nil_darwin_native_heap_budget SHARED tests/darwin_native_heap_budget.c) + add_executable(nil_darwin_native_heap_budget_control tests/darwin_native_heap_budget_control.c) + foreach(nil_heap_target IN ITEMS nil_darwin_native_heap_budget nil_darwin_native_heap_budget_control) + target_compile_features(${nil_heap_target} PRIVATE c_std_11) + target_compile_options(${nil_heap_target} PRIVATE -fno-builtin -Wall -Wextra -Werror -mmacosx-version-min=14.0) + target_link_options(${nil_heap_target} PRIVATE -mmacosx-version-min=14.0) + endforeach() + set(NIL_MEMORY_LIMIT_ARGUMENTS + --heap-budget-library $ + --heap-budget-control $ + --output-parent "${CMAKE_BINARY_DIR}/test-evidence") endif() add_test( NAME ny.native-managed-lifetime @@ -994,6 +1008,12 @@ if(BUILD_TESTING) --compiler $ --llvm "${NIL_ENABLE_LLVM}" ${NIL_MEMORY_LIMIT_ARGUMENTS}) set_tests_properties(ny.native-owned-lifetime PROPERTIES LABELS "native;ownership;rc" SKIP_RETURN_CODE 77 TIMEOUT 240) + if(APPLE) + # Includes independent allocator controls and a second execution of + # each unchanged native fixture with real frees deliberately withheld. + set_tests_properties(ny.native-managed-lifetime PROPERTIES TIMEOUT 240) + set_tests_properties(ny.native-owned-lifetime PROPERTIES TIMEOUT 360) + endif() # Matrix lexer conformance: the sample corpus must tokenise clean, and # each negative fixture must be rejected for the stated reason. add_executable(matrix_lex_dump tests/matrix/lex_dump.cpp) diff --git a/docs/platform-support.md b/docs/platform-support.md index 64e53e13..ea11caa6 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -204,3 +204,10 @@ alone do not establish Windows or macOS support, and declarations do not establi native/JIT execution. Cross-platform validation of these additions is in progress. Their evidence must join the older desktop, tooling and SDK baselines before declaring current-source parity. + +The private Windows LLVM-ON build at `965f159` passes all six focused web-runtime +contracts with no skips, including VM socket cleanup and the native timestamp +routes. The corresponding Linux contracts pass. Hosted Intel/ARM acceptance and +current macOS editor coverage are still in progress. The socket/byte/entropy +native limitation is common to Linux, Windows and macOS; those APIs currently +require the interpreter on all three hosts. diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 2c0f2fb4..eaeaeaed 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -91,6 +91,38 @@ passes. The Linux rebuild and Matrix, timestamp and freestanding contracts pass. These corrections require fresh hosted execution; the original failures remain under `parity-web-20261111/github-web-43113972394` and the private Windows evidence. +At `965f159`, the fresh Windows LLVM-ON source build and all six focused contracts +pass with no skips: 118 adapter assertions, 184 VM lifecycle assertions, 129 +timestamp helper assertions, 38 Matrix fixtures, ten loopback cases and the public +timestamp route contract. The full run took 103.79 seconds and preserved the old +dirty guest checkout and accepted SDK. Its report SHA is +`d1d823e482290dd3044d872a97d1ca79ecadfb22a1e858a1a4972bf8f3fe1649`. +A separate eight-case Windows probe confirms that textual and binary source maps +embed canonical long paths even when the compiler receives an 8.3 TEMP alias. + +## Darwin allocation retention and editor coverage + +The two native lifetime programs remain unchanged. Darwin now has a test-only +interposer to enforce **96 MiB of live requested native heap allocations**. +This metric excludes allocator overhead, the fixed tracker, mapped images and +allocation paths outside the intercepted API; it is not a claim of whole-process +address-space/commit parity. The harness audits the generated binary's allocator +imports, requires an injection handshake, checks the unchanged loops' allocation +counts and repeats each exact executable while withholding real target frees. +Retained untouched allocations must also fail, so a sampled footprint cannot +substitute for the allocation-retention check. Linux RLIMIT_AS and Windows Job +tests retain their original behavior. + +GCC and Clang portable accounting controls pass: over 2 GiB allocated and freed +with a 64 KiB peak, plus realloc growth/shrink/failure, alignment and overflow +controls. Untouched retention and withheld-free controls reject at 96 MiB. +These local controls do not establish Mach-O injection. The new application +workflow requires real Intel and ARM execution, the existing native timestamp +routes, and the full VS Code extension-host campaign with no recorded exclusions. +It uses a fresh profile and Microsoft's architecture-specific VS Code 1.146.0 +archives, pinned by SHA256 from the official update service. All results are +development/build-tree checks; no installed SDK or user editor is activated. + Native socket/byte/entropy parity needs a binary-safe value boundary. The new VM APIs deliberately carry embedded NUL and arbitrary bytes in `text`; the current native text ABI and several LLVM operations use null-terminated strings diff --git a/tests/darwin_native_heap_budget.c b/tests/darwin_native_heap_budget.c new file mode 100644 index 00000000..27f18134 --- /dev/null +++ b/tests/darwin_native_heap_budget.c @@ -0,0 +1,322 @@ +/* Test-only allocator retention oracle for owned, non-hardened child binaries. + * DYLD's self-image exclusion lets replacements call the original entry point: + * https://github.com/apple-oss-distributions/dyld/blob/main/include/mach-o/dyld-interposing.h + * No production allocator counters, sampled RSS, or privileged kernel settings. + * Build with -fno-builtin and a macOS >=14 deployment target for typed malloc. + */ +#include "darwin_native_heap_budget.h" +#include +#include +#include +#include +#include +#include + +#if defined(__APPLE__) +#include +#include +#include +#define ORIGINAL(name) name +#elif defined(NIL_HEAP_BUDGET_PORTABLE_TEST) +/* Linux --wrap exercises the identical accounting and controls locally. It + * does not validate Mach-O interposition and cannot admit Darwin coverage. */ +extern void *__real_malloc(size_t); +extern void *__real_calloc(size_t, size_t); +extern void *__real_realloc(void *, size_t); +extern void __real_free(void *); +extern void *__real_aligned_alloc(size_t, size_t); +extern int __real_posix_memalign(void **, size_t, size_t); +extern void *__real_valloc(size_t); +#define ORIGINAL(name) __real_##name +#else +#error This test requires Darwin (or the explicit portable accounting check). +#endif + +#define TABLE_COUNT (1u << 17) +#define TOMBSTONE ((void *)(uintptr_t)1) +struct entry { void *pointer; size_t bytes; }; +static struct entry entries[TABLE_COUNT]; /* fixed 2 MiB on the supported hosts */ +static atomic_flag tracker_lock = ATOMIC_FLAG_INIT; +static struct nil_heap_snapshot stats = { + .version = NIL_HEAP_ABI_VERSION, .budget = NIL_HEAP_BUDGET_BYTES +}; +static int report_fd = -1, ready, withhold, fail_realloc; +static char token[33]; + +static void lock_tracker(void) { + while (atomic_flag_test_and_set_explicit(&tracker_lock, memory_order_acquire)) {} +} +static void unlock_tracker(void) { + atomic_flag_clear_explicit(&tracker_lock, memory_order_release); +} + +/* Formatting and reporting cannot allocate, even on budget/metadata failure. */ +static char *append_text(char *at, const char *text) { + while (*text) *at++ = *text++; + return at; +} +static char *append_number(char *at, uint64_t number) { + char reverse[24]; size_t used = 0; + do { reverse[used++] = (char)('0' + number % 10); number /= 10; } while (number); + while (used) *at++ = reverse[--used]; + return at; +} +static char *field(char *at, const char *name, uint64_t value) { + at = append_text(at, ",\""); at = append_text(at, name); + at = append_text(at, "\":"); return append_number(at, value); +} +static void emit_locked(const char *event) { + char line[2048], *at = line; + if (report_fd < 0) return; + at = append_text(at, "{\"event\":\""); at = append_text(at, event); + at = append_text(at, "\",\"token\":\""); at = append_text(at, token); + at = append_text(at, "\""); + at = field(at, "pid", (uint64_t)getpid()); +#define EMIT(member) at = field(at, #member, stats.member) + EMIT(version); EMIT(budget); EMIT(live); EMIT(peak); EMIT(cumulative); + EMIT(allocations); EMIT(frees); EMIT(reallocations); EMIT(failures); + EMIT(unknown_frees); EMIT(withheld); EMIT(target_size); + EMIT(target_allocations); EMIT(target_frees); EMIT(denied_request); EMIT(typed_calls); +#undef EMIT + at = append_text(at, "}\n"); + size_t left = (size_t)(at - line); const char *next = line; + while (left) { + ssize_t written = write(report_fd, next, left); + if (written < 0 && errno == EINTR) continue; + if (written <= 0) _exit(NIL_HEAP_TRACKER_EXIT); + next += written; left -= (size_t)written; + } +} +static void fail_locked(const char *event, int code) { + emit_locked(event); + _exit(code); +} + +static size_t slot_for(void *pointer, int inserting) { + size_t first_dead = TABLE_COUNT; + size_t start = ((uintptr_t)pointer >> 4) * UINT64_C(11400714819323198485); + for (size_t offset = 0; offset < TABLE_COUNT; ++offset) { + size_t slot = (start + offset) & (TABLE_COUNT - 1); + if (entries[slot].pointer == pointer) return slot; + if (entries[slot].pointer == TOMBSTONE && first_dead == TABLE_COUNT) first_dead = slot; + if (!entries[slot].pointer) return inserting ? (first_dead == TABLE_COUNT ? slot : first_dead) : TABLE_COUNT; + } + return inserting ? first_dead : TABLE_COUNT; +} +static void insert_locked(void *pointer, size_t bytes) { + size_t slot = slot_for(pointer, 1); + if (slot == TABLE_COUNT) fail_locked("table-full", NIL_HEAP_TRACKER_EXIT); + if (entries[slot].pointer == pointer) fail_locked("duplicate-pointer", NIL_HEAP_TRACKER_EXIT); + entries[slot].pointer = pointer; entries[slot].bytes = bytes; +} +static void remove_locked(size_t slot) { + entries[slot].pointer = TOMBSTONE; entries[slot].bytes = 0; +} +static void reserve_locked(size_t bytes) { + if (bytes > stats.budget - stats.live) { + stats.denied_request = bytes; + fail_locked("budget", NIL_HEAP_BUDGET_EXIT); + } + stats.live += bytes; + if (stats.live > stats.peak) stats.peak = stats.live; +} +static void reserve(size_t bytes, int typed) { + lock_tracker(); stats.typed_calls += (uint64_t)typed; + reserve_locked(bytes); unlock_tracker(); +} +static void successful_request_locked(size_t bytes) { + if (UINT64_MAX - stats.cumulative < bytes) fail_locked("counter-overflow", NIL_HEAP_TRACKER_EXIT); + stats.cumulative += bytes; ++stats.allocations; + if (bytes == stats.target_size) ++stats.target_allocations; +} +static void *finish_allocate(void *pointer, size_t bytes) { + int saved_errno = errno; + lock_tracker(); + if (pointer) { insert_locked(pointer, bytes); successful_request_locked(bytes); } + else { stats.live -= bytes; ++stats.failures; } + unlock_tracker(); errno = saved_errno; return pointer; +} +static int calloc_size(size_t count, size_t size, size_t *bytes, int typed) { + if (size && count > SIZE_MAX / size) { + lock_tracker(); ++stats.failures; stats.typed_calls += (uint64_t)typed; unlock_tracker(); + errno = ENOMEM; return 0; + } + *bytes = count * size; return 1; +} +static int release(void *pointer, int typed) { + if (!pointer) return 1; + lock_tracker(); stats.typed_calls += (uint64_t)typed; + size_t slot = slot_for(pointer, 0); + if (slot == TABLE_COUNT) ++stats.unknown_frees; + else { + size_t bytes = entries[slot].bytes; + if (withhold && bytes == stats.target_size) { + ++stats.withheld; unlock_tracker(); return 0; + } + stats.live -= bytes; ++stats.frees; + if (bytes == stats.target_size) ++stats.target_frees; + remove_locked(slot); + } + unlock_tracker(); return 1; +} + +/* Remove the old key before calling libc: realloc may release it and another + * thread may reuse that address. Keep its bytes reserved until libc returns. + * No tracker lock is held across a real allocator call. */ +struct realloc_state { size_t old_bytes, reserved; int tracked, denied; }; +static struct realloc_state begin_realloc(void *pointer, size_t bytes, int typed) { + struct realloc_state state = {0, 0, 0, 0}; + lock_tracker(); ++stats.reallocations; stats.typed_calls += (uint64_t)typed; + if (fail_realloc) { fail_realloc = 0; ++stats.failures; state.denied = 1; unlock_tracker(); return state; } + size_t slot = pointer ? slot_for(pointer, 0) : TABLE_COUNT; + if (slot != TABLE_COUNT) { + state.old_bytes = entries[slot].bytes; state.tracked = 1; remove_locked(slot); + } + state.reserved = bytes > state.old_bytes ? bytes - state.old_bytes : 0; + reserve_locked(state.reserved); unlock_tracker(); return state; +} +static void *finish_realloc(void *old_pointer, void *pointer, size_t bytes, struct realloc_state state) { + int saved_errno = errno; + lock_tracker(); stats.live -= state.reserved; + if (!pointer && bytes != 0) { + if (state.tracked) insert_locked(old_pointer, state.old_bytes); + ++stats.failures; + } else { + stats.live -= state.old_bytes; + if (state.tracked) { + ++stats.frees; + if (state.old_bytes == stats.target_size) ++stats.target_frees; + } + if (pointer) { + stats.live += bytes; insert_locked(pointer, bytes); successful_request_locked(bytes); + } + } + unlock_tracker(); errno = saved_errno; return pointer; +} + +static void *budget_malloc(size_t bytes) { + reserve(bytes, 0); return finish_allocate(ORIGINAL(malloc)(bytes), bytes); +} +static void *budget_calloc(size_t count, size_t size) { + size_t bytes; if (!calloc_size(count, size, &bytes, 0)) return NULL; + reserve(bytes, 0); return finish_allocate(ORIGINAL(calloc)(count, size), bytes); +} +static void budget_free(void *pointer) { + if (release(pointer, 0)) ORIGINAL(free)(pointer); +} +static void *budget_realloc(void *pointer, size_t bytes) { + struct realloc_state state = begin_realloc(pointer, bytes, 0); + if (state.denied) { errno = ENOMEM; return NULL; } + return finish_realloc(pointer, ORIGINAL(realloc)(pointer, bytes), bytes, state); +} +static void *budget_aligned_alloc(size_t alignment, size_t bytes) { + reserve(bytes, 0); return finish_allocate(ORIGINAL(aligned_alloc)(alignment, bytes), bytes); +} +static int budget_posix_memalign(void **out, size_t alignment, size_t bytes) { + reserve(bytes, 0); void *pointer = NULL; + int result = ORIGINAL(posix_memalign)(&pointer, alignment, bytes); + finish_allocate(result == 0 ? pointer : NULL, bytes); + if (result == 0) *out = pointer; + return result; +} +static void *budget_valloc(size_t bytes) { + reserve(bytes, 0); return finish_allocate(ORIGINAL(valloc)(bytes), bytes); +} + +#if defined(__APPLE__) +DYLD_INTERPOSE(budget_malloc, malloc) +DYLD_INTERPOSE(budget_calloc, calloc) +DYLD_INTERPOSE(budget_free, free) +DYLD_INTERPOSE(budget_realloc, realloc) +DYLD_INTERPOSE(budget_aligned_alloc, aligned_alloc) +DYLD_INTERPOSE(budget_posix_memalign, posix_memalign) +DYLD_INTERPOSE(budget_valloc, valloc) + +#if __MAC_OS_X_VERSION_MAX_ALLOWED >= 140000 +/* Modern Apple Clang can rewrite ordinary calls to typed allocation symbols. + * Intercept these actual entry points too, preserving the type argument. */ +static void *budget_type_malloc(size_t bytes, malloc_type_id_t type) { + reserve(bytes, 1); return finish_allocate(malloc_type_malloc(bytes, type), bytes); +} +static void *budget_type_calloc(size_t count, size_t size, malloc_type_id_t type) { + size_t bytes; if (!calloc_size(count, size, &bytes, 1)) return NULL; + reserve(bytes, 1); return finish_allocate(malloc_type_calloc(count, size, type), bytes); +} +static void budget_type_free(void *pointer, malloc_type_id_t type) { + if (release(pointer, 1)) malloc_type_free(pointer, type); +} +static void *budget_type_realloc(void *pointer, size_t bytes, malloc_type_id_t type) { + struct realloc_state state = begin_realloc(pointer, bytes, 1); + if (state.denied) { errno = ENOMEM; return NULL; } + return finish_realloc(pointer, malloc_type_realloc(pointer, bytes, type), bytes, state); +} +static void *budget_type_aligned_alloc(size_t alignment, size_t bytes, malloc_type_id_t type) { + reserve(bytes, 1); return finish_allocate(malloc_type_aligned_alloc(alignment, bytes, type), bytes); +} +static int budget_type_posix_memalign(void **out, size_t alignment, size_t bytes, malloc_type_id_t type) { + reserve(bytes, 1); void *pointer = NULL; + int result = malloc_type_posix_memalign(&pointer, alignment, bytes, type); + finish_allocate(result == 0 ? pointer : NULL, bytes); + if (result == 0) *out = pointer; + return result; +} +static void *budget_type_valloc(size_t bytes, malloc_type_id_t type) { + reserve(bytes, 1); return finish_allocate(malloc_type_valloc(bytes, type), bytes); +} +DYLD_INTERPOSE(budget_type_malloc, malloc_type_malloc) +DYLD_INTERPOSE(budget_type_calloc, malloc_type_calloc) +DYLD_INTERPOSE(budget_type_free, malloc_type_free) +DYLD_INTERPOSE(budget_type_realloc, malloc_type_realloc) +DYLD_INTERPOSE(budget_type_aligned_alloc, malloc_type_aligned_alloc) +DYLD_INTERPOSE(budget_type_posix_memalign, malloc_type_posix_memalign) +DYLD_INTERPOSE(budget_type_valloc, malloc_type_valloc) +#endif +#else +void *__wrap_malloc(size_t bytes) { return budget_malloc(bytes); } +void *__wrap_calloc(size_t count, size_t size) { return budget_calloc(count, size); } +void *__wrap_realloc(void *pointer, size_t bytes) { return budget_realloc(pointer, bytes); } +void __wrap_free(void *pointer) { budget_free(pointer); } +void *__wrap_aligned_alloc(size_t alignment, size_t bytes) { return budget_aligned_alloc(alignment, bytes); } +int __wrap_posix_memalign(void **out, size_t alignment, size_t bytes) { return budget_posix_memalign(out, alignment, bytes); } +void *__wrap_valloc(size_t bytes) { return budget_valloc(bytes); } +#endif + +int nil_heap_budget_snapshot(struct nil_heap_snapshot *out, size_t size) { + if (!ready || !out || size != sizeof(*out)) return 0; + lock_tracker(); *out = stats; unlock_tracker(); return 1; +} +void nil_heap_budget_fail_next_realloc(void) { + lock_tracker(); fail_realloc = 1; unlock_tracker(); +} + +__attribute__((constructor)) static void initialize_budget(void) { + const char *path = getenv("NIL_DARWIN_HEAP_REPORT"); + const char *given_token = getenv("NIL_DARWIN_HEAP_TOKEN"); + const char *target = getenv("NIL_DARWIN_HEAP_TARGET"); + const char *mode = getenv("NIL_DARWIN_HEAP_WITHHOLD"); + uint64_t target_size; + int withhold_mode; + if (!path || !*path || !given_token || strlen(given_token) != 32 || !target || !mode) + _exit(NIL_HEAP_TRACKER_EXIT); + for (size_t index = 0; index < 32; ++index) { + char value = given_token[index]; + if (!((value >= '0' && value <= '9') || (value >= 'a' && value <= 'f'))) + _exit(NIL_HEAP_TRACKER_EXIT); + token[index] = value; + } + if (strcmp(target, "16384") == 0) target_size = 16384; + else if (strcmp(target, "65536") == 0) target_size = 65536; + else _exit(NIL_HEAP_TRACKER_EXIT); + if (strcmp(mode, "0") == 0) withhold_mode = 0; + else if (strcmp(mode, "1") == 0) withhold_mode = 1; + else _exit(NIL_HEAP_TRACKER_EXIT); + int descriptor = open(path, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600); + if (descriptor < 0) _exit(NIL_HEAP_TRACKER_EXIT); + lock_tracker(); + stats.target_size = target_size; withhold = withhold_mode; report_fd = descriptor; + ready = 1; emit_locked("ready"); unlock_tracker(); +} +__attribute__((destructor)) static void finalize_budget(void) { + if (!ready) return; + lock_tracker(); emit_locked("final"); close(report_fd); report_fd = -1; unlock_tracker(); +} diff --git a/tests/darwin_native_heap_budget.h b/tests/darwin_native_heap_budget.h new file mode 100644 index 00000000..de05f444 --- /dev/null +++ b/tests/darwin_native_heap_budget.h @@ -0,0 +1,26 @@ +#ifndef NIL_DARWIN_NATIVE_HEAP_BUDGET_H +#define NIL_DARWIN_NATIVE_HEAP_BUDGET_H + +#include +#include + +#define NIL_HEAP_BUDGET_BYTES (UINT64_C(96) * 1024 * 1024) +#define NIL_HEAP_BUDGET_EXIT 86 +#define NIL_HEAP_TRACKER_EXIT 87 +#define NIL_HEAP_ABI_VERSION 1 + +/* Test ABI only. Bytes are intercepted requested payloads, including in-flight + * reservations, not malloc usable sizes, RSS, VM reservations, or process commit. + * The fixed tracker table and allocation paths that bypass these entry points + * are outside this metric. The harness independently checks fixture coverage. */ +struct nil_heap_snapshot { + uint64_t version, budget, live, peak, cumulative, allocations, frees; + uint64_t reallocations, failures, unknown_frees, withheld; + uint64_t target_size, target_allocations, target_frees; + uint64_t denied_request, typed_calls; +}; + +typedef int (*nil_heap_snapshot_fn)(struct nil_heap_snapshot *, size_t); +typedef void (*nil_heap_fail_realloc_fn)(void); + +#endif diff --git a/tests/darwin_native_heap_budget_control.c b/tests/darwin_native_heap_budget_control.c new file mode 100644 index 00000000..912ff484 --- /dev/null +++ b/tests/darwin_native_heap_budget_control.c @@ -0,0 +1,139 @@ +/* Independent consumers exercise actual libc calls, including untouched calloc. + * -fno-builtin keeps the optimizer from removing allocation/free pairs. */ +#include "darwin_native_heap_budget.h" +#include +#include +#include +#include +#include +#if defined(__APPLE__) +#include +#include +#endif + +static nil_heap_snapshot_fn snapshot; +static nil_heap_fail_realloc_fn fail_next_realloc; +static void *volatile retained[2048]; +static volatile unsigned char observed; + +static void write_all(int fd, const char *text) { + size_t left = strlen(text); + while (left) { + ssize_t written = write(fd, text, left); + if (written < 0 && errno == EINTR) continue; + if (written <= 0) _exit(91); + text += written; left -= (size_t)written; + } +} +static void require(int condition, const char *message) { + if (!condition) { write_all(2, message); _exit(90); } +} +static struct nil_heap_snapshot state(void) { + struct nil_heap_snapshot result; + require(snapshot(&result, sizeof(result)), "snapshot unavailable\n"); + require(result.version == NIL_HEAP_ABI_VERSION && result.budget == NIL_HEAP_BUDGET_BYTES, + "snapshot ABI or budget differs\n"); + return result; +} +static void live_is(uint64_t bytes) { + require(state().live == bytes, "live requested bytes differ\n"); +} + +static void positive(void) { + struct nil_heap_snapshot before = state(); + unsigned char *value = malloc(16384); + require(value != NULL, "malloc failed\n"); live_is(before.live + 16384); + value[0] = 19; value[16383] = 23; + value = realloc(value, 65536); + require(value && value[0] == 19 && value[16383] == 23, "realloc growth lost content\n"); + live_is(before.live + 65536); + fail_next_realloc(); errno = 0; + void *failed = realloc(value, 131072); + require(!failed && errno == ENOMEM && value[0] == 19 && value[16383] == 23, + "failed realloc invalidated original allocation\n"); + live_is(before.live + 65536); + value = realloc(value, 4096); + require(value && value[0] == 19, "realloc shrink lost content\n"); + live_is(before.live + 4096); + free(value); live_is(before.live); + value = realloc(NULL, 8192); + require(value != NULL, "realloc NULL failed\n"); live_is(before.live + 8192); + free(value); live_is(before.live); + value = malloc(0); + live_is(before.live); free(value); live_is(before.live); + value = calloc(2, 4096); + require(value != NULL, "calloc failed\n"); live_is(before.live + 8192); + for (size_t index = 0; index < 8192; ++index) require(value[index] == 0, "calloc not zeroed\n"); + free(value); live_is(before.live); + volatile size_t huge_count = SIZE_MAX / 3 + 1; + errno = 0; + require(!calloc(huge_count, 3) && errno == ENOMEM, "calloc overflow accepted\n"); + live_is(before.live); + value = aligned_alloc(64, 4096); + require(value && (uintptr_t)value % 64 == 0, "aligned_alloc failed\n"); + live_is(before.live + 4096); free(value); live_is(before.live); + void *aligned = NULL; + require(posix_memalign(&aligned, 256, 8192) == 0 && aligned && (uintptr_t)aligned % 256 == 0, + "posix_memalign failed\n"); + live_is(before.live + 8192); free(aligned); live_is(before.live); + value = valloc(4096); + require(value && (uintptr_t)value % (size_t)getpagesize() == 0, "valloc failed\n"); + live_is(before.live + 4096); free(value); live_is(before.live); + free(NULL); live_is(before.live); +#if defined(__APPLE__) && __MAC_OS_X_VERSION_MAX_ALLOWED >= 140000 + uint64_t typed_before = state().typed_calls; + value = malloc_type_malloc(16384, 0); + require(value != NULL, "typed malloc failed\n"); live_is(before.live + 16384); + value = malloc_type_realloc(value, 32768, 0); + require(value != NULL, "typed realloc failed\n"); live_is(before.live + 32768); + malloc_type_free(value, 0); live_is(before.live); + value = malloc_type_calloc(2, 4096, 0); + require(value && value[0] == 0 && value[8191] == 0, "typed calloc failed\n"); + live_is(before.live + 8192); malloc_type_free(value, 0); live_is(before.live); + value = malloc_type_aligned_alloc(64, 4096, 0); + require(value && (uintptr_t)value % 64 == 0, "typed aligned_alloc failed\n"); + live_is(before.live + 4096); malloc_type_free(value, 0); live_is(before.live); + aligned = NULL; + require(malloc_type_posix_memalign(&aligned, 256, 8192, 0) == 0 && aligned, + "typed posix_memalign failed\n"); + live_is(before.live + 8192); malloc_type_free(aligned, 0); live_is(before.live); + value = malloc_type_valloc(4096, 0); + require(value != NULL, "typed valloc failed\n"); + live_is(before.live + 4096); malloc_type_free(value, 0); live_is(before.live); + require(state().typed_calls >= typed_before + 11, "typed interception absent\n"); +#endif + /* >2 GiB cumulatively; never more than 64 KiB of this loop remains live. */ + for (size_t index = 0; index < 34000; ++index) { + value = calloc(1, 65536); + require(value != NULL, "sustained calloc failed\n"); + observed = value[0]; observed = value[65535]; + free(value); + } + live_is(before.live); + struct nil_heap_snapshot after = state(); + require(after.cumulative - before.cumulative > UINT64_C(2) * 1024 * 1024 * 1024, + "sustained allocation coverage too small\n"); + require(after.target_allocations >= 34000 && after.target_frees >= 34000, + "sustained target interception incomplete\n"); + require(after.reallocations >= before.reallocations + 4 && after.failures >= before.failures + 2, + "realloc/failure controls incomplete\n"); + write_all(1, "heap-controls-ok\n"); +} + +int main(int argc, char **argv) { + snapshot = (nil_heap_snapshot_fn)dlsym(RTLD_DEFAULT, "nil_heap_budget_snapshot"); + fail_next_realloc = (nil_heap_fail_realloc_fn)dlsym(RTLD_DEFAULT, "nil_heap_budget_fail_next_realloc"); + require(snapshot && fail_next_realloc, "heap budget injection absent\n"); + require(argc == 2, "expected one control mode\n"); + (void)state(); + if (strcmp(argv[1], "positive") == 0) { positive(); return 0; } + require(strcmp(argv[1], "retain") == 0, "unknown control mode\n"); + /* Do not touch payload pages: a footprint-only replacement can miss this. + * Even if the oracle fails, this control requests at most 128 MiB. */ + for (size_t index = 0; index < 2048; ++index) { + retained[index] = calloc(1, 65536); + require(retained[index] != NULL, "retention control allocator failed first\n"); + } + require(0, "retention control exceeded its independent safety ceiling\n"); + return 90; +} diff --git a/tests/tools/darwin_native_heap_budget.py b/tests/tools/darwin_native_heap_budget.py new file mode 100644 index 00000000..28b06e97 --- /dev/null +++ b/tests/tools/darwin_native_heap_budget.py @@ -0,0 +1,217 @@ +"""Bounded Darwin-only requested-allocation retention proof for native fixtures. + +This is not a whole-process address-space, committed-memory, or RSS limit. +The unchanged native fixture must independently demonstrate its expected calloc +payload count, and withheld real frees must make that same executable fail. +The tracker is injected only into fresh owned consumers, never the compiler or +the Python process-ownership gate. Raw successes and failures are retained. +""" +from __future__ import annotations + +import json +import os +from pathlib import Path +import re +import secrets +import sys +import tempfile +import time + +import ordinary_io_parity_contract as support + +ROOT = Path(__file__).resolve().parents[2] +BUDGET = 96 * 1024 * 1024 +METRIC = '96 MiB live requested native heap allocations (Darwin test interposer)' +ALLOCATORS = {'malloc', 'calloc', 'realloc', 'free', 'aligned_alloc', 'posix_memalign', 'valloc'} +ALLOCATORS |= {'malloc_type_' + name for name in tuple(ALLOCATORS)} + + +def add_arguments(parser): + parser.add_argument('--heap-budget-library', type=Path) + parser.add_argument('--heap-budget-control', type=Path) + parser.add_argument('--output-parent', type=Path, + help='Retain Darwin raw evidence in a fresh child directory') + + +class DarwinHeapBudget: + def __init__(self, args, name, *, seconds, fixtures): + if sys.platform != 'darwin': + raise AssertionError('Darwin interposition requires actual macOS execution') + if not args.heap_budget_library or not args.heap_budget_control: + raise AssertionError('Darwin lifetime tests require the heap budget library and control executable') + self.library = args.heap_budget_library.resolve(strict=True) + self.control = args.heap_budget_control.resolve(strict=True) + parent = args.output_parent.resolve() if args.output_parent else Path(tempfile.gettempdir()) + parent.mkdir(parents=True, exist_ok=True) + self.output = Path(tempfile.mkdtemp(prefix='darwin-' + name + '-', dir=parent)) + self.work = self.output / 'work' + self.logs = self.output / 'logs' + self.work.mkdir(); self.logs.mkdir() + self.environment = {key: value for key, value in os.environ.items() + if not key.startswith(('NY_', 'NIL_', 'DYLD_'))} + self.deadline = time.monotonic() + seconds + self.ownership = support.load_ownership(ROOT) + self.expected_fixtures = fixtures + self.report = {'schema': 'nil.darwin-native-heap-budget.v1', 'status': 'FAIL', + 'metric': METRIC, 'budget_bytes': BUDGET, + 'scope': 'intercepted malloc-family requested payload retention; excludes tracker storage, ' + 'allocator overhead, VM mappings and allocation paths bypassing these functions', + 'whole_process_memory_parity': False, 'whole_seconds': seconds, + 'commands': [], 'runs': [], 'imports': [], 'fixtures': [], + 'inputs': {}} + inputs = [self.library, self.control, args.compiler.resolve(strict=True), Path(__file__), + ROOT / ('tests/tools/native_' + name.replace('-', '_') + '.py'), + ROOT / 'tests/darwin_native_heap_budget.c', + ROOT / 'tests/darwin_native_heap_budget.h', + ROOT / 'tests/darwin_native_heap_budget_control.c', + ROOT / 'validation/ny-3d/scripts/debugger_process.py'] + for path in inputs: + self.report['inputs'][str(path)] = support.file_record(path) + + def __enter__(self): + # Preparation is inside the lifetime script's with block so its failure + # also reaches __exit__ and retains the report. + return str(self.work) + + def __exit__(self, kind, error, traceback): + if error is not None: + self.report['error'] = str(error) + else: + try: + if len(self.report['fixtures']) != self.expected_fixtures: + raise AssertionError('not all requested native fixtures completed') + for path, record in self.report['inputs'].items(): + if support.file_record(Path(path)) != record: + raise AssertionError('contract input changed during execution: ' + path) + self.report['status'] = 'PASS' + except Exception as failure: + self.report['error'] = str(failure) + self.report['cleanup_verified'] = bool(self.report['commands']) and all( + row.get('cleanup_verified') is True for row in self.report['commands']) + if not self.report['cleanup_verified']: + self.report['status'] = 'FAIL' + path = self.output / 'report.json' + path.write_text(json.dumps(self.report, indent=2) + '\n', encoding='utf-8') + print(self.report['status'] + ': ' + str(path)) + if error is None and self.report['status'] != 'PASS': + raise AssertionError(self.report.get('error', 'owned cleanup failed')) + return False + + def command(self, name, command, *, stdout=b'', stderr=b'', exit_code=0, seconds=30): + remaining = self.deadline - time.monotonic() + if remaining <= 0: + raise AssertionError('whole Darwin heap contract deadline exhausted') + row = support.run_process(self.ownership, [str(part) for part in command], self.work, + self.environment, self.logs, name, min(seconds, remaining)) + row['expected_exit_code'] = exit_code + self.report['commands'].append(row) + actual = (self.logs / row['stdout']).read_bytes() + errors = (self.logs / row['stderr']).read_bytes() + if (row.get('exit_code') != exit_code or not row.get('cleanup_verified') or + row.get('error') or (stdout is not None and actual != stdout) or errors != stderr): + raise AssertionError(name + ': output/exit/deadline/owned-cleanup contract failed') + row['status'] = 'PASS' # Expected negative exits are evidence, not skipped tests. + return actual + + def inspect(self, name, executable, *, fixture=False): + imports = self.command(name + '-imports', ['/usr/bin/nm', '-u', executable], stdout=None) + self.command(name + '-dependencies', ['/usr/bin/otool', '-L', executable], stdout=None) + candidates = set(re.findall(rb'\b_((?:malloc|calloc|realloc|free|aligned_alloc|posix_memalign|valloc)[A-Za-z0-9_]*)\b', imports)) + candidates = {value.decode('ascii') for value in candidates} + # Queries do not allocate. Zone allocation, reallocf, and future typed + # entry points remain fail-closed until explicitly covered and tested. + candidates -= {'malloc_size', 'malloc_good_size', 'malloc_zone_from_ptr', 'malloc_default_zone'} + unsupported = candidates - ALLOCATORS + row = {'name': name, 'path': str(executable), 'allocator_imports': sorted(candidates), + 'unsupported_allocator_imports': sorted(unsupported), **support.file_record(executable)} + self.report['imports'].append(row) + if unsupported: + raise AssertionError(name + ': uncovered allocator imports: ' + ', '.join(sorted(unsupported))) + if fixture and (not candidates & {'calloc', 'malloc_type_calloc'} or + not candidates & {'free', 'malloc_type_free'}): + raise AssertionError(name + ': expected native runtime calloc/free imports absent') + + def injected(self, name, executable, arguments=(), *, target, withhold=False, + wanted=b'', negative=False): + events_path = self.logs / (name + '.heap.jsonl') + token = secrets.token_hex(16) + # The ownership helper starts a Python gate. Supplying DYLD_* in its + # environment would instrument that gate before exec. env assignments + # are parsed only after the gate execs this system utility. + command = ['/usr/bin/env', 'DYLD_INSERT_LIBRARIES=' + str(self.library), + 'NIL_DARWIN_HEAP_REPORT=' + str(events_path), + 'NIL_DARWIN_HEAP_TOKEN=' + token, + 'NIL_DARWIN_HEAP_TARGET=' + str(target), + 'NIL_DARWIN_HEAP_WITHHOLD=' + ('1' if withhold else '0'), + str(executable), *arguments] + self.command(name, command, stdout=wanted, exit_code=86 if negative else 0, seconds=45) + record = support.file_record(events_path, 8192) + events = [json.loads(line) for line in events_path.read_text(encoding='ascii').splitlines()] + if (len(events) != 2 or events[0].get('event') != 'ready' or + events[1].get('event') != ('budget' if negative else 'final')): + raise AssertionError(name + ': missing or malformed injection/final handshake') + for event in events: + if (event.get('token') != token or event.get('version') != 1 or + event.get('budget') != BUDGET or event.get('target_size') != target or + not isinstance(event.get('pid'), int) or event['pid'] <= 0 or + event['pid'] == os.getpid() or not 0 <= event.get('live', -1) <= event.get('peak', -1) <= BUDGET): + raise AssertionError(name + ': wrong process/configuration/accounting handshake') + initial, final = events + if initial['pid'] != final['pid']: + raise AssertionError(name + ': report crossed process identity') + for key in ('allocations', 'frees', 'cumulative', 'target_allocations', 'target_frees', 'typed_calls'): + if final[key] < initial[key]: + raise AssertionError(name + ': allocator counters moved backwards') + if negative: + if (not 0 < final['denied_request'] <= target or + final['live'] < BUDGET - 2 * target or + final['target_allocations'] * target < BUDGET * .8 or + (withhold and final['withheld'] == 0)): + raise AssertionError(name + ': retention did not cause the bounded allocation rejection') + elif final['withheld'] or final['denied_request']: + raise AssertionError(name + ': ordinary fixture used negative-control behavior') + self.report['runs'].append({'name': name, 'expected_budget_rejection': negative, + 'withhold_target_frees': withhold, 'events': events, + 'report_file': events_path.name, **record}) + return final + + def prepare(self): + self.command('host-version', ['/usr/bin/sw_vers'], stdout=None) + self.command('host-machine', ['/usr/bin/uname', '-m'], stdout=None) + self.command('clang-version', ['/usr/bin/xcrun', 'clang', '--version'], stdout=None) + self.command('sdk-version', ['/usr/bin/xcrun', '--show-sdk-version'], stdout=None) + self.inspect('interposer', self.library) + self.inspect('control', self.control) + self.command('injection-absent', [self.control, 'positive'], exit_code=90, + stderr=b'heap budget injection absent\n') + positive = self.injected('control-positive', self.control, ['positive'], target=65536, + wanted=b'heap-controls-ok\n') + if (positive['cumulative'] <= 2 * 1024**3 or positive['target_allocations'] < 34000 or + positive['target_frees'] < 34000 or positive['failures'] < 2): + raise AssertionError('positive/realloc/failure control coverage insufficient') + self.injected('control-retained-untouched', self.control, ['retain'], target=65536, negative=True) + self.injected('control-withheld-free', self.control, ['positive'], target=65536, + withhold=True, negative=True) + + def compile(self, name, command): + self.command(name, command, seconds=90) + + def fixture(self, name, source, executable, expected, *, target, count): + source_record = support.file_record(source) + executable_record = support.file_record(executable) + self.inspect(name, executable, fixture=True) + final = self.injected(name, executable, target=target, wanted=expected.encode('utf-8')) + if (final['target_allocations'] < count or final['target_frees'] < count - 1 or + final['cumulative'] < target * count): + raise AssertionError(name + ': expected unchanged-loop payload coverage absent') + # The exact same native executable must fail when actual target frees + # are withheld. No Ny source mutation or runtime GC counters are used. + self.injected(name + '-withheld-free', executable, target=target, + withhold=True, negative=True) + if (support.file_record(source) != source_record or + support.file_record(executable) != executable_record): + raise AssertionError(name + ': source or executable changed between controls') + self.report['fixtures'].append({'name': name, 'source': str(source), + 'source_record': source_record, 'executable_record': executable_record, + 'expected_target_allocations': count, 'target_bytes': target, + 'positive_and_withheld_free_replay': 'PASS'}) diff --git a/tests/tools/native_managed_lifetime.py b/tests/tools/native_managed_lifetime.py index a7892ff9..27882bb7 100644 --- a/tests/tools/native_managed_lifetime.py +++ b/tests/tools/native_managed_lifetime.py @@ -11,6 +11,7 @@ import tempfile from native_memory_support import memory_metric, run_memory_bounded, verify_memory_limit +from darwin_native_heap_budget import DarwinHeapBudget, METRIC, add_arguments def main() -> int: @@ -19,11 +20,15 @@ def main() -> int: parser.add_argument("--stdlib", type=Path, required=True) parser.add_argument("--llvm", choices=("ON", "OFF"), required=True) parser.add_argument("--memory-launcher", type=Path) + add_arguments(parser) args = parser.parse_args() - if sys.platform not in ("linux", "win32") or args.llvm == "OFF": + if sys.platform not in ("linux", "win32", "darwin") or args.llvm == "OFF": return 77 - with tempfile.TemporaryDirectory(prefix="ny-managed-lifetime café ") as temporary: + darwin = (DarwinHeapBudget(args, 'managed-lifetime', seconds=210, fixtures=1) + if sys.platform == 'darwin' else None) + context = darwin if darwin else tempfile.TemporaryDirectory(prefix="ny-managed-lifetime café ") + with context as temporary: root = Path(temporary) source = root / "sustained.ny" output = root / ("sustained.exe" if os.name == "nt" else "sustained") @@ -41,20 +46,28 @@ def main() -> int: end fn ''', encoding="utf-8", newline='\n') environment = {key: value for key, value in os.environ.items() if not key.startswith("NY_")} + if darwin: + darwin.prepare() verify_memory_limit(args.memory_launcher, cwd=root, environment=environment) - compiled = subprocess.run([str(args.compiler.resolve()), "--quiet", "--no-stdlib", - "--sdk-path", str(args.stdlib.resolve()), "--native", "--output", str(output), str(source)], - cwd=root, env=environment, text=True, capture_output=True, timeout=90) - if compiled.returncode != 0 or compiled.stdout != "" or compiled.stderr != "": - raise AssertionError(compiled) + command = [str(args.compiler.resolve()), "--quiet", "--no-stdlib", + "--sdk-path", str(args.stdlib.resolve()), "--native", "--output", str(output), str(source)] + if darwin: + darwin.compile('compile-managed', command) + else: + compiled = subprocess.run(command, cwd=root, env=environment, text=True, capture_output=True, timeout=90) + if compiled.returncode != 0 or compiled.stdout != "" or compiled.stderr != "": + raise AssertionError(compiled) # Apply to the compiled consumer only, never compiler/JIT host. # The unchanged loop allocates >625 MiB cumulatively. - result = run_memory_bounded(output, cwd=root, environment=environment, - launcher=args.memory_launcher) - if result.returncode != 0 or result.stdout != "799980000\n" or result.stderr != "": - raise AssertionError(result) - print("native managed lifetime: sustained allocation fits " + memory_metric()) + if darwin: + darwin.fixture('managed', source, output, "799980000\n", target=16384, count=40000) + else: + result = run_memory_bounded(output, cwd=root, environment=environment, + launcher=args.memory_launcher) + if result.returncode != 0 or result.stdout != "799980000\n" or result.stderr != "": + raise AssertionError(result) + print("native managed lifetime: sustained allocation fits " + (METRIC if darwin else memory_metric())) return 0 diff --git a/tests/tools/native_owned_lifetime.py b/tests/tools/native_owned_lifetime.py index 2072be4d..5bd5dd54 100644 --- a/tests/tools/native_owned_lifetime.py +++ b/tests/tools/native_owned_lifetime.py @@ -8,6 +8,7 @@ import sys import tempfile from native_memory_support import memory_metric, run_memory_bounded, verify_memory_limit +from darwin_native_heap_budget import DarwinHeapBudget, METRIC, add_arguments def main() -> int: @@ -15,8 +16,9 @@ def main() -> int: parser.add_argument("--compiler", type=Path, required=True) parser.add_argument("--llvm", choices=("ON", "OFF"), required=True) parser.add_argument("--memory-launcher", type=Path) + add_arguments(parser) args = parser.parse_args() - if sys.platform not in ("linux", "win32") or args.llvm == "OFF": + if sys.platform not in ("linux", "win32", "darwin") or args.llvm == "OFF": return 77 source_text = """struct LifetimeProblem code:i32 @@ -55,24 +57,34 @@ def main() -> int: end fn """ environment = {key: value for key, value in os.environ.items() if not key.startswith("NY_")} - with tempfile.TemporaryDirectory(prefix="ny-owned-lifetime café ") as temporary: + darwin = (DarwinHeapBudget(args, 'owned-lifetime', seconds=330, fixtures=2) + if sys.platform == 'darwin' else None) + context = darwin if darwin else tempfile.TemporaryDirectory(prefix="ny-owned-lifetime café ") + with context as temporary: root = Path(temporary) + if darwin: + darwin.prepare() verify_memory_limit(args.memory_launcher, cwd=root, environment=environment) source = root / "sustained.ny" source.write_text(source_text, encoding="utf-8", newline='\n') for name, flags in (("managed", []), ("unmanaged", ["--unmanaged"])): output = root / (name + (".exe" if os.name == "nt" else "")) - compiled = subprocess.run([str(args.compiler.resolve()), "--no-stdlib", "--quiet", - "--native", *flags, "--output", str(output), str(source)], cwd=root, - env=environment, stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=90) - if compiled.returncode != 0 or compiled.stderr: - raise AssertionError(f"{name} compile: {compiled}") - ran = run_memory_bounded(output, cwd=root, environment=environment, - launcher=args.memory_launcher) - if ran.returncode != 0 or ran.stderr or ran.stdout != "35994000\n": - raise AssertionError(f"{name} lifetime: {ran}") + command = [str(args.compiler.resolve()), "--no-stdlib", "--quiet", + "--native", *flags, "--output", str(output), str(source)] + if darwin: + darwin.compile('compile-' + name, command) + darwin.fixture(name, source, output, "35994000\n", target=65536, count=36000) + else: + compiled = subprocess.run(command, cwd=root, + env=environment, stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=90) + if compiled.returncode != 0 or compiled.stderr: + raise AssertionError(f"{name} compile: {compiled}") + ran = run_memory_bounded(output, cwd=root, environment=environment, + launcher=args.memory_launcher) + if ran.returncode != 0 or ran.stderr or ran.stdout != "35994000\n": + raise AssertionError(f"{name} lifetime: {ran}") print("Owned normal/return/continue/fault/recover scopes: >2 GiB cumulative storage fits " + - memory_metric() + " in both memory modes") + (METRIC if darwin else memory_metric()) + " in both memory modes") return 0 From 2632bc655127150f3aadc198ac2519ec595d43bb Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 18:16:57 -0500 Subject: [PATCH 08/39] Implement Darwin compiler content identity and qualify application tooling --- .../workflows/macos-application-parity.yml | 26 +- CMakeLists.txt | 16 + README.md | 8 +- common/content_digest.hpp | 46 ++- docs/platform-support.md | 14 +- docs/standard-library/ui.md | 10 +- nilbuild/digest_phase.hpp | 52 ++++ nilbuild/main.cpp | 74 ++++- nilcompiler/CMakeLists.txt | 5 + .../src/compiler_module_fingerprint.hpp | 4 + .../compiler_module_fingerprint_darwin.cpp | 282 ++++++++++++++++++ .../compiler_module_fingerprint_darwin.hpp | 241 +++++++++++++++ nilcompiler/src/main.cpp | 5 +- notebook/2026-11-11-web-runtime-parity.md | 36 +++ scripts/ci/macos-toolchain-census.py | 80 +++++ tests/compiler_module_darwin_tests.cpp | 186 ++++++++++++ tests/darwin_native_heap_budget.c | 36 ++- tests/tools/analysis_dependencies.py | 8 +- tests/tools/incremental_build_contract.py | 59 +++- 19 files changed, 1130 insertions(+), 58 deletions(-) create mode 100644 nilbuild/digest_phase.hpp create mode 100644 nilcompiler/src/compiler_module_fingerprint_darwin.cpp create mode 100644 nilcompiler/src/compiler_module_fingerprint_darwin.hpp create mode 100644 scripts/ci/macos-toolchain-census.py create mode 100644 tests/compiler_module_darwin_tests.cpp diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index 0d9ea3f7..a2d564a7 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -16,6 +16,7 @@ name: macOS application platform checks - 'common/**' - 'nillinker/process_runner*' - 'validation/ny-3d/scripts/debugger_process.py' + - 'scripts/ci/macos-toolchain-census.py' - 'tests/darwin_native_heap_budget*' - 'tests/tools/darwin_native_heap_budget.py' - 'tests/tools/native_managed_lifetime.py' @@ -89,7 +90,7 @@ jobs: set -o pipefail cmake --build build/application --parallel 2 --target \ nilcompiler nilinterpreter nilassembler nillsp nildebugger nilbuild niltest aperture \ - nil_vscode_host_runner nil_file_time_support_tests \ + nil_vscode_host_runner nil_file_time_support_tests nil_compiler_module_darwin_tests \ 2>&1 | tee build/application-evidence/build.log - name: Build independent allocation controls id: heap @@ -99,15 +100,21 @@ jobs: cmake --build build/application --parallel 2 --target \ nil_darwin_native_heap_budget nil_darwin_native_heap_budget_control \ 2>&1 | tee build/application-evidence/heap-build.log + - name: Record selected native SDK and linker inputs + if: ${{ !cancelled() && steps.tools.outcome == 'success' }} + timeout-minutes: 2 + run: >- + python3 -B scripts/ci/macos-toolchain-census.py --build build/application + --output build/application-evidence/toolchain-census - name: Verify actual native lifetime and timestamp contracts timeout-minutes: 15 run: | python3 - <<'PYTHON' import json, subprocess, xml.etree.ElementTree as ET from pathlib import Path - expected = {'ny.native-managed-lifetime', 'ny.native-owned-lifetime', + expected = {'compiler.darwin-module-fingerprint', 'ny.native-managed-lifetime', 'ny.native-owned-lifetime', 'nil.file-time-support', 'nil.file-modified-at-contract'} - pattern = r'^(ny[.]native-(managed|owned)-lifetime|nil[.]file-(time-support|modified-at-contract))$' + pattern = r'^(compiler[.]darwin-module-fingerprint|ny[.]native-(managed|owned)-lifetime|nil[.]file-(time-support|modified-at-contract))$' base = ['ctest', '--test-dir', 'build/application', '-R', pattern] root = Path('build/application-evidence').resolve() inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) @@ -165,7 +172,18 @@ jobs: NIL_VSCODE_HOST_TIMEOUT_MS: '900000' run: | set -o pipefail - export VSCODE_EXECUTABLE="$GITHUB_WORKSPACE/build/editor-download/Visual Studio Code.app/Contents/MacOS/Electron" + VSCODE_EXECUTABLE="$(python3 - <<'PYTHON' + import plistlib + from pathlib import Path + app = Path('build/editor-download/Visual Studio Code.app/Contents').resolve(strict=True) + with (app/'Info.plist').open('rb') as stream: name = plistlib.load(stream)['CFBundleExecutable'] + assert name and Path(name).name == name + executable = app/'MacOS'/name + assert executable.is_file() + print(executable) + PYTHON + )" + export VSCODE_EXECUTABLE export NIL_VSCODE_TOOL_DIRECTORY="$GITHUB_WORKSPACE/build/application/bin" export NIL_VSCODE_HOST_RUNNER="$NIL_VSCODE_TOOL_DIRECTORY/nil_vscode_host_runner" export NIL_VSCODE_SDK_PATH="$GITHUB_WORKSPACE" diff --git a/CMakeLists.txt b/CMakeLists.txt index 43c3485e..8fd4785c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -117,6 +117,17 @@ if(BUILD_TESTING) endif() add_test(NAME nilbuild.compiler-module-fingerprints COMMAND $) set_tests_properties(nilbuild.compiler-module-fingerprints PROPERTIES LABELS "tools;project;rc" TIMEOUT 30) + add_executable(nil_compiler_module_darwin_tests tests/compiler_module_darwin_tests.cpp) + target_compile_features(nil_compiler_module_darwin_tests PRIVATE cxx_std_17) + if(APPLE) + target_sources(nil_compiler_module_darwin_tests PRIVATE nilcompiler/src/compiler_module_fingerprint_darwin.cpp) + target_compile_options(nil_compiler_module_darwin_tests PRIVATE -fblocks) + endif() + add_test(NAME compiler.darwin-module-fingerprint COMMAND $) + set_tests_properties(compiler.darwin-module-fingerprint PROPERTIES LABELS "tools;project;rc" TIMEOUT 60) + if(WIN32) + target_link_libraries(nil_compiler_module_darwin_tests PRIVATE advapi32) + endif() # Test-only launcher for isolated editor integration runs. Reuse the # toolchain's bounded process/Job cleanup; never ship it in the SDK. @@ -832,6 +843,11 @@ if(BUILD_TESTING) --cxx "${CMAKE_C_COMPILER}" --llvm "${NIL_ENABLE_LLVM}") set_tests_properties(nilbuild.incremental-contract PROPERTIES LABELS "tools;project;rc" TIMEOUT 240) + if(APPLE) + # Three real builds read complete dyld cache contents. The measured + # first Intel hash took 104 s; fake/mutation cases keep their limits. + set_tests_properties(nilbuild.incremental-contract PROPERTIES TIMEOUT 900) + endif() add_test(NAME nilbuild.compiler-identity-contract COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/incremental_build_contract.py" --builder $ --compiler $ diff --git a/README.md b/README.md index fc81e8f8..20cb263a 100644 --- a/README.md +++ b/README.md @@ -255,12 +255,12 @@ assembly and executable-context embedding, inspection and dead-strip retention checks. Both macOS development SDKs also pass the full installed standard-library campaign, isolated PostgreSQL application routes, installed debugger checks and execution checkpoints. Both hosts now pass the underlying -AppKit provider and interpreter language-level desktop checks. Windows uses its +AppKit provider and interpreter/native language-level desktop checks. Windows uses its native Win32 provider; Linux uses GTK4. macOS editor integration and a complete current SDK desktop campaign remain open. The newer socket, entropy and text-byte -intrinsics are interpreter-backed; their cross-platform acceptance is being -brought up to date. File timestamps now share an implementation across interpreter -and native routes, with Linux text/binary NIL, JIT and AOT checks passing. +intrinsics pass the same interpreter contracts on Linux, Windows and both Macs. +File timestamps share an implementation across interpreter and native routes, +with Linux and Windows text/binary NIL, JIT and AOT checks passing. Incremental reuse and native memory-limit tests also remain open, so full platform parity is not yet established. The [platform support matrix](docs/platform-support.md) separates verified diff --git a/common/content_digest.hpp b/common/content_digest.hpp index 7bcb6a5b..03cc87ec 100644 --- a/common/content_digest.hpp +++ b/common/content_digest.hpp @@ -2,6 +2,7 @@ // SHA-256 content identities for build inputs. Timestamps are not cache keys. #include +#include #include #include #include @@ -9,9 +10,13 @@ #include #include #include +#include +#if defined(__APPLE__) +#include +#endif namespace nil::detail { -class sha256 { +class portable_sha256 { std::array state{0x6a09e667,0xbb67ae85,0x3c6ef372,0xa54ff53a, 0x510e527f,0x9b05688c,0x1f83d9ab,0x5be0cd19}; std::array block{}; @@ -58,14 +63,49 @@ class sha256 { return out.str(); } }; +#if defined(__APPLE__) +// The identity is still ordinary SHA-256 over every byte. Apple's implementation +// avoids spending minutes in the scalar fallback on the dyld backing files. +class sha256 { + CC_SHA256_CTX state{}; +public: + sha256() { + if (CC_SHA256_Init(&state) != 1) throw std::runtime_error("cannot initialize SHA-256"); + } + void update(const char *data, std::size_t length) { + while (length) { + const auto count = static_cast(std::min(length, std::numeric_limits::max())); + if (CC_SHA256_Update(&state, data, count) != 1) throw std::runtime_error("cannot update SHA-256"); + data += count; + length -= count; + } + } + std::string finish() { + unsigned char bytes[CC_SHA256_DIGEST_LENGTH]; + if (CC_SHA256_Final(bytes, &state) != 1) throw std::runtime_error("cannot finish SHA-256"); + constexpr char digits[] = "0123456789abcdef"; + std::string result; + result.reserve(sizeof(bytes) * 2); + for (auto byte : bytes) { result += digits[byte >> 4]; result += digits[byte & 15]; } + return result; + } +}; +#else +using sha256 = portable_sha256; +#endif inline std::string content_digest(const std::string &text) { sha256 hash; hash.update(text.data(),text.size());return hash.finish(); } -inline std::string file_digest(const std::filesystem::path &path) { +template inline std::string file_digest(const std::filesystem::path &path, Check check) { + check(); if(!std::filesystem::is_regular_file(path)) throw std::runtime_error("expected a regular build input: "+path.string()); std::ifstream input(path,std::ios::binary); if(!input) throw std::runtime_error("cannot read build input: "+path.string()); sha256 hash; char buffer[65536]; - while(input.read(buffer,sizeof(buffer))||input.gcount()) hash.update(buffer,static_cast(input.gcount())); + while(input.read(buffer,sizeof(buffer))||input.gcount()) { + check(); + hash.update(buffer,static_cast(input.gcount())); + } if(input.bad()) throw std::runtime_error("cannot read build input: "+path.string()); return hash.finish(); } +inline std::string file_digest(const std::filesystem::path &path) { return file_digest(path, [] {}); } } // namespace nil::detail diff --git a/docs/platform-support.md b/docs/platform-support.md index ea11caa6..49bb244b 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -183,9 +183,11 @@ the complete admission rules. campaign does not qualify cross-process keyboard injection or accessibility; the language interaction suite currently runs on Linux and Windows. Earlier private GTK and AppKit prerequisite probes are retained as historical evidence, - but GTK is no longer the macOS product provider. The language-level AppKit - display campaign uses LLVM OFF and therefore establishes the interpreter route, - not native/JIT application or editor acceptance. + but GTK is no longer the macOS product provider. The LLVM-ON campaign at + `965f159` passes all five selected CTests on both hosts, including all three + Matrix desktop cases on each interpreter/native route. The retained full + language reports verify all six cases; this is separate from editor acceptance + and the cross-process keyboard/accessibility exclusion. - Editor desktop integration, untested provider configurations and release-package acceptance remain separate from the passing installed campaigns. Shipping a Mach-O tool alone does not qualify these surfaces. @@ -207,7 +209,9 @@ declaring current-source parity. The private Windows LLVM-ON build at `965f159` passes all six focused web-runtime contracts with no skips, including VM socket cleanup and the native timestamp -routes. The corresponding Linux contracts pass. Hosted Intel/ARM acceptance and -current macOS editor coverage are still in progress. The socket/byte/entropy +routes. At `19bd5f1`, the hosted campaign passes the same six interpreter contracts +on all four hosts, with exact inventories, no skips and independent raw-result +replay. The tested PR merge and head revisions are retained separately. Current +macOS editor coverage is still in progress. The socket/byte/entropy native limitation is common to Linux, Windows and macOS; those APIs currently require the interpreter on all three hosts. diff --git a/docs/standard-library/ui.md b/docs/standard-library/ui.md index d835a284..9329a530 100644 --- a/docs/standard-library/ui.md +++ b/docs/standard-library/ui.md @@ -5,9 +5,9 @@ events and explicit model bindings. Ny programs can import the same module. The VM and native compiler use one desktop provider per platform, each behind the same checked boundary: a GTK4 provider on Linux and a Win32 provider on Windows. Windows remains a development target, not a qualified release platform. -The macOS provider is AppKit. Its provider, lifecycle and interpreter -language-level desktop contracts pass on Intel and Apple Silicon; current SDK, -native/JIT application and editor acceptance remain separate. AppKit replaces the +The macOS provider is AppKit. Its provider, lifecycle and interpreter/native +language-level desktop contracts pass on Intel and Apple Silicon; current SDK +and editor acceptance remain separate. AppKit replaces the earlier private GTK-on-macOS experiments. See [platform support](../platform-support.md). Examples display the framework generation as **NIL²**, with **NIL^2** as the plain-text spelling. This presentation does not change numeric release or @@ -71,8 +71,8 @@ possible GTK configuration. AppKit is the supported source implementation. On macOS, creation and polling must run on the process main thread, which is where AppKit requires its windows to be used. Ordinary interpreter, JIT and -native entry routes use the main thread. The interpreter display campaign passes; -native/JIT application acceptance still needs its own campaign. DAP currently executes the +native entry routes use the main thread. The LLVM-ON display campaign verifies +the interpreter and native application cases on both architectures. DAP currently executes the debuggee on a worker, so DAP with macOS UI is outside this port, and pausing a text or JSON debug session does not automatically pump window events. AppKit supplies no box or grid container, so, as on Win32, the provider owns the diff --git a/nilbuild/digest_phase.hpp b/nilbuild/digest_phase.hpp new file mode 100644 index 00000000..e2e41fa1 --- /dev/null +++ b/nilbuild/digest_phase.hpp @@ -0,0 +1,52 @@ +#pragma once + +#include "../common/content_digest.hpp" +#include +#include +#include + +namespace nilbuild { +// A single preparation OR verification phase owns this map. Never retain it +// across compilation, analysis refresh, dependency layers, or build commands. +// Only shared compiler/toolchain inputs use it; source dependency checks stay +// independent. Selected path spellings remain identities, including symlinks. +class digest_phase { + std::map values; + std::set excluded; + bool excluded_path(const std::filesystem::path &path) const { + const auto spelling = std::filesystem::absolute(path).lexically_normal().string(); + if (excluded.count(spelling)) return true; + std::error_code error; + const auto canonical = std::filesystem::canonical(path, error); + if (error || excluded.count(canonical.string())) return true; + // Canonicalization resolves symlinks, not hard links. Most shared inputs + // have one link, so avoid a per-source identity scan in that common case. + const auto links = std::filesystem::hard_link_count(path, error); + if (error) return true; + if (links > 1) for (const auto &source : excluded) { + const bool same = std::filesystem::equivalent(path, source, error); + if (same) return true; + // Missing/changed sources are handled by the independent source check; + // don't memoize while their physical identity cannot be compared. + if (error) return true; + } + return false; + } +public: + void exclude(const std::filesystem::path &path) { + excluded.insert(std::filesystem::absolute(path).lexically_normal().string()); + std::error_code error; + const auto canonical = std::filesystem::canonical(path, error); + if (!error) excluded.insert(canonical.string()); + } + void inherit_exclusions(const digest_phase &preparation) { excluded = preparation.excluded; } + template std::string read(const std::filesystem::path &path, Read digest) { + if (excluded_path(path)) return digest(path); + const auto key = std::filesystem::absolute(path).lexically_normal().string(); + const auto found = values.find(key); + if (found != values.end()) return found->second; + return values.emplace(key, digest(path)).first->second; + } + bool contains(const std::string &path) const { return !excluded_path(path) && values.count(path) != 0; } +}; +} // namespace nilbuild diff --git a/nilbuild/main.cpp b/nilbuild/main.cpp index 36ebed6b..6d4047df 100644 --- a/nilbuild/main.cpp +++ b/nilbuild/main.cpp @@ -5,6 +5,7 @@ #include "../common/tool_discovery.hpp" #include "process.hpp" #include "toolchain_inputs.hpp" +#include "digest_phase.hpp" #include "nil_version.hpp" #include @@ -658,14 +659,19 @@ void check_cancelled() { if (nilbuild::pending_signal) throw build_cancelled{128 + nilbuild::pending_signal}; } using input_map = std::map; -void add_input(input_map &inputs, const fs::path &path) { - inputs[fs::absolute(path).lexically_normal().string()] = nil::detail::file_digest(path); +std::string checked_digest(const fs::path &path) { + return nil::detail::file_digest(path, check_cancelled); +} +void add_input(input_map &inputs, const fs::path &path, nilbuild::digest_phase *phase = nullptr) { + inputs[fs::absolute(path).lexically_normal().string()] = phase ? phase->read(path, checked_digest) : checked_digest(path); check_cancelled(); } -void verify_inputs(const input_map &inputs) { +void verify_inputs(const input_map &inputs, nilbuild::digest_phase *phase = nullptr, + const nilbuild::digest_phase *shared = nullptr) { for (const auto &[path, digest] : inputs) { check_cancelled(); - if (nil::detail::file_digest(path) != digest) + const auto current = phase && shared && shared->contains(path) ? phase->read(path, checked_digest) : checked_digest(path); + if (current != digest) throw std::runtime_error("build input changed during the build; retry: " + path); } } @@ -694,10 +700,11 @@ bool compiler_files_complete(const json &report) { // The compiler tells us which runtime and linker it would select. Query that // driver for startup objects, libraries and its linker rather than guessing a // host installation layout. Missing identities disable reuse, not compilation. -bool native_inputs(input_map &inputs, const json &report, const project &value) { +bool native_inputs(input_map &inputs, const json &report, const project &value, + nilbuild::digest_phase *phase = nullptr) { const auto runtime = report.object.find("nativeRuntime"), linker = report.object.find("nativeLinker"); if (runtime == report.object.end() || runtime->second.type != json::kind::string) return false; - add_input(inputs, runtime->second.text); + add_input(inputs, runtime->second.text, phase); if (linker == report.object.end() || linker->second.type != json::kind::string) return false; std::vector library_paths; const auto reported_paths = report.object.find("nativeLibraryPaths"); @@ -711,7 +718,7 @@ bool native_inputs(input_map &inputs, const json &report, const project &value) const auto files = nilbuild::native_toolchain_files(linker->second.text, value.directory, runtime->second.text, library_paths); if (!files) return false; - for (const auto &file : *files) add_input(inputs, file); + for (const auto &file : *files) add_input(inputs, file, phase); return true; } struct build_task { @@ -729,6 +736,13 @@ std::string build_key(const project &value, const build_task &task, const std::v auto description = describe(value); description.object["schema"] = json::str("nil.build.inputs.v1"); description.object["target"] = json::str(task.native ? "native" : "vm"); +#if defined(__APPLE__) +#if defined(__aarch64__) || defined(__arm64__) + description.object["hostArchitecture"] = json::str("arm64"); +#else + description.object["hostArchitecture"] = json::str("x86_64"); +#endif +#endif description.object["inputs"] = json::map(); for (const auto &[path, digest] : *task.inputs) description.object["inputs"].object[path] = json::str(digest); auto environment = json::map(); @@ -737,6 +751,12 @@ std::string build_key(const project &value, const build_task &task, const std::v environment.object[name] = value ? json::str(value) : json{}; } description.object["environment"] = std::move(environment); +#if defined(__APPLE__) + for (const auto *name : {"DEVELOPER_DIR", "TOOLCHAINS"}) { + const char *value = std::getenv(name); + description.object["environment"].object[name] = value ? json::str(value) : json{}; + } +#endif if (task.native && report.object.count("nativeLibraryPaths")) description.object["nativeLibraryPaths"] = report.at("nativeLibraryPaths"); description.object["references"] = json::list(); @@ -798,6 +818,27 @@ int build(const project &root, const options &opts, const fs::path &compiler, co if (!status) status = analyzed; } if (status) return status; + // Darwin's shared cache is several GiB. Hash its complete contents once + // for this preparation phase, shared across ready projects and targets. + // Other hosts retain their existing read behavior. Sources/manifests never + // use this memo, and verification below always starts with an empty one. + nilbuild::digest_phase prepared_shared; + nilbuild::digest_phase *preparation = nullptr; +#if defined(__APPLE__) + // Exclude every source/manifest before any shared input is memoized. A + // compiler report can legitimately overlap inventories; even a symlinked + // source advertised as a module must keep independent fresh checks. + for (const auto &value : graph.projects) { + prepared_shared.exclude(value.file); prepared_shared.exclude(value.entry); + } + for (const auto &report : reports) { + const auto dependencies = report.object.find("dependencies"); + if (dependencies == report.object.end() || dependencies->second.type != json::kind::array) continue; + for (const auto &dependency : dependencies->second.array) + prepared_shared.exclude(string_value(dependency.at("path"), "dependency path")); + } + preparation = &prepared_shared; +#endif std::vector tasks; std::vector commands; for (std::size_t i = 0; i < ready.size(); ++i) { @@ -823,14 +864,15 @@ int build(const project &root, const options &opts, const fs::path &compiler, co for (const bool native : {false, true}) { if ((native && opts.target == "vm") || (!native && opts.target == "native")) continue; build_task task{index, native, native ? value.native_output() : value.vm_output(), sources.value_or(input_map{}), {}, {}, - sources.has_value() && compiler_files_complete(reports[i])}; + sources.has_value() && compiler_files_complete(reports[i]), {}}; if (task.inputs) { - add_input(*task.inputs, value.file); add_input(*task.inputs, value.entry); add_input(*task.inputs, compiler); add_input(*task.inputs, self); + add_input(*task.inputs, value.file); add_input(*task.inputs, value.entry); + add_input(*task.inputs, compiler, preparation); add_input(*task.inputs, self, preparation); if (reports[i].object.count("compilerFiles")) { - for (const auto &file : string_list(reports[i].at("compilerFiles"), "compilerFiles")) add_input(*task.inputs, file); + for (const auto &file : string_list(reports[i].at("compilerFiles"), "compilerFiles")) add_input(*task.inputs, file, preparation); } if (native) { - if (!native_inputs(task.native_files, reports[i], value)) task.cacheable = false; + if (!native_inputs(task.native_files, reports[i], value, preparation)) task.cacheable = false; task.inputs->insert(task.native_files.begin(), task.native_files.end()); } } @@ -864,6 +906,12 @@ int build(const project &root, const options &opts, const fs::path &compiler, co const auto compiled = nilbuild::execute_many(commands, opts.jobs); for (const auto &result : compiled) std::cerr << result.output << result.error; if (const auto failed = failed_batch(compiled)) return failed; + nilbuild::digest_phase verified_shared; + nilbuild::digest_phase *verification = nullptr; +#if defined(__APPLE__) + verified_shared.inherit_exclusions(prepared_shared); + verification = &verified_shared; +#endif // Re-resolve the graph after a compilation before publishing cacheable // artifacts. This catches new shadowing files as well as modified inputs. for (std::size_t i = 0; i < ready.size(); ++i) { @@ -885,7 +933,7 @@ int build(const project &root, const options &opts, const fs::path &compiler, co compiler_module_difference(reports[i], current)); for (const auto &task : tasks) if (task.project_index == index && task.native && task.cacheable) { input_map current_native; - if (!native_inputs(current_native, current, value) || current_native != task.native_files || + if (!native_inputs(current_native, current, value, verification) || current_native != task.native_files || (current.object.count("nativeLibraryPaths") && (!reports[i].object.count("nativeLibraryPaths") || nilbuild::dump(current.at("nativeLibraryPaths")) != @@ -897,7 +945,7 @@ int build(const project &root, const options &opts, const fs::path &compiler, co const auto temporary = task.staging->path / task.output.filename(); if (!fs::is_regular_file(fs::symlink_status(temporary))) throw std::runtime_error("compiler succeeded without creating a regular output artifact"); if (task.native && !executable(temporary)) throw std::runtime_error("compiler native artifact is not executable"); - if (task.inputs) verify_inputs(*task.inputs); + if (task.inputs) verify_inputs(*task.inputs, verification, preparation); protect_inputs(graph.projects[task.project_index]); } check_cancelled(); diff --git a/nilcompiler/CMakeLists.txt b/nilcompiler/CMakeLists.txt index 85605901..b29a9cda 100644 --- a/nilcompiler/CMakeLists.txt +++ b/nilcompiler/CMakeLists.txt @@ -210,6 +210,11 @@ if(NILCOMPILER_POSTGRESQL_INCLUDE_DIR) endif() add_executable(nilcompiler src/main.cpp) +if(APPLE) + target_sources(nilcompiler PRIVATE src/compiler_module_fingerprint_darwin.cpp) + set_source_files_properties(src/compiler_module_fingerprint_darwin.cpp PROPERTIES + COMPILE_OPTIONS "-fblocks") +endif() if(WIN32) target_sources(nilcompiler PRIVATE ../nillinker/process_runner.cpp) target_link_libraries(nilcompiler PRIVATE advapi32) diff --git a/nilcompiler/src/compiler_module_fingerprint.hpp b/nilcompiler/src/compiler_module_fingerprint.hpp index d951ce0f..44113c60 100644 --- a/nilcompiler/src/compiler_module_fingerprint.hpp +++ b/nilcompiler/src/compiler_module_fingerprint.hpp @@ -20,6 +20,10 @@ namespace nil::detail::compiler_modules { namespace fs = std::filesystem; using files = std::set; +#if defined(__APPLE__) +// Defined in the Darwin-only translation unit (-fblocks). +bool collect_darwin(files &observed); +#endif // Linux file-backed mappings are a separate identity graph from source inputs. // Keep valid observations on failure; callers may still protect/hash them, but diff --git a/nilcompiler/src/compiler_module_fingerprint_darwin.cpp b/nilcompiler/src/compiler_module_fingerprint_darwin.cpp new file mode 100644 index 00000000..c8d6fbe3 --- /dev/null +++ b/nilcompiler/src/compiler_module_fingerprint_darwin.cpp @@ -0,0 +1,282 @@ +#include "compiler_module_fingerprint.hpp" +#include "compiler_module_fingerprint_darwin.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace nil::detail::compiler_modules { +namespace { +using namespace darwin; +constexpr std::size_t maximum_commands = 65536; +bool read_self(std::uint64_t address, void *destination, std::size_t size) { + if (!address || size > maximum_commands || address > UINT64_MAX - size) return false; + mach_vm_size_t copied = 0; + return mach_vm_read_overwrite(mach_task_self(), address, size, + reinterpret_cast(destination), &copied) == KERN_SUCCESS && copied == size; +} +loaded_image inspect(const mach_header *pointer, std::int64_t slide, bool derive_slide = false) { + loaded_image row; + row.address = reinterpret_cast(pointer); + row.slide = slide; + mach_header_64 header{}; + require(read_self(row.address, &header, sizeof(header)) && header.magic == MH_MAGIC_64 && + header.ncmds && header.ncmds <= 1024 && header.sizeofcmds <= maximum_commands && + header.ncmds <= header.sizeofcmds / sizeof(load_command)); + row.cpu = static_cast(header.cputype); row.flags = header.flags; + std::array commands{}; + require(read_self(end(row.address, sizeof(header)), commands.data(), header.sizeofcmds)); + std::size_t offset = 0; + bool text = false, identifier = false; + for (std::uint32_t i = 0; i < header.ncmds; ++i) { + require(offset <= header.sizeofcmds && sizeof(load_command) <= header.sizeofcmds - offset); + load_command command{}; + std::memcpy(&command, commands.data() + offset, sizeof(command)); + require(command.cmdsize >= sizeof(command) && command.cmdsize % 8 == 0 && command.cmdsize <= header.sizeofcmds - offset); + if (command.cmd == LC_UUID) { + require(!identifier && command.cmdsize == sizeof(uuid_command)); + uuid_command item{}; std::memcpy(&item, commands.data() + offset, sizeof(item)); + std::copy(std::begin(item.uuid), std::end(item.uuid), row.id.begin()); + identifier = true; + } + if (command.cmd == LC_SEGMENT_64) { + require(command.cmdsize >= sizeof(segment_command_64)); + segment_command_64 item{}; std::memcpy(&item, commands.data() + offset, sizeof(item)); + require(item.nsects <= (command.cmdsize - sizeof(item)) / sizeof(section_64) && + sizeof(item) + item.nsects * sizeof(section_64) == command.cmdsize); + end(item.vmaddr, item.vmsize); + if (std::strncmp(item.segname, "__TEXT", sizeof(item.segname)) == 0) { + require(!text && item.vmsize); + const auto actual_slide = image_slide(row.address, item.vmaddr); + if (derive_slide) row.slide = actual_slide; + else require(actual_slide == row.slide); + row.text_size = item.vmsize; text = true; + } + } + offset += command.cmdsize; + } + require(text && offset == header.sizeofcmds); + // An ordinary executable built without LC_UUID is still covered by its full + // file digest. Cached images must carry a matching UUID in attribute(). + return row; +} +struct image_state { + std::mutex mutex; + std::vector rows; + std::uint64_t generation = 0; + bool failed = false; +}; +// dyld callbacks cannot be unregistered and may run during process teardown. +image_state &state() { static auto *value = new image_state; return *value; } +void added(const mach_header *header, intptr_t slide) { + auto &owner = state(); + try { + auto row = inspect(header, slide); + std::lock_guard lock(owner.mutex); + ++owner.generation; + if (owner.rows.size() >= 4096) owner.failed = true; + else owner.rows.push_back(std::move(row)); + } catch (...) { + std::lock_guard lock(owner.mutex); + ++owner.generation; owner.failed = true; + } +} +void removed(const mach_header *header, intptr_t) { + auto &owner = state(); + std::lock_guard lock(owner.mutex); + ++owner.generation; + const auto address = reinterpret_cast(header); + owner.rows.erase(std::remove_if(owner.rows.begin(), owner.rows.end(), + [&](const auto &row) { return row.address == address; }), owner.rows.end()); +} +template Function symbol(const char *name) { + dlerror(); void *pointer = dlsym(RTLD_DEFAULT, name); + require(pointer && !dlerror()); + return reinterpret_cast(pointer); +} +using opaque = void *; +struct api { + opaque (*create)() = symbol("dyld_process_create_for_current_task"); + void (*dispose)(opaque) = symbol("dyld_process_dispose"); + opaque (*snapshot)(opaque, int *) = symbol("dyld_process_snapshot_create_for_process"); + void (*free_snapshot)(opaque) = symbol("dyld_process_snapshot_dispose"); + opaque (*cache)(opaque) = symbol("dyld_process_snapshot_get_shared_cache"); + void (*cache_uuid)(opaque, unsigned char(*)[16]) = symbol("dyld_shared_cache_copy_uuid"); + std::uint64_t (*base)(opaque) = symbol("dyld_shared_cache_get_base_address"); + std::uint64_t (*size)(opaque) = symbol("dyld_shared_cache_get_mapped_size"); + bool (*private_mapping)(opaque) = symbol("dyld_shared_cache_is_mapped_private"); + void (*files)(opaque, void(^)(const char *)) = symbol("dyld_shared_cache_for_each_file"); +}; +struct snapshot { + observation active; + std::vector paths; +}; +snapshot capture(const api &functions, opaque process) { + int error = -1; + std::unique_ptr owner(functions.snapshot(process, &error), functions.free_snapshot); + require(owner && !error); + const auto cache = functions.cache(owner.get()); + require(cache && !functions.private_mapping(cache)); + snapshot result; + unsigned char id[16]{}; + functions.cache_uuid(cache, &id); + std::copy(std::begin(id), std::end(id), result.active.cache_id.begin()); + result.active.base = functions.base(cache); result.active.size = functions.size(cache); + __block bool complete = true; + // A pointer capture avoids copying the mutable C++ owner into the block. + auto *paths = &result.paths; + functions.files(cache, ^(const char *path) { + if (!path || path[0] != '/' || strnlen(path, 4096) == 4096 || paths->size() >= maximum_files) { complete = false; return; } + try { paths->emplace_back(path); } catch (...) { complete = false; } + }); + require(complete && !result.paths.empty()); + std::sort(result.paths.begin(), result.paths.end()); + require(std::adjacent_find(result.paths.begin(), result.paths.end()) == result.paths.end()); + return result; +} +dyld_all_image_infos loader() { + task_dyld_info_data_t task{}; + mach_msg_type_number_t count = TASK_DYLD_INFO_COUNT; + require(task_info(mach_task_self(), TASK_DYLD_INFO, reinterpret_cast(&task), &count) == KERN_SUCCESS && + count == TASK_DYLD_INFO_COUNT && task.all_image_info_format == TASK_DYLD_ALL_IMAGE_INFO_64 && + task.all_image_info_size >= offsetof(dyld_all_image_infos, dyldPath) + sizeof(const char *)); + dyld_all_image_infos info{}; + require(read_self(task.all_image_info_addr, &info, offsetof(dyld_all_image_infos, dyldPath) + sizeof(const char *)) && + info.version >= 15 && info.dyldImageLoadAddress && info.sharedCacheBaseAddress && + info.infoArray && info.infoArrayCount && info.infoArrayCount <= 4096); + return info; +} +bool same_file(const struct stat &a, const struct stat &b) { + return a.st_dev == b.st_dev && a.st_ino == b.st_ino && a.st_size == b.st_size && + a.st_mtimespec.tv_sec == b.st_mtimespec.tv_sec && a.st_mtimespec.tv_nsec == b.st_mtimespec.tv_nsec && + a.st_ctimespec.tv_sec == b.st_ctimespec.tv_sec && a.st_ctimespec.tv_nsec == b.st_ctimespec.tv_nsec; +} +void retain_readable(files &observed, const std::string &path) { + if (path.empty() || path.front() != '/' || path.find('\0') != std::string::npos) return; + std::error_code error; + if (fs::is_regular_file(fs::u8path(path), error) && !error && + static_cast(std::ifstream(fs::u8path(path), std::ios::binary))) observed.insert(path); +} +} // namespace + +bool collect_darwin(files &observed) { + try { + // Preserve independently known ordinary inputs even on an older host + // without the optional shared-cache APIs. This inventory alone never + // certifies completeness; cached virtual install names are not files. + const auto ordinary_count = _dyld_image_count(); + require(ordinary_count <= 4096); + for (std::uint32_t i = 0; i < ordinary_count; ++i) { + const auto *path = _dyld_get_image_name(i); + if (path && strnlen(path, 4096) < 4096) retain_readable(observed, path); + } + api functions; + std::unique_ptr process(functions.create(), functions.dispose); + require(static_cast(process)); + // Resolve/prewarm optional API and digest machinery before observing loader + // generations. Their libraries are themselves part of the resulting set. + CC_SHA256_CTX digest{}; require(CC_SHA256_Init(&digest) == 1); + (void)capture(functions, process.get()); + static std::once_flag registration; + std::call_once(registration, [] { + (void)state(); + _dyld_register_func_for_add_image(added); + _dyld_register_func_for_remove_image(removed); + }); + auto before = capture(functions, process.get()); + const auto loaded = loader(); + require(loaded.sharedCacheBaseAddress == before.active.base && + std::equal(std::begin(loaded.sharedCacheUUID), std::end(loaded.sharedCacheUUID), before.active.cache_id.begin())); + before.active.slide = loaded.sharedCacheSlide; + before.active.loader = reinterpret_cast(loaded.dyldImageLoadAddress); + std::uint64_t generation = 0; + { + auto &owner = state(); + std::lock_guard lock(owner.mutex); + require(!owner.failed && owner.rows.size() == loaded.infoArrayCount); + before.active.images = owner.rows; generation = owner.generation; + } + if (std::none_of(before.active.images.begin(), before.active.images.end(), + [&](const auto &row) { return row.address == before.active.loader; })) + before.active.images.push_back(inspect(loaded.dyldImageLoadAddress, 0, true)); + for (auto &row : before.active.images) { + Dl_info named{}; + if (dladdr(reinterpret_cast(row.address), &named) && + reinterpret_cast(named.dli_fbase) == row.address && named.dli_fname && + strnlen(named.dli_fname, 4096) < 4096) row.ordinary_path = named.dli_fname; + if (row.ordinary_path.empty() && row.address == before.active.loader && loaded.dyldPath && + strnlen(loaded.dyldPath, 4096) < 4096) row.ordinary_path = loaded.dyldPath; + retain_readable(observed, row.ordinary_path); + } + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(15); + std::size_t remaining = 8U * 1024 * 1024; + std::vector caches; + std::set> backing_ids; + for (const auto &path : before.paths) { + const int descriptor = open(path.c_str(), O_RDONLY | O_CLOEXEC | O_NONBLOCK); + require(descriptor >= 0); + struct close_file { int value; ~close_file() { close(value); } } cleanup{descriptor}; + struct stat first{}, last{}, named{}; + require(!fstat(descriptor, &first) && S_ISREG(first.st_mode) && first.st_size > 0 && + static_cast(first.st_size) <= maximum_bytes && backing_ids.emplace(first.st_dev, first.st_ino).second); + // A readable API-reported backing file remains protected even if a + // later metadata check is unsupported or another backing file is absent. + observed.insert(path); + const reader read = [&](std::uint64_t offset, std::size_t count) { + require(std::chrono::steady_clock::now() < deadline && count <= remaining); + end(offset, count, static_cast(first.st_size)); + remaining -= count; + bytes data(count); + std::size_t copied = 0; + while (copied < count) { + require(std::chrono::steady_clock::now() < deadline); + const auto size = pread(descriptor, data.data() + copied, count - copied, static_cast(offset + copied)); + if (size < 0 && errno == EINTR) continue; + require(size > 0); copied += static_cast(size); + } + return data; + }; + const auto prefix = read(0, 104); + caches.push_back(parse(path, static_cast(first.st_size), read, + identity(prefix, 88) == before.active.cache_id)); + require(!fstat(descriptor, &last) && !stat(path.c_str(), &named) && same_file(first, last) && same_file(first, named)); + } +#if defined(__aarch64__) || defined(__arm64__) + constexpr std::uint32_t cpu = 0x0100000cU; +#else + constexpr std::uint32_t cpu = 0x01000007U; +#endif + const auto paths = attribute(before.active, caches, cpu); + for (const auto &path : paths) { + std::error_code error; + require(fs::is_regular_file(fs::u8path(path), error) && !error && static_cast(std::ifstream(fs::u8path(path), std::ios::binary))); + } + const auto after = capture(functions, process.get()); + const auto loaded_after = loader(); + require(after.paths == before.paths && after.active.cache_id == before.active.cache_id && + after.active.base == before.active.base && after.active.size == before.active.size && + loaded_after.infoArrayChangeTimestamp == loaded.infoArrayChangeTimestamp && + loaded_after.sharedCacheBaseAddress == loaded.sharedCacheBaseAddress && + loaded_after.sharedCacheSlide == loaded.sharedCacheSlide); + { + auto &owner = state(); std::lock_guard lock(owner.mutex); + require(!owner.failed && owner.generation == generation); + } + observed.insert(paths.begin(), paths.end()); + return true; + } catch (...) { + return false; + } +} +} // namespace nil::detail::compiler_modules diff --git a/nilcompiler/src/compiler_module_fingerprint_darwin.hpp b/nilcompiler/src/compiler_module_fingerprint_darwin.hpp new file mode 100644 index 00000000..b9bb6051 --- /dev/null +++ b/nilcompiler/src/compiler_module_fingerprint_darwin.hpp @@ -0,0 +1,241 @@ +#pragma once + +// Bounded, portable parsing and attribution for the Darwin module collector. +// UUIDs select backing files; the builder hashes their entire contents. Neither +// UUIDs nor timestamps are persistent cache identities. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace nil::detail::compiler_modules::darwin { +using uuid = std::array; +using bytes = std::vector; +constexpr std::size_t maximum_images = 16384, maximum_files = 128; +constexpr std::uint64_t maximum_bytes = 8ULL * 1024 * 1024 * 1024; +inline void require(bool condition) { if (!condition) throw std::runtime_error("incomplete Darwin module identity"); } +inline std::uint64_t end(std::uint64_t begin, std::uint64_t size, + std::uint64_t limit = UINT64_MAX) { + require(begin <= limit && size <= limit - begin); + return begin + size; +} +inline std::uint64_t number(const bytes &data, std::size_t offset, std::size_t width) { + require(width <= 8 && offset <= data.size() && width <= data.size() - offset); + std::uint64_t value = 0; + for (std::size_t i = 0; i < width; ++i) value |= std::uint64_t(data[offset + i]) << (8 * i); + return value; +} +inline uuid identity(const bytes &data, std::size_t offset) { + require(offset <= data.size() && 16 <= data.size() - offset); + uuid value{}; + std::copy_n(data.begin() + offset, value.size(), value.begin()); + require(value != uuid{}); + return value; +} +struct mapping { + std::uint64_t address = 0, size = 0, offset = 0; + std::uint32_t maximum = 0, initial = 0; +}; +struct subcache { uuid id{}; std::uint64_t offset = 0; std::string suffix; }; +struct cache_image { uuid id{}; std::uint64_t address = 0, size = 0; }; +struct cache_file { + std::string path, magic; + uuid id{}; + std::uint64_t size = 0, base = 0, region_size = 0; + std::vector mappings; + std::vector children; + std::vector images; +}; +using reader = std::function; +inline void disjoint(std::vector> ranges) { + std::sort(ranges.begin(), ranges.end()); + for (std::size_t i = 0; i < ranges.size(); ++i) { + const auto finish = end(ranges[i].first, ranges[i].second); + require(ranges[i].second != 0 && (i + 1 == ranges.size() || finish <= ranges[i + 1].first)); + } +} +inline cache_file parse(const std::string &path, std::uint64_t size, + const reader &read, bool image_table = false) { + require(!path.empty() && path.front() == '/' && path.find('\0') == std::string::npos && size && size <= maximum_bytes); + const auto first = read(0, 24); + require(first.size() == 24); + std::string magic(first.begin(), first.begin() + 16); + while (!magic.empty() && (magic.back() == '\0' || magic.back() == ' ')) magic.pop_back(); + require(magic == "dyld_v1 arm64e" || magic == "dyld_v1 arm64" || + magic == "dyld_v1 x86_64" || magic == "dyld_v1 x86_64h"); + const auto header_size = number(first, 16, 4), count = number(first, 20, 4); + require(header_size >= 400 && header_size <= 4096 && count > 0 && count <= maximum_files); + const auto header = read(0, static_cast(header_size)); + require(header.size() == header_size); + cache_file result; + result.path = path; result.magic = magic; result.size = size; + result.id = identity(header, 88); + result.base = number(header, 224, 8); result.region_size = number(header, 232, 8); + end(result.base, result.region_size); + const auto table = [&](std::uint64_t offset, std::uint64_t entries, std::size_t stride, + std::size_t maximum = maximum_files) { + require(entries <= maximum && entries <= UINT64_MAX / stride); + end(offset, entries * stride, size); + auto data = read(offset, static_cast(entries * stride)); + require(data.size() == entries * stride); + return data; + }; + const auto mappings = table(header_size, count, 32); + std::vector> addresses, offsets; + for (std::size_t i = 0; i < count; ++i) { + const auto at = i * 32; + mapping item{number(mappings, at, 8), number(mappings, at + 8, 8), number(mappings, at + 16, 8), + static_cast(number(mappings, at + 24, 4)), static_cast(number(mappings, at + 28, 4))}; + require(item.size && !(item.initial & ~item.maximum)); + end(item.offset, item.size, size); + addresses.emplace_back(item.address, item.size); offsets.emplace_back(item.offset, item.size); + result.mappings.push_back(item); + } + disjoint(addresses); disjoint(offsets); + const auto slide_count = number(header, 316, 4); + require(slide_count == count); + const auto slides = table(number(header, 312, 4), slide_count, 56); + for (std::size_t i = 0; i < count; ++i) { + const auto at = i * 56; + const auto &item = result.mappings[i]; + require(item.address == number(slides, at, 8) && item.size == number(slides, at + 8, 8) && + item.offset == number(slides, at + 16, 8) && item.maximum == number(slides, at + 48, 4) && + item.initial == number(slides, at + 52, 4)); + end(number(slides, at + 24, 8), number(slides, at + 32, 8), size); + } + const bool modern = header_size > 456; + const auto children_count = number(header, 396, 4); + const std::size_t stride = modern ? 56 : 24; + const auto children = table(number(header, 392, 4), children_count, stride); + std::set ids; + std::set suffixes; + std::set child_offsets; + for (std::size_t i = 0; i < children_count; ++i) { + subcache child{identity(children, i * stride), number(children, i * stride + 16, 8), {}}; + if (modern) { + const auto begin = children.begin() + i * stride + 24; + const auto finish = std::find(begin, begin + 32, 0); + require(finish != begin + 32); + child.suffix.assign(begin, finish); + require(child.suffix.size() > 1 && child.suffix.front() == '.' && child.suffix.back() != '.' && + child.suffix.find("..") == std::string::npos && + child.suffix.find_first_not_of(".abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789") == std::string::npos); + } else child.suffix = "." + std::to_string(i + 1); + require(child.id != result.id && child.offset && ids.insert(child.id).second && + suffixes.insert(child.suffix).second && child_offsets.insert(child.offset).second); + result.children.push_back(std::move(child)); + } + end(number(header, 40, 8), number(header, 48, 8), size); + end(number(header, 72, 8), number(header, 80, 8), size); + if (image_table) { + const auto image_count = number(header, 144, 8); + require(image_count > 0); + const auto images = table(number(header, 136, 8), image_count, 32, maximum_images); + std::set image_addresses; + for (std::size_t i = 0; i < image_count; ++i) { + cache_image image{identity(images, i * 32), number(images, i * 32 + 16, 8), number(images, i * 32 + 24, 4)}; + require(image.size && image_addresses.insert(image.address).second); + end(image.address, image.size); + result.images.push_back(image); + } + } + return result; +} +struct loaded_image { + uuid id{}; + std::uint64_t address = 0, text_size = 0; + std::int64_t slide = 0; + std::uint32_t cpu = 0, flags = 0; + std::string ordinary_path; +}; +inline std::int64_t image_slide(std::uint64_t address, std::uint64_t unslid) { + if (address >= unslid) { + require(address - unslid <= static_cast(INT64_MAX)); + return static_cast(address - unslid); + } + const auto magnitude = unslid - address; + require(magnitude <= (std::uint64_t{1} << 63)); + return magnitude == (std::uint64_t{1} << 63) ? INT64_MIN : -static_cast(magnitude); +} +struct observation { + uuid cache_id{}; + std::uint64_t base = 0, size = 0, slide = 0, loader = 0; + std::vector images; +}; +// This attribution is equivalent to Linux/Windows loaded-path identity, not +// vnode attestation. The whole selected files remain subject to content hashing +// and the builder's independent post-compilation graph/content verification. +inline std::set attribute(const observation &active, + const std::vector &caches, + std::uint32_t cpu) { + require(!caches.empty() && caches.size() <= maximum_files && !active.images.empty() && active.images.size() <= 4096); + const cache_file *root = nullptr; + std::map by_path; + std::set cache_ids; + std::uint64_t total = 0; + for (const auto &file : caches) { + require(by_path.emplace(file.path, &file).second && cache_ids.insert(file.id).second); + total = end(total, file.size, maximum_bytes); + if (file.id == active.cache_id) { require(!root); root = &file; } + } + require(root && root->children.size() + 1 == caches.size() && root->base && + root->region_size == active.size && end(root->base, active.slide) == active.base); + const auto first_mapping = [](const cache_file &file) { + require(!file.mappings.empty()); + return std::min_element(file.mappings.begin(), file.mappings.end(), + [](const auto &a, const auto &b) { return a.address < b.address; })->address; + }; + require(first_mapping(*root) == root->base && !root->images.empty()); + for (const auto &child : root->children) { + const auto found = by_path.find(root->path + child.suffix); + require(found != by_path.end()); + const auto &file = *found->second; + require(file.id == child.id && file.children.empty() && file.magic == root->magic && + first_mapping(file) == end(root->base, child.offset)); + } + std::vector> ranges; + for (const auto &file : caches) for (const auto &map : file.mappings) { + require(map.address >= root->base); + end(map.address - root->base, map.size, root->region_size); + ranges.emplace_back(map.address, map.size); + } + disjoint(ranges); + std::map images; + for (const auto &image : root->images) require(images.emplace(image.address, &image).second); + std::set result; + for (const auto &file : caches) result.insert(file.path); + std::set seen; + bool loader_seen = false, cached = false; + const bool arm = root->magic.find("arm64") != std::string::npos; + require(cpu == (arm ? 0x0100000cU : 0x01000007U)); + for (const auto &image : active.images) { + require(image.cpu == cpu && image.address && image.text_size && seen.insert(image.address).second); + loader_seen |= image.address == active.loader; + const mapping *mapped = nullptr; + for (const auto &file : caches) for (const auto &map : file.mappings) { + const auto begin = end(map.address, active.slide); + if (image.address >= begin && image.address < end(begin, map.size)) { require(!mapped); mapped = ↦ } + } + if (mapped) { + require(image.address >= active.slide && image.slide >= 0 && static_cast(image.slide) == active.slide && (image.flags & 0x80000000U)); + const auto unslid = image.address - active.slide; + const auto found = images.find(unslid); + require(found != images.end() && found->second->id == image.id && found->second->size == image.text_size && + (mapped->initial & 5) == 5 && end(unslid, image.text_size) <= end(mapped->address, mapped->size)); + cached = true; + } else { + require(!(image.address >= active.base && image.address < end(active.base, active.size))); + require(!image.ordinary_path.empty() && image.ordinary_path.front() == '/' && image.ordinary_path.find('\0') == std::string::npos); + result.insert(image.ordinary_path); + } + } + require(loader_seen && cached); + return result; +} +} // namespace nil::detail::compiler_modules::darwin diff --git a/nilcompiler/src/main.cpp b/nilcompiler/src/main.cpp index 9b0f1b65..0ea32186 100644 --- a/nilcompiler/src/main.cpp +++ b/nilcompiler/src/main.cpp @@ -1673,8 +1673,11 @@ static int compile_main(int argc, char *argv[], ny::analysis_session *analysis) // a false graph change between separate analysis processes. analysis->compiler_files_complete = nil::detail::compiler_modules::collect_windows(analysis->compiler_files); +#elif defined(__APPLE__) + analysis->compiler_files_complete = + nil::detail::compiler_modules::collect_darwin(analysis->compiler_files); #else - // Darwin loaded images/shared-cache backing identity is not yet complete. + // Unknown hosts cannot establish a complete loaded-module inventory. // Successful source analysis still reports and protects all consumed files. analysis->compiler_files_complete = false; #endif diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index eaeaeaed..4c1f2e45 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -100,6 +100,35 @@ dirty guest checkout and accepted SDK. Its report SHA is A separate eight-case Windows probe confirms that textual and binary source maps embed canonical long paths even when the compiler receives an 8.3 TEMP alias. +The completed hosted web-runtime run `43116117693` passes all six contracts on +Linux, Windows, Intel macOS and ARM macOS. Independent replay checked 148 artifact +hashes and each host's timestamp byte streams. Its checked-out PR merge is +`cec4364d`, with requested head `19bd5f1` retained as a separate identity. AppKit +run `43115312796` passes all five CTests and all six interpreter/native desktop +cases on both architectures at exact head `965f159`. The LLVM-enabled Linux +source workflow at `19bd5f1` also passes all 579 CTests with no skips; its LLVM-OFF +and extension jobs pass. + +## Darwin compiler cache identity + +The compiler now has a bounded Darwin loaded-image collector. It matches actual +headers (including dyld) to exact cache mappings and UUID tables, preserves +ordinary loaded paths and returns the complete backing-file set for ordinary +full-content SHA256 hashing. UUIDs are attribution metadata, never cache keys. +Unknown APIs/profiles or unstable observations remain uncached; known readable +inputs are retained for output protection even when the inventory is incomplete. +CommonCrypto computes the same digest as the portable SHA implementation. + +The builder hashes shared toolchain files once per preparation phase and once +again in a fresh verification phase. It never carries those values across builds +or compilation boundaries. Consumed sources and manifests, including symlink and +hard-link aliases, are excluded from memoization. Seventy portable controls, +36 explicit identity checks and retained real Intel/ARM metadata replay pass. +The coordinated Linux builder, fingerprint and source-protection regressions +also pass. Actual Apple API execution is a required hosted test. Native linker +input closure is being implemented separately; compiler identity alone does not +establish native incremental reuse. + ## Darwin allocation retention and editor coverage The two native lifetime programs remain unchanged. Darwin now has a test-only @@ -123,6 +152,13 @@ It uses a fresh profile and Microsoft's architecture-specific VS Code 1.146.0 archives, pinned by SHA256 from the official update service. All results are development/build-tree checks; no installed SDK or user editor is activated. +The first application campaign at `19bd5f1` exposed two setup errors. The public +Apple SDK does not ship dyld's interposition convenience header; the fixture now +declares its own pointer pairs in the documented Mach-O interposing section. +The pinned editor archive uses a different executable name; the launcher now +reads `CFBundleExecutable` from its own verified app bundle. Both failures are +retained, and actual injection/editor acceptance remains pending a corrected run. + Native socket/byte/entropy parity needs a binary-safe value boundary. The new VM APIs deliberately carry embedded NUL and arbitrary bytes in `text`; the current native text ABI and several LLVM operations use null-terminated strings diff --git a/scripts/ci/macos-toolchain-census.py b/scripts/ci/macos-toolchain-census.py new file mode 100644 index 00000000..b3edda63 --- /dev/null +++ b/scripts/ci/macos-toolchain-census.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python3 +"""Bounded read-only SDK/link-plan evidence; this does not enable build reuse.""" +from __future__ import annotations +import argparse +import hashlib +import json +import os +from pathlib import Path +import subprocess +import sys +import time + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--build', required=True, type=Path) + parser.add_argument('--output', required=True, type=Path) + args = parser.parse_args() + assert sys.platform == 'darwin' + args.output.mkdir(parents=True, exist_ok=False) + deadline = time.monotonic() + 90 + report = {'status': 'FAIL', 'scope': 'SDK stub census and dry link plan only', 'commands': []} + + def run(name, argv): + remaining = deadline - time.monotonic() + assert remaining > 0, 'census deadline exhausted' + result = subprocess.run([str(value) for value in argv], stdin=subprocess.DEVNULL, + capture_output=True, timeout=min(20, remaining)) + for kind, data in [('stdout', result.stdout), ('stderr', result.stderr)]: + assert len(data) <= 2*1024*1024, 'census output exceeded bound' + (args.output/(name+'.'+kind)).write_bytes(data) + report['commands'].append({'name': name, 'argv': [str(value) for value in argv], + 'exit_code': result.returncode}) + result.check_returncode() + return result.stdout.decode('utf-8').strip() + + try: + cache = (args.build/'CMakeCache.txt').read_text() + drivers = [line.split('=', 1)[1] for line in cache.splitlines() + if line.startswith('NIL_NATIVE_LINKER_DRIVER:STRING=')] + assert len(drivers) == 1 + driver = Path(drivers[0]); assert driver.is_absolute() and driver.is_file() + sdk = Path(run('sdk', ['/usr/bin/xcrun', '--sdk', 'macosx', '--show-sdk-path'])) + assert sdk.is_absolute() and sdk.is_dir() + archive = (args.build/'lib/libnil.a').resolve(strict=True) + run('selected-driver', [driver, '--version']) + run('native-link-plan', [driver, '-###', '-pthread', '-o', '/dev/null', archive, + '-framework', 'Cocoa', '-framework', 'CoreGraphics']) + run('linker', ['/usr/bin/xcrun', '--find', 'ld']) + rows, entries, total = [], 0, 0 + for root in [sdk/'usr/lib', sdk/'System/Library/Frameworks']: + for directory, directories, files in os.walk(root, followlinks=False): + assert time.monotonic() < deadline, 'SDK census deadline exhausted' + entries += len(directories) + len(files) + assert entries <= 250000, 'SDK namespace exceeds census bound' + for name in sorted(files): + path = Path(directory)/name + if path.suffix != '.tbd': + continue + assert len(rows) < 20000, 'too many SDK stubs' + with path.open('rb') as stream: + prefix = stream.read(2048) + total += len(prefix) + assert total <= 32*1024*1024, 'SDK prefix budget exceeded' + rows.append({'path': path.relative_to(sdk).as_posix(), + 'resolved': str(path.resolve()), 'bytes': path.stat().st_size, + 'prefix': prefix.decode('utf-8', errors='replace'), + 'prefix_sha256': hashlib.sha256(prefix).hexdigest()}) + assert rows, 'no SDK stubs observed' + report.update(status='PASS', sdk=str(sdk), driver=str(driver), runtime=str(archive), + entries=entries, prefix_bytes=total, stubs=rows) + except Exception as error: + report['error'] = str(error) + (args.output/'report.json').write_text(json.dumps(report, indent=2)+'\n') + print(report['status'] + ': ' + str(args.output/'report.json')) + return 0 if report['status'] == 'PASS' else 1 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/tests/compiler_module_darwin_tests.cpp b/tests/compiler_module_darwin_tests.cpp new file mode 100644 index 00000000..18b501ad --- /dev/null +++ b/tests/compiler_module_darwin_tests.cpp @@ -0,0 +1,186 @@ +#include "../nilcompiler/src/compiler_module_fingerprint.hpp" +#include "../nilcompiler/src/compiler_module_fingerprint_darwin.hpp" +#include "../nilbuild/digest_phase.hpp" +#include "../common/private_directory.hpp" + +#include +#include +#include + +namespace d = nil::detail::compiler_modules::darwin; +namespace fs = std::filesystem; +namespace { +unsigned checks = 0; +void check(bool value, const char *message) { ++checks; if (!value) throw std::runtime_error(message); } +template void rejected(Function invoke, const char *message) { + bool failed = false; + try { invoke(); } catch (const std::exception &) { failed = true; } + check(failed, message); +} +void put(d::bytes &data, std::size_t offset, std::uint64_t value, std::size_t width = 8) { + for (std::size_t i = 0; i < width; ++i) data.at(offset + i) = static_cast(value >> (8 * i)); +} +d::uuid id(unsigned char value) { d::uuid result{}; result.back() = value; return result; } +void put_id(d::bytes &data, std::size_t offset, unsigned char value) { data.at(offset + 15) = value; } +d::bytes cache(bool root) { + d::bytes data(8192); + const std::string magic = "dyld_v1 arm64e"; + std::copy(magic.begin(), magic.end(), data.begin()); + put(data, 16, 512, 4); put(data, 20, 1, 4); put_id(data, 88, root ? 1 : 2); + put(data, 224, 0x100000); put(data, 232, 0x10000); + put(data, 312, 544, 4); put(data, 316, 1, 4); + for (const auto offset : {512U, 544U}) { + put(data, offset, root ? 0x100000 : 0x108000); put(data, offset + 8, 8192); + put(data, offset + (offset == 512 ? 24 : 48), 5, 4); + put(data, offset + (offset == 512 ? 28 : 52), 5, 4); + } + if (root) { + put(data, 392, 600, 4); put(data, 396, 1, 4); put_id(data, 600, 2); put(data, 616, 0x8000); + data[624] = '.'; data[625] = '1'; + put(data, 136, 1024); put(data, 144, 2); + put_id(data, 1024, 3); put(data, 1040, 0x101000); put(data, 1048, 0x100, 4); + put_id(data, 1056, 4); put(data, 1072, 0x109000); put(data, 1080, 0x100, 4); + } + return data; +} +d::cache_file parse(const d::bytes &data, bool root) { + return d::parse(root ? "/cache" : "/cache.1", data.size(), [&](std::uint64_t offset, std::size_t size) { + d::end(offset, size, data.size()); + return d::bytes(data.begin() + offset, data.begin() + offset + size); + }, root); +} +d::observation active() { + d::observation result; + result.cache_id = id(1); result.base = 0x1100000; result.size = 0x10000; + result.slide = 0x1000000; result.loader = 0x1109000; + result.images = {{id(3), 0x1101000, 0x100, 0x1000000, 0x100000c, 0x80000000U, {}}, + {id(4), result.loader, 0x100, 0x1000000, 0x100000c, 0x80000000U, {}}, + {id(3), 0x5000000, 0x100, 0, 0x100000c, 0x80000000U, "/ordinary"}}; + return result; +} +void attribution() { + const auto root = cache(true), child = cache(false); + const std::vector files{parse(root, true), parse(child, false)}; + check(d::attribute(active(), files, 0x100000c) == std::set{"/cache", "/cache.1", "/ordinary"}, + "complete backing set and same-UUID outside-cache image stay content inputs"); + auto ordinary_loader = active(); + ordinary_loader.loader = ordinary_loader.images.back().address; + ordinary_loader.images.back().slide = -4096; + check(d::attribute(ordinary_loader, files, 0x100000c).count("/ordinary"), "ordinary dyld uses its own slide and full file identity"); + ordinary_loader.images.back().ordinary_path.clear(); + rejected([&] { d::attribute(ordinary_loader, files, 0x100000c); }, "ordinary dyld without readable-path attribution is incomplete"); + check(d::image_slide(0x1000, 0x2000) == -4096 && d::image_slide(0x2000, 0x1000) == 4096, + "actual header derives positive and negative ordinary loader slides"); + auto mutate_image = [&](auto mutation) { auto value = active(); mutation(value); d::attribute(value, files, 0x100000c); }; + rejected([&] { mutate_image([](auto &v) { ++v.images[0].address; }); }, "off-by-one header cannot match cache image"); + rejected([&] { mutate_image([](auto &v) { ++v.images[0].slide; }); }, "wrong slide cannot match cache image"); + rejected([&] { mutate_image([](auto &v) { v.images[0].id = id(8); }); }, "wrong UUID cannot match cache image"); + rejected([&] { mutate_image([](auto &v) { v.images[0].address = v.base + 0x4000; }); }, "envelope gap is not a cache mapping"); + rejected([&] { mutate_image([](auto &v) { v.images[0].text_size = 0x2000; }); }, "TEXT bounds cannot cross mappings"); + rejected([&] { mutate_image([](auto &v) { v.images[0].flags = 0; }); }, "cached image flag required"); + rejected([&] { mutate_image([](auto &v) { v.images[2].ordinary_path.clear(); }); }, "ordinary image needs actual path"); + rejected([&] { mutate_image([](auto &v) { v.images.erase(v.images.begin() + 1); }); }, "dyld loader must be accounted for"); + rejected([&] { mutate_image([](auto &v) { v.images.push_back(v.images[0]); }); }, "duplicate header rejected"); + rejected([&] { mutate_image([](auto &v) { v.cache_id = id(7); }); }, "wrong active root rejected"); + rejected([&] { mutate_image([](auto &v) { ++v.size; }); }, "active and disk cache bounds must match"); + auto mutate_files = [&](auto mutation) { auto value = files; mutation(value); d::attribute(active(), value, 0x100000c); }; + rejected([&] { mutate_files([](auto &v) { v.pop_back(); }); }, "missing backing file rejected"); + rejected([&] { mutate_files([](auto &v) { v.push_back(v[1]); }); }, "duplicate backing file rejected"); + rejected([&] { mutate_files([](auto &v) { v[1].id = id(5); }); }, "subcache UUID correspondence required"); + rejected([&] { mutate_files([](auto &v) { v[1].path += ".other"; }); }, "root suffix set must match exact file list"); + rejected([&] { mutate_files([](auto &v) { v[1].mappings[0].address = 0x100000; }); }, "overlapping cache mappings rejected"); + rejected([&] { mutate_files([](auto &v) { v[0].images.push_back(v[0].images[0]); }); }, "duplicate image table address rejected"); + rejected([&] { mutate_files([](auto &v) { v[0].children[0].offset += 1; }); }, "subcache VM offset correspondence required"); + rejected([&] { d::attribute(active(), files, 0x1000007); }, "target architecture must match cache"); + for (const auto offset : {16U, 20U, 312U, 316U, 392U, 396U, 136U, 144U}) { + auto invalid = root; put(invalid, offset, UINT32_MAX, 4); + rejected([&] { parse(invalid, true); }, "out-of-range metadata rejected"); + } + auto truncated = root; truncated.resize(100); + rejected([&] { parse(truncated, true); }, "truncated metadata rejected"); + auto overflow = root; put(overflow, 512, UINT64_MAX - 5); + rejected([&] { parse(overflow, true); }, "overflowing address rejected"); + auto suffix = root; suffix[626] = '/'; + rejected([&] { parse(suffix, true); }, "subcache suffix cannot introduce path traversal"); +} +void digests(const fs::path &directory) { + check(nil::detail::content_digest("") == "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "empty SHA256 vector"); + check(nil::detail::content_digest("abc") == "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", "abc SHA256 vector"); + check(nil::detail::content_digest(std::string(1000000, 'a')) == "cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0", "million-a SHA256 vector"); + for (const std::size_t size : {1U, 55U, 56U, 63U, 64U, 65U, 4095U, 4096U, 65535U, 65536U, 65537U, 1048593U}) { + std::string value(size, '\0'); + for (std::size_t i = 0; i < size; ++i) value[i] = static_cast(i * 31); + nil::detail::portable_sha256 portable; + portable.update(value.data(), value.size()); + const auto expected = portable.finish(); + check(nil::detail::content_digest(value) == expected, "platform and scalar SHA256 agree at boundaries"); + const auto file = directory / "input"; + { std::ofstream output(file, std::ios::binary); output.write(value.data(), value.size()); } + check(nil::detail::file_digest(file) == expected, "file SHA256 hashes every byte including NUL"); + } + const auto file = directory / "input"; + int cancellation = 0; + rejected([&] { nil::detail::file_digest(file, [&] { if (++cancellation == 3) throw std::runtime_error("cancel"); }); }, "hash checks cancellation between bounded chunks"); + check(cancellation == 3, "cancellation stops the hash immediately"); + nilbuild::digest_phase prepared; + unsigned reads = 0; + const auto read = [&](const fs::path &path) { ++reads; return nil::detail::file_digest(path); }; + const auto before = prepared.read(file, read); + check(prepared.read(file, read) == before && reads == 1, "same preparation phase coalesces shared input"); + const auto stamp = fs::last_write_time(file); + { std::fstream output(file, std::ios::binary | std::ios::in | std::ios::out); output.put('Z'); } + fs::last_write_time(file, stamp); + nilbuild::digest_phase verified; + check(verified.read(file, read) != before && reads == 2, "verification reads new bytes despite same length and timestamp"); + nilbuild::digest_phase next_build; + check(next_build.read(file, read) == verified.read(file, read) && reads == 3, "new invocation cannot reuse a previous phase digest"); + nilbuild::digest_phase overlap; + overlap.exclude(file); + const auto old_reads = reads; + const auto current = overlap.read(file, read); + { std::fstream output(file, std::ios::binary | std::ios::in | std::ios::out); output.put('Q'); } + fs::last_write_time(file, stamp); + check(overlap.read(file, read) != current && reads == old_reads + 2 && !overlap.contains(file.string()), + "consumed source advertised as a shared module is never memoized"); + const auto alias = +#ifdef _WIN32 + directory / "." / file.filename(); +#else + directory / "alias"; + fs::create_symlink(file.filename(), alias); +#endif + const auto alias_reads = reads; + (void)overlap.read(alias, read); (void)overlap.read(alias, read); + check(reads == alias_reads + 2 && !overlap.contains(alias.string()), "source alias cannot bypass memo exclusion"); + nilbuild::digest_phase overlap_verification; + overlap_verification.inherit_exclusions(overlap); + const auto verification_reads = reads; + (void)overlap_verification.read(alias, read); (void)overlap_verification.read(alias, read); + check(reads == verification_reads + 2, "verification inherits source exclusions without preparation digests"); + const auto hard_link = directory / "hard-link"; + fs::create_hard_link(file, hard_link); + const auto hard_link_reads = reads; + (void)overlap_verification.read(hard_link, read); (void)overlap_verification.read(hard_link, read); + check(reads == hard_link_reads + 2 && !overlap_verification.contains(hard_link.string()), + "source hard-link alias cannot enter the shared memo"); +} +} // namespace +int main() { + const auto directory = nil::detail::create_private_directory(fs::temp_directory_path(), "nil-darwin-modules-"); + try { + attribution(); digests(directory); +#ifdef __APPLE__ + nil::detail::compiler_modules::files first, second; + check(nil::detail::compiler_modules::collect_darwin(first), "actual Darwin module attribution completes"); + check(nil::detail::compiler_modules::collect_darwin(second) && first == second, "actual Darwin module inventory is stable"); + check(std::any_of(first.begin(), first.end(), [](const auto &path) { return path.find("dyld_shared_cache") != std::string::npos; }), "actual backing files are ordinary digest inputs"); +#endif + fs::remove_all(directory); + std::cout << "Darwin module identity and digest phases: " << checks << " checks passed\n"; + return 0; + } catch (const std::exception &error) { + std::error_code ignored; fs::remove_all(directory, ignored); + std::cerr << error.what() << '\n'; + return 1; + } +} diff --git a/tests/darwin_native_heap_budget.c b/tests/darwin_native_heap_budget.c index 27f18134..75b5b22a 100644 --- a/tests/darwin_native_heap_budget.c +++ b/tests/darwin_native_heap_budget.c @@ -14,7 +14,13 @@ #if defined(__APPLE__) #include -#include +/* The public SDK omits dyld's convenience macro header. The loader consumes + * pointer pairs in this documented Mach-O section; keep the declaration here. */ +#define NIL_HEAP_INTERPOSE(replacement, original) \ + __attribute__((used, section("__DATA,__interpose,interposing"))) \ + static const struct { const void *new_function, *old_function; } \ + nil_heap_pair_##replacement = { (const void *)(uintptr_t)&replacement, \ + (const void *)(uintptr_t)&original }; #include #define ORIGINAL(name) name #elif defined(NIL_HEAP_BUDGET_PORTABLE_TEST) @@ -224,13 +230,13 @@ static void *budget_valloc(size_t bytes) { } #if defined(__APPLE__) -DYLD_INTERPOSE(budget_malloc, malloc) -DYLD_INTERPOSE(budget_calloc, calloc) -DYLD_INTERPOSE(budget_free, free) -DYLD_INTERPOSE(budget_realloc, realloc) -DYLD_INTERPOSE(budget_aligned_alloc, aligned_alloc) -DYLD_INTERPOSE(budget_posix_memalign, posix_memalign) -DYLD_INTERPOSE(budget_valloc, valloc) +NIL_HEAP_INTERPOSE(budget_malloc, malloc) +NIL_HEAP_INTERPOSE(budget_calloc, calloc) +NIL_HEAP_INTERPOSE(budget_free, free) +NIL_HEAP_INTERPOSE(budget_realloc, realloc) +NIL_HEAP_INTERPOSE(budget_aligned_alloc, aligned_alloc) +NIL_HEAP_INTERPOSE(budget_posix_memalign, posix_memalign) +NIL_HEAP_INTERPOSE(budget_valloc, valloc) #if __MAC_OS_X_VERSION_MAX_ALLOWED >= 140000 /* Modern Apple Clang can rewrite ordinary calls to typed allocation symbols. @@ -263,13 +269,13 @@ static int budget_type_posix_memalign(void **out, size_t alignment, size_t bytes static void *budget_type_valloc(size_t bytes, malloc_type_id_t type) { reserve(bytes, 1); return finish_allocate(malloc_type_valloc(bytes, type), bytes); } -DYLD_INTERPOSE(budget_type_malloc, malloc_type_malloc) -DYLD_INTERPOSE(budget_type_calloc, malloc_type_calloc) -DYLD_INTERPOSE(budget_type_free, malloc_type_free) -DYLD_INTERPOSE(budget_type_realloc, malloc_type_realloc) -DYLD_INTERPOSE(budget_type_aligned_alloc, malloc_type_aligned_alloc) -DYLD_INTERPOSE(budget_type_posix_memalign, malloc_type_posix_memalign) -DYLD_INTERPOSE(budget_type_valloc, malloc_type_valloc) +NIL_HEAP_INTERPOSE(budget_type_malloc, malloc_type_malloc) +NIL_HEAP_INTERPOSE(budget_type_calloc, malloc_type_calloc) +NIL_HEAP_INTERPOSE(budget_type_free, malloc_type_free) +NIL_HEAP_INTERPOSE(budget_type_realloc, malloc_type_realloc) +NIL_HEAP_INTERPOSE(budget_type_aligned_alloc, malloc_type_aligned_alloc) +NIL_HEAP_INTERPOSE(budget_type_posix_memalign, malloc_type_posix_memalign) +NIL_HEAP_INTERPOSE(budget_type_valloc, malloc_type_valloc) #endif #else void *__wrap_malloc(size_t bytes) { return budget_malloc(bytes); } diff --git a/tests/tools/analysis_dependencies.py b/tests/tools/analysis_dependencies.py index f3b7a615..04091b9d 100644 --- a/tests/tools/analysis_dependencies.py +++ b/tests/tools/analysis_dependencies.py @@ -83,12 +83,10 @@ def consumed(report, expected): assert all(Path(path).is_absolute() and Path(path).is_file() for path in compiler_files) if sys.platform.startswith('linux'): assert str(compiler) in compiler_files, compiler_files - if sys.platform.startswith('linux') or os.name == 'nt': + if sys.platform.startswith('linux') or os.name == 'nt' or sys.platform == 'darwin': assert report['compilerFilesComplete'], report - elif sys.platform == 'darwin': - # Source certification remains available while loaded-image/shared - # cache identity is deliberately incomplete on this host. - assert not report['compilerFilesComplete'] and report['dependenciesComplete'], report + if sys.platform == 'darwin': + assert any('dyld_shared_cache' in path for path in compiler_files), compiler_files # Preserving the timestamp cannot conceal an edit; the digest describes bytes. previous = nested.stat() diff --git a/tests/tools/incremental_build_contract.py b/tests/tools/incremental_build_contract.py index e0b1010c..dc0bf716 100644 --- a/tests/tools/incremental_build_contract.py +++ b/tests/tools/incremental_build_contract.py @@ -191,6 +191,13 @@ def event(kind): module.write_bytes(next_contents.read_bytes()) os.utime(module, ns=(previous.st_atime_ns,previous.st_mtime_ns)) next_contents.unlink() +next_cache=entry.parent/'cache-content-after' +if next_cache.is_file(): + module=entry.parent/'dyld_shared_cache_arm64e.1' + previous=module.stat() + module.write_bytes(next_cache.read_bytes()) + os.utime(module, ns=(previous.st_atime_ns,previous.st_mtime_ns)) + next_cache.unlink() event('finish') ''', newline='\n') script.chmod(0o755) @@ -204,7 +211,11 @@ def run(*arguments, cwd=root, status=0, tool=fake): command = [str(builder), *map(str, arguments)] if arguments[0] in ('build', 'run', 'check'): command += ['--compiler', str(tool), '--sdk', str(stdlib)] - result = subprocess.run(command, cwd=cwd, text=True, capture_output=True, timeout=60) + # Actual whole-cache SHA on Intel has measured a 104 s first pass. + # Keep fake/containment checks at 60 s and bound real Darwin builds + # separately; this changes no assertion or cache admission rule. + timeout = 300 if sys.platform == 'darwin' and tool == compiler else 60 + result = subprocess.run(command, cwd=cwd, text=True, capture_output=True, timeout=timeout) checks += 1 assert result.returncode == status, (command, result.returncode, result.stdout, result.stderr) return result @@ -298,6 +309,24 @@ def create(directory, name, references=()): assert artifact.read_bytes()==original and state.read_bytes()==old_state run('build','--project',project) assert artifact.read_bytes()!=original and not state.exists() + + # A compiler may report a source or manifest as a loaded input as + # well. Shared toolchain hashing must not hide its fresh source + # check, including preserved-length/timestamp mutation. + project=create(root/'overlapping-source-module','app') + dependency=project.parent/'dependency.input' + dependency.write_bytes(b'source version 1') + (project.parent/'compiler-modules.json').write_text(json.dumps([ + str(project), str(project.parent/'main.matrix'), str(dependency)])) + run('build','--project',project) + artifact=project.parent/'build/app.nil';state=artifact.parent/'.nilbuild/app.nil.nildata' + original,old_state=artifact.read_bytes(),state.read_bytes() + stamp=dependency.stat() + (project.parent/'dependency-content-after').write_bytes(b'source version 2') + rejected=run('build','--project',project,'--rebuild',status=2) + assert 'build input changed during the build' in rejected.stderr and str(dependency) in rejected.stderr,rejected.stderr + assert artifact.read_bytes()==original and state.read_bytes()==old_state + assert dependency.stat().st_mtime_ns==stamp.st_mtime_ns print(f'nilbuild compiler identity: {checks} explicit certification, uncached build and source protection checks passed') return @@ -505,8 +534,32 @@ def create(directory, name, references=()): assert content_artifact.read_bytes()!=old_artifact assert 'up to date:' in run('build','--project',content_project).stderr - # Positive parity remains required here, including on Darwin where - # incomplete loaded-image discovery deliberately prevents reuse. + # Whole shared-cache files use the same ordinary content identities. + # Changing a tail byte with the header/length/timestamp unchanged must + # invalidate a hit and reject publication if it occurs during compile. + cache_project=create(root/'cache-content','cache-content') + cache_dir=cache_project.parent + backing=cache_dir/'dyld_shared_cache_arm64e.1' + cache_bytes=b'unchanged cache header and UUID' + bytes(1024 * 1024) + b'1' + backing.write_bytes(cache_bytes) + (cache_dir/'compiler-modules.json').write_text(json.dumps([str(backing)])) + run('build','--project',cache_project) + assert 'up to date:' in run('build','--project',cache_project).stderr + stamp=backing.stat() + backing.write_bytes(cache_bytes[:-1]+b'2') + os.utime(backing,ns=(stamp.st_atime_ns,stamp.st_mtime_ns)) + assert 'up to date:' not in run('build','--project',cache_project).stderr + cache_artifact=cache_dir/'build/cache-content.nil' + cache_state=cache_dir/'build/.nilbuild/cache-content.nil.nildata' + original_cache_artifact,original_cache_state=cache_artifact.read_bytes(),cache_state.read_bytes() + (cache_dir/'main.matrix').write_text('version two\n',newline='\n') + (cache_dir/'cache-content-after').write_bytes(cache_bytes[:-1]+b'3') + changed=run('build','--project',cache_project,'--rebuild',status=2) + assert 'build input changed during the build' in changed.stderr and str(backing) in changed.stderr,changed.stderr + assert cache_artifact.read_bytes()==original_cache_artifact and cache_state.read_bytes()==original_cache_state + assert backing.stat().st_mtime_ns==stamp.st_mtime_ns and backing.read_bytes()==cache_bytes[:-1]+b'3' + + # Positive reuse is required on every supported host. # Containment is covered independently by --compiler-identity-only. # Real compiler smoke: unchanged imports skip compilation for both routes. actual=root/'real';actual.mkdir() From f79b94b30008d81b54f75248acca5d294754cbbb Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 18:30:38 -0500 Subject: [PATCH 09/39] Qualify Darwin toolchain inputs and fix MSVC freestanding hooks --- .../workflows/macos-application-parity.yml | 87 ++- CMakeLists.txt | 20 + docs/platform-support.md | 15 +- nilbuild/darwin_toolchain_inputs.hpp | 497 ++++++++++++++++++ nilbuild/toolchain_inputs.hpp | 56 ++ .../freestanding_runtime.cpp | 12 + notebook/2026-11-11-web-runtime-parity.md | 30 ++ scripts/ci/macos-toolchain-census.py | 4 + tests/darwin_native_heap_budget.c | 34 +- tests/darwin_toolchain_inputs_tests.cpp | 213 ++++++++ tests/freestanding_hook_contract.cpp | 23 + tests/tools/darwin_native_heap_budget.py | 31 +- vscode/test/run-host.cjs | 8 +- 13 files changed, 997 insertions(+), 33 deletions(-) create mode 100644 nilbuild/darwin_toolchain_inputs.hpp create mode 100644 tests/darwin_toolchain_inputs_tests.cpp create mode 100644 tests/freestanding_hook_contract.cpp diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index a2d564a7..af2e51c7 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -69,6 +69,7 @@ jobs: cache: npm cache-dependency-path: vscode/package-lock.json - name: Configure LLVM applications + id: configure timeout-minutes: 8 run: | mkdir -p build/application-evidence @@ -83,23 +84,45 @@ jobs: "-DLLVM_DIR=$nil_llvm_prefix/lib/cmake/llvm" \ -DNILCOMPILER_SQLITE_INCLUDE_DIR=OFF -DNILCOMPILER_POSTGRESQL_INCLUDE_DIR=OFF \ 2>&1 | tee build/application-evidence/configure.log + - name: Build independent allocation controls + id: heap + timeout-minutes: 3 + run: | + set -o pipefail + cmake --build build/application --parallel 2 --target \ + nil_darwin_native_heap_budget nil_darwin_native_heap_budget_control \ + 2>&1 | tee build/application-evidence/heap-build.log + - name: Qualify allocator injection before building applications + if: ${{ !cancelled() && steps.heap.outcome == 'success' }} + timeout-minutes: 4 + run: | + set -o pipefail + python3 -B tests/tools/darwin_native_heap_budget.py --controls-only \ + --heap-budget-library build/application/lib/libnil_darwin_native_heap_budget.dylib \ + --heap-budget-control build/application/bin/nil_darwin_native_heap_budget_control \ + --output-parent build/application/test-evidence \ + 2>&1 | tee build/application-evidence/heap-controls.log + - name: Preserve early allocation evidence + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: macos-heap-controls-${{ matrix.host.arch }}-${{ github.sha }}-${{ github.run_attempt }} + path: | + build/application/test-evidence/darwin-heap-controls-*/ + build/application-evidence/heap-*.log + if-no-files-found: warn + retention-days: 14 - name: Build application tools id: tools + if: ${{ !cancelled() && steps.configure.outcome == 'success' }} timeout-minutes: 40 run: | set -o pipefail cmake --build build/application --parallel 2 --target \ nilcompiler nilinterpreter nilassembler nillsp nildebugger nilbuild niltest aperture \ nil_vscode_host_runner nil_file_time_support_tests nil_compiler_module_darwin_tests \ + nil_darwin_toolchain_inputs_tests nil_freestanding_hook_default nil_freestanding_hook_override \ 2>&1 | tee build/application-evidence/build.log - - name: Build independent allocation controls - id: heap - timeout-minutes: 3 - run: | - set -o pipefail - cmake --build build/application --parallel 2 --target \ - nil_darwin_native_heap_budget nil_darwin_native_heap_budget_control \ - 2>&1 | tee build/application-evidence/heap-build.log - name: Record selected native SDK and linker inputs if: ${{ !cancelled() && steps.tools.outcome == 'success' }} timeout-minutes: 2 @@ -107,14 +130,16 @@ jobs: python3 -B scripts/ci/macos-toolchain-census.py --build build/application --output build/application-evidence/toolchain-census - name: Verify actual native lifetime and timestamp contracts - timeout-minutes: 15 + if: ${{ !cancelled() && steps.tools.outcome == 'success' && steps.heap.outcome == 'success' }} + timeout-minutes: 16 run: | python3 - <<'PYTHON' import json, subprocess, xml.etree.ElementTree as ET from pathlib import Path expected = {'compiler.darwin-module-fingerprint', 'ny.native-managed-lifetime', 'ny.native-owned-lifetime', - 'nil.file-time-support', 'nil.file-modified-at-contract'} - pattern = r'^(compiler[.]darwin-module-fingerprint|ny[.]native-(managed|owned)-lifetime|nil[.]file-(time-support|modified-at-contract))$' + 'nil.file-time-support', 'nil.file-modified-at-contract', + 'nil.freestanding-hook-default', 'nil.freestanding-hook-override'} + pattern = r'^(compiler[.]darwin-module-fingerprint|ny[.]native-(managed|owned)-lifetime|nil[.]file-(time-support|modified-at-contract)|nil[.]freestanding-hook-(default|override))$' base = ['ctest', '--test-dir', 'build/application', '-R', pattern] root = Path('build/application-evidence').resolve() inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) @@ -123,7 +148,7 @@ jobs: with (root/'ctest.log').open('wb') as output: run = subprocess.run(base+['--no-tests=error', '--output-on-failure', '--output-junit', str(root/'results.xml')], stdout=output, - stderr=subprocess.STDOUT, timeout=850) + stderr=subprocess.STDOUT, timeout=910) print((root/'ctest.log').read_text(errors='replace')) run.check_returncode() cases = list(ET.parse(root/'results.xml').getroot().iter('testcase')) @@ -131,6 +156,31 @@ jobs: assert all(case.get('status') == 'run' and all(case.find(kind) is None for kind in ['failure','error','skipped']) for case in cases) PYTHON + - name: Verify default native toolchain and real incremental reuse + if: ${{ !cancelled() && steps.tools.outcome == 'success' }} + timeout-minutes: 20 + run: | + python3 - <<'PYTHON' + import json, subprocess, xml.etree.ElementTree as ET + from pathlib import Path + expected = {'nil.darwin-toolchain-inputs', 'nilbuild.incremental-contract'} + pattern = r'^(nil[.]darwin-toolchain-inputs|nilbuild[.]incremental-contract)$' + base = ['ctest', '--test-dir', 'build/application', '-R', pattern] + root = Path('build/application-evidence').resolve() + inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) + (root/'incremental-inventory.json').write_bytes(inventory) + assert {case['name'] for case in json.loads(inventory)['tests']} == expected + with (root/'incremental.log').open('wb') as output: + run = subprocess.run(base+['--no-tests=error', '--output-on-failure', + '--output-junit', str(root/'incremental-results.xml')], stdout=output, + stderr=subprocess.STDOUT, timeout=1100) + print((root/'incremental.log').read_text(errors='replace')) + run.check_returncode() + cases = list(ET.parse(root/'incremental-results.xml').getroot().iter('testcase')) + assert len(cases) == len(expected) and {case.get('name') for case in cases} == expected + assert all(case.get('status') == 'run' and all(case.find(kind) is None + for kind in ['failure','error','skipped']) for case in cases) + PYTHON - name: Prepare pinned VS Code in this runner's private build directory id: editor if: ${{ !cancelled() && steps.tools.outcome == 'success' }} @@ -138,7 +188,7 @@ jobs: run: | # Version 1.146.0; architecture-specific SHA256 values from Microsoft's # update API on 2026-11-11. No auto-update or system installation. - mkdir -p build/editor-download build/editor-tmp + mkdir -p build/editor-download curl --fail --location --max-time 180 --max-filesize 800000000 \ "https://vscode.download.prss.microsoft.com/dbazure/download/stable/69047191fc427f2f92828279aae1d90d8eeda7ab/$NIL_EDITOR_ARCHIVE" \ -o build/editor-download/editor.zip @@ -187,7 +237,6 @@ jobs: export NIL_VSCODE_TOOL_DIRECTORY="$GITHUB_WORKSPACE/build/application/bin" export NIL_VSCODE_HOST_RUNNER="$NIL_VSCODE_TOOL_DIRECTORY/nil_vscode_host_runner" export NIL_VSCODE_SDK_PATH="$GITHUB_WORKSPACE" - export TMPDIR="$GITHUB_WORKSPACE/build/editor-tmp" for nil_subset in NIL_VSCODE_NILDATA_ONLY NIL_VSCODE_LANGUAGE_VALUES_ONLY \ NIL_VSCODE_MEMBER_SIGNATURE_ONLY NIL_VSCODE_COMPLETION_ONLY NIL_VSCODE_SEMANTIC_ONLY \ NIL_VSCODE_WORKSPACE_ONLY NIL_VSCODE_ANALYSIS_ONLY NIL_VSCODE_ANALYSIS_PROBE_ONLY; do @@ -213,6 +262,15 @@ jobs: for source, name in [('build/application/test-evidence','contracts'), ('build/vscode-host-smoke','editor-reports')]: if Path(source).is_dir(): shutil.copytree(source, root/name) + for launch in Path('build/vscode-host-smoke').glob('*-launch.json'): + record = json.loads(launch.read_text()) + temporary = Path(record['temporary']) + assert record['platform'] == 'darwin' and temporary.parent == Path('/tmp') + assert temporary.name.startswith('nil vscode host café 中文-') + profile = temporary/'profile' + assert str(profile) == record['profile'] and not profile.is_symlink() + logs = profile/'logs' + if logs.is_dir(): shutil.copytree(logs, root/'editor-logs'/record['runId']) dirty = subprocess.check_output(['git','status','--porcelain','--untracked-files=no'], text=True, timeout=15) revision = subprocess.check_output(['git','rev-parse','HEAD'], text=True, timeout=15).strip() files = {} @@ -236,6 +294,5 @@ jobs: build/application-evidence/ build/application/Testing/Temporary/LastTest.log build/application/CMakeCache.txt - build/editor-tmp/**/profile/logs/** if-no-files-found: error retention-days: 14 diff --git a/CMakeLists.txt b/CMakeLists.txt index 8fd4785c..3c6543d4 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -101,6 +101,19 @@ install( install(FILES LICENSE DESTINATION ".") if(BUILD_TESTING) + foreach(nil_hook_profile IN ITEMS default override) + add_executable(nil_freestanding_hook_${nil_hook_profile} tests/freestanding_hook_contract.cpp) + target_compile_features(nil_freestanding_hook_${nil_hook_profile} PRIVATE cxx_std_17) + target_link_libraries(nil_freestanding_hook_${nil_hook_profile} PRIVATE nilruntime_freestanding) + if(nil_hook_profile STREQUAL "override") + target_compile_definitions(nil_freestanding_hook_${nil_hook_profile} PRIVATE NIL_TEST_HOST_OVERRIDE=1) + endif() + if(UNIX AND NOT APPLE) + target_link_options(nil_freestanding_hook_${nil_hook_profile} PRIVATE -no-pie) + endif() + add_test(NAME nil.freestanding-hook-${nil_hook_profile} COMMAND $) + set_tests_properties(nil.freestanding-hook-${nil_hook_profile} PROPERTIES LABELS "runtime;rc" TIMEOUT 15) + endforeach() if(WIN32) # Keep the public backend declarations usable after real Windows RPC # headers, even in an LLVM-OFF build that normally hides this interface. @@ -128,6 +141,13 @@ if(BUILD_TESTING) if(WIN32) target_link_libraries(nil_compiler_module_darwin_tests PRIVATE advapi32) endif() + add_executable(nil_darwin_toolchain_inputs_tests tests/darwin_toolchain_inputs_tests.cpp) + target_compile_features(nil_darwin_toolchain_inputs_tests PRIVATE cxx_std_17) + if(WIN32) + target_link_libraries(nil_darwin_toolchain_inputs_tests PRIVATE advapi32) + endif() + add_test(NAME nil.darwin-toolchain-inputs COMMAND $) + set_tests_properties(nil.darwin-toolchain-inputs PROPERTIES LABELS "tools;project;rc" TIMEOUT 60) # Test-only launcher for isolated editor integration runs. Reuse the # toolchain's bounded process/Job cleanup; never ship it in the SDK. diff --git a/docs/platform-support.md b/docs/platform-support.md index 49bb244b..b00bf9f7 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -149,14 +149,13 @@ the complete admission rules. ## Remaining macOS work -- Native incremental dependency discovery and reuse remain incomplete. Darwin - compiler loaded-library identity remains incomplete in production. A separate - native diagnostic now matches all observed loaded images to ordinary files or - exact shared-cache mappings on both architectures and hashes every cache file - twice. Reading roughly 5 GB took 15.057 then 14.056 seconds on ARM, and 104.462 - then 15.351 seconds on Intel. This is not yet a practical cache-key integration; - native toolchain dependency discovery also remains open. Both VM and native - builds deliberately rebuild without complete identity. +- Native incremental reuse is still being qualified. The compiler now resolves + loaded images to ordinary files and exact shared-cache backing files for full + content hashing; its actual Apple Silicon collector test passes at `2632bc6`. + The builder uses separate preparation and verification digest phases without + caching source/manifest reads. Native linker dependency collection and real + VM/native reuse still require both hosted architectures to pass. Incomplete + or unsupported dependency profiles deliberately rebuild. - A separate [development SDK workflow](../.github/workflows/macos-sdk.yml) now implements native Darwin consumer admission, root install layout, pinned documentation tooling, and staged dependency/relocation checks. Both diff --git a/nilbuild/darwin_toolchain_inputs.hpp b/nilbuild/darwin_toolchain_inputs.hpp new file mode 100644 index 00000000..27ceaf14 --- /dev/null +++ b/nilbuild/darwin_toolchain_inputs.hpp @@ -0,0 +1,497 @@ +#pragma once + +// Conservative closure for nilbuild's ordinary Apple-clang native invocation. +// This deliberately inventories a superset of SDK linkable inputs; unsupported +// drivers, search profiles, stub formats or escaping dependencies disable reuse. +// Every returned file is content-hashed by nilbuild, and the complete namespace +// is collected again before publication. No timestamps certify cache identity. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace nilbuild::darwin_toolchain_detail { +namespace fs = std::filesystem; + +inline std::string trim(const std::string &text) { + const auto first = text.find_first_not_of(" \r\n\t"); + return first == std::string::npos ? "" : text.substr(first, text.find_last_not_of(" \r\n\t") - first + 1); +} +inline bool regular(const fs::path &path) { + std::error_code error; + return fs::is_regular_file(path, error) && !error; +} +inline bool inside(const fs::path &path, const fs::path &root) { + const auto value = path.lexically_normal(), base = root.lexically_normal(); + auto at = value.begin(); + for (auto part = base.begin(); part != base.end(); ++part, ++at) + if (at == value.end() || *at != *part) return false; + return true; +} +inline bool clean_path(const fs::path &path) { + const auto text = path.string(); + return path.is_absolute() && text.find_first_of("\r\n\0", 0, 3) == std::string::npos && + path == path.lexically_normal(); +} + +// clang -### quotes every command argument. Interpret only its double-quoted +// command syntax; never execute text through a shell or accept response files. +inline std::optional> quoted_arguments(const std::string &line) { + std::vector result; + std::size_t at = 0; + while (at < line.size()) { + while (at < line.size() && std::isspace(static_cast(line[at]))) ++at; + if (at == line.size()) break; + if (line[at++] != '"') return {}; + std::string value; + bool closed = false; + while (at < line.size()) { + char byte = line[at++]; + if (byte == '"') { closed = true; break; } + if (byte == '\\') { + if (at == line.size() || (line[at] != '\\' && line[at] != '"')) return {}; + byte = line[at++]; + } + if (byte == '\0' || byte == '\n' || byte == '\r') return {}; + value += byte; + } + if (!closed || value.empty()) return {}; + result.push_back(std::move(value)); + if (at < line.size() && !std::isspace(static_cast(line[at]))) return {}; + } + return result.empty() ? std::nullopt : std::optional>(std::move(result)); +} + +struct link_plan { + fs::path installed, linker, sdk, lto, compiler_rt; + std::string architecture; +}; +inline std::optional parse_plan(const std::string &text, const fs::path &runtime) { + if (text.size() > 1024 * 1024 || text.find('\0') != std::string::npos) return {}; + link_plan plan; + std::vector command; + std::istringstream lines(text); + std::string line; + bool apple = false; + while (std::getline(lines, line)) { + line = trim(line); + if (line.empty()) continue; + if (line.rfind("Apple clang version ", 0) == 0 && !apple) { apple = true; continue; } + if (line.rfind("Target: ", 0) == 0 || line == "Thread model: posix") continue; + if (line.rfind("InstalledDir: ", 0) == 0 && plan.installed.empty()) { + plan.installed = line.substr(14); continue; + } + if (line.front() == '"' && command.empty()) { + const auto parsed = quoted_arguments(line); + if (!parsed) return {}; + command = *parsed; continue; + } + // Includes default configuration-file diagnostics and extra compiler jobs. + return {}; + } + if (!apple || !clean_path(plan.installed) || command.empty() || + plan.installed.filename() != "bin" || plan.installed.parent_path().filename() != "usr") return {}; + const auto installation = plan.installed.parent_path().parent_path().filename(); + if (installation != "XcodeDefault.xctoolchain" && installation != "CommandLineTools") return {}; + plan.linker = command.front(); + if (plan.linker != plan.installed / "ld") return {}; + bool dynamic = false, platform = false, output = false, input = false, cxx = false, system = false; + std::set frameworks; + for (std::size_t index = 1; index < command.size(); ++index) { + const auto &option = command[index]; + const auto next = [&]() -> std::optional { + if (index >= command.size() - 1) { index = command.size(); return {}; } + return command[++index]; + }; + if (option == "-demangle" || option == "-no_deduplicate" || option == "-adhoc_codesign") continue; + if (option == "-dynamic" && !dynamic) { dynamic = true; continue; } + if (option == "-lc++" && !cxx) { cxx = true; continue; } + if (option == "-lSystem" && !system) { system = true; continue; } + if (option == "-arch" && plan.architecture.empty()) { + const auto value = next(); if (!value || (*value != "arm64" && *value != "x86_64")) return {}; + plan.architecture = *value; continue; + } + if (option == "-platform_version" && !platform) { + const auto name = next(), minimum = next(), sdk_version = next(); + if (!name || *name != "macos" || !minimum || !sdk_version || minimum->empty() || sdk_version->empty() || + minimum->find_first_not_of("0123456789.") != std::string::npos || + sdk_version->find_first_not_of("0123456789.") != std::string::npos) return {}; + platform = true; continue; + } + if (option == "-syslibroot" && plan.sdk.empty()) { + const auto value = next(); if (!value || !clean_path(*value)) return {}; + plan.sdk = *value; continue; + } + if (option == "-lto_library" && plan.lto.empty()) { + const auto value = next(); if (!value || !clean_path(*value)) return {}; + plan.lto = *value; continue; + } + if (option == "-mllvm") { + const auto value = next(); if (!value || *value != "-enable-linkonceodr-outlining") return {}; + continue; + } + if (option == "-o" && !output) { + const auto value = next(); if (!value || *value != "/dev/null") return {}; + output = true; continue; + } + if (option == "-framework") { + const auto value = next(); + if (!value || (*value != "Cocoa" && *value != "CoreGraphics") || !frameworks.insert(*value).second) return {}; + continue; + } + if (option == runtime.string() && !input) { input = true; continue; } + const fs::path path(option); + if (plan.compiler_rt.empty() && clean_path(path) && path.filename() == "libclang_rt.osx.a" && + inside(path, plan.installed.parent_path() / "lib" / "clang")) { + plan.compiler_rt = path; continue; + } + return {}; // Includes custom -L/-F, -force_load, plugins, and response files. + } + if (!dynamic || !platform || !output || !input || !cxx || !system || frameworks.size() != 2 || + plan.architecture.empty() || plan.sdk.empty() || plan.lto.empty() || plan.compiler_rt.empty() || + plan.lto != plan.installed.parent_path() / "lib" / "libLTO.dylib" || + plan.sdk.filename().string().rfind("MacOSX", 0) != 0 || plan.sdk.extension() != ".sdk") return {}; + return plan; +} + +inline bool system_install_name(const std::string &name) { + const fs::path path(name); + // Install names are always POSIX paths, including in portable parser tests. + return !name.empty() && name.front() == '/' && name.find_first_of("\\\r\n\0 \t", 0, 6) == std::string::npos && + path == path.lexically_normal() && (inside(path, "/usr/lib") || inside(path, "/System/Library/Frameworks") || + inside(path, "/System/Library/PrivateFrameworks")); +} + +// A conservative lexical admission check for Apple's generated YAML TBD v4. +// Re-export paths must be explicit system install names. Escapes, aliases, custom tags, +// relative re-exports and rpath names require a richer parser and disable reuse. +// Inventorying every SDK stub covers all architecture/OS re-export choices. +inline bool supported_stub(const std::string &text, std::set *references = nullptr) { + if (text.size() > 32 * 1024 * 1024 || text.find('\0') != std::string::npos || + text.find('\\') != std::string::npos || text.rfind("--- !tapi-tbd", 0) != 0) return false; + std::istringstream lines(text); std::string line; + bool version = false, install = false, in_libraries = false; + std::string libraries; + auto scalar = [](std::string value) { + value = trim(value); + if (value.size() >= 2 && (value.front() == '\'' || value.front() == '"') && value.back() == value.front()) + value = value.substr(1, value.size() - 2); + return value; + }; + const auto admit_name = [&](const std::string &name) { + if (!system_install_name(name)) return false; + const fs::path path(name); + if (!path.extension().empty() && path.extension() != ".dylib" && path.extension() != ".tbd") return false; + for (const auto &part : path) + if (part == "Headers" || part == "PrivateHeaders" || part == "Modules") return false; + if (references) { references->insert(name); if (references->size() > 65536) return false; } + return true; + }; + while (std::getline(lines, line)) { + line = trim(line); + if (line.empty() || line.front() == '#') continue; + if (line.rfind("---", 0) == 0) { if (line != "--- !tapi-tbd") return false; continue; } + if (line.rfind("tbd-version:", 0) == 0) { + if (trim(line.substr(12)) != "4") return false; + version = true; continue; + } + if (line.rfind("install-name:", 0) == 0) { + // LC_ID_DYLIB and $ld$install_name export strings describe output/interface + // identity; they do not name additional files read by the linker. + // Generated SDKs contain @rpath and historical AppleInternal IDs here. + if (scalar(line.substr(13)).empty()) return false; + install = true; + } + const auto split = line.find(':'); + const auto key = split == std::string::npos ? std::string{} : trim(line.substr(0, split)); + if (key == "reexported-libraries" && !trim(line.substr(split + 1)).empty()) return false; + if (!line.empty() && (line.front() == '"' || line.front() == '\'')) { + const auto end_quote = line.find(line.front(), 1); + if (end_quote != std::string::npos && trim(line.substr(end_quote + 1)).rfind(':', 0) == 0) return false; + } + if (key == "libraries" || key == "library") { + if (in_libraries) return false; + libraries = trim(line.substr(split + 1)); + if (libraries.empty() || libraries.front() != '[') return false; + in_libraries = true; + } else if (in_libraries) libraries += ' ' + line; + if (in_libraries && libraries.find(']') != std::string::npos) { + if (libraries.back() != ']') return false; + std::istringstream values(libraries.substr(1, libraries.size() - 2)); std::string value; + while (std::getline(values, value, ',')) if (!admit_name(scalar(value))) return false; + in_libraries = false; + } + // Reject YAML indirection outside quoted strings, so it cannot manufacture + // an unobserved re-export value from an anchor or tagged scalar. + bool quoted = false; char quote = 0; + for (std::size_t at = 0; at < line.size(); ++at) { + const char byte = line[at]; + if ((byte == '\'' || byte == '"') && (!quoted || quote == byte)) { + quoted = !quoted; quote = byte; continue; + } + if (!quoted && (byte == '&' || byte == '*' || byte == '!' || byte == '|' || byte == '>' || + byte == '{' || byte == '}')) return false; + } + } + return version && install && !in_libraries; +} + +struct image_description { + std::vector libraries, rpaths; + bool system_loader = false; +}; +inline std::optional parse_image(const fs::path &path, + const std::string &listing, + const std::string &commands) { + if (listing.size() > 1024 * 1024 || commands.size() > 4 * 1024 * 1024 || + listing.find('\0') != std::string::npos || commands.find('\0') != std::string::npos || + commands.find("LC_DYLD_ENVIRONMENT") != std::string::npos) return {}; + image_description result; + std::istringstream lines(listing); std::string line; bool heading = false; + while (std::getline(lines, line)) { + if (line.empty()) continue; + if (line.front() != '\t' && line.front() != ' ') { + if (line.rfind(path.string(), 0) != 0 || line.back() != ':') return {}; + heading = true; continue; + } + line = trim(line); const auto marker = line.find(" (compatibility version "); + if (marker == std::string::npos) return {}; + result.libraries.push_back(line.substr(0, marker)); + } + if (!heading) return {}; + std::istringstream load_commands(commands); bool rpath = false, loader = false; + while (std::getline(load_commands, line)) { + line = trim(line); + if (line == "cmd LC_RPATH") { if (rpath || loader) return {}; rpath = true; } + else if (line == "cmd LC_LOAD_DYLINKER") { if (rpath || loader) return {}; loader = true; } + else if (rpath && line.rfind("path ", 0) == 0) { + const auto marker = line.rfind(" (offset "); + if (marker == std::string::npos || marker <= 5) return {}; + result.rpaths.push_back(line.substr(5, marker - 5)); rpath = false; + } else if (loader && line.rfind("name ", 0) == 0) { + const auto marker = line.rfind(" (offset "); + if (marker == std::string::npos || line.substr(5, marker - 5) != "/usr/lib/dyld") return {}; + result.system_loader = true; loader = false; + } else if ((rpath || loader) && line.rfind("cmdsize ", 0) != 0) return {}; + } + return rpath || loader ? std::nullopt : std::optional(std::move(result)); +} + +class collector { +public: + using probe = std::function(const fs::path &)>; +private: + fs::path driver, cwd, runtime, host; + link_plan plan; + probe dependencies; + std::set files_seen, active_directories; + std::set stub_references; + std::set inspected_contexts; + std::map image_descriptions; + std::size_t entries = 0; + + fs::path host_path(const fs::path &path) const { return host / path.relative_path(); } + bool add(const fs::path &path) { + if (!clean_path(path) || !regular(path) || files_seen.size() >= 65536) return false; + if (path.extension() == ".a") { + std::ifstream archive(path, std::ios::binary); char magic[8]{}; + archive.read(magic, sizeof(magic)); + if (archive.gcount() == 8 && std::string(magic, sizeof(magic)) == "!\n") return false; + } + files_seen.insert(path); return true; + } + bool read_stub(const fs::path &path) { + std::ifstream stream(path, std::ios::binary); std::string text; char block[8192]; + while (stream.read(block, sizeof(block)) || stream.gcount()) { + text.append(block, static_cast(stream.gcount())); + if (text.size() > 32 * 1024 * 1024) return false; + } + return !stream.bad() && supported_stub(text, &stub_references); + } + bool tree(const fs::path &path, bool sdk, unsigned depth = 0) { + std::error_code error; + if (depth > 32 || ++entries > 1000000) return false; + const auto canonical = fs::canonical(path, error); + if (error || !active_directories.insert(canonical).second) return false; + fs::directory_iterator iterator(path, error), end; + if (error) return false; + for (; iterator != end; iterator.increment(error)) { + if (error || ++entries > 1000000) return false; + const auto entry = iterator->path(); + const auto status = fs::status(entry, error); + if (error) return false; + if (fs::is_directory(status)) { + // Header/resource trees cannot be selected as linked library files. + const auto name = entry.filename(); + if (name == "Headers" || name == "PrivateHeaders" || name == "Modules") continue; + if (!tree(entry, sdk, depth + 1)) return false; + } else if (fs::is_regular_file(status)) { + const auto extension = entry.extension(); + if (extension == ".tbd" || extension == ".dylib" || extension == ".so" || extension == ".a" || extension == ".o" || extension.empty()) { + if (sdk && extension.empty()) { + std::ifstream stream(entry, std::ios::binary); char prefix[16]{}; + stream.read(prefix, sizeof(prefix)); + const auto size = static_cast(stream.gcount()); + if (stream.bad()) return false; + const std::string bytes(prefix, size); + if (bytes.rfind("--- !tapi-tbd", 0) == 0) { + if (!add(entry) || !read_stub(entry)) return false; + } else if (bytes.rfind("!\n", 0) == 0 || bytes.rfind("!\n", 0) == 0 || + (size >= 4 && (bytes.compare(0, 4, "\xfe\xed\xfa\xce", 4) == 0 || bytes.compare(0, 4, "\xce\xfa\xed\xfe", 4) == 0 || + bytes.compare(0, 4, "\xfe\xed\xfa\xcf", 4) == 0 || bytes.compare(0, 4, "\xcf\xfa\xed\xfe", 4) == 0 || + bytes.compare(0, 4, "\xca\xfe\xba\xbe", 4) == 0 || bytes.compare(0, 4, "\xbe\xba\xfe\xca", 4) == 0 || + bytes.compare(0, 4, "\xca\xfe\xba\xbf", 4) == 0 || bytes.compare(0, 4, "\xbf\xba\xfe\xca", 4) == 0))) return false; + continue; // Non-linkable metadata such as framework PkgInfo. + } + if (!add(entry) || (extension == ".tbd" && !read_stub(entry))) return false; + if (sdk && (extension == ".dylib" || extension == ".so")) return false; // binary SDK stubs need load-command analysis + } + } else return false; + } + active_directories.erase(canonical); + return !error; + } + bool image(const fs::path &path, const fs::path &executable_directory, unsigned depth = 0, + const std::vector &inherited_rpaths = {}, std::vector *effective_rpaths = nullptr) { + if (depth > 32 || !add(path)) return false; + std::error_code error; + const auto physical = fs::canonical(path, error); + if (error || !add(physical)) return false; + auto cached = image_descriptions.find(path); + if (cached == image_descriptions.end()) { + const auto description = dependencies(path); + if (!description) return false; + cached = image_descriptions.emplace(path, *description).first; + } + const auto &description = cached->second; + if (physical.parent_path() != path.parent_path()) { + const auto loader_relative = [](const std::string &name) { return name.rfind("@loader_path", 0) == 0; }; + if (std::any_of(description.rpaths.begin(), description.rpaths.end(), loader_relative) || + std::any_of(description.libraries.begin(), description.libraries.end(), loader_relative)) return false; + } + // Compiler identity separately proves the active system dyld image and all + // shared-cache backing bytes. Include an ordinary loader path as well when + // the host exposes one; an alternative LC_LOAD_DYLINKER is never admitted. + if (description.system_loader && regular(host_path("/usr/lib/dyld")) && !add(host_path("/usr/lib/dyld"))) return false; + std::vector rpaths; + for (const auto &name : description.rpaths) { + fs::path selected; + if (name.rfind("@loader_path/", 0) == 0) selected = (physical.parent_path() / name.substr(13)).lexically_normal(); + else if (name.rfind("@executable_path/", 0) == 0) selected = (executable_directory / name.substr(17)).lexically_normal(); + else selected = name; + if (!clean_path(selected) || (!inside(selected, plan.installed.parent_path()) && + !system_install_name(selected.string()))) return false; + rpaths.push_back(selected); + } + rpaths.insert(rpaths.end(), inherited_rpaths.begin(), inherited_rpaths.end()); + std::vector unique_rpaths; + for (const auto &root : rpaths) + if (std::find(unique_rpaths.begin(), unique_rpaths.end(), root) == unique_rpaths.end()) unique_rpaths.push_back(root); + rpaths = std::move(unique_rpaths); + if (effective_rpaths) *effective_rpaths = rpaths; + std::string context = path.string() + '\0' + executable_directory.string(); + for (const auto &root : rpaths) context += '\0' + root.string(); + if (!inspected_contexts.insert(context).second) return true; + if (inspected_contexts.size() > 1024) return false; + for (const auto &name : description.libraries) { + fs::path selected; + if (name.rfind("@rpath/", 0) == 0) { + const auto tail = fs::path(name.substr(7)); + if (tail.has_parent_path()) return false; + for (const auto &root : rpaths) { + const auto candidate = root / tail; + if (regular(candidate)) { selected = candidate; break; } + } + // A dylib's install ID may use its parent's rpath and appear in otool + // -L. It names this file, not an additional dependency. + if (selected.empty() && path.filename() == tail && path.extension() == ".dylib") selected = path; + if (selected.empty()) return false; + } else if (name.rfind("@loader_path/", 0) == 0) { + selected = (physical.parent_path() / name.substr(13)).lexically_normal(); + if (!inside(selected, plan.installed.parent_path())) return false; + } else if (system_install_name(name)) { + selected = host_path(name); + // Non-file system images are represented by the compiler's complete + // shared-cache backing-file superset. Ordinary overrides are included. + if (!regular(selected)) continue; + } else { + selected = name; + if (!clean_path(selected) || !inside(selected, plan.installed.parent_path())) return false; + } + if (selected != path && !image(selected, executable_directory, depth + 1, rpaths)) return false; + } + return true; + } + bool adjacent_helpers() { + const auto directory = plan.installed.parent_path() / "lib"; + std::error_code error; + fs::directory_iterator iterator(directory, error), end; + if (error) return false; + for (; iterator != end; iterator.increment(error)) { + if (error || ++entries > 1000000) return false; + if (iterator->path().extension() == ".dylib" && !add(iterator->path())) return false; + } + return !error; + } +public: + collector(fs::path selected_driver, fs::path directory, fs::path selected_runtime, + link_plan selected_plan, probe inspect_image, fs::path host_root = "/") + : driver(std::move(selected_driver)), cwd(std::move(directory)), runtime(std::move(selected_runtime)), + host(std::move(host_root)), plan(std::move(selected_plan)), dependencies(std::move(inspect_image)) {} + + std::optional> files() { + std::error_code error; + const auto actual_driver = fs::canonical(driver, error); + if (error) return {}; + const auto actual_compiler = fs::canonical(plan.installed / "clang++", error); + if (error) return {}; + const auto actual_linker = fs::canonical(plan.linker, error); + if (error) return {}; + std::vector linker_rpaths; + if (!clean_path(driver) || !clean_path(cwd) || !add(runtime) || !image(driver, actual_driver.parent_path()) || + !image(plan.installed / "clang++", actual_compiler.parent_path()) || + !image(plan.linker, actual_linker.parent_path(), 0, {}, &linker_rpaths) || + !image(plan.lto, actual_linker.parent_path(), 0, linker_rpaths) || !add(plan.compiler_rt)) return {}; + if (!add(plan.sdk / "SDKSettings.json") || !add(host_path("/System/Library/CoreServices/SystemVersion.plist"))) return {}; + if (fs::exists(plan.sdk / "AppleInternal", error) || error) return {}; + for (const auto &directory : {plan.sdk / "usr/lib", plan.sdk / "System/Library/Frameworks"}) + if (!tree(directory, true)) return {}; + if (!adjacent_helpers()) return {}; + // With exactly one syslibroot, Apple's linker omits absent standard search + // directories instead of falling back to host /usr/local or /Library. + // https://github.com/apple-oss-distributions/ld64/blob/main/src/ld/Options.cpp + // Keep optional SDK search namespaces in the fresh inventory when present. + for (const auto &relative : {"usr/local/lib", "Library/Frameworks", "System/Library/PrivateFrameworks"}) { + const auto directory = plan.sdk / relative; + const bool present = fs::exists(directory, error); + if (error || (present && !tree(directory, true))) return {}; + } + // Absolute re-export lookup differs from standard -L/-F lookup: when the + // SDK/inlined stub cannot satisfy it, ld can try the raw host pathname. + // Until ordinary host link-input closure is modeled, refuse that profile. + // Re-checking these absences catches a newly installed fallback as well. + for (const auto &name : stub_references) { + const auto relative = fs::path(name).relative_path(); + auto sdk_stub = plan.sdk / relative; sdk_stub.replace_extension(".tbd"); + if (regular(plan.sdk / relative)) { + if (!files_seen.count(plan.sdk / relative)) return {}; + continue; + } + if (regular(sdk_stub)) { + if (!files_seen.count(sdk_stub)) return {}; + continue; + } + auto host_stub = host_path(name); host_stub.replace_extension(".tbd"); + if (regular(host_path(name)) || regular(host_stub)) return {}; + } + return std::vector(files_seen.begin(), files_seen.end()); + } +}; +} // namespace nilbuild::darwin_toolchain_detail diff --git a/nilbuild/toolchain_inputs.hpp b/nilbuild/toolchain_inputs.hpp index 8524cdcf..ab604050 100644 --- a/nilbuild/toolchain_inputs.hpp +++ b/nilbuild/toolchain_inputs.hpp @@ -2,6 +2,7 @@ #include "process.hpp" #include "windows_toolchain_inputs.hpp" +#include "darwin_toolchain_inputs.hpp" #include #include @@ -11,6 +12,9 @@ #include #include #include +#if defined(__APPLE__) +#include +#endif namespace nilbuild { struct process_cancelled : std::exception { @@ -258,6 +262,58 @@ inline std::optional> native_toolchain_files( }; return windows_toolchain_detail::collector(driver, directory, runtime, library_paths, lib ? lib : "", inspect).files(); +#elif defined(__APPLE__) + // nilbuild currently forwards no -l/--link options. Expanding that command + // surface requires expanding this closure before admitting those projects. + if (!library_paths.empty() || runtime.empty()) return std::nullopt; + for (char **entry = *_NSGetEnviron(); entry && *entry; ++entry) { + const std::string item(*entry); + const auto equal = item.find('='); + if (equal == std::string::npos || equal + 1 == item.size()) continue; + const auto name = item.substr(0, equal); + if (name.rfind("DYLD_", 0) == 0 || name.rfind("LD_", 0) == 0 || name.rfind("CCC_", 0) == 0 || + name.rfind("CLANG_CONFIG_", 0) == 0 || name == "LIBRARY_PATH" || name == "COMPILER_PATH" || + name == "GCC_EXEC_PREFIX" || name == "CPATH" || name == "CPLUS_INCLUDE_PATH" || + name == "C_INCLUDE_PATH" || name == "OBJC_INCLUDE_PATH" || name == "ARCHFLAGS" || + name == "IPHONEOS_DEPLOYMENT_TARGET" || name == "TVOS_DEPLOYMENT_TARGET" || + name == "WATCHOS_DEPLOYMENT_TARGET") return std::nullopt; + } + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(120); + unsigned queries = 0; + const auto query = [&](const std::vector &command, bool driver_plan = false) + -> std::optional { + const auto now = std::chrono::steady_clock::now(); + if (++queries > 128 || now >= deadline) return {}; + process_options options; + options.close_stdin = true; options.cleanup_descendants = true; + options.time_limit = std::min(std::chrono::milliseconds(15000), + std::chrono::duration_cast(deadline - now)); + options.output_limit = 4 * 1024 * 1024; options.total_output_limit = 4 * 1024 * 1024; + const auto result = execute_many({{command, directory}}, 1, options).front(); + if (result.cancelled) throw process_cancelled(result.status); + if (result.status || result.timed_out || result.output_limited || (!driver_plan && !result.error.empty())) return {}; + return driver_plan ? result.output + result.error : result.output; + }; + const auto output = query({driver.string(), "-###", "-pthread", "-o", "/dev/null", runtime.string(), + "-framework", "Cocoa", "-framework", "CoreGraphics"}, true); + if (!output) return {}; + const auto plan = darwin_toolchain_detail::parse_plan(*output, runtime); + if (!plan) return {}; +#if defined(__aarch64__) || defined(__arm64__) + if (plan->architecture != "arm64") return {}; +#elif defined(__x86_64__) + if (plan->architecture != "x86_64") return {}; +#else + return {}; +#endif + const auto inspect = [&](const std::filesystem::path &image) + -> std::optional { + const auto libraries = query({"/usr/bin/otool", "-arch", plan->architecture, "-L", image.string()}); + const auto commands = query({"/usr/bin/otool", "-arch", plan->architecture, "-l", image.string()}); + if (!libraries || !commands) return {}; + return darwin_toolchain_detail::parse_image(image, *libraries, *commands); + }; + return darwin_toolchain_detail::collector(driver, directory, runtime, *plan, inspect).files(); #else (void)driver; (void)directory; (void)runtime; (void)library_paths; return std::nullopt; diff --git a/nilruntime-freestanding/freestanding_runtime.cpp b/nilruntime-freestanding/freestanding_runtime.cpp index 1cf958d9..8ad745e7 100644 --- a/nilruntime-freestanding/freestanding_runtime.cpp +++ b/nilruntime-freestanding/freestanding_runtime.cpp @@ -72,7 +72,19 @@ extern "C" { // to zero -- which cannot be represented once the consumer links above 4 GiB, // and fails the link for a symbol nobody intended to use. A definition is // always reachable and needs no null check. +#if defined(_MSC_VER) +// COFF's link-time fallback preserves the host override without exporting a +// competing strong definition. x86 C symbols have one extra leading underscore. +void __ny_host_fail(const char *message) noexcept; +#if defined(_M_IX86) +#pragma comment(linker, "/alternatename:___ny_host_fail=___ny_default_host_fail") +#else +#pragma comment(linker, "/alternatename:__ny_host_fail=__ny_default_host_fail") +#endif +void __ny_default_host_fail(const char *message) noexcept { +#else __attribute__((weak)) void __ny_host_fail(const char *message) noexcept { +#endif (void)message; } diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 4c1f2e45..d084c460 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -129,6 +129,17 @@ also pass. Actual Apple API execution is a required hosted test. Native linker input closure is being implemented separately; compiler identity alone does not establish native incremental reuse. +The first native closure implementation hashes the selected Apple driver, +linker and helper libraries, compiler runtime, and the SDK's linkable namespace. +It resolves loader search paths per executable and preserves selected aliases. +Unknown linker options, external fallback files and injected loader/search +environments disable reuse. Fifty-nine portable checks and all 407 complete +TBD files in the ARM census pass; the other 808 captured prefixes are truncated +and require actual hosted full-file reads. The observed `-L/usr/local/lib` +driver option is still outside the admitted profile, so this implementation +does not yet establish default native cache parity. The application workflow +now requires the actual complete VM/native incremental contract on both hosts. + ## Darwin allocation retention and editor coverage The two native lifetime programs remain unchanged. Darwin now has a test-only @@ -159,6 +170,25 @@ The pinned editor archive uses a different executable name; the launcher now reads `CFBundleExecutable` from its own verified app bundle. Both failures are retained, and actual injection/editor acceptance remains pending a corrected run. +At `2632bc6`, application campaign `43118620968` (PR merge `4eac1494`) +passes the real compiler loaded-image collector and all native timestamp routes +on both architectures. Both allocation controls exit before their report +handshake; this is retained as a failed test, not a memory-retention pass. The +editor now launches but its deep profile exceeds macOS's Unix socket path limit. +Its owned fixture now uses a short temporary root while retaining Unicode paths; +launch records identify the exact retained profile logs. Both artifact manifests +were replayed locally: 104 matching files each and unchanged source. + +The complete Windows `2632bc6` build exposed an unconditional GNU weak-function +attribute in the freestanding archive. MSVC now uses a COFF alternate symbol for +the default failure hook. Separate executable link controls require the real +archive with and without an application-provided hook; the Linux archive and +existing freestanding execution contract pass. Actual MSVC archive creation, +default/override linking and both executions now pass all six phases in the +private VM, with owned cleanup and unchanged source. The full Windows campaign +will use the next committed source; its earlier failure remains recorded. The +current `2632bc6` LLVM-enabled Linux campaign also passes all 580 CTests, no skips. + Native socket/byte/entropy parity needs a binary-safe value boundary. The new VM APIs deliberately carry embedded NUL and arbitrary bytes in `text`; the current native text ABI and several LLVM operations use null-terminated strings diff --git a/scripts/ci/macos-toolchain-census.py b/scripts/ci/macos-toolchain-census.py index b3edda63..7caa263f 100644 --- a/scripts/ci/macos-toolchain-census.py +++ b/scripts/ci/macos-toolchain-census.py @@ -20,6 +20,10 @@ def main(): args.output.mkdir(parents=True, exist_ok=False) deadline = time.monotonic() + 90 report = {'status': 'FAIL', 'scope': 'SDK stub census and dry link plan only', 'commands': []} + # Only explicit tool-selection inputs; never dump the runner environment. + report['selection_environment'] = {name: os.environ.get(name) for name in ( + 'LIBRARY_PATH', 'COMPILER_PATH', 'SDKROOT', 'DEVELOPER_DIR', 'TOOLCHAINS', + 'DYLD_LIBRARY_PATH', 'DYLD_FALLBACK_LIBRARY_PATH', 'LDFLAGS')} def run(name, argv): remaining = deadline - time.monotonic() diff --git a/tests/darwin_native_heap_budget.c b/tests/darwin_native_heap_budget.c index 75b5b22a..be03eba4 100644 --- a/tests/darwin_native_heap_budget.c +++ b/tests/darwin_native_heap_budget.c @@ -71,6 +71,27 @@ static char *field(char *at, const char *name, uint64_t value) { at = append_text(at, ",\""); at = append_text(at, name); at = append_text(at, "\":"); return append_number(at, value); } +/* Initialization itself can fail before the report file is available. Keep + * these failures distinguishable from a consumer's retained-allocation exit, + * without asking stdio or the allocator to format the diagnostic. */ +__attribute__((noreturn)) static void tracker_error(const char *event) { + int saved_errno = errno; + char line[256], *at = line; + at = append_text(at, "nil heap tracker: "); + at = append_text(at, event); + at = append_text(at, "; errno="); + at = append_number(at, (uint64_t)saved_errno); + at = append_text(at, "\n"); + const char *next = line; + size_t left = (size_t)(at - line); + while (left) { + ssize_t written = write(STDERR_FILENO, next, left); + if (written < 0 && errno == EINTR) continue; + if (written <= 0) break; + next += written; left -= (size_t)written; + } + _exit(NIL_HEAP_TRACKER_EXIT); +} static void emit_locked(const char *event) { char line[2048], *at = line; if (report_fd < 0) return; @@ -89,12 +110,13 @@ static void emit_locked(const char *event) { while (left) { ssize_t written = write(report_fd, next, left); if (written < 0 && errno == EINTR) continue; - if (written <= 0) _exit(NIL_HEAP_TRACKER_EXIT); + if (written <= 0) tracker_error("report-write"); next += written; left -= (size_t)written; } } static void fail_locked(const char *event, int code) { emit_locked(event); + if (code == NIL_HEAP_TRACKER_EXIT) tracker_error(event); _exit(code); } @@ -303,21 +325,21 @@ __attribute__((constructor)) static void initialize_budget(void) { uint64_t target_size; int withhold_mode; if (!path || !*path || !given_token || strlen(given_token) != 32 || !target || !mode) - _exit(NIL_HEAP_TRACKER_EXIT); + tracker_error("initialization-environment"); for (size_t index = 0; index < 32; ++index) { char value = given_token[index]; if (!((value >= '0' && value <= '9') || (value >= 'a' && value <= 'f'))) - _exit(NIL_HEAP_TRACKER_EXIT); + tracker_error("initialization-token"); token[index] = value; } if (strcmp(target, "16384") == 0) target_size = 16384; else if (strcmp(target, "65536") == 0) target_size = 65536; - else _exit(NIL_HEAP_TRACKER_EXIT); + else tracker_error("initialization-target"); if (strcmp(mode, "0") == 0) withhold_mode = 0; else if (strcmp(mode, "1") == 0) withhold_mode = 1; - else _exit(NIL_HEAP_TRACKER_EXIT); + else tracker_error("initialization-withhold"); int descriptor = open(path, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600); - if (descriptor < 0) _exit(NIL_HEAP_TRACKER_EXIT); + if (descriptor < 0) tracker_error("report-open"); lock_tracker(); stats.target_size = target_size; withhold = withhold_mode; report_fd = descriptor; ready = 1; emit_locked("ready"); unlock_tracker(); diff --git a/tests/darwin_toolchain_inputs_tests.cpp b/tests/darwin_toolchain_inputs_tests.cpp new file mode 100644 index 00000000..62866839 --- /dev/null +++ b/tests/darwin_toolchain_inputs_tests.cpp @@ -0,0 +1,213 @@ +#include "../nilbuild/darwin_toolchain_inputs.hpp" +#include "../common/content_digest.hpp" +#include "../common/private_directory.hpp" + +#include +#include +#include + +namespace fs = std::filesystem; +namespace detail = nilbuild::darwin_toolchain_detail; +namespace { +int checks = 0; +void check(bool value, const char *message) { + ++checks; + if (!value) throw std::runtime_error(message); +} +void write(const fs::path &path, const std::string &text) { + fs::create_directories(path.parent_path()); + std::ofstream stream(path, std::ios::binary); stream << text; + if (!stream) throw std::runtime_error("cannot write fixture"); +} +std::string quote(const std::string &text) { + std::string result = "\""; + for (const auto byte : text) { if (byte == '"' || byte == '\\') result += '\\'; result += byte; } + return result + '"'; +} +std::string stub(const std::string &name = "/usr/lib/libSystem.B.dylib") { + return "--- !tapi-tbd\ntbd-version: 4\ntargets: [ arm64-macos, x86_64-macos ]\ninstall-name: " + name + + "\nreexported-libraries:\n - targets: [ arm64-macos ]\n libraries: [ '/usr/lib/system/libdyld.dylib',\n" + " '/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation' ]\n" + "exports:\n - targets: [ arm64-macos ]\n symbols: [ _one, '$ld$hide$os10.4$_two' ]\n...\n"; +} +std::string plan_text(const detail::link_plan &plan, const fs::path &runtime) { + std::string result = "Apple clang version 17.0.0 (clang-1700.0.13.5)\nTarget: arm64-apple-darwin24.6.0\n" + "Thread model: posix\nInstalledDir: " + plan.installed.string() + "\n "; + for (const auto &value : std::vector{plan.linker.string(), "-demangle", "-lto_library", plan.lto.string(), + "-dynamic", "-arch", plan.architecture, "-platform_version", "macos", "15.0.0", "15.5", "-syslibroot", + plan.sdk.string(), "-mllvm", "-enable-linkonceodr-outlining", "-o", "/dev/null", runtime.string(), + "-framework", "Cocoa", "-framework", "CoreGraphics", "-lc++", "-lSystem", plan.compiler_rt.string()}) + result += quote(value) + ' '; + return result + '\n'; +} +std::map identities(const std::vector &files) { + std::map result; + for (const auto &path : files) result[path.string()] = nil::detail::file_digest(path); + return result; +} +void parsing(const detail::link_plan &plan, const fs::path &runtime) { + const auto text = plan_text(plan, runtime); + const auto parsed = detail::parse_plan(text, runtime); + check(parsed && parsed->sdk == plan.sdk && parsed->linker == plan.linker && + parsed->architecture == "arm64", "real Apple17 plan shape resolves without losing paths"); + auto intel = plan; intel.architecture = "x86_64"; + check(detail::parse_plan(plan_text(intel, runtime), runtime).has_value(), "Intel Apple17 plan resolves"); + for (const auto &extra : {" \"@response.rsp\"", " \"-L/other/lib\"", " \"-framework\" \"Unknown\"", " \"-force_load\" \"other.a\""}) { + auto changed = text; changed.insert(changed.size() - 1, extra); + check(!detail::parse_plan(changed, runtime), "unmodeled link inputs fail closed"); + } + check(!detail::parse_plan("Configuration file: hidden.cfg\n" + text, runtime), "opaque driver configs fail closed"); + check(!detail::parse_plan(text + " \"second-job\"\n", runtime), "extra tool jobs fail closed"); + auto truncated = text.substr(0, text.find(quote("-platform_version"))) + "\"-platform_version\"\n"; + check(!detail::parse_plan(truncated, runtime), "truncated multi-value options fail closed without indexing past argv"); + check(!detail::quoted_arguments("\"unterminated"), "unclosed command quote rejected"); + check(!detail::quoted_arguments("\"one\"unquoted"), "adjacent unquoted text rejected"); + check(!detail::quoted_arguments("\"escaped\\nnewline\""), "unknown command escapes rejected"); + check(detail::supported_stub(stub()), "generated YAML TBDv4 re-export lists supported"); + auto aligned = stub(); aligned.replace(aligned.find("tbd-version: 4"), 14, "tbd-version: 4"); + check(detail::supported_stub(aligned), "actual SDK aligned version field supported"); + for (const auto &name : {"@rpath/library.dylib", "/AppleInternal/build/library.dylib"}) + check(detail::supported_stub(stub(name)), "SDK install IDs are interface data, not re-export file reads"); + for (const auto &name : {"relative.dylib", "@rpath/library.dylib", "/tmp/library.dylib", "/usr/lib/../../tmp/library.dylib"}) + check(!detail::supported_stub(stub() + "libraries: [ '" + name + "' ]\n"), "escaping re-export paths rejected"); + auto external = stub(); external.replace(external.find("/usr/lib/system/libdyld.dylib"), 27, "/outside/library.dylib"); + check(!detail::supported_stub(external), "escaping re-export rejected"); + check(!detail::supported_stub(stub() + "libraries: [ relative ]\n"), "relative re-export rejected"); + check(detail::supported_stub(stub() + "libraries: [ '/System/Library/Frameworks/F.framework/Resources/hidden' ]\n"), + "actual ATS/ImageIO resource libraries are included in modeled SDK trees"); + check(!detail::supported_stub(stub() + "libraries: [ '/usr/lib/unmodeled.extension' ]\n"), "unmodeled library filename types rejected"); + check(!detail::supported_stub(stub() + "libraries: *alias\n"), "YAML aliases cannot hide dependencies"); + check(!detail::supported_stub(stub() + "reexported-libraries: [{ libraries: [ relative ] }]\n"), + "unmodeled flow mappings cannot hide relative re-exports"); + check(!detail::supported_stub(stub() + "\"libraries\": [ relative ]\n"), "quoted YAML keys require richer parsing"); + check(!detail::supported_stub(stub() + "libraries: [ '/usr/lib/\\u0061.dylib' ]\n"), "escaped path spelling rejected"); + check(!detail::supported_stub(stub() + "libraries: [ '/usr/lib/one.dylib'\n"), "unfinished re-export list rejected"); + const auto image = plan.linker; + const auto description = detail::parse_image(image, + image.string() + ":\n\t@rpath/libtapi.dylib (compatibility version 1.0.0, current version 1.0.0)\n", + "Load command 1\n cmd LC_RPATH\n cmdsize 32\n path @loader_path/../lib (offset 12)\n"); + check(description && description->libraries.size() == 1 && description->rpaths == + std::vector{"@loader_path/../lib"}, "otool image and rpath inputs parsed together"); + check(!detail::parse_image(image, "bad format\n", ""), "unknown image listing rejected"); + check(!detail::parse_image(image, image.string() + ":\n", "cmd LC_RPATH\n"), "truncated rpath rejected"); + check(!detail::parse_image(image, image.string() + ":\n", "cmd LC_DYLD_ENVIRONMENT\n"), "embedded loader environment rejected"); + check(!detail::parse_image(image, image.string() + ":\n", "cmd LC_LOAD_DYLINKER\ncmdsize 32\nname /unmodeled/dyld (offset 12)\n"), + "unmodeled dynamic loader rejected"); + const auto loader = detail::parse_image(image, image.string() + ":\n", + "cmd LC_LOAD_DYLINKER\ncmdsize 32\nname /usr/lib/dyld (offset 12)\n"); + check(loader && loader->system_loader, "standard system loader explicitly recognized"); +} +void closure(const fs::path &root) { + const auto project = root / "project", host = root / "host"; + fs::create_directories(project); + detail::link_plan plan; + plan.installed = root / "Xcode test.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin"; + plan.linker = plan.installed / "ld"; + plan.sdk = root / "SDKs/MacOSX15.5.sdk"; + plan.lto = plan.installed.parent_path() / "lib/libLTO.dylib"; + plan.compiler_rt = plan.installed.parent_path() / "lib/clang/17/lib/darwin/libclang_rt.osx.a"; + plan.architecture = "arm64"; + const auto driver = root / "selected clang++", runtime = root / "nil.a"; + for (const auto &path : {driver, plan.installed / "clang++", plan.linker, plan.lto, + plan.installed.parent_path() / "lib/libtapi.dylib", plan.compiler_rt, runtime}) + write(path, "fixture bytes"); + write(plan.sdk / "SDKSettings.json", "SDK15.5"); + write(host / "System/Library/CoreServices/SystemVersion.plist", "macOS15"); + write(plan.sdk / "usr/lib/libSystem.tbd", stub()); + write(plan.sdk / "usr/lib/system/libdyld.tbd", stub("/usr/lib/system/libdyld.dylib")); + write(plan.sdk / "System/Library/Frameworks/Cocoa.framework/Cocoa.tbd", stub("/System/Library/Frameworks/Cocoa.framework/Cocoa")); + write(plan.sdk / "System/Library/Frameworks/CoreGraphics.framework/CoreGraphics.tbd", stub("/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics")); + std::map descriptions; + descriptions[plan.linker] = {{"@rpath/libtapi.dylib", "@rpath/libLTO.dylib", "/usr/lib/libSystem.B.dylib"}, {"@loader_path/../lib"}}; + descriptions[plan.lto] = {{"@rpath/libLTO.dylib", "/usr/lib/libSystem.B.dylib"}, {}}; + const auto inspect = [&](const fs::path &path) -> std::optional { return descriptions[path]; }; + const auto collect = [&]() { return detail::collector(driver, project, runtime, plan, inspect, host).files(); }; + parsing(plan, runtime); + auto files = collect(); + check(files.has_value(), "default SDK/toolchain closure resolves"); + auto before = identities(*files); + for (const auto &path : {driver, plan.linker, plan.lto, plan.compiler_rt, runtime, + plan.installed.parent_path() / "lib/libtapi.dylib", plan.sdk / "usr/lib/system/libdyld.tbd"}) + check(before.count(path.string()) != 0, "driver, helpers, runtime and transitive stub bytes included"); + const auto shared = plan.installed.parent_path() / "lib/shared.dylib"; + const auto driver_leaf = plan.installed.parent_path() / "lib/driver-set/leaf.dylib"; + const auto linker_leaf = plan.installed.parent_path() / "lib/linker-set/leaf.dylib"; + for (const auto &path : {shared, driver_leaf, linker_leaf}) write(path, "context-specific image"); + descriptions[driver] = {{shared.string()}, {driver_leaf.parent_path().string()}}; + descriptions[plan.linker].libraries.push_back(shared.string()); + descriptions[plan.linker].rpaths.push_back(linker_leaf.parent_path().string()); + descriptions[shared] = {{"@rpath/leaf.dylib"}, {}}; + files = collect(); + check(files && identities(*files).count(driver_leaf.string()) && identities(*files).count(linker_leaf.string()), + "shared image is resolved in both driver and linker inherited-runpath contexts"); + descriptions[driver] = {}; + descriptions[plan.linker].libraries.pop_back(); descriptions[plan.linker].rpaths.pop_back(); + const auto other_driver = plan.installed.parent_path() / "alternate-bin/clang++"; + const auto executable_leaf = plan.installed.parent_path() / "alternate-bin/helper.dylib"; + write(other_driver, "other driver"); write(executable_leaf, "root-specific helper"); + descriptions[other_driver] = {{"@rpath/helper.dylib"}, {"@executable_path/."}}; + files = detail::collector(other_driver, project, runtime, plan, inspect, host).files(); + check(files && identities(*files).count(executable_leaf.string()), "executable_path follows the current process root"); + files = collect(); before = identities(*files); + const auto leaf = plan.sdk / "usr/lib/system/libdyld.tbd"; + const auto timestamp = fs::last_write_time(leaf); + write(leaf, stub("/usr/lib/system/libdyld.dylib") + "# same-timestamp change\n"); + fs::last_write_time(leaf, timestamp); + files = collect(); + check(files && identities(*files) != before, "stub content changes invalidate despite preserved timestamps"); + before = identities(*files); + write(plan.sdk / "usr/lib/new-shadow.tbd", stub("/usr/lib/new-shadow.dylib")); + files = collect(); check(files && identities(*files) != before, "new SDK shadow changes input namespace"); + before = identities(*files); + write(plan.installed.parent_path() / "lib/new-helper.dylib", "new helper"); + files = collect(); check(files && identities(*files) != before, "new toolchain helper changes namespace"); + write(plan.sdk / "usr/lib/escape.tbd", stub() + "libraries: [ '/unmodeled/library.dylib' ]\n"); + check(!collect(), "external SDK re-export disables reuse"); + fs::remove(plan.sdk / "usr/lib/escape.tbd"); + descriptions[plan.linker].libraries.push_back("/unmodeled/helper.dylib"); + check(!collect(), "external ordinary toolchain dependency disables reuse"); + descriptions[plan.linker].libraries.pop_back(); + descriptions[plan.linker].rpaths.push_back("/unmodeled"); + check(!collect(), "unmodeled rpath disables reuse"); + descriptions[plan.linker].rpaths.pop_back(); + write(host / "System/Library/Frameworks/CoreFoundation.framework/CoreFoundation", "ordinary host fallback"); + check(!collect(), "new ordinary host fallback for absent SDK re-export disables reuse"); + fs::remove(host / "System/Library/Frameworks/CoreFoundation.framework/CoreFoundation"); + write(runtime, "!\n"); + check(!collect(), "thin archive external members cannot evade identity"); + write(runtime, "fixture bytes"); + fs::create_directories(plan.sdk / "AppleInternal"); + check(!collect(), "AppleInternal implicit order-file policies disable default-profile reuse"); + fs::remove(plan.sdk / "AppleInternal"); +#ifndef _WIN32 + const auto alias = plan.sdk / "usr/lib/alias.tbd"; + fs::create_symlink("system/libdyld.tbd", alias); + files = collect(); check(files && identities(*files).count(alias.string()), "selected file alias retained"); + before = identities(*files); + fs::remove(alias); fs::create_symlink("libSystem.tbd", alias); + files = collect(); check(files && identities(*files) != before, "file alias retarget invalidates even when both old targets exist"); + const auto directory_alias = plan.sdk / "usr/lib/current"; + fs::create_directory_symlink("system", directory_alias); + files = collect(); check(files && identities(*files).count((directory_alias / "libdyld.tbd").string()), + "directory alias preserves its selected child paths"); + fs::create_directory_symlink(".", plan.sdk / "usr/lib/cycle"); + check(!collect(), "directory symlink cycles fail closed within bounds"); + fs::remove(plan.sdk / "usr/lib/cycle"); +#endif + fs::remove(plan.lto); + check(!collect(), "missing required linker helper disables reuse"); +} +} // namespace + +int main() { + const auto root = nil::detail::create_private_directory(fs::temp_directory_path(), "nil-darwin-closure-"); + try { + closure(root); + fs::remove_all(root); + std::cout << "Darwin toolchain closure: " << checks << " checks passed\n"; + return 0; + } catch (const std::exception &error) { + std::error_code ignored; fs::remove_all(root, ignored); + std::cerr << error.what() << '\n'; return 1; + } +} diff --git a/tests/freestanding_hook_contract.cpp b/tests/freestanding_hook_contract.cpp new file mode 100644 index 00000000..aada5c64 --- /dev/null +++ b/tests/freestanding_hook_contract.cpp @@ -0,0 +1,23 @@ +// Link the actual freestanding archive into an owned hosted test. The default +// hook must resolve, and an application-supplied hook must replace it. +#include + +extern "C" std::uint32_t nyrt_abi_version() noexcept; +extern "C" void __ny_host_fail(const char *) noexcept; + +#if defined(NIL_TEST_HOST_OVERRIDE) +static const char *observed = nullptr; +extern "C" void __ny_host_fail(const char *message) noexcept { observed = message; } +#endif + +int main() { + // Require the runtime object even in the override case, so a duplicate + // strong definition cannot escape the link check through archive laziness. + if (nyrt_abi_version() != 1) return 1; + static const char message[] = "owned failure hook probe"; + __ny_host_fail(message); +#if defined(NIL_TEST_HOST_OVERRIDE) + if (observed != message) return 2; +#endif + return 0; +} diff --git a/tests/tools/darwin_native_heap_budget.py b/tests/tools/darwin_native_heap_budget.py index 28b06e97..e84b17c5 100644 --- a/tests/tools/darwin_native_heap_budget.py +++ b/tests/tools/darwin_native_heap_budget.py @@ -8,6 +8,7 @@ """ from __future__ import annotations +import argparse import json import os from pathlib import Path @@ -59,12 +60,14 @@ def __init__(self, args, name, *, seconds, fixtures): 'whole_process_memory_parity': False, 'whole_seconds': seconds, 'commands': [], 'runs': [], 'imports': [], 'fixtures': [], 'inputs': {}} - inputs = [self.library, self.control, args.compiler.resolve(strict=True), Path(__file__), - ROOT / ('tests/tools/native_' + name.replace('-', '_') + '.py'), + inputs = [self.library, self.control, Path(__file__), ROOT / 'tests/darwin_native_heap_budget.c', ROOT / 'tests/darwin_native_heap_budget.h', ROOT / 'tests/darwin_native_heap_budget_control.c', ROOT / 'validation/ny-3d/scripts/debugger_process.py'] + if fixtures: + inputs.extend([args.compiler.resolve(strict=True), + ROOT / ('tests/tools/native_' + name.replace('-', '_') + '.py')]) for path in inputs: self.report['inputs'][str(path)] = support.file_record(path) @@ -109,7 +112,11 @@ def command(self, name, command, *, stdout=b'', stderr=b'', exit_code=0, seconds errors = (self.logs / row['stderr']).read_bytes() if (row.get('exit_code') != exit_code or not row.get('cleanup_verified') or row.get('error') or (stdout is not None and actual != stdout) or errors != stderr): - raise AssertionError(name + ': output/exit/deadline/owned-cleanup contract failed') + raise AssertionError(name + ': output/exit/deadline/owned-cleanup contract failed; ' + f"exit={row.get('exit_code')} expected={exit_code}, " + f"cleanup={row.get('cleanup_verified')}, " + f"error={row.get('error')!r}, stdout={actual[:1024]!r}, " + f"stderr={errors[:1024]!r}") row['status'] = 'PASS' # Expected negative exits are evidence, not skipped tests. return actual @@ -215,3 +222,21 @@ def fixture(self, name, source, executable, expected, *, target, count): 'source_record': source_record, 'executable_record': executable_record, 'expected_target_allocations': count, 'target_bytes': target, 'positive_and_withheld_free_replay': 'PASS'}) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + add_arguments(parser) + parser.add_argument('--controls-only', action='store_true', required=True, + help='Qualify injection and allocation accounting without building native fixtures') + args = parser.parse_args() + harness = DarwinHeapBudget(args, 'heap-controls', seconds=180, fixtures=0) + with harness: + # The identical positive, untouched-retention, and withheld-free controls + # used by both lifetime contracts run before the expensive compiler build. + # Passing this command does not qualify either native lifetime fixture. + harness.prepare() + + +if __name__ == '__main__': + main() diff --git a/vscode/test/run-host.cjs b/vscode/test/run-host.cjs index daac7d10..7547180f 100644 --- a/vscode/test/run-host.cjs +++ b/vscode/test/run-host.cjs @@ -9,13 +9,19 @@ const { resolveEditor, hostTimeout, writeIsolatedProfileSettings, validateReport const extensionRoot = path.resolve(__dirname, ".."); const repository = path.resolve(extensionRoot, ".."); -const temporary = fs.mkdtempSync(path.join(os.tmpdir(), "nil vscode host café 中文-")); +// macOS stores its editor IPC socket below the profile. Its ordinary temporary +// directory (or a deep CI TMPDIR) can exceed the Unix socket path limit before +// the editor starts. Keep this owned fixture short, retaining Unicode paths. +const temporary = fs.mkdtempSync(path.join(process.platform === "darwin" ? "/tmp" : os.tmpdir(), "nil vscode host café 中文-")); const workspace = path.join(temporary, "workspace"); const profile = path.join(temporary, "profile"); const reportDirectory = path.join(repository, "build", "vscode-host-smoke"); fs.mkdirSync(path.join(workspace, ".vscode"), { recursive: true }); fs.mkdirSync(reportDirectory, { recursive: true }); const runId = randomUUID(); +fs.writeFileSync(path.join(reportDirectory, `${runId}-launch.json`), JSON.stringify({ + runId, temporary, workspace, profile, platform: process.platform +}, null, 2)); const report = path.join(reportDirectory, `${runId}.json`); const tools = path.resolve(process.env.NIL_VSCODE_TOOL_DIRECTORY || process.env.NY_VSCODE_TOOL_DIRECTORY || path.join(repository, "build", "cmake", "bin")); const executable = resolveEditor(process.env.VSCODE_EXECUTABLE); From a5181386aade200894cf26be59325bde438f76d1 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 18:48:28 -0500 Subject: [PATCH 10/39] Handle Darwin allocator nesting and qualify native object contracts --- .../workflows/macos-application-parity.yml | 4 +- CMakeLists.txt | 15 +- README.md | 13 +- docs/platform-support.md | 2 +- nilbuild/darwin_toolchain_inputs.hpp | 75 ++++++++++ nilbuild/toolchain_inputs.hpp | 18 ++- notebook/2026-11-11-web-runtime-parity.md | 37 +++++ tests/darwin_native_heap_budget.c | 101 ++++++++++--- tests/darwin_native_heap_budget_control.c | 7 + tests/darwin_native_heap_budget_nested.c | 117 +++++++++++++++ tests/darwin_toolchain_inputs_tests.cpp | 78 +++++++++- .../tools/darwin_heap_accounting_contract.py | 98 ++++++++++++ tests/tools/darwin_native_heap_budget.py | 12 +- tests/tools/native_symbol_surface.py | 141 ++++++++++++++---- vscode/test/suite.cjs | 12 +- 15 files changed, 653 insertions(+), 77 deletions(-) create mode 100644 tests/darwin_native_heap_budget_nested.c create mode 100644 tests/tools/darwin_heap_accounting_contract.py diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index af2e51c7..87ca7108 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -138,8 +138,8 @@ jobs: from pathlib import Path expected = {'compiler.darwin-module-fingerprint', 'ny.native-managed-lifetime', 'ny.native-owned-lifetime', 'nil.file-time-support', 'nil.file-modified-at-contract', - 'nil.freestanding-hook-default', 'nil.freestanding-hook-override'} - pattern = r'^(compiler[.]darwin-module-fingerprint|ny[.]native-(managed|owned)-lifetime|nil[.]file-(time-support|modified-at-contract)|nil[.]freestanding-hook-(default|override))$' + 'nil.freestanding-hook-default', 'nil.freestanding-hook-override', 'nil.native-symbol-surface'} + pattern = r'^(compiler[.]darwin-module-fingerprint|ny[.]native-(managed|owned)-lifetime|nil[.]file-(time-support|modified-at-contract)|nil[.]freestanding-hook-(default|override)|nil[.]native-symbol-surface)$' base = ['ctest', '--test-dir', 'build/application', '-R', pattern] root = Path('build/application-evidence').resolve() inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) diff --git a/CMakeLists.txt b/CMakeLists.txt index 3c6543d4..7b4ad42b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -689,6 +689,12 @@ if(BUILD_TESTING) # test suite when testing was explicitly enabled. find_package(Python3 3.8 REQUIRED COMPONENTS Interpreter) if(Python3_Interpreter_FOUND) + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_test(NAME nil.darwin-heap-accounting + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/darwin_heap_accounting_contract.py" + --cc "${CMAKE_C_COMPILER}" --output-parent "${CMAKE_BINARY_DIR}/test-evidence") + set_tests_properties(nil.darwin-heap-accounting PROPERTIES LABELS "runtime;native;rc" TIMEOUT 120) + endif() add_test(NAME nilbuild.llvm-dia-configuration COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/llvm_dia_cmake_contract.py" --helper "${CMAKE_CURRENT_SOURCE_DIR}/cmake/NilLlvmDia.cmake" @@ -790,9 +796,16 @@ if(BUILD_TESTING) # consumer has to supply every undefined symbol deliberately, so this # boundary widens only on purpose. if(NIL_ENABLE_LLVM) + find_program(NIL_TEST_LLVM_NM NAMES llvm-nm llvm-nm-18 + HINTS "${LLVM_DIR}/../../../bin" "${LLVM_DIR}/../bin") + set(NIL_SYMBOL_TOOL_ARGUMENTS) + if(NIL_TEST_LLVM_NM) + list(APPEND NIL_SYMBOL_TOOL_ARGUMENTS --llvm-nm "${NIL_TEST_LLVM_NM}") + endif() add_test(NAME nil.native-symbol-surface COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/native_symbol_surface.py" - --compiler $ --assembler $) + --compiler $ --assembler $ + ${NIL_SYMBOL_TOOL_ARGUMENTS}) set_tests_properties(nil.native-symbol-surface PROPERTIES LABELS "native;abi;rc" SKIP_RETURN_CODE 77 TIMEOUT 180) endif() diff --git a/README.md b/README.md index 20cb263a..ea664dc5 100644 --- a/README.md +++ b/README.md @@ -108,10 +108,11 @@ The framework includes: [testing guide](docs/tools/testing.md). - **Desktop applications.** The Matrix-authored [`nil.std.ui`](docs/standard-library/ui.md) provides windows, layout containers, - standard controls, typed events, and explicit model synchronization. Its GTK 4 - provider supports VM and native execution on Linux and Windows development - builds. GTK is an external runtime dependency; Windows requires an interactive, - visible desktop. This does not extend Windows release qualification. + standard controls, typed events, and explicit model synchronization. VM and + native desktop checks pass with GTK4 on Linux, Win32 on Windows and AppKit on + Intel and Apple Silicon macOS. Linux requires an external GTK4 runtime; + desktop tests require an interactive session. Installed SDK and release + qualification remain separate from these development checks. - **Aperture runtime diagnostics.** Memory profiles record runtime allocations, frees, retained storage, peaks and source locations for VM and native programs. Saved JSON reports support inspection and automated analysis. See @@ -260,8 +261,8 @@ native Win32 provider; Linux uses GTK4. macOS editor integration and a complete current SDK desktop campaign remain open. The newer socket, entropy and text-byte intrinsics pass the same interpreter contracts on Linux, Windows and both Macs. File timestamps share an implementation across interpreter and native routes, -with Linux and Windows text/binary NIL, JIT and AOT checks passing. -Incremental reuse and native memory-limit tests also remain open, so full platform +with text/binary NIL, JIT and AOT checks passing on all four hosts. +Darwin incremental reuse and native allocation-retention tests remain open, so full platform parity is not yet established. The [platform support matrix](docs/platform-support.md) separates verified development behavior from remaining SDK, provider and release work. diff --git a/docs/platform-support.md b/docs/platform-support.md index b00bf9f7..8ed3d8d9 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -151,7 +151,7 @@ the complete admission rules. - Native incremental reuse is still being qualified. The compiler now resolves loaded images to ordinary files and exact shared-cache backing files for full - content hashing; its actual Apple Silicon collector test passes at `2632bc6`. + content hashing; its actual collector test passes on both Macs at `2632bc6`. The builder uses separate preparation and verification digest phases without caching source/manifest reads. Native linker dependency collection and real VM/native reuse still require both hosted architectures to pass. Incomplete diff --git a/nilbuild/darwin_toolchain_inputs.hpp b/nilbuild/darwin_toolchain_inputs.hpp index 27ceaf14..8fbeb4c0 100644 --- a/nilbuild/darwin_toolchain_inputs.hpp +++ b/nilbuild/darwin_toolchain_inputs.hpp @@ -72,6 +72,7 @@ inline std::optional> quoted_arguments(const std::strin struct link_plan { fs::path installed, linker, sdk, lto, compiler_rt; std::string architecture; + bool local_libraries = false; }; inline std::optional parse_plan(const std::string &text, const fs::path &runtime) { if (text.size() > 1024 * 1024 || text.find('\0') != std::string::npos) return {}; @@ -114,6 +115,15 @@ inline std::optional parse_plan(const std::string &text, const fs::pa if (option == "-dynamic" && !dynamic) { dynamic = true; continue; } if (option == "-lc++" && !cxx) { cxx = true; continue; } if (option == "-lSystem" && !system) { system = true; continue; } + // Apple's selected driver emits this explicit search path on ordinary + // hosted installations. Unlike ld's implicit default it can fall back to + // the host when the selected SDK has no usr/local/lib directory. + if (option == "-L/usr/local/lib") { plan.local_libraries = true; continue; } + if (option == "-L") { + const auto value = next(); + if (!value || *value != "/usr/local/lib") return {}; + plan.local_libraries = true; continue; + } if (option == "-arch" && plan.architecture.empty()) { const auto value = next(); if (!value || (*value != "arm64" && *value != "x86_64")) return {}; plan.architecture = *value; continue; @@ -161,6 +171,15 @@ inline std::optional parse_plan(const std::string &text, const fs::pa return plan; } +// The two explicit archives are the only link inputs that can carry object +// autolinks in this profile. An unmodeled LC_LINKER_OPTION can request libraries +// outside the command's fixed c++/System/framework roots. otool must successfully +// inspect the selected architecture before the caller invokes the collector. +inline bool supported_archive_commands(const std::string &text) { + return !text.empty() && text.size() <= 4 * 1024 * 1024 && text.find('\0') == std::string::npos && + text.find("LC_LINKER_OPTION") == std::string::npos; +} + inline bool system_install_name(const std::string &name) { const fs::path path(name); // Install names are always POSIX paths, including in portable parser tests. @@ -440,6 +459,61 @@ class collector { } return !error; } + bool local_library_inputs() { + if (!plan.local_libraries) return true; + std::error_code error; + // Options.cpp:4380-4429: an explicit absolute -L uses the SDK directory + // when it exists; unlike a standard search directory it otherwise falls + // back to the host. Inspect this choice anew on every collection. + const auto sdk_directory = plan.sdk / "usr/local/lib"; + const bool sdk_directory_exists = fs::is_directory(sdk_directory, error); + if (error && error != std::errc::no_such_file_or_directory) return false; + if (sdk_directory_exists) return true; // Complete SDK tree was collected. + error.clear(); + const auto directory = host_path("/usr/local/lib"); + const bool host_directory_exists = fs::is_directory(directory, error); + if (error && error != std::errc::no_such_file_or_directory) return false; + if (!host_directory_exists) return true; + + // The default command's two -l roots and every architecture/OS SDK + // re-export form a conservative set of names the linker can look up here. + // This is a fresh namespace lookup, not a list cached from the last build: + // additions, removals, aliases and new transitive TBD names are observed. + // Unrelated Homebrew libraries cannot be chosen by these roots; inspecting + // their dyld dependencies would incorrectly model them as executed tools. + std::set candidates = {"libc++.tbd", "libc++.dylib", "libc++.so", "libc++.a", + "libSystem.tbd", "libSystem.dylib", "libSystem.so", "libSystem.a"}; + std::set checked; + for (;;) { + for (const auto &reference : stub_references) { + const auto path = fs::path(reference), leaf = path.filename(); + // Options.cpp:1153-1199 searches frameworks via -F, and never searches + // -L for a .dylib embedded inside a framework. Other indirect library + // lookups use the literal leaf and its .tbd replacement (no lib prefix). + if (reference.find('/' + leaf.string() + ".framework/") != std::string::npos || + (path.extension() == ".dylib" && reference.find(".framework/") != std::string::npos)) continue; + candidates.insert(leaf.string()); + auto stub = leaf; stub.replace_extension(".tbd"); candidates.insert(stub.string()); + } + if (candidates.size() > 131072) return false; + bool changed = false; + for (const auto &name : candidates) { + if (!checked.insert(name).second) continue; + changed = true; + const auto path = directory / name; + error.clear(); const auto status = fs::status(path, error); + if (error == std::errc::no_such_file_or_directory || status.type() == fs::file_type::not_found) continue; + if (error || !fs::is_regular_file(status)) return false; + // A selected ordinary binary/archive requires its own link-input + // dependency closure; don't confuse that with the tools' dyld graph. + // Admit only supported text stubs, including a stub behind an alias + // with a .dylib/extensionless name, and collect its re-exports as well. + if (!add(path) || !read_stub(path)) return false; + } + if (!changed) break; + } + return true; + } public: collector(fs::path selected_driver, fs::path directory, fs::path selected_runtime, link_plan selected_plan, probe inspect_image, fs::path host_root = "/") @@ -473,6 +547,7 @@ class collector { const bool present = fs::exists(directory, error); if (error || (present && !tree(directory, true))) return {}; } + if (!local_library_inputs()) return {}; // Absolute re-export lookup differs from standard -L/-F lookup: when the // SDK/inlined stub cannot satisfy it, ld can try the raw host pathname. // Until ordinary host link-input closure is modeled, refuse that profile. diff --git a/nilbuild/toolchain_inputs.hpp b/nilbuild/toolchain_inputs.hpp index ab604050..bc693339 100644 --- a/nilbuild/toolchain_inputs.hpp +++ b/nilbuild/toolchain_inputs.hpp @@ -266,13 +266,21 @@ inline std::optional> native_toolchain_files( // nilbuild currently forwards no -l/--link options. Expanding that command // surface requires expanding this closure before admitting those projects. if (!library_paths.empty() || runtime.empty()) return std::nullopt; + bool local_library_environment = false; for (char **entry = *_NSGetEnviron(); entry && *entry; ++entry) { const std::string item(*entry); const auto equal = item.find('='); if (equal == std::string::npos || equal + 1 == item.size()) continue; const auto name = item.substr(0, equal); + if (name == "LIBRARY_PATH") { + // This exact search profile is modeled below and already appears in the + // build key. Empty components/other directories are intentionally not + // inferred or sanitized; they require a broader search closure. + if (item.substr(equal + 1) != "/usr/local/lib") return std::nullopt; + local_library_environment = true; continue; + } if (name.rfind("DYLD_", 0) == 0 || name.rfind("LD_", 0) == 0 || name.rfind("CCC_", 0) == 0 || - name.rfind("CLANG_CONFIG_", 0) == 0 || name == "LIBRARY_PATH" || name == "COMPILER_PATH" || + name.rfind("CLANG_CONFIG_", 0) == 0 || name == "COMPILER_PATH" || name == "GCC_EXEC_PREFIX" || name == "CPATH" || name == "CPLUS_INCLUDE_PATH" || name == "C_INCLUDE_PATH" || name == "OBJC_INCLUDE_PATH" || name == "ARCHFLAGS" || name == "IPHONEOS_DEPLOYMENT_TARGET" || name == "TVOS_DEPLOYMENT_TARGET" || @@ -298,7 +306,7 @@ inline std::optional> native_toolchain_files( "-framework", "Cocoa", "-framework", "CoreGraphics"}, true); if (!output) return {}; const auto plan = darwin_toolchain_detail::parse_plan(*output, runtime); - if (!plan) return {}; + if (!plan || (local_library_environment && !plan->local_libraries)) return {}; #if defined(__aarch64__) || defined(__arm64__) if (plan->architecture != "arm64") return {}; #elif defined(__x86_64__) @@ -306,6 +314,12 @@ inline std::optional> native_toolchain_files( #else return {}; #endif + // Archives can carry hidden -l/-framework directives in object load commands. + // Keep that unsupported until their names join the modeled search roots. + for (const auto &archive : {runtime, plan->compiler_rt}) { + const auto commands = query({"/usr/bin/otool", "-arch", plan->architecture, "-l", archive.string()}); + if (!commands || !darwin_toolchain_detail::supported_archive_commands(*commands)) return {}; + } const auto inspect = [&](const std::filesystem::path &image) -> std::optional { const auto libraries = query({"/usr/bin/otool", "-arch", plan->architecture, "-L", image.string()}); diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index d084c460..8cdfcb41 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -140,6 +140,17 @@ driver option is still outside the admitted profile, so this implementation does not yet establish default native cache parity. The application workflow now requires the actual complete VM/native incremental contract on both hosts. +The next closure slice models the observed explicit `/usr/local/lib` search +option using Apple's SDK rerooting rules and a fresh candidate lookup on every +collection. Supported stub shadows and their transitive re-exports are hashed; +unmodeled binary shadows and archive autolinks disable reuse. Seventy-seven +Linux checks and sanitizer controls pass. The portable fixture's path keys now +use generic separators; actual MSVC compilation and all 71 Windows controls +pass (the additional six Linux controls require symlink creation). The source +campaign's original Windows fixture failure is retained. Hosted default native +reuse remains required; `f79b94b` proves VM reuse on ARM but correctly refuses +the still-unmodeled native search profile in that revision. + ## Darwin allocation retention and editor coverage The two native lifetime programs remain unchanged. Darwin now has a test-only @@ -189,6 +200,32 @@ private VM, with owned cleanup and unchanged source. The full Windows campaign will use the next committed source; its earlier failure remains recorded. The current `2632bc6` LLVM-enabled Linux campaign also passes all 580 CTests, no skips. +The `f79b94b` early Darwin controls identify the initialization failure as a +duplicate-pointer accounting event: libmalloc can nest typed and untyped public +allocator calls. The fixture now accounts only for the outer request using a +fixed per-thread table, avoiding first-use TLS allocation. An explicit nesting +shim reproduces the old failure; ordinary/nested positive, untouched-retention +and withheld-free controls pass under both GCC and Clang. They include startup, +realloc preservation and overlapping threads. Coverage counts now subtract the +ready snapshot. Actual Darwin qualification remains pending this correction. + +The shortened Mac editor profile launches successfully. At `f79b94b` the ARM +Problems check fails because the test requests `/tmp` diagnostics while the +compiler correctly reports the physical `/private/tmp` file. Retained terminal +output and Problems entries show the expected compiler source, line and message. +The harness now checks physical file identity with its existing alias helper, +while still requiring those diagnostic fields. This is not an editor pass until +the complete real-host suite finishes; no checks are excluded. + +The native symbol-surface harness now reads COFF and Mach-O through explicit +LLVM tooling instead of skipping those hosts. It preserves the same runtime +import allowlists and checks freestanding entry/host interop and IR attributes; +ELF x86_64 relocation assertions remain identified as that format's extra +coverage. Six actual Clang-emitted format/architecture fixtures validate the +reader. All sixteen generic checks pass with the real Windows `f79b94b` compiler +and assembler; Linux passes those plus both ELF relocation checks. Mach-O +framework execution remains in the next hosted campaign. + Native socket/byte/entropy parity needs a binary-safe value boundary. The new VM APIs deliberately carry embedded NUL and arbitrary bytes in `text`; the current native text ABI and several LLVM operations use null-terminated strings diff --git a/tests/darwin_native_heap_budget.c b/tests/darwin_native_heap_budget.c index be03eba4..b4ca1e1a 100644 --- a/tests/darwin_native_heap_budget.c +++ b/tests/darwin_native_heap_budget.c @@ -7,6 +7,7 @@ #include "darwin_native_heap_budget.h" #include #include +#include #include #include #include @@ -26,14 +27,18 @@ #elif defined(NIL_HEAP_BUDGET_PORTABLE_TEST) /* Linux --wrap exercises the identical accounting and controls locally. It * does not validate Mach-O interposition and cannot admit Darwin coverage. */ -extern void *__real_malloc(size_t); -extern void *__real_calloc(size_t, size_t); -extern void *__real_realloc(void *, size_t); -extern void __real_free(void *); -extern void *__real_aligned_alloc(size_t, size_t); -extern int __real_posix_memalign(void **, size_t, size_t); -extern void *__real_valloc(size_t); +#if defined(NIL_HEAP_BUDGET_NESTED_TEST) +#define ORIGINAL(name) nil_heap_nested_##name +#else #define ORIGINAL(name) __real_##name +#endif +extern void *ORIGINAL(malloc)(size_t); +extern void *ORIGINAL(calloc)(size_t, size_t); +extern void *ORIGINAL(realloc)(void *, size_t); +extern void ORIGINAL(free)(void *); +extern void *ORIGINAL(aligned_alloc)(size_t, size_t); +extern int ORIGINAL(posix_memalign)(void **, size_t, size_t); +extern void *ORIGINAL(valloc)(size_t); #else #error This test requires Darwin (or the explicit portable accounting check). #endif @@ -48,6 +53,8 @@ static struct nil_heap_snapshot stats = { }; static int report_fd = -1, ready, withhold, fail_realloc; static char token[33]; +#define ACTIVE_THREADS 256u +static struct { pthread_t owner; int active; } active_calls[ACTIVE_THREADS]; static void lock_tracker(void) { while (atomic_flag_test_and_set_explicit(&tracker_lock, memory_order_acquire)) {} @@ -120,6 +127,38 @@ static void fail_locked(const char *event, int code) { _exit(code); } +/* The original allocator can call another interposed public entry point: + * notably Darwin's typed/untyped compatibility paths during initialization. + * Charge only the outer request, including realloc's internal malloc/free. + * Do not use C/C++ TLS: Darwin's first TLV lookup may itself allocate. pthread + * self identity is already available to dyld before library initialization. + * This fixed table has no lifetime allocation and fails closed if exhausted. */ +static size_t enter_allocator(void) { + const pthread_t self = pthread_self(); + size_t vacant = ACTIVE_THREADS; + lock_tracker(); + for (size_t index = 0; index < ACTIVE_THREADS; ++index) { + if (!active_calls[index].active) { if (vacant == ACTIVE_THREADS) vacant = index; } + else if (pthread_equal(active_calls[index].owner, self)) { + unlock_tracker(); return ACTIVE_THREADS; + } + } + if (vacant == ACTIVE_THREADS) fail_locked("active-thread-limit", NIL_HEAP_TRACKER_EXIT); + active_calls[vacant].owner = self; active_calls[vacant].active = 1; + unlock_tracker(); return vacant; +} +static void leave_allocator(size_t slot) { + int saved_errno = errno; + lock_tracker(); active_calls[slot].active = 0; unlock_tracker(); + errno = saved_errno; +} +static void *leave_result(void *result, size_t slot) { + leave_allocator(slot); return result; +} +#define ENTER_OR_FORWARD(call) \ + const size_t active_slot = enter_allocator(); \ + if (active_slot == ACTIVE_THREADS) return (call) + static size_t slot_for(void *pointer, int inserting) { size_t first_dead = TABLE_COUNT; size_t start = ((uintptr_t)pointer >> 4) * UINT64_C(11400714819323198485); @@ -223,32 +262,42 @@ static void *finish_realloc(void *old_pointer, void *pointer, size_t bytes, stru } static void *budget_malloc(size_t bytes) { - reserve(bytes, 0); return finish_allocate(ORIGINAL(malloc)(bytes), bytes); + ENTER_OR_FORWARD(ORIGINAL(malloc)(bytes)); + reserve(bytes, 0); return leave_result(finish_allocate(ORIGINAL(malloc)(bytes), bytes), active_slot); } static void *budget_calloc(size_t count, size_t size) { - size_t bytes; if (!calloc_size(count, size, &bytes, 0)) return NULL; - reserve(bytes, 0); return finish_allocate(ORIGINAL(calloc)(count, size), bytes); + ENTER_OR_FORWARD(ORIGINAL(calloc)(count, size)); + size_t bytes; if (!calloc_size(count, size, &bytes, 0)) return leave_result(NULL, active_slot); + reserve(bytes, 0); return leave_result(finish_allocate(ORIGINAL(calloc)(count, size), bytes), active_slot); } static void budget_free(void *pointer) { + const size_t active_slot = enter_allocator(); + if (active_slot == ACTIVE_THREADS) { ORIGINAL(free)(pointer); return; } if (release(pointer, 0)) ORIGINAL(free)(pointer); + leave_allocator(active_slot); } static void *budget_realloc(void *pointer, size_t bytes) { + ENTER_OR_FORWARD(ORIGINAL(realloc)(pointer, bytes)); struct realloc_state state = begin_realloc(pointer, bytes, 0); - if (state.denied) { errno = ENOMEM; return NULL; } - return finish_realloc(pointer, ORIGINAL(realloc)(pointer, bytes), bytes, state); + if (state.denied) { errno = ENOMEM; return leave_result(NULL, active_slot); } + return leave_result(finish_realloc(pointer, ORIGINAL(realloc)(pointer, bytes), bytes, state), active_slot); } static void *budget_aligned_alloc(size_t alignment, size_t bytes) { - reserve(bytes, 0); return finish_allocate(ORIGINAL(aligned_alloc)(alignment, bytes), bytes); + ENTER_OR_FORWARD(ORIGINAL(aligned_alloc)(alignment, bytes)); + reserve(bytes, 0); return leave_result(finish_allocate(ORIGINAL(aligned_alloc)(alignment, bytes), bytes), active_slot); } static int budget_posix_memalign(void **out, size_t alignment, size_t bytes) { + ENTER_OR_FORWARD(ORIGINAL(posix_memalign)(out, alignment, bytes)); reserve(bytes, 0); void *pointer = NULL; int result = ORIGINAL(posix_memalign)(&pointer, alignment, bytes); finish_allocate(result == 0 ? pointer : NULL, bytes); if (result == 0) *out = pointer; + leave_allocator(active_slot); return result; } static void *budget_valloc(size_t bytes) { - reserve(bytes, 0); return finish_allocate(ORIGINAL(valloc)(bytes), bytes); + ENTER_OR_FORWARD(ORIGINAL(valloc)(bytes)); + reserve(bytes, 0); return leave_result(finish_allocate(ORIGINAL(valloc)(bytes), bytes), active_slot); } #if defined(__APPLE__) @@ -264,32 +313,42 @@ NIL_HEAP_INTERPOSE(budget_valloc, valloc) /* Modern Apple Clang can rewrite ordinary calls to typed allocation symbols. * Intercept these actual entry points too, preserving the type argument. */ static void *budget_type_malloc(size_t bytes, malloc_type_id_t type) { - reserve(bytes, 1); return finish_allocate(malloc_type_malloc(bytes, type), bytes); + ENTER_OR_FORWARD(malloc_type_malloc(bytes, type)); + reserve(bytes, 1); return leave_result(finish_allocate(malloc_type_malloc(bytes, type), bytes), active_slot); } static void *budget_type_calloc(size_t count, size_t size, malloc_type_id_t type) { - size_t bytes; if (!calloc_size(count, size, &bytes, 1)) return NULL; - reserve(bytes, 1); return finish_allocate(malloc_type_calloc(count, size, type), bytes); + ENTER_OR_FORWARD(malloc_type_calloc(count, size, type)); + size_t bytes; if (!calloc_size(count, size, &bytes, 1)) return leave_result(NULL, active_slot); + reserve(bytes, 1); return leave_result(finish_allocate(malloc_type_calloc(count, size, type), bytes), active_slot); } static void budget_type_free(void *pointer, malloc_type_id_t type) { + const size_t active_slot = enter_allocator(); + if (active_slot == ACTIVE_THREADS) { malloc_type_free(pointer, type); return; } if (release(pointer, 1)) malloc_type_free(pointer, type); + leave_allocator(active_slot); } static void *budget_type_realloc(void *pointer, size_t bytes, malloc_type_id_t type) { + ENTER_OR_FORWARD(malloc_type_realloc(pointer, bytes, type)); struct realloc_state state = begin_realloc(pointer, bytes, 1); - if (state.denied) { errno = ENOMEM; return NULL; } - return finish_realloc(pointer, malloc_type_realloc(pointer, bytes, type), bytes, state); + if (state.denied) { errno = ENOMEM; return leave_result(NULL, active_slot); } + return leave_result(finish_realloc(pointer, malloc_type_realloc(pointer, bytes, type), bytes, state), active_slot); } static void *budget_type_aligned_alloc(size_t alignment, size_t bytes, malloc_type_id_t type) { - reserve(bytes, 1); return finish_allocate(malloc_type_aligned_alloc(alignment, bytes, type), bytes); + ENTER_OR_FORWARD(malloc_type_aligned_alloc(alignment, bytes, type)); + reserve(bytes, 1); return leave_result(finish_allocate(malloc_type_aligned_alloc(alignment, bytes, type), bytes), active_slot); } static int budget_type_posix_memalign(void **out, size_t alignment, size_t bytes, malloc_type_id_t type) { + ENTER_OR_FORWARD(malloc_type_posix_memalign(out, alignment, bytes, type)); reserve(bytes, 1); void *pointer = NULL; int result = malloc_type_posix_memalign(&pointer, alignment, bytes, type); finish_allocate(result == 0 ? pointer : NULL, bytes); if (result == 0) *out = pointer; + leave_allocator(active_slot); return result; } static void *budget_type_valloc(size_t bytes, malloc_type_id_t type) { - reserve(bytes, 1); return finish_allocate(malloc_type_valloc(bytes, type), bytes); + ENTER_OR_FORWARD(malloc_type_valloc(bytes, type)); + reserve(bytes, 1); return leave_result(finish_allocate(malloc_type_valloc(bytes, type), bytes), active_slot); } NIL_HEAP_INTERPOSE(budget_type_malloc, malloc_type_malloc) NIL_HEAP_INTERPOSE(budget_type_calloc, malloc_type_calloc) diff --git a/tests/darwin_native_heap_budget_control.c b/tests/darwin_native_heap_budget_control.c index 912ff484..d4a668a1 100644 --- a/tests/darwin_native_heap_budget_control.c +++ b/tests/darwin_native_heap_budget_control.c @@ -15,6 +15,9 @@ static nil_heap_snapshot_fn snapshot; static nil_heap_fail_realloc_fn fail_next_realloc; static void *volatile retained[2048]; static volatile unsigned char observed; +#if defined(NIL_HEAP_BUDGET_NESTED_TEST) +extern void nil_heap_nested_verify_threads(void); +#endif static void write_all(int fd, const char *text) { size_t left = strlen(text); @@ -126,6 +129,10 @@ int main(int argc, char **argv) { require(snapshot && fail_next_realloc, "heap budget injection absent\n"); require(argc == 2, "expected one control mode\n"); (void)state(); +#if defined(NIL_HEAP_BUDGET_NESTED_TEST) + if (strcmp(argv[1], "positive") == 0 && strcmp(getenv("NIL_DARWIN_HEAP_WITHHOLD"), "0") == 0) + nil_heap_nested_verify_threads(); +#endif if (strcmp(argv[1], "positive") == 0) { positive(); return 0; } require(strcmp(argv[1], "retain") == 0, "unknown control mode\n"); /* Do not touch payload pages: a footprint-only replacement can miss this. diff --git a/tests/darwin_native_heap_budget_nested.c b/tests/darwin_native_heap_budget_nested.c new file mode 100644 index 00000000..028aa141 --- /dev/null +++ b/tests/darwin_native_heap_budget_nested.c @@ -0,0 +1,117 @@ +/* Linux-only allocator shim for the Darwin oracle's accounting regression. + * Real libc requests deliberately reenter other interposed public entry points, + * as Darwin typed/untyped compatibility paths can do. This does not qualify + * Mach-O injection: the hosted controls still must pass on each architecture. */ +#if !defined(NIL_HEAP_BUDGET_PORTABLE_TEST) || !defined(NIL_HEAP_BUDGET_NESTED_TEST) +#error This shim is only for the explicit portable nested-accounting test. +#endif +#include "darwin_native_heap_budget.h" +#include +#include +#include +#include +#include +#include +#include + +extern void *__real_malloc(size_t); +extern void *__real_calloc(size_t, size_t); +extern void *__real_realloc(void *, size_t); +extern void __real_free(void *); +extern void *__real_aligned_alloc(size_t, size_t); +extern int __real_posix_memalign(void **, size_t, size_t); +extern void *__real_valloc(size_t); +extern int nil_heap_budget_snapshot(struct nil_heap_snapshot *, size_t); + +/* TLS is safe in this Linux-only shim, unlike the Darwin interposer itself. + * Volatile prevents libc's compiler-visible leaf attributes from eliminating + * the state change around calls that GNU --wrap redirects back into this TU. */ +static _Thread_local volatile int nested; +static atomic_int parallel; +static pthread_barrier_t inside, allocated, releasing; + +void *nil_heap_nested_malloc(size_t bytes) { + if (nested) return __real_malloc(bytes); + nested = 1; void *result = calloc(1, bytes); nested = 0; return result; +} +void *nil_heap_nested_calloc(size_t count, size_t size) { + if (nested) return __real_calloc(count, size); + if (size && count > SIZE_MAX / size) { errno = ENOMEM; return NULL; } + nested = 1; + if (atomic_load(¶llel) && count * size == 65536) pthread_barrier_wait(&inside); + void *result = malloc(count * size); + if (result) memset(result, 0, count * size); + nested = 0; return result; +} +void nil_heap_nested_free(void *pointer) { + if (nested) { __real_free(pointer); return; } + nested = 1; free(pointer); nested = 0; +} +void *nil_heap_nested_realloc(void *pointer, size_t bytes) { + if (nested) return __real_realloc(pointer, bytes); + nested = 1; + void *result = bytes ? malloc(bytes) : NULL; + if (result && pointer) { + size_t available = malloc_usable_size(pointer); + memcpy(result, pointer, bytes < available ? bytes : available); + } + if (result || !bytes) free(pointer); + nested = 0; return result; +} +void *nil_heap_nested_aligned_alloc(size_t alignment, size_t bytes) { + if (nested) return __real_aligned_alloc(alignment, bytes); + nested = 1; void *result = NULL; + int error = posix_memalign(&result, alignment, bytes); + nested = 0; if (error) errno = error; return result; +} +int nil_heap_nested_posix_memalign(void **out, size_t alignment, size_t bytes) { + if (nested) return __real_posix_memalign(out, alignment, bytes); + nested = 1; int result = posix_memalign(out, alignment, bytes); nested = 0; return result; +} +void *nil_heap_nested_valloc(size_t bytes) { + if (nested) return __real_valloc(bytes); + nested = 1; void *result = NULL; + int error = posix_memalign(&result, (size_t)getpagesize(), bytes); + nested = 0; if (error) errno = error; return result; +} + +__attribute__((noreturn)) static void failed(void) { + static const char message[] = "nested allocator accounting differs\n"; + const ssize_t written = write(2, message, sizeof(message) - 1); + (void)written; _exit(92); +} +/* Reproduce the actual hosted failure before the report constructor opens its + * file: without outer-call accounting this exits duplicate-pointer/87. */ +__attribute__((constructor(101))) static void early_nested_request(void) { + void *result = malloc(1024); if (!result) failed(); free(result); +} +static void *parallel_request(void *unused) { + (void)unused; + void *result = calloc(1, 65536); if (!result) failed(); + pthread_barrier_wait(&allocated); + pthread_barrier_wait(&releasing); + free(result); return NULL; +} +void nil_heap_nested_verify_threads(void) { + struct nil_heap_snapshot before, during, after; + pthread_t threads[4]; + if (!nil_heap_budget_snapshot(&before, sizeof(before))) failed(); + if (pthread_barrier_init(&inside, NULL, 5) || pthread_barrier_init(&allocated, NULL, 5) || + pthread_barrier_init(&releasing, NULL, 5)) failed(); + atomic_store(¶llel, 1); + for (size_t index = 0; index < 4; ++index) + if (pthread_create(&threads[index], NULL, parallel_request, NULL)) failed(); + /* All four original allocators are active together. A process-global + * reentry flag would incorrectly exclude the other three outer requests. */ + pthread_barrier_wait(&inside); + pthread_barrier_wait(&allocated); + if (!nil_heap_budget_snapshot(&during, sizeof(during)) || + during.live != before.live + 4 * 65536 || during.allocations != before.allocations + 4 || + during.target_allocations != before.target_allocations + 4) failed(); + pthread_barrier_wait(&releasing); + for (size_t index = 0; index < 4; ++index) if (pthread_join(threads[index], NULL)) failed(); + atomic_store(¶llel, 0); + if (!nil_heap_budget_snapshot(&after, sizeof(after)) || after.live != before.live || + after.frees != before.frees + 4 || after.unknown_frees != before.unknown_frees) failed(); + pthread_barrier_destroy(&inside); pthread_barrier_destroy(&allocated); pthread_barrier_destroy(&releasing); +} diff --git a/tests/darwin_toolchain_inputs_tests.cpp b/tests/darwin_toolchain_inputs_tests.cpp index 62866839..b1ca140e 100644 --- a/tests/darwin_toolchain_inputs_tests.cpp +++ b/tests/darwin_toolchain_inputs_tests.cpp @@ -38,11 +38,12 @@ std::string plan_text(const detail::link_plan &plan, const fs::path &runtime) { plan.sdk.string(), "-mllvm", "-enable-linkonceodr-outlining", "-o", "/dev/null", runtime.string(), "-framework", "Cocoa", "-framework", "CoreGraphics", "-lc++", "-lSystem", plan.compiler_rt.string()}) result += quote(value) + ' '; + if (plan.local_libraries) result += quote("-L/usr/local/lib") + ' '; return result + '\n'; } std::map identities(const std::vector &files) { std::map result; - for (const auto &path : files) result[path.string()] = nil::detail::file_digest(path); + for (const auto &path : files) result[path.generic_string()] = nil::detail::file_digest(path); return result; } void parsing(const detail::link_plan &plan, const fs::path &runtime) { @@ -52,6 +53,17 @@ void parsing(const detail::link_plan &plan, const fs::path &runtime) { parsed->architecture == "arm64", "real Apple17 plan shape resolves without losing paths"); auto intel = plan; intel.architecture = "x86_64"; check(detail::parse_plan(plan_text(intel, runtime), runtime).has_value(), "Intel Apple17 plan resolves"); + auto local = plan; local.local_libraries = true; + const auto local_plan = detail::parse_plan(plan_text(local, runtime), runtime); + check(local_plan && local_plan->local_libraries, "observed explicit local library search is modeled"); + auto separate_local = text; separate_local.insert(separate_local.size() - 1, " \"-L\" \"/usr/local/lib\""); + const auto separate_plan = detail::parse_plan(separate_local, runtime); + check(separate_plan && separate_plan->local_libraries, "separate-argument local search is equivalent"); + check(detail::supported_archive_commands("Archive : fixture.a\nfixture.a(member.o):\nLoad command 1\ncmd LC_SEGMENT_64\n"), + "ordinary archive load commands admit the fixed link roots"); + check(!detail::supported_archive_commands("Archive : fixture.a\ncmd LC_LINKER_OPTION\ncount 2\nstring #1 -l\nstring #2 hidden\n"), + "archive autolinks require additional modeled library roots"); + check(!detail::supported_archive_commands(""), "empty archive inspection cannot certify absent autolinks"); for (const auto &extra : {" \"@response.rsp\"", " \"-L/other/lib\"", " \"-framework\" \"Unknown\"", " \"-force_load\" \"other.a\""}) { auto changed = text; changed.insert(changed.size() - 1, extra); check(!detail::parse_plan(changed, runtime), "unmodeled link inputs fail closed"); @@ -128,7 +140,7 @@ void closure(const fs::path &root) { auto before = identities(*files); for (const auto &path : {driver, plan.linker, plan.lto, plan.compiler_rt, runtime, plan.installed.parent_path() / "lib/libtapi.dylib", plan.sdk / "usr/lib/system/libdyld.tbd"}) - check(before.count(path.string()) != 0, "driver, helpers, runtime and transitive stub bytes included"); + check(before.count(path.generic_string()) != 0, "driver, helpers, runtime and transitive stub bytes included"); const auto shared = plan.installed.parent_path() / "lib/shared.dylib"; const auto driver_leaf = plan.installed.parent_path() / "lib/driver-set/leaf.dylib"; const auto linker_leaf = plan.installed.parent_path() / "lib/linker-set/leaf.dylib"; @@ -138,7 +150,7 @@ void closure(const fs::path &root) { descriptions[plan.linker].rpaths.push_back(linker_leaf.parent_path().string()); descriptions[shared] = {{"@rpath/leaf.dylib"}, {}}; files = collect(); - check(files && identities(*files).count(driver_leaf.string()) && identities(*files).count(linker_leaf.string()), + check(files && identities(*files).count(driver_leaf.generic_string()) && identities(*files).count(linker_leaf.generic_string()), "shared image is resolved in both driver and linker inherited-runpath contexts"); descriptions[driver] = {}; descriptions[plan.linker].libraries.pop_back(); descriptions[plan.linker].rpaths.pop_back(); @@ -147,7 +159,7 @@ void closure(const fs::path &root) { write(other_driver, "other driver"); write(executable_leaf, "root-specific helper"); descriptions[other_driver] = {{"@rpath/helper.dylib"}, {"@executable_path/."}}; files = detail::collector(other_driver, project, runtime, plan, inspect, host).files(); - check(files && identities(*files).count(executable_leaf.string()), "executable_path follows the current process root"); + check(files && identities(*files).count(executable_leaf.generic_string()), "executable_path follows the current process root"); files = collect(); before = identities(*files); const auto leaf = plan.sdk / "usr/lib/system/libdyld.tbd"; const auto timestamp = fs::last_write_time(leaf); @@ -182,18 +194,72 @@ void closure(const fs::path &root) { #ifndef _WIN32 const auto alias = plan.sdk / "usr/lib/alias.tbd"; fs::create_symlink("system/libdyld.tbd", alias); - files = collect(); check(files && identities(*files).count(alias.string()), "selected file alias retained"); + files = collect(); check(files && identities(*files).count(alias.generic_string()), "selected file alias retained"); before = identities(*files); fs::remove(alias); fs::create_symlink("libSystem.tbd", alias); files = collect(); check(files && identities(*files) != before, "file alias retarget invalidates even when both old targets exist"); const auto directory_alias = plan.sdk / "usr/lib/current"; fs::create_directory_symlink("system", directory_alias); - files = collect(); check(files && identities(*files).count((directory_alias / "libdyld.tbd").string()), + files = collect(); check(files && identities(*files).count((directory_alias / "libdyld.tbd").generic_string()), "directory alias preserves its selected child paths"); fs::create_directory_symlink(".", plan.sdk / "usr/lib/cycle"); check(!collect(), "directory symlink cycles fail closed within bounds"); fs::remove(plan.sdk / "usr/lib/cycle"); #endif + plan.local_libraries = true; + files = collect(); + check(files.has_value(), "explicit local search permits an absent host directory"); + const auto local = host / "usr/local/lib"; + write(local / "libunrelated.dylib", "unrelated ordinary binary"); + files = collect(); + check(files.has_value(), "unrelated host library cannot become an input of fixed default link roots"); + write(local / "libc++.tbd", stub("/usr/lib/libc++.1.dylib")); + files = collect(); + check(files && identities(*files).count((local / "libc++.tbd").generic_string()), + "direct c++ text stub shadow is captured"); + before = identities(*files); + const auto local_time = fs::last_write_time(local / "libc++.tbd"); + write(local / "libc++.tbd", stub("/usr/lib/libc++.1.dylib") + "# same-time shadow change\n"); + fs::last_write_time(local / "libc++.tbd", local_time); + files = collect(); + check(files && identities(*files) != before, "host stub bytes invalidate with preserved timestamps"); + before = identities(*files); + write(local / "libdyld.tbd", stub("/usr/lib/system/libdyld.dylib") + + "libraries: [ '/usr/lib/extra-dependency.dylib' ]\n"); + write(local / "extra-dependency.tbd", stub("/usr/lib/extra-dependency.dylib")); + files = collect(); + check(files && identities(*files) != before && identities(*files).count((local / "extra-dependency.tbd").generic_string()), + "new SDK indirect shadow and its additional re-export are collected to a fixed point"); + fs::remove(local / "extra-dependency.tbd"); + write(local / "extra-dependency.dylib", "unmodeled ordinary dependency"); + check(!collect(), "new transitive ordinary shadow disables reuse"); + fs::remove(local / "extra-dependency.dylib"); + write(local / "libSystem.a", "!\n"); + check(!collect(), "direct static library shadow requires additional link-input closure"); + fs::remove(local / "libSystem.a"); + write(local / "libdyld.dylib", "unmodeled ordinary shadow"); + check(!collect(), "ordinary SDK indirect library shadow disables reuse"); + write(plan.sdk / "usr/local/lib/sdk-local.tbd", stub("/usr/lib/sdk-local.dylib")); + files = collect(); + check(files && !identities(*files).count((local / "libc++.tbd").generic_string()) && + identities(*files).count((plan.sdk / "usr/local/lib/sdk-local.tbd").generic_string()), + "SDK local directory reroots explicit search and hides ordinary host shadows"); + fs::remove_all(plan.sdk / "usr/local/lib"); + check(!collect(), "removing SDK local directory observes newly exposed host shadow"); + fs::remove(local / "libdyld.dylib"); +#ifndef _WIN32 + const auto local_alias = local / "libSystem.tbd"; + fs::create_symlink("libc++.tbd", local_alias); + files = collect(); + check(files && identities(*files).count(local_alias.generic_string()), "host candidate alias name is retained"); + before = identities(*files); + fs::remove(local_alias); fs::create_symlink("libdyld.tbd", local_alias); + files = collect(); + check(files && identities(*files) != before, "host candidate alias retarget invalidates"); +#endif + plan.local_libraries = false; + write(local / "libSystem.dylib", "unmodeled host default"); + check(collect().has_value(), "implicit standard local path alone never falls back outside the SDK"); fs::remove(plan.lto); check(!collect(), "missing required linker helper disables reuse"); } diff --git a/tests/tools/darwin_heap_accounting_contract.py b/tests/tools/darwin_heap_accounting_contract.py new file mode 100644 index 00000000..ed8b9860 --- /dev/null +++ b/tests/tools/darwin_heap_accounting_contract.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Linux --wrap regression for Darwin heap-oracle bookkeeping, not Mach-O proof.""" +from __future__ import annotations + +import argparse +import json +import os +from pathlib import Path +import sys +import tempfile + +import ordinary_io_parity_contract as support + +ROOT = Path(__file__).resolve().parents[2] +BUDGET = 96 * 1024 * 1024 + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--cc', default='cc') + parser.add_argument('--output-parent', type=Path) + args = parser.parse_args() + if not sys.platform.startswith('linux'): + raise AssertionError('The portable --wrap control requires Linux; use real injection on Darwin') + parent = args.output_parent.resolve() if args.output_parent else Path(tempfile.gettempdir()) + parent.mkdir(parents=True, exist_ok=True) + output = Path(tempfile.mkdtemp(prefix='nil-heap-accounting-', dir=parent)) + ownership = support.load_ownership(ROOT) + environment = {key: value for key, value in os.environ.items() + if not key.startswith(('NIL_', 'NY_', 'DYLD_'))} + report = {'schema': 'nil.heap-accounting-portable.v1', 'status': 'FAIL', + 'darwin_injection_qualified': False, 'commands': [], 'cases': []} + + def command(name, argv, *, variables=None, code=0, stdout=b''): + row = support.run_process(ownership, list(map(str, argv)), ROOT, + environment | (variables or {}), output, name, 30) + report['commands'].append(row) + actual = (output / row['stdout']).read_bytes() + errors = (output / row['stderr']).read_bytes() + if (row.get('exit_code') != code or not row.get('cleanup_verified') or + row.get('error') or actual != stdout or errors): + raise AssertionError(f'{name}: exit={row.get("exit_code")} expected={code}; ' + f'stdout={actual[:1024]!r}; stderr={errors[:1024]!r}') + row['status'] = 'PASS' + + try: + for nested in (False, True): + kind = 'nested' if nested else 'ordinary' + executable = output / kind + argv = [args.cc, '-std=c11', '-D_DEFAULT_SOURCE', '-DNIL_HEAP_BUDGET_PORTABLE_TEST', + '-Wall', '-Wextra', '-Werror', '-O2', '-fno-builtin', '-rdynamic', '-pthread', + ROOT / 'tests/darwin_native_heap_budget.c', + ROOT / 'tests/darwin_native_heap_budget_control.c'] + if nested: + argv += ['-DNIL_HEAP_BUDGET_NESTED_TEST', ROOT / 'tests/darwin_native_heap_budget_nested.c'] + argv += ['-ldl'] + ['-Wl,--wrap=' + name for name in + ('malloc', 'calloc', 'realloc', 'free', 'aligned_alloc', 'posix_memalign', 'valloc')] + command(kind + '-build', argv + ['-o', executable]) + for case, mode, withhold in (('positive', 'positive', False), + ('retained', 'retain', False), ('withheld', 'positive', True)): + name = kind + '-' + case + events_path = output / (name + '.jsonl') + negative = case != 'positive' + command(name, [executable, mode], code=86 if negative else 0, + stdout=b'' if negative else b'heap-controls-ok\n', variables={ + 'NIL_DARWIN_HEAP_REPORT': str(events_path), 'NIL_DARWIN_HEAP_TOKEN': 'a' * 32, + 'NIL_DARWIN_HEAP_TARGET': '65536', 'NIL_DARWIN_HEAP_WITHHOLD': str(int(withhold))}) + events = [json.loads(line) for line in events_path.read_text().splitlines()] + if len(events) != 2 or events[0]['event'] != 'ready': + raise AssertionError(name + ': initialization/final event pair absent') + initial, final = events + if final['event'] != ('budget' if negative else 'final') or final['budget'] != BUDGET: + raise AssertionError(name + ': wrong terminal event or budget') + if nested and (initial['cumulative'] != 1024 or initial['allocations'] != 1 or initial['live']): + raise AssertionError(name + ': pre-initialization nested request was not counted exactly once') + if negative: + if not (BUDGET - 65536 <= final['live'] <= BUDGET and final['denied_request'] == 65536): + raise AssertionError(name + ': retained payload did not trigger the real budget') + if withhold and not final['withheld']: + raise AssertionError(name + ': actual target frees were not withheld') + elif (final['live'] or final['cumulative'] - initial['cumulative'] <= 2 * 1024**3 or + final['target_allocations'] - initial['target_allocations'] < 34000 + (4 if nested else 0) + or final['unknown_frees'] - initial['unknown_frees']): + raise AssertionError(name + ': sustained/nested/threaded allocation coverage differs') + report['cases'].append({'name': name, 'events': events}) + report['status'] = 'PASS' + except Exception as error: + report['error'] = str(error) + raise + finally: + report['cleanup_verified'] = bool(report['commands']) and all( + item.get('cleanup_verified') for item in report['commands']) + (output / 'report.json').write_text(json.dumps(report, indent=2) + '\n') + print(report['status'] + ': ' + str(output / 'report.json')) + + +if __name__ == '__main__': + main() diff --git a/tests/tools/darwin_native_heap_budget.py b/tests/tools/darwin_native_heap_budget.py index e84b17c5..6a1c6d90 100644 --- a/tests/tools/darwin_native_heap_budget.py +++ b/tests/tools/darwin_native_heap_budget.py @@ -166,21 +166,25 @@ def injected(self, name, executable, arguments=(), *, target, withhold=False, initial, final = events if initial['pid'] != final['pid']: raise AssertionError(name + ': report crossed process identity') - for key in ('allocations', 'frees', 'cumulative', 'target_allocations', 'target_frees', 'typed_calls'): + since_ready = {} + for key in ('allocations', 'frees', 'cumulative', 'target_allocations', 'target_frees', + 'typed_calls', 'reallocations', 'failures', 'unknown_frees', 'withheld'): if final[key] < initial[key]: raise AssertionError(name + ': allocator counters moved backwards') + since_ready[key] = final[key] - initial[key] if negative: if (not 0 < final['denied_request'] <= target or final['live'] < BUDGET - 2 * target or - final['target_allocations'] * target < BUDGET * .8 or - (withhold and final['withheld'] == 0)): + since_ready['target_allocations'] * target < BUDGET * .8 or + (withhold and since_ready['withheld'] == 0)): raise AssertionError(name + ': retention did not cause the bounded allocation rejection') elif final['withheld'] or final['denied_request']: raise AssertionError(name + ': ordinary fixture used negative-control behavior') self.report['runs'].append({'name': name, 'expected_budget_rejection': negative, 'withhold_target_frees': withhold, 'events': events, + 'since_ready': since_ready, 'report_file': events_path.name, **record}) - return final + return since_ready def prepare(self): self.command('host-version', ['/usr/bin/sw_vers'], stdout=None) diff --git a/tests/tools/native_symbol_surface.py b/tests/tools/native_symbol_surface.py index 343dd6ff..50acc2dc 100644 --- a/tests/tools/native_symbol_surface.py +++ b/tests/tools/native_symbol_surface.py @@ -7,13 +7,18 @@ satisfy with stubs it should never need, and a libc symbol arriving by accident is exactly the kind of drift that is invisible on a hosted host. -The ELF symbol table is read directly rather than shelling out to nm, so the -contract does not depend on binutils being installed. +The ELF symbol table is read directly. COFF and Mach-O use an explicitly +provided llvm-nm, with no demangling and only the Darwin ABI's one leading +underscore removed. The relocation checks apply only to ELF x86_64; the +import/export and LLVM-IR checks run on every supported object profile. """ from __future__ import annotations import argparse +from functools import partial from pathlib import Path +import platform +import re import struct import subprocess import sys @@ -41,7 +46,65 @@ freestanding_concat_imports = {'nyrt_str_length', 'nyrt_mem_copy', 'nyrt_alloc'} -def undefined_symbols(path: Path) -> set[str]: +def object_profile(path: Path) -> tuple[str, str]: + """Validate a single supported relocatable object, never an archive/image.""" + data = path.read_bytes() + if data.startswith(b'\x7fELF'): + if len(data) < 64 or data[4:7] != b'\x02\x01\x01': + raise AssertionError(f'expected ELF64 little-endian version 1: {path}') + kind, machine = struct.unpack_from(' command_bytes // 8 or command_bytes > len(data) - 32): + raise AssertionError(f'expected x86_64/arm64 Mach-O MH_OBJECT: {path}') + return 'Mach-O', 'x86_64' if cpu == 0x1000007 else 'arm64' + if len(data) >= 20: + machine, sections, _timestamp, _symbols, _symbol_count, optional, flags = struct.unpack_from(' set[str]: + names = set() + for name in output.splitlines(): + # The fixture's ABI names are ASCII identifiers; prefixes such as an + # archive/member heading, an address/type column, or demangled spaces + # must never silently become a symbol. Preserve all language underscores. + if not re.fullmatch(r'[A-Za-z_.$?@][A-Za-z_0-9.$?@]*', name): + raise AssertionError(f'unexpected llvm-nm just-symbols output: {name!r}') + if profile[0] == 'Mach-O': + if not name.startswith('_'): + raise AssertionError(f'Mach-O external symbol lacks Darwin ABI underscore: {name!r}') + name = name[1:] + if not name or name in names: + raise AssertionError(f'empty or duplicate external symbol: {name!r}') + names.add(name) + return names + + +def undefined_symbols(path: Path, llvm_nm: Path | None = None) -> tuple[set[str], set[str]]: + profile = object_profile(path) + if profile[0] != 'ELF': + if llvm_nm is None or not llvm_nm.is_file(): + raise AssertionError(f'{profile[0]} symbol checks require an explicit existing --llvm-nm executable') + sets = [] + for selection in ('--undefined-only', '--defined-only'): + command = [str(llvm_nm), '--format=just-symbols', '--extern-only', + '--no-demangle', selection, str(path)] + result = subprocess.run(command, stdin=subprocess.DEVNULL, capture_output=True, + text=True, timeout=30) + if result.returncode or result.stderr: + raise AssertionError((command, result.returncode, result.stdout, result.stderr)) + sets.append(nm_names(result.stdout, profile)) + return sets[0], sets[1] data = path.read_bytes() if data[:4] != b'\x7fELF': raise SystemExit(f'not an ELF object: {path}') @@ -117,6 +180,8 @@ def absolute_relocation_count(obj: Path) -> int: misbehaving. Read from the section headers directly, so the check needs no binutils. """ + if object_profile(obj) != ('ELF', 'x86_64'): + raise AssertionError('32-bit absolute relocation check requires ELF x86_64') data = obj.read_bytes() if data[:4] != b'\x7fELF' or data[4] != 2 or data[5] != 1: raise SystemExit(f'expected a little-endian 64-bit object: {obj}') @@ -143,7 +208,7 @@ def absolute_relocation_count(obj: Path) -> int: def emit(compiler: Path, assembler: Path, root: Path, name: str, source: str, freestanding: bool = False, no_red_zone: bool = False, code_model: str | None = None, - ir: Path | None = None): + ir: Path | None = None, llvm_nm: Path | None = None): path = root / f'{name}.ny' path.write_text(source, newline='\n') nil = root / f'{name}.nil' @@ -162,23 +227,24 @@ def emit(compiler: Path, assembler: Path, root: Path, name: str, source: str, result = subprocess.run(list(map(str, command)), stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=120) assert result.returncode == 0, (command, result.returncode, result.stdout, result.stderr) - return undefined_symbols(obj) + return undefined_symbols(obj, llvm_nm) def main() -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('--compiler', type=Path, required=True) parser.add_argument('--assembler', type=Path, required=True) + parser.add_argument('--llvm-nm', type=Path, + help='Explicit LLVM symbol reader, required for COFF and Mach-O objects') args = parser.parse_args() - if sys.platform != 'linux': - print('SKIP: the native symbol surface is pinned on ELF hosts') - return 77 checks = 0 with tempfile.TemporaryDirectory(prefix='nil-symbol-surface-') as directory: root = Path(directory) + emit_case = partial(emit, args.compiler, args.assembler, root, llvm_nm=args.llvm_nm) trivial = 'fn main() -> i64 {\n let total: i64 = 0;\n return total;\n}\n' - plain, plain_exports = emit(args.compiler, args.assembler, root, 'plain', trivial) + plain, plain_exports = emit_case('plain', trivial) + profile = object_profile(root / 'plain.o') assert plain <= runtime_abi, ( 'an object that uses no hosted facility imports only the native runtime ABI; ' f'unexpected: {sorted(plain - runtime_abi)}') @@ -194,7 +260,7 @@ def main() -> int: ' let b: text = "right";\n' ' let joined: text = a + b;\n' ' return 0;\n}\n') - concat, _ = emit(args.compiler, args.assembler, root, 'concat', + concat, _ = emit_case('concat', concatenating) assert concat_imports <= concat, ( 'a concatenating program must still import the helper dependencies; ' @@ -211,8 +277,7 @@ def main() -> int: # dependencies become runtime hooks, and no libc name may survive -- # a freestanding consumer satisfying calloc at all is the reverse- # engineered-shim problem this exists to remove. - free_concat, _ = emit(args.compiler, args.assembler, root, - 'free_concat', concatenating, freestanding=True) + free_concat, _ = emit_case('free_concat', concatenating, freestanding=True) assert freestanding_concat_imports <= free_concat, ( 'a freestanding concatenating program must reach memory through the ' 'runtime hooks; missing: ' @@ -235,7 +300,7 @@ def main() -> int: assert 'main' in plain_exports and '__ny_entry' not in plain_exports, ( f'a hosted object is entered through main; exports: {sorted(plain_exports)}') checks += 1 - free, free_exports = emit(args.compiler, args.assembler, root, 'free', + free, free_exports = emit_case('free', trivial, freestanding=True) assert '__ny_entry' in free_exports and 'main' not in free_exports, ( 'a freestanding object is entered through __ny_entry and defines no C main; ' @@ -256,14 +321,14 @@ def main() -> int: '@interop("host")\n' f'fn {host_symbol}(message: text) -> i64\n' 'end fn\n\n') - unused, _ = emit(args.compiler, args.assembler, root, 'host_unused', + unused, _ = emit_case('host_unused', declaration + 'fn main() -> i64 {\n return 0;\n}\n', freestanding=True) assert host_symbol not in unused, ( 'a host symbol that is never called must not be imported; ' f'imports: {sorted(unused)}') checks += 1 - called, _ = emit(args.compiler, args.assembler, root, 'host_called', + called, _ = emit_case('host_called', declaration + 'fn main() -> i64 {\n' ' let greeting: text = "from ny";\n' f' return {host_symbol}(greeting);\n' @@ -288,31 +353,31 @@ def main() -> int: # string constants of a program that never asked to be low. The large # model removes them, and the first assertion keeps the second from # passing because the program had no addresses to materialize. - emit(args.compiler, args.assembler, root, 'small', concatenating, - freestanding=True) - small_absolutes = absolute_relocation_count(root / 'small.o') - assert small_absolutes > 0, ( - 'the default code model is expected to use 32-bit absolute ' - 'relocations; this check would otherwise prove nothing') - checks += 1 - emit(args.compiler, args.assembler, root, 'large', concatenating, - freestanding=True, code_model='large') - large_absolutes = absolute_relocation_count(root / 'large.o') - assert large_absolutes == 0, ( - 'an object emitted for the large code model must have no 32-bit ' - f'absolute relocations; found {large_absolutes}') - checks += 1 + relocation_checks = 0 + if profile == ('ELF', 'x86_64'): + emit_case('small', concatenating, freestanding=True) + small_absolutes = absolute_relocation_count(root / 'small.o') + assert small_absolutes > 0, ( + 'the default code model is expected to use 32-bit absolute ' + 'relocations; this check would otherwise prove nothing') + relocation_checks += 1 + emit_case('large', concatenating, freestanding=True, code_model='large') + large_absolutes = absolute_relocation_count(root / 'large.o') + assert large_absolutes == 0, ( + 'an object emitted for the large code model must have no 32-bit ' + f'absolute relocations; found {large_absolutes}') + relocation_checks += 1 # The red zone is a separate decision from the entry shape: a target # whose interrupt entry switches stacks keeps it, a kernel-reentrant # one cannot. So it is its own option and is checked on its own. hosted_ir = root / 'hosted.ll' - emit(args.compiler, args.assembler, root, 'hosted_ir', trivial, ir=hosted_ir) + emit_case('hosted_ir', trivial, ir=hosted_ir) assert definitions_without_noredzone(hosted_ir), ( 'the red zone stays available unless it is turned off') checks += 1 guarded_ir = root / 'guarded.ll' - emit(args.compiler, args.assembler, root, 'guarded_ir', trivial, + emit_case('guarded_ir', trivial, freestanding=True, no_red_zone=True, ir=guarded_ir) missing = definitions_without_noredzone(guarded_ir) assert not missing, ( @@ -320,7 +385,19 @@ def main() -> int: f'without it: {missing}') checks += 1 - print(f'Native symbol surface: {checks} checks passed') + expected_format = {'linux': 'ELF', 'win32': 'COFF', 'darwin': 'Mach-O'}.get(sys.platform) + expected_architecture = {'x86_64': 'x86_64', 'amd64': 'x86_64', + 'arm64': 'arm64', 'aarch64': 'arm64'}.get(platform.machine().lower()) + assert profile == (expected_format, expected_architecture), ( + f'default object profile {profile} differs from host {(expected_format, expected_architecture)}') + assert all(object_profile(path) == profile for path in root.glob('*.o')), ( + 'object profile changed between native surface fixtures') + + print(f'Native symbol surface ({profile[0]} {profile[1]}): {checks} generic ABI/IR checks passed') + if relocation_checks: + print(f'ELF x86_64 small/large-model relocation checks: {relocation_checks} passed') + else: + print(f'Relocation coverage is limited to ELF x86_64; {profile[0]} {profile[1]} is outside that profile') return 0 diff --git a/vscode/test/suite.cjs b/vscode/test/suite.cjs index b77a41c2..ba7f162a 100644 --- a/vscode/test/suite.cjs +++ b/vscode/test/suite.cjs @@ -275,8 +275,16 @@ exports.run = async function run() { await eventually(() => events.length > start, "rejected compiler task"); assert.equal(events[start].task, "compile"); assert.notEqual(events[start].exitCode, 0); - const diagnostics = await eventually(() => { - const items = vscode.languages.getDiagnostics(invalid.uri).filter((item) => item.source === "nilcompiler"); + const diagnostics = await eventually(async () => { + // Compiler diagnostics use physical source paths. On macOS /tmp and + // /private/tmp identify the same file but have different URI spellings. + // Require the correct file identity and compiler source in Problems. + const items = []; + for (const [uri, entries] of vscode.languages.getDiagnostics()) { + if (uri.scheme === "file" && await sameFile(uri.fsPath, invalid.uri.fsPath)) { + items.push(...entries.filter((item) => item.source === "nilcompiler")); + } + } return items.length && items; }, "Matrix compiler problem matcher"); assert.ok(diagnostics.some((item) => item.range.start.line === 2 && /Type mismatch/.test(item.message))); From 837625258103292b1c869d4fa32f12fba1179290 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 19:04:24 -0500 Subject: [PATCH 11/39] Keep DAP execution on the main thread and strengthen platform acceptance --- .../workflows/macos-application-parity.yml | 65 +++++- CMakeLists.txt | 8 + docs/standard-library/ui.md | 8 +- nilbuild/darwin_toolchain_inputs.hpp | 109 +++++---- nilbuild/main.cpp | 10 +- nilbuild/toolchain_inputs.hpp | 43 ++-- nildebugger/CMakeLists.txt | 13 ++ nildebugger/README.md | 10 +- nildebugger/dap.cpp | 184 ++++++++++++---- nildebugger/tests/dap_contract_support.py | 208 ++++++++++++++++++ nildebugger/tests/dap_desktop.py | 167 ++++++++++++++ nildebugger/tests/dap_lifecycle.py | 111 ++++++++++ nildebugger/tests/dap_owner_tests.cpp | 154 +++++++++++++ notebook/2026-11-11-web-runtime-parity.md | 52 +++++ tests/darwin_native_heap_budget.c | 1 + tests/darwin_native_heap_budget_control.c | 29 ++- tests/darwin_native_inputs_tests.cpp | 45 ++++ tests/darwin_toolchain_inputs_tests.cpp | 16 +- .../tools/darwin_heap_accounting_contract.py | 20 +- tests/tools/darwin_native_heap_budget.py | 8 + tests/tools/native_owned_lifetime.py | 5 +- 21 files changed, 1144 insertions(+), 122 deletions(-) create mode 100644 nildebugger/tests/dap_contract_support.py create mode 100644 nildebugger/tests/dap_desktop.py create mode 100644 nildebugger/tests/dap_lifecycle.py create mode 100644 nildebugger/tests/dap_owner_tests.cpp create mode 100644 tests/darwin_native_inputs_tests.cpp diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index 87ca7108..d9617808 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -17,10 +17,12 @@ name: macOS application platform checks - 'nillinker/process_runner*' - 'validation/ny-3d/scripts/debugger_process.py' - 'scripts/ci/macos-toolchain-census.py' - - 'tests/darwin_native_heap_budget*' + - 'tests/darwin*' + - 'tests/compiler_module_darwin_tests.cpp' - 'tests/tools/darwin_native_heap_budget.py' - 'tests/tools/native_managed_lifetime.py' - 'tests/tools/native_owned_lifetime.py' + - 'tests/tools/native_symbol_surface.py' - 'tests/tools/file_modified_at_contract.py' - 'vscode/**' workflow_dispatch: @@ -112,6 +114,32 @@ jobs: build/application-evidence/heap-*.log if-no-files-found: warn retention-days: 14 + - name: Build early native dependency probe + id: native_inputs + if: ${{ !cancelled() && steps.configure.outcome == 'success' }} + timeout-minutes: 8 + run: | + set -o pipefail + cmake --build build/application --parallel 2 --target nil_darwin_native_inputs_tests \ + 2>&1 | tee build/application-evidence/native-inputs-build.log + - name: Check the selected native dependency profile + if: ${{ !cancelled() && steps.native_inputs.outcome == 'success' }} + timeout-minutes: 4 + run: | + set -o pipefail + ctest --test-dir build/application -R '^nil[.]darwin-native-inputs$' \ + --no-tests=error --output-on-failure --output-junit native-inputs-results.xml \ + 2>&1 | tee build/application-evidence/native-inputs.log + - name: Preserve early native dependency evidence + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: macos-native-inputs-${{ matrix.host.arch }}-${{ github.sha }}-${{ github.run_attempt }} + path: | + build/application-evidence/native-inputs*.log + build/application/native-inputs-results.xml + if-no-files-found: warn + retention-days: 14 - name: Build application tools id: tools if: ${{ !cancelled() && steps.configure.outcome == 'success' }} @@ -122,7 +150,39 @@ jobs: nilcompiler nilinterpreter nilassembler nillsp nildebugger nilbuild niltest aperture \ nil_vscode_host_runner nil_file_time_support_tests nil_compiler_module_darwin_tests \ nil_darwin_toolchain_inputs_tests nil_freestanding_hook_default nil_freestanding_hook_override \ + nildebugger_core_tests nildebugger_source_metadata_tests nildebugger_dap_owner_tests \ 2>&1 | tee build/application-evidence/build.log + - name: Verify debugger protocols and owned desktop sessions + if: ${{ !cancelled() && steps.tools.outcome == 'success' }} + timeout-minutes: 12 + run: | + set -o pipefail + python3 - <<'PYTHON' + import json, subprocess, xml.etree.ElementTree as ET + from pathlib import Path + expected = {'nildebugger.'+name for name in + ('core', 'json-protocol', 'dap-protocol', 'source-metadata', 'dap-lifecycle', 'dap-owner')} + base = ['ctest', '--test-dir', 'build/application', '-R', + r'^nildebugger[.](core|json-protocol|dap-protocol|source-metadata|dap-lifecycle|dap-owner)$'] + root = Path('build/application-evidence').resolve() + inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) + (root/'debugger-inventory.json').write_bytes(inventory) + assert {case['name'] for case in json.loads(inventory)['tests']} == expected + with (root/'debugger.log').open('wb') as output: + run = subprocess.run(base+['--no-tests=error', '--output-on-failure', + '--output-junit', str(root/'debugger-results.xml')], stdout=output, + stderr=subprocess.STDOUT, timeout=480) + print((root/'debugger.log').read_text(errors='replace')) + run.check_returncode() + cases = list(ET.parse(root/'debugger-results.xml').getroot().iter('testcase')) + assert len(cases) == len(expected) and {case.get('name') for case in cases} == expected + assert all(case.get('status') == 'run' and all(case.find(kind) is None + for kind in ['failure','error','skipped']) for case in cases) + PYTHON + python3 -B nildebugger/tests/dap_desktop.py \ + --debugger build/application/bin/nildebugger --compiler build/application/bin/nilcompiler \ + --nilstd nilstd/src --output-parent build/application/test-evidence \ + 2>&1 | tee build/application-evidence/debugger-desktop.log - name: Record selected native SDK and linker inputs if: ${{ !cancelled() && steps.tools.outcome == 'success' }} timeout-minutes: 2 @@ -189,7 +249,8 @@ jobs: # Version 1.146.0; architecture-specific SHA256 values from Microsoft's # update API on 2026-11-11. No auto-update or system installation. mkdir -p build/editor-download - curl --fail --location --max-time 180 --max-filesize 800000000 \ + curl --fail --location --max-time 120 --max-filesize 800000000 \ + --retry 2 --retry-all-errors --retry-delay 2 --retry-max-time 180 \ "https://vscode.download.prss.microsoft.com/dbazure/download/stable/69047191fc427f2f92828279aae1d90d8eeda7ab/$NIL_EDITOR_ARCHIVE" \ -o build/editor-download/editor.zip python3 - <<'PYTHON' diff --git a/CMakeLists.txt b/CMakeLists.txt index 7b4ad42b..770712b9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -148,6 +148,14 @@ if(BUILD_TESTING) endif() add_test(NAME nil.darwin-toolchain-inputs COMMAND $) set_tests_properties(nil.darwin-toolchain-inputs PROPERTIES LABELS "tools;project;rc" TIMEOUT 60) + if(APPLE) + add_executable(nil_darwin_native_inputs_tests tests/darwin_native_inputs_tests.cpp) + target_compile_features(nil_darwin_native_inputs_tests PRIVATE cxx_std_17) + add_dependencies(nil_darwin_native_inputs_tests nilruntime) + add_test(NAME nil.darwin-native-inputs COMMAND $ + "${NIL_NATIVE_LINKER_DRIVER}" "$") + set_tests_properties(nil.darwin-native-inputs PROPERTIES LABELS "tools;project;rc" TIMEOUT 180) + endif() # Test-only launcher for isolated editor integration runs. Reuse the # toolchain's bounded process/Job cleanup; never ship it in the SDK. diff --git a/docs/standard-library/ui.md b/docs/standard-library/ui.md index 9329a530..d2d6401d 100644 --- a/docs/standard-library/ui.md +++ b/docs/standard-library/ui.md @@ -72,9 +72,11 @@ possible GTK configuration. AppKit is the supported source implementation. On macOS, creation and polling must run on the process main thread, which is where AppKit requires its windows to be used. Ordinary interpreter, JIT and native entry routes use the main thread. The LLVM-ON display campaign verifies -the interpreter and native application cases on both architectures. DAP currently executes the -debuggee on a worker, so DAP with macOS UI is outside this port, and pausing a -text or JSON debug session does not automatically pump window events. AppKit +the interpreter and native application cases on both architectures. DAP now +keeps the VM and its UI resources on the process main thread across source +stops. Its real GTK ownership/event tests pass; Win32 and AppKit debugger +desktop qualification is being added separately from ordinary UI execution. +Paused debug sessions do not automatically pump window events. AppKit supplies no box or grid container, so, as on Win32, the provider owns the layout pass and sizes come from each control's fitting size. diff --git a/nilbuild/darwin_toolchain_inputs.hpp b/nilbuild/darwin_toolchain_inputs.hpp index 8fbeb4c0..4faaeeed 100644 --- a/nilbuild/darwin_toolchain_inputs.hpp +++ b/nilbuild/darwin_toolchain_inputs.hpp @@ -20,6 +20,14 @@ namespace nilbuild::darwin_toolchain_detail { namespace fs = std::filesystem; +inline void explain(std::string *reason, std::string message) { + if (!reason || !reason->empty()) return; + if (message.size() > 1024) message.resize(1024); + for (char &byte : message) + if (static_cast(byte) < 32 || byte == 127) byte = '?'; + *reason = std::move(message); +} + inline std::string trim(const std::string &text) { const auto first = text.find_first_not_of(" \r\n\t"); return first == std::string::npos ? "" : text.substr(first, text.find_last_not_of(" \r\n\t") - first + 1); @@ -315,14 +323,24 @@ class collector { std::set inspected_contexts; std::map image_descriptions; std::size_t entries = 0; + std::string *reason; + + bool fail(const char *message, const fs::path &path = {}) { + if (reason) explain(reason, std::string(message) + (path.empty() ? "" : ": " + path.string())); + return false; + } + std::optional> refuse(const char *message, const fs::path &path = {}) { + fail(message, path); return {}; + } fs::path host_path(const fs::path &path) const { return host / path.relative_path(); } bool add(const fs::path &path) { - if (!clean_path(path) || !regular(path) || files_seen.size() >= 65536) return false; + if (!clean_path(path) || !regular(path)) return fail("required input is not a normalized regular file", path); + if (files_seen.size() >= 65536) return fail("input file inventory exceeded 65536 entries", path); if (path.extension() == ".a") { std::ifstream archive(path, std::ios::binary); char magic[8]{}; archive.read(magic, sizeof(magic)); - if (archive.gcount() == 8 && std::string(magic, sizeof(magic)) == "!\n") return false; + if (archive.gcount() == 8 && std::string(magic, sizeof(magic)) == "!\n") return fail("thin archive has unmodeled external members", path); } files_seen.insert(path); return true; } @@ -330,24 +348,25 @@ class collector { std::ifstream stream(path, std::ios::binary); std::string text; char block[8192]; while (stream.read(block, sizeof(block)) || stream.gcount()) { text.append(block, static_cast(stream.gcount())); - if (text.size() > 32 * 1024 * 1024) return false; + if (text.size() > 32 * 1024 * 1024) return fail("TBD input exceeds 32 MiB", path); } - return !stream.bad() && supported_stub(text, &stub_references); + if (stream.bad()) return fail("could not read TBD input", path); + return supported_stub(text, &stub_references) || fail("unsupported TBD dependency profile", path); } bool tree(const fs::path &path, bool sdk, unsigned depth = 0) { std::error_code error; - if (depth > 32 || ++entries > 1000000) return false; + if (depth > 32 || ++entries > 1000000) return fail("SDK traversal exceeded depth or entry bound", path); const auto canonical = fs::canonical(path, error); - if (error || !active_directories.insert(canonical).second) return false; + if (error || !active_directories.insert(canonical).second) return fail("unreadable or cyclic SDK directory", path); fs::directory_iterator iterator(path, error), end; - if (error) return false; + if (error) return fail("could not enumerate SDK directory", path); for (; iterator != end; iterator.increment(error)) { - if (error || ++entries > 1000000) return false; + if (error || ++entries > 1000000) return fail("SDK enumeration failed or exceeded entry bound", path); const auto entry = iterator->path(); const auto status = fs::status(entry, error); - if (error) return false; + if (error) return fail("could not stat SDK entry", entry); if (fs::is_directory(status)) { - // Header/resource trees cannot be selected as linked library files. + // Header/module trees cannot be selected as linked library files. const auto name = entry.filename(); if (name == "Headers" || name == "PrivateHeaders" || name == "Modules") continue; if (!tree(entry, sdk, depth + 1)) return false; @@ -358,7 +377,7 @@ class collector { std::ifstream stream(entry, std::ios::binary); char prefix[16]{}; stream.read(prefix, sizeof(prefix)); const auto size = static_cast(stream.gcount()); - if (stream.bad()) return false; + if (stream.bad()) return fail("could not inspect extensionless SDK entry", entry); const std::string bytes(prefix, size); if (bytes.rfind("--- !tapi-tbd", 0) == 0) { if (!add(entry) || !read_stub(entry)) return false; @@ -366,34 +385,36 @@ class collector { (size >= 4 && (bytes.compare(0, 4, "\xfe\xed\xfa\xce", 4) == 0 || bytes.compare(0, 4, "\xce\xfa\xed\xfe", 4) == 0 || bytes.compare(0, 4, "\xfe\xed\xfa\xcf", 4) == 0 || bytes.compare(0, 4, "\xcf\xfa\xed\xfe", 4) == 0 || bytes.compare(0, 4, "\xca\xfe\xba\xbe", 4) == 0 || bytes.compare(0, 4, "\xbe\xba\xfe\xca", 4) == 0 || - bytes.compare(0, 4, "\xca\xfe\xba\xbf", 4) == 0 || bytes.compare(0, 4, "\xbf\xba\xfe\xca", 4) == 0))) return false; + bytes.compare(0, 4, "\xca\xfe\xba\xbf", 4) == 0 || bytes.compare(0, 4, "\xbf\xba\xfe\xca", 4) == 0))) return fail("unmodeled extensionless binary SDK input", entry); continue; // Non-linkable metadata such as framework PkgInfo. } if (!add(entry) || (extension == ".tbd" && !read_stub(entry))) return false; - if (sdk && (extension == ".dylib" || extension == ".so")) return false; // binary SDK stubs need load-command analysis + if (sdk && (extension == ".dylib" || extension == ".so")) return fail("unmodeled binary SDK input", entry); } - } else return false; + } else return fail("SDK entry is not a regular file or directory", entry); } active_directories.erase(canonical); - return !error; + return !error || fail("SDK enumeration failed", path); } bool image(const fs::path &path, const fs::path &executable_directory, unsigned depth = 0, const std::vector &inherited_rpaths = {}, std::vector *effective_rpaths = nullptr) { - if (depth > 32 || !add(path)) return false; + if (depth > 32) return fail("tool image dependency depth exceeded 32", path); + if (!add(path)) return false; std::error_code error; const auto physical = fs::canonical(path, error); - if (error || !add(physical)) return false; + if (error) return fail("could not resolve tool image", path); + if (!add(physical)) return false; auto cached = image_descriptions.find(path); if (cached == image_descriptions.end()) { const auto description = dependencies(path); - if (!description) return false; + if (!description) return fail("tool image inspection failed", path); cached = image_descriptions.emplace(path, *description).first; } const auto &description = cached->second; if (physical.parent_path() != path.parent_path()) { const auto loader_relative = [](const std::string &name) { return name.rfind("@loader_path", 0) == 0; }; if (std::any_of(description.rpaths.begin(), description.rpaths.end(), loader_relative) || - std::any_of(description.libraries.begin(), description.libraries.end(), loader_relative)) return false; + std::any_of(description.libraries.begin(), description.libraries.end(), loader_relative)) return fail("cross-directory image alias uses loader-relative paths", path); } // Compiler identity separately proves the active system dyld image and all // shared-cache backing bytes. Include an ordinary loader path as well when @@ -406,7 +427,7 @@ class collector { else if (name.rfind("@executable_path/", 0) == 0) selected = (executable_directory / name.substr(17)).lexically_normal(); else selected = name; if (!clean_path(selected) || (!inside(selected, plan.installed.parent_path()) && - !system_install_name(selected.string()))) return false; + !system_install_name(selected.string()))) return fail("tool rpath escapes modeled roots", selected); rpaths.push_back(selected); } rpaths.insert(rpaths.end(), inherited_rpaths.begin(), inherited_rpaths.end()); @@ -418,12 +439,12 @@ class collector { std::string context = path.string() + '\0' + executable_directory.string(); for (const auto &root : rpaths) context += '\0' + root.string(); if (!inspected_contexts.insert(context).second) return true; - if (inspected_contexts.size() > 1024) return false; + if (inspected_contexts.size() > 1024) return fail("tool image contexts exceeded 1024", path); for (const auto &name : description.libraries) { fs::path selected; if (name.rfind("@rpath/", 0) == 0) { const auto tail = fs::path(name.substr(7)); - if (tail.has_parent_path()) return false; + if (tail.has_parent_path()) return fail("tool rpath dependency has an unmodeled suffix", name); for (const auto &root : rpaths) { const auto candidate = root / tail; if (regular(candidate)) { selected = candidate; break; } @@ -431,10 +452,10 @@ class collector { // A dylib's install ID may use its parent's rpath and appear in otool // -L. It names this file, not an additional dependency. if (selected.empty() && path.filename() == tail && path.extension() == ".dylib") selected = path; - if (selected.empty()) return false; + if (selected.empty()) return fail("tool rpath dependency could not be resolved", name); } else if (name.rfind("@loader_path/", 0) == 0) { selected = (physical.parent_path() / name.substr(13)).lexically_normal(); - if (!inside(selected, plan.installed.parent_path())) return false; + if (!inside(selected, plan.installed.parent_path())) return fail("loader-relative dependency escapes toolchain", selected); } else if (system_install_name(name)) { selected = host_path(name); // Non-file system images are represented by the compiler's complete @@ -442,7 +463,7 @@ class collector { if (!regular(selected)) continue; } else { selected = name; - if (!clean_path(selected) || !inside(selected, plan.installed.parent_path())) return false; + if (!clean_path(selected) || !inside(selected, plan.installed.parent_path())) return fail("tool dependency escapes modeled roots", selected); } if (selected != path && !image(selected, executable_directory, depth + 1, rpaths)) return false; } @@ -452,12 +473,12 @@ class collector { const auto directory = plan.installed.parent_path() / "lib"; std::error_code error; fs::directory_iterator iterator(directory, error), end; - if (error) return false; + if (error) return fail("could not enumerate adjacent toolchain helpers", directory); for (; iterator != end; iterator.increment(error)) { - if (error || ++entries > 1000000) return false; + if (error || ++entries > 1000000) return fail("adjacent helper enumeration failed or exceeded entry bound", directory); if (iterator->path().extension() == ".dylib" && !add(iterator->path())) return false; } - return !error; + return !error || fail("adjacent helper enumeration failed", directory); } bool local_library_inputs() { if (!plan.local_libraries) return true; @@ -467,12 +488,12 @@ class collector { // back to the host. Inspect this choice anew on every collection. const auto sdk_directory = plan.sdk / "usr/local/lib"; const bool sdk_directory_exists = fs::is_directory(sdk_directory, error); - if (error && error != std::errc::no_such_file_or_directory) return false; + if (error && error != std::errc::no_such_file_or_directory) return fail("could not inspect SDK local library directory", sdk_directory); if (sdk_directory_exists) return true; // Complete SDK tree was collected. error.clear(); const auto directory = host_path("/usr/local/lib"); const bool host_directory_exists = fs::is_directory(directory, error); - if (error && error != std::errc::no_such_file_or_directory) return false; + if (error && error != std::errc::no_such_file_or_directory) return fail("could not inspect host local library directory", directory); if (!host_directory_exists) return true; // The default command's two -l roots and every architecture/OS SDK @@ -495,7 +516,7 @@ class collector { candidates.insert(leaf.string()); auto stub = leaf; stub.replace_extension(".tbd"); candidates.insert(stub.string()); } - if (candidates.size() > 131072) return false; + if (candidates.size() > 131072) return fail("local-library candidate bound exceeded", directory); bool changed = false; for (const auto &name : candidates) { if (!checked.insert(name).second) continue; @@ -503,7 +524,7 @@ class collector { const auto path = directory / name; error.clear(); const auto status = fs::status(path, error); if (error == std::errc::no_such_file_or_directory || status.type() == fs::file_type::not_found) continue; - if (error || !fs::is_regular_file(status)) return false; + if (error || !fs::is_regular_file(status)) return fail("local-library candidate is unreadable or not regular", path); // A selected ordinary binary/archive requires its own link-input // dependency closure; don't confuse that with the tools' dyld graph. // Admit only supported text stubs, including a stub behind an alias @@ -516,25 +537,26 @@ class collector { } public: collector(fs::path selected_driver, fs::path directory, fs::path selected_runtime, - link_plan selected_plan, probe inspect_image, fs::path host_root = "/") + link_plan selected_plan, probe inspect_image, fs::path host_root = "/", std::string *failure_reason = nullptr) : driver(std::move(selected_driver)), cwd(std::move(directory)), runtime(std::move(selected_runtime)), - host(std::move(host_root)), plan(std::move(selected_plan)), dependencies(std::move(inspect_image)) {} + host(std::move(host_root)), plan(std::move(selected_plan)), dependencies(std::move(inspect_image)), reason(failure_reason) {} std::optional> files() { std::error_code error; const auto actual_driver = fs::canonical(driver, error); - if (error) return {}; + if (error) return refuse("could not resolve selected driver", driver); const auto actual_compiler = fs::canonical(plan.installed / "clang++", error); - if (error) return {}; + if (error) return refuse("could not resolve selected compiler", plan.installed / "clang++"); const auto actual_linker = fs::canonical(plan.linker, error); - if (error) return {}; + if (error) return refuse("could not resolve selected linker", plan.linker); std::vector linker_rpaths; - if (!clean_path(driver) || !clean_path(cwd) || !add(runtime) || !image(driver, actual_driver.parent_path()) || + if (!clean_path(driver) || !clean_path(cwd)) return refuse("driver or working directory is not normalized"); + if (!add(runtime) || !image(driver, actual_driver.parent_path()) || !image(plan.installed / "clang++", actual_compiler.parent_path()) || !image(plan.linker, actual_linker.parent_path(), 0, {}, &linker_rpaths) || !image(plan.lto, actual_linker.parent_path(), 0, linker_rpaths) || !add(plan.compiler_rt)) return {}; if (!add(plan.sdk / "SDKSettings.json") || !add(host_path("/System/Library/CoreServices/SystemVersion.plist"))) return {}; - if (fs::exists(plan.sdk / "AppleInternal", error) || error) return {}; + if (fs::exists(plan.sdk / "AppleInternal", error) || error) return refuse("AppleInternal SDK profile or lookup failure", plan.sdk); for (const auto &directory : {plan.sdk / "usr/lib", plan.sdk / "System/Library/Frameworks"}) if (!tree(directory, true)) return {}; if (!adjacent_helpers()) return {}; @@ -545,7 +567,8 @@ class collector { for (const auto &relative : {"usr/local/lib", "Library/Frameworks", "System/Library/PrivateFrameworks"}) { const auto directory = plan.sdk / relative; const bool present = fs::exists(directory, error); - if (error || (present && !tree(directory, true))) return {}; + if (error) return refuse("could not inspect optional SDK directory", directory); + if (present && !tree(directory, true)) return {}; } if (!local_library_inputs()) return {}; // Absolute re-export lookup differs from standard -L/-F lookup: when the @@ -556,15 +579,15 @@ class collector { const auto relative = fs::path(name).relative_path(); auto sdk_stub = plan.sdk / relative; sdk_stub.replace_extension(".tbd"); if (regular(plan.sdk / relative)) { - if (!files_seen.count(plan.sdk / relative)) return {}; + if (!files_seen.count(plan.sdk / relative)) return refuse("SDK re-export missing from inventory", plan.sdk / relative); continue; } if (regular(sdk_stub)) { - if (!files_seen.count(sdk_stub)) return {}; + if (!files_seen.count(sdk_stub)) return refuse("SDK re-export stub missing from inventory", sdk_stub); continue; } auto host_stub = host_path(name); host_stub.replace_extension(".tbd"); - if (regular(host_path(name)) || regular(host_stub)) return {}; + if (regular(host_path(name)) || regular(host_stub)) return refuse("unmodeled ordinary host re-export fallback", name); } return std::vector(files_seen.begin(), files_seen.end()); } diff --git a/nilbuild/main.cpp b/nilbuild/main.cpp index 6d4047df..2aca39e0 100644 --- a/nilbuild/main.cpp +++ b/nilbuild/main.cpp @@ -701,7 +701,7 @@ bool compiler_files_complete(const json &report) { // driver for startup objects, libraries and its linker rather than guessing a // host installation layout. Missing identities disable reuse, not compilation. bool native_inputs(input_map &inputs, const json &report, const project &value, - nilbuild::digest_phase *phase = nullptr) { + nilbuild::digest_phase *phase = nullptr, std::string *reason = nullptr) { const auto runtime = report.object.find("nativeRuntime"), linker = report.object.find("nativeLinker"); if (runtime == report.object.end() || runtime->second.type != json::kind::string) return false; add_input(inputs, runtime->second.text, phase); @@ -716,7 +716,7 @@ bool native_inputs(input_map &inputs, const json &report, const project &value, } } const auto files = nilbuild::native_toolchain_files(linker->second.text, value.directory, - runtime->second.text, library_paths); + runtime->second.text, library_paths, reason); if (!files) return false; for (const auto &file : *files) add_input(inputs, file, phase); return true; @@ -872,7 +872,11 @@ int build(const project &root, const options &opts, const fs::path &compiler, co for (const auto &file : string_list(reports[i].at("compilerFiles"), "compilerFiles")) add_input(*task.inputs, file, preparation); } if (native) { - if (!native_inputs(task.native_files, reports[i], value, preparation)) task.cacheable = false; + std::string reason; + if (!native_inputs(task.native_files, reports[i], value, preparation, &reason)) { + task.cacheable = false; + if (!reason.empty()) std::cerr << "nilbuild: native reuse unavailable: " << reason << '\n'; + } task.inputs->insert(task.native_files.begin(), task.native_files.end()); } } diff --git a/nilbuild/toolchain_inputs.hpp b/nilbuild/toolchain_inputs.hpp index bc693339..41454172 100644 --- a/nilbuild/toolchain_inputs.hpp +++ b/nilbuild/toolchain_inputs.hpp @@ -241,7 +241,8 @@ class collector { inline std::optional> native_toolchain_files( const std::filesystem::path &driver, const std::filesystem::path &directory, const std::filesystem::path &runtime = {}, - const std::vector &library_paths = {}) { + const std::vector &library_paths = {}, std::string *reason = nullptr) { + if (reason) reason->clear(); #if defined(__linux__) (void)runtime; (void)library_paths; return toolchain_detail::collector(driver, directory).files(); @@ -263,9 +264,12 @@ inline std::optional> native_toolchain_files( return windows_toolchain_detail::collector(driver, directory, runtime, library_paths, lib ? lib : "", inspect).files(); #elif defined(__APPLE__) + const auto unavailable = [&](std::string message) -> std::optional> { + darwin_toolchain_detail::explain(reason, std::move(message)); return {}; + }; // nilbuild currently forwards no -l/--link options. Expanding that command // surface requires expanding this closure before admitting those projects. - if (!library_paths.empty() || runtime.empty()) return std::nullopt; + if (!library_paths.empty() || runtime.empty()) return unavailable("custom library paths or missing native runtime"); bool local_library_environment = false; for (char **entry = *_NSGetEnviron(); entry && *entry; ++entry) { const std::string item(*entry); @@ -276,7 +280,7 @@ inline std::optional> native_toolchain_files( // This exact search profile is modeled below and already appears in the // build key. Empty components/other directories are intentionally not // inferred or sanitized; they require a broader search closure. - if (item.substr(equal + 1) != "/usr/local/lib") return std::nullopt; + if (item.substr(equal + 1) != "/usr/local/lib") return unavailable("unmodeled LIBRARY_PATH search profile"); local_library_environment = true; continue; } if (name.rfind("DYLD_", 0) == 0 || name.rfind("LD_", 0) == 0 || name.rfind("CCC_", 0) == 0 || @@ -284,14 +288,16 @@ inline std::optional> native_toolchain_files( name == "GCC_EXEC_PREFIX" || name == "CPATH" || name == "CPLUS_INCLUDE_PATH" || name == "C_INCLUDE_PATH" || name == "OBJC_INCLUDE_PATH" || name == "ARCHFLAGS" || name == "IPHONEOS_DEPLOYMENT_TARGET" || name == "TVOS_DEPLOYMENT_TARGET" || - name == "WATCHOS_DEPLOYMENT_TARGET") return std::nullopt; + name == "WATCHOS_DEPLOYMENT_TARGET") return unavailable("unmodeled tool-selection environment variable: " + name); } const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(120); unsigned queries = 0; const auto query = [&](const std::vector &command, bool driver_plan = false) -> std::optional { const auto now = std::chrono::steady_clock::now(); - if (++queries > 128 || now >= deadline) return {}; + if (++queries > 128 || now >= deadline) { + darwin_toolchain_detail::explain(reason, "toolchain inspection exceeded query or time budget"); return {}; + } process_options options; options.close_stdin = true; options.cleanup_descendants = true; options.time_limit = std::min(std::chrono::milliseconds(15000), @@ -299,35 +305,46 @@ inline std::optional> native_toolchain_files( options.output_limit = 4 * 1024 * 1024; options.total_output_limit = 4 * 1024 * 1024; const auto result = execute_many({{command, directory}}, 1, options).front(); if (result.cancelled) throw process_cancelled(result.status); - if (result.status || result.timed_out || result.output_limited || (!driver_plan && !result.error.empty())) return {}; + if (result.status || result.timed_out || result.output_limited || (!driver_plan && !result.error.empty())) { + darwin_toolchain_detail::explain(reason, "toolchain query failed (status=" + std::to_string(result.status) + + ", timeout=" + std::to_string(result.timed_out) + ", output_limit=" + std::to_string(result.output_limited) + + "): " + command.front() + (driver_plan ? " [driver link plan]" : " " + command.back()) + + (result.error.empty() ? "" : "; stderr=" + result.error.substr(0, 512))); + return {}; + } return driver_plan ? result.output + result.error : result.output; }; const auto output = query({driver.string(), "-###", "-pthread", "-o", "/dev/null", runtime.string(), "-framework", "Cocoa", "-framework", "CoreGraphics"}, true); if (!output) return {}; const auto plan = darwin_toolchain_detail::parse_plan(*output, runtime); - if (!plan || (local_library_environment && !plan->local_libraries)) return {}; + if (!plan) return unavailable("selected Apple driver link plan is outside the supported profile"); + if (local_library_environment && !plan->local_libraries) return unavailable("LIBRARY_PATH was not represented by the modeled link plan"); #if defined(__aarch64__) || defined(__arm64__) - if (plan->architecture != "arm64") return {}; + if (plan->architecture != "arm64") return unavailable("link plan architecture differs from running ARM64 host"); #elif defined(__x86_64__) - if (plan->architecture != "x86_64") return {}; + if (plan->architecture != "x86_64") return unavailable("link plan architecture differs from running x86-64 host"); #else - return {}; + return unavailable("unsupported running host architecture"); #endif // Archives can carry hidden -l/-framework directives in object load commands. // Keep that unsupported until their names join the modeled search roots. for (const auto &archive : {runtime, plan->compiler_rt}) { const auto commands = query({"/usr/bin/otool", "-arch", plan->architecture, "-l", archive.string()}); - if (!commands || !darwin_toolchain_detail::supported_archive_commands(*commands)) return {}; + if (!commands) return {}; + if (!darwin_toolchain_detail::supported_archive_commands(*commands)) + return unavailable("native archive has unmodeled autolinks or empty inspection: " + archive.string()); } const auto inspect = [&](const std::filesystem::path &image) -> std::optional { const auto libraries = query({"/usr/bin/otool", "-arch", plan->architecture, "-L", image.string()}); const auto commands = query({"/usr/bin/otool", "-arch", plan->architecture, "-l", image.string()}); if (!libraries || !commands) return {}; - return darwin_toolchain_detail::parse_image(image, *libraries, *commands); + const auto description = darwin_toolchain_detail::parse_image(image, *libraries, *commands); + if (!description) darwin_toolchain_detail::explain(reason, "unsupported otool image/load-command output: " + image.string()); + return description; }; - return darwin_toolchain_detail::collector(driver, directory, runtime, *plan, inspect).files(); + return darwin_toolchain_detail::collector(driver, directory, runtime, *plan, inspect, "/", reason).files(); #else (void)driver; (void)directory; (void)runtime; (void)library_paths; return std::nullopt; diff --git a/nildebugger/CMakeLists.txt b/nildebugger/CMakeLists.txt index d539ecb9..901e77d8 100644 --- a/nildebugger/CMakeLists.txt +++ b/nildebugger/CMakeLists.txt @@ -62,6 +62,19 @@ if(BUILD_TESTING) "$" "$" ) set_tests_properties(nildebugger.dap-protocol PROPERTIES TIMEOUT 90 LABELS "debugger;rc") + add_test(NAME nildebugger.dap-lifecycle + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/dap_lifecycle.py" + --debugger "$" --compiler "$" + --output-parent "${CMAKE_BINARY_DIR}/test-evidence") + set_tests_properties(nildebugger.dap-lifecycle PROPERTIES TIMEOUT 180 LABELS "debugger;rc") + add_executable(nildebugger_dap_owner_tests + tests/dap_owner_tests.cpp debugger_core.cpp dap.cpp ../nillsp/json_rpc.cpp) + target_compile_features(nildebugger_dap_owner_tests PRIVATE cxx_std_17) + target_include_directories(nildebugger_dap_owner_tests PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}" "${CMAKE_CURRENT_SOURCE_DIR}/../nillsp") + target_link_libraries(nildebugger_dap_owner_tests PRIVATE nilcompiler_core Threads::Threads) + add_test(NAME nildebugger.dap-owner COMMAND nildebugger_dap_owner_tests) + set_tests_properties(nildebugger.dap-owner PROPERTIES TIMEOUT 20 LABELS "debugger;rc") add_executable(nildebugger_source_metadata_tests tests/source_metadata_tests.cpp) target_compile_features(nildebugger_source_metadata_tests PRIVATE cxx_std_17) target_link_libraries(nildebugger_source_metadata_tests PRIVATE nilcompiler_core) diff --git a/nildebugger/README.md b/nildebugger/README.md index 269f0838..94790518 100644 --- a/nildebugger/README.md +++ b/nildebugger/README.md @@ -36,12 +36,18 @@ Only one launch is accepted per adapter process. It exposes the main VM as one thread; isolated parallel workers finish before their parent instruction returns and are not separately step-debugged. -DAP execution runs on one worker; the control thread never inspects a running -VM. `pause` and `terminate` request a cooperative stop between instructions, so +DAP keeps VM creation, execution, inspection and disposal on the process main +thread. A separate protocol reader accepts control requests while the VM runs; +it never inspects running VM state. Source stops preserve the UI provider's +owning thread. `pause` and `terminate` request a cooperative stop between instructions, so blocking foreign calls, sleeps, and a running parallel-map instruction must return before they take effect. Program stdout is streamed incrementally as bounded output events, including while execution is running. The adapter rejects oversized or malformed framing (8 KiB headers, 1 MiB requests). +Disconnect, EOF and complete malformed frames stop the owned session without +requiring a client to close stdin after disconnect. An adapter transport failure +gets a best-effort terminal notice; an open input stream still belongs to the +client until disconnect or EOF. Paused sessions do not pump desktop events. Native-executable attach/DWARF, data breakpoints, hit counts, logpoints, variable mutation, reverse execution, arbitrary expression execution, and interactive diff --git a/nildebugger/dap.cpp b/nildebugger/dap.cpp index 281d24df..482bf11b 100644 --- a/nildebugger/dap.cpp +++ b/nildebugger/dap.cpp @@ -5,6 +5,9 @@ #include #include #include +#include +#include +#include #include #include #include @@ -130,12 +133,11 @@ class Adapter { debugger_.get_vm().setOutputStream(program_output_); debugger_.get_vm().setInputStream(program_input_); } - ~Adapter() { - debugger_.request_pause(); - if (worker_.joinable()) worker_.join(); - } + enum class Dispatch { handled, main_thread, disconnect }; - bool dispatch(const Json& request) { + // Only the framing/control thread passes control_only. Every operation + // below that flag's return point touches the stopped VM on process main. + Dispatch dispatch(const Json& request, bool control_only) { std::lock_guard lock(session_mutex_); int seq = -1; std::string command; @@ -158,7 +160,7 @@ class Adapter { {"supportsConditionalBreakpoints", true}, {"supportsEvaluateForHovers", true}, {"supportsTerminateRequest", true}})); - return true; + return Dispatch::handled; } if (!initialized_) throw std::runtime_error("initialize is required"); if (command == "disconnect" || command == "terminate") { @@ -166,16 +168,18 @@ class Adapter { debugger_.request_pause(); response(seq, command); if (!running_.load()) event("terminated"); - return command != "disconnect"; + return command == "disconnect" ? Dispatch::disconnect : Dispatch::handled; } if (command == "pause") { require_thread(args); if (!running_.load()) throw std::runtime_error("the program is not running"); debugger_.request_pause(); response(seq, command); - return true; + return Dispatch::handled; } + if (terminate_requested_.load()) throw std::runtime_error("the debug session has terminated"); if (running_.load()) throw std::runtime_error("pause the program before inspection or mutation"); + if (control_only) return Dispatch::main_thread; if (command == "launch") { if (launched_) throw std::runtime_error("one launch is supported per adapter process"); const auto program = string_arg(args, "program"); @@ -303,7 +307,7 @@ class Adapter { const int depth = frame_depth(args); if (!frame_is_mapped(depth)) { response(seq, command, json::object({{"scopes", Json::Array{}}})); - return true; + return Dispatch::handled; } Json::Object values; for (const auto& local : debugger_.get_source_locals(depth)) { @@ -349,7 +353,7 @@ class Adapter { } catch (const std::exception& error) { response(seq, command, json::object({}), false, error.what()); } - return true; + return Dispatch::handled; } private: @@ -413,49 +417,78 @@ class Adapter { void start(const std::string& action, const std::function& acknowledge) { if (next_frame_ == std::numeric_limits::max()) throw std::runtime_error("frame identifier space exhausted; start a new debug session"); - if (worker_.joinable()) worker_.join(); variables_.clear(); frame_depths_.clear(); depth_frames_.clear(); debugger_.clear_pause_request(); + pending_action_ = action; running_.store(true); try { - worker_ = std::thread([this, action] { - // dispatch holds this lock until the worker is created and its - // successful response is sent. No debuggee output precedes that - // response, and thread-start failure yields only one failure reply. - { std::lock_guard starting(session_mutex_); } - if (action == "entry") debugger_.stop_at_entry(); - else if (action == "next") debugger_.step_over(); - else if (action == "stepIn") debugger_.step_into(); - else if (action == "stepOut") debugger_.step_out(); - else debugger_.continue_cooperatively(); - std::lock_guard lock(session_mutex_); - running_.store(false); - if (terminate_requested_.load()) { - debugger_.detach(); - event("terminated"); - } else if (debugger_.get_state() == ExecutionState::Terminated) { - const auto requested = debugger_.get_vm().getRequestedExitCode(); - event("exited", json::object({{"exitCode", int64_t(requested ? *requested : debugger_.get_vm().getProgramExitCode())}})); - event("terminated"); - } else { - auto body = json::object({{"reason", debugger_.last_stop_reason()}, - {"threadId", int64_t(1)}, {"allThreadsStopped", true}}); - if (!debugger_.last_error().empty()) body.mut_object()["text"] = safe_text(debugger_.last_error()); - if (debugger_.last_breakpoint_id() > 0 && debugger_.last_stop_reason() == "breakpoint") - body.mut_object()["hitBreakpointIds"] = json::array({Json(int64_t(debugger_.last_breakpoint_id()))}); - event("stopped", std::move(body)); - } - }); + acknowledge(); + event("continued", json::object({{"threadId", int64_t(1)}, {"allThreadsContinued", true}})); } catch (...) { + pending_action_.clear(); running_.store(false); throw; } - acknowledge(); - event("continued", json::object({{"threadId", int64_t(1)}, {"allThreadsContinued", true}})); } +public: + // The reader may request only this atomic instruction-boundary stop. + // Debugger construction, execution, inspection and destruction stay on the + // calling process main thread, as required by AppKit and UI handle owners. + void request_stop() { debugger_.request_pause(); } + + void execute_pending() { + if (pending_action_.empty()) return; + const auto action = std::move(pending_action_); + pending_action_.clear(); + if (action == "entry") debugger_.stop_at_entry(); + else if (action == "next") debugger_.step_over(); + else if (action == "stepIn") debugger_.step_into(); + else if (action == "stepOut") debugger_.step_out(); + else debugger_.continue_cooperatively(); + std::lock_guard lock(session_mutex_); + running_.store(false); + if (terminate_requested_.load()) { + debugger_.detach(); + event("terminated"); + } else if (debugger_.get_state() == ExecutionState::Terminated) { + const auto requested = debugger_.get_vm().getRequestedExitCode(); + event("exited", json::object({{"exitCode", int64_t(requested ? *requested : debugger_.get_vm().getProgramExitCode())}})); + event("terminated"); + } else { + auto body = json::object({{"reason", debugger_.last_stop_reason()}, + {"threadId", int64_t(1)}, {"allThreadsStopped", true}}); + if (!debugger_.last_error().empty()) body.mut_object()["text"] = safe_text(debugger_.last_error()); + if (debugger_.last_breakpoint_id() > 0 && debugger_.last_stop_reason() == "breakpoint") + body.mut_object()["hitBreakpointIds"] = json::array({Json(int64_t(debugger_.last_breakpoint_id()))}); + event("stopped", std::move(body)); + } + } + + void fail_execution() { + std::lock_guard lock(session_mutex_); + pending_action_.clear(); + running_.store(false); + terminate_requested_.store(true); + debugger_.detach(); + if (!failure_reported_) { + failure_reported_ = true; + // Ordinary VM faults are stopped events handled above. This path + // covers adapter/transport failures. Make a best-effort terminal + // notice, but a broken output stream must not prevent the reader + // from accepting disconnect or observing EOF on its own thread. + try { + event("output", json::object({{"category", "stderr"}, + {"output", "Debug adapter failed; disconnect and restart this session.\n"}})); + } catch (...) {} + try { event("terminated"); } catch (...) {} + } + } + +private: + std::ostream& output_; std::mutex output_mutex_; std::mutex session_mutex_; @@ -464,13 +497,14 @@ class Adapter { OutputBuffer output_buffer_; std::ostream program_output_; std::istringstream program_input_; // explicit EOF; control stdin is never debuggee input - std::thread worker_; + std::string pending_action_; // process-main-thread only std::atomic running_{false}; std::atomic terminate_requested_{false}; bool initialized_ = false; bool launched_ = false; bool configured_ = false; bool stop_on_entry_ = true; + bool failure_reported_ = false; int next_frame_ = 1; int next_variable_ = 1; std::map frame_depths_; @@ -483,8 +517,70 @@ class Adapter { int run_dap(std::istream& input, std::ostream& output) { try { Adapter adapter(output); - Json request; - while (read_request(input, request)) if (!adapter.dispatch(request)) break; + std::mutex mailbox_mutex; + std::condition_variable mailbox_changed; + std::optional pending; + bool reader_done = false; + std::exception_ptr reader_error, execution_error; + // The reader owns normal transport shutdown. It never waits for more + // input after disconnect, and EOF/malformed framing always requests a + // stop before main joins it. There is at most one bounded request in + // flight, so a pipelined client cannot build an unbounded work queue. + std::thread reader([&] { + try { + Json request; + while (read_request(input, request)) { + const auto result = adapter.dispatch(request, true); + if (result == Adapter::Dispatch::disconnect) break; + if (result != Adapter::Dispatch::main_thread) continue; + std::unique_lock lock(mailbox_mutex); + pending = std::move(request); + mailbox_changed.notify_one(); + mailbox_changed.wait(lock, [&] { return !pending; }); + } + } catch (...) { + reader_error = std::current_exception(); + } + adapter.request_stop(); + { + std::lock_guard lock(mailbox_mutex); + reader_done = true; + } + mailbox_changed.notify_one(); + }); + for (;;) { + Json request; + { + std::unique_lock lock(mailbox_mutex); + mailbox_changed.wait(lock, [&] { return pending || reader_done; }); + if (!pending) break; + request = std::move(*pending); + } + try { + adapter.dispatch(request, false); + } catch (...) { + if (!execution_error) execution_error = std::current_exception(); + adapter.fail_execution(); + } + { + std::lock_guard lock(mailbox_mutex); + pending.reset(); + } + // Release the reader before executing a continue/step. It can now + // accept pause/terminate while main runs real VM instructions. + mailbox_changed.notify_one(); + try { + adapter.execute_pending(); + } catch (...) { + if (!execution_error) execution_error = std::current_exception(); + adapter.fail_execution(); + // Keep servicing the transport until its reader owns shutdown; + // never unwind and join a thread blocked on an open stdin. + } + } + reader.join(); + if (reader_error) std::rethrow_exception(reader_error); + if (execution_error) std::rethrow_exception(execution_error); return 0; } catch (const std::exception& error) { // stdout is reserved for valid DAP frames even for malformed input. diff --git a/nildebugger/tests/dap_contract_support.py b/nildebugger/tests/dap_contract_support.py new file mode 100644 index 00000000..938e80f3 --- /dev/null +++ b/nildebugger/tests/dap_contract_support.py @@ -0,0 +1,208 @@ +"""Bounded, retained DAP evidence with exact child-tree ownership.""" +from __future__ import annotations + +import hashlib +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import time + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / 'validation/ny-3d/scripts')) +from debugger_process import CaptureBudget, OwnedProcess, PipeReader, write_pipe + + +def output_directory(parent, prefix): + if parent: + parent = parent.resolve() + parent.mkdir(parents=True, exist_ok=True) + return Path(tempfile.mkdtemp(prefix=prefix, dir=parent)) + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def environment(): + return {key: value for key, value in os.environ.items() + if not key.startswith(('NY_', 'NIL_'))} + + +def compile_program(compiler, source, output, record, env, *, binary=False, nilstd=None): + command = [compiler, '--quiet', '--no-bundle', '--debug-info', '--no-stdlib'] + if nilstd: + command += ['--sdk-path', nilstd] + if binary: + command += ['--emit-binary-nil'] + command += ['--output', output, source] + command = [str(part) for part in command] + owner = OwnedProcess(command, cwd=output.parent, env=env, stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdout, stderr = owner.communicate(timeout=90) + output.with_suffix('.compile.stdout').write_bytes(stdout) + output.with_suffix('.compile.stderr').write_bytes(stderr) + record.append({'argv': command, 'exit_code': owner.process.returncode, + 'source_sha256': digest(source), 'cleanup_verified': True}) + assert owner.process.returncode == 0, (command, stdout, stderr) + assert output.read_bytes().startswith(b'NIL\0') == binary + + +class Client: + def __init__(self, debugger, root, record, env, timeout=45): + self.record = record + record.update(status='FAIL', natural_exit=False, cleanup_verified=False, requests=[]) + self.deadline = time.monotonic() + timeout + self.owner = OwnedProcess([str(debugger), '--dap'], cwd=root, env=env, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + self.budget = CaptureBudget(self.owner.terminate, maximum=4 * 1024 * 1024) + self.stdout = PipeReader(self.owner.process.stdout, self.budget) + self.wire = (root / (record['name'] + '.wire')).open('wb') + self.errors = (root / (record['name'] + '.stderr')).open('wb') + self.stderr = PipeReader(self.owner.process.stderr, self.budget, self.errors) + self.buffer = b'' + self.pending = [] + self.seq = 0 + self.last_sequence = 0 + self.observed = [] + self.stdout_eof = False + self.finished = False + + def remaining(self, maximum=8): + remaining = min(maximum, self.deadline - time.monotonic()) + assert remaining > 0, 'DAP session deadline expired' + self.budget.check() + return remaining + + def send(self, data): + write_pipe(self.owner.process.stdin, data, self.remaining(), self.owner.terminate) + + def _read(self): + chunk = self.stdout.read(self.remaining()) + self.wire.write(chunk) + self.wire.flush() + self.stdout_eof = not chunk + assert chunk, 'adapter stdout ended before the expected frame' + self.buffer += chunk + + def frame(self): + while b'\r\n\r\n' not in self.buffer: + assert len(self.buffer) <= 8192, 'oversized DAP response header' + self._read() + header, self.buffer = self.buffer.split(b'\r\n\r\n', 1) + assert header.startswith(b'Content-Length: ') and b'\r\n' not in header, header + size = int(header[16:]) + assert 0 < size <= 1024 * 1024 + while len(self.buffer) < size: + self._read() + body, self.buffer = self.buffer[:size], self.buffer[size:] + message = json.loads(body) + assert message['seq'] > self.last_sequence, message + self.last_sequence = message['seq'] + assert len(self.observed) < 4096, 'too many adapter frames' + self.observed.append({key: message[key] for key in ('seq', 'type', 'event', 'request_seq') if key in message}) + return message + + def wait(self, predicate): + while True: + self.remaining() + for index, message in enumerate(self.pending): + if predicate(message): + return self.pending.pop(index) + assert len(self.pending) < 4096, 'too many unconsumed adapter messages' + self.pending.append(self.frame()) + + def request(self, command, arguments=None, success=True): + self.seq += 1 + seq = self.seq + body = json.dumps({'seq': seq, 'type': 'request', 'command': command, + 'arguments': arguments or {}}).encode() + first_frame = len(self.observed) + self.send(f'Content-Length: {len(body)}\r\n\r\n'.encode() + body) + response = self.wait(lambda message: message.get('type') == 'response' and + message.get('request_seq') == seq) + self.record['requests'].append({'command': command, 'success': response.get('success')}) + assert response['success'] is success, response + if success and command in ('configurationDone', 'continue', 'next', 'stepIn', 'stepOut'): + # Searching a message queue alone would hide output emitted before + # the acknowledgement. Verify the actual observed wire sequence. + preceding = self.observed[first_frame:] + assert len(preceding) == 1 and preceding[0].get('request_seq') == seq, preceding + continued = self.frame() + assert continued.get('event') == 'continued', continued + self.record['requests'][-1]['start_wire_order_verified'] = True + return response['body'] + + def event(self, name): + return self.wait(lambda message: message.get('type') == 'event' and + message.get('event') == name)['body'] + + def initialize(self, program, *, stop_on_entry=False): + self.request('initialize', {'adapterID': 'nil-contract', 'pathFormat': 'path'}) + self.request('launch', {'program': str(program), 'stopOnEntry': stop_on_entry}) + self.event('initialized') + + def evaluate(self, name): + return self.request('evaluate', {'expression': name})['result'] + + def stopped(self, reason): + stopped = self.event('stopped') + assert stopped['reason'] == reason, stopped + return self.request('stackTrace', {'threadId': 1})['stackFrames'][0] + + def natural_exit(self, expected=0): + # Do not close stdin or signal the child to make this oracle pass. + deadline = time.monotonic() + self.remaining(5) + while self.owner.poll() is None: + assert time.monotonic() < deadline, 'adapter required forced cleanup instead of natural shutdown' + self.budget.check() + time.sleep(.01) + self.record['stdin_open_at_exit'] = not self.owner.process.stdin.closed + self.record['natural_exit'] = True + self.owner.wait(max(.01, deadline - time.monotonic())) + self.record['exit_code'] = self.owner.process.returncode + assert self.owner.process.returncode == expected, self.record + self.record['status'] = 'PASS' + + def close(self): + if self.finished: + return + self.finished = True + try: + if self.owner.poll() is None: + self.owner.kill() + else: + self.owner.wait(5) + self.owner.finish() + self.record['cleanup_verified'] = True + finally: + if not self.owner.process.stdin.closed: + self.owner.process.stdin.close() + try: + # Preserve remaining framed bytes too; the reader is bounded + # even when a failed assertion stops consuming protocol events. + while not self.stdout_eof: + chunk = self.stdout.read(5) + self.wire.write(chunk) + self.stdout_eof = not chunk + finally: + self.stdout.close() + self.stderr.close() + self.wire.close() + self.errors.close() + self.budget.check() + + +def session(debugger, root, report, name, env, action): + record = {'name': name} + report['sessions'].append(record) + client = Client(debugger, root, record, env) + try: + action(client) + except BaseException as error: + record.update(status='FAIL', error=str(error)) + raise + finally: + client.close() diff --git a/nildebugger/tests/dap_desktop.py b/nildebugger/tests/dap_desktop.py new file mode 100644 index 00000000..5ec74db9 --- /dev/null +++ b/nildebugger/tests/dap_desktop.py @@ -0,0 +1,167 @@ +#!/usr/bin/env python3 +"""Real UI handles and queued events survive DAP source stops on process main. + +This requires a usable desktop provider. Linux owns a short-lived private Xvfb; +Windows/macOS use an explicitly selected interactive test session. Programmatic +activation targets only the fixture's own button; no external input is sent. +Paused-window responsiveness or physical keyboard injection is not asserted. +""" +from __future__ import annotations + +import argparse +from contextlib import nullcontext +import importlib.util +import json +from pathlib import Path +import sys + +import dap_contract_support as support + +SOURCE = '''book dap-desktop +read nil.std.ui +read nil.std.testing +read nil.std.threading +read nil.std.core.result +read nil.std.core.option + +next-event receives surface as Window returns Event begin paragraph + variable attempts as number is 0 + repeat while attempts is less than 200 + value pending is require-ok of PendingEvent, text with (poll-event with surface), "poll owned window" + consider pending + when Option.Some holding notification then give notification + when Option.None then evaluate sleep-ms with 5 + end consider + set attempts to attempts plus 1 + end repeat + evaluate panic with "owned button activation did not produce an event" +end paragraph + +main returns override integer begin paragraph + value surface is require-ok of Window, text with (create-window with "NIL DAP owned desktop", 360, 180), "create window on process main" + value button is require-ok of Widget, text with (create-button with (window-root with surface), "before"), "create button" + evaluate require-ok of boolean, text with (show-window with surface), "show window" + value created is surface.handle + evaluate require-ok of boolean, text with (set-widget-text with button, "after"), "mutate retained button" + value after is require-ok of text, text with (widget-text with button), "read retained button" + when after does not equal "after" then give 71 end when + evaluate require-ok of boolean, text with (activate-widget with button), "activate retained button" + value queued is button.handle + value click is next-event with surface + value matched is event-for with click, button + value clicked is is-click with click + when not matched then give 72 end when + when not clicked then give 73 end when + value snapshot is click.window + evaluate require-ok of boolean, text with (set-widget-text with button, "final"), "mutate after snapshot" + value finaltext is require-ok of text, text with (widget-text with button), "read after snapshot" + when finaltext does not equal "final" then give 74 end when + evaluate require-ok of boolean, text with (close-window with surface), "close owned window" + value retained is snapshot.handle + when retained does not equal created then give 75 end when + say "desktop-owner-pass" + give 0 +end paragraph +''' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--debugger', type=Path, required=True) + parser.add_argument('--compiler', type=Path, required=True) + parser.add_argument('--nilstd', type=Path, required=True) + parser.add_argument('--xvfb', type=Path) + parser.add_argument('--output-parent', type=Path) + args = parser.parse_args() + debugger, compiler, nilstd = (path.resolve(strict=True) for path in (args.debugger, args.compiler, args.nilstd)) + root = support.output_directory(args.output_parent, 'nil-dap-desktop-') + report = {'schema': 'nil.dap-desktop.v1', 'status': 'FAIL', 'commands': [], 'sessions': [], + 'tools': {str(path): support.digest(path) for path in (debugger, compiler)}, + 'scope': 'VM UI ownership and event retention across stops; no paused host pumping or external input'} + try: + display = nullcontext(support.environment()) + if sys.platform.startswith('linux'): + assert args.xvfb, '--xvfb is required for the owned Linux desktop' + spec = importlib.util.spec_from_file_location('nil_dap_private_display', support.ROOT / 'tests/tools/ui_display_contract.py') + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + display = module.isolated_display(args.xvfb.resolve(strict=True)) + source = root / 'desktop.matrix' + source.write_text(SOURCE, encoding='utf-8', newline='\n') + lines = {marker: next(index for index, line in enumerate(SOURCE.splitlines(), 1) if marker in line) + for marker in ('value created', 'value queued', 'value snapshot', 'value retained')} + with display as env: + # Keep the selected private display, while avoiding incidental SDK + # runtime overrides from the invoking developer shell. + env = {key: value for key, value in env.items() if not key.startswith(('NY_', 'NIL_'))} + for binary in (False, True): + encoding = 'binary' if binary else 'text' + program = root / (encoding + '.nil') + support.compile_program(compiler, source, program, report['commands'], env, + binary=binary, nilstd=nilstd) + + def initialize(client): + client.initialize(program) + points = client.request('setBreakpoints', {'source': {'path': str(source)}, + 'breakpoints': [{'line': line} for line in lines.values()]})['breakpoints'] + assert len(points) == 4 and all(point['verified'] for point in points), points + client.request('configurationDone') + frame = client.stopped('breakpoint') + assert frame['line'] == lines['value created'], frame + surface, button = int(client.evaluate('surface.handle')), int(client.evaluate('button.handle')) + assert surface > 0 and button > 0 and surface != button + client.record['window_handle'], client.record['button_handle'] = surface, button + return surface, button + + def retained_handles(client): + surface, button = initialize(client) + # Each next executes on the same owner, including calls into + # the real provider using handles created before the stop. + for unused in range(3): + client.request('next', {'threadId': 1}) + client.stopped('step') + assert int(client.evaluate('surface.handle')) == surface + assert int(client.evaluate('button.handle')) == button + assert client.evaluate('after') == '"after"' + client.request('continue', {'threadId': 1}) + frame = client.stopped('breakpoint') + assert frame['line'] == lines['value queued'], frame + assert int(client.evaluate('button.handle')) == button + client.request('continue', {'threadId': 1}) + frame = client.stopped('breakpoint') + assert frame['line'] == lines['value snapshot'], frame + flags = {name: client.evaluate(name) for name in ('matched', 'clicked')} + assert flags == {'matched': '1', 'clicked': '1'}, flags + client.request('continue', {'threadId': 1}) + frame = client.stopped('breakpoint') + assert frame['line'] == lines['value retained'], frame + assert int(client.evaluate('snapshot.handle')) == surface + assert client.evaluate('finaltext') == '"final"' + client.request('continue', {'threadId': 1}) + assert 'desktop-owner-pass' in client.event('output')['output'] + assert client.event('exited')['exitCode'] == 0 + client.event('terminated') + client.request('disconnect') + client.natural_exit() + assert client.record['stdin_open_at_exit'] + client.record['queued_event_retained'] = True + + def stopped_disconnect(client): + initialize(client) + # VM destruction must dispose the still-open window on main. + client.request('disconnect') + client.natural_exit() + assert client.record['stdin_open_at_exit'] + + support.session(debugger, root, report, encoding + '-retention', env, retained_handles) + support.session(debugger, root, report, encoding + '-disconnect', env, stopped_disconnect) + report['status'] = 'PASS' + except BaseException as error: + report['error'] = str(error) + (root / 'report.json').write_text(json.dumps(report, indent=2) + '\n') + print(report['status'] + ': ' + str(root / 'report.json')) + return 0 if report['status'] == 'PASS' else 1 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/nildebugger/tests/dap_lifecycle.py b/nildebugger/tests/dap_lifecycle.py new file mode 100644 index 00000000..9688794e --- /dev/null +++ b/nildebugger/tests/dap_lifecycle.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +"""DAP pause/resume and transport shutdown must preserve one execution owner.""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +import dap_contract_support as support + +SOURCE = '''fn main() -> i32 + let mut value: i64 = 0 + println("ready") + while true + value = value + 1 + end while + return 0 +end fn +''' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--debugger', type=Path, required=True) + parser.add_argument('--compiler', type=Path, required=True) + parser.add_argument('--output-parent', type=Path) + args = parser.parse_args() + debugger, compiler = args.debugger.resolve(strict=True), args.compiler.resolve(strict=True) + root = support.output_directory(args.output_parent, 'nil-dap-lifecycle-') + report = {'schema': 'nil.dap-lifecycle.v1', 'status': 'FAIL', 'commands': [], 'sessions': [], + 'tools': {str(path): support.digest(path) for path in (debugger, compiler)}} + env = support.environment() + try: + source = root / 'loop.ny' + source.write_text(SOURCE, encoding='utf-8', newline='\n') + for binary in (False, True): + name = 'binary' if binary else 'text' + program = root / (name + '.nil') + support.compile_program(compiler, source, program, report['commands'], env, binary=binary) + + def running(client): + client.initialize(program) + client.request('configurationDone') + assert 'ready' in client.event('output')['output'] + + def pause_resume(client): + running(client) + client.request('stackTrace', {'threadId': 1}, success=False) + previous = -1 + for unused in range(5): + client.request('pause', {'threadId': 1}) + client.stopped('pause') + value = int(client.evaluate('value')) + assert value >= previous + previous = value + client.request('continue', {'threadId': 1}) + client.request('terminate') + client.event('terminated') + client.request('continue', {'threadId': 1}, success=False) + client.request('disconnect') + client.natural_exit() + assert client.record['stdin_open_at_exit'] + + def disconnect_running(client): + running(client) + client.request('disconnect') + client.natural_exit() + assert client.record['stdin_open_at_exit'] + + def disconnect_stopped(client): + client.initialize(program, stop_on_entry=True) + client.request('configurationDone') + client.stopped('entry') + client.request('disconnect') + client.natural_exit() + assert client.record['stdin_open_at_exit'] + + def eof_running(client): + running(client) + client.owner.process.stdin.close() + client.natural_exit() + assert not client.record['stdin_open_at_exit'] + + def malformed_running(client): + running(client) + client.send(b'Content-Length: -1\r\n\r\n') + client.natural_exit(1) + assert client.record['stdin_open_at_exit'] + + def partial_running(client, partial): + running(client) + client.send(partial) + client.owner.process.stdin.close() + client.natural_exit(1) + + for label, action in [('pause-resume', pause_resume), ('disconnect-running', disconnect_running), + ('disconnect-stopped', disconnect_stopped), ('eof-running', eof_running), + ('malformed-running', malformed_running), + ('partial-header', lambda client: partial_running(client, b'Content-Length: 10\r')), + ('partial-body', lambda client: partial_running(client, b'Content-Length: 10\r\n\r\n{}'))]: + support.session(debugger, root, report, name + '-' + label, env, action) + report['status'] = 'PASS' + except BaseException as error: + report['error'] = str(error) + (root / 'report.json').write_text(json.dumps(report, indent=2) + '\n') + print(report['status'] + ': ' + str(root / 'report.json')) + return 0 if report['status'] == 'PASS' else 1 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/nildebugger/tests/dap_owner_tests.cpp b/nildebugger/tests/dap_owner_tests.cpp new file mode 100644 index 00000000..48902f0e --- /dev/null +++ b/nildebugger/tests/dap_owner_tests.cpp @@ -0,0 +1,154 @@ +#include "dap.hpp" +#include "json_rpc.hpp" +#include "../../common/private_directory.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using Json = ny::json::Value; +void check(bool condition, const char* message) { + if (!condition) throw std::runtime_error(message); +} +std::string request(int seq, const char* command, Json arguments = ny::json::object({})) { + const auto body = ny::json::stringify(ny::json::object({{"seq", int64_t(seq)}, + {"type", "request"}, {"command", command}, {"arguments", std::move(arguments)}})); + return "Content-Length: " + std::to_string(body.size()) + "\r\n\r\n" + body; +} + +// Only the reader consumes this stream. Its next command is released by an +// observed DAP event, so premature EOF cannot pause the VM before the marker. +// Even a missing event returns EOF within five seconds, allowing normal join. +class GatedInput final : public std::streambuf { +public: + explicit GatedInput(std::vector segments) : segments_(std::move(segments)) { + select(0); + } + void release(unsigned stage) { + std::lock_guard lock(mutex_); + ready_ = std::max(ready_, stage); + changed_.notify_one(); + } + bool timed_out() const { return timed_out_; } // read only after run_dap joined its reader +private: + void select(unsigned stage) { + auto& segment = segments_.at(stage); + setg(segment.data(), segment.data(), segment.data() + segment.size()); + } + int_type underflow() override { + if (++stage_ >= segments_.size()) return traits_type::eof(); + std::unique_lock lock(mutex_); + if (!changed_.wait_for(lock, std::chrono::seconds(5), [&] { return ready_ >= stage_; })) { + timed_out_ = true; + return traits_type::eof(); + } + select(stage_); + return traits_type::to_int_type(*gptr()); + } + std::vector segments_; + unsigned stage_ = 0, ready_ = 0; + bool timed_out_ = false; + std::mutex mutex_; + std::condition_variable changed_; +}; + +// Observe complete framed output emitted by run_dap itself. Adapter writes are +// serialized by its output mutex. Capture the actual writer of each debuggee +// stdout event, not the protocol reader or a recycled operating-system ID. +class ObservedOutput final : public std::streambuf { +public: + explicit ObservedOutput(GatedInput& input) : input_(input), owner_(std::this_thread::get_id()) {} + unsigned markers = 0; + bool all_on_main = true, valid = true; +private: + int_type overflow(int_type byte) override { + if (!traits_type::eq_int_type(byte, traits_type::eof())) { + const char value = traits_type::to_char_type(byte); + xsputn(&value, 1); + } + return traits_type::not_eof(byte); + } + std::streamsize xsputn(const char* bytes, std::streamsize count) override { + buffer_.append(bytes, static_cast(count)); + if (buffer_.size() > 1024 * 1024) { valid = false; input_.release(2); return count; } + for (;;) { + const auto split = buffer_.find("\r\n\r\n"); + if (split == std::string::npos) break; + if (buffer_.rfind("Content-Length: ", 0) != 0) { valid = false; input_.release(2); break; } + const auto size = static_cast(std::stoul(buffer_.substr(16, split - 16))); + if (size > 1024 * 1024) { valid = false; input_.release(2); break; } + if (buffer_.size() < split + 4 + size) break; + auto message = ny::json::parse(buffer_.substr(split + 4, size), nullptr); + buffer_.erase(0, split + 4 + size); + if (!message || !message->is_object()) { valid = false; input_.release(2); continue; } + const auto* event = message->get("event"); + if (!event || !event->is_string()) continue; + if (event->as_string() == "stopped") input_.release(1); + if (event->as_string() == "terminated") input_.release(2); + if (event->as_string() != "output") continue; + const auto* body = message->get("body"); + const auto* text = body ? body->get("output") : nullptr; + if (text && text->is_string() && + (text->as_string() == "owner-first" || text->as_string() == "owner-second")) { + ++markers; + all_on_main = all_on_main && std::this_thread::get_id() == owner_; + } + } + return count; + } + GatedInput& input_; + const std::thread::id owner_; + std::string buffer_; +}; +} // namespace + +int main() { + const auto root = nil::detail::create_private_directory(std::filesystem::temp_directory_path(), "nil-dap-owner-"); + try { + const auto program = root / "owner.nil", source = root / "owner.ny"; + const auto quoted = ny::json::stringify(Json(source.string())); + std::ofstream file(program, std::ios::binary); + file << "module dap_owner\nconst A = \"owner-first\"\nconst B = \"owner-second\"\nfunction main -> i32:\n"; + const std::vector instructions = {"PUSH_CONST_STR A", "CALL println 1", + "PUSH_CONST_STR B", "CALL println 1", "PUSH_CONST 0", "RET"}; + for (unsigned index = 0; index < instructions.size(); ++index) + file << " # source_debug " << index << ' ' << quoted << " 0\n " << instructions[index] + << " ; line " << index / 2 + 1 << '\n'; + file << "end function\n"; + file.close(); + check(static_cast(file), "could not write owned NIL fixture"); + GatedInput input_buffer({ + request(1, "initialize") + request(2, "launch", ny::json::object({ + {"program", program.string()}, {"stopOnEntry", false}})) + + request(3, "setBreakpoints", ny::json::object({{"source", ny::json::object({{"path", source.string()}})}, + {"breakpoints", ny::json::array({ny::json::object({{"line", int64_t(2)}})})}})) + + request(4, "configurationDone"), + request(5, "continue", ny::json::object({{"threadId", int64_t(1)}})), request(6, "disconnect")}); + ObservedOutput output_buffer(input_buffer); + std::istream input(&input_buffer); + std::ostream output(&output_buffer); + const int status = ny::debug::run_dap(input, output); + check(status == 0, "adapter failed during owner proof"); + check(!input_buffer.timed_out(), "adapter did not release bounded input gate"); + check(output_buffer.valid && output_buffer.markers == 2, "both real debuggee output markers are required"); + check(output_buffer.all_on_main, "debuggee output was executed off the calling process main thread"); + std::filesystem::remove_all(root); + std::cout << "DAP owner: both execution segments ran on process main\n"; + return 0; + } catch (const std::exception& error) { + std::error_code ignored; std::filesystem::remove_all(root, ignored); + std::cerr << error.what() << '\n'; + return 1; + } +} diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 8cdfcb41..9990e6d4 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -226,6 +226,58 @@ reader. All sixteen generic checks pass with the real Windows `f79b94b` compiler and assembler; Linux passes those plus both ELF relocation checks. Mach-O framework execution remains in the next hosted campaign. +At `a518138`, both actual Mac allocator control sets pass. ARM also passes the +unchanged managed-lifetime workload and all sixteen generic Mach-O symbol +checks. Owned-lifetime coverage exposes a fixture assumption: `owned(16384)` +allocates 65,540 bytes, including its four-byte guard, rather than 65,536. +The target size is corrected with the Ny source unchanged. Twelve portable +control cases pass under GCC and Clang. The exact retained Ny workload, linked +against the real Linux runtime with accounting wrappers, observes 36,000 +allocations and frees in each memory mode; withholding those exact frees rejects +at 1,535 retained blocks. The 96 MiB bound is unchanged. Hosted confirmation +of both owned memory modes remains required. + +The Mac synthetic toolchain fixture now canonicalizes its private temporary +root, avoiding the incidental `/var` alias while retaining deliberate symlink +controls. Native dependency refusal reports the first bounded, sanitized reason; +environment values are never included. Eighty portable controls pass. An early +actual-host probe builds only the runtime and dependency collector, so further +SDK/linker failures are visible before the LLVM application build finishes. +It validates input discovery only; full-content identity and real incremental +reuse remain separate mandatory tests. + +The Windows `f79b94b` full run finishes with 583 registrations: 579 pass, the +known portable path-fixture failure, and three skips. Follow-up evidence fixes +the path fixture, executes the sixteen COFF checks and passes both desktop gates +in Nathan's session (ten language VM/native UI/keyboard cases plus window +discovery). All 1,138 retained route successes and 78 expected native rejections +replay correctly; PostgreSQL's six routes and owned cluster shutdown pass too. +Source, installed SDK and the original dirty guest checkout remain unchanged. +The real editor campaign is separate and still running; its first launch missed +a required Windows test-only Node setting and is retained as a setup failure. + +## DAP execution ownership + +DAP previously constructed a new worker for every continue or source step. +AppKit cannot create or poll its windows there, and desktop resources could +cross owning threads after a stop. VM construction, loading, execution, +inspection and disposal now stay on process main; one protocol reader forwards +bounded requests and accepts cooperative pause/terminate while instructions run. + +Fourteen lifecycle sessions verify normal, EOF, malformed-frame and disconnect +paths, including clients that keep stdin open. A strict wire oracle requires +acknowledgement before continued/debuggee events. Four real GTK desktop sessions +exercise retained handles and queued events across stops, and natural disconnect +with an open window. All existing debugger tests pass. GTK alone cannot prove +the fix: the old adapter can reuse pthread IDs and pass that test. A deterministic +stream observer instead checks the actual writer of two debuggee output events; +it passes the new adapter and rejects the exact previous implementation. + +Win32 and AppKit DAP desktop confirmation remains required. Paused windows are +not pumped, and adapter transport failures use best-effort terminal notification +before the reader accepts disconnect/EOF; cancellable arbitrary input streams +are outside this slice. No new native debugger or checkpoint profile is claimed. + Native socket/byte/entropy parity needs a binary-safe value boundary. The new VM APIs deliberately carry embedded NUL and arbitrary bytes in `text`; the current native text ABI and several LLVM operations use null-terminated strings diff --git a/tests/darwin_native_heap_budget.c b/tests/darwin_native_heap_budget.c index b4ca1e1a..77763667 100644 --- a/tests/darwin_native_heap_budget.c +++ b/tests/darwin_native_heap_budget.c @@ -393,6 +393,7 @@ __attribute__((constructor)) static void initialize_budget(void) { } if (strcmp(target, "16384") == 0) target_size = 16384; else if (strcmp(target, "65536") == 0) target_size = 65536; + else if (strcmp(target, "65540") == 0) target_size = 65540; else tracker_error("initialization-target"); if (strcmp(mode, "0") == 0) withhold_mode = 0; else if (strcmp(mode, "1") == 0) withhold_mode = 1; diff --git a/tests/darwin_native_heap_budget_control.c b/tests/darwin_native_heap_budget_control.c index d4a668a1..d3388cfb 100644 --- a/tests/darwin_native_heap_budget_control.c +++ b/tests/darwin_native_heap_budget_control.c @@ -123,6 +123,26 @@ static void positive(void) { write_all(1, "heap-controls-ok\n"); } +static void guarded_positive(void) { + const struct nil_heap_snapshot before = state(); + require(before.target_size == 65540, "guarded target differs\n"); + /* The unchanged owned(16384) native fixture requests 64 KiB plus + * its one-i32 guard element. Pin that real allocation/free size too. */ + for (size_t index = 0; index < 34000; ++index) { + unsigned char *value = calloc(1, 65540); + require(value != NULL, "guarded calloc failed\n"); + observed = value[0]; observed = value[65539]; + free(value); + } + live_is(before.live); + const struct nil_heap_snapshot after = state(); + require(after.target_allocations - before.target_allocations == 34000 && + after.target_frees - before.target_frees == 34000 && + after.cumulative - before.cumulative >= UINT64_C(34000) * 65540, + "guarded payload interception incomplete\n"); + write_all(1, "guarded-controls-ok\n"); +} + int main(int argc, char **argv) { snapshot = (nil_heap_snapshot_fn)dlsym(RTLD_DEFAULT, "nil_heap_budget_snapshot"); fail_next_realloc = (nil_heap_fail_realloc_fn)dlsym(RTLD_DEFAULT, "nil_heap_budget_fail_next_realloc"); @@ -134,11 +154,14 @@ int main(int argc, char **argv) { nil_heap_nested_verify_threads(); #endif if (strcmp(argv[1], "positive") == 0) { positive(); return 0; } - require(strcmp(argv[1], "retain") == 0, "unknown control mode\n"); + if (strcmp(argv[1], "guarded-positive") == 0) { guarded_positive(); return 0; } + const int guarded = strcmp(argv[1], "guarded-retain") == 0; + require(guarded || strcmp(argv[1], "retain") == 0, "unknown control mode\n"); + require(state().target_size == (guarded ? 65540 : 65536), "retained target differs\n"); /* Do not touch payload pages: a footprint-only replacement can miss this. - * Even if the oracle fails, this control requests at most 128 MiB. */ + * Even if the oracle fails, this requests at most 128 MiB plus 8 KiB. */ for (size_t index = 0; index < 2048; ++index) { - retained[index] = calloc(1, 65536); + retained[index] = calloc(1, guarded ? 65540 : 65536); require(retained[index] != NULL, "retention control allocator failed first\n"); } require(0, "retention control exceeded its independent safety ceiling\n"); diff --git a/tests/darwin_native_inputs_tests.cpp b/tests/darwin_native_inputs_tests.cpp new file mode 100644 index 00000000..9ade70a3 --- /dev/null +++ b/tests/darwin_native_inputs_tests.cpp @@ -0,0 +1,45 @@ +#include "../nilbuild/toolchain_inputs.hpp" + +#include +#include +#include +#include +#include + +int main(int argc, char** argv) { + if (argc != 3) { + std::cerr << "usage: nil_darwin_native_inputs_tests DRIVER RUNTIME\n"; + return 2; + } + try { + nilbuild::cancellation_scope cancellation; + namespace fs = std::filesystem; + const auto driver = fs::absolute(argv[1]).lexically_normal(); + const auto runtime = fs::absolute(argv[2]).lexically_normal(); + std::string reason; + const auto files = nilbuild::native_toolchain_files( + driver, fs::current_path(), runtime, {}, &reason); + if (!files || files->empty()) + throw std::runtime_error(reason.empty() ? "native dependency collector returned no files or reason" : reason); + if (std::find(files->begin(), files->end(), runtime) == files->end()) + throw std::runtime_error("native dependency inventory omitted the selected runtime"); + for (const auto& file : *files) { + std::error_code error; + if (!file.is_absolute() || !fs::is_regular_file(file, error) || error) + throw std::runtime_error("native input is not an absolute regular file: " + file.string()); + } + // This validates actual default-profile collection only. Content + // hashing and unchanged-build reuse remain the incremental contract. + std::cout << "PASS: native dependency collection includes runtime and " + << files->size() << " regular inputs (no content-hash/reuse claim)\n"; + return 0; + } catch (const nilbuild::process_cancelled& error) { + std::cerr << "FAIL: native dependency collection cancelled\n"; + return error.status; + } catch (const std::exception& error) { + std::string reason; + nilbuild::darwin_toolchain_detail::explain(&reason, error.what()); + std::cerr << "FAIL: " << reason << '\n'; + return 1; + } +} diff --git a/tests/darwin_toolchain_inputs_tests.cpp b/tests/darwin_toolchain_inputs_tests.cpp index b1ca140e..bab75c88 100644 --- a/tests/darwin_toolchain_inputs_tests.cpp +++ b/tests/darwin_toolchain_inputs_tests.cpp @@ -262,11 +262,25 @@ void closure(const fs::path &root) { check(collect().has_value(), "implicit standard local path alone never falls back outside the SDK"); fs::remove(plan.lto); check(!collect(), "missing required linker helper disables reuse"); + std::string reason; + check(!detail::collector(driver, project, runtime, plan, inspect, host, &reason).files() && + reason.find("libLTO.dylib") != std::string::npos, + "failed dependency closure explains its first missing helper"); + reason.clear(); + detail::explain(&reason, "unsafe\r\n\t" + std::string(2048, 'x')); + check(reason.size() == 1024 && reason.find_first_of("\r\n\t") == std::string::npos, + "untrusted tool/path diagnostics are bounded and control sanitized"); + const auto first_reason = reason; + detail::explain(&reason, "later failure"); + check(reason == first_reason, "later generic failures preserve the first concrete reason"); } } // namespace int main() { - const auto root = nil::detail::create_private_directory(fs::temp_directory_path(), "nil-darwin-closure-"); + // macOS's temporary parent may be a /var -> /private/var alias. Keep the + // synthetic toolchain rooted consistently; alias behavior is tested below + // through deliberate fixture links rather than an incidental host prefix. + const auto root = fs::canonical(nil::detail::create_private_directory(fs::temp_directory_path(), "nil-darwin-closure-")); try { closure(root); fs::remove_all(root); diff --git a/tests/tools/darwin_heap_accounting_contract.py b/tests/tools/darwin_heap_accounting_contract.py index ed8b9860..161d8306 100644 --- a/tests/tools/darwin_heap_accounting_contract.py +++ b/tests/tools/darwin_heap_accounting_contract.py @@ -56,15 +56,20 @@ def command(name, argv, *, variables=None, code=0, stdout=b''): argv += ['-ldl'] + ['-Wl,--wrap=' + name for name in ('malloc', 'calloc', 'realloc', 'free', 'aligned_alloc', 'posix_memalign', 'valloc')] command(kind + '-build', argv + ['-o', executable]) - for case, mode, withhold in (('positive', 'positive', False), - ('retained', 'retain', False), ('withheld', 'positive', True)): + for case, mode, withhold, target in ( + ('positive', 'positive', False, 65536), ('retained', 'retain', False, 65536), + ('withheld', 'positive', True, 65536), + ('guarded-positive', 'guarded-positive', False, 65540), + ('guarded-retained', 'guarded-retain', False, 65540), + ('guarded-withheld', 'guarded-positive', True, 65540)): name = kind + '-' + case events_path = output / (name + '.jsonl') - negative = case != 'positive' + negative = not case.endswith('positive') + positive_output = b'heap-controls-ok\n' if target == 65536 else b'guarded-controls-ok\n' command(name, [executable, mode], code=86 if negative else 0, - stdout=b'' if negative else b'heap-controls-ok\n', variables={ + stdout=b'' if negative else positive_output, variables={ 'NIL_DARWIN_HEAP_REPORT': str(events_path), 'NIL_DARWIN_HEAP_TOKEN': 'a' * 32, - 'NIL_DARWIN_HEAP_TARGET': '65536', 'NIL_DARWIN_HEAP_WITHHOLD': str(int(withhold))}) + 'NIL_DARWIN_HEAP_TARGET': str(target), 'NIL_DARWIN_HEAP_WITHHOLD': str(int(withhold))}) events = [json.loads(line) for line in events_path.read_text().splitlines()] if len(events) != 2 or events[0]['event'] != 'ready': raise AssertionError(name + ': initialization/final event pair absent') @@ -74,12 +79,13 @@ def command(name, argv, *, variables=None, code=0, stdout=b''): if nested and (initial['cumulative'] != 1024 or initial['allocations'] != 1 or initial['live']): raise AssertionError(name + ': pre-initialization nested request was not counted exactly once') if negative: - if not (BUDGET - 65536 <= final['live'] <= BUDGET and final['denied_request'] == 65536): + if not (BUDGET - target <= final['live'] <= BUDGET and final['denied_request'] == target): raise AssertionError(name + ': retained payload did not trigger the real budget') if withhold and not final['withheld']: raise AssertionError(name + ': actual target frees were not withheld') elif (final['live'] or final['cumulative'] - initial['cumulative'] <= 2 * 1024**3 or - final['target_allocations'] - initial['target_allocations'] < 34000 + (4 if nested else 0) + final['target_allocations'] - initial['target_allocations'] < + 34000 + (4 if nested and target == 65536 else 0) or final['unknown_frees'] - initial['unknown_frees']): raise AssertionError(name + ': sustained/nested/threaded allocation coverage differs') report['cases'].append({'name': name, 'events': events}) diff --git a/tests/tools/darwin_native_heap_budget.py b/tests/tools/darwin_native_heap_budget.py index 6a1c6d90..9e182b5b 100644 --- a/tests/tools/darwin_native_heap_budget.py +++ b/tests/tools/darwin_native_heap_budget.py @@ -203,6 +203,14 @@ def prepare(self): self.injected('control-retained-untouched', self.control, ['retain'], target=65536, negative=True) self.injected('control-withheld-free', self.control, ['positive'], target=65536, withhold=True, negative=True) + guarded = self.injected('control-guarded-positive', self.control, ['guarded-positive'], + target=65540, wanted=b'guarded-controls-ok\n') + if guarded['target_allocations'] != 34000 or guarded['target_frees'] != 34000: + raise AssertionError('guarded native-buffer allocation/free control coverage differs') + self.injected('control-guarded-retained', self.control, ['guarded-retain'], target=65540, + negative=True) + self.injected('control-guarded-withheld', self.control, ['guarded-positive'], target=65540, + withhold=True, negative=True) def compile(self, name, command): self.command(name, command, seconds=90) diff --git a/tests/tools/native_owned_lifetime.py b/tests/tools/native_owned_lifetime.py index 5bd5dd54..c7df9e70 100644 --- a/tests/tools/native_owned_lifetime.py +++ b/tests/tools/native_owned_lifetime.py @@ -73,7 +73,10 @@ def main() -> int: "--native", *flags, "--output", str(output), str(source)] if darwin: darwin.compile('compile-' + name, command) - darwin.fixture(name, source, output, "35994000\n", target=65536, count=36000) + # The native raw-buffer ABI allocates one additional i32 guard + # element, so this unchanged owned(16384) loop requests + # exactly 65540 bytes for each payload, in both memory modes. + darwin.fixture(name, source, output, "35994000\n", target=65540, count=36000) else: compiled = subprocess.run(command, cwd=root, env=environment, stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=90) From 8aaf3aed42f24f54b547b92565e71b9e2cf433b9 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 19:18:44 -0500 Subject: [PATCH 12/39] Scope SQL loader paths to execution and record platform acceptance --- .../workflows/macos-application-parity.yml | 8 +- .github/workflows/macos-bringup.yml | 8 +- .github/workflows/macos-sdk.yml | 12 ++- CMakeLists.txt | 9 ++ README.md | 7 +- docs/platform-support.md | 45 ++++++++-- docs/standard-library/ui.md | 4 +- notebook/2026-11-11-web-runtime-parity.md | 63 +++++++++++-- scripts/ci/macos-sdk.py | 4 +- scripts/ci/test_macos_sdk.py | 16 ++++ tests/database_postgresql_isolated.py | 34 +++++-- tests/routes/ny_route_matrix.py | 20 +++-- tests/runtime_provider_environment.py | 31 +++++++ tests/test_database_postgresql_isolated.py | 42 +++++++++ tests/test_runtime_provider_environment.py | 89 +++++++++++++++++++ vscode/test/project-checks.cjs | 19 ++-- vscode/test/suite.cjs | 9 +- 17 files changed, 371 insertions(+), 49 deletions(-) create mode 100644 tests/runtime_provider_environment.py create mode 100644 tests/test_runtime_provider_environment.py diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index d9617808..1bbfe5dd 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -310,7 +310,13 @@ jobs: report = json.loads(Path('build/vscode-host-smoke/latest.json').read_text()) assert report['platform'] == 'darwin' and report['status'] == 'PASS' assert report['vscode'] == '1.146.0' and not report['skippedChecks'] - assert len(report['checks']) >= 100 and all(row['status'] == 'PASS' for row in report['checks']) + expected_modules = ['analyzer-checks', 'project-checks', 'workspace-checks', 'semantic-checks', + 'completion-checks', 'member-signature-checks', 'language-values-checks', + 'memory-profile-checks', 'niltest-checks'] + assert report['fullSuiteCompleted'] is True and report['completedModules'] == expected_modules + checks = report['checks'] + assert checks and all(row['status'] == 'PASS' and row['name'] for row in checks) + assert len({row['name'] for row in checks}) == len(checks) PYTHON - name: Preserve complete or partial evidence if: always() diff --git a/.github/workflows/macos-bringup.yml b/.github/workflows/macos-bringup.yml index 75cda122..a11af12f 100644 --- a/.github/workflows/macos-bringup.yml +++ b/.github/workflows/macos-bringup.yml @@ -53,6 +53,7 @@ jobs: run: | set -o pipefail python3 -B scripts/ci/test_macos_evidence.py -v 2>&1 | tee build/macos-evidence/collector-controls.log + python3 -B tests/test_runtime_provider_environment.py -v 2>&1 | tee build/macos-evidence/provider-environment-controls.log - name: Observe image toolchain and optional providers timeout-minutes: 3 run: | @@ -66,9 +67,12 @@ jobs: test -f "$nil_macos_sdk/usr/include/sqlite3.h" { echo "NIL_CI_SQLITE_INCLUDE=$nil_macos_sdk/usr/include" - echo "DYLD_FALLBACK_LIBRARY_PATH=/usr/local/lib:/usr/lib" + # Inert declaration: provider harnesses apply it only to runtime + # children. Never export a DYLD override to compiler/build steps. + echo "NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH=/usr/local/lib:/usr/lib" } >> "$GITHUB_ENV" printf 'SQLite: Apple SDK headers at %s; system dylib provider\n' "$nil_macos_sdk/usr/include" > build/macos-evidence/providers.txt + echo 'SQL loader paths are runtime-only declarations; JIT combines compilation/execution in one process' >> build/macos-evidence/providers.txt # Use one installed PostgreSQL distribution for headers, client # library, binaries and shared data; never mix independent versions. if nil_pg_prefix="$(brew --prefix postgresql@16 2>/dev/null)" && @@ -84,7 +88,7 @@ jobs: echo "NIL_TEST_POSTGRES_BIN=$nil_pg_prefix/bin" echo "NIL_TEST_POSTGRES_SHARE=$nil_pg_share" echo "NIL_CI_PQ_INCLUDE=$nil_pg_include" - echo "DYLD_FALLBACK_LIBRARY_PATH=$nil_pg_library:/usr/local/lib:/usr/lib" + echo "NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH=$nil_pg_library:/usr/local/lib:/usr/lib" } >> "$GITHUB_ENV" "$nil_pg_prefix/bin/pg_config" --version --includedir --libdir --sharedir >> build/macos-evidence/providers.txt else diff --git a/.github/workflows/macos-sdk.yml b/.github/workflows/macos-sdk.yml index cf90600e..cfb6f38f 100644 --- a/.github/workflows/macos-sdk.yml +++ b/.github/workflows/macos-sdk.yml @@ -63,6 +63,7 @@ jobs: python3 -B tests/sdk_consumer_routes_tests.py -v 2>&1 | tee build/macos-sdk-evidence/consumer-controls.log python3 -B scripts/ci/test_macos_postgresql_provider.py -v 2>&1 | tee build/macos-sdk-evidence/provider-controls.log python3 -B scripts/ci/test_macos_debugger.py -v 2>&1 | tee build/macos-sdk-evidence/debugger-controls.log + python3 -B tests/test_runtime_provider_environment.py -v 2>&1 | tee build/macos-sdk-evidence/provider-environment-controls.log - name: Fetch hash-pinned official Pandoc id: pandoc timeout-minutes: 5 @@ -91,7 +92,9 @@ jobs: nil_macos_sdk="$(xcrun --sdk macosx --show-sdk-path)" test -f "$nil_macos_sdk/usr/include/sqlite3.h" echo "NIL_CI_SQLITE_INCLUDE=$nil_macos_sdk/usr/include" >> "$GITHUB_ENV" - echo "DYLD_FALLBACK_LIBRARY_PATH=/usr/local/lib:/usr/lib" >> "$GITHUB_ENV" + # Keep loader paths inert during compiler and SDK build processes. + # The SQL/route harnesses apply this declaration only to execution. + echo "NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH=/usr/local/lib:/usr/lib" >> "$GITHUB_ENV" printf 'SQLite: Apple SDK headers at %s; system dylib\n' "$nil_macos_sdk/usr/include" > build/macos-sdk-evidence/providers.txt if [ "$NIL_CI_POSTGRES_PROFILE" = private16 ]; then python3 -B - <<'PY' @@ -103,7 +106,7 @@ jobs: provider.validate_provider(report) values = {'NIL_TEST_POSTGRES_BIN': report['bin_dir'], 'NIL_TEST_POSTGRES_SHARE': report['share_dir'], 'NIL_CI_PQ_INCLUDE': report['include_dir'], - 'DYLD_FALLBACK_LIBRARY_PATH': report['library_dir'] + ':/usr/lib'} + 'NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH': report['library_dir'] + ':/usr/lib'} with open(os.environ['GITHUB_ENV'], 'a') as stream: for key, value in values.items(): assert isinstance(value, str) and '\n' not in value and '\r' not in value @@ -123,7 +126,7 @@ jobs: echo "NIL_TEST_POSTGRES_BIN=$nil_pg_prefix/bin" echo "NIL_TEST_POSTGRES_SHARE=$nil_pg_share" echo "NIL_CI_PQ_INCLUDE=$nil_pg_include" - echo "DYLD_FALLBACK_LIBRARY_PATH=$nil_pg_library:/usr/local/lib:/usr/lib" + echo "NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH=$nil_pg_library:/usr/local/lib:/usr/lib" } >> "$GITHUB_ENV" "$nil_pg_prefix/bin/pg_config" --version --includedir --libdir --sharedir >> build/macos-sdk-evidence/providers.txt else @@ -131,9 +134,10 @@ jobs: echo 'PostgreSQL: coherent installed provider absent; actual CTest skips/failures retained' >> build/macos-sdk-evidence/providers.txt fi { - echo 'GTK: explicit macOS provider available in source; this SDK campaign does not provision or qualify desktop use' + echo 'AppKit: native macOS provider; this SDK campaign does not provision or qualify desktop use' echo 'Installed candidate checks: inherited DYLD/source/tool overrides removed; Apple toolchain PATH; system SQLite only' echo 'Private PostgreSQL installed checks use a separate explicitly declared provider environment' + echo 'SQL loader paths apply only to runtime children; JIT is a combined compilation/execution process' echo 'Homebrew LLVM dependencies remain external development prerequisites, not bundled portable libraries' } >> build/macos-sdk-evidence/providers.txt { diff --git a/CMakeLists.txt b/CMakeLists.txt index 770712b9..7bfb4943 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -321,6 +321,12 @@ if(BUILD_TESTING) target_compile_features(nil_database_postgresql_tests PRIVATE cxx_std_17) add_test(NAME nil.database-postgresql-client COMMAND $) set_tests_properties(nil.database-postgresql-client PROPERTIES LABELS "database;runtime" SKIP_RETURN_CODE 77 TIMEOUT 30) + if(APPLE AND NOT "$ENV{NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH}" STREQUAL "") + # The optional provider path belongs to this executing runtime only. + # Keep compiler/build processes free of test-only dyld selection inputs. + set_tests_properties(nil.database-postgresql-client PROPERTIES + ENVIRONMENT "DYLD_FALLBACK_LIBRARY_PATH=$ENV{NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH}") + endif() add_executable(nilagent_ipc_tests tests/nilagent_ipc_tests.cpp) target_link_libraries(nilagent_ipc_tests PRIVATE Threads::Threads) @@ -1719,6 +1725,9 @@ if(BUILD_TESTING) --compiler $ --interpreter $ --stdlib "${CMAKE_CURRENT_SOURCE_DIR}/nilstd/src" --llvm "${NIL_ENABLE_LLVM}") set_tests_properties(nil.reflection-consumer PROPERTIES LABELS "analysis;reflection;routes" TIMEOUT 120) + add_test(NAME nil.runtime-provider-environment + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/test_runtime_provider_environment.py") + set_tests_properties(nil.runtime-provider-environment PROPERTIES LABELS "database;tools;routes" TIMEOUT 30) add_test(NAME nil.database-postgresql-fixture COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/test_database_postgresql_isolated.py") set_tests_properties(nil.database-postgresql-fixture PROPERTIES LABELS "database;tools" TIMEOUT 30) diff --git a/README.md b/README.md index ea664dc5..23393feb 100644 --- a/README.md +++ b/README.md @@ -249,6 +249,8 @@ such as `say formatted "value={5}"`; it works on interpreted and native routes. The current version is **1.0.0-rc.1**. Linux x86_64 is the reference development platform. The Windows x86_64 development SDK has [paired Linux acceptance and installed tooling coverage](notebook/2026-11-02-windows-port.md). +The newer Windows source campaign also passes the Win32 desktop contracts and +all 88 real VS Code host groups; a fresh SDK still needs separate qualification. Intel and Apple Silicon macOS have passing native build baselines with LLVM enabled and disabled, real interpreter/checkpoint execution, and representative LLVM JIT/AOT execution. Both macOS architectures also pass bounded native inline @@ -262,8 +264,9 @@ current SDK desktop campaign remain open. The newer socket, entropy and text-byt intrinsics pass the same interpreter contracts on Linux, Windows and both Macs. File timestamps share an implementation across interpreter and native routes, with text/binary NIL, JIT and AOT checks passing on all four hosts. -Darwin incremental reuse and native allocation-retention tests remain open, so full platform -parity is not yet established. +Both Macs now pass the native allocation-retention contracts and desktop debugger +sessions. Darwin incremental reuse, complete Mac editor coverage and current SDK +qualification remain open, so full platform parity is not yet established. The [platform support matrix](docs/platform-support.md) separates verified development behavior from remaining SDK, provider and release work. Windows and macOS release qualification remains pending. The diff --git a/docs/platform-support.md b/docs/platform-support.md index 8ed3d8d9..6d434a58 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -9,10 +9,10 @@ release boundary. A newer source checkout does not update an installed SDK. | Host | Verified development baseline | Remaining boundary | | --- | --- | --- | -| Linux x86_64 | Reference host. The clean `5630388` SDK passed the full suite, installed consumer routes, shared runtime, debugger, Unicode/I/O, execution checkpoints and GTK checks. | The reviewed Ubuntu release toolchain is separate from this development SDK. New release binaries still require the release gates. | -| Windows x86_64 | The same `5630388` source has paired Linux acceptance: interpreter/bundles, LLVM JIT/AOT, object/link routes, installed tooling, checkpoints, GTK and VS Code. | Development acceptance on the provisioned Windows VM; bare-host installation and Windows release qualification remain open. GTK and PostgreSQL require external runtimes. | -| macOS 15 Intel x86_64 | At `0c88d2f`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Earlier LLVM-ON/OFF, native assembly and embedded-context baselines also pass. | Full source suite: 567 passed, one failed and three skipped out of 571. Incremental reuse fails; UI and two native memory-limit tests skip. Desktop/editor and release acceptance remain open. | -| macOS 15 Apple Silicon arm64 | At `3414ce5`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Native assembly and embedded context also pass. | Full source suite: 567 passed, one failed and three skipped out of 571. Incremental reuse fails; UI and two native memory-limit tests skip. Desktop/editor and release acceptance remain open. | +| Linux x86_64 | Reference host. The clean `5630388` SDK passed installed consumer routes, shared runtime, debugger, Unicode/I/O, checkpoints and GTK checks. The newer `8376252` LLVM-ON source suite passes all 586 CTests. | The reviewed Ubuntu release toolchain is separate from this development SDK. New release binaries still require the release gates. | +| Windows x86_64 | The `5630388` installed SDK retains paired Linux acceptance. The newer private `f79b94b` source build passes 579 of 583 registered tests; separately pinned follow-ups resolve its fixture failure and execute the three skipped contracts. Real Win32 desktop and all 88 VS Code host groups pass. | The `8376252` debugger also passes six CTests and four real Win32 desktop sessions. Current-source SDK qualification remains open. Bare-host installation and Windows release qualification remain open. PostgreSQL requires an external runtime; the current UI provider is Win32. | +| macOS 15 Intel x86_64 | At `0c88d2f`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Earlier LLVM-ON/OFF, native assembly and embedded-context baselines also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse, full editor acceptance, a current-source SDK and release qualification remain open. | +| macOS 15 Apple Silicon arm64 | At `3414ce5`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Native assembly and embedded context also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse, full editor acceptance, a current-source SDK and release qualification remain open. | LLVM-OFF builds exercise frontend/interpreter behavior and rejection of disabled native operations. Their successful tests do not establish native execution. @@ -165,10 +165,13 @@ the complete admission rules. its coherent client/server and installed source/binary routes on both architectures; the scoped-address validation correction passes on both hosts. The corrected Intel retry at `0c88d2f` completes all installed campaigns, as recorded above. -- The two native lifetime stress tests still lack a demonstrated Darwin memory - bound. Native diagnostics show that the fixed 96 MiB virtual-memory cap is - rejected at process setup; larger diagnostic profiles do not replace it. - Successful lifetime execution routes remain covered by the ordinary suite. +- Both Mac hosts pass the managed and owned native lifetime contracts at + `8376252`, including both owned memory modes. Each owned workload allocates + and frees 36,000 guarded buffers; mandatory withheld-free controls reach the + 96 MiB retained-payload limit. The allocator controls also pass more than 2 GiB + of cumulative allocation. This measures intercepted malloc-family payloads, + not total address space, allocator metadata or mappings. The old Darwin + virtual-address cap remains unusable and is not represented as passing. - Desktop UI now uses one provider per platform: GTK on Linux and Win32 on Windows. The Windows provider passes the full desktop contract on a real interactive session -- provider contract, lifecycle release observation, the @@ -202,7 +205,8 @@ are currently interpreter-backed. File timestamps share a stable native-metadata implementation across interpreter and native routes: Linux text NIL, binary NIL, JIT and AOT all pass unchanged-file and changed-metadata checks. Linux results alone do not establish Windows or macOS support, and declarations do not establish -native/JIT execution. Cross-platform validation of these additions is in progress. +native/JIT execution. The actual native timestamp contract also passes on Windows +and both Mac architectures; full current-source packaging remains in progress. Their evidence must join the older desktop, tooling and SDK baselines before declaring current-source parity. @@ -214,3 +218,26 @@ replay. The tested PR merge and head revisions are retained separately. Current macOS editor coverage is still in progress. The socket/byte/entropy native limitation is common to Linux, Windows and macOS; those APIs currently require the interpreter on all three hosts. + +## Current Windows source and editor follow-up + +At `f79b94b`, the full private Windows build records 579 passes, one portable +path-fixture failure and three skips across 583 registrations. The corrected +fixture passes 71 MSVC checks separately. The formerly ELF-only symbol contract +now passes all sixteen generic ABI/IR checks against actual COFF objects. +Nathan's interactive session passes all ten Matrix VM/native desktop and input +cases, 137 provider checks, 235 lifecycle checks and window discovery. The retained +route reports contain 1,138 successes and 78 expected native rejections; the +six PostgreSQL routes and owned-cluster shutdown also pass. + +The full real VS Code host passes 88 distinct groups with no exclusions, using +the `a518138` test-only physical-file alias correction with `f79b94b` products. +Extension unit coverage is 161 passes and one Windows-inapplicable POSIX FIFO +skip. An outer evidence wrapper incorrectly expected at least 100 host groups; +its raw failure is retained alongside independent replay of every artifact hash, +all six successful process phases and cleanup. This composed source evidence +does not replace the accepted SDK. The main-thread DAP execution change at +`8376252` subsequently passes all six debugger CTests, fourteen lifecycle sessions +and four actual desktop sessions on Windows and both Mac architectures. Linux +also passes all 586 source CTests at this revision. Details are in the +[web-runtime parity notebook](../notebook/2026-11-11-web-runtime-parity.md). diff --git a/docs/standard-library/ui.md b/docs/standard-library/ui.md index d2d6401d..5d53728b 100644 --- a/docs/standard-library/ui.md +++ b/docs/standard-library/ui.md @@ -74,8 +74,8 @@ where AppKit requires its windows to be used. Ordinary interpreter, JIT and native entry routes use the main thread. The LLVM-ON display campaign verifies the interpreter and native application cases on both architectures. DAP now keeps the VM and its UI resources on the process main thread across source -stops. Its real GTK ownership/event tests pass; Win32 and AppKit debugger -desktop qualification is being added separately from ordinary UI execution. +stops. Its real GTK, Win32 and AppKit ownership/event tests pass, including +retained handles and queued events across source steps and disconnect cleanup. Paused debug sessions do not automatically pump window events. AppKit supplies no box or grid container, so, as on Win32, the provider owns the layout pass and sizes come from each control's fitting size. diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 9990e6d4..1c60b649 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -253,8 +253,16 @@ in Nathan's session (ten language VM/native UI/keyboard cases plus window discovery). All 1,138 retained route successes and 78 expected native rejections replay correctly; PostgreSQL's six routes and owned cluster shutdown pass too. Source, installed SDK and the original dirty guest checkout remain unchanged. -The real editor campaign is separate and still running; its first launch missed -a required Windows test-only Node setting and is retained as a setup failure. +The real Windows editor campaign now passes all 88 check groups without exclusions, +using the `a518138` physical-file alias helper against the unchanged `f79b94b` +products. Its first launch missed a required test-only Node setting and is +retained as a setup failure. The second run's private outer wrapper incorrectly +required 100 groups; the authoritative full-host report and all six process phases +pass. Independent replay verifies 34 retained file hashes, twelve raw streams, +matching run identity, cleanup and unchanged source/tools. The raw outer failure +remains preserved. Extension unit tests pass 161 cases with the one explicitly +inapplicable POSIX FIFO case skipped. Future host reports record full-suite +completion and the exact completed module list instead of an arbitrary group count. ## DAP execution ownership @@ -273,10 +281,35 @@ the fix: the old adapter can reuse pthread IDs and pass that test. A determinist stream observer instead checks the actual writer of two debuggee output events; it passes the new adapter and rejects the exact previous implementation. -Win32 and AppKit DAP desktop confirmation remains required. Paused windows are -not pumped, and adapter transport failures use best-effort terminal notification -before the reader accepts disconnect/EOF; cancellable arbitrary input streams -are outside this slice. No new native debugger or checkpoint profile is claimed. +At `8376252`, Windows passes all six exact debugger CTests, fourteen lifecycle +sessions and four real Nathan desktop sessions. Independent replay parses 318 +raw DAP frames and verifies 42 acknowledgement/continued orderings. Both actual +Mac hosts also pass those six CTests and four AppKit desktop sessions in run +`43124577210` (tested PR merge `b3661e8`). Linux's complete LLVM-ON source suite +passes all 586 registrations without skips in run `43124577017`. + +The same Mac campaign passes all eight selected native contracts on each host: +managed lifetime, both owned memory modes, generic Mach-O ABI checks, timestamps, +compiler identity and default/overridden host hooks. Independent replay verifies +306 retained file hashes per host and the raw allocation events. Each owned mode +allocates/frees all 36,000 guarded buffers; withholding frees reaches the 96 MiB +retained-payload bound. This does not claim a whole-process virtual-memory bound. +The shared six web-runtime contracts also pass on all four hosts at this revision; +148 retained file hashes and exact inventories replay successfully. + +Both Mac editor sessions reach 25 successful groups before the aggregate +45-second project check/build/run deadline expires. The project check succeeds +and the VM build reaches post-compilation verification; the current real cache +contract takes roughly 110 seconds. The editor harness now records each command's +duration and allows 180 seconds for this three-command Darwin group, retaining +the full-host bound. Full editor acceptance remains open. Early native probes +identify the remaining cache refusals: an arm64e system launcher inspected as +arm64, and a non-linkable EmailCore plugin embedded in the SDK. + +Paused windows are not pumped, and adapter transport failures use best-effort +terminal notification before the reader accepts disconnect/EOF; cancellable +arbitrary input streams are outside this slice. No new native debugger or +checkpoint profile is claimed. Native socket/byte/entropy parity needs a binary-safe value boundary. The new VM APIs deliberately carry embedded NUL and arbitrary bytes in `text`; the @@ -294,6 +327,24 @@ consumers together, adapt C-string FFI boundaries explicitly, and reject mixed old/new object and runtime ABIs. A hidden pointer-length registry would retain the same migration burden while adding lifetime and registration ambiguity. +## SQL provider environment scope + +The full Mac source/SDK workflows previously exported SQL's +`DYLD_FALLBACK_LIBRARY_PATH` to every process, including the builder. That changes +native dependency selection and correctly prevents cache reuse under the modeled +default profile. The workflows now carry an inert test declaration and apply it +only to PostgreSQL server/client and language execution processes. Compilation, +assembly and AOT linking keep their ordinary environment; JIT explicitly remains +a combined compilation/execution process. External callers without the declaration +retain their ambient environment. Conflicting declared and ambient DYLD settings +fail explicitly rather than being silently cleared. + +The direct PostgreSQL client CTest receives its path only through its own test +property. Four environment controls inspect all six route branches; seventeen +isolated-server controls, fifteen SDK controls and the existing route/consumer +controls pass. A real bounded Linux SQL smoke still passes all six routes. +Actual Mac source and SDK campaigns must confirm the new provider scope. + ## Preserved web handoff The interrupted web tree contains the listener-to-engine replacement, template diff --git a/scripts/ci/macos-sdk.py b/scripts/ci/macos-sdk.py index da33377d..bb061537 100644 --- a/scripts/ci/macos-sdk.py +++ b/scripts/ci/macos-sdk.py @@ -120,12 +120,12 @@ def run(name, command, timeout, *, cwd=ROOT, clean=False, provider_library=None) variables = environment() if clean else os.environ.copy() if provider_library is not None: require(clean and Path(provider_library).is_dir(), "declared provider requires clean environment and library directory") - variables["DYLD_FALLBACK_LIBRARY_PATH"] = str(provider_library) + ":/usr/lib" + variables["NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH"] = str(provider_library) + ":/usr/lib" row = support.run_process(ownership, [str(x) for x in command], cwd, variables, OUTPUT, name, timeout) row["environment_profile"] = "Apple toolchain; no inherited NIL/NY/DYLD/tool overrides" if clean else "recorded build/provider environment" if provider_library is not None: - row["environment_profile"] = "Apple toolchain; inherited overrides removed; one explicit PostgreSQL provider" + row["environment_profile"] = "Apple toolchain; inherited overrides removed; PostgreSQL loader path declared for runtime children only" row["declared_provider_library"] = str(provider_library) save(name + ".json", row) require(row["status"] == "PASS" and row["cleanup_verified"], diff --git a/scripts/ci/test_macos_sdk.py b/scripts/ci/test_macos_sdk.py index e5624f8a..786983ae 100644 --- a/scripts/ci/test_macos_sdk.py +++ b/scripts/ci/test_macos_sdk.py @@ -17,6 +17,22 @@ class CandidateTests(unittest.TestCase): + def test_declared_provider_parent_does_not_export_loader_override(self): + with tempfile.TemporaryDirectory() as temporary: + seen = {} + + def run_process(owner, argv, cwd, environment, output, name, timeout): + seen.update(environment) + return {'status': 'PASS', 'cleanup_verified': True} + + support = mock.Mock(run_process=run_process) + with mock.patch.object(sdk, 'load', return_value=support), \ + mock.patch.object(sdk, 'environment', return_value={'PATH': '/usr/bin:/bin'}), \ + mock.patch.object(sdk, 'save'): + sdk.run('postgresql-test', ['/owned/test'], 5, clean=True, provider_library=temporary) + self.assertNotIn('DYLD_FALLBACK_LIBRARY_PATH', seen) + self.assertEqual(seen['NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH'], temporary + ':/usr/lib') + def test_payload_seal_covers_bytes_modes_and_namespace(self): with tempfile.TemporaryDirectory() as temporary: root = Path(temporary) diff --git a/tests/database_postgresql_isolated.py b/tests/database_postgresql_isolated.py index 4d0d8e69..39588b25 100644 --- a/tests/database_postgresql_isolated.py +++ b/tests/database_postgresql_isolated.py @@ -23,6 +23,7 @@ REPO = Path(__file__).resolve().parents[1] sys.path.insert(0, str(REPO / 'validation/ny-3d/scripts')) from debugger_process import CaptureBudget, OwnedProcess, PipeReader +from runtime_provider_environment import DECLARATION, provider_environments def require(condition, message): @@ -216,7 +217,17 @@ def close(self): def run_fixture(args, report): env = clean_environment(os.environ) + if DECLARATION in os.environ: + env[DECLARATION] = os.environ[DECLARATION] + env, runtime_env, declared_provider = provider_environments(env) env['NIL_STDLIB_PATH'] = str(args.stdlib.resolve()) + runtime_env['NIL_STDLIB_PATH'] = env['NIL_STDLIB_PATH'] + if declared_provider is not None: + report['runtime_loader'] = { + 'dyld_fallback_library_path': declared_provider, + 'scope': 'server/client and route execution only; JIT combines compilation and execution', + 'compile_environment': 'no ambient DYLD overrides', + } root = Path(tempfile.mkdtemp(prefix='nil-db-pg-')) report.update(work_directory=str(root), work_directory_retained=True) password, address, server = '', '', None @@ -248,7 +259,7 @@ def record(name, output): if args.postgres_share: init += ['-L', str(args.postgres_share.resolve())] report['phase'] = 'initialize' - status, output = run(init, env, root, 30) + status, output = run(init, runtime_env, root, 30) record('initdb.log', output) pwfile.unlink() require(status == 0, 'private PostgreSQL initialization failed') @@ -262,6 +273,7 @@ def record(name, output): reservation.close() address = 'hostaddr=127.0.0.1 dbname=postgres user=nil_test' env.update(PGPASSWORD=password, PGPORT=str(port), PGSSLMODE='disable', PGGSSENCMODE='disable') + runtime_env.update(PGPASSWORD=password, PGPORT=str(port), PGSSLMODE='disable', PGGSSENCMODE='disable') else: port = 5432 # Scoped by the newly-created private socket directory. address = 'host=' + conninfo_quote(socket_directory) + ' dbname=postgres user=nil_test' @@ -280,7 +292,7 @@ def record(name, output): command = ([binary('pg_ctl'), '-D', str(data), '-w', '-t', '30', 'start'] if windows else [binary('postgres'), '-D', str(data)]) report['server_launcher'] = 'pg_ctl-restricted-token' if windows else 'postgres-foreground' - server = Server(command, env, root, launcher=windows) + server = Server(command, runtime_env, root, launcher=windows) deadline = time.monotonic() + 30 while not server.ready(data, port): require(time.monotonic() < deadline, 'private PostgreSQL startup deadline exceeded') @@ -294,17 +306,17 @@ def record(name, output): report['phase'] = 'authentication-controls' check = [binary('psql'), '-X', '-w', '-qAt', '-h', '127.0.0.1', '-U', 'nil_test', '-d', 'postgres', '-c', 'SELECT 1'] - rejected, output = run(check, dict(env, PGPASSWORD=password + '-wrong'), root, 10) + rejected, output = run(check, dict(runtime_env, PGPASSWORD=password + '-wrong'), root, 10) record('authentication-negative.log', output) require(rejected != 0, 'loopback server accepted an incorrect credential') - accepted, output = run(check, env, root, 10) + accepted, output = run(check, runtime_env, root, 10) record('authentication-positive.log', output) require(accepted == 0 and output.replace(b'\r\n', b'\n') == b'1\n', 'loopback server did not accept the fixture credential') report['authentication_controls'] = 'wrong-password-rejected; correct-password-accepted' if args.provider_test: report['phase'] = 'provider-tests' - status, output = run([str(args.provider_test.resolve()), address], env, root, 30) + status, output = run([str(args.provider_test.resolve()), address], runtime_env, root, 30) record('provider.log', output) require(status == 0, 'PostgreSQL provider lifecycle tests failed') source = root / 'postgresql.ny' @@ -316,7 +328,12 @@ def record(name, output): '--expected-exit', '0', '--routes', 'all', '--with-stdlib', '--timeout', '30', '--work-dir', str(args.report_dir / 'routes'), '--report-json', str(args.report_dir / 'routes.json')] report['phase'] = 'language-routes' - status, output = run(command, env, root, 180) + route_env = dict(env) + if declared_provider is not None: + # Keep the route harness and its compiler/AOT-link children clean; + # it applies this inert declaration only to execution phases. + route_env[DECLARATION] = declared_provider + status, output = run(command, route_env, root, 180) record('routes.log', output) require(status == 0, 'PostgreSQL language route matrix failed') matrix = json.loads((args.report_dir / 'routes.json').read_bytes()) @@ -324,7 +341,7 @@ def record(name, output): report['all_routes_executed'] = report['routes']['PASS'] == 6 report['phase'] = 'shutdown' shutdown_attempted = True - server.graceful_stop([binary('pg_ctl'), '-D', str(data)], env, root, record) + server.graceful_stop([binary('pg_ctl'), '-D', str(data)], runtime_env, root, record) report['graceful_shutdown'] = True success = True finally: @@ -336,7 +353,7 @@ def record(name, output): if report.get('server_owned_and_ready') and not shutdown_attempted: shutdown_attempted = True try: - server.graceful_stop([binary('pg_ctl'), '-D', str(data)], env, root, record) + server.graceful_stop([binary('pg_ctl'), '-D', str(data)], runtime_env, root, record) report['graceful_shutdown'] = True except Exception: report['graceful_shutdown'] = False @@ -350,6 +367,7 @@ def record(name, output): finally: pwfile.unlink(missing_ok=True) env.pop('PGPASSWORD', None) + runtime_env.pop('PGPASSWORD', None) if success: shutil.rmtree(root) require(not root.exists(), 'private PostgreSQL directory survived cleanup') diff --git a/tests/routes/ny_route_matrix.py b/tests/routes/ny_route_matrix.py index eb18e852..0462e4ae 100644 --- a/tests/routes/ny_route_matrix.py +++ b/tests/routes/ny_route_matrix.py @@ -26,6 +26,9 @@ from pathlib import Path from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from runtime_provider_environment import provider_environments + EXECUTION_ROUTES: Tuple[str, ...] = ( "text-interpreter", @@ -1341,11 +1344,18 @@ def run_route( max_output_chars: int, ) -> Dict[str, Any]: route_dir.mkdir(parents=True, exist_ok=False) + env, runtime_env, declared_provider = provider_environments(env) result = new_route_result(route, route_dir) checks: List[Dict[str, Any]] = result["checks"] def record_command(name: str, process: CommandResult) -> None: result["commands"][name] = process.as_dict(max_output_chars) + if declared_provider is not None: + result["commands"][name]["runtime_loader"] = { + "dyld_fallback_library_path": None if name == "compile" else declared_provider, + "role": "build" if name == "compile" else + "combined JIT compilation/execution" if name == "compile_and_run" else "execution", + } if route in ("text-interpreter", "binary-interpreter"): binary = route == "binary-interpreter" @@ -1385,7 +1395,7 @@ def record_command(name: str, process: CommandResult) -> None: ], route_dir, timeout_seconds, - env, + runtime_env, ) record_command("run", run_process) check_process_contract( @@ -1427,7 +1437,7 @@ def record_command(name: str, process: CommandResult) -> None: ) if compile_process.exit_code == 0 and artifact["valid"]: run_process = run_command( - [str(output)], route_dir, timeout_seconds, env + [str(output)], route_dir, timeout_seconds, runtime_env ) record_command("run", run_process) check_process_contract( @@ -1455,7 +1465,7 @@ def record_command(name: str, process: CommandResult) -> None: jit_command, route_dir, timeout_seconds, - env, + runtime_env, ) record_command("compile_and_run", jit_process) if llvm_is_unavailable(jit_process): @@ -1544,7 +1554,7 @@ def record_command(name: str, process: CommandResult) -> None: [str(expected_executable), *program_args], route_dir, timeout_seconds, - env, + runtime_env, ) record_command("run", run_process) check_process_contract( @@ -1592,7 +1602,7 @@ def record_command(name: str, process: CommandResult) -> None: elif route == "interpreter-reject": run_process = run_command( - [str(interpreter), str(source)], route_dir, timeout_seconds, env + [str(interpreter), str(source)], route_dir, timeout_seconds, runtime_env ) record_command("run", run_process) check_process_contract( diff --git a/tests/runtime_provider_environment.py b/tests/runtime_provider_environment.py new file mode 100644 index 00000000..bee80b5b --- /dev/null +++ b/tests/runtime_provider_environment.py @@ -0,0 +1,31 @@ +"""Explicit test-only Darwin provider paths, separated from build processes.""" +from __future__ import annotations + +from pathlib import PurePosixPath +import sys + +DECLARATION = 'NIL_TEST_DARWIN_RUNTIME_LIBRARY_PATH' +LOADER_VARIABLE = 'DYLD_FALLBACK_LIBRARY_PATH' + + +def provider_environments(environment, *, platform_name=None): + """Return build/runtime environments and the optional declared search path. + + Without the CI/test declaration, ambient loader settings retain their + existing meaning. With it, a conflicting loader override is an error, not + a setting silently removed to make a cache or qualification test pass. + """ + build = dict(environment) + declared = build.pop(DECLARATION, None) + if declared is None: + return build, dict(build), None + if (platform_name or sys.platform) != 'darwin': + raise ValueError('the declared runtime library search path requires macOS') + if (not isinstance(declared, str) or not declared or len(declared) > 32768 or + any(char in declared for char in '\0\r\n') or + any(not part or not PurePosixPath(part).is_absolute() for part in declared.split(':'))): + raise ValueError('runtime library path must contain ordered absolute nonempty Darwin directories') + if any(key.startswith('DYLD_') and value for key, value in build.items()): + raise ValueError('declared runtime library path conflicts with ambient DYLD overrides') + runtime = dict(build, **{LOADER_VARIABLE: declared}) + return build, runtime, declared diff --git a/tests/test_database_postgresql_isolated.py b/tests/test_database_postgresql_isolated.py index 52ad6472..b305bb64 100644 --- a/tests/test_database_postgresql_isolated.py +++ b/tests/test_database_postgresql_isolated.py @@ -1,5 +1,6 @@ """Fixture isolation/cleanup regressions; no PostgreSQL installation required.""" import io +from functools import partial import json from pathlib import Path, PurePosixPath import shutil @@ -45,6 +46,47 @@ def test_environment_removes_provider_overrides_but_preserves_toolchain(self): 'NY_HOME': 'other', 'nil_stdlib_path': 'other', 'PATH': 'tools', 'LIB': 'libs', 'INCLUDE': 'headers'}), {'PATH': 'tools', 'LIB': 'libs', 'INCLUDE': 'headers'}) + def test_declared_provider_is_runtime_only_through_server_and_route_harness(self): + with tempfile.TemporaryDirectory(prefix='nil-pg-env-') as directory: + report_dir = Path(directory) + args = SimpleNamespace(stdlib=report_dir, report_dir=report_dir, + postgres_bin=report_dir, postgres_share=None, provider_test=report_dir / 'provider-test', + compiler=report_dir / 'compiler', interpreter=report_dir / 'interpreter', require_all_routes=True) + server = Mock(launcher=False, output=io.BytesIO(), error=io.BytesIO()) + server.ready.return_value = True + server.owner.process.returncode = 0 + report = {} + commands = [] + declared = '/private/provider/lib:/usr/lib' + + def run(command, env, cwd, timeout): + commands.append((list(map(str, command)), dict(env))) + if Path(command[0]).stem == 'initdb': + (Path(cwd) / 'data').mkdir() + if Path(command[0]).stem == 'psql': + return (1, b'') if env['PGPASSWORD'].endswith('-wrong') else (0, b'1\n') + if any(str(part).endswith('ny_route_matrix.py') for part in command): + (report_dir / 'routes.json').write_text(json.dumps(self.route_report())) + return 0, b'' + + with patch.dict(fixture.os.environ, {fixture.DECLARATION: declared}, clear=True), \ + patch.object(fixture, 'provider_environments', partial(fixture.provider_environments, platform_name='darwin')), \ + patch.object(fixture, 'private_permissions'), \ + patch.object(fixture, 'run', side_effect=run), \ + patch.object(fixture, 'Server', return_value=server) as launch: + fixture.run_fixture(args, report) + self.assertEqual(len(commands), 5 if fixture.os.name == 'nt' else 3) + for command, env in commands[:-1]: + self.assertEqual(env['DYLD_FALLBACK_LIBRARY_PATH'], declared, command) + self.assertNotIn(fixture.DECLARATION, env) + route_parent = commands[-1][1] + self.assertNotIn('DYLD_FALLBACK_LIBRARY_PATH', route_parent) + self.assertEqual(route_parent[fixture.DECLARATION], declared) + self.assertEqual(launch.call_args.args[1]['DYLD_FALLBACK_LIBRARY_PATH'], declared) + self.assertEqual(server.graceful_stop.call_args.args[1]['DYLD_FALLBACK_LIBRARY_PATH'], declared) + self.assertTrue(report['all_routes_executed']) + self.assertTrue(report['owned_process_cleanup']) + def test_windows_configuration_is_loopback_and_authenticated(self): config, hba = fixture.configuration(True, Path('unused'), 24567) self.assertIn("listen_addresses = '127.0.0.1'", config) diff --git a/tests/test_runtime_provider_environment.py b/tests/test_runtime_provider_environment.py new file mode 100644 index 00000000..77602e6b --- /dev/null +++ b/tests/test_runtime_provider_environment.py @@ -0,0 +1,89 @@ +"""Provider loader declarations affect execution, never compile/link children.""" +from functools import partial +import importlib.util +from pathlib import Path +import sys +import tempfile +import unittest +from unittest.mock import patch + +from runtime_provider_environment import DECLARATION, LOADER_VARIABLE, provider_environments + +ROOT = Path(__file__).resolve().parents[1] +SPEC = importlib.util.spec_from_file_location('provider_route_matrix', ROOT / 'tests/routes/ny_route_matrix.py') +routes = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = routes +SPEC.loader.exec_module(routes) + + +class ProviderEnvironmentTests(unittest.TestCase): + def test_ambient_overrides_are_unchanged_without_explicit_declaration(self): + original = {'PATH': '/tools', LOADER_VARIABLE: '/ambient', 'DYLD_LIBRARY_PATH': '/other'} + for host in ('darwin', 'linux', 'win32'): + build, runtime, declared = provider_environments(original, platform_name=host) + self.assertEqual(build, original) + self.assertEqual(runtime, original) + self.assertIsNone(declared) + self.assertIsNot(build, runtime) + + def test_declared_path_is_runtime_only_and_preserves_order(self): + value = '/private/provider with space/lib:/usr/lib' + original = {'PATH': '/tools', DECLARATION: value, 'NIL_STDLIB_PATH': '/sdk/nilstd'} + build, runtime, declared = provider_environments(original, platform_name='darwin') + self.assertEqual(declared, value) + self.assertNotIn(LOADER_VARIABLE, build) + self.assertNotIn(DECLARATION, build) + self.assertEqual(runtime[LOADER_VARIABLE], value) + self.assertEqual(runtime['NIL_STDLIB_PATH'], '/sdk/nilstd') + self.assertEqual(original[DECLARATION], value) + + def test_conflicts_and_ambiguous_declarations_fail_before_execution(self): + for invalid in ('', '/lib:', 'relative:/usr/lib', '/lib\nother', '/lib\0other'): + with self.subTest(invalid=invalid), self.assertRaises(ValueError): + provider_environments({DECLARATION: invalid}, platform_name='darwin') + for name in (LOADER_VARIABLE, 'DYLD_INSERT_LIBRARIES', 'DYLD_LIBRARY_PATH'): + with self.subTest(name=name), self.assertRaisesRegex(ValueError, 'conflicts'): + provider_environments({DECLARATION: '/provider:/usr/lib', name: '/ambient'}, platform_name='darwin') + with self.assertRaisesRegex(ValueError, 'macOS'): + provider_environments({DECLARATION: '/provider:/usr/lib'}, platform_name='linux') + + def test_each_route_assigns_loader_path_to_actual_execution_stage(self): + for route in routes.EXECUTION_ROUTES: + with self.subTest(route=route), tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + seen = [] + declared = '/private/qualified/lib:/usr/lib' + + def command(argv, cwd, timeout, env): + seen.append((list(argv), dict(env))) + return routes.CommandResult(list(map(str, argv)), str(cwd), 0, '', '', 0.0) + + def artifact(*unused): + return {'valid': True, 'errors': [], 'metadata': {}} + + with patch.object(routes, 'provider_environments', partial(provider_environments, platform_name='darwin')), \ + patch.object(routes, 'run_command', side_effect=command), \ + patch.object(routes, 'validate_text_nil', side_effect=artifact), \ + patch.object(routes, 'validate_binary_nil', side_effect=artifact), \ + patch.object(routes, 'validate_bundle', side_effect=artifact), \ + patch.object(routes, 'validate_native_executable', side_effect=artifact): + result = routes.run_route(route, root / route, Path('/compiler'), Path('/interpreter'), + Path('/fixture.ny'), '', '', 0, 5, [], [], [], True, + {'PATH': '/tools', DECLARATION: declared}, 1024) + self.assertEqual(result['status'], 'PASS', result) + if route == 'llvm-jit': + self.assertEqual(len(seen), 1) + self.assertEqual(seen[0][1][LOADER_VARIABLE], declared) + self.assertEqual(result['commands']['compile_and_run']['runtime_loader']['role'], + 'combined JIT compilation/execution') + else: + self.assertEqual(len(seen), 2) + self.assertNotIn(LOADER_VARIABLE, seen[0][1]) + self.assertEqual(seen[1][1][LOADER_VARIABLE], declared) + self.assertIsNone(result['commands']['compile']['runtime_loader']['dyld_fallback_library_path']) + self.assertEqual(result['commands']['run']['runtime_loader']['dyld_fallback_library_path'], declared) + self.assertTrue(all(DECLARATION not in env for _, env in seen)) + + +if __name__ == '__main__': + unittest.main() diff --git a/vscode/test/project-checks.cjs b/vscode/test/project-checks.cjs index 40c2c8ae..a3618fdd 100644 --- a/vscode/test/project-checks.cjs +++ b/vscode/test/project-checks.cjs @@ -59,13 +59,22 @@ exports.run = async ({ check, workspace }) => { ["native", "ny.checkNativeProject", "ny.buildNativeProject", "ny.runNativeProject"] ]) { await check(`${language}: ${target} project check, build and run tasks`, async () => { - assert.equal(await vscode.commands.executeCommand(checkCommand), 0); - assert.equal(await vscode.commands.executeCommand(buildCommand), 0); - assert.equal(await vscode.commands.executeCommand(runCommand), 0); + const phases = []; + for (const command of [checkCommand, buildCommand, runCommand]) { + const start = Date.now(); + console.log(`Starting ${language} ${target} project command: ${command}`); + assert.equal(await vscode.commands.executeCommand(command), 0, command); + const milliseconds = Date.now() - start; + phases.push({ command, milliseconds }); + console.log(`Completed ${language} ${target} project command: ${command} (${milliseconds} ms)`); + } const output = path.join(directory, "build", "hello" + (target === "vm" ? ".nil" : process.platform === "win32" ? ".exe" : "")); assert.ok((await fs.stat(output)).size > 0); - return { target, output }; - }); + return { target, output, phases }; + // Darwin fingerprints the complete dyld backing files in independent + // preparation/verification phases. This group executes three real tools; + // retain those checks with a bounded aggregate budget and phase timings. + }, process.platform === "darwin" ? 180_000 : 45_000); } await check(`${language}: semantic rules and project warning policy agree with project checks`, async () => { const warning = language === "ny" ? 'import nil.std.text\nfn main() -> i32\n try_char_at("abc", 0)\n return 0\n println("unreachable")\nend fn\n' : diff --git a/vscode/test/suite.cjs b/vscode/test/suite.cjs index ba7f162a..098b3987 100644 --- a/vscode/test/suite.cjs +++ b/vscode/test/suite.cjs @@ -41,15 +41,17 @@ exports.run = async function run() { const checks = []; const runId = process.env.NIL_VSCODE_TEST_RUN_ID; assert.ok(runId, "Isolated host run identity is required."); - const report = { status: "RUNNING", runId, vscode: vscode.version, platform: process.platform, workspace, checks, skippedChecks: [] }; - const check = async (name, operation) => { + const report = { status: "RUNNING", runId, vscode: vscode.version, platform: process.platform, workspace, checks, skippedChecks: [], + fullSuiteCompleted: false, completedModules: [] }; + const check = async (name, operation, timeout = 45_000) => { const start = Date.now(); - const detail = await bounded(operation(), name, 45_000); + const detail = await bounded(operation(), name, timeout); checks.push({ name, status: "PASS", milliseconds: Date.now() - start, ...(detail || {}) }); console.log(`PASS ${name}`); }; const runModule = async name => { await require(`./${name}.cjs`).run({ check, workspace }); + report.completedModules.push(name); }; const events = []; report.tasks = events; @@ -378,6 +380,7 @@ exports.run = async function run() { for (const module of ["analyzer-checks", "project-checks", "workspace-checks", "semantic-checks", "completion-checks", "member-signature-checks", "language-values-checks", "memory-profile-checks", "niltest-checks"]) await runModule(module); + report.fullSuiteCompleted = true; report.status = "PASS"; } catch (error) { report.status = "FAIL"; From 860750e3f47930be1dd91ac2d3e41c136f0e6e2b Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 19:20:50 -0500 Subject: [PATCH 13/39] Model universal Darwin tools and non-linkable SDK resources --- nilbuild/darwin_toolchain_inputs.hpp | 151 +++++++++++++++++++--- nilbuild/toolchain_inputs.hpp | 11 +- notebook/2026-11-11-web-runtime-parity.md | 10 ++ tests/darwin_toolchain_inputs_tests.cpp | 116 +++++++++++++++++ 4 files changed, 266 insertions(+), 22 deletions(-) diff --git a/nilbuild/darwin_toolchain_inputs.hpp b/nilbuild/darwin_toolchain_inputs.hpp index 4faaeeed..437a6e77 100644 --- a/nilbuild/darwin_toolchain_inputs.hpp +++ b/nilbuild/darwin_toolchain_inputs.hpp @@ -7,10 +7,12 @@ // is collected again before publication. No timestamps certify cache identity. #include #include +#include #include #include #include #include +#include #include #include #include @@ -188,6 +190,111 @@ inline bool supported_archive_commands(const std::string &text) { text.find("LC_LINKER_OPTION") == std::string::npos; } +inline bool macho_magic(const std::string &bytes) { + if (bytes.size() < 4) return false; + const auto prefix = bytes.substr(0, 4); + return prefix == std::string("\xfe\xed\xfa\xce", 4) || prefix == std::string("\xce\xfa\xed\xfe", 4) || + prefix == std::string("\xfe\xed\xfa\xcf", 4) || prefix == std::string("\xcf\xfa\xed\xfe", 4) || + prefix == std::string("\xca\xfe\xba\xbe", 4) || prefix == std::string("\xbe\xba\xfe\xca", 4) || + prefix == std::string("\xca\xfe\xba\xbf", 4) || prefix == std::string("\xbf\xba\xfe\xca", 4); +} + +// An SDK may ship runnable plug-ins alongside its linkable interfaces. Apple +// ld's ordinary dylib parser rejects MH_EXECUTE and MH_BUNDLE; this profile +// never passes -bundle_loader. Ignore only positively identified such images, +// including every universal slice. A dylib/object, malformed header or unknown +// file remains unmodeled. Sources: cctools mach-o/{loader,fat}.h and ld64's +// src/ld/parsers/macho_dylib_file.cpp, Parser::validFile. +inline std::optional> macho_file_types(const fs::path &path) { + std::vector kinds; + std::error_code error; + const auto size = fs::file_size(path, error); + if (error || size > static_cast(std::numeric_limits::max())) return {}; + std::ifstream file(path, std::ios::binary); + const auto read = [&](std::uint64_t offset, std::size_t count) -> std::optional { + if (offset > size || count > size - offset || count > 4 * 1024 * 1024) return {}; + std::string bytes(count, '\0'); file.clear(); file.seekg(static_cast(offset)); + if (!file.read(bytes.data(), static_cast(count))) return {}; + return bytes; + }; + const auto number = [](const std::string &bytes, std::size_t at, unsigned count, bool big) { + std::uint64_t value = 0; + for (unsigned index = 0; index < count; ++index) + value |= std::uint64_t(static_cast(bytes[at + index])) << + (8 * (big ? count - index - 1 : index)); + return value; + }; + const auto slice = [&](std::uint64_t offset, std::uint64_t extent, + std::optional> cpu) { + const auto header = read(offset, 28); + if (!header || extent < 28) return false; + const auto magic = header->substr(0, 4); + const bool big = magic == std::string("\xfe\xed\xfa\xce", 4) || magic == std::string("\xfe\xed\xfa\xcf", 4); + const bool wide = magic == std::string("\xfe\xed\xfa\xcf", 4) || magic == std::string("\xcf\xfa\xed\xfe", 4); + if (!big && magic != std::string("\xce\xfa\xed\xfe", 4) && magic != std::string("\xcf\xfa\xed\xfe", 4)) return false; + const auto header_size = wide ? 32U : 28U; + if (extent < header_size || !read(offset, header_size)) return false; + if (cpu && (number(*header, 4, 4, big) != cpu->first || (number(*header, 8, 4, big) & 0x00ffffffU) != (cpu->second & 0x00ffffffU))) return false; + const auto kind = number(*header, 12, 4, big); + const auto count = number(*header, 16, 4, big), bytes = number(*header, 20, 4, big); + if (count > 65536 || bytes > 4 * 1024 * 1024 || bytes > extent - header_size) return false; + const auto commands = read(offset + header_size, static_cast(bytes)); + if (!commands) return false; + std::size_t at = 0; + for (std::uint64_t index = 0; index < count; ++index) { + if (at > commands->size() || commands->size() - at < 8) return false; + const auto length = number(*commands, at + 4, 4, big); + if (length < 8 || length % 4 || length > commands->size() - at) return false; + at += static_cast(length); + } + if (at != commands->size()) return false; + kinds.push_back(static_cast(kind)); + return true; + }; + const auto header = read(0, 8); + if (!header || !macho_magic(*header)) return {}; + const auto magic = header->substr(0, 4); + const bool fat_big = magic == std::string("\xca\xfe\xba\xbe", 4) || magic == std::string("\xca\xfe\xba\xbf", 4); + const bool fat_little = magic == std::string("\xbe\xba\xfe\xca", 4) || magic == std::string("\xbf\xba\xfe\xca", 4); + if (!fat_big && !fat_little) { + if (!slice(0, size, {})) return {}; + return kinds; + } + const bool wide = magic == std::string("\xca\xfe\xba\xbf", 4) || magic == std::string("\xbf\xba\xfe\xca", 4); + const auto count = number(*header, 4, 4, fat_big); + const auto stride = wide ? 32U : 20U; + if (!count || count > 64 || size < 8 + count * stride) return {}; + const auto table = read(8, static_cast(count * stride)); + if (!table) return {}; + // ld64 also recognizes one legacy ARM64 descriptor beyond nfat_arch in + // FAT_MAGIC files. Do not silently omit a hidden potentially-linkable slice. + if (!wide && size >= 8 + (count + 1) * stride) { + const auto extra = read(8 + count * stride, stride); + if (!extra) return {}; + if (number(*extra, 0, 4, fat_big) == 0x0100000c && number(*extra, 4, 4, fat_big) <= 1) return {}; + } + std::vector> extents; + for (std::uint64_t index = 0; index < count; ++index) { + const auto at = static_cast(index * stride); + const auto offset = number(*table, at + 8, wide ? 8 : 4, fat_big); + const auto extent = number(*table, at + (wide ? 16 : 12), wide ? 8 : 4, fat_big); + const auto alignment = number(*table, at + (wide ? 24 : 16), 4, fat_big); + if (alignment > 63 || offset < 8 + count * stride || offset > size || extent > size - offset || + offset % (std::uint64_t(1) << alignment) || (wide && number(*table, at + 28, 4, fat_big))) return {}; + for (const auto &previous : extents) + if (offset < previous.second && previous.first < offset + extent) return {}; + extents.emplace_back(offset, offset + extent); + if (!slice(offset, extent, std::make_pair(number(*table, at, 4, fat_big), number(*table, at + 4, 4, fat_big)))) return {}; + } + return kinds; +} + +inline bool non_linkable_macho(const fs::path &path) { + const auto kinds = macho_file_types(path); + return kinds && !kinds->empty() && std::all_of(kinds->begin(), kinds->end(), + [](std::uint32_t kind) { return kind == 2 || kind == 8; }); // MH_EXECUTE / MH_BUNDLE +} + inline bool system_install_name(const std::string &name) { const fs::path path(name); // Install names are always POSIX paths, including in portable parser tests. @@ -381,11 +488,11 @@ class collector { const std::string bytes(prefix, size); if (bytes.rfind("--- !tapi-tbd", 0) == 0) { if (!add(entry) || !read_stub(entry)) return false; - } else if (bytes.rfind("!\n", 0) == 0 || bytes.rfind("!\n", 0) == 0 || - (size >= 4 && (bytes.compare(0, 4, "\xfe\xed\xfa\xce", 4) == 0 || bytes.compare(0, 4, "\xce\xfa\xed\xfe", 4) == 0 || - bytes.compare(0, 4, "\xfe\xed\xfa\xcf", 4) == 0 || bytes.compare(0, 4, "\xcf\xfa\xed\xfe", 4) == 0 || - bytes.compare(0, 4, "\xca\xfe\xba\xbe", 4) == 0 || bytes.compare(0, 4, "\xbe\xba\xfe\xca", 4) == 0 || - bytes.compare(0, 4, "\xca\xfe\xba\xbf", 4) == 0 || bytes.compare(0, 4, "\xbf\xba\xfe\xca", 4) == 0))) return fail("unmodeled extensionless binary SDK input", entry); + } else if (bytes.rfind("!\n", 0) == 0 || bytes.rfind("!\n", 0) == 0) { + return fail("unmodeled extensionless archive SDK input", entry); + } else if (macho_magic(bytes) && !non_linkable_macho(entry)) { + return fail("unmodeled or malformed extensionless Mach-O SDK input", entry); + } continue; // Non-linkable metadata such as framework PkgInfo. } if (!add(entry) || (extension == ".tbd" && !read_stub(entry))) return false; @@ -441,31 +548,37 @@ class collector { if (!inspected_contexts.insert(context).second) return true; if (inspected_contexts.size() > 1024) return fail("tool image contexts exceeded 1024", path); for (const auto &name : description.libraries) { - fs::path selected; + std::vector selected; if (name.rfind("@rpath/", 0) == 0) { const auto tail = fs::path(name.substr(7)); if (tail.has_parent_path()) return fail("tool rpath dependency has an unmodeled suffix", name); + // Tool inspection inventories every universal slice (the OS shim may + // be arm64e while the emitted link target is arm64). Its merged rpaths + // cannot safely pick only one winner: include every existing candidate + // so a different architecture/context's first match cannot be hidden. for (const auto &root : rpaths) { const auto candidate = root / tail; - if (regular(candidate)) { selected = candidate; break; } + if (regular(candidate)) selected.push_back(candidate); } - // A dylib's install ID may use its parent's rpath and appear in otool - // -L. It names this file, not an additional dependency. - if (selected.empty() && path.filename() == tail && path.extension() == ".dylib") selected = path; + // A dylib's install ID is self metadata, not an extra file dependency. + if (selected.empty() && path.filename() == tail && path.extension() == ".dylib") selected.push_back(path); if (selected.empty()) return fail("tool rpath dependency could not be resolved", name); } else if (name.rfind("@loader_path/", 0) == 0) { - selected = (physical.parent_path() / name.substr(13)).lexically_normal(); - if (!inside(selected, plan.installed.parent_path())) return fail("loader-relative dependency escapes toolchain", selected); + const auto candidate = (physical.parent_path() / name.substr(13)).lexically_normal(); + if (!inside(candidate, plan.installed.parent_path())) return fail("loader-relative dependency escapes toolchain", candidate); + selected.push_back(candidate); } else if (system_install_name(name)) { - selected = host_path(name); - // Non-file system images are represented by the compiler's complete - // shared-cache backing-file superset. Ordinary overrides are included. - if (!regular(selected)) continue; + const auto candidate = host_path(name); + // Actual system images without ordinary files are covered by the + // compiler's complete active shared-cache backing-file superset. + if (regular(candidate)) selected.push_back(candidate); } else { - selected = name; - if (!clean_path(selected) || !inside(selected, plan.installed.parent_path())) return fail("tool dependency escapes modeled roots", selected); + const fs::path candidate = name; + if (!clean_path(candidate) || !inside(candidate, plan.installed.parent_path())) return fail("tool dependency escapes modeled roots", candidate); + selected.push_back(candidate); } - if (selected != path && !image(selected, executable_directory, depth + 1, rpaths)) return false; + for (const auto &candidate : selected) + if (candidate != path && !image(candidate, executable_directory, depth + 1, rpaths)) return false; } return true; } diff --git a/nilbuild/toolchain_inputs.hpp b/nilbuild/toolchain_inputs.hpp index 41454172..2132f87c 100644 --- a/nilbuild/toolchain_inputs.hpp +++ b/nilbuild/toolchain_inputs.hpp @@ -288,7 +288,7 @@ inline std::optional> native_toolchain_files( name == "GCC_EXEC_PREFIX" || name == "CPATH" || name == "CPLUS_INCLUDE_PATH" || name == "C_INCLUDE_PATH" || name == "OBJC_INCLUDE_PATH" || name == "ARCHFLAGS" || name == "IPHONEOS_DEPLOYMENT_TARGET" || name == "TVOS_DEPLOYMENT_TARGET" || - name == "WATCHOS_DEPLOYMENT_TARGET") return unavailable("unmodeled tool-selection environment variable: " + name); + name == "WATCHOS_DEPLOYMENT_TARGET" || name == "OTOOL_PATH") return unavailable("unmodeled tool-selection environment variable: " + name); } const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(120); unsigned queries = 0; @@ -337,8 +337,13 @@ inline std::optional> native_toolchain_files( } const auto inspect = [&](const std::filesystem::path &image) -> std::optional { - const auto libraries = query({"/usr/bin/otool", "-arch", plan->architecture, "-L", image.string()}); - const auto commands = query({"/usr/bin/otool", "-arch", plan->architecture, "-l", image.string()}); + const auto kinds = darwin_toolchain_detail::macho_file_types(image); + if (!kinds || !std::all_of(kinds->begin(), kinds->end(), [](std::uint32_t kind) { return kind == 2 || kind == 6; })) { + darwin_toolchain_detail::explain(reason, "unsupported or ambiguous Mach-O tool slices: " + image.string()); + return {}; + } + const auto libraries = query({"/usr/bin/otool", "-arch", "all", "-L", image.string()}); + const auto commands = query({"/usr/bin/otool", "-arch", "all", "-l", image.string()}); if (!libraries || !commands) return {}; const auto description = darwin_toolchain_detail::parse_image(image, *libraries, *commands); if (!description) darwin_toolchain_detail::explain(reason, "unsupported otool image/load-command output: " + image.string()); diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 1c60b649..0b822bff 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -306,6 +306,16 @@ the full-host bound. Full editor acceptance remains open. Early native probes identify the remaining cache refusals: an arm64e system launcher inspected as arm64, and a non-linkable EmailCore plugin embedded in the SDK. +The native collector now inspects every tool slice, including Apple's arm64e +launcher, and conservatively follows every existing candidate from merged +runpaths. SDK omission requires a structurally valid executable/bundle in every +slice; object/dylib/stub, mixed, unknown and malformed images still disable reuse. +The ordinary link plan does not admit `-bundle_loader`. `OTOOL_PATH` is rejected +as an unmodeled tool selector. All 131 portable controls pass strict warnings +and ASan/UBSan, including transitive alternate-runpath dependencies and malformed +universal headers. This implementation still needs the actual hosted probe and +positive native cache reuse to pass. + Paused windows are not pumped, and adapter transport failures use best-effort terminal notification before the reader accepts disconnect/EOF; cancellable arbitrary input streams are outside this slice. No new native debugger or diff --git a/tests/darwin_toolchain_inputs_tests.cpp b/tests/darwin_toolchain_inputs_tests.cpp index bab75c88..808ae7ba 100644 --- a/tests/darwin_toolchain_inputs_tests.cpp +++ b/tests/darwin_toolchain_inputs_tests.cpp @@ -19,6 +19,81 @@ void write(const fs::path &path, const std::string &text) { std::ofstream stream(path, std::ios::binary); stream << text; if (!stream) throw std::runtime_error("cannot write fixture"); } +void integer(std::string &bytes, std::size_t offset, std::uint64_t value, unsigned width, bool big) { + for (unsigned index = 0; index < width; ++index) + bytes.at(offset + index) = static_cast((value >> (8 * (big ? width - index - 1 : index))) & 255); +} +std::string macho(std::uint32_t kind, bool wide = true, bool big = false, + std::uint32_t cpu = 0x0100000c, std::uint32_t subtype = 2) { + std::string bytes(wide ? 32 : 28, '\0'); + integer(bytes, 0, wide ? 0xfeedfacf : 0xfeedface, 4, big); + integer(bytes, 4, cpu, 4, big); integer(bytes, 8, subtype, 4, big); + integer(bytes, 12, kind, 4, big); + return bytes; +} +std::string universal(std::uint32_t first = 2, std::uint32_t second = 8, + bool wide = false, bool big = true) { + std::string bytes(160, '\0'); + integer(bytes, 0, wide ? 0xcafebabf : 0xcafebabe, 4, big); integer(bytes, 4, 2, 4, big); + for (unsigned index = 0; index < 2; ++index) { + const auto at = 8 + index * (wide ? 32 : 20), offset = 96 + index * 32; + const auto cpu = index ? 0x01000007U : 0x0100000cU, subtype = index ? 3U : 2U; + integer(bytes, at, cpu, 4, big); integer(bytes, at + 4, subtype, 4, big); + integer(bytes, at + 8, offset, wide ? 8 : 4, big); + integer(bytes, at + (wide ? 16 : 12), 32, wide ? 8 : 4, big); + integer(bytes, at + (wide ? 24 : 16), 5, 4, big); + bytes.replace(offset, 32, macho(index ? second : first, true, false, cpu, subtype)); + } + return bytes; +} +void machos(const fs::path &root) { + const auto path = root / "mach-o-control"; + const auto accepts = [&](const std::string &bytes) { write(path, bytes); return detail::non_linkable_macho(path); }; + const auto malformed = [&](const std::string &bytes) { write(path, bytes); return !detail::macho_file_types(path); }; + for (bool wide : {false, true}) for (bool big : {false, true}) { + check(accepts(macho(2, wide, big)), "both byte orders and widths admit executable SDK resources"); + check(accepts(macho(8, wide, big)), "both byte orders and widths admit bundle SDK resources"); + check(accepts(universal(2, 8, wide, big)), "every universal executable/bundle slice is positively identified"); + } + for (const auto kind : {1U, 6U, 9U, 0xffffffffU}) { + check(!accepts(macho(kind)), "object/dylib/stub/unknown file types cannot be ignored as SDK resources"); + check(!accepts(universal(8, kind)), "one linkable or unknown universal slice prevents omission"); + } + check(malformed("ordinary metadata"), "ordinary file is not positively identified as a Mach-O resource"); + check(malformed("!\n"), "an archive is not a Mach-O resource"); + check(malformed(macho(8).substr(0, 31)), "truncated 64-bit header is rejected"); + auto commands = macho(8); commands.resize(40); + integer(commands, 16, 1, 4, false); integer(commands, 20, 8, 4, false); + integer(commands, 32, 1, 4, false); integer(commands, 36, 8, 4, false); + check(accepts(commands), "complete bounded load commands are accepted"); + check(malformed(commands.substr(0, 39)), "truncated load commands are rejected"); + auto changed = commands; integer(changed, 36, 7, 4, false); + check(malformed(changed), "undersized load command is rejected"); + changed = commands; integer(changed, 16, 0, 4, false); + check(malformed(changed), "unaccounted load command bytes are rejected"); + changed = commands; integer(changed, 20, 4 * 1024 * 1024 + 1, 4, false); + check(malformed(changed), "excessive load command extent is rejected"); + changed = universal(); integer(changed, 4, 65, 4, true); + check(malformed(changed), "excessive universal slice count is rejected"); + check(malformed(universal().substr(0, 47)), "truncated universal descriptor table is rejected"); + changed = universal(); integer(changed, 16, 144, 4, true); + check(malformed(changed), "universal slice extending beyond file is rejected"); + changed = universal(); integer(changed, 36, 96, 4, true); + check(malformed(changed), "overlapping universal slices are rejected"); + changed = universal(); integer(changed, 16, 32, 4, true); + check(malformed(changed), "universal slice cannot overlap its descriptor table"); + changed = universal(); integer(changed, 24, 64, 4, true); + check(malformed(changed), "invalid universal alignment is rejected before shifting"); + changed = universal(2, 8, true); integer(changed, 36, 1, 4, true); + check(malformed(changed), "fat64 reserved field must be zero"); + changed = universal(); integer(changed, 100, 0x01000007, 4, false); + check(malformed(changed), "universal descriptor must match its slice CPU type"); + changed = universal(); integer(changed, 104, 0x80000002, 4, false); + check(accepts(changed), "CPU subtype capabilities are masked as in Apple ld"); + changed = universal(); integer(changed, 48, 0x0100000c, 4, true); integer(changed, 52, 0, 4, true); + check(malformed(changed), "legacy hidden ARM64 descriptor cannot escape universal inspection"); + fs::remove(path); +} std::string quote(const std::string &text) { std::string result = "\""; for (const auto byte : text) { if (byte == '"' || byte == '\\') result += '\\'; result += byte; } @@ -100,6 +175,13 @@ void parsing(const detail::link_plan &plan, const fs::path &runtime) { "Load command 1\n cmd LC_RPATH\n cmdsize 32\n path @loader_path/../lib (offset 12)\n"); check(description && description->libraries.size() == 1 && description->rpaths == std::vector{"@loader_path/../lib"}, "otool image and rpath inputs parsed together"); + const auto universal_description = detail::parse_image(image, + image.string() + " (architecture arm64e):\n\t@rpath/arm.dylib (compatibility version 1.0.0, current version 1.0.0)\n" + + image.string() + " (architecture x86_64):\n\t@rpath/intel.dylib (compatibility version 1.0.0, current version 1.0.0)\n", + image.string() + " (architecture arm64e):\ncmd LC_RPATH\npath @loader_path/../lib/arm (offset 12)\n" + + image.string() + " (architecture x86_64):\ncmd LC_RPATH\npath @loader_path/../lib/intel (offset 12)\n"); + check(universal_description && universal_description->libraries.size() == 2 && universal_description->rpaths.size() == 2, + "all-architecture tool output preserves both dependency and runpath sets"); check(!detail::parse_image(image, "bad format\n", ""), "unknown image listing rejected"); check(!detail::parse_image(image, image.string() + ":\n", "cmd LC_RPATH\n"), "truncated rpath rejected"); check(!detail::parse_image(image, image.string() + ":\n", "cmd LC_DYLD_ENVIRONMENT\n"), "embedded loader environment rejected"); @@ -152,6 +234,21 @@ void closure(const fs::path &root) { files = collect(); check(files && identities(*files).count(driver_leaf.generic_string()) && identities(*files).count(linker_leaf.generic_string()), "shared image is resolved in both driver and linker inherited-runpath contexts"); + const auto arm_branch = plan.installed.parent_path() / "lib/arm/branch.dylib"; + const auto intel_branch = plan.installed.parent_path() / "lib/intel/branch.dylib"; + const auto arm_child = arm_branch.parent_path() / "child.dylib"; + const auto intel_child = intel_branch.parent_path() / "child.dylib"; + for (const auto &path : {arm_branch, intel_branch, arm_child, intel_child}) write(path, "slice-specific image"); + descriptions[driver] = {{"@rpath/branch.dylib"}, {arm_branch.parent_path().string(), intel_branch.parent_path().string()}}; + descriptions[arm_branch] = {{"@loader_path/child.dylib"}, {}}; + descriptions[intel_branch] = {{"@loader_path/child.dylib"}, {}}; + files = collect(); + check(files && identities(*files).count(arm_branch.generic_string()) && identities(*files).count(arm_child.generic_string()) && + identities(*files).count(intel_branch.generic_string()) && identities(*files).count(intel_child.generic_string()), + "merged universal runpaths retain both same-name candidates and their separate dependency graphs"); + descriptions[intel_branch].libraries.push_back("/unmodeled/second-slice.dylib"); + check(!collect(), "an unsupported dependency in a later universal runpath candidate fails closed"); + descriptions[intel_branch].libraries.pop_back(); descriptions[driver] = {}; descriptions[plan.linker].libraries.pop_back(); descriptions[plan.linker].rpaths.pop_back(); const auto other_driver = plan.installed.parent_path() / "alternate-bin/clang++"; @@ -170,6 +267,24 @@ void closure(const fs::path &root) { before = identities(*files); write(plan.sdk / "usr/lib/new-shadow.tbd", stub("/usr/lib/new-shadow.dylib")); files = collect(); check(files && identities(*files) != before, "new SDK shadow changes input namespace"); + const auto plugin = plan.sdk / "System/Library/PrivateFrameworks/EmailCore.framework/Versions/A/PlugIns/EmailCore.wkbundle/Contents/MacOS/EmailCore"; + for (const auto kind : {2U, 8U}) { + write(plugin, macho(kind)); files = collect(); + check(files && !identities(*files).count(plugin.generic_string()), "positively nonlinkable SDK plugin image is not a native link input"); + } + write(plugin, universal()); files = collect(); + check(files && !identities(*files).count(plugin.generic_string()), "all-nonlinkable universal SDK plugin is omitted"); + for (const auto kind : {1U, 6U, 9U, 0xffffffffU}) { + write(plugin, macho(kind)); + check(!collect(), "SDK plugin location cannot exempt an object, dylib, stub or unknown Mach-O type"); + } + write(plugin, macho(8).substr(0, 31)); + check(!collect(), "truncated SDK plugin image is not silently omitted"); + write(plugin, universal(8, 6)); + check(!collect(), "mixed universal plugin cannot hide a linkable SDK slice"); + write(plugin, "!\n"); + check(!collect(), "extensionless archive in plugin location still disables reuse"); + fs::remove(plugin); before = identities(*files); write(plan.installed.parent_path() / "lib/new-helper.dylib", "new helper"); files = collect(); check(files && identities(*files) != before, "new toolchain helper changes namespace"); @@ -282,6 +397,7 @@ int main() { // through deliberate fixture links rather than an incidental host prefix. const auto root = fs::canonical(nil::detail::create_private_directory(fs::temp_directory_path(), "nil-darwin-closure-")); try { + machos(root); closure(root); fs::remove_all(root); std::cout << "Darwin toolchain closure: " << checks << " checks passed\n"; From 40017f1496dee796bdaa493b6347d7cbe167ae14 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 19:27:43 -0500 Subject: [PATCH 14/39] Correct AppKit interaction state and entry editing limits --- nilcompiler/src/ui_runtime_appkit.mm | 99 ++++++++++++++---- notebook/2026-11-11-web-runtime-parity.md | 24 +++++ tests/ui_runtime_appkit_tests.mm | 122 +++++++++++++++++++++- 3 files changed, 224 insertions(+), 21 deletions(-) diff --git a/nilcompiler/src/ui_runtime_appkit.mm b/nilcompiler/src/ui_runtime_appkit.mm index 58c096af..99943c2b 100644 --- a/nilcompiler/src/ui_runtime_appkit.mm +++ b/nilcompiler/src/ui_runtime_appkit.mm @@ -53,10 +53,17 @@ bool from_ns(NSString *value, std::string &out) { if (!value) { out.clear(); return true; } const char *bytes = [value UTF8String]; if (!bytes) return false; - out.assign(bytes); + // NSString may contain U+0000. Preserve its bytes so valid_text rejects it + // instead of accepting the prefix before the first NUL. + out.assign(bytes, [value lengthOfBytesUsingEncoding:NSUTF8StringEncoding]); return true; } +bool valid_entry(NSString *value) { + std::string text; + return from_ns(value, text) && valid_text(text) && entry_text_fits(text); +} + // A window server session is required before any window is created, so the // failure names the cause instead of surfacing a late AppKit error. bool window_server_available() { @@ -81,12 +88,34 @@ @implementation NyUiContainer - (BOOL)isFlipped { return YES; } @end +// Validate edits before the field editor commits them, including paste and +// replacements in the middle of a string. NSString.length counts UTF-16 code +// units; the common validator counts the contract's Unicode scalar values. +@interface NyUiEntryFormatter : NSFormatter +@end +@implementation NyUiEntryFormatter +- (NSString *)stringForObjectValue:(id)object { + return [object isKindOfClass:[NSString class]] ? object : nil; +} +- (BOOL)getObjectValue:(id *)object forString:(NSString *)string errorDescription:(NSString **)error { + if (!ny::ui::valid_entry(string)) return NO; + if (object) *object = string; + return YES; +} +- (BOOL)isPartialStringValid:(NSString **)string proposedSelectedRange:(NSRangePointer)selection + originalString:(NSString *)original originalSelectedRange:(NSRange)originalSelection + errorDescription:(NSString **)error { + return string && ny::ui::valid_entry(*string); +} +@end + // The delivery block is built inside Context::Impl, which is private to // Context; a block carries that access, where naming the type here could not. typedef void (^NyUiDeliver)(id sender, long long kind); @interface NyUiTarget : NSObject @property(nonatomic, copy) NyUiDeliver deliver; +@property(nonatomic, copy) void (^relayout)(NSWindow *); - (void)buttonAction:(id)sender; - (void)checkAction:(id)sender; @end @@ -102,6 +131,9 @@ - (BOOL)windowShouldClose:(NSWindow *)sender { // Matrix decides when to close; AppKit must not destroy a live handle. return NO; } +- (void)windowDidResize:(NSNotification *)notification { + if (self.relayout) self.relayout(notification.object); +} @end namespace ny::ui { @@ -122,7 +154,6 @@ - (BOOL)windowShouldClose:(NSWindow *)sender { std::thread::id owner = std::this_thread::get_id(); std::map nodes; std::map handles; // view or window -> handle - std::map focus_targets; std::deque events; std::map snapshots; std::size_t event_bytes = 0, snapshot_bytes = 0, text_bytes = 0; @@ -200,6 +231,34 @@ bool effective_sensitive(const Node &node) { } return true; } + NSView *focused_view(std::int64_t window) { + const auto *top = find(window); + if (!top || !top->window_object) return nil; + NSResponder *responder = top->window_object.firstResponder; + if (!responder) return nil; + // AppKit temporarily focuses the shared field editor, not NSTextField. + for (const auto &[id, candidate] : nodes) { + (void)id; + if (candidate.window == window && candidate.kind == Kind::entry && + [(NSTextField *)candidate.view currentEditor] == responder) return candidate.view; + } + return [responder isKindOfClass:[NSView class]] ? (NSView *)responder : nil; + } + bool focused(const Node &node) { + NSView *view = focused_view(node.window); + return view && (view == node.view || [view isDescendantOf:node.view]); + } + void sync_sensitivity(std::int64_t window) { + for (auto &[id, node] : nodes) { + (void)id; + if (node.window != window || container(node.kind)) continue; + const bool enabled = effective_sensitive(node); + if (!enabled && focused(node)) { + if (auto *top = find(window)) [top->window_object makeFirstResponder:nil]; + } + ((NSControl *)node.view).enabled = enabled ? YES : NO; + } + } bool effective_visible(const Node &node) { for (const Node *current = &node;;) { const auto found = current->properties.find("visible"); @@ -223,6 +282,7 @@ void enqueue(std::int64_t handle, std::int64_t kind) noexcept { try { auto *node = find(handle); if (!node || node->suppress || tearing_down) return; + if (kind != 4 && !effective_sensitive(*node)) return; Event event{kind, handle, node->window, 0, {}}; if (kind == 2) { auto *field = ((NSTextField *)node->view); @@ -433,6 +493,12 @@ void relayout(std::int64_t window) { const auto handle = impl->handle_of((__bridge void *)sender); if (handle) impl->enqueue(handle, static_cast(kind)); }; + created.relayout = ^(NSWindow *window) { + auto impl = weak.lock(); + if (!impl || !impl->active.load() || impl->tearing_down) return; + const auto handle = impl->handle_of((__bridge void *)window); + if (handle) impl->relayout(handle); + }; return created; } @@ -449,7 +515,7 @@ void dispose() noexcept { for (auto &entry : nodes) entry.second.view = nil; } if (target) { target.deliver = nil; target = nil; } - nodes.clear(); handles.clear(); focus_targets.clear(); + nodes.clear(); handles.clear(); events.clear(); snapshots.clear(); event_bytes = snapshot_bytes = text_bytes = 0; tearing_down = false; @@ -572,6 +638,7 @@ void dispose() noexcept { NSTextField *field = [[NSTextField alloc] initWithFrame:NSZeroRect]; field.stringValue = text; field.editable = YES; field.selectable = YES; field.bezeled = YES; + field.formatter = [[NyUiEntryFormatter alloc] init]; field.delegate = impl_->target; node.view = field; } else if (kind == "checkbox") { @@ -602,6 +669,9 @@ void dispose() noexcept { [owner->view addSubview:view]; impl_->nodes.emplace(handle, std::move(node)); impl_->handles.emplace((__bridge void *)view, handle); + auto &inserted = impl_->nodes.at(handle); + if (!container(inserted.kind)) + ((NSControl *)view).enabled = impl_->effective_sensitive(inserted) ? YES : NO; impl_->text_bytes += label.size(); impl_->relayout(impl_->nodes.at(handle).window); } catch (...) { @@ -693,8 +763,8 @@ void dispose() noexcept { if (!impl_->effective_sensitive(*node)) return failure("widget cannot receive keyboard focus"); auto *window = impl_->find(node->window); if (!window || !window->window_object) return failure("widget cannot receive keyboard focus"); - impl_->focus_targets[node->window] = handle; - [window->window_object makeFirstResponder:node->view]; + if (![window->window_object makeFirstResponder:node->view]) + return failure("widget could not receive keyboard focus"); return success(); } const bool flag = property == "visible" || property == "sensitive" || property == "hexpand" || property == "vexpand" || property == "checked"; @@ -721,7 +791,7 @@ void dispose() noexcept { impl_->relayout(node->window); } } else if (property == "sensitive") { - if (!container(node->kind)) ((NSControl *)node->view).enabled = value ? YES : NO; + impl_->sync_sensitivity(node->window); } else if (property == "checked") { ++node->suppress; ((NSButton *)node->view).state = value ? NSControlStateValueOn : NSControlStateValueOff; @@ -754,20 +824,12 @@ void dispose() noexcept { } if (property == "effective_visible") return success(impl_->effective_visible(*node) ? 1 : 0); if (property == "sensitive") { - if (container(node->kind)) { - auto found = node->properties.find("sensitive"); - return success(found == node->properties.end() || found->second ? 1 : 0); - } - return success(((NSControl *)node->view).enabled ? 1 : 0); + const auto found = node->properties.find("sensitive"); + return success(found == node->properties.end() || found->second ? 1 : 0); } if (property == "effective_sensitive") return success(impl_->effective_sensitive(*node) ? 1 : 0); if (property == "focus") { - auto found = impl_->focus_targets.find(node->window); - if (found == impl_->focus_targets.end()) return success(0); - if (found->second == handle) return success(1); - auto *target = impl_->find(found->second); - // A compound widget's internal child counts as the widget's own focus. - return success(target && [target->view isDescendantOf:node->view] ? 1 : 0); + return success(impl_->focused(*node) ? 1 : 0); } if (property == "width" || property == "height") { if (node->kind == Kind::window) { @@ -922,8 +984,6 @@ void dispose() noexcept { if (!targeted(it->second.target)) { ++it; continue; } impl_->snapshot_bytes -= it->second.text.size(); it = impl_->snapshots.erase(it); } - auto focus = impl_->focus_targets.find(window); - if (focus != impl_->focus_targets.end() && targeted(focus->second)) impl_->focus_targets.erase(focus); impl_->relayout(window); return success(); } @@ -958,7 +1018,6 @@ void dispose() noexcept { if (it->second.window != window) { ++it; continue; } impl_->snapshot_bytes -= it->second.text.size(); it = impl_->snapshots.erase(it); } - impl_->focus_targets.erase(window); return success(); } } // namespace ny::ui diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 0b822bff..af7e0e47 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -337,6 +337,30 @@ consumers together, adapt C-string FFI boundaries explicitly, and reject mixed old/new object and runtime ABIs. A hidden pointer-length registry would retain the same migration burden while adding lifetime and registration ambiguity. +## AppKit interaction behavior follow-up + +A comparison with the actual native controls found gaps beyond the old +cross-process input fixture restriction. AppKit's container sensitivity affected +only model flags, focus queries remembered the last explicit request, native +window resizes did not rerun layout, and editing could exceed the shared entry +scalar limit. The provider now propagates effective sensitivity while preserving +local flags, maps the real responder/field editor back to its widget, relayouts +on resize and validates edits through an NSFormatter. NSString conversion retains +embedded NUL bytes so the common text validator rejects them. + +New real-control assertions cover disabled descendants and later children, +nested re-enable behavior, externally changed focus, field-editor termination, +native resize, astral Unicode at the 16,384-scalar limit, bounded replacement and +NUL rejection. Source review found no blocker; actual hosted compilation and +window-server execution remain required for these new cases. + +Shared layout behavior is a separate follow-up: Win32 and AppKit retain hidden +children's layout slots, represent spanned grid tracks by their starting track, +and do not distribute extra grid space using expansion flags. Exact cross-toolkit +geometry is not currently specified by the public contract. These observations +are retained for an explicit layout contract rather than being called macOS-only +implementation gaps. + ## SQL provider environment scope The full Mac source/SDK workflows previously exported SQL's diff --git a/tests/ui_runtime_appkit_tests.mm b/tests/ui_runtime_appkit_tests.mm index c5aaf846..5fa0e31c 100644 --- a/tests/ui_runtime_appkit_tests.mm +++ b/tests/ui_runtime_appkit_tests.mm @@ -64,6 +64,125 @@ void rejected(const ny::ui::Result &result, const char *fragment, const char *wh return 0; } +NSWindow *native_window(NSString *title) { + NSWindow *found = nil; + for (NSWindow *candidate in NSApp.windows) { + if (![candidate.title isEqualToString:title]) continue; + require(found == nil, "fixture window title is unique within this process"); + found = candidate; + } + require(found != nil, "fixture window belongs to this application"); + return found; +} + +NSControl *native_control(NSView *root, NSString *text) { + if ([root isKindOfClass:[NSButton class]] && [((NSButton *)root).title isEqualToString:text]) + return (NSControl *)root; + if ([root isKindOfClass:[NSTextField class]] && [((NSTextField *)root).stringValue isEqualToString:text]) + return (NSControl *)root; + for (NSView *child in root.subviews) + if (NSControl *found = native_control(child, text)) return found; + return nil; +} + +// Native changes deliberately bypass Context setters: otherwise stale cached +// focus and model-only sensitivity can pass without affecting real controls. +void interaction_state_contract() { + @autoreleasepool { + ny::ui::Context context; + const auto window = ok(context.window("NIL AppKit interaction state", 420, 280), "create state window"); + const auto group = ok(context.widget("column", window, ""), "create state group"); + const auto nested = ok(context.widget("row", group, ""), "create nested state group"); + const auto entry = ok(context.widget("entry", nested, "editable"), "create state entry"); + const auto box = ok(context.widget("checkbox", nested, "Toggle state"), "create state checkbox"); + const auto button = ok(context.widget("button", group, "Local disabled"), "create state button"); + const auto sibling = ok(context.widget("button", window, "Outside group"), "create state sibling"); + ok(context.set_number(window, "visible", 1), "show state window"); + NSWindow *top = native_window(@"NIL AppKit interaction state"); + auto *field = (NSTextField *)native_control(top.contentView, @"editable"); + auto *check = (NSButton *)native_control(top.contentView, @"Toggle state"); + auto *local = (NSButton *)native_control(top.contentView, @"Local disabled"); + auto *outside = (NSButton *)native_control(top.contentView, @"Outside group"); + require(field && check && local && outside, "all fixture controls are native children"); + + ok(context.set_number(entry, "focus", 1), "focus state entry"); + require(field.currentEditor == top.firstResponder, "entry uses the native field editor"); + require(ok(context.read_number(entry, "focus"), "read field focus") == 1, "field editor counts as entry focus"); + require([top makeFirstResponder:outside], "native responder changes without a provider request"); + require(ok(context.read_number(entry, "focus"), "old field focus") == 0, "old focus request is not cached as current focus"); + require(ok(context.read_number(sibling, "focus"), "new native focus") == 1, "native focus change reaches the provider"); + require(ok(context.read_number(group, "focus"), "group focus outside") == 0, "unrelated group does not own focus"); + require([top makeFirstResponder:field], "native field focus changes without a provider request"); + require(ok(context.read_number(nested, "focus"), "nested field focus") == 1, "field editor belongs to its logical ancestors"); + + ok(context.set_number(button, "sensitive", 0), "disable local button"); + ok(context.set_number(group, "sensitive", 0), "disable ancestor of focused entry"); + require(!field.enabled && !check.enabled && !local.enabled && outside.enabled, + "ancestor sensitivity disables native descendants and preserves siblings"); + require(field.currentEditor == nil && ok(context.read_number(entry, "focus"), "disabled entry focus") == 0, + "disabling a focused subtree ends native editing"); + require(ok(context.read_number(box, "sensitive"), "local checkbox sensitivity") == 1, + "ancestor disable preserves the local enabled flag"); + [check performClick:nil]; [local performClick:nil]; + require(check.state == NSControlStateValueOff && next_event(context, window) == 0, + "native activation of disabled descendants changes neither state nor events"); + const auto late = ok(context.widget("button", nested, "Created disabled"), "create child under disabled ancestor"); + auto *late_native = native_control(top.contentView, @"Created disabled"); + require(late_native && !late_native.enabled, "new native children inherit disabled state"); + rejected(context.set_number(late, "focus", 1), "focus", "disabled child cannot receive focus"); + ok(context.set_number(nested, "sensitive", 0), "disable nested ancestor locally"); + ok(context.set_number(group, "sensitive", 1), "restore outer ancestor"); + require(!field.enabled && !check.enabled && !late_native.enabled && !local.enabled, + "restoring outer ancestor preserves nested and local disabled flags"); + ok(context.set_number(nested, "sensitive", 1), "restore nested ancestor"); + require(field.enabled && check.enabled && late_native.enabled && !local.enabled, + "restoring nested ancestor restores only locally enabled controls"); + [check performClick:nil]; + const auto event = next_event(context, window); + require(event != 0 && ok(context.event_number(event, "kind"), "native toggle kind") == 3 && + ok(context.event_number(event, "target"), "native toggle target") == box && + ok(context.event_number(event, "checked"), "native toggle state") == 1, + "restored native checkbox delivers its actual state change"); + ok(context.release_event(event), "release native toggle"); + + ok(context.set_number(group, "hexpand", 1), "expand state group"); + const auto old_width = ok(context.read_number(group, "width"), "state group initial width"); + [top setContentSize:NSMakeSize(620, 380)]; + (void)next_event(context, window); + require(ok(context.read_number(group, "width"), "state group resized width") > old_width, + "native window resize relayouts expanding children without a provider size write"); + + ok(context.set_text(entry, std::string(16383, 'a')), "set entry one scalar below limit"); + ok(context.set_number(entry, "focus", 1), "focus bounded entry"); + NSTextView *editor = (NSTextView *)field.currentEditor; + require(editor != nil, "bounded entry has an actual field editor"); + [editor setSelectedRange:NSMakeRange(editor.string.length, 0)]; + [editor insertText:@"\U0001f642" replacementRange:NSMakeRange(NSNotFound, 0)]; + const auto inserted = next_event(context, window); + require(inserted != 0 && ok(context.event_number(inserted, "kind"), "boundary edit kind") == 2, + "native edit at exactly 16384 scalars produces a text event"); + const std::string expected = std::string(16383, 'a') + "\xf0\x9f\x99\x82"; + require(context.event_text(inserted).text == expected && context.read_text(entry).text == expected, + "entry limit counts astral characters as one scalar, not two UTF-16 units"); + ok(context.release_event(inserted), "release boundary edit"); + [editor insertText:@"x" replacementRange:NSMakeRange(NSNotFound, 0)]; + require(context.read_text(entry).text == expected && next_event(context, window) == 0, + "native input beyond the scalar limit is rejected without a change event"); + [editor insertText:@"ok" replacementRange:NSMakeRange(0, editor.string.length)]; + const auto replaced = next_event(context, window); + require(replaced != 0 && context.read_text(entry).text == "ok", "a bounded whole-field replacement remains editable"); + ok(context.release_event(replaced), "release replacement edit"); + + // An invalid NSString arriving from the toolkit must not become a valid + // prefix at the C++ boundary, even if it bypasses the editing formatter. + require([top makeFirstResponder:outside], "finish native entry editing"); + const unichar nul_text[] = {'o', 'k', 0, 'x'}; + field.stringValue = [NSString stringWithCharacters:nul_text length:4]; + rejected(context.read_text(entry), "UTF-8", "native NUL is rejected rather than truncated"); + ok(context.close(window), "close state fixture"); + } +} + void contract() { ny::ui::Context context; const auto window = ok(context.window("NIL AppKit provider", 420, 320), "create window"); @@ -266,10 +385,11 @@ int main(int argc, char **argv) { if (mode == "--display") { require(desktop, "--display requires a window server session"); contract(); + interaction_state_contract(); std::cout << "UI runtime: " << checks << " checks passed (--display)\n"; return 0; } - if (desktop) contract(); + if (desktop) { contract(); interaction_state_contract(); } else { ny::ui::Context context; rejected(context.window("no session", 320, 240), "window server", From 1ddaa32053ba7e0132ddb208869e755a32e59194 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 19:40:20 -0500 Subject: [PATCH 15/39] Reject unmodeled SDK archive fallback and handle editor file aliases --- nilbuild/darwin_toolchain_inputs.hpp | 57 ++++++++++++++++++++++- notebook/2026-11-11-web-runtime-parity.md | 43 +++++++++++++---- tests/darwin_toolchain_inputs_tests.cpp | 33 +++++++++++++ vscode/test/project-checks.cjs | 21 +++++++-- 4 files changed, 139 insertions(+), 15 deletions(-) diff --git a/nilbuild/darwin_toolchain_inputs.hpp b/nilbuild/darwin_toolchain_inputs.hpp index 437a6e77..6ea439c5 100644 --- a/nilbuild/darwin_toolchain_inputs.hpp +++ b/nilbuild/darwin_toolchain_inputs.hpp @@ -648,6 +648,61 @@ class collector { } return true; } + bool direct_library_inputs() { + // Options.cpp:861-873 searches each directory for dylib/TBD, .so, then + // archive before advancing. Only the two command-line -l roots may select + // an archive: supported TBD re-exports name dylibs/frameworks explicitly. + // A selected archive could carry new LC_LINKER_OPTION roots, so require + // captured text interfaces for these two lookups. Unrelated SDK archives + // and archives shadowed by an earlier interface remain harmless inputs. + // Selection precedes architecture parsing: InputFiles.cpp's constructor / + // parseWorkerThread ignores or errors on incompatible inputs, never resumes + // findLibrary. textstub_dylib_file.cpp's architecture fallback reparses the + // same selected interface, so it cannot expose a shadowed archive either. + std::vector directories; + const auto directory = [&](const fs::path &path) -> std::optional { + std::error_code error; + const bool present = fs::is_directory(path, error); + if (error && error != std::errc::no_such_file_or_directory) { + fail("could not inspect direct library search directory", path); return {}; + } + return present; + }; + const auto sdk_local = plan.sdk / "usr/local/lib"; + const auto has_sdk_local = directory(sdk_local); + if (!has_sdk_local) return false; + if (plan.local_libraries) { + if (*has_sdk_local) directories.push_back(sdk_local); + else { + const auto local = host_path("/usr/local/lib"); + const auto present = directory(local); + if (!present) return false; + if (*present) directories.push_back(local); + } + } + // The one-SDK profile omits absent implicit host directories. Explicit -L + // comes first; the implicit SDK order is /usr/lib, then /usr/local/lib. + directories.push_back(plan.sdk / "usr/lib"); + if (*has_sdk_local) directories.push_back(sdk_local); + for (const auto *name : {"c++", "System"}) { + bool selected = false; + for (const auto &root : directories) { + for (const auto *extension : {".tbd", ".dylib", ".so", ".a"}) { + const auto path = root / (std::string("lib") + name + extension); + std::error_code error; + const auto status = fs::status(path, error); + if (error == std::errc::no_such_file_or_directory || status.type() == fs::file_type::not_found) continue; + if (error || !fs::is_regular_file(status)) return fail("direct library candidate is unreadable or not regular", path); + if (path.extension() == ".a") return fail("selected SDK archive has unmodeled autolink dependencies", path); + if (!files_seen.count(path) || !read_stub(path)) return fail("direct library root is not a captured text interface", path); + selected = true; break; + } + if (selected) break; + } + if (!selected) return fail("required direct library text interface was not found", name); + } + return true; + } public: collector(fs::path selected_driver, fs::path directory, fs::path selected_runtime, link_plan selected_plan, probe inspect_image, fs::path host_root = "/", std::string *failure_reason = nullptr) @@ -683,7 +738,7 @@ class collector { if (error) return refuse("could not inspect optional SDK directory", directory); if (present && !tree(directory, true)) return {}; } - if (!local_library_inputs()) return {}; + if (!local_library_inputs() || !direct_library_inputs()) return {}; // Absolute re-export lookup differs from standard -L/-F lookup: when the // SDK/inlined stub cannot satisfy it, ld can try the raw host pathname. // Until ordinary host link-input closure is modeled, refuse that profile. diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index af7e0e47..10bba5b3 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -337,6 +337,30 @@ consumers together, adapt C-string FFI boundaries explicitly, and reject mixed old/new object and runtime ABIs. A hidden pointer-length registry would retain the same migration burden while adding lifetime and registration ambiguity. +At `860750e`, both actual Mac native-input probes collect 8,321 regular inputs. +Both native incremental steps also pass: ARM's verified report contains 131 +closure and 78 real/fake incremental checks, including positive VM/native reuse +and invalidation of changed bytes with unchanged timestamps. Intel's successful +step takes roughly 387 seconds; its final raw artifact is still pending. + +A subsequent source review closes an additional cache-admission gap: an SDK +archive selected for the direct c++/System library roots could add unmodeled +link options. The collector now follows the supported linker's directory order +and requires the selected roots to be captured text interfaces. It permits +unrelated or shadowed archives while rejecting a newly exposed archive fallback. +All 143 strict-warning and ASan/UBSan checks pass; the old source fails the new +stub-removal regression. Default hosted acceptance of this refinement remains +required. + +The ARM editor at `860750e` completes 32 groups, including all four Ny/Matrix +VM/native project check/build/run groups. Measured totals are approximately +48–50 seconds for VM and 90–93 seconds for native. Its next failure is a lookup +assumption for a closed imported file: the correct single diagnostic is present +under `/private/tmp`, while the test reads `/tmp`. The test now collects findings +by physical file identity without deduplicating them, preserving the exact +one-finding, source, location, dirty-buffer and external-edit checks. Full editor +acceptance remains pending. + ## AppKit interaction behavior follow-up A comparison with the actual native controls found gaps beyond the old @@ -351,15 +375,16 @@ embedded NUL bytes so the common text validator rejects them. New real-control assertions cover disabled descendants and later children, nested re-enable behavior, externally changed focus, field-editor termination, native resize, astral Unicode at the 16,384-scalar limit, bounded replacement and -NUL rejection. Source review found no blocker; actual hosted compilation and -window-server execution remain required for these new cases. - -Shared layout behavior is a separate follow-up: Win32 and AppKit retain hidden -children's layout slots, represent spanned grid tracks by their starting track, -and do not distribute extra grid space using expansion flags. Exact cross-toolkit -geometry is not currently specified by the public contract. These observations -are retained for an explicit layout contract rather than being called macOS-only -implementation gaps. +NUL rejection. At `40017f1`, hosted run `43127447798` passes the real provider on +both Macs: 156 display checks and 41 in-process keyboard checks per host, plus +availability. This targeted run does not yet qualify the new Matrix input shim. + +Further comparison confirms actual shared layout gaps in Win32 and AppKit: +hidden children retain slots, a lone spanning control can be clipped to a +fraction of its requested size, and grids ignore expansion. Nested containers +also fail to inherit descendant expansion. These affect real layouts beyond +font/pixel differences and are being corrected with shared bounded allocation +math and native geometry assertions against the same public Context API. ## SQL provider environment scope diff --git a/tests/darwin_toolchain_inputs_tests.cpp b/tests/darwin_toolchain_inputs_tests.cpp index 808ae7ba..db754537 100644 --- a/tests/darwin_toolchain_inputs_tests.cpp +++ b/tests/darwin_toolchain_inputs_tests.cpp @@ -208,6 +208,7 @@ void closure(const fs::path &root) { write(plan.sdk / "SDKSettings.json", "SDK15.5"); write(host / "System/Library/CoreServices/SystemVersion.plist", "macOS15"); write(plan.sdk / "usr/lib/libSystem.tbd", stub()); + write(plan.sdk / "usr/lib/libc++.tbd", stub("/usr/lib/libc++.1.dylib")); write(plan.sdk / "usr/lib/system/libdyld.tbd", stub("/usr/lib/system/libdyld.dylib")); write(plan.sdk / "System/Library/Frameworks/Cocoa.framework/Cocoa.tbd", stub("/System/Library/Frameworks/Cocoa.framework/Cocoa")); write(plan.sdk / "System/Library/Frameworks/CoreGraphics.framework/CoreGraphics.tbd", stub("/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics")); @@ -220,6 +221,38 @@ void closure(const fs::path &root) { auto files = collect(); check(files.has_value(), "default SDK/toolchain closure resolves"); auto before = identities(*files); + write(plan.sdk / "usr/lib/libunrelated.a", "!\n"); + check(collect().has_value(), "unrelated SDK archive cannot introduce autolinks into fixed direct roots"); + for (const auto *name : {"c++", "System"}) { + const auto direct = plan.sdk / "usr/lib" / (std::string("lib") + name); + const auto archive = fs::path(direct.string() + ".a"), interface = fs::path(direct.string() + ".tbd"); + write(archive, "!\n"); + check(collect().has_value(), "same-directory text interface precedes a direct archive fallback"); + const auto text = name == std::string("c++") ? stub("/usr/lib/libc++.1.dylib") : stub(); + auto incompatible = text; + const std::string compatible_targets = "arm64-macos, x86_64-macos"; + incompatible.replace(incompatible.find(compatible_targets), compatible_targets.size(), "i386-macos"); + write(interface, incompatible); + check(collect().has_value(), "architecture parsing cannot resume path search into a shadowed archive (no link success claim)"); + fs::remove(interface); + std::string archive_reason; + check(!detail::collector(driver, project, runtime, plan, inspect, host, &archive_reason).files() && + archive_reason.find("selected SDK archive") != std::string::npos, + "removing direct stub exposes archive autolinks and disables reuse"); + fs::remove(archive); + check(!collect(), "missing direct text root cannot certify successful native reuse"); + write(interface, text); + } + const auto early_archive = plan.sdk / "usr/local/lib/libc++.a"; + write(early_archive, "!\n"); + check(collect().has_value(), "implicit SDK usr/lib interface precedes a later local archive"); + plan.local_libraries = true; + check(!collect(), "explicit SDK-local archive precedes the standard SDK text interface"); + write(plan.sdk / "usr/local/lib/libc++.tbd", stub("/usr/lib/libc++.1.dylib")); + check(collect().has_value(), "earlier SDK-local text interface safely shadows its archive"); + fs::remove_all(plan.sdk / "usr/local/lib"); + plan.local_libraries = false; + files = collect(); before = identities(*files); for (const auto &path : {driver, plan.linker, plan.lto, plan.compiler_rt, runtime, plan.installed.parent_path() / "lib/libtapi.dylib", plan.sdk / "usr/lib/system/libdyld.tbd"}) check(before.count(path.generic_string()) != 0, "driver, helpers, runtime and transitive stub bytes included"); diff --git a/vscode/test/project-checks.cjs b/vscode/test/project-checks.cjs index a3618fdd..61a1b51b 100644 --- a/vscode/test/project-checks.cjs +++ b/vscode/test/project-checks.cjs @@ -10,6 +10,17 @@ async function eventually(operation, description) { throw new Error(`Timed out: ${description}`); } const findings = uri => vscode.languages.getDiagnostics(uri).filter(item => item.source === "nil-analysis"); +// Imported files which are not open are reported by their compiler-resolved +// path. A workspace may use a physical alias (notably /tmp on Darwin). Inspect +// every matching URI without deduplication, so duplicate Problems still fail. +async function importedFindings(uri) { + const results = []; + for (const [candidate, diagnostics] of vscode.languages.getDiagnostics()) { + if (candidate.scheme === "file" && await sameFile(candidate.fsPath, uri.fsPath)) + results.push(...diagnostics.filter(item => item.source === "nil-analysis")); + } + return results; +} async function replace(document, text) { const edit = new vscode.WorkspaceEdit(); edit.replace(document.uri, new vscode.Range(document.positionAt(0), document.positionAt(document.getText().length)), text); @@ -114,7 +125,7 @@ exports.run = async ({ check, workspace }) => { await vscode.window.showTextDocument(document); await eventually(() => vscode.commands.executeCommand("ny.analyzeCurrentFile"), "current compiler check"); } - const errors = findings(depUri).filter(item => item.severity === vscode.DiagnosticSeverity.Error); + const errors = (await importedFindings(depUri)).filter(item => item.severity === vscode.DiagnosticSeverity.Error); assert.equal(errors.length, 1, "Two entry reports should not duplicate a dependency finding."); assert.equal(errors[0].range.start.line, 3, "UTF-8 and CRLF offsets must map the imported source exactly."); const depDocument = await vscode.workspace.openTextDocument(depUri); @@ -122,17 +133,17 @@ exports.run = async ({ check, workspace }) => { await replace(depDocument, broken.replace('"bad"', "1")); await vscode.window.showTextDocument(entries[0]); await eventually(() => vscode.commands.executeCommand("ny.analyzeCurrentFile"), "current compiler check"); - await eventually(() => findings(depUri).length === 0, "dirty import diagnostic removal"); + await eventually(async () => (await importedFindings(depUri)).length === 0, "dirty import diagnostic removal"); assert.equal(await fs.readFile(dependency, "utf8"), broken, "Analysis must not save imported buffers."); assert.equal(await depDocument.save(), true); - await eventually(() => entries.every(document => findings(document.uri).length === 0) && findings(depUri).length === 0, "saved import diagnostics refreshed"); + await eventually(async () => entries.every(document => findings(document.uri).length === 0) && (await importedFindings(depUri)).length === 0, "saved import diagnostics refreshed"); // A disk-only edit must invalidate entries even when the import is closed. await vscode.window.showTextDocument(depDocument); await vscode.commands.executeCommand("workbench.action.closeActiveEditor"); await fs.writeFile(dependency, broken); - await eventually(() => findings(depUri).some(item => item.range.start.line === 3), "external import change refreshed Problems"); + await eventually(async () => (await importedFindings(depUri)).some(item => item.range.start.line === 3), "external import change refreshed Problems"); await fs.writeFile(dependency, broken.replace('"bad"', "1")); - await eventually(() => findings(depUri).length === 0, "external import repair cleared Problems"); + await eventually(async () => (await importedFindings(depUri)).length === 0, "external import repair cleared Problems"); return { importedLine: 4, duplicateCount: errors.length }; }); }; From 1efbe6623445cfcc486b8b7a7794dd112c357224 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 19:48:03 -0500 Subject: [PATCH 16/39] Align native UI layouts and qualify AppKit language input --- .github/workflows/macos-appkit-provider.yml | 5 +- CMakeLists.txt | 22 +- README.md | 4 +- docs/platform-support.md | 16 +- docs/standard-library/ui.md | 21 +- nilcompiler/src/ui_layout.hpp | 156 ++++++++++++++ nilcompiler/src/ui_runtime_appkit.mm | 179 +++++++++------- nilcompiler/src/ui_runtime_win32.cpp | 176 ++++++++------- notebook/2026-11-11-web-runtime-parity.md | 32 ++- .../tools/test_ui_appkit_display_contract.py | 200 ++++++++++++++++++ tests/tools/ui_display_contract.py | 139 ++++++++++-- tests/ui_appkit_input_driver.mm | 176 +++++++++++++++ tests/ui_layout_provider_contract.hpp | 85 ++++++++ tests/ui_layout_tests.cpp | 100 +++++++++ tests/ui_runtime_appkit_tests.mm | 4 +- tests/ui_runtime_tests.cpp | 3 +- tests/ui_runtime_win32_tests.cpp | 4 +- vscode/test/project-checks.cjs | 4 +- 18 files changed, 1140 insertions(+), 186 deletions(-) create mode 100644 nilcompiler/src/ui_layout.hpp create mode 100644 tests/tools/test_ui_appkit_display_contract.py create mode 100644 tests/ui_appkit_input_driver.mm create mode 100644 tests/ui_layout_provider_contract.hpp create mode 100644 tests/ui_layout_tests.cpp diff --git a/.github/workflows/macos-appkit-provider.yml b/.github/workflows/macos-appkit-provider.yml index ea6d641e..dd871ac5 100644 --- a/.github/workflows/macos-appkit-provider.yml +++ b/.github/workflows/macos-appkit-provider.yml @@ -130,7 +130,7 @@ jobs: timeout-minutes: 40 run: >- cmake --build build/ci --parallel 2 --target nilcompiler nilinterpreter - niltest nil_ui_runtime_appkit_tests nil_ui_lifecycle_tests + niltest nil_ui_runtime_appkit_tests nil_ui_lifecycle_tests nil_ui_appkit_input_driver nil_ui_layout_tests - name: Language-level desktop contract timeout-minutes: 15 run: | @@ -140,7 +140,7 @@ jobs: python3 - <<'PY' import xml.etree.ElementTree as ET cases = list(ET.parse('build/ci/appkit-display-results.xml').getroot().iter('testcase')) - expected = {'nil.ui-runtime', 'nil.ui-windows-display-harness', 'nil.ui-display', + expected = {'nil.ui-runtime', 'nil.ui-layout', 'nil.ui-windows-display-harness', 'nil.ui-appkit-display-harness', 'nil.ui-display', 'nil.ui-intrinsic-contract', 'nil.ui-models'} assert len(cases) == len(expected) and {case.get('name') for case in cases} == expected assert all(case.get('status') == 'run' and case.find('skipped') is None and @@ -155,5 +155,6 @@ jobs: build/ci/appkit-display-results.xml build/ci/Testing/Temporary/LastTest.log build/ci/CMakeCache.txt + build/ci/appkit-input-evidence/ if-no-files-found: error retention-days: 14 diff --git a/CMakeLists.txt b/CMakeLists.txt index 7bfb4943..25dd1346 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -367,6 +367,11 @@ if(BUILD_TESTING) add_test(NAME nil.etp-codec COMMAND $) set_tests_properties(nil.etp-codec PROPERTIES LABELS "tools;ipc;etp" TIMEOUT 30) + add_executable(nil_ui_layout_tests tests/ui_layout_tests.cpp) + target_compile_features(nil_ui_layout_tests PRIVATE cxx_std_17) + add_test(NAME nil.ui-layout COMMAND $) + set_tests_properties(nil.ui-layout PROPERTIES LABELS "ui;layout;parity" TIMEOUT 30) + # The provider fixture follows the provider: tests/ui_runtime_tests.cpp # inspects loaded GTK modules and symbol origins, so it describes the GTK # provider and cannot speak for Win32. @@ -382,6 +387,12 @@ if(BUILD_TESTING) target_include_directories(nil_ui_runtime_appkit_tests PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/nilcompiler/src") target_compile_features(nil_ui_runtime_appkit_tests PRIVATE cxx_std_17) set(NIL_UI_PROVIDER_TEST nil_ui_runtime_appkit_tests) + # Test-only input into the owning process's AppKit responder chain. + # No install/export rule: SDKs never carry or load this fixture. + add_library(nil_ui_appkit_input_driver SHARED tests/ui_appkit_input_driver.mm) + target_compile_features(nil_ui_appkit_input_driver PRIVATE cxx_std_17) + target_compile_options(nil_ui_appkit_input_driver PRIVATE -fobjc-arc) + target_link_libraries(nil_ui_appkit_input_driver PRIVATE "-framework Cocoa") else() add_executable(nil_ui_runtime_tests tests/ui_runtime_tests.cpp) target_link_libraries(nil_ui_runtime_tests PRIVATE nilruntime Threads::Threads) @@ -774,8 +785,17 @@ if(BUILD_TESTING) add_test(NAME nil.ui-windows-display-harness COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/test_windows_ui_display_contract.py") set_tests_properties(nil.ui-windows-display-harness PROPERTIES LABELS "ui;tools;windows" TIMEOUT 45) + add_test(NAME nil.ui-appkit-display-harness + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/test_ui_appkit_display_contract.py") + set_tests_properties(nil.ui-appkit-display-harness PROPERTIES LABELS "ui;tools;apple" TIMEOUT 30) set(NIL_UI_DISPLAY_CONTRACT "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/ui_display_contract.py") set(NIL_UI_DISPLAY_TIMEOUT 240) + set(NIL_UI_DISPLAY_APPKIT_ARGS) + if(APPLE) + set(NIL_UI_DISPLAY_TIMEOUT 420) + set(NIL_UI_DISPLAY_APPKIT_ARGS --appkit-input-driver $ + --appkit-evidence-directory "${CMAKE_CURRENT_BINARY_DIR}/appkit-input-evidence") + endif() if(WIN32) set(NIL_UI_DISPLAY_CONTRACT "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/windows_ui_display_contract.py") set(NIL_UI_DISPLAY_TIMEOUT 660) @@ -790,7 +810,7 @@ if(BUILD_TESTING) --interaction-source "${CMAKE_CURRENT_SOURCE_DIR}/tests/ui/gtk_interaction.matrix" --provider-test $ --lifecycle-test $ - --llvm "${NIL_ENABLE_LLVM}") + --llvm "${NIL_ENABLE_LLVM}" ${NIL_UI_DISPLAY_APPKIT_ARGS}) set_tests_properties(nil.ui-display PROPERTIES LABELS "ui;routes" SKIP_RETURN_CODE 77 TIMEOUT "${NIL_UI_DISPLAY_TIMEOUT}") add_test(NAME nil.ui-intrinsic-contract COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/ui_intrinsic_contract.py" diff --git a/README.md b/README.md index 23393feb..85ef4551 100644 --- a/README.md +++ b/README.md @@ -264,8 +264,8 @@ current SDK desktop campaign remain open. The newer socket, entropy and text-byt intrinsics pass the same interpreter contracts on Linux, Windows and both Macs. File timestamps share an implementation across interpreter and native routes, with text/binary NIL, JIT and AOT checks passing on all four hosts. -Both Macs now pass the native allocation-retention contracts and desktop debugger -sessions. Darwin incremental reuse, complete Mac editor coverage and current SDK +Both Macs now pass native allocation-retention contracts, desktop debugger +sessions and native incremental reuse. Complete Mac editor coverage and current SDK qualification remain open, so full platform parity is not yet established. The [platform support matrix](docs/platform-support.md) separates verified development behavior from remaining SDK, provider and release work. diff --git a/docs/platform-support.md b/docs/platform-support.md index 6d434a58..3df97b33 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -11,8 +11,8 @@ release boundary. A newer source checkout does not update an installed SDK. | --- | --- | --- | | Linux x86_64 | Reference host. The clean `5630388` SDK passed installed consumer routes, shared runtime, debugger, Unicode/I/O, checkpoints and GTK checks. The newer `8376252` LLVM-ON source suite passes all 586 CTests. | The reviewed Ubuntu release toolchain is separate from this development SDK. New release binaries still require the release gates. | | Windows x86_64 | The `5630388` installed SDK retains paired Linux acceptance. The newer private `f79b94b` source build passes 579 of 583 registered tests; separately pinned follow-ups resolve its fixture failure and execute the three skipped contracts. Real Win32 desktop and all 88 VS Code host groups pass. | The `8376252` debugger also passes six CTests and four real Win32 desktop sessions. Current-source SDK qualification remains open. Bare-host installation and Windows release qualification remain open. PostgreSQL requires an external runtime; the current UI provider is Win32. | -| macOS 15 Intel x86_64 | At `0c88d2f`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Earlier LLVM-ON/OFF, native assembly and embedded-context baselines also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse, full editor acceptance, a current-source SDK and release qualification remain open. | -| macOS 15 Apple Silicon arm64 | At `3414ce5`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Native assembly and embedded context also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse, full editor acceptance, a current-source SDK and release qualification remain open. | +| macOS 15 Intel x86_64 | At `0c88d2f`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Earlier LLVM-ON/OFF, native assembly and embedded-context baselines also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse passes at `860750e`; full editor acceptance, a current-source SDK and release qualification remain open. | +| macOS 15 Apple Silicon arm64 | At `3414ce5`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Native assembly and embedded context also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse passes at `860750e`; full editor acceptance, a current-source SDK and release qualification remain open. | LLVM-OFF builds exercise frontend/interpreter behavior and rejection of disabled native operations. Their successful tests do not establish native execution. @@ -149,13 +149,14 @@ the complete admission rules. ## Remaining macOS work -- Native incremental reuse is still being qualified. The compiler now resolves +- Native incremental reuse passes on both Macs at `860750e`. The compiler resolves loaded images to ordinary files and exact shared-cache backing files for full content hashing; its actual collector test passes on both Macs at `2632bc6`. The builder uses separate preparation and verification digest phases without - caching source/manifest reads. Native linker dependency collection and real - VM/native reuse still require both hosted architectures to pass. Incomplete - or unsupported dependency profiles deliberately rebuild. + caching source/manifest reads. Both actual linker-dependency probes collect + 8,321 inputs, and both real VM/native reuse and invalidation campaigns pass. + A subsequent SDK archive-fallback refinement still needs hosted qualification. + Incomplete or unsupported dependency profiles deliberately rebuild. - A separate [development SDK workflow](../.github/workflows/macos-sdk.yml) now implements native Darwin consumer admission, root install layout, pinned documentation tooling, and staged dependency/relocation checks. Both @@ -178,7 +179,8 @@ the complete admission rules. Matrix desktop suite and the interaction suite that drives real keyboard input -- while a session without an interactive window station still skips. On macOS the provider is AppKit. Its published contract passes against a - real window server session on both hosted architectures, 93 checks each, + real window server session on both hosted architectures; the newer `40017f1` + provider passes 156 display and 41 in-process keyboard checks per host, and `nil.ui-display` now passes there too: the provider fixture, the lifecycle fixture and the Matrix desktop suite. Real keyboard input is covered in process by the AppKit provider fixture. The current AppKit diff --git a/docs/standard-library/ui.md b/docs/standard-library/ui.md index 5d53728b..ee0b1c0f 100644 --- a/docs/standard-library/ui.md +++ b/docs/standard-library/ui.md @@ -80,6 +80,14 @@ Paused debug sessions do not automatically pump window events. AppKit supplies no box or grid container, so, as on Win32, the provider owns the layout pass and sizes come from each control's fitting size. +Hidden children reserve no layout space. A grid span preserves the child's +full requested size, and expanding tracks share available space. Containers +inherit visible descendants' expansion requests unless explicitly overridden; +cross-axis fill is independent of requesting extra major-axis space. Win32 and +AppKit share bounded integer allocation, cached measurement and iterative +placement. Native geometry assertions compare these behaviors against GTK +without requiring identical fonts or pixel measurements. + ## Windows and widgets `Window` and `Widget` are distinct records containing checked provider handles. @@ -205,6 +213,12 @@ global input injection, change focus or switch desktops. Desktop interaction cas run sequentially with `niltest --jobs 1`; see [compiler-backed testing](../tools/testing.md) for filtering, reports and timeouts. +The macOS harness uses a test-only library inside each owning process to send +input through its verified AppKit responder chain. It exercises the same Matrix +interaction source and records process-bound evidence without Accessibility +permissions or global input. This library is not installed in the SDK. Hosted +qualification of this new language-level input path is still pending. + The same display harness checks real GTK window counts during VM pauses and after completion, faults, resets, reloads and destruction. Native context completion/restart releases windows, while copied text remains valid across @@ -212,7 +226,6 @@ managed collection. CI installs the optional GTK/Xvfb dependencies so desktop coverage is required there; local builds without them report a test skip. The Windows harness's `--require-display` mode turns a missing desktop into a qualification failure. Optional headless mode checks an explicitly configured -or executable-adjacent GTK deployment before allowing a skip; a broken selected -runtime or any failure after starting a display fixture is never a skip. A -headless skip is not desktop qualification. Separate loader and noninteractive -provider probes cover DLL selection and rejection before GTK initialization. +desktop before allowing a skip; any failure after starting a display fixture +remains a failure. A headless skip is not desktop qualification. Linux loader +probes separately cover GTK library selection and rejection before initialization. diff --git a/nilcompiler/src/ui_layout.hpp b/nilcompiler/src/ui_layout.hpp new file mode 100644 index 00000000..b20ee6da --- /dev/null +++ b/nilcompiler/src/ui_layout.hpp @@ -0,0 +1,156 @@ +#pragma once + +// Toolkit-independent allocation for the checked ui.v1 row/column/grid model. +// Coordinates and node counts have the same bounds as the public provider. +#include +#include +#include +#include +#include +#include + +namespace ny::ui::layout { +using Pixel = std::int64_t; + +// Native widget coordinates are signed 32-bit integers on Win32. Extremely +// sparse/nested valid grids can exceed that addressable extent; keep their +// requisitions and off-screen coordinates representable on every provider. +inline Pixel native_extent(Pixel value) noexcept { + return std::max(Pixel(0), std::min(Pixel(std::numeric_limits::max()), value)); +} + +inline bool expands(std::optional local, bool visible, bool descendants) { + return visible && local.value_or(descendants); +} + +struct Item { + std::int64_t start = 0, span = 1; + Pixel natural = 0; + bool expand = false, visible = true; +}; + +struct Axis { + Pixel natural = 0, allocated = 0; + std::vector positions, sizes; + + Pixel extent(std::int64_t start, std::int64_t span) const { + const auto last = static_cast(start + span - 1); + return positions.at(last) + sizes.at(last) - positions.at(static_cast(start)); + } +}; + +namespace detail { +// Range additions and range sums over the occupied tracks. A second instance +// indexes only expanding tracks, so a 65,536-cell span needs logarithmic work +// even when its spare pixels belong to a sparse subset of tracks. +class Additions { + std::vector delta_, weighted_; + static void add(std::vector &tree, std::size_t index, Pixel value) { + for (; index < tree.size(); index += index & (~index + 1)) tree[index] += value; + } + static Pixel sum(const std::vector &tree, std::size_t index) { + Pixel value = 0; + for (; index; index -= index & (~index + 1)) value += tree[index]; + return value; + } + Pixel prefix(std::size_t end) const { + return static_cast(end) * sum(delta_, end) - sum(weighted_, end); + } +public: + explicit Additions(std::size_t count) : delta_(count + 1), weighted_(count + 1) {} + void increase(std::size_t begin, std::size_t end, Pixel amount) { + if (begin == end || !amount) return; + add(delta_, begin + 1, amount); add(delta_, end + 1, -amount); + add(weighted_, begin + 1, amount * static_cast(begin)); + add(weighted_, end + 1, -amount * static_cast(end)); + } + Pixel total(std::size_t begin, std::size_t end) const { return prefix(end) - prefix(begin); } + Pixel at(std::size_t index) const { return sum(delta_, index + 1); } +}; +} // namespace detail + +inline Axis arrange(const std::vector &items, Pixel spacing, Pixel available = 0) { + if (items.size() > 4096 || spacing < 0 || spacing > 4096 || available < 0 || + available > std::numeric_limits::max()) + throw std::invalid_argument("UI layout bounds exceeded"); + std::size_t extent = 0; + for (const auto &item : items) { + if (item.start < 0 || item.span < 1 || item.start > 65535 || item.span > 65536 - item.start || + item.natural < 0 || item.natural > std::numeric_limits::max()) + throw std::invalid_argument("UI layout item bounds exceeded"); + if (item.visible) extent = std::max(extent, static_cast(item.start + item.span)); + } + Axis result; + result.positions.resize(extent); result.sizes.resize(extent); + if (!extent) return result; + std::vector coverage(extent + 1), requested_expand(extent + 1); + std::vector initial(extent); + std::vector single_expand(extent); + for (const auto &item : items) { + if (!item.visible) continue; + const auto begin = static_cast(item.start), end = begin + static_cast(item.span); + ++coverage[begin]; --coverage[end]; + if (item.span == 1) { + initial[begin] = std::max(initial[begin], item.natural); + single_expand[begin] = single_expand[begin] || item.expand; + } + } + std::vector single_prefix(extent + 1); + for (std::size_t index = 0; index < extent; ++index) + single_prefix[index + 1] = single_prefix[index] + (single_expand[index] ? 1 : 0); + // An expanding span uses an already-expanding track if it has one. Otherwise + // all its tracks may expand. Applying these requests together avoids order + // dependence between different spanning controls. + for (const auto &item : items) { + if (!item.visible || !item.expand || item.span == 1) continue; + const auto begin = static_cast(item.start), end = begin + static_cast(item.span); + if (single_prefix[end] == single_prefix[begin]) { ++requested_expand[begin]; --requested_expand[end]; } + } + std::vector occupied_prefix(extent + 1), expanded_prefix(1, 0); + std::vector base_prefix(1, 0); + std::vector expanding; + int covered = 0, requested = 0; + for (std::size_t index = 0; index < extent; ++index) { + covered += coverage[index]; requested += requested_expand[index]; + occupied_prefix[index + 1] = occupied_prefix[index] + (covered ? 1 : 0); + if (!covered) continue; + expanding.push_back(single_expand[index] || requested > 0); + expanded_prefix.push_back(expanded_prefix.back() + (expanding.back() ? 1 : 0)); + base_prefix.push_back(base_prefix.back() + initial[index]); + } + const auto count = expanding.size(); + detail::Additions all(count), expanded(expanded_prefix.back()); + const auto sum = [&](std::size_t begin, std::size_t end) { + return base_prefix[end] - base_prefix[begin] + all.total(begin, end) + + expanded.total(expanded_prefix[begin], expanded_prefix[end]); + }; + const auto distribute = [&](std::size_t begin, std::size_t end, Pixel extra, bool use_expanding) { + auto &changes = use_expanding ? expanded : all; + if (use_expanding) { begin = expanded_prefix[begin]; end = expanded_prefix[end]; } + const auto selected = static_cast(end - begin); + if (extra <= 0 || !selected) return; + changes.increase(begin, end, extra / selected); + changes.increase(begin, begin + static_cast(extra % selected), 1); + }; + for (const auto &item : items) { + if (!item.visible || item.span == 1) continue; + const auto begin = occupied_prefix[static_cast(item.start)]; + const auto end = occupied_prefix[static_cast(item.start + item.span)]; + const Pixel current = sum(begin, end) + spacing * static_cast(end - begin - 1); + distribute(begin, end, item.natural - current, expanded_prefix[end] != expanded_prefix[begin]); + } + result.natural = sum(0, count) + spacing * static_cast(count - 1); + if (available > result.natural && expanded_prefix.back()) + distribute(0, count, available - result.natural, true); + std::size_t active = 0; + for (std::size_t coordinate = 0; coordinate < extent; ++coordinate) { + if (occupied_prefix[coordinate + 1] == occupied_prefix[coordinate]) continue; + if (active) result.allocated += spacing; + result.positions[coordinate] = result.allocated; + const Pixel size = initial[coordinate] + all.at(active) + + (expanding[active] ? expanded.at(expanded_prefix[active]) : 0); + result.sizes[coordinate] = size; result.allocated += size; ++active; + } + return result; +} +} // namespace ny::ui::layout diff --git a/nilcompiler/src/ui_runtime_appkit.mm b/nilcompiler/src/ui_runtime_appkit.mm index 99943c2b..dd9179fd 100644 --- a/nilcompiler/src/ui_runtime_appkit.mm +++ b/nilcompiler/src/ui_runtime_appkit.mm @@ -17,9 +17,11 @@ // and this file owns the layout pass, exactly as the Win32 provider does. #include "ui_runtime.hpp" #include "ui_runtime_shared.hpp" +#include "ui_layout.hpp" #include #include +#include #include #include #include @@ -307,6 +309,23 @@ void enqueue(std::int64_t handle, std::int64_t kind) noexcept { if (node.parent == handle) result.push_back(id); return result; } + std::map> layout_children_cache; + std::map> layout_expansion_cache; + std::map layout_natural_cache; + bool layout_visible(const Node &node) const { + const auto value = node.properties.find("visible"); + return value == node.properties.end() || value->second != 0; + } + const std::vector &layout_children(std::int64_t handle) const { + static const std::vector empty; + const auto found = layout_children_cache.find(handle); + return found == layout_children_cache.end() ? empty : found->second; + } + bool expands(const Node &node, const char *axis) const { + const auto id = handle_of((__bridge void *)node.view); + const auto found = layout_expansion_cache.find(id); + return found != layout_expansion_cache.end() && (axis[0] == 'h' ? found->second.first : found->second.second); + } std::int64_t property(const Node &node, const char *name, std::int64_t fallback = 0) { auto found = node.properties.find(name); if (found != node.properties.end()) return found->second; @@ -315,6 +334,8 @@ void enqueue(std::int64_t handle, std::int64_t kind) noexcept { return fallback; } NSSize natural(std::int64_t handle) { + const auto cached = layout_natural_cache.find(handle); + if (cached != layout_natural_cache.end()) return cached->second; auto *node = find(handle); if (!node) return NSMakeSize(0, 0); NSSize size = NSMakeSize(0, 0); @@ -330,23 +351,20 @@ NSSize natural(std::int64_t handle) { break; } default: { - const auto kids = children_of(handle); + const auto kids = layout_children(handle); const auto spacing = static_cast(property(*node, "spacing")); if (node->kind == Kind::grid) { - std::map widths, heights; + std::vector columns, rows; for (auto id : kids) { - auto *child = find(id); - if (!child) continue; + const auto &child = nodes.at(id); const auto wanted = natural(id); - auto &width = widths[child->placement.column]; - auto &height = heights[child->placement.row]; - width = std::max(width, wanted.width / std::max(1, child->placement.columns)); - height = std::max(height, wanted.height / std::max(1, child->placement.rows)); + columns.push_back({child.placement.column, child.placement.columns, + static_cast(std::ceil(wanted.width)), expands(child, "hexpand")}); + rows.push_back({child.placement.row, child.placement.rows, + static_cast(std::ceil(wanted.height)), expands(child, "vexpand")}); } - for (const auto &entry : widths) size.width += entry.second; - for (const auto &entry : heights) size.height += entry.second; - if (!widths.empty()) size.width += spacing * (widths.size() - 1); - if (!heights.empty()) size.height += spacing * (heights.size() - 1); + size.width = static_cast(layout::arrange(columns, spacing).natural); + size.height = static_cast(layout::arrange(rows, spacing).natural); } else { const bool vertical = node->kind != Kind::row; for (auto id : kids) { @@ -366,16 +384,25 @@ NSSize natural(std::int64_t handle) { size.width += property(*node, "margin_start") + property(*node, "margin_end"); size.height += property(*node, "margin_top") + property(*node, "margin_bottom"); } + size.width = layout::native_extent(static_cast(std::ceil(size.width))); + size.height = layout::native_extent(static_cast(std::ceil(size.height))); + layout_natural_cache.emplace(handle, size); return size; } - void place(std::int64_t handle, NSRect area) { + void place_one(std::int64_t handle, NSRect area, std::vector> &pending) { auto *node = find(handle); if (!node) return; + const auto right = layout::native_extent(static_cast(area.origin.x + area.size.width)); + const auto bottom = layout::native_extent(static_cast(area.origin.y + area.size.height)); + area.origin.x = layout::native_extent(static_cast(area.origin.x)); + area.origin.y = layout::native_extent(static_cast(area.origin.y)); + area.size.width = std::max(0, right - area.origin.x); + area.size.height = std::max(0, bottom - area.origin.y); if (node->kind != Kind::window) { - area.origin.x += property(*node, "margin_start"); - area.origin.y += property(*node, "margin_top"); - area.size.width -= property(*node, "margin_start") + property(*node, "margin_end"); - area.size.height -= property(*node, "margin_top") + property(*node, "margin_bottom"); + area.origin.x = layout::native_extent(static_cast(area.origin.x) + property(*node, "margin_start")); + area.origin.y = layout::native_extent(static_cast(area.origin.y) + property(*node, "margin_top")); + area.size.width = right - property(*node, "margin_end") - area.origin.x; + area.size.height = bottom - property(*node, "margin_bottom") - area.origin.y; if (area.size.width < 0) area.size.width = 0; if (area.size.height < 0) area.size.height = 0; node->allocation = area; @@ -396,85 +423,89 @@ void place(std::int64_t handle, NSRect area) { if (inner.size.width < 0) inner.size.width = 0; if (inner.size.height < 0) inner.size.height = 0; - const auto kids = children_of(handle); + const auto kids = layout_children(handle); if (kids.empty()) return; const auto spacing = static_cast(property(*node, "spacing")); if (node->kind == Kind::grid) { - std::map widths, heights; + std::vector columns, rows; for (auto id : kids) { - auto *child = find(id); - if (!child) continue; + const auto &child = nodes.at(id); const auto wanted = natural(id); - auto &width = widths[child->placement.column]; - auto &height = heights[child->placement.row]; - width = std::max(width, wanted.width / std::max(1, child->placement.columns)); - height = std::max(height, wanted.height / std::max(1, child->placement.rows)); + columns.push_back({child.placement.column, child.placement.columns, + static_cast(std::ceil(wanted.width)), expands(child, "hexpand")}); + rows.push_back({child.placement.row, child.placement.rows, + static_cast(std::ceil(wanted.height)), expands(child, "vexpand")}); } - std::map column_offset, row_offset; - CGFloat x = inner.origin.x; - for (auto &entry : widths) { column_offset[entry.first] = x; x += entry.second + spacing; } - CGFloat y = inner.origin.y; - for (auto &entry : heights) { row_offset[entry.first] = y; y += entry.second + spacing; } + const auto horizontal = layout::arrange(columns, static_cast(spacing), + static_cast(inner.size.width)); + const auto vertical = layout::arrange(rows, static_cast(spacing), + static_cast(inner.size.height)); for (auto id : kids) { - auto *child = find(id); - if (!child) continue; - CGFloat width = 0, height = 0; - for (std::int64_t span = 0; span < child->placement.columns; ++span) { - auto found = widths.find(child->placement.column + span); - if (found != widths.end()) width += found->second + (span ? spacing : 0); - } - for (std::int64_t span = 0; span < child->placement.rows; ++span) { - auto found = heights.find(child->placement.row + span); - if (found != heights.end()) height += found->second + (span ? spacing : 0); - } - const CGFloat left = column_offset.count(child->placement.column) - ? column_offset[child->placement.column] : inner.origin.x; - const CGFloat top = row_offset.count(child->placement.row) - ? row_offset[child->placement.row] : inner.origin.y; - place(id, NSMakeRect(left, top, width, height)); + const auto &child = nodes.at(id); + const auto left = inner.origin.x + horizontal.positions.at(static_cast(child.placement.column)); + const auto top = inner.origin.y + vertical.positions.at(static_cast(child.placement.row)); + const auto width = horizontal.extent(child.placement.column, child.placement.columns); + const auto height = vertical.extent(child.placement.row, child.placement.rows); + pending.emplace_back(id, NSMakeRect(left, top, width, height)); } return; } const bool vertical = node->kind != Kind::row; - CGFloat used = spacing * (kids.size() - 1); - std::size_t expanding = 0; - std::vector wanted; - wanted.reserve(kids.size()); + std::vector items; for (auto id : kids) { - wanted.push_back(natural(id)); - used += vertical ? wanted.back().height : wanted.back().width; - auto *child = find(id); - if (child && property(*child, vertical ? "vexpand" : "hexpand")) ++expanding; + const auto &child = nodes.at(id); + const auto wanted = natural(id); + items.push_back({static_cast(items.size()), 1, + static_cast(std::ceil(vertical ? wanted.height : wanted.width)), + expands(child, vertical ? "vexpand" : "hexpand")}); } - const CGFloat span = vertical ? inner.size.height : inner.size.width; - const CGFloat extra = span > used ? span - used : 0; - const CGFloat share = expanding ? extra / expanding : 0; - - CGFloat offset = vertical ? inner.origin.y : inner.origin.x; + const auto axis = layout::arrange(items, static_cast(spacing), + static_cast(vertical ? inner.size.height : inner.size.width)); for (std::size_t index = 0; index < kids.size(); ++index) { - auto *child = find(kids[index]); - if (!child) continue; - const bool expand = property(*child, vertical ? "vexpand" : "hexpand") != 0; - const CGFloat length = (vertical ? wanted[index].height : wanted[index].width) + (expand ? share : 0); - NSRect target; - if (vertical) { - const CGFloat width = property(*child, "hexpand") - ? inner.size.width : std::min(wanted[index].width, inner.size.width); - target = NSMakeRect(inner.origin.x, offset, width, length); - } else { - const CGFloat height = property(*child, "vexpand") - ? inner.size.height : std::min(wanted[index].height, inner.size.height); - target = NSMakeRect(offset, inner.origin.y, length, height); + const auto offset = axis.positions[index], length = axis.sizes[index]; + const NSRect target = vertical + ? NSMakeRect(inner.origin.x, inner.origin.y + offset, inner.size.width, length) + : NSMakeRect(inner.origin.x + offset, inner.origin.y, length, inner.size.height); + pending.emplace_back(kids[index], target); + } + } + void place(std::int64_t handle, NSRect area) { + std::vector> pending{{handle, area}}; + while (!pending.empty()) { + const auto next = pending.back(); pending.pop_back(); + place_one(next.first, next.second, pending); + } + } + void measure_layout(std::int64_t window) { + layout_children_cache.clear(); layout_expansion_cache.clear(); layout_natural_cache.clear(); + for (const auto &[id, node] : nodes) + if (node.window == window && node.parent && layout_visible(node)) layout_children_cache[node.parent].push_back(id); + // Handles increase at creation and a parent must already exist. Reverse + // order therefore measures each descendant before its parent, without + // recursive expansion/measurement or stack growth for deeply nested UI. + for (auto it = nodes.rbegin(); it != nodes.rend(); ++it) { + const auto &[id, node] = *it; + if (node.window != window) continue; + bool horizontal = false, vertical = false; + for (auto child : layout_children(id)) { + const auto &flags = layout_expansion_cache.at(child); + horizontal = horizontal || flags.first; vertical = vertical || flags.second; } - place(kids[index], target); - offset += length + spacing; + const auto flag = [&](const char *name, bool descendants) { + const auto local = node.properties.find(name); + return layout::expands(local == node.properties.end() ? std::nullopt : std::optional(local->second != 0), + layout_visible(node), descendants); + }; + layout_expansion_cache.emplace(id, std::make_pair(flag("hexpand", horizontal), flag("vexpand", vertical))); + (void)natural(id); } } void relayout(std::int64_t window) { auto *node = find(window); if (!node || node->kind != Kind::window) return; + measure_layout(window); NSRect content = [node->window_object contentRectForFrameRect:node->window_object.frame]; if (content.size.width <= 0 || content.size.height <= 0) content = NSMakeRect(0, 0, static_cast(property(*node, "width", 1)), diff --git a/nilcompiler/src/ui_runtime_win32.cpp b/nilcompiler/src/ui_runtime_win32.cpp index 578a0c72..ddeb11ee 100644 --- a/nilcompiler/src/ui_runtime_win32.cpp +++ b/nilcompiler/src/ui_runtime_win32.cpp @@ -14,6 +14,7 @@ // sensitivity inherit, and clipping behave, the way the contract describes. #include "ui_runtime.hpp" #include "ui_runtime_shared.hpp" +#include "ui_layout.hpp" #include #include @@ -268,6 +269,23 @@ struct Context::Impl : std::enable_shared_from_this { if (node.parent == handle) result.push_back(id); return result; } + std::map> layout_children_cache; + std::map> layout_expansion_cache; + std::map layout_natural_cache; + bool layout_visible(const Node &node) const { + const auto value = node.properties.find("visible"); + return value == node.properties.end() || value->second != 0; + } + const std::vector &layout_children(std::int64_t handle) const { + static const std::vector empty; + const auto found = layout_children_cache.find(handle); + return found == layout_children_cache.end() ? empty : found->second; + } + bool expands(const Node &node, const char *axis) const { + const auto id = handle_of(node.object); + const auto found = layout_expansion_cache.find(id); + return found != layout_expansion_cache.end() && (axis[0] == 'h' ? found->second.first : found->second.second); + } std::int64_t property(const Node &node, const char *name, std::int64_t fallback = 0) { auto found = node.properties.find(name); if (found != node.properties.end()) return found->second; @@ -275,7 +293,10 @@ struct Context::Impl : std::enable_shared_from_this { if (all != node.properties.end() && std::string(name).rfind("margin_", 0) == 0) return all->second; return fallback; } + static LONG native_pixel(layout::Pixel value) { return static_cast(layout::native_extent(value)); } SIZE natural(std::int64_t handle) { + const auto cached = layout_natural_cache.find(handle); + if (cached != layout_natural_cache.end()) return cached->second; auto *node = find(handle); if (!node) return SIZE{0, 0}; SIZE size{0, 0}; @@ -302,51 +323,49 @@ struct Context::Impl : std::enable_shared_from_this { break; } default: { - const auto kids = children_of(handle); + const auto kids = layout_children(handle); const auto spacing = static_cast(property(*node, "spacing")); if (node->kind == Kind::grid) { - std::map widths, heights; + std::vector columns, rows; for (auto id : kids) { - auto *child = find(id); - if (!child) continue; + const auto &child = nodes.at(id); const auto wanted = natural(id); - auto &width = widths[child->placement.column]; - auto &height = heights[child->placement.row]; - width = (std::max)(width, static_cast(wanted.cx / (std::max)(std::int64_t(1), child->placement.columns))); - height = (std::max)(height, static_cast(wanted.cy / (std::max)(std::int64_t(1), child->placement.rows))); + columns.push_back({child.placement.column, child.placement.columns, + static_cast(wanted.cx), expands(child, "hexpand")}); + rows.push_back({child.placement.row, child.placement.rows, + static_cast(wanted.cy), expands(child, "vexpand")}); } - for (const auto &[column, width] : widths) { (void)column; size.cx += width; } - for (const auto &[row, height] : heights) { (void)row; size.cy += height; } - if (!widths.empty()) size.cx += spacing * static_cast(widths.size() - 1); - if (!heights.empty()) size.cy += spacing * static_cast(heights.size() - 1); + size.cx = native_pixel(layout::arrange(columns, spacing).natural); + size.cy = native_pixel(layout::arrange(rows, spacing).natural); } else { const bool vertical = node->kind != Kind::row; for (auto id : kids) { const auto wanted = natural(id); - if (vertical) { size.cx = (std::max)(size.cx, wanted.cx); size.cy += wanted.cy; } - else { size.cy = (std::max)(size.cy, wanted.cy); size.cx += wanted.cx; } + if (vertical) { size.cx = (std::max)(size.cx, wanted.cx); size.cy = native_pixel(layout::Pixel(size.cy) + wanted.cy); } + else { size.cy = (std::max)(size.cy, wanted.cy); size.cx = native_pixel(layout::Pixel(size.cx) + wanted.cx); } } if (!kids.empty()) { const auto gaps = spacing * static_cast(kids.size() - 1); - if (vertical) size.cy += gaps; else size.cx += gaps; + if (vertical) size.cy = native_pixel(layout::Pixel(size.cy) + gaps); else size.cx = native_pixel(layout::Pixel(size.cx) + gaps); } } break; } } if (node->kind != Kind::window) { - size.cx += static_cast(property(*node, "margin_start") + property(*node, "margin_end")); - size.cy += static_cast(property(*node, "margin_top") + property(*node, "margin_bottom")); + size.cx = native_pixel(layout::Pixel(size.cx) + property(*node, "margin_start") + property(*node, "margin_end")); + size.cy = native_pixel(layout::Pixel(size.cy) + property(*node, "margin_top") + property(*node, "margin_bottom")); } + layout_natural_cache.emplace(handle, size); return size; } - void place(std::int64_t handle, RECT area) { + void place_one(std::int64_t handle, RECT area, std::vector> &pending) { auto *node = find(handle); if (!node) return; if (node->kind != Kind::window) { - area.left += static_cast(property(*node, "margin_start")); + area.left = native_pixel(layout::Pixel(area.left) + property(*node, "margin_start")); area.right -= static_cast(property(*node, "margin_end")); - area.top += static_cast(property(*node, "margin_top")); + area.top = native_pixel(layout::Pixel(area.top) + property(*node, "margin_top")); area.bottom -= static_cast(property(*node, "margin_bottom")); if (area.right < area.left) area.right = area.left; if (area.bottom < area.top) area.bottom = area.top; @@ -372,83 +391,90 @@ struct Context::Impl : std::enable_shared_from_this { if (inner.right < inner.left) inner.right = inner.left; if (inner.bottom < inner.top) inner.bottom = inner.top; - const auto kids = children_of(handle); + const auto kids = layout_children(handle); if (kids.empty()) return; const auto spacing = static_cast(property(*node, "spacing")); if (node->kind == Kind::grid) { - std::map widths, heights; + std::vector columns, rows; for (auto id : kids) { - auto *child = find(id); - if (!child) continue; + const auto &child = nodes.at(id); const auto wanted = natural(id); - auto &width = widths[child->placement.column]; - auto &height = heights[child->placement.row]; - width = (std::max)(width, static_cast(wanted.cx / (std::max)(std::int64_t(1), child->placement.columns))); - height = (std::max)(height, static_cast(wanted.cy / (std::max)(std::int64_t(1), child->placement.rows))); + columns.push_back({child.placement.column, child.placement.columns, + static_cast(wanted.cx), expands(child, "hexpand")}); + rows.push_back({child.placement.row, child.placement.rows, + static_cast(wanted.cy), expands(child, "vexpand")}); } - std::map column_offset, row_offset; - LONG x = inner.left; - for (auto &[column, width] : widths) { column_offset[column] = x; x += width + spacing; } - LONG y = inner.top; - for (auto &[row, height] : heights) { row_offset[row] = y; y += height + spacing; } + const auto horizontal = layout::arrange(columns, static_cast(spacing), + static_cast(inner.right - inner.left)); + const auto vertical = layout::arrange(rows, static_cast(spacing), + static_cast(inner.bottom - inner.top)); for (auto id : kids) { - auto *child = find(id); - if (!child) continue; - LONG width = 0, height = 0; - for (std::int64_t span = 0; span < child->placement.columns; ++span) { - auto found = widths.find(child->placement.column + span); - if (found != widths.end()) width += found->second + (span ? spacing : 0); - } - for (std::int64_t span = 0; span < child->placement.rows; ++span) { - auto found = heights.find(child->placement.row + span); - if (found != heights.end()) height += found->second + (span ? spacing : 0); - } - const LONG left = column_offset.count(child->placement.column) ? column_offset[child->placement.column] : inner.left; - const LONG top = row_offset.count(child->placement.row) ? row_offset[child->placement.row] : inner.top; - place(id, RECT{left, top, left + width, top + height}); + const auto &child = nodes.at(id); + const auto left = inner.left + horizontal.positions.at(static_cast(child.placement.column)); + const auto top = inner.top + vertical.positions.at(static_cast(child.placement.row)); + const auto width = horizontal.extent(child.placement.column, child.placement.columns); + const auto height = vertical.extent(child.placement.row, child.placement.rows); + pending.emplace_back(id, RECT{native_pixel(left), native_pixel(top), + native_pixel(left + width), native_pixel(top + height)}); } return; } const bool vertical = node->kind != Kind::row; - LONG used = spacing * static_cast(kids.size() - 1); - std::size_t expanding = 0; - std::vector wanted; - wanted.reserve(kids.size()); + std::vector items; for (auto id : kids) { - wanted.push_back(natural(id)); - used += vertical ? wanted.back().cy : wanted.back().cx; - auto *child = find(id); - if (child && property(*child, vertical ? "vexpand" : "hexpand")) ++expanding; + const auto &child = nodes.at(id); + const auto wanted = natural(id); + items.push_back({static_cast(items.size()), 1, + static_cast(vertical ? wanted.cy : wanted.cx), + expands(child, vertical ? "vexpand" : "hexpand")}); } - const LONG span = vertical ? inner.bottom - inner.top : inner.right - inner.left; - const LONG extra = span > used ? span - used : 0; - const LONG share = expanding ? extra / static_cast(expanding) : 0; - - LONG offset = vertical ? inner.top : inner.left; + const auto axis = layout::arrange(items, static_cast(spacing), + static_cast(vertical ? inner.bottom - inner.top : inner.right - inner.left)); for (std::size_t index = 0; index < kids.size(); ++index) { - auto *child = find(kids[index]); - if (!child) continue; - const bool expand = property(*child, vertical ? "vexpand" : "hexpand") != 0; - LONG length = (vertical ? wanted[index].cy : wanted[index].cx) + (expand ? share : 0); - RECT target; - if (vertical) { - const LONG width = property(*child, "hexpand") ? inner.right - inner.left - : (std::min)(wanted[index].cx, inner.right - inner.left); - target = RECT{inner.left, offset, inner.left + width, offset + length}; - } else { - const LONG height = property(*child, "vexpand") ? inner.bottom - inner.top - : (std::min)(wanted[index].cy, inner.bottom - inner.top); - target = RECT{offset, inner.top, offset + length, inner.top + height}; + const auto offset = axis.positions[index], length = axis.sizes[index]; + const RECT target = vertical + ? RECT{inner.left, native_pixel(inner.top + offset), inner.right, native_pixel(inner.top + offset + length)} + : RECT{native_pixel(inner.left + offset), inner.top, native_pixel(inner.left + offset + length), inner.bottom}; + pending.emplace_back(kids[index], target); + } + } + void place(std::int64_t handle, RECT area) { + std::vector> pending{{handle, area}}; + while (!pending.empty()) { + const auto next = pending.back(); pending.pop_back(); + place_one(next.first, next.second, pending); + } + } + void measure_layout(std::int64_t window) { + layout_children_cache.clear(); layout_expansion_cache.clear(); layout_natural_cache.clear(); + for (const auto &[id, node] : nodes) + if (node.window == window && node.parent && layout_visible(node)) layout_children_cache[node.parent].push_back(id); + // Handles increase at creation and a parent must already exist. Reverse + // order therefore measures each descendant before its parent, without + // recursive expansion/measurement or stack growth for deeply nested UI. + for (auto it = nodes.rbegin(); it != nodes.rend(); ++it) { + const auto &[id, node] = *it; + if (node.window != window) continue; + bool horizontal = false, vertical = false; + for (auto child : layout_children(id)) { + const auto &flags = layout_expansion_cache.at(child); + horizontal = horizontal || flags.first; vertical = vertical || flags.second; } - place(kids[index], target); - offset += length + spacing; + const auto flag = [&](const char *name, bool descendants) { + const auto local = node.properties.find(name); + return layout::expands(local == node.properties.end() ? std::nullopt : std::optional(local->second != 0), + layout_visible(node), descendants); + }; + layout_expansion_cache.emplace(id, std::make_pair(flag("hexpand", horizontal), flag("vexpand", vertical))); + (void)natural(id); } } void relayout(std::int64_t window) { auto *node = find(window); if (!node || node->kind != Kind::window) return; + measure_layout(window); RECT client{}; if (!GetClientRect(node->object, &client) || (client.right == 0 && client.bottom == 0)) client = RECT{0, 0, static_cast(property(*node, "width", 1)), diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 10bba5b3..cc6e62ed 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -341,7 +341,9 @@ At `860750e`, both actual Mac native-input probes collect 8,321 regular inputs. Both native incremental steps also pass: ARM's verified report contains 131 closure and 78 real/fake incremental checks, including positive VM/native reuse and invalidation of changed bytes with unchanged timestamps. Intel's successful -step takes roughly 387 seconds; its final raw artifact is still pending. +step takes roughly 387 seconds. Its final 306-file artifact manifest has also +been independently verified, with all six debugger, eight native and two +incremental CTests passing without skips and source unchanged. A subsequent source review closes an additional cache-admission gap: an SDK archive selected for the direct c++/System library roots could add unmodeled @@ -361,6 +363,14 @@ by physical file identity without deduplicating them, preserving the exact one-finding, source, location, dirty-buffer and external-edit checks. Full editor acceptance remains pending. +The Intel editor at the same revision completes 26 groups. Its VM project +check/build/run takes approximately 104 seconds; the subsequent native build +alone takes 144 seconds, leaving too little of the three-minute aggregate +budget for the separate run task. The native project group now has a five-minute +Darwin budget; VM retains three minutes, other checks retain their bounds and +the full host still has a fifteen-minute deadline. Phase timings and exact +successful exit assertions remain required. The original failure is retained. + ## AppKit interaction behavior follow-up A comparison with the actual native controls found gaps beyond the old @@ -386,6 +396,26 @@ also fail to inherit descendant expansion. These affect real layouts beyond font/pixel differences and are being corrected with shared bounded allocation math and native geometry assertions against the same public Context API. +The shared allocator now preserves full spanning requisitions, removes hidden +children and gaps, propagates expansion with explicit overrides and fills the +cross axis. Native coordinate conversion saturates extreme nested grids, and +per-layout child/expansion/measurement caches with iterative placement avoid +recursive depth and repeated-subtree work. Strict GCC, ASan/UBSan and actual +MSVC each pass 21,443 portable checks. The identical public geometry fixture +passes in the real GTK reference (4,603 provider checks) and Nathan's Win32 +desktop (217 provider checks). The Windows report is retained at +`ui-layout-windows-v1/desktop-69b47920e5/evidence/report.json`, SHA-256 +`db3b5a9e9d9737460ea75d50828e265eb5bf557eeb490331ec43cbb60ba6c5ed`. +These are pinned working-tree builds; AppKit and final-SHA SDK checks remain. + +The new test-only AppKit input library drives only its own verified fixture +window through AppKit responders. The unchanged Matrix interaction suite must +complete both cases on each interpreter/native route and emit matching token, +parent/child PID and phase evidence. Seven portable harness controls pass, +including failed/timeout stream retention and malformed-evidence rejection. +The library is excluded from SDK installation. Actual hosted compilation and +input execution are the next gate; no Accessibility or global-input claim is made. + ## SQL provider environment scope The full Mac source/SDK workflows previously exported SQL's diff --git a/tests/tools/test_ui_appkit_display_contract.py b/tests/tools/test_ui_appkit_display_contract.py new file mode 100644 index 00000000..865fcf67 --- /dev/null +++ b/tests/tools/test_ui_appkit_display_contract.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""Portable evidence/launch controls; actual AppKit input needs a Mac desktop.""" +import contextlib +import copy +import importlib.util +import io +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +from types import SimpleNamespace +import unittest +from unittest import mock + +spec = importlib.util.spec_from_file_location('appkit_display_contract', + Path(__file__).with_name('ui_display_contract.py')) +contract = importlib.util.module_from_spec(spec) +spec.loader.exec_module(contract) + + +class AppKitHarnessTests(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory(prefix='nil-appkit-harness-unit-') + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name).resolve() + self.driver = self.root / 'test-only.dylib' + self.driver.write_bytes(b'not a loadable library; mock launch only') + self.token = 'a' * 32 + self.parent = 700 + + def evidence(self, directory=None, token=None, route='vm'): + directory = directory or self.root + token = token or self.token + tests = [] + for offset, (name, stage, keys) in enumerate((('keyboard_events', 3, 8), ('close_request', 1, 0)), 1): + record = {'schema': 'nil.appkit.input.v1', 'token': token, 'route': route, + 'test': name, 'pid': self.parent + offset, 'parentPid': self.parent, + 'status': 'passed', 'reason': 'fixture', 'stage': stage, 'keys': keys, + 'scope': 'owned-process AppKit responder input; no Accessibility or global input'} + (directory / f'{token}-{record["pid"]}.json').write_text(json.dumps(record), encoding='utf-8') + tests.append({'name': name, 'suite': 'nil.std.ui.gtk.interaction', 'route': route, + 'status': 'passed', 'exitCode': 77, 'signal': 0, 'timedOut': False, + 'cancelled': False, 'stdout': 'language assertions passed', + 'stderr': 'NIL_APPKIT_INPUT ' + json.dumps(record) + '\n'}) + return {'schema': 'nil.test.results.v1', 'errors': [], 'tests': tests} + + def case_directory(self): + directory = self.root / 'evidence' + directory.mkdir() + return directory + + def test_exact_language_cases_and_private_input_records_are_required(self): + directory = self.case_directory() + report = self.evidence(directory) + self.assertEqual(len(contract.validate_appkit_interaction(report, 'vm', directory, + self.token, self.parent)), 2) + for mutate in (lambda value: value['tests'].pop(), + lambda value: value['tests'].append(value['tests'][0]), + lambda value: value['tests'][0].update(route='native'), + lambda value: value['tests'][0].update(status='skipped'), + lambda value: value['tests'][0].update(exitCode=0), + lambda value: value['tests'][0].update(timedOut=True), + lambda value: value['tests'][0].update(signal=15), + lambda value: value['tests'][0].update(cancelled=True), + lambda value: value['tests'][0].update(stderr=''), + lambda value: value['errors'].append('compile failed')): + altered = copy.deepcopy(report) + mutate(altered) + with self.assertRaises(AssertionError): + contract.validate_appkit_interaction(altered, 'vm', directory, self.token, self.parent) + + def test_foreign_pid_token_route_or_incomplete_delivery_cannot_qualify(self): + directory = self.case_directory() + report = self.evidence(directory) + for change in ({'parentPid': self.parent + 90}, {'token': 'b' * 32}, {'route': 'native'}, + {'pid': self.parent}, {'pid': -1}, {'pid': True}, {'test': 'close_request'}, + {'keys': 7}, {'stage': 2}, {'status': 'failed'}): + altered = copy.deepcopy(report) + record = json.loads(altered['tests'][0]['stderr'].removeprefix('NIL_APPKIT_INPUT ')) + record.update(change) + altered['tests'][0]['stderr'] = 'NIL_APPKIT_INPUT ' + json.dumps(record) + with self.subTest(change=change), self.assertRaises(AssertionError): + contract.validate_appkit_interaction(altered, 'vm', directory, self.token, self.parent) + # Python bool compares equal to 0/1; typed record fields must reject it + # even when the corresponding language case's expected value is 0/1. + for field in ('stage', 'keys'): + altered = copy.deepcopy(report) + record = json.loads(altered['tests'][1]['stderr'].removeprefix('NIL_APPKIT_INPUT ')) + record[field] = bool(record[field]) + altered['tests'][1]['stderr'] = 'NIL_APPKIT_INPUT ' + json.dumps(record) + with self.subTest(boolean_field=field), self.assertRaises(AssertionError): + contract.validate_appkit_interaction(altered, 'vm', directory, self.token, self.parent) + + def test_extra_duplicate_missing_or_tampered_records_fail(self): + directory = self.case_directory() + report = self.evidence(directory) + duplicate = copy.deepcopy(report) + duplicate['tests'][0]['stderr'] *= 2 + with self.assertRaises(AssertionError): + contract.validate_appkit_interaction(duplicate, 'vm', directory, self.token, self.parent) + extra = directory / 'unexpected.json' + extra.write_text('{}') + with self.assertRaises(AssertionError): + contract.validate_appkit_interaction(report, 'vm', directory, self.token, self.parent) + extra.unlink() + path = next(directory.iterdir()) + original = path.read_text() + path.write_text('{}') + with self.assertRaises(AssertionError): + contract.validate_appkit_interaction(report, 'vm', directory, self.token, self.parent) + path.unlink() + with self.assertRaises(AssertionError): + contract.validate_appkit_interaction(report, 'vm', directory, self.token, self.parent) + path.write_text(original) + + def test_explicit_injection_preserves_environment_and_rejects_ambient_composition(self): + original = {'PATH': '/fixture/bin', 'DYLD_FALLBACK_LIBRARY_PATH': '/fixture/runtime'} + result = contract.appkit_interaction_environment(original, self.driver, self.root, self.token, 'vm') + self.assertEqual(result['PATH'], original['PATH']) + self.assertEqual(result['DYLD_FALLBACK_LIBRARY_PATH'], original['DYLD_FALLBACK_LIBRARY_PATH']) + self.assertNotIn('DYLD_INSERT_LIBRARIES', original) + self.assertEqual(result['DYLD_INSERT_LIBRARIES'], str(self.driver)) + for ambient in (dict(original, DYLD_INSERT_LIBRARIES='/other.dylib'), + dict(original, NIL_TEST_APPKIT_INPUT_TOKEN=self.token)): + with self.assertRaises(AssertionError): + contract.appkit_interaction_environment(ambient, self.driver, self.root, self.token, 'vm') + with self.assertRaises(AssertionError): + contract.appkit_interaction_environment(original, self.root / 'absent', self.root, self.token, 'vm') + + def test_actual_route_orchestration_scopes_injection_and_retains_complete_evidence(self): + source = self.root / 'unchanged.matrix' + source.write_text('owned test fixture') + args = SimpleNamespace(interaction_source=source, appkit_input_driver=self.driver, + appkit_evidence_directory=self.root / 'runs', llvm='ON') + ambient = {'PATH': '/owned/tools'} + seen = [] + def checked(command, environment, timeout, capture_prefix): + self.assertEqual(timeout, 120) + self.assertEqual(command[-1], source) + route = command[command.index('--target') + 1] + self.assertEqual(environment['NIL_TEST_APPKIT_INPUT_ROUTE'], route) + self.assertEqual(environment['DYLD_INSERT_LIBRARIES'], str(self.driver)) + self.assertEqual(capture_prefix.name, route) + seen.append(route) + report = self.evidence(Path(environment['NIL_TEST_APPKIT_INPUT_DIRECTORY']), + environment['NIL_TEST_APPKIT_INPUT_TOKEN'], route) + return SimpleNamespace(stdout=json.dumps(report), stderr='', pid=self.parent) + with mock.patch.object(contract, 'checked', side_effect=checked), contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(contract.run_appkit_interaction(args, ['owned-niltest', 'run'], ambient), 4) + self.assertEqual(seen, ['vm', 'native']) + self.assertEqual(ambient, {'PATH': '/owned/tools'}) + report = json.loads(next(args.appkit_evidence_directory.glob('*/report.json')).read_text()) + self.assertEqual(report['status'], 'passed') + self.assertEqual([route['languageCases'] for route in report['routes']], [2, 2]) + self.assertEqual(report['crossProcessAccessibility'], 'not exercised') + + def test_failed_language_completion_retains_failure_instead_of_accepting_input_only(self): + source = self.root / 'unchanged.matrix' + source.write_text('owned test fixture') + args = SimpleNamespace(interaction_source=source, appkit_input_driver=self.driver, + appkit_evidence_directory=self.root / 'runs', llvm='OFF') + def failed(command, environment, timeout, capture_prefix): + report = self.evidence(Path(environment['NIL_TEST_APPKIT_INPUT_DIRECTORY']), + environment['NIL_TEST_APPKIT_INPUT_TOKEN']) + report['tests'][0]['exitCode'] = 1 + return SimpleNamespace(stdout=json.dumps(report), stderr='', pid=self.parent) + with mock.patch.object(contract, 'checked', side_effect=failed), contextlib.redirect_stdout(io.StringIO()): + with self.assertRaises(AssertionError): + contract.run_appkit_interaction(args, ['owned-niltest', 'run'], {}) + report = json.loads(next(args.appkit_evidence_directory.glob('*/report.json')).read_text()) + self.assertEqual(report['status'], 'failed') + + def test_owned_process_must_exit_naturally_with_success(self): + prefix = self.root / 'raw' + result = contract.checked([sys.executable, '-c', 'print("owned")'], os.environ.copy(), + timeout=5, capture_prefix=prefix) + self.assertEqual(result.stdout.strip(), 'owned') + self.assertEqual(prefix.with_suffix('.stdout').read_text(), 'owned\n') + self.assertGreater(result.pid, 0) + with self.assertRaises(AssertionError): + contract.checked([sys.executable, '-c', + 'import sys; print("failed stdout"); print("failed stderr", file=sys.stderr); raise SystemExit(7)'], + os.environ.copy(), timeout=5, capture_prefix=prefix) + self.assertEqual(prefix.with_suffix('.stdout').read_text(), 'failed stdout\n') + self.assertEqual(prefix.with_suffix('.stderr').read_text(), 'failed stderr\n') + # The production Windows display harness uses Jobs. This POSIX helper + # never owns a timed-out Windows process in the real display workflow. + if os.name != 'nt': + with self.assertRaises(subprocess.TimeoutExpired): + contract.checked([sys.executable, '-c', + 'import sys,time; print("partial stdout", flush=True); print("partial stderr", file=sys.stderr, flush=True); time.sleep(30)'], + os.environ.copy(), timeout=.2, capture_prefix=prefix) + self.assertEqual(prefix.with_suffix('.stdout').read_text(), 'partial stdout\n') + self.assertEqual(prefix.with_suffix('.stderr').read_text(), 'partial stderr\n') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/tools/ui_display_contract.py b/tests/tools/ui_display_contract.py index 9a85dec9..f6a04424 100644 --- a/tests/tools/ui_display_contract.py +++ b/tests/tools/ui_display_contract.py @@ -7,6 +7,7 @@ from contextlib import contextmanager import ctypes as c import ctypes.util +import hashlib import json import os from pathlib import Path @@ -229,20 +230,43 @@ def check_errors(self): assert not unexpected, ('unexpected X11 protocol error', unexpected) -def checked(command, env, timeout=180): +def checked(command, env, timeout=180, capture_prefix=None): process = subprocess.Popen(list(map(str, command)), env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, start_new_session=True) + output, errors = '', '' + def partial(value): + return value.decode('utf-8', errors='replace') if isinstance(value, bytes) else value or '' try: - output, errors = process.communicate(timeout=timeout) + try: + output, errors = process.communicate(timeout=timeout) + except subprocess.TimeoutExpired as expired: + # communicate() can expose bytes even in text mode. Keep the raw + # partial streams before cancellation, then drain the owned child + # once more so its cancellation diagnostics are preserved as well. + output, errors = partial(expired.output), partial(expired.stderr) + terminate(process) + try: + output, errors = process.communicate(timeout=3) + except subprocess.TimeoutExpired as draining: + output = partial(draining.output) or output + errors = partial(draining.stderr) or errors + raise assert process.returncode == 0, (command, process.returncode, output, errors) - return subprocess.CompletedProcess(command, process.returncode, output, errors) + result = subprocess.CompletedProcess(command, process.returncode, output, errors) + result.pid = process.pid + return result finally: - # Give niltest time to forward cancellation to its separately grouped - # compiler/test children before killing the harness session. - terminate(process) - if process.stdout: process.stdout.close() - if process.stderr: process.stderr.close() + try: + if capture_prefix is not None: + Path(str(capture_prefix) + '.stdout').write_text(output, encoding='utf-8') + Path(str(capture_prefix) + '.stderr').write_text(errors, encoding='utf-8') + finally: + # Give niltest time to forward cancellation to its separately grouped + # compiler/test children before killing the harness session. + terminate(process) + if process.stdout: process.stdout.close() + if process.stderr: process.stderr.close() def validate_desktop_report(report, llvm): @@ -259,6 +283,91 @@ def validate_desktop_report(report, llvm): for test in tests), report +def appkit_interaction_environment(environment, driver, directory, token, route): + """Explicit fixture injection; ordinary display/build commands keep their env.""" + assert driver.is_absolute() and driver.is_file(), driver + assert directory.is_absolute() and directory.is_dir(), directory + assert len(token) == 32 and all(byte in '0123456789abcdef' for byte in token), token + assert route in ('vm', 'native'), route + assert not environment.get('DYLD_INSERT_LIBRARIES'), 'AppKit input fixture cannot compose ambient injection' + assert not any(key.startswith('NIL_TEST_APPKIT_INPUT_') for key in environment), 'AppKit fixture declaration already present' + # niltest's compiler/linker children inherit this declaration too. The shim + # is inert until its own process owns the precise interaction window. This + # is deliberately never used as evidence for an uninjected cache profile. + return dict(environment, DYLD_INSERT_LIBRARIES=str(driver), + NIL_TEST_APPKIT_INPUT_TOKEN=token, NIL_TEST_APPKIT_INPUT_DIRECTORY=str(directory), + NIL_TEST_APPKIT_INPUT_ROUTE=route) + + +def validate_appkit_interaction(report, route, directory, token, parent_pid): + expected = {'keyboard_events': (3, 8), 'close_request': (1, 0)} + assert report['schema'] == 'nil.test.results.v1' and not report['errors'], report + tests = report['tests'] + assert len(tests) == len(expected), report + assert {(test['name'], test['suite'], test['route']) for test in tests} == { + (name, 'nil.std.ui.gtk.interaction', route) for name in expected}, report + pids, paths, records = set(), set(), [] + for test in tests: + assert test['status'] == 'passed' and test['exitCode'] == 77 and not ( + test['signal'] or test['timedOut'] or test['cancelled']), test + markers = [line.removeprefix('NIL_APPKIT_INPUT ') for line in test['stderr'].splitlines() + if line.startswith('NIL_APPKIT_INPUT ')] + assert len(markers) == 1 and len(markers[0]) <= 8192, test + record = json.loads(markers[0]) + pid = record['pid'] + assert type(pid) is int and 0 < pid <= 0x7fffffff and pid != parent_pid and pid not in pids, record + assert all(type(record[field]) is int for field in ('parentPid', 'stage', 'keys')), record + assert (record['schema'], record['token'], record['route'], record['test'], record['parentPid'], + record['status'], record['stage'], record['keys']) == ( + 'nil.appkit.input.v1', token, route, test['name'], parent_pid, + 'passed', *expected[test['name']]), record + assert record['scope'] == 'owned-process AppKit responder input; no Accessibility or global input', record + path = directory / f'{token}-{pid}.json' + assert path.is_file() and not path.is_symlink() and path.stat().st_size <= 8192, path + assert json.loads(path.read_text(encoding='utf-8')) == record, (path, record) + pids.add(pid); paths.add(path); records.append(record) + assert set(directory.iterdir()) == paths, ('unexpected/missing input evidence', directory) + return records + + +def run_appkit_interaction(args, base, environment): + assert args.interaction_source and args.appkit_input_driver, 'macOS interaction qualification requires its input fixture' + driver = args.appkit_input_driver.resolve(strict=True) + source = args.interaction_source.resolve(strict=True) + parent = args.appkit_evidence_directory + if parent: + parent.mkdir(parents=True, exist_ok=True) + root = Path(tempfile.mkdtemp(prefix='nil-appkit-input-', dir=parent)).resolve() + summary = {'schema': 'nil.appkit.interaction.v1', 'status': 'running', 'routes': [], + 'inputDriver': str(driver), 'inputDriverSha256': hashlib.sha256(driver.read_bytes()).hexdigest(), + 'source': str(source), 'sourceSha256': hashlib.sha256(source.read_bytes()).hexdigest(), + 'scope': 'in-process AppKit responder input; compiler children inherit inert test injection; not default-cache proof', + 'crossProcessAccessibility': 'not exercised', 'evidenceDirectory': str(root)} + print(f'AppKit interaction evidence: {root}', flush=True) + try: + for route in (['vm', 'native'] if args.llvm == 'ON' else ['vm']): + directory = root / route + directory.mkdir(mode=0o700) + token = secrets.token_hex(16) + env = appkit_interaction_environment(environment, driver, directory, token, route) + command = [*base, '--target', route, '--filter', 'nil.std.ui.gtk.interaction', source] + result = checked(command, env, timeout=120, capture_prefix=root / route) + report = json.loads(result.stdout) + print(json.dumps(report, indent=2), flush=True) + records = validate_appkit_interaction(report, route, directory, token, result.pid) + summary['routes'].append({'route': route, 'parentPid': result.pid, + 'naturalCompletion': True, 'languageCases': len(report['tests']), + 'inputs': records}) + summary['status'] = 'passed' + return sum(route['languageCases'] for route in summary['routes']) + except BaseException as error: + summary.update(status='failed', error=str(error)) + raise + finally: + (root / 'report.json').write_text(json.dumps(summary, indent=2) + '\n', encoding='utf-8') + print(json.dumps(summary, indent=2), flush=True) + + def darwin(args): """The AppKit contract. macOS has no detachable display to create, so the runner's own window server session is used; without one there is nothing @@ -287,14 +396,10 @@ def darwin(args): # missing, duplicated or skipped cases rather than accepting any green subset. print(json.dumps(report, indent=2), flush=True) validate_desktop_report(report, args.llvm) - # The interaction suite is deliberately not run here. It needs keystrokes - # delivered to another process, and on macOS posting events to a different - # process requires accessibility trust that a hosted runner does not grant. - # Real input is covered in-process instead, by the provider fixture's own - # input probe, so this is a limit of cross-process injection rather than of - # the provider. Saying so beats quietly reporting a narrower contract. - print(f'macOS desktop contract passed: {len(report["tests"])} language cases, ' - 'interaction suite not covered (cross-process input needs accessibility trust)') + interaction_cases = run_appkit_interaction(args, base, env) + print(f'macOS desktop contract passed: {len(report["tests"])} desktop + ' + f'{interaction_cases} interaction language cases; in-process AppKit input, ' + 'cross-process Accessibility not exercised') return 0 @@ -310,6 +415,8 @@ def main(): parser.add_argument('--interaction-source', type=Path) parser.add_argument('--provider-test', type=Path) parser.add_argument('--lifecycle-test', type=Path) + parser.add_argument('--appkit-input-driver', type=Path) + parser.add_argument('--appkit-evidence-directory', type=Path) parser.add_argument('--llvm', choices=('ON', 'OFF'), default='OFF') args = parser.parse_args() if sys.platform == 'darwin': diff --git a/tests/ui_appkit_input_driver.mm b/tests/ui_appkit_input_driver.mm new file mode 100644 index 00000000..f2398afb --- /dev/null +++ b/tests/ui_appkit_input_driver.mm @@ -0,0 +1,176 @@ +/* + * Copyright (C) 2008 - 2026 Execute Software. All rights reserved. + * SPDX-License-Identifier: MIT + */ +// Test-only in-process input for the unchanged Matrix interaction suite. +// This library is never installed. It neither requests Accessibility trust nor +// posts global/cross-process input. The provider's poll-driven main run loop +// invokes the timer; actual AppKit responders then generate provider events. +#import +#import + +#include +#include +#include +#include +#include +#include + +@interface NilAppKitInputDriver : NSObject +@property(nonatomic, strong) NSTimer *timer; +@property(nonatomic, copy) NSString *token; +@property(nonatomic, copy) NSString *directory; +@property(nonatomic, copy) NSString *route; +@property(nonatomic, copy) NSString *test; +@property(nonatomic, weak) NSWindow *window; +@property(nonatomic, weak) NSTextField *entry; +@property(nonatomic, weak) NSButton *checkbox; +@property(nonatomic, weak) NSButton *button; +@property(nonatomic) NSTimeInterval deadline; +@property(nonatomic) NSUInteger stage; +@property(nonatomic) NSUInteger keys; +@property(nonatomic) pid_t parent; +@property(nonatomic) BOOL driving; +- (void)tick:(NSTimer *)timer; +@end + +// Keep the callback receiver alive through its final report, even when timer +// invalidation removes the run loop's last reference during that callback. +static NilAppKitInputDriver *active_driver; + +@implementation NilAppKitInputDriver +- (void)finish:(NSString *)status reason:(NSString *)reason { + [self.timer invalidate]; self.timer = nil; + NSDictionary *record = @{@"schema": @"nil.appkit.input.v1", @"token": self.token, + @"route": self.route, @"test": self.test ?: @"unknown", @"pid": @(getpid()), + @"parentPid": @(self.parent), @"status": status, @"reason": reason, + @"keys": @(self.keys), @"stage": @(self.stage), + @"scope": @"owned-process AppKit responder input; no Accessibility or global input"}; + NSData *data = [NSJSONSerialization dataWithJSONObject:record options:NSJSONWritingSortedKeys error:nullptr]; + NSString *path = [self.directory stringByAppendingPathComponent: + [NSString stringWithFormat:@"%@-%d.json", self.token, getpid()]]; + const int fd = open(path.fileSystemRepresentation, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600); + if (!data || fd < 0) { std::fputs("NIL AppKit input evidence creation failed\n", stderr); _exit(85); } + const char *bytes = (const char *)data.bytes; + std::size_t pending = data.length; + while (pending) { + const auto count = write(fd, bytes, pending); + if (count < 0 && errno == EINTR) continue; + if (count <= 0) { close(fd); _exit(85); } + bytes += count; pending -= static_cast(count); + } + if (close(fd)) _exit(85); + std::fputs("NIL_APPKIT_INPUT ", stderr); + std::fwrite(data.bytes, 1, data.length, stderr); + std::fputc('\n', stderr); std::fflush(stderr); + if (![status isEqualToString:@"passed"]) _exit(85); +} +- (void)fail:(NSString *)reason { [self finish:@"failed" reason:reason]; } +- (BOOL)findControls:(NSView *)root { + if ([root isKindOfClass:[NSTextField class]] && ((NSTextField *)root).editable) { + if (self.entry) return NO; + self.entry = (NSTextField *)root; + } else if ([root isKindOfClass:[NSButton class]]) { + NSButton *control = (NSButton *)root; + if ([control.title isEqualToString:@"Enable setting"]) { + if (self.checkbox) return NO; + self.checkbox = control; + } else if ([control.title isEqualToString:@"Finish probe"]) { + if (self.button) return NO; + self.button = control; + } else return NO; + } + for (NSView *child in root.subviews) if (![self findControls:child]) return NO; + return YES; +} +- (void)key:(NSString *)characters code:(unsigned short)code { + NSWindow *window = self.window; + if (!window || !window.visible) { [self fail:@"owned fixture window disappeared before input"]; return; } + const auto event = ^NSEvent *(NSEventType type) { + return [NSEvent keyEventWithType:type location:NSZeroPoint modifierFlags:0 + timestamp:NSProcessInfo.processInfo.systemUptime windowNumber:window.windowNumber + context:nil characters:characters charactersIgnoringModifiers:characters + isARepeat:NO keyCode:code]; + }; + NSEvent *down = event(NSEventTypeKeyDown), *up = event(NSEventTypeKeyUp); + if (!down || !up) { [self fail:@"could not construct owned-window key events"]; return; } + // Queue key-up first so a control's synchronous key tracking can consume it; + // direct key-down targets this verified NSWindow even if the app is inactive. + // No activation policy, system focus, HID tap or other process is modified. + [NSApp postEvent:up atStart:NO]; + self.driving = YES; + [window sendEvent:down]; + self.driving = NO; + ++self.keys; +} +- (void)tick:(NSTimer *)timer { + (void)timer; + @autoreleasepool { + if (self.driving) return; + if (![NSThread isMainThread]) { [self fail:@"input timer left the process main thread"]; return; } + if (!self.test) { + if (!NSApp) return; // Discovery/compiler processes never create fixture windows. + NSWindow *found = nil; + for (NSWindow *candidate in NSApp.windows) { + if (!candidate.visible || !([candidate.title isEqualToString:@"NIL2 UI keyboard probe"] || + [candidate.title isEqualToString:@"NIL2 UI close probe"])) continue; + if (found) { self.test = @"ambiguous"; [self fail:@"more than one owned fixture window"]; return; } + found = candidate; + } + if (!found) return; + self.window = found; + self.deadline = NSProcessInfo.processInfo.systemUptime + 10; + if ([found.title isEqualToString:@"NIL2 UI close probe"]) { + self.test = @"close_request"; + [found performClose:nil]; + if (!found.visible) { [self fail:@"close request destroyed the live language handle"]; return; } + self.stage = 1; + [self finish:@"passed" reason:@"owned window-manager close action delivered"]; + return; + } + self.test = @"keyboard_events"; + if (![self findControls:found.contentView] || !self.entry || !self.checkbox || !self.button || + !self.entry.enabled || !self.checkbox.enabled || !self.button.enabled || + self.entry.currentEditor != found.firstResponder || self.entry.stringValue.length) { + [self fail:@"keyboard fixture controls or initial language-requested focus did not match"]; return; + } + self.stage = 1; + for (NSString *character in @[@"M", @"a", @"t", @"r", @"i", @"x"]) [self key:character code:0]; + return; + } + if (NSProcessInfo.processInfo.systemUptime > self.deadline) { + [self fail:@"owned input phases exceeded ten seconds"]; return; + } + if (self.stage == 1) { + if (![self.entry.stringValue isEqualToString:@"Matrix"]) return; + if (![self.window makeFirstResponder:self.checkbox]) { [self fail:@"checkbox refused in-process focus"]; return; } + self.stage = 2; [self key:@" " code:49]; + } else if (self.stage == 2) { + if (self.checkbox.state != NSControlStateValueOn) return; + if (![self.window makeFirstResponder:self.button]) { [self fail:@"button refused in-process focus"]; return; } + self.stage = 3; [self key:@" " code:49]; + [self finish:@"passed" reason:@"owned entry text and checkbox state observed; button key delivered"]; + } + } +} +@end + +__attribute__((constructor)) static void nil_appkit_input_start() { + const char *token = std::getenv("NIL_TEST_APPKIT_INPUT_TOKEN"); + if (!token) return; + const char *directory = std::getenv("NIL_TEST_APPKIT_INPUT_DIRECTORY"); + const char *route = std::getenv("NIL_TEST_APPKIT_INPUT_ROUTE"); + if (std::strlen(token) != 32 || std::strspn(token, "0123456789abcdef") != 32 || + !directory || *directory != '/' || !route || (std::strcmp(route, "vm") && std::strcmp(route, "native"))) _exit(85); + @autoreleasepool { + NilAppKitInputDriver *driver = [[NilAppKitInputDriver alloc] init]; + active_driver = driver; + driver.token = [NSString stringWithUTF8String:token]; + driver.directory = [NSString stringWithUTF8String:directory]; + driver.route = [NSString stringWithUTF8String:route]; + driver.parent = getppid(); + driver.timer = [NSTimer timerWithTimeInterval:0.01 target:driver selector:@selector(tick:) + userInfo:nil repeats:YES]; + [[NSRunLoop mainRunLoop] addTimer:driver.timer forMode:NSDefaultRunLoopMode]; + } +} diff --git a/tests/ui_layout_provider_contract.hpp b/tests/ui_layout_provider_contract.hpp new file mode 100644 index 00000000..71150070 --- /dev/null +++ b/tests/ui_layout_provider_contract.hpp @@ -0,0 +1,85 @@ +#pragma once +#include "ui_runtime.hpp" +#include +#include +#include + +// The same public-provider assertions run on GTK, Win32 and AppKit. Compare +// allocation changes and equal-control requisitions, never font-specific pixels. +template +void ui_layout_provider_contract(Check check) { + ny::ui::Context context; + const auto ok = [&](const ny::ui::Result &result) { + check(static_cast(result), result.error.c_str()); return result.number; + }; + const auto width = [&](std::int64_t widget) { return ok(context.read_number(widget, "width")); }; + const auto height = [&](std::int64_t widget) { return ok(context.read_number(widget, "height")); }; + const auto await = [&](std::int64_t window, auto condition, const char *message) { + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(3); + bool passed = false; + do { + const auto event = ok(context.poll(window)); + if (event) ok(context.release_event(event)); + if (condition()) { passed = true; break; } + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } while (std::chrono::steady_clock::now() < deadline); + check(passed, message); + }; + + const auto window = ok(context.window("NIL layout visibility and expansion", 720, 240)); + const auto row = ok(context.widget("row", window, "")); + const auto fixed = ok(context.widget("button", row, "Fixed sibling")); + const auto nested = ok(context.widget("column", row, "")); + const auto entry = ok(context.widget("entry", nested, "entry")); + ok(context.set_number(row, "spacing", 7)); + ok(context.set_number(entry, "hexpand", 1)); + ok(context.set_number(window, "visible", 1)); + await(window, [&] { return width(row) > 600 && width(entry) > 300 && width(fixed) > 0; }, + "visible descendants propagate expansion through nested containers"); + const auto before_hidden = width(entry), fixed_width = width(fixed); + ok(context.set_number(fixed, "visible", 0)); + await(window, [&] { return width(entry) >= before_hidden + fixed_width; }, + "hiding a child releases its size and spacing to the visible expanding sibling"); + ok(context.set_number(fixed, "visible", 1)); + await(window, [&] { return width(entry) == before_hidden; }, "showing a child restores the original layout"); + const auto expanded = width(nested); + ok(context.set_number(nested, "hexpand", 0)); + await(window, [&] { return width(nested) < expanded; }, "explicit false overrides descendant expansion"); + ok(context.set_number(nested, "hexpand", 1)); + await(window, [&] { return width(nested) == expanded; }, "explicit true restores expansion"); + ok(context.set_number(window, "width", 900)); + await(window, [&] { return width(entry) > before_hidden + 100; }, "resizing reaches nested expanding controls"); + ok(context.close(window)); + + const auto grid_window = ok(context.window("NIL layout spans and grid growth", 640, 320)); + const auto reference_row = ok(context.widget("row", grid_window, "")); + const std::string title = "A complete spanning control"; + const auto reference = ok(context.widget("button", reference_row, title)); + const auto grid = ok(context.widget("grid", grid_window, "")); + const auto spanning = ok(context.widget("button", grid, title)); + ok(context.grid_attach(grid, spanning, 0, 0, 3, 2)); + ok(context.set_number(grid, "spacing", 3)); + ok(context.set_number(grid_window, "visible", 1)); + await(grid_window, [&] { return width(reference) > 0 && height(reference) > 0 && + width(spanning) >= width(reference) && height(spanning) >= height(reference); }, + "a lone multi-column and multi-row child preserves its full natural requisition"); + const auto fixed_span_width = width(spanning), fixed_span_height = height(spanning); + ok(context.set_number(spanning, "hexpand", 1)); + ok(context.set_number(spanning, "vexpand", 1)); + await(grid_window, [&] { return width(spanning) > fixed_span_width + 100 && height(spanning) > fixed_span_height + 100; }, + "grid tracks distribute available horizontal and vertical space to expanding spans"); + const auto old_grid_width = width(spanning), old_grid_height = height(spanning); + ok(context.set_number(grid_window, "width", 800)); + ok(context.set_number(grid_window, "height", 480)); + await(grid_window, [&] { return width(spanning) > old_grid_width + 100 && height(spanning) > old_grid_height + 100; }, + "native grid allocations continue to expand after resizing"); + ok(context.close(grid_window)); + + const auto fill_window = ok(context.window("NIL layout cross-axis fill", 480, 180)); + const auto label = ok(context.widget("label", fill_window, "fill")); + ok(context.set_number(label, "hexpand", 0)); + ok(context.set_number(fill_window, "visible", 1)); + await(fill_window, [&] { return width(label) == width(fill_window) && width(label) > 400; }, + "default cross-axis fill is independent of requesting extra major-axis space"); + ok(context.close(fill_window)); +} diff --git a/tests/ui_layout_tests.cpp b/tests/ui_layout_tests.cpp new file mode 100644 index 00000000..8e24c0ee --- /dev/null +++ b/tests/ui_layout_tests.cpp @@ -0,0 +1,100 @@ +#include "../nilcompiler/src/ui_layout.hpp" + +#include +#include +#include +#include + +namespace { +unsigned checks = 0; +void check(bool value, const char *message) { + ++checks; + if (!value) { std::cerr << "UI layout: " << message << '\n'; std::exit(1); } +} +using ny::ui::layout::arrange; +using ny::ui::layout::expands; +using ny::ui::layout::Item; + +void invariants(const std::vector &items, std::int64_t spacing, std::int64_t available) { + const auto measured = arrange(items, spacing), placed = arrange(items, spacing, available); + check(placed.natural == measured.natural, "allocation does not alter requisition"); + std::vector occupied(placed.sizes.size()); + bool expandable = false; + for (const auto &item : items) { + if (!item.visible) continue; + expandable = expandable || item.expand; + check(placed.extent(item.start, item.span) >= item.natural, "every spanning control fits its full requisition"); + for (std::int64_t i = 0; i < item.span; ++i) occupied[static_cast(item.start + i)] = true; + } + std::int64_t end = 0; bool previous = false; + for (std::size_t index = 0; index < occupied.size(); ++index) { + if (!occupied[index]) continue; + check(placed.sizes[index] >= 0, "track size is nonnegative"); + check(placed.positions[index] == end + (previous ? spacing : 0), "occupied tracks have exactly one gap and never overlap"); + end = placed.positions[index] + placed.sizes[index]; previous = true; + } + check(end == placed.allocated, "all allocated pixels are accounted for"); + check(placed.allocated == (expandable ? std::max(available, measured.natural) : measured.natural), + "spare pixels go only to expanding tracks with no rounding loss"); +} +} + +int main() { + const auto lone = arrange({{0, 3, 101, false}}, 5); + check(lone.natural == 101 && lone.extent(0, 3) == 101, "lone spanning control is not divided down to its first cell"); + check(lone.sizes.size() == 3 && lone.sizes[0] == 31 && lone.sizes[1] == 30 && lone.sizes[2] == 30, + "span remainder uses deterministic integer pixels"); + const auto empty = arrange({{0, 1, 900, true, false}}, 7, 300); + check(empty.natural == 0 && empty.allocated == 0 && empty.sizes.empty(), "hidden-only container has no dimensions or gaps"); + const auto shown = arrange({{0, 1, 40, false}, {1, 1, 20, true}, {2, 1, 30, false}}, 5, 200); + const auto hidden = arrange({{0, 1, 40, false, false}, {1, 1, 20, true}, {2, 1, 30, false}}, 5, 200); + check(hidden.sizes[1] == shown.sizes[1] + 45 && hidden.positions[1] == 0, + "hiding a sibling releases its full requisition and gap to the expanding control"); + const auto mixed = arrange({{0, 1, 10, false}, {1, 1, 20, true}, {0, 3, 90, true}}, 3, 240); + check(mixed.sizes[0] == 10 && mixed.sizes[2] == 0 && mixed.sizes[1] == 224, + "expanding span uses an existing expanding track without growing fixed tracks"); + const auto last = arrange({{65535, 1, 17, false}}, 4096); + check(last.natural == 17 && last.positions[65535] == 0, "unused coordinate gaps do not reserve empty tracks"); + const auto maximum_span = arrange({{0, 65536, 65537, true}}, 0, 131073); + check(maximum_span.natural == 65537 && maximum_span.extent(0, 65536) == 131073 && maximum_span.sizes[0] == 3, + "full coordinate-range span preserves requisition and expansion remainder"); + + const bool leaf = expands(true, true, false); + check(expands(std::nullopt, true, expands(std::nullopt, true, leaf)), "nested containers inherit descendant expansion by default"); + check(!expands(false, true, leaf), "explicit false overrides descendant expansion"); + check(!expands(std::nullopt, true, expands(true, false, false)), "hidden descendants cannot request expansion"); + check(expands(true, true, false), "explicit true expands an empty visible container"); + check(!expands(true, false, true), "hidden container does not expand"); + + std::int64_t nested = 1; + for (unsigned level = 0; level < 12; ++level) { + const auto grid = arrange({{0, 1, nested}, {1, 65535, 1}}, 4096); + const auto next = ny::ui::layout::native_extent(grid.natural); + check(next >= nested && next <= std::numeric_limits::max(), + "nested sparse grids saturate representable native requisitions without wrapping"); + nested = next; + } + check(nested == std::numeric_limits::max(), "nested maximum spans reach the native coordinate limit safely"); + + // A small independently checked domain covers overlapping axis spans from + // controls on different rows, sparse coordinates, hidden nodes and resizing. + std::mt19937 random(0x4e494cU); + for (unsigned trial = 0; trial < 160; ++trial) { + std::vector items; + for (unsigned i = 0, count = 1 + random() % 20; i < count; ++i) { + const auto start = static_cast(random() % 48); + items.push_back({start, static_cast(1 + random() % (64 - start)), static_cast(random() % 400), + random() % 3 == 0, random() % 4 != 0}); + } + invariants(items, random() % 12, random() % 3000); + } + // The public maxima must not turn each span into a 65,536-step scan. + std::vector maximal; + for (std::int64_t index = 0; index < 4096; ++index) maximal.push_back({0, 65536, 65536 + index, index == 0}); + const auto bounded = arrange(maximal, 1, 500000); + check(bounded.natural == 69631 && bounded.allocated == 500000, + "4096 full-range spanning constraints remain bounded and conserve allocation"); + for (const auto &item : maximal) + check(bounded.extent(item.start, item.span) >= item.natural, "maximum-domain child is never clipped"); + std::cout << "UI layout: " << checks << " checks passed\n"; +} diff --git a/tests/ui_runtime_appkit_tests.mm b/tests/ui_runtime_appkit_tests.mm index 5fa0e31c..70813f77 100644 --- a/tests/ui_runtime_appkit_tests.mm +++ b/tests/ui_runtime_appkit_tests.mm @@ -15,6 +15,7 @@ // main() runs on the process main thread, which is where AppKit requires the // provider to be used. #include "ui_runtime.hpp" +#include "ui_layout_provider_contract.hpp" #include #include @@ -385,11 +386,12 @@ int main(int argc, char **argv) { if (mode == "--display") { require(desktop, "--display requires a window server session"); contract(); + ui_layout_provider_contract(require); interaction_state_contract(); std::cout << "UI runtime: " << checks << " checks passed (--display)\n"; return 0; } - if (desktop) { contract(); interaction_state_contract(); } + if (desktop) { contract(); interaction_state_contract(); ui_layout_provider_contract(require); } else { ny::ui::Context context; rejected(context.window("no session", 320, 240), "window server", diff --git a/tests/ui_runtime_tests.cpp b/tests/ui_runtime_tests.cpp index 8a9e8463..cbd4b7d2 100644 --- a/tests/ui_runtime_tests.cpp +++ b/tests/ui_runtime_tests.cpp @@ -3,6 +3,7 @@ * SPDX-License-Identifier: MIT */ #include "ui_runtime.hpp" +#include "ui_layout_provider_contract.hpp" #include "ui_registry_observer.hpp" #include @@ -468,7 +469,7 @@ int main(int argc, char **argv) { if (mode == "--darwin-mixed-provider") preload_mixed_provider(); #endif basic_validation(); - if (mode == "--display") display_tests(); + if (mode == "--display") { display_tests(); ui_layout_provider_contract(require); } else if (mode == "--no-display") no_display(); else if (mode == "--noninteractive-display") noninteractive_display(); else if (mode == "--darwin-worker-first") { diff --git a/tests/ui_runtime_win32_tests.cpp b/tests/ui_runtime_win32_tests.cpp index 549767fb..bb400c66 100644 --- a/tests/ui_runtime_win32_tests.cpp +++ b/tests/ui_runtime_win32_tests.cpp @@ -14,6 +14,7 @@ // process has an interactive window station and otherwise checks that window // creation is refused, so the same test is meaningful in a service session. #include "ui_runtime.hpp" +#include "ui_layout_provider_contract.hpp" #include #include @@ -315,11 +316,12 @@ int main(int argc, char **argv) { if (mode == "--display") { require(desktop, "--display requires an interactive window station"); contract(); + ui_layout_provider_contract(require); input_probe(false); std::cout << "UI runtime: " << checks << " checks passed (--display)\n"; return 0; } - if (desktop) contract(); + if (desktop) { contract(); ui_layout_provider_contract(require); } else { // A service session has no desktop: creation must fail, and say why. ny::ui::Context context; diff --git a/vscode/test/project-checks.cjs b/vscode/test/project-checks.cjs index 61a1b51b..dbd048fd 100644 --- a/vscode/test/project-checks.cjs +++ b/vscode/test/project-checks.cjs @@ -85,7 +85,9 @@ exports.run = async ({ check, workspace }) => { // Darwin fingerprints the complete dyld backing files in independent // preparation/verification phases. This group executes three real tools; // retain those checks with a bounded aggregate budget and phase timings. - }, process.platform === "darwin" ? 180_000 : 45_000); + // Intel hosted native builds measured 144 seconds before the separate + // run command, which verifies the same dependency closure again. + }, process.platform === "darwin" ? (target === "native" ? 300_000 : 180_000) : 45_000); } await check(`${language}: semantic rules and project warning policy agree with project checks`, async () => { const warning = language === "ny" ? 'import nil.std.text\nfn main() -> i32\n try_char_at("abc", 0)\n return 0\n println("unreachable")\nend fn\n' : From 60d4c99e41d81eb0a3063ff6d210abdda4171d9d Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 19:54:39 -0500 Subject: [PATCH 17/39] Avoid Win32 interface macro in Darwin closure fixture --- tests/darwin_toolchain_inputs_tests.cpp | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/darwin_toolchain_inputs_tests.cpp b/tests/darwin_toolchain_inputs_tests.cpp index db754537..85f5fdfc 100644 --- a/tests/darwin_toolchain_inputs_tests.cpp +++ b/tests/darwin_toolchain_inputs_tests.cpp @@ -225,23 +225,23 @@ void closure(const fs::path &root) { check(collect().has_value(), "unrelated SDK archive cannot introduce autolinks into fixed direct roots"); for (const auto *name : {"c++", "System"}) { const auto direct = plan.sdk / "usr/lib" / (std::string("lib") + name); - const auto archive = fs::path(direct.string() + ".a"), interface = fs::path(direct.string() + ".tbd"); + const auto archive = fs::path(direct.string() + ".a"), text_interface = fs::path(direct.string() + ".tbd"); write(archive, "!\n"); check(collect().has_value(), "same-directory text interface precedes a direct archive fallback"); const auto text = name == std::string("c++") ? stub("/usr/lib/libc++.1.dylib") : stub(); auto incompatible = text; const std::string compatible_targets = "arm64-macos, x86_64-macos"; incompatible.replace(incompatible.find(compatible_targets), compatible_targets.size(), "i386-macos"); - write(interface, incompatible); + write(text_interface, incompatible); check(collect().has_value(), "architecture parsing cannot resume path search into a shadowed archive (no link success claim)"); - fs::remove(interface); + fs::remove(text_interface); std::string archive_reason; check(!detail::collector(driver, project, runtime, plan, inspect, host, &archive_reason).files() && archive_reason.find("selected SDK archive") != std::string::npos, "removing direct stub exposes archive autolinks and disables reuse"); fs::remove(archive); check(!collect(), "missing direct text root cannot certify successful native reuse"); - write(interface, text); + write(text_interface, text); } const auto early_archive = plan.sdk / "usr/local/lib/libc++.a"; write(early_archive, "!\n"); From 5a9840350682ec8934d53b19d0d5edd96884f626 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 20:05:45 -0500 Subject: [PATCH 18/39] Keep AppKit test injection out of compiler processes --- .github/workflows/macos-appkit-provider.yml | 3 +- CMakeLists.txt | 17 +++- docs/platform-support.md | 2 +- notebook/2026-11-11-web-runtime-parity.md | 34 +++++++ .../tools/test_ui_appkit_display_contract.py | 93 ++++++++++++++++++- tests/tools/ui_display_contract.py | 30 ++++-- tests/ui_appkit_compile_driver.cpp | 62 +++++++++++++ 7 files changed, 224 insertions(+), 17 deletions(-) create mode 100644 tests/ui_appkit_compile_driver.cpp diff --git a/.github/workflows/macos-appkit-provider.yml b/.github/workflows/macos-appkit-provider.yml index dd871ac5..c320fef2 100644 --- a/.github/workflows/macos-appkit-provider.yml +++ b/.github/workflows/macos-appkit-provider.yml @@ -130,7 +130,8 @@ jobs: timeout-minutes: 40 run: >- cmake --build build/ci --parallel 2 --target nilcompiler nilinterpreter - niltest nil_ui_runtime_appkit_tests nil_ui_lifecycle_tests nil_ui_appkit_input_driver nil_ui_layout_tests + niltest nil_ui_runtime_appkit_tests nil_ui_lifecycle_tests nil_ui_appkit_input_driver + nil_ui_appkit_compile_driver nil_ui_layout_tests - name: Language-level desktop contract timeout-minutes: 15 run: | diff --git a/CMakeLists.txt b/CMakeLists.txt index 25dd1346..b1c52336 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -371,6 +371,12 @@ if(BUILD_TESTING) target_compile_features(nil_ui_layout_tests PRIVATE cxx_std_17) add_test(NAME nil.ui-layout COMMAND $) set_tests_properties(nil.ui-layout PROPERTIES LABELS "ui;layout;parity" TIMEOUT 30) + if(UNIX) + # Also built on Linux for portable executable-adapter controls. + # This fixture has no install/export rule. + add_executable(nil_ui_appkit_compile_driver tests/ui_appkit_compile_driver.cpp) + target_compile_features(nil_ui_appkit_compile_driver PRIVATE cxx_std_17) + endif() # The provider fixture follows the provider: tests/ui_runtime_tests.cpp # inspects loaded GTK modules and symbol origins, so it describes the GTK @@ -785,8 +791,16 @@ if(BUILD_TESTING) add_test(NAME nil.ui-windows-display-harness COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/test_windows_ui_display_contract.py") set_tests_properties(nil.ui-windows-display-harness PROPERTIES LABELS "ui;tools;windows" TIMEOUT 45) + set(NIL_UI_APPKIT_COMPILE_TEST_ARGS) + if(TARGET nil_ui_appkit_compile_driver) + set(NIL_UI_APPKIT_COMPILE_TEST_ARGS --compile-driver $) + if(TARGET nil_ui_appkit_input_driver) + list(APPEND NIL_UI_APPKIT_COMPILE_TEST_ARGS --input-driver $) + endif() + endif() add_test(NAME nil.ui-appkit-display-harness - COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/test_ui_appkit_display_contract.py") + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/test_ui_appkit_display_contract.py" + ${NIL_UI_APPKIT_COMPILE_TEST_ARGS}) set_tests_properties(nil.ui-appkit-display-harness PROPERTIES LABELS "ui;tools;apple" TIMEOUT 30) set(NIL_UI_DISPLAY_CONTRACT "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/ui_display_contract.py") set(NIL_UI_DISPLAY_TIMEOUT 240) @@ -794,6 +808,7 @@ if(BUILD_TESTING) if(APPLE) set(NIL_UI_DISPLAY_TIMEOUT 420) set(NIL_UI_DISPLAY_APPKIT_ARGS --appkit-input-driver $ + --appkit-compile-driver $ --appkit-evidence-directory "${CMAKE_CURRENT_BINARY_DIR}/appkit-input-evidence") endif() if(WIN32) diff --git a/docs/platform-support.md b/docs/platform-support.md index 3df97b33..fda45ed0 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -9,7 +9,7 @@ release boundary. A newer source checkout does not update an installed SDK. | Host | Verified development baseline | Remaining boundary | | --- | --- | --- | -| Linux x86_64 | Reference host. The clean `5630388` SDK passed installed consumer routes, shared runtime, debugger, Unicode/I/O, checkpoints and GTK checks. The newer `8376252` LLVM-ON source suite passes all 586 CTests. | The reviewed Ubuntu release toolchain is separate from this development SDK. New release binaries still require the release gates. | +| Linux x86_64 | Reference host. The clean `5630388` SDK passed installed consumer routes, shared runtime, debugger, Unicode/I/O, checkpoints and GTK checks. The newer `1efbe66` LLVM-ON source suite passes all 589 CTests without skips. | The reviewed Ubuntu release toolchain is separate from this development SDK. New release binaries still require the release gates. | | Windows x86_64 | The `5630388` installed SDK retains paired Linux acceptance. The newer private `f79b94b` source build passes 579 of 583 registered tests; separately pinned follow-ups resolve its fixture failure and execute the three skipped contracts. Real Win32 desktop and all 88 VS Code host groups pass. | The `8376252` debugger also passes six CTests and four real Win32 desktop sessions. Current-source SDK qualification remains open. Bare-host installation and Windows release qualification remain open. PostgreSQL requires an external runtime; the current UI provider is Win32. | | macOS 15 Intel x86_64 | At `0c88d2f`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Earlier LLVM-ON/OFF, native assembly and embedded-context baselines also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse passes at `860750e`; full editor acceptance, a current-source SDK and release qualification remain open. | | macOS 15 Apple Silicon arm64 | At `3414ce5`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Native assembly and embedded context also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse passes at `860750e`; full editor acceptance, a current-source SDK and release qualification remain open. | diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index cc6e62ed..30ebb2a8 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -416,6 +416,40 @@ including failed/timeout stream retention and malformed-evidence rejection. The library is excluded from SDK installation. Actual hosted compilation and input execution are the next gate; no Accessibility or global-input claim is made. +At `1efbe66`, Linux's complete LLVM-ON source suite passes 589/589 CTests +without skips in 377.54 seconds (run `43129952409`); LLVM-OFF and extension +jobs also pass. The four-host web-runtime campaign (`43129952445`) again +passes the exact six selected contracts per host, with all 148 retained files +independently hash-verified. Local Linux's full desktop contract also passes +all ten VM/native language cases after supplying the existing private Xvfb +prerequisite and building the lifecycle fixture; earlier prerequisite failures +and skips are retained separately. + +Hosted AppKit run `43129956297` passes all seven selected CTests on Intel, +including all ten desktop/interaction language cases. ARM passes six of seven: +layout, provider and all desktop cases pass, and both VM interaction cases pass. +Native interaction compilation fails because the test-only arm64 input library +is inherited by Apple's arm64e compiler launcher. This is an input-harness +scope error; a follow-up must keep application injection out of compiler +children before ARM's language interaction gate can pass. Retained route +reports and per-process input records have been independently replayed. + +The correction uses a same-architecture test compiler adapter: it validates +the explicit compiler/input declaration, removes only fixture-owned injection +and variables, then replaces itself with the real compiler. Ordinary arguments, +environment, PID and exit status are preserved; niltest and application children +retain the input driver. Ten controls pass, including actual subprocess checks +and rejection of mismatched injection or compiler targets. The adapter has no +SDK install rule. Hosted ARM acceptance remains required. + +The first fresh Windows SDK build at `1efbe66` exposes a separate fixture +portability error: Win32's COM headers define `interface`, which collides with +a local variable in the Darwin closure test. Commit `60d4c99` changes only that +identifier. Actual MSVC then passes 137 closure checks; Linux passes 143 +(six additional POSIX symlink controls). The failed build remains recorded, +and the clean Windows SDK retry uses that exact new commit. No runtime code +or accepted installed SDK changed in this correction. + ## SQL provider environment scope The full Mac source/SDK workflows previously exported SQL's diff --git a/tests/tools/test_ui_appkit_display_contract.py b/tests/tools/test_ui_appkit_display_contract.py index 865fcf67..4d5e3dbf 100644 --- a/tests/tools/test_ui_appkit_display_contract.py +++ b/tests/tools/test_ui_appkit_display_contract.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 """Portable evidence/launch controls; actual AppKit input needs a Mac desktop.""" import contextlib +import argparse import copy import importlib.util import io @@ -19,6 +20,12 @@ contract = importlib.util.module_from_spec(spec) spec.loader.exec_module(contract) +options = argparse.ArgumentParser(add_help=False) +options.add_argument('--compile-driver', type=Path) +options.add_argument('--input-driver', type=Path) +fixture_options, remaining = options.parse_known_args() +sys.argv = [sys.argv[0], *remaining] + class AppKitHarnessTests(unittest.TestCase): def setUp(self): @@ -117,7 +124,7 @@ def test_extra_duplicate_missing_or_tampered_records_fail(self): def test_explicit_injection_preserves_environment_and_rejects_ambient_composition(self): original = {'PATH': '/fixture/bin', 'DYLD_FALLBACK_LIBRARY_PATH': '/fixture/runtime'} - result = contract.appkit_interaction_environment(original, self.driver, self.root, self.token, 'vm') + result = contract.appkit_interaction_environment(original, self.driver, self.root, self.token, 'vm', self.driver) self.assertEqual(result['PATH'], original['PATH']) self.assertEqual(result['DYLD_FALLBACK_LIBRARY_PATH'], original['DYLD_FALLBACK_LIBRARY_PATH']) self.assertNotIn('DYLD_INSERT_LIBRARIES', original) @@ -125,14 +132,15 @@ def test_explicit_injection_preserves_environment_and_rejects_ambient_compositio for ambient in (dict(original, DYLD_INSERT_LIBRARIES='/other.dylib'), dict(original, NIL_TEST_APPKIT_INPUT_TOKEN=self.token)): with self.assertRaises(AssertionError): - contract.appkit_interaction_environment(ambient, self.driver, self.root, self.token, 'vm') + contract.appkit_interaction_environment(ambient, self.driver, self.root, self.token, 'vm', self.driver) with self.assertRaises(AssertionError): - contract.appkit_interaction_environment(original, self.root / 'absent', self.root, self.token, 'vm') + contract.appkit_interaction_environment(original, self.root / 'absent', self.root, self.token, 'vm', self.driver) def test_actual_route_orchestration_scopes_injection_and_retains_complete_evidence(self): source = self.root / 'unchanged.matrix' source.write_text('owned test fixture') args = SimpleNamespace(interaction_source=source, appkit_input_driver=self.driver, + appkit_compile_driver=self.driver, compiler=Path(sys.executable).resolve(), appkit_evidence_directory=self.root / 'runs', llvm='ON') ambient = {'PATH': '/owned/tools'} seen = [] @@ -142,13 +150,15 @@ def checked(command, environment, timeout, capture_prefix): route = command[command.index('--target') + 1] self.assertEqual(environment['NIL_TEST_APPKIT_INPUT_ROUTE'], route) self.assertEqual(environment['DYLD_INSERT_LIBRARIES'], str(self.driver)) + self.assertEqual(environment['NIL_TEST_APPKIT_INPUT_COMPILER'], str(args.compiler)) + self.assertEqual(command[command.index('--compiler') + 1], args.appkit_compile_driver) self.assertEqual(capture_prefix.name, route) seen.append(route) report = self.evidence(Path(environment['NIL_TEST_APPKIT_INPUT_DIRECTORY']), environment['NIL_TEST_APPKIT_INPUT_TOKEN'], route) return SimpleNamespace(stdout=json.dumps(report), stderr='', pid=self.parent) with mock.patch.object(contract, 'checked', side_effect=checked), contextlib.redirect_stdout(io.StringIO()): - self.assertEqual(contract.run_appkit_interaction(args, ['owned-niltest', 'run'], ambient), 4) + self.assertEqual(contract.run_appkit_interaction(args, ['owned-niltest', 'run', '--compiler', args.compiler], ambient), 4) self.assertEqual(seen, ['vm', 'native']) self.assertEqual(ambient, {'PATH': '/owned/tools'}) report = json.loads(next(args.appkit_evidence_directory.glob('*/report.json')).read_text()) @@ -160,6 +170,7 @@ def test_failed_language_completion_retains_failure_instead_of_accepting_input_o source = self.root / 'unchanged.matrix' source.write_text('owned test fixture') args = SimpleNamespace(interaction_source=source, appkit_input_driver=self.driver, + appkit_compile_driver=self.driver, compiler=Path(sys.executable).resolve(), appkit_evidence_directory=self.root / 'runs', llvm='OFF') def failed(command, environment, timeout, capture_prefix): report = self.evidence(Path(environment['NIL_TEST_APPKIT_INPUT_DIRECTORY']), @@ -168,7 +179,7 @@ def failed(command, environment, timeout, capture_prefix): return SimpleNamespace(stdout=json.dumps(report), stderr='', pid=self.parent) with mock.patch.object(contract, 'checked', side_effect=failed), contextlib.redirect_stdout(io.StringIO()): with self.assertRaises(AssertionError): - contract.run_appkit_interaction(args, ['owned-niltest', 'run'], {}) + contract.run_appkit_interaction(args, ['owned-niltest', 'run', '--compiler', args.compiler], {}) report = json.loads(next(args.appkit_evidence_directory.glob('*/report.json')).read_text()) self.assertEqual(report['status'], 'failed') @@ -196,5 +207,77 @@ def test_owned_process_must_exit_naturally_with_success(self): self.assertEqual(prefix.with_suffix('.stderr').read_text(), 'partial stderr\n') +if fixture_options.compile_driver: + class CompiledAdapterTests(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory(prefix='nil-appkit-adapter-unit-') + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name).resolve() + self.wrapper = fixture_options.compile_driver.resolve(strict=True) + if sys.platform == 'darwin': + self.assertIsNotNone(fixture_options.input_driver, 'Darwin adapter controls need the real compatible input dylib') + self.driver = fixture_options.input_driver.resolve(strict=True) + else: + self.driver = self.root / 'input.dylib' + self.driver.write_bytes(b'non-Darwin environment control; never loaded') + # No inherited override is removed silently. The control requires a + # clean caller and then supplies its own explicit fixture profile. + self.assertFalse(os.environ.get('DYLD_INSERT_LIBRARIES')) + self.env = contract.appkit_interaction_environment(dict(os.environ), self.driver, self.root, + 'b' * 32, 'native', Path(sys.executable).resolve(strict=True)) + self.env['NIL_TEST_PRESERVED_SETTING'] = 'ordinary value with spaces $ and quotes' + self.env['DYLD_FALLBACK_LIBRARY_PATH'] = str(self.root) + + def test_real_adapter_cleans_compile_environment_and_preserves_arguments(self): + program = ('import json,os,sys; print(json.dumps({' + '"env":{key:os.environ.get(key) for key in ("PATH","NIL_TEST_PRESERVED_SETTING","DYLD_FALLBACK_LIBRARY_PATH","DYLD_INSERT_LIBRARIES")},' + '"fixtureNames":[key for key in os.environ if key.startswith("NIL_TEST_APPKIT_INPUT_")],' + '"args":sys.argv[1:],"pid":os.getpid(),"parentPid":os.getppid()}))') + process = subprocess.Popen([str(self.wrapper), '-c', program, 'argument with spaces', '$literal'], + env=self.env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) + try: + output, errors = process.communicate(timeout=10) + finally: + if process.poll() is None: + process.kill(); process.communicate(timeout=3) + self.assertEqual(process.returncode, 0, errors) + observed = json.loads(output) + self.assertEqual(observed['pid'], process.pid, 'exec must preserve the owned compiler PID') + self.assertEqual(observed['parentPid'], os.getpid(), 'exec must preserve the parent relationship') + self.assertEqual(observed['args'], ['argument with spaces', '$literal']) + self.assertIsNone(observed['env']['DYLD_INSERT_LIBRARIES']) + self.assertEqual(observed['fixtureNames'], []) + self.assertEqual(observed['env']['NIL_TEST_PRESERVED_SETTING'], self.env['NIL_TEST_PRESERVED_SETTING']) + self.assertEqual(observed['env']['PATH'], self.env['PATH']) + self.assertEqual(observed['env'].get('DYLD_FALLBACK_LIBRARY_PATH'), self.env['DYLD_FALLBACK_LIBRARY_PATH']) + # The parent environment still drives niltest's application route. + self.assertEqual(self.env['DYLD_INSERT_LIBRARIES'], str(self.driver)) + + def test_real_adapter_propagates_compiler_status(self): + result = subprocess.run([str(self.wrapper), '-c', 'raise SystemExit(7)'], env=self.env, + capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 7, result.stderr) + + def test_real_adapter_rejects_mismatched_missing_and_recursive_profiles(self): + other = self.root / 'other-driver.dylib' + other.write_bytes(b'declared only; never loaded') + nonexecutable = self.root / 'nonexecutable-compiler' + nonexecutable.write_bytes(b'not executable') + for change in ({'NIL_TEST_APPKIT_INPUT_DRIVER': str(other)}, + {'DYLD_INSERT_LIBRARIES': ''}, + {'NIL_TEST_APPKIT_INPUT_DRIVER': ''}, + {'NIL_TEST_APPKIT_INPUT_COMPILER': str(self.root / 'absent')}, + {'NIL_TEST_APPKIT_INPUT_COMPILER': 'relative/compiler'}, + {'NIL_TEST_APPKIT_INPUT_COMPILER': str(self.wrapper)}, + {'NIL_TEST_APPKIT_INPUT_COMPILER': str(nonexecutable)}, + {'NIL_TEST_APPKIT_INPUT_DIRECTORY': str(self.root / 'absent')}): + with self.subTest(change=change): + result = subprocess.run([str(self.wrapper), '-c', 'print("UNEXPECTED COMPILER EXECUTION")'], + env=dict(self.env, **change), capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 85, result.stderr) + self.assertNotIn('UNEXPECTED COMPILER EXECUTION', result.stdout) + self.assertIn('AppKit test compiler adapter:', result.stderr) + + if __name__ == '__main__': unittest.main() diff --git a/tests/tools/ui_display_contract.py b/tests/tools/ui_display_contract.py index f6a04424..e959c206 100644 --- a/tests/tools/ui_display_contract.py +++ b/tests/tools/ui_display_contract.py @@ -283,20 +283,22 @@ def validate_desktop_report(report, llvm): for test in tests), report -def appkit_interaction_environment(environment, driver, directory, token, route): +def appkit_interaction_environment(environment, driver, directory, token, route, compiler): """Explicit fixture injection; ordinary display/build commands keep their env.""" assert driver.is_absolute() and driver.is_file(), driver + assert compiler.is_absolute() and compiler.is_file(), compiler assert directory.is_absolute() and directory.is_dir(), directory assert len(token) == 32 and all(byte in '0123456789abcdef' for byte in token), token assert route in ('vm', 'native'), route assert not environment.get('DYLD_INSERT_LIBRARIES'), 'AppKit input fixture cannot compose ambient injection' assert not any(key.startswith('NIL_TEST_APPKIT_INPUT_') for key in environment), 'AppKit fixture declaration already present' - # niltest's compiler/linker children inherit this declaration too. The shim - # is inert until its own process owns the precise interaction window. This - # is deliberately never used as evidence for an uninjected cache profile. + # The same-architecture test compiler adapter validates this declaration, + # then removes it before exec'ing the real compiler. niltest and its actual + # application children keep the input driver; system tools never load it. return dict(environment, DYLD_INSERT_LIBRARIES=str(driver), NIL_TEST_APPKIT_INPUT_TOKEN=token, NIL_TEST_APPKIT_INPUT_DIRECTORY=str(directory), - NIL_TEST_APPKIT_INPUT_ROUTE=route) + NIL_TEST_APPKIT_INPUT_ROUTE=route, NIL_TEST_APPKIT_INPUT_DRIVER=str(driver), + NIL_TEST_APPKIT_INPUT_COMPILER=str(compiler)) def validate_appkit_interaction(report, route, directory, token, parent_pid): @@ -331,8 +333,15 @@ def validate_appkit_interaction(report, route, directory, token, parent_pid): def run_appkit_interaction(args, base, environment): - assert args.interaction_source and args.appkit_input_driver, 'macOS interaction qualification requires its input fixture' + assert args.interaction_source and args.appkit_input_driver and args.appkit_compile_driver, 'macOS interaction qualification requires its input and compiler fixtures' driver = args.appkit_input_driver.resolve(strict=True) + compile_driver = args.appkit_compile_driver.resolve(strict=True) + compiler = args.compiler.resolve(strict=True) + assert compile_driver.is_file() and compiler.is_file(), (compile_driver, compiler) + command_base = list(base) + compiler_slot = command_base.index('--compiler') + 1 + assert Path(command_base[compiler_slot]).resolve() == compiler, 'niltest real compiler must match the explicit fixture declaration' + command_base[compiler_slot] = compile_driver source = args.interaction_source.resolve(strict=True) parent = args.appkit_evidence_directory if parent: @@ -340,8 +349,10 @@ def run_appkit_interaction(args, base, environment): root = Path(tempfile.mkdtemp(prefix='nil-appkit-input-', dir=parent)).resolve() summary = {'schema': 'nil.appkit.interaction.v1', 'status': 'running', 'routes': [], 'inputDriver': str(driver), 'inputDriverSha256': hashlib.sha256(driver.read_bytes()).hexdigest(), + 'compileDriver': str(compile_driver), 'compileDriverSha256': hashlib.sha256(compile_driver.read_bytes()).hexdigest(), + 'compiler': str(compiler), 'source': str(source), 'sourceSha256': hashlib.sha256(source.read_bytes()).hexdigest(), - 'scope': 'in-process AppKit responder input; compiler children inherit inert test injection; not default-cache proof', + 'scope': 'in-process AppKit responder input; test compiler adapter removes fixture injection before compilation; not default-cache proof', 'crossProcessAccessibility': 'not exercised', 'evidenceDirectory': str(root)} print(f'AppKit interaction evidence: {root}', flush=True) try: @@ -349,8 +360,8 @@ def run_appkit_interaction(args, base, environment): directory = root / route directory.mkdir(mode=0o700) token = secrets.token_hex(16) - env = appkit_interaction_environment(environment, driver, directory, token, route) - command = [*base, '--target', route, '--filter', 'nil.std.ui.gtk.interaction', source] + env = appkit_interaction_environment(environment, driver, directory, token, route, compiler) + command = [*command_base, '--target', route, '--filter', 'nil.std.ui.gtk.interaction', source] result = checked(command, env, timeout=120, capture_prefix=root / route) report = json.loads(result.stdout) print(json.dumps(report, indent=2), flush=True) @@ -416,6 +427,7 @@ def main(): parser.add_argument('--provider-test', type=Path) parser.add_argument('--lifecycle-test', type=Path) parser.add_argument('--appkit-input-driver', type=Path) + parser.add_argument('--appkit-compile-driver', type=Path) parser.add_argument('--appkit-evidence-directory', type=Path) parser.add_argument('--llvm', choices=('ON', 'OFF'), default='OFF') args = parser.parse_args() diff --git a/tests/ui_appkit_compile_driver.cpp b/tests/ui_appkit_compile_driver.cpp new file mode 100644 index 00000000..fbd66bba --- /dev/null +++ b/tests/ui_appkit_compile_driver.cpp @@ -0,0 +1,62 @@ +/* + * Copyright (C) 2008 - 2026 Execute Software. All rights reserved. + * SPDX-License-Identifier: MIT + */ +// Test-only niltest compiler adapter. Native application execution still uses +// the injected AppKit input driver; discovery/compilation and system linker +// children must not inherit it (Apple's launcher can have an arm64e slice). +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +namespace { +int fail(const char *message) { + std::fprintf(stderr, "AppKit test compiler adapter: %s\n", message); + return 85; +} +const char *setting(const char *name) { + const char *value = std::getenv(name); + return value && *value && std::strlen(value) <= 32768 ? value : nullptr; +} +bool regular_absolute(const char *value) { + if (!value || !fs::path(value).is_absolute()) return false; + std::error_code error; + return fs::is_regular_file(value, error) && !error; +} +} + +int main(int argc, char **argv) { + const char *compiler = setting("NIL_TEST_APPKIT_INPUT_COMPILER"); + const char *driver = setting("NIL_TEST_APPKIT_INPUT_DRIVER"); + const char *injection = setting("DYLD_INSERT_LIBRARIES"); + const char *token = setting("NIL_TEST_APPKIT_INPUT_TOKEN"); + const char *directory = setting("NIL_TEST_APPKIT_INPUT_DIRECTORY"); + const char *route = setting("NIL_TEST_APPKIT_INPUT_ROUTE"); + if (!regular_absolute(compiler) || !regular_absolute(driver)) return fail("declared compiler and input driver must be absolute regular files"); + if (!injection || std::strcmp(injection, driver)) return fail("injection does not match the declared fixture driver"); + if (!token || std::strlen(token) != 32 || std::strspn(token, "0123456789abcdef") != 32 || + !route || (std::strcmp(route, "vm") && std::strcmp(route, "native"))) return fail("invalid fixture token or route"); + std::error_code error; + if (!directory || !fs::path(directory).is_absolute() || !fs::is_directory(directory, error) || error) + return fail("fixture evidence directory is not an absolute directory"); + if (fs::equivalent(compiler, argv[0], error) || error) return fail("recursive or unresolved compiler adapter target"); + // Keep a private copy before removing the declaration that owns getenv's + // storage. All ordinary environment settings and arguments remain intact. + std::string selected(compiler); + std::vector arguments; + arguments.reserve(static_cast(argc) + 1); + arguments.push_back(selected.data()); + for (int index = 1; index < argc; ++index) arguments.push_back(argv[index]); + arguments.push_back(nullptr); + for (const char *name : {"DYLD_INSERT_LIBRARIES", "NIL_TEST_APPKIT_INPUT_COMPILER", + "NIL_TEST_APPKIT_INPUT_DRIVER", "NIL_TEST_APPKIT_INPUT_TOKEN", + "NIL_TEST_APPKIT_INPUT_DIRECTORY", "NIL_TEST_APPKIT_INPUT_ROUTE"}) + if (unsetenv(name)) return fail("could not remove fixture-owned environment declaration"); + execv(selected.c_str(), arguments.data()); + return fail("could not execute the declared compiler"); +} From 11370affc68bddb14d25efd3450c187894d29d58 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 20:16:28 -0500 Subject: [PATCH 19/39] Match editor fixture paths to captured source authority --- docs/platform-support.md | 15 +++++---- docs/standard-library/ui.md | 6 ++-- notebook/2026-11-11-web-runtime-parity.md | 21 +++++++++++- vscode/test/semantic-checks.cjs | 39 ++++++++++++++++++++--- vscode/test/workspace-checks.cjs | 6 ++-- 5 files changed, 70 insertions(+), 17 deletions(-) diff --git a/docs/platform-support.md b/docs/platform-support.md index fda45ed0..2dc6e517 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -183,15 +183,18 @@ the complete admission rules. provider passes 156 display and 41 in-process keyboard checks per host, and `nil.ui-display` now passes there too: the provider fixture, the lifecycle fixture and the Matrix desktop suite. Real keyboard input is - covered in process by the AppKit provider fixture. The current AppKit - campaign does not qualify cross-process keyboard injection or accessibility; - the language interaction suite currently runs on Linux and Windows. Earlier + covered in process by the AppKit provider fixture. At `5a98403`, both Macs + pass all seven selected UI CTests and all ten desktop/interaction Matrix + cases across interpreter/native routes. The test-only input driver uses + each owning application's responders; its compiler adapter keeps injection + out of host compilation. This does not qualify cross-process keyboard + injection or accessibility. Earlier private GTK and AppKit prerequisite probes are retained as historical evidence, but GTK is no longer the macOS product provider. The LLVM-ON campaign at `965f159` passes all five selected CTests on both hosts, including all three - Matrix desktop cases on each interpreter/native route. The retained full - language reports verify all six cases; this is separate from editor acceptance - and the cross-process keyboard/accessibility exclusion. + Matrix desktop cases on each interpreter/native route. That earlier six-case + baseline is superseded by the ten-case campaign above; editor acceptance + remains separate. - Editor desktop integration, untested provider configurations and release-package acceptance remain separate from the passing installed campaigns. Shipping a Mach-O tool alone does not qualify these surfaces. diff --git a/docs/standard-library/ui.md b/docs/standard-library/ui.md index ee0b1c0f..5561994f 100644 --- a/docs/standard-library/ui.md +++ b/docs/standard-library/ui.md @@ -216,8 +216,10 @@ for filtering, reports and timeouts. The macOS harness uses a test-only library inside each owning process to send input through its verified AppKit responder chain. It exercises the same Matrix interaction source and records process-bound evidence without Accessibility -permissions or global input. This library is not installed in the SDK. Hosted -qualification of this new language-level input path is still pending. +permissions or global input. A test compiler adapter removes that injection +before invoking the real compiler and host linker. Neither fixture is installed +in the SDK. At `5a98403`, both hosted Mac architectures pass all ten desktop and +interaction cases across the interpreter and native routes. The same display harness checks real GTK window counts during VM pauses and after completion, faults, resets, reloads and destruction. Native context diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 30ebb2a8..49c5dc28 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -440,7 +440,10 @@ and variables, then replaces itself with the real compiler. Ordinary arguments, environment, PID and exit status are preserved; niltest and application children retain the input driver. Ten controls pass, including actual subprocess checks and rejection of mismatched injection or compiler targets. The adapter has no -SDK install rule. Hosted ARM acceptance remains required. +SDK install rule. At `5a98403`, hosted run `43132072573` passes all seven UI +CTests on both architectures, with no case skips. Both retain all six desktop +and four interaction language cases. Root independently replayed each route's +niltest report against the token/PID input files and unchanged Matrix source. The first fresh Windows SDK build at `1efbe66` exposes a separate fixture portability error: Win32's COM headers define `interface`, which collides with @@ -450,6 +453,22 @@ identifier. Actual MSVC then passes 137 closure checks; Linux passes 143 and the clean Windows SDK retry uses that exact new commit. No runtime code or accepted installed SDK changed in this correction. +Both `1efbe66` Mac editor runs complete 37 groups, including all project and +real imported-file checks, before a fake compiler emits an alias path rather +than its captured canonical source identity. A real Linux editor under aliased +TMPDIR reproduces that failure. The fixture now returns the canonical captured +snapshot without changing its text. The full alias run then exposes the same +assumption in semantic navigation expectations for clean, uncaptured disk +sources. Those expectations now accept physical identity only for clean saved +targets without tabs; dirty, untitled and open-tab targets still require exact +URIs. Reference cardinality and duplicate checks remain strict, and new checks +verify that even clean open imports retain their editor URI. + +The corrected real Linux host passes all 88 groups, all nine modules and no +exclusions in 44.7 seconds using VS Code 1.144.1. Analyzer bytes are unchanged. +The complete failing/corrected chain is retained under +`editor-workspace-alias-linux-q8g3r09h`; hosted Mac confirmation remains open. + ## SQL provider environment scope The full Mac source/SDK workflows previously exported SQL's diff --git a/vscode/test/semantic-checks.cjs b/vscode/test/semantic-checks.cjs index a886e552..a858519b 100644 --- a/vscode/test/semantic-checks.cjs +++ b/vscode/test/semantic-checks.cjs @@ -2,7 +2,26 @@ const assert = require("node:assert/strict"); const fs = require("node:fs/promises"); const path = require("node:path"); const vscode = require("vscode"); -const { createFileAlias } = require("./compiler-fixture-support.cjs"); +const { createFileAlias, sameExistingFile } = require("./compiler-fixture-support.cjs"); + +function hasTab(document) { + const expected = document.uri.toString(); + return vscode.window.tabGroups.all.some(group => group.tabs.some(tab => { + if (tab.input instanceof vscode.TabInputText) return tab.input.uri.toString() === expected; + if (tab.input instanceof vscode.TabInputTextDiff) { + return tab.input.original.toString() === expected || tab.input.modified.toString() === expected; + } + return false; + })); +} +function targetUriMatches(uri, document) { + if (uri.toString() === document.uri.toString()) return true; + // Captured editor inputs must retain their authoritative URI, even when an + // open tab is clean. A clean cached model with no tab is only a disk source; + // the real compiler returns its canonical physical path (/private/tmp on Mac). + return uri.scheme === "file" && document.uri.scheme === "file" && + !document.isDirty && !hasTab(document) && sameExistingFile(uri.fsPath, document.uri.fsPath); +} const pause = ms => new Promise(resolve => setTimeout(resolve, ms)); async function eventually(operation, description) { @@ -53,7 +72,7 @@ async function expectDefinition(document, useContext, useToken, target, declarat const [result] = results; assert.ok(result.originSelectionRange && result.targetSelectionRange && result.targetRange, "Semantic navigation must return a LocationLink with exact original-source selections."); - assert.equal(result.targetUri.toString(), target.uri.toString()); + assert.ok(targetUriMatches(result.targetUri, target), "Definition must retain an editor input URI or name the exact uncaptured disk file."); assert.ok(result.originSelectionRange.isEqual(origin), "Origin must select the actual source spelling."); assert.ok(result.targetSelectionRange.isEqual(selection), "Target must select the resolved declaration token."); assert.ok(result.targetRange.isEqual(selection), "Target range must not widen to an unrelated statement."); @@ -89,9 +108,14 @@ async function expectReferences(document, context, token, occurrences) { }, `references of ${context}`); // A nonempty wrong result must fail immediately, including duplicate // lexical LSP guesses or a partial set presented as complete references. - assert.deepEqual(results.map(item => locationKey(item.uri, item.range)).sort(), - occurrences.map(([source, sourceContext, sourceToken = token]) => - locationKey(source.uri, rangeIn(source, sourceContext, sourceToken))).sort()); + const remaining = occurrences.map(([source, sourceContext, sourceToken = token]) => + ({ source, range: rangeIn(source, sourceContext, sourceToken) })); + assert.equal(results.length, remaining.length, "References must include every certified occurrence exactly once."); + for (const result of results) { + const index = remaining.findIndex(expected => targetUriMatches(result.uri, expected.source) && result.range.isEqual(expected.range)); + assert.notEqual(index, -1, `Unexpected or duplicate reference: ${locationKey(result.uri, result.range)}`); + remaining.splice(index, 1); + } return results; } async function rename(document, context, token, newName) { @@ -191,6 +215,11 @@ exports.run = async ({ check, workspace }) => { const header = dependencyLanguage === "ny" ? `fn ${declared}(` : `public ${declared} receives`; await analyze(entry); await expectDefinition(entry, use, use, dependency, header, declared); + await vscode.window.showTextDocument(dependency, { preview: false }); + await analyze(entry); + const cleanOpen = await expectDefinition(entry, use, use, dependency, header, declared); + assert.equal(dependency.isDirty, false); + assert.equal(cleanOpen.targetUri.toString(), dependency.uri.toString(), "Even a clean open import must retain its editor URI."); const comment = dependencyLanguage === "ny" ? "# é😀 dirty declaration\r\n\r\n" : "!!! é😀 dirty declaration\r\n\r\n"; const dirtyText = comment + depText.replace(/\n/g, "\r\n"); await replace(dependency, dirtyText); diff --git a/vscode/test/workspace-checks.cjs b/vscode/test/workspace-checks.cjs index f6592840..0f5fca76 100644 --- a/vscode/test/workspace-checks.cjs +++ b/vscode/test/workspace-checks.cjs @@ -139,15 +139,15 @@ process.stdin.on('data', chunk => input += chunk); process.stdin.on('end', () => { const request = JSON.parse(input); const dependencies = request.sources.filter(item => sameExistingFile(item.path, ${JSON.stringify(dependencyPath)})); - const dependency = dependencies[0]; + const dependency = dependencies[0] && canonicalSnapshot(dependencies[0]); const ours = sameExistingFile(primary, ${JSON.stringify(entryPath)}); fs.appendFileSync(${JSON.stringify(log)}, JSON.stringify({pid:process.pid,primary,text:dependency?.text,count:dependencies.length}) + '\\n'); const stale = dependency?.text.includes('WAIT'); const diagnostics = ours && dependency ? [{code:stale?'STALE':'IMPORT_CURRENT',severity:'warning',message:'Imported finding', - path:${JSON.stringify(dependencyPath)},start:6,length:2,hint:'',fix:{start:6,length:2,replacement:'',description:'Do not offer imported edits'}}] : []; + path:dependency.path,start:6,length:2,hint:'',fix:{start:6,length:2,replacement:'',description:'Do not offer imported edits'}}] : []; setTimeout(() => process.stdout.write(JSON.stringify({schema:'nil.analysis.v1',source:primary, language:primary.endsWith('.ny')?'ny':'matrix',success:true,reflection:{},diagnostics, - sources:ours && dependency ? [{path:${JSON.stringify(dependencyPath)},text:dependency.text}] : []})), ours && stale ? 5000 : 0); + sources:ours && dependency ? [dependency] : []})), ours && stale ? 5000 : 0); }); `); try { From b22f311bb787dadeed0108414e07a3f3dd41953f Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 20:29:46 -0500 Subject: [PATCH 20/39] Enforce freestanding SDK imports across native object formats --- .../workflows/macos-application-parity.yml | 36 ++- .github/workflows/macos-sdk.yml | 14 + CMakeLists.txt | 14 +- nilassembler/tests/nilassembler_cli_tests.py | 2 + .../freestanding_runtime.cpp | 11 + notebook/2026-11-11-web-runtime-parity.md | 43 +++ scripts/sdk_freestanding_imports.py | 222 +++++++++++++++ scripts/validate-sdk.py | 87 +----- tests/freestanding_hook_contract.cpp | 27 ++ tests/sdk_freestanding_imports_tests.py | 254 ++++++++++++++++++ tests/tools/context_manifest.py | 3 +- tests/tools/project_build_contract.py | 20 +- 12 files changed, 647 insertions(+), 86 deletions(-) create mode 100644 scripts/sdk_freestanding_imports.py create mode 100644 tests/sdk_freestanding_imports_tests.py diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index 1bbfe5dd..869b4b26 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -6,6 +6,7 @@ name: macOS application platform checks - '.github/workflows/macos-application-parity.yml' - 'CMakeLists.txt' - 'nilcompiler/**' + - 'nilruntime-freestanding/**' - 'nilinterpreter/**' - 'nilstd/**' - 'nilbuild/**' @@ -24,6 +25,13 @@ name: macOS application platform checks - 'tests/tools/native_owned_lifetime.py' - 'tests/tools/native_symbol_surface.py' - 'tests/tools/file_modified_at_contract.py' + - 'tests/tools/project_build_contract.py' + - 'tests/tools/context_manifest.py' + - 'nilassembler/tests/nilassembler_cli_tests.py' + - 'scripts/sdk_freestanding_imports.py' + - 'scripts/validate-sdk.py' + - 'tests/sdk_freestanding_imports_tests.py' + - 'tests/freestanding_hook_contract.cpp' - 'vscode/**' workflow_dispatch: @@ -147,7 +155,7 @@ jobs: run: | set -o pipefail cmake --build build/application --parallel 2 --target \ - nilcompiler nilinterpreter nilassembler nillsp nildebugger nilbuild niltest aperture \ + nilcompiler nilinterpreter nilassembler nillinker nillsp nildebugger nilbuild niltest aperture \ nil_vscode_host_runner nil_file_time_support_tests nil_compiler_module_darwin_tests \ nil_darwin_toolchain_inputs_tests nil_freestanding_hook_default nil_freestanding_hook_override \ nildebugger_core_tests nildebugger_source_metadata_tests nildebugger_dap_owner_tests \ @@ -216,6 +224,32 @@ jobs: assert all(case.get('status') == 'run' and all(case.find(kind) is None for kind in ['failure','error','skipped']) for case in cases) PYTHON + - name: Verify project, assembler, context and archive contracts + if: ${{ !cancelled() && steps.tools.outcome == 'success' }} + timeout-minutes: 20 + run: | + python3 - <<'PYTHON' + import json, subprocess, xml.etree.ElementTree as ET + from pathlib import Path + expected = {'nilbuild.project-contract', 'nilassembler.cli', + 'ny.tools.context-manifest', 'nil.freestanding-archive-imports'} + pattern = r'^(nilbuild[.]project-contract|nilassembler[.]cli|ny[.]tools[.]context-manifest|nil[.]freestanding-archive-imports)$' + base = ['ctest', '--test-dir', 'build/application', '-R', pattern] + root = Path('build/application-evidence').resolve() + inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) + (root/'sdk-contract-inventory.json').write_bytes(inventory) + assert {case['name'] for case in json.loads(inventory)['tests']} == expected + with (root/'sdk-contracts.log').open('wb') as output: + run = subprocess.run(base+['--no-tests=error', '--output-on-failure', + '--output-junit', str(root/'sdk-contract-results.xml')], stdout=output, + stderr=subprocess.STDOUT, timeout=1150) + print((root/'sdk-contracts.log').read_text(errors='replace')) + run.check_returncode() + cases = list(ET.parse(root/'sdk-contract-results.xml').getroot().iter('testcase')) + assert len(cases) == len(expected) and {case.get('name') for case in cases} == expected + assert all(case.get('status') == 'run' and all(case.find(kind) is None + for kind in ['failure','error','skipped']) for case in cases) + PYTHON - name: Verify default native toolchain and real incremental reuse if: ${{ !cancelled() && steps.tools.outcome == 'success' }} timeout-minutes: 20 diff --git a/.github/workflows/macos-sdk.yml b/.github/workflows/macos-sdk.yml index cfb6f38f..65365ada 100644 --- a/.github/workflows/macos-sdk.yml +++ b/.github/workflows/macos-sdk.yml @@ -215,6 +215,20 @@ jobs: ctest --test-dir build/ci --output-on-failure --parallel 2 --no-tests=error --timeout 360 \ --test-output-size-passed 16384 --test-output-size-failed 65536 \ --output-junit ctest-results.xml 2>&1 | tee build/macos-sdk-evidence/ctest.log + - name: Upload full CTest evidence immediately + if: always() + timeout-minutes: 3 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: macos-sdk-ctest-${{ matrix.host.arch }}-${{ github.sha }}-${{ github.run_attempt }} + path: | + build/macos-sdk-evidence/registration.json + build/macos-sdk-evidence/capability-registration.json + build/macos-sdk-evidence/sdk-build-evidence.json + build/macos-sdk-evidence/ctest.log + build/ci/ctest-results.xml + if-no-files-found: warn + retention-days: 14 - name: Install isolated development stage id: install # A complete failed suite is still recorded, and still fails this job. diff --git a/CMakeLists.txt b/CMakeLists.txt index b1c52336..f5ad0fc9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -902,8 +902,14 @@ if(BUILD_TESTING) --llvm "${NIL_ENABLE_LLVM}" --cxx "${CMAKE_C_COMPILER}" ) + set(NIL_PROJECT_CONTRACT_TIMEOUT 180) + if(APPLE) + # The mixed-language contract performs multiple independently + # fingerprinted VM/native builds; each retains its own deadline. + set(NIL_PROJECT_CONTRACT_TIMEOUT 900) + endif() set_tests_properties(nilbuild.project-contract PROPERTIES - LABELS "tools;project;analysis;rc" TIMEOUT 180) + LABELS "tools;project;analysis;rc" TIMEOUT "${NIL_PROJECT_CONTRACT_TIMEOUT}") if(UNIX) # exec identity, POSIX controlling terminals and FIFOs do not # exist on Windows. The common contract tests Windows' resident @@ -1011,6 +1017,12 @@ if(BUILD_TESTING) COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/sdk_qualification_tests.py" ) set_tests_properties(ny.sdk-qualification PROPERTIES LABELS "sdk;rc" TIMEOUT 30) + # Inspect the actual runtime archive on every host, independently of + # LLVM availability. Unknown/malformed object members fail closed. + add_test(NAME nil.freestanding-archive-imports + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/sdk_freestanding_imports_tests.py" + --archive $) + set_tests_properties(nil.freestanding-archive-imports PROPERTIES LABELS "sdk;native;abi;rc" TIMEOUT 120) add_test( NAME ny.release-artifacts COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/release_artifact_tests.py" diff --git a/nilassembler/tests/nilassembler_cli_tests.py b/nilassembler/tests/nilassembler_cli_tests.py index d16c5ba8..4003c77e 100644 --- a/nilassembler/tests/nilassembler_cli_tests.py +++ b/nilassembler/tests/nilassembler_cli_tests.py @@ -122,6 +122,8 @@ def link_and_run( if sys.platform.startswith("linux"): command.append("-no-pie") command.extend(["-o", str(executable), str(object_path), str(runtime)]) + if sys.platform == "darwin": + command.extend(["-framework", "Cocoa", "-framework", "CoreGraphics"]) linked = run(command, cwd) if linked.returncode != 0: raise AssertionError( diff --git a/nilruntime-freestanding/freestanding_runtime.cpp b/nilruntime-freestanding/freestanding_runtime.cpp index 8ad745e7..704a36bb 100644 --- a/nilruntime-freestanding/freestanding_runtime.cpp +++ b/nilruntime-freestanding/freestanding_runtime.cpp @@ -163,7 +163,14 @@ void *nyrt_alloc(std::uint64_t size) noexcept { heap_used = start + size; // Zeroed because the helper this serves replaced calloc, and generated code // is entitled to assume a fresh block holds no earlier vessel's bytes. +#if defined(_MSC_VER) + // MSVC has no -ffreestanding equivalent: observable byte writes prevent + // its optimizer from replacing this loop with an unavailable memset. + volatile unsigned char *const zero = block; + for (std::uint64_t index = 0; index < size; ++index) zero[index] = 0; +#else for (std::uint64_t index = 0; index < size; ++index) block[index] = 0; +#endif return block; } @@ -171,7 +178,11 @@ void *nyrt_alloc(std::uint64_t size) noexcept { // has no libc to call and these are not the host's decision to make. void *nyrt_mem_copy(void *destination, const void *source, std::uint64_t size) noexcept { +#if defined(_MSC_VER) + volatile unsigned char *out = static_cast(destination); +#else unsigned char *out = static_cast(destination); +#endif const unsigned char *in = static_cast(source); for (std::uint64_t index = 0; index < size; ++index) out[index] = in[index]; return destination; diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 49c5dc28..2ece137d 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -469,6 +469,49 @@ exclusions in 44.7 seconds using VS Code 1.144.1. Analyzer bytes are unchanged. The complete failing/corrected chain is retained under `editor-workspace-alias-linux-q8g3r09h`; hosted Mac confirmation remains open. +## SDK binary and full-suite follow-up + +The clean Windows `60d4c99` candidate records 586 passes and only the two +headless desktop skips from 588 registrations, and stages eleven root tools, +eighteen checked library modules and compiled documentation. A separate binary +audit nevertheless finds MSVC-generated `memcpy` and `memset` imports in its +freestanding archive. The old validator inspected only ELF objects and silently +accepted other formats. That candidate therefore remains unsuitable for SDK +parity despite its otherwise passing recorded gates. + +MSVC-specific volatile byte destinations prevent those two loop substitutions +without changing GCC/Clang optimization. A new bounded archive reader checks +ELF, COFF and Mach-O symbol tables and rejects malformed or mixed profiles; +unknown formats no longer look import-free. Actual MSVC A/B archives prove the +old imports disappear, while default/override hooks and the expanded heap/copy +checks still pass. Both corrected no-CRT PE hosts link with `/NODEFAULTLIB`, +import only the explicitly selected host `ExitProcess`, execute and exit zero. +Fifteen parser groups, twelve actual Clang cross-format positive/negative +archives, Linux's built runtime and the hosted ARM SDK archive also pass. +The new CTest and SDK validator both enforce the archive boundary. Final Windows +evidence is `freestanding-current-v3/report.json`, SHA-256 +`3bbf0218c33e105d69fa9a0df31bc38fadfd32043fd2bb4983c63dd215c1677a`. +A corrected clean Windows SDK must replace the candidate qualification. + +The ARM `1efbe66` SDK preserves all 268 payload files through relocation and +passes all 1,184 installed consumer routes, 146 debugger checks/15 sessions, +execution checkpoints and twelve PostgreSQL source/binary routes. All 920 +retained evidence files were independently hash-verified. Its source suite is +583 passes, four failures and one ELF-only freestanding execution skip; the +installed results do not erase those failures. In addition to the now-corrected +AppKit input fixture, two manual native-link fixtures omit Cocoa/CoreGraphics, +and the first fingerprinted project build exceeds its 45-second deadline. + +The raw host links now carry the runtime's existing framework dependencies. +Only real Darwin project build/run calls receive the measured VM/native budgets +(180/300 seconds), retaining explicit shorter process/error-test deadlines and +phase timings; their complete CTest is bounded to 900 seconds. The application +workflow now admits the exact project, assembler, embedded-context and archive +contracts as well as its existing native/editor checks. Seven relevant local +CTests pass in 12.99 seconds; actual hosted follow-up remains required. Full SDK +workflows also upload their source inventory immediately, before lengthy +installed campaigns, so failed gates are visible without waiting for packaging. + ## SQL provider environment scope The full Mac source/SDK workflows previously exported SQL's diff --git a/scripts/sdk_freestanding_imports.py b/scripts/sdk_freestanding_imports.py new file mode 100644 index 00000000..de4862ca --- /dev/null +++ b/scripts/sdk_freestanding_imports.py @@ -0,0 +1,222 @@ +#!/usr/bin/env python3 +"""Read freestanding archive imports without requiring nm on an SDK consumer. + +Supported objects are ELF64LE, ordinary 64-bit COFF, and thin Mach-O64LE, +for x86_64/arm64. Unknown members (including bitcode/import objects), malformed +symbol tables, and unsupported encodings fail closed. This is an import-surface +check, not an executable loader or a replacement for native link/run contracts. +""" +from __future__ import annotations + +import struct +from pathlib import Path + +from sdk_artifact_validation import ( + ARCHIVE_MAGIC, MAXIMUM_ARCHIVE_BYTES, MAXIMUM_ARCHIVE_MEMBERS, + MAXIMUM_ARCHIVE_SYMBOLS, is_static_archive, +) + +ALLOWED_IMPORTS = frozenset({'__ny_host_fail'}) +REQUIRED_DEFINITIONS = frozenset({'nyrt_abi_version', 'nyrt_context_begin', 'nyrt_context_end'}) +INDEX_NAMES = frozenset({b'/', b'/SYM64/', b'__.SYMDEF', b'__.SYMDEF SORTED', + b'__.SYMDEF_64', b'__.SYMDEF_64 SORTED'}) + + +def require(condition: bool, message: str) -> None: + if not condition: + raise ValueError(message) + + +def extent(data: bytes, start: int, size: int) -> bytes: + require(0 <= start <= len(data) and 0 <= size <= len(data) - start, + 'object table extends beyond its member') + return data[start:start + size] + + +def name_at(table: bytes, index: int) -> str: + require(0 <= index < len(table), 'symbol name offset outside its string table') + end = table.find(b'\0', index) + require(end != -1, 'unterminated symbol name') + return table[index:end].decode('utf-8', 'strict') + + +def elf_symbols(data: bytes) -> tuple[set[str], set[str]]: + require(len(data) >= 64 and data[:7] == b'\x7fELF\x02\x01\x01', 'unsupported ELF encoding') + kind, machine, version = struct.unpack_from('> 4 not in (1, 2): # STB_GLOBAL/STB_WEAK, including weak imports. + continue + require(shndx < count or shndx in (0xfff1, 0xfff2), + 'unsupported or out-of-range external ELF symbol section') + name = name_at(strings, index) + require(bool(name), 'empty external ELF symbol') + (undefined if shndx == 0 else defined).add(name) + require(found == 1, 'expected exactly one ELF symbol table') + return undefined, defined + + +def coff_symbols(data: bytes) -> tuple[set[str], set[str]]: + require(len(data) >= 20, 'truncated COFF header') + machine, sections, _time, start, count, optional, flags = struct.unpack_from(' 0 and optional == 0 and not flags & 2, + 'unsupported COFF object profile') + extent(data, 20, sections * 40) + require(start >= 20 + sections * 40 and 0 < count <= MAXIMUM_ARCHIVE_SYMBOLS, + 'invalid COFF symbol table') + extent(data, start, count * 18 + 4) + string_start = start + count * 18 + string_size, = struct.unpack_from('= 4, 'invalid COFF string table') + strings = extent(data, string_start, string_size) + undefined, defined = set(), set() + index = 0 + while index < count: + position = start + index * 18 + raw_name, value, section, _type, storage, auxiliaries = struct.unpack_from('<8sIhHBB', data, position) + require(auxiliaries < count - index, 'COFF auxiliary symbols exceed table') + if storage in (2, 105): # IMAGE_SYM_CLASS_EXTERNAL / WEAK_EXTERNAL. + if raw_name[:4] == b'\0' * 4: + name_offset, = struct.unpack_from('= 4, 'COFF symbol references string-table length') + name = name_at(strings, name_offset) + else: + name = raw_name.split(b'\0', 1)[0].decode('utf-8', 'strict') + require(bool(name) and -1 <= section <= sections, 'invalid external COFF symbol') + # A nonzero value with section zero denotes common storage. + (undefined if section == 0 and value == 0 else defined).add(name) + index += 1 + auxiliaries + return undefined, defined + + +def macho_symbols(data: bytes) -> tuple[set[str], set[str]]: + require(len(data) >= 32, 'truncated Mach-O header') + magic, cpu, _subtype, kind, commands, size, _flags, _reserved = struct.unpack_from('= 8 and length % 8 == 0 and length <= end - position, + 'invalid Mach-O load command size') + if command == 2: # LC_SYMTAB + require(length == 24 and symtab is None, 'invalid or duplicate Mach-O symbol table') + symtab = struct.unpack_from('= 72, 'truncated Mach-O segment command') + count, = struct.unpack_from(' 1, 'missing Mach-O C ABI underscore') + name = raw_name[1:] # Exactly one ABI underscore; preserve language underscores. + (undefined if symbol_type == 0 and value == 0 else defined).add(name) + return undefined, defined + + +def object_symbols(data: bytes) -> tuple[set[str], set[str]]: + if data.startswith(b'\x7fELF'): + return elf_symbols(data) + if data.startswith(b'\xcf\xfa\xed\xfe'): + return macho_symbols(data) + if data[:2] in (b'\x64\x86', b'\x64\xaa'): + return coff_symbols(data) + raise ValueError('unsupported archive member object format') + + +def freestanding_archive_imports(archive: Path) -> set[str]: + """Return unresolved external names, or raise on any uninspected member.""" + require(is_static_archive(archive), 'invalid archive framing or runtime symbol index') + # Bound the read itself as well as the earlier stat, including a replaced file. + with archive.open('rb') as stream: + data = stream.read(MAXIMUM_ARCHIVE_BYTES + 1) + require(len(data) <= MAXIMUM_ARCHIVE_BYTES and data.startswith(ARCHIVE_MAGIC), 'invalid archive size/magic') + undefined, defined = set(), set() + offset = len(ARCHIVE_MAGIC) + members = 0 + objects = 0 + profile = None + while offset < len(data): + members += 1 + require(members <= MAXIMUM_ARCHIVE_MEMBERS, 'too many archive members') + header = extent(data, offset, 60) + require(header[58:] == b'`\n' and header[48:58].strip().isdigit(), 'invalid archive member header') + size = int(header[48:58].strip()) + body = extent(data, offset + 60, size) + name = header[:16].rstrip(b' ') + if name.startswith(b'#1/'): + require(name[3:].isdigit(), 'invalid BSD extended member name') + length = int(name[3:]) + require(0 < length <= min(size, 65536), 'invalid BSD member name length') + name, body = body[:length].rstrip(b'\0'), body[length:] + if len(name) > 1 and name.endswith(b'/') and name not in (b'//', b'/SYM64/'): + name = name[:-1] + if name not in INDEX_NAMES and name != b'//': + imports, exports = object_symbols(body) + if body.startswith(b'\x7fELF'): + member_profile = ('ELF', struct.unpack_from(' 0 and REQUIRED_DEFINITIONS <= defined, + 'archive members do not define the required runtime ABI') + return undefined - defined diff --git a/scripts/validate-sdk.py b/scripts/validate-sdk.py index bcc2f5b2..de88bfdb 100755 --- a/scripts/validate-sdk.py +++ b/scripts/validate-sdk.py @@ -24,6 +24,7 @@ is_nil_library, is_nil_v2_library, is_static_archive, nil_library_format, nil_library_version, ) +from sdk_freestanding_imports import ALLOWED_IMPORTS, freestanding_archive_imports from sdk_qualification import LEGACY_TOOLS, APERTURE_TOOLS, TOOLS, qualification_errors from sdk_sources import is_source, module_artifact_path, canonical_std_namespace from sdk_runtime import (STATIC_NAMES, SHARED_NAMES, RUNTIME_PAIRS, @@ -47,9 +48,6 @@ FREESTANDING_RUNTIME_PATHS = tuple( "nillib/native/" + name for name in FREESTANDING_STATIC_NAMES ) -# Synthesised by the linker rather than supplied by anyone, so its presence -# says nothing about what the archive needs from a host. -LINKER_SYNTHESISED_SYMBOLS = frozenset({"_GLOBAL_OFFSET_TABLE_"}) SHA256_PATTERN = re.compile(r"[0-9a-f]{64}") REVISION_PATTERN = re.compile(r"(?:[0-9a-f]{40}|[0-9a-f]{64})") MODULE_PATTERN = re.compile( @@ -505,89 +503,18 @@ def validate_freestanding_runtime( ) return if archive is not None: - unexpected = sorted( - freestanding_archive_imports(archive) - LINKER_SYNTHESISED_SYMBOLS - ) + try: + unexpected = sorted(freestanding_archive_imports(archive) - ALLOWED_IMPORTS) + except (OSError, ValueError) as error: + errors.append("native_freestanding_runtime import validation failed: " + str(error)) + return if unexpected: errors.append( - "native_freestanding_runtime must import nothing but the weak " + "native_freestanding_runtime must import nothing but the " "host failure hook; found: {}".format(unexpected) ) -def freestanding_archive_imports(archive: Path) -> set[str]: - """Undefined, non-weak symbol names in an ELF static archive. - - The ELF symbol tables are read directly rather than through nm, so the - check does not depend on binutils being installed where an SDK is - validated. A non-ELF archive returns nothing: the import surface is - pinned on ELF hosts by the repository's own tests, and this record must - not fail a Windows or macOS package for lacking a reader. - """ - try: - data = archive.read_bytes() - except OSError: - return set() - if not data.startswith(b"!\n"): - return set() - undefined: set[str] = set() - offset = 8 - while offset + 60 <= len(data): - header = data[offset:offset + 60] - if header[58:60] != b"`\n": - break - try: - size = int(header[48:58].decode("ascii").strip() or "0") - except ValueError: - break - body = data[offset + 60:offset + 60 + size] - offset += 60 + size + (size & 1) - if body[:4] != b"\x7fELF" or body[4] != 2 or body[5] != 1: - continue - undefined |= elf_undefined_symbols(body) - return undefined - - -def elf_undefined_symbols(data: bytes) -> set[str]: - """Named SHN_UNDEF symbols in one little-endian 64-bit ELF object.""" - try: - section_offset = int.from_bytes(data[0x28:0x30], "little") - section_size = int.from_bytes(data[0x3a:0x3c], "little") - section_count = int.from_bytes(data[0x3c:0x3e], "little") - names_index = int.from_bytes(data[0x3e:0x40], "little") - - def section(index: int) -> tuple[int, int, int, int, int]: - base = section_offset + index * section_size - name = int.from_bytes(data[base:base + 4], "little") - offset = int.from_bytes(data[base + 0x18:base + 0x20], "little") - size = int.from_bytes(data[base + 0x20:base + 0x28], "little") - link = int.from_bytes(data[base + 0x28:base + 0x2c], "little") - entry = int.from_bytes(data[base + 0x38:base + 0x40], "little") - return name, offset, size, link, entry - - def text(table: int, index: int) -> str: - end = data.index(b"\0", table + index) - return data[table + index:end].decode("utf-8", "replace") - - names_offset = section(names_index)[1] - undefined: set[str] = set() - for index in range(section_count): - name, offset, size, link, entry = section(index) - if text(names_offset, name) != ".symtab" or not entry: - continue - strings = section(link)[1] - for position in range(offset, offset + size, entry): - name_index = int.from_bytes(data[position:position + 4], "little") - info = data[position + 4] - shndx = int.from_bytes(data[position + 6:position + 8], "little") - # STB_WEAK is a host-supplied hook, not a requirement. - if name_index and shndx == 0 and info >> 4 != 2: - undefined.add(text(strings, name_index)) - return undefined - except (IndexError, ValueError): - return set() - - def validate_shared_runtime(value: Any, files: dict[str, Path], errors: list[str]) -> None: key = validate_file_record(value, "native_shared_runtime", files, errors) if key is None: diff --git a/tests/freestanding_hook_contract.cpp b/tests/freestanding_hook_contract.cpp index aada5c64..52c22dde 100644 --- a/tests/freestanding_hook_contract.cpp +++ b/tests/freestanding_hook_contract.cpp @@ -4,6 +4,12 @@ extern "C" std::uint32_t nyrt_abi_version() noexcept; extern "C" void __ny_host_fail(const char *) noexcept; +extern "C" void nyrt_heap_init(void *, std::uint64_t) noexcept; +extern "C" void *nyrt_alloc(std::uint64_t) noexcept; +extern "C" void *nyrt_mem_copy(void *, const void *, std::uint64_t) noexcept; +extern "C" std::uint64_t nyrt_str_length(const char *) noexcept; +extern "C" std::uint64_t nyrt_heap_used() noexcept; +extern "C" void nyrt_context_end() noexcept; #if defined(NIL_TEST_HOST_OVERRIDE) static const char *observed = nullptr; @@ -19,5 +25,26 @@ int main() { #if defined(NIL_TEST_HOST_OVERRIDE) if (observed != message) return 2; #endif + alignas(16) unsigned char arena[128]; + for (auto &byte : arena) byte = 0xa5; + nyrt_heap_init(arena, sizeof(arena)); + auto *first = static_cast(nyrt_alloc(33)); + if (first != arena || nyrt_heap_used() != 33) return 3; + for (unsigned i = 0; i < 33; ++i) if (first[i] != 0) return 4; + for (unsigned i = 33; i < sizeof(arena); ++i) if (arena[i] != 0xa5) return 5; + const unsigned char source[] = {0, 0xff, 17, 0x80, 41}; + if (nyrt_mem_copy(first + 1, source, sizeof(source)) != first + 1) return 6; + for (unsigned i = 0; i < sizeof(source); ++i) if (first[i + 1] != source[i]) return 7; + if (first[0] || first[6]) return 8; + if (nyrt_mem_copy(nullptr, nullptr, 0) != nullptr) return 9; + if (nyrt_alloc(0) != arena + 48 || nyrt_heap_used() != 49 || arena[48]) return 10; + if (nyrt_alloc(UINT64_MAX) != nullptr || nyrt_heap_used() != 49) return 11; + if (nyrt_str_length(nullptr) || nyrt_str_length("") || nyrt_str_length("ab\0c") != 2) return 12; + nyrt_context_end(); + if (nyrt_heap_used() || nyrt_alloc(128) != arena) return 13; + for (auto byte : arena) if (byte) return 14; + if (nyrt_alloc(1) != nullptr) return 15; + nyrt_heap_init(nullptr, 128); + if (nyrt_alloc(1) != nullptr) return 16; return 0; } diff --git a/tests/sdk_freestanding_imports_tests.py b/tests/sdk_freestanding_imports_tests.py new file mode 100644 index 00000000..0c5f459f --- /dev/null +++ b/tests/sdk_freestanding_imports_tests.py @@ -0,0 +1,254 @@ +#!/usr/bin/env python3 +"""Cross-format parser controls and actual built freestanding archive contract.""" +from __future__ import annotations + +import argparse +import hashlib +import importlib.util +from pathlib import Path +import struct +import sys +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / 'scripts')) +from sdk_freestanding_imports import (ALLOWED_IMPORTS, REQUIRED_DEFINITIONS, + freestanding_archive_imports, object_symbols) +from sdk_native_archive_validation import ar_member, sysv_index_payload, bsd_index_payload + + +def symbols(imports): + return [(name, True) for name in sorted(REQUIRED_DEFINITIONS)] + [(name, False) for name in imports] + + +def elf(imports=(), *, machine=62, weak=False): + strings = bytearray(b'\0') + entries = bytearray(24) + for name, defined in symbols(imports): + index = len(strings) + strings.extend(name.encode() + b'\0') + entries.extend(struct.pack('\n' + index + names + member + extra + + +class Imports(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory(prefix='nil-freestanding-imports-') + self.addCleanup(self.temporary.cleanup) + self.path = Path(self.temporary.name) / 'runtime.a' + + def read(self, value): + self.path.write_bytes(value) + return freestanding_archive_imports(self.path) + + def test_all_six_profiles_and_archive_names(self): + for make, machine in ((elf, 62), (elf, 183), (coff, 0x8664), (coff, 0xaa64), + (macho, 0x1000007), (macho, 0x100000c)): + for naming in ({}, {'bsd': True}, {'gnu': True}): + with self.subTest(format=make.__name__, machine=machine, naming=naming): + self.assertEqual(self.read(archive(make(machine=machine), **naming)), set()) + self.assertEqual(self.read(archive(make(['memcpy', 'memset', '__ny_host_fail'], machine=machine), + **naming)), {'memcpy', 'memset', '__ny_host_fail'}) + + def test_weak_import_is_not_silently_ignored(self): + for make in (elf, coff, macho): + with self.subTest(format=make.__name__): + self.assertEqual(self.read(archive(make(['memcpy'], weak=True))), {'memcpy'}) + + def test_definitions_resolve_imports(self): + for make in (elf, coff, macho): + self.assertEqual(self.read(archive(make(['nyrt_abi_version']))), set()) + + def test_got_exception_applies_only_to_elf(self): + self.assertEqual(self.read(archive(elf(['_GLOBAL_OFFSET_TABLE_']))), set()) + for make in (coff, macho): + self.assertEqual(self.read(archive(make(['_GLOBAL_OFFSET_TABLE_']))), {'_GLOBAL_OFFSET_TABLE_'}) + self.assertNotIn('_GLOBAL_OFFSET_TABLE_', ALLOWED_IMPORTS) + + def test_incompatible_member_cannot_resolve_an_import(self): + for second in (elf(machine=183), coff(), macho()): + # An incompatible second member is rejected even when it does not + # happen to resolve an import; cross-target archives are unsupported. + with self.assertRaises(ValueError): + self.read(archive(elf(['memcpy']), extra=ar_member(second, b'second.o/'))) + + def test_external_definitions_require_real_sections(self): + e = bytearray(elf(['memcpy'])) + section, = struct.unpack_from('\n', valid[:-1], valid + b'x', b'!\n' + valid[8:]): + with self.assertRaises(ValueError): + self.read(value) + + def test_malformed_symbol_and_string_tables(self): + mutations = [] + e = bytearray(elf(['memcpy'])) + section, = struct.unpack_from(' dict[str, object]: raise AssertionError("Darwin context retention requires the host clang++ driver") stripped = root / (artifact.stem + "-dead-strip") success(run([driver, "-pthread", "-Wl,-dead_strip", "-o", stripped, - obj, tools["runtime"]]), "Darwin dead-strip linking") + obj, tools["runtime"], "-framework", "Cocoa", "-framework", "CoreGraphics"]), + "Darwin dead-strip linking") if inspect(stripped) != source_manifest: raise AssertionError("Darwin dead stripping removed or changed context") success(run([stripped]), "dead-stripped context execution", EXPECTED) diff --git a/tests/tools/project_build_contract.py b/tests/tools/project_build_contract.py index 296ddcbd..4b3fede9 100644 --- a/tests/tools/project_build_contract.py +++ b/tests/tools/project_build_contract.py @@ -140,8 +140,15 @@ def save(value=base, *, file=project): file.write_text(json.dumps(value), encoding='utf8', newline='\n') def invoke(command, *flags, status=0, cwd=root, explicit_tools=True, sdk=True, - project_flag=True, environment=None, timeout=45): + project_flag=True, environment=None, timeout=None): nonlocal count + measured_build = sys.platform == 'darwin' and command in ('build', 'run') and explicit_tools + if timeout is None: + # Darwin independently hashes full loaded-image/shared-cache + # inputs before and after real builds. Hosted VM builds have + # exceeded 45 seconds; native builds measured 144 seconds before execution. + native = any(left == '--target' and right == 'native' for left, right in zip(flags, flags[1:])) + timeout = (300 if native else 180) if measured_build else 45 call = [args.builder, command] if project_flag: call += ['--project', str(project)] @@ -150,8 +157,15 @@ def invoke(command, *flags, status=0, cwd=root, explicit_tools=True, sdk=True, if sdk: call += ['--sdk', args.stdlib] call += list(map(str, flags)) - result = subprocess.run(call, cwd=cwd, text=True, capture_output=True, - encoding='utf-8', timeout=timeout, env=environment) + if measured_build: + print(f'nilbuild contract: starting {command} ({timeout}s bound)', flush=True) + started = time.monotonic() + try: + result = subprocess.run(call, cwd=cwd, text=True, capture_output=True, + encoding='utf-8', timeout=timeout, env=environment) + finally: + if measured_build: + print(f'nilbuild contract: {command} elapsed {time.monotonic() - started:.3f}s', flush=True) assert result.returncode == status, (call, result.returncode, result.stdout, result.stderr) count += 1 return result From 9fb6c04ad7a8f5a0c65128daa48945fc84839aa6 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 20:30:45 -0500 Subject: [PATCH 21/39] Reject unsupported ELF symbol bindings in SDK archives --- scripts/sdk_freestanding_imports.py | 4 +++- tests/sdk_freestanding_imports_tests.py | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/scripts/sdk_freestanding_imports.py b/scripts/sdk_freestanding_imports.py index de4862ca..73add971 100644 --- a/scripts/sdk_freestanding_imports.py +++ b/scripts/sdk_freestanding_imports.py @@ -65,7 +65,9 @@ def elf_symbols(data: bytes) -> tuple[set[str], set[str]]: strings = extent(data, sections[link][4], sections[link][5]) for position in range(start, start + size, entry_size): index, info, _other, shndx, _value, _size = struct.unpack_from('> 4 not in (1, 2): # STB_GLOBAL/STB_WEAK, including weak imports. + binding = info >> 4 + require(binding in (0, 1, 2), 'unsupported ELF symbol binding') + if binding == 0: # Skip STB_LOCAL, retain GLOBAL and WEAK imports. continue require(shndx < count or shndx in (0xfff1, 0xfff2), 'unsupported or out-of-range external ELF symbol section') diff --git a/tests/sdk_freestanding_imports_tests.py b/tests/sdk_freestanding_imports_tests.py index 0c5f459f..324d6089 100644 --- a/tests/sdk_freestanding_imports_tests.py +++ b/tests/sdk_freestanding_imports_tests.py @@ -189,7 +189,8 @@ def test_malformed_symbol_and_string_tables(self): section, = struct.unpack_from(' Date: Wed, 11 Nov 2026 20:55:07 -0500 Subject: [PATCH 22/39] Share bounded Windows executable replacement across tools --- .../workflows/macos-application-parity.yml | 2 +- CMakeLists.txt | 5 + common/atomic_file_replace.hpp | 48 +++++ nilcompiler/src/main.cpp | 38 +--- nillinker/main.cpp | 20 +- tests/atomic_file_replace_tests.cpp | 182 ++++++++++++++++++ 6 files changed, 242 insertions(+), 53 deletions(-) create mode 100644 common/atomic_file_replace.hpp create mode 100644 tests/atomic_file_replace_tests.cpp diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index 869b4b26..06873a4a 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -15,7 +15,7 @@ name: macOS application platform checks - 'nillsp/**' - 'aperture/**' - 'common/**' - - 'nillinker/process_runner*' + - 'nillinker/**' - 'validation/ny-3d/scripts/debugger_process.py' - 'scripts/ci/macos-toolchain-census.py' - 'tests/darwin*' diff --git a/CMakeLists.txt b/CMakeLists.txt index f5ad0fc9..2c474b89 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -115,6 +115,11 @@ if(BUILD_TESTING) set_tests_properties(nil.freestanding-hook-${nil_hook_profile} PROPERTIES LABELS "runtime;rc" TIMEOUT 15) endforeach() if(WIN32) + add_executable(nil_atomic_file_replace_tests tests/atomic_file_replace_tests.cpp) + target_compile_features(nil_atomic_file_replace_tests PRIVATE cxx_std_17) + target_link_libraries(nil_atomic_file_replace_tests PRIVATE advapi32) + add_test(NAME nil.atomic-file-replace COMMAND $) + set_tests_properties(nil.atomic-file-replace PROPERTIES LABELS "tools;windows;publication;rc" TIMEOUT 15) # Keep the public backend declarations usable after real Windows RPC # headers, even in an LLVM-OFF build that normally hides this interface. add_library(nil_native_windows_header_contract OBJECT tests/native_windows_header_contract.cpp) diff --git a/common/atomic_file_replace.hpp b/common/atomic_file_replace.hpp new file mode 100644 index 00000000..e65db249 --- /dev/null +++ b/common/atomic_file_replace.hpp @@ -0,0 +1,48 @@ +#ifndef NIL_COMMON_ATOMIC_FILE_REPLACE_HPP +#define NIL_COMMON_ATOMIC_FILE_REPLACE_HPP + +#include +#include + +#ifdef _WIN32 +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#endif + +namespace nil::detail { + +inline bool replace_output_file(const std::filesystem::path &staging, + const std::filesystem::path &output, + std::error_code &error) { +#ifdef _WIN32 + // A recently executed image or a reader without delete sharing can briefly + // prevent replacement. Retain the compiler's bounded retry policy for every + // executable publisher. Persistent locks and access failures still fail; + // never unlink the old output or relax its permissions to make progress. + for (int attempt = 0;; ++attempt) { + if (MoveFileExW(staging.c_str(), output.c_str(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH)) { + error.clear(); + return true; + } + const DWORD code = GetLastError(); + const bool transient = code == ERROR_ACCESS_DENIED || + code == ERROR_SHARING_VIOLATION || + code == ERROR_LOCK_VIOLATION; + if (!transient || attempt >= 50) { + error = std::error_code(static_cast(code), std::system_category()); + return false; + } + Sleep(20); + } +#else + std::filesystem::rename(staging, output, error); + return !error; +#endif +} + +} // namespace nil::detail + +#endif diff --git a/nilcompiler/src/main.cpp b/nilcompiler/src/main.cpp index 0ea32186..5ca71936 100644 --- a/nilcompiler/src/main.cpp +++ b/nilcompiler/src/main.cpp @@ -7,6 +7,7 @@ #include "../../common/native_executable_path.hpp" #include "../../common/windows_utf8_manifest.hpp" #include "../../common/windows_private_storage.hpp" +#include "../../common/atomic_file_replace.hpp" #include "compiler_module_fingerprint.hpp" #ifdef _WIN32 #include "../../nillinker/process_runner.hpp" @@ -316,37 +317,6 @@ create_private_staging_directory(const std::filesystem::path &output, return std::nullopt; } -static bool replace_output_file(const std::filesystem::path &staging, - const std::filesystem::path &output, - std::error_code &error) { -#ifdef _WIN32 - // Windows can refuse to replace an executable that ran moments ago: the - // image section outlives the process for a short while, and a scanner may - // hold a freshly written file open while it inspects it. Both clear on their - // own within milliseconds, and neither means the compiler did anything - // wrong, so retry briefly rather than fail the build. Observed as - // "Failed to install linked executable: Access is denied." on a test that - // compiles to the same output path twice and runs it in between. - for (int attempt = 0;; ++attempt) { - if (MoveFileExW(staging.c_str(), output.c_str(), - MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH)) - return true; - const DWORD code = GetLastError(); - const bool transient = code == ERROR_ACCESS_DENIED || - code == ERROR_SHARING_VIOLATION || - code == ERROR_LOCK_VIOLATION; - if (!transient || attempt >= 50) { - error = std::error_code(static_cast(code), std::system_category()); - return false; - } - Sleep(20); - } -#else - std::filesystem::rename(staging, output, error); - return !error; -#endif -} - static bool ensure_staged_regular_file_permissions( const std::filesystem::path &path, std::filesystem::perms required_owner_permissions, std::error_code &error) { @@ -2006,7 +1976,7 @@ static int compile_main(int argc, char *argv[], ny::analysis_session *analysis) if (!ensure_staged_regular_file_permissions(nil_out_path, std::filesystem::perms::owner_read | std::filesystem::perms::owner_write, output_error) || - !replace_output_file(nil_out_path, output_path, output_error)) { + !nil::detail::replace_output_file(nil_out_path, output_path, output_error)) { std::cerr << "Error: failed to install complete test artifact: " << output_error.message() << '\n'; return 1; @@ -2130,7 +2100,7 @@ static int compile_main(int argc, char *argv[], ny::analysis_session *analysis) } output_error.clear(); - if (!replace_output_file(staging_path, out_exe, output_error)) { + if (!nil::detail::replace_output_file(staging_path, out_exe, output_error)) { std::cerr << "Error: failed to install bundled executable: " << output_error.message() << "\n"; return 1; @@ -2398,7 +2368,7 @@ static int compile_main(int argc, char *argv[], ny::analysis_session *analysis) } std::error_code install_error; - if (!replace_output_file(link_output_path, exe_path, install_error)) { + if (!nil::detail::replace_output_file(link_output_path, exe_path, install_error)) { std::cerr << "[Native] Failed to install linked executable: " << install_error.message() << "\n"; return 1; diff --git a/nillinker/main.cpp b/nillinker/main.cpp index 5015db7e..5ef5a067 100644 --- a/nillinker/main.cpp +++ b/nillinker/main.cpp @@ -6,6 +6,7 @@ #include "../common/standard_streams.hpp" #include "../common/windows_utf8_manifest.hpp" #include "../common/windows_private_storage.hpp" +#include "../common/atomic_file_replace.hpp" #include "nillinker.hpp" #include "process_runner.hpp" #include "secure_file_io.hpp" @@ -992,23 +993,6 @@ bool secure_executable(const fs::path &path, std::error_code &error) { return true; } -bool replace_output_file(const fs::path &staging, const fs::path &output, - std::error_code &error) { -#ifdef _WIN32 - if (MoveFileExW(staging.c_str(), output.c_str(), - MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH)) { - error.clear(); - return true; - } - error = - std::error_code(static_cast(GetLastError()), std::system_category()); - return false; -#else - fs::rename(staging, output, error); - return !error; -#endif -} - void print_usage(std::ostream &out) { out << "Usage: nillinker [options] object...\n" << "\nOptions:\n" @@ -1602,7 +1586,7 @@ bool Linker::link(const fs::path &requested_output) { output_error.message(); return false; } - if (!replace_output_file(staged_output, output, output_error)) { + if (!nil::detail::replace_output_file(staged_output, output, output_error)) { error_ = "cannot atomically publish linked executable to '" + display_path(output) + "': " + output_error.message(); return false; diff --git a/tests/atomic_file_replace_tests.cpp b/tests/atomic_file_replace_tests.cpp new file mode 100644 index 00000000..29511141 --- /dev/null +++ b/tests/atomic_file_replace_tests.cpp @@ -0,0 +1,182 @@ +// Exercise real Windows sharing/access failures against the publication helper. +// Every file and handle belongs to this fixture; no installed SDK is modified. +#include "../common/atomic_file_replace.hpp" +#include "../common/private_directory.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef _WIN32 +#error This contract requires native Windows file-sharing semantics. +#endif + +namespace { +namespace fs = std::filesystem; +using Clock = std::chrono::steady_clock; +int checks = 0; + +void check(bool value, const char *message) { + ++checks; + if (!value) throw std::runtime_error(message); +} + +class Handle { +public: + explicit Handle(HANDLE value) : value_(value) {} + ~Handle() { if (value_ != INVALID_HANDLE_VALUE) CloseHandle(value_); } + Handle(const Handle &) = delete; + Handle &operator=(const Handle &) = delete; + HANDLE get() const { return value_; } + bool close() { + const HANDLE value = value_; + value_ = INVALID_HANDLE_VALUE; + return CloseHandle(value) != FALSE; + } +private: + HANDLE value_; +}; + +class Join { +public: + explicit Join(std::thread &thread) : thread_(thread) {} + ~Join() { if (thread_.joinable()) thread_.join(); } +private: + std::thread &thread_; +}; + +class ReadOnly { +public: + explicit ReadOnly(const fs::path &path) : path_(path) { + check(SetFileAttributesW(path_.c_str(), FILE_ATTRIBUTE_READONLY) != FALSE, + "set owned target read-only"); + } + ~ReadOnly() { SetFileAttributesW(path_.c_str(), FILE_ATTRIBUTE_NORMAL); } +private: + fs::path path_; +}; + +void write(const fs::path &path, const std::string &bytes) { + std::ofstream stream(path, std::ios::binary | std::ios::trunc); + stream.write(bytes.data(), static_cast(bytes.size())); + stream.close(); + check(!stream.fail(), "write owned file bytes"); +} + +std::string read(const fs::path &path) { + std::ifstream stream(path, std::ios::binary); + check(stream.is_open(), "open owned file for byte oracle"); + const std::string bytes((std::istreambuf_iterator(stream)), {}); + check(!stream.bad(), "read owned file bytes"); + return bytes; +} + +bool transient(DWORD error) { + return error == ERROR_ACCESS_DENIED || error == ERROR_SHARING_VIOLATION || + error == ERROR_LOCK_VIOLATION; +} + +void bounded(Clock::time_point start) { + check(Clock::now() - start < std::chrono::seconds(5), + "publication attempt exceeded five-second fixture bound"); +} + +void contracts(const fs::path &root) { + const auto source = root / L"staging space \u03bb.tmp"; + const auto output = root / L"output space \u03bb.exe"; + const std::string old_bytes("old\0\xff", 5), new_bytes("new\0\x80payload", 12); + std::error_code error = std::make_error_code(std::errc::io_error); + write(source, new_bytes); + check(nil::detail::replace_output_file(source, output, error), "publish initially absent output"); + check(!error && !fs::exists(source) && read(output) == new_bytes, + "successful publication clears stale error and consumes staging file"); + + write(output, old_bytes); + write(source, new_bytes); + { + Handle held(CreateFileW(output.c_str(), GENERIC_READ, + FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr, + OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr)); + check(held.get() != INVALID_HANDLE_VALUE, "open transient target without delete sharing"); + // The linker's old single-attempt operation must fail under this exact + // real OS condition, before another thread is allowed to release it. + const BOOL single = MoveFileExW(source.c_str(), output.c_str(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH); + const DWORD single_error = GetLastError(); + check(single == FALSE && transient(single_error), "old single attempt rejects held target"); + check(read(output) == old_bytes && read(source) == new_bytes, + "single-attempt negative control preserves both files"); + std::atomic closed{false}; + std::thread releaser([&] { + Sleep(160); + closed.store(held.close()); + }); + Join join(releaser); + const auto start = Clock::now(); + const bool replaced = nil::detail::replace_output_file(source, output, error); + releaser.join(); + bounded(start); + check(closed.load(), "owned transient handle closed successfully"); + check(replaced && !error, "retry publishes after transient handle release"); + check(!fs::exists(source) && read(output) == new_bytes, + "transient success publishes exact new bytes"); + } + + write(output, old_bytes); + write(source, new_bytes); + { + Handle held(CreateFileW(output.c_str(), GENERIC_READ, + FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr, + OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr)); + check(held.get() != INVALID_HANDLE_VALUE, "open permanently held target"); + const auto start = Clock::now(); + const bool replaced = nil::detail::replace_output_file(source, output, error); + bounded(start); + check(!replaced && transient(static_cast(error.value())), + "permanent sharing failure remains a reported error"); + check(read(output) == old_bytes && read(source) == new_bytes, + "permanent sharing failure preserves old output and staging bytes"); + } + { + ReadOnly readonly(output); + const auto start = Clock::now(); + const bool replaced = nil::detail::replace_output_file(source, output, error); + bounded(start); + check(!replaced && error.value() == ERROR_ACCESS_DENIED, "read-only target fails without permission changes"); + check((GetFileAttributesW(output.c_str()) & FILE_ATTRIBUTE_READONLY) != 0, + "failed publication leaves read-only attribute intact"); + check(read(output) == old_bytes && read(source) == new_bytes, + "read-only failure preserves both files"); + } + check(fs::remove(source), "remove only the owned staging file for missing-source control"); + const auto start = Clock::now(); + const bool replaced = nil::detail::replace_output_file(source, output, error); + bounded(start); + check(!replaced && error.value() == ERROR_FILE_NOT_FOUND, "missing staging file is reported"); + check(read(output) == old_bytes && !fs::exists(source), "missing-source failure preserves output"); +} +} // namespace + +int main() { + fs::path root; + try { + root = nil::detail::create_private_directory(fs::temp_directory_path(), "nil-atomic-replace-"); + contracts(root); + fs::remove_all(root); + check(!fs::exists(root), "owned fixture directory removed"); + std::cout << "Windows atomic replacement: " << checks << " checks passed\n"; + return 0; + } catch (const std::exception &error) { + if (!root.empty()) { + std::error_code ignored; + fs::remove_all(root, ignored); + } + std::cerr << error.what() << '\n'; + return 1; + } +} From eca43f1d01997d777f943feb2a7a877964ec0084 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 21:27:22 -0500 Subject: [PATCH 23/39] Qualify Darwin tool discovery and honor SDK build limits --- .../workflows/macos-application-parity.yml | 11 +- CMakeLists.txt | 3 + README.md | 12 +- docs/platform-support.md | 11 +- docs/release/v1-profile.md | 15 ++- notebook/2026-11-11-web-runtime-parity.md | 109 ++++++++++++++++-- scripts/build-sdk.sh | 4 +- tests/tools/tool_discovery_contract.py | 34 +++++- 8 files changed, 165 insertions(+), 34 deletions(-) diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index 06873a4a..36f6ea05 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -26,6 +26,7 @@ name: macOS application platform checks - 'tests/tools/native_symbol_surface.py' - 'tests/tools/file_modified_at_contract.py' - 'tests/tools/project_build_contract.py' + - 'tests/tools/tool_discovery_contract.py' - 'tests/tools/context_manifest.py' - 'nilassembler/tests/nilassembler_cli_tests.py' - 'scripts/sdk_freestanding_imports.py' @@ -224,16 +225,16 @@ jobs: assert all(case.get('status') == 'run' and all(case.find(kind) is None for kind in ['failure','error','skipped']) for case in cases) PYTHON - - name: Verify project, assembler, context and archive contracts + - name: Verify project, discovery, assembler, context and archive contracts if: ${{ !cancelled() && steps.tools.outcome == 'success' }} - timeout-minutes: 20 + timeout-minutes: 30 run: | python3 - <<'PYTHON' import json, subprocess, xml.etree.ElementTree as ET from pathlib import Path - expected = {'nilbuild.project-contract', 'nilassembler.cli', + expected = {'nilbuild.project-contract', 'nilbuild.tool-discovery-contract', 'nilassembler.cli', 'ny.tools.context-manifest', 'nil.freestanding-archive-imports'} - pattern = r'^(nilbuild[.]project-contract|nilassembler[.]cli|ny[.]tools[.]context-manifest|nil[.]freestanding-archive-imports)$' + pattern = r'^(nilbuild[.](project-contract|tool-discovery-contract)|nilassembler[.]cli|ny[.]tools[.]context-manifest|nil[.]freestanding-archive-imports)$' base = ['ctest', '--test-dir', 'build/application', '-R', pattern] root = Path('build/application-evidence').resolve() inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) @@ -242,7 +243,7 @@ jobs: with (root/'sdk-contracts.log').open('wb') as output: run = subprocess.run(base+['--no-tests=error', '--output-on-failure', '--output-junit', str(root/'sdk-contract-results.xml')], stdout=output, - stderr=subprocess.STDOUT, timeout=1150) + stderr=subprocess.STDOUT, timeout=1750) print((root/'sdk-contracts.log').read_text(errors='replace')) run.check_returncode() cases = list(ET.parse(root/'sdk-contract-results.xml').getroot().iter('testcase')) diff --git a/CMakeLists.txt b/CMakeLists.txt index 2c474b89..e6927f1f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -957,6 +957,9 @@ if(BUILD_TESTING) --compiler $ --interpreter $ --stdlib "${CMAKE_CURRENT_SOURCE_DIR}/nilstd/src") set_tests_properties(nilbuild.tool-discovery-contract PROPERTIES LABELS "tools;project;rc" TIMEOUT 180) + if(APPLE) + set_tests_properties(nilbuild.tool-discovery-contract PROPERTIES TIMEOUT 600) + endif() add_test(NAME nil.platform.raw-byte-routes COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/raw_byte_routes_contract.py" --compiler $ --interpreter $ diff --git a/README.md b/README.md index 85ef4551..55c67745 100644 --- a/README.md +++ b/README.md @@ -247,7 +247,8 @@ such as `say formatted "value={5}"`; it works on interpreted and native routes. ## Project status The current version is **1.0.0-rc.1**. Linux x86_64 is the reference development -platform. The Windows x86_64 development SDK has +platform; the `9fb6c04` LLVM-enabled source suite passes all 590 CTests without +skips. The Windows x86_64 development SDK has [paired Linux acceptance and installed tooling coverage](notebook/2026-11-02-windows-port.md). The newer Windows source campaign also passes the Win32 desktop contracts and all 88 real VS Code host groups; a fresh SDK still needs separate qualification. @@ -259,14 +260,15 @@ checks. Both macOS development SDKs also pass the full installed standard-library campaign, isolated PostgreSQL application routes, installed debugger checks and execution checkpoints. Both hosts now pass the underlying AppKit provider and interpreter/native language-level desktop checks. Windows uses its -native Win32 provider; Linux uses GTK4. macOS editor integration and a complete -current SDK desktop campaign remain open. The newer socket, entropy and text-byte +native Win32 provider; Linux uses GTK4. Both Mac architectures pass all 88 real +editor groups at `9fb6c04`. Current SDK qualification and an Intel project-tool +timeout follow-up remain open. The newer socket, entropy and text-byte intrinsics pass the same interpreter contracts on Linux, Windows and both Macs. File timestamps share an implementation across interpreter and native routes, with text/binary NIL, JIT and AOT checks passing on all four hosts. Both Macs now pass native allocation-retention contracts, desktop debugger -sessions and native incremental reuse. Complete Mac editor coverage and current SDK -qualification remain open, so full platform parity is not yet established. +sessions and native incremental reuse. Full platform parity awaits those +remaining SDK and project-tool checks. The [platform support matrix](docs/platform-support.md) separates verified development behavior from remaining SDK, provider and release work. Windows and macOS release qualification remains pending. The diff --git a/docs/platform-support.md b/docs/platform-support.md index 2dc6e517..e2589af3 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -9,10 +9,10 @@ release boundary. A newer source checkout does not update an installed SDK. | Host | Verified development baseline | Remaining boundary | | --- | --- | --- | -| Linux x86_64 | Reference host. The clean `5630388` SDK passed installed consumer routes, shared runtime, debugger, Unicode/I/O, checkpoints and GTK checks. The newer `1efbe66` LLVM-ON source suite passes all 589 CTests without skips. | The reviewed Ubuntu release toolchain is separate from this development SDK. New release binaries still require the release gates. | +| Linux x86_64 | Reference host. The clean `5630388` SDK passed installed consumer routes, shared runtime, debugger, Unicode/I/O, checkpoints and GTK checks. The newer `9fb6c04` LLVM-ON source suite passes all 590 CTests without skips. | The reviewed Ubuntu release toolchain is separate from this development SDK. New release binaries still require the release gates. | | Windows x86_64 | The `5630388` installed SDK retains paired Linux acceptance. The newer private `f79b94b` source build passes 579 of 583 registered tests; separately pinned follow-ups resolve its fixture failure and execute the three skipped contracts. Real Win32 desktop and all 88 VS Code host groups pass. | The `8376252` debugger also passes six CTests and four real Win32 desktop sessions. Current-source SDK qualification remains open. Bare-host installation and Windows release qualification remain open. PostgreSQL requires an external runtime; the current UI provider is Win32. | -| macOS 15 Intel x86_64 | At `0c88d2f`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Earlier LLVM-ON/OFF, native assembly and embedded-context baselines also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse passes at `860750e`; full editor acceptance, a current-source SDK and release qualification remain open. | -| macOS 15 Apple Silicon arm64 | At `3414ce5`, the relocated development SDK passes its installed standard-library campaign, twelve isolated PostgreSQL source/binary routes, installed debugger/checkpoints and corrected numeric-address validation. Native assembly and embedded context also pass. | That SDK baseline has 567 passes, one failure and three skips. Newer source checks pass AppKit VM/native UI, DAP desktop, native lifetime, timestamp and ABI contracts on both Macs. Native incremental reuse passes at `860750e`; full editor acceptance, a current-source SDK and release qualification remain open. | +| macOS 15 Intel x86_64 | The relocated `1efbe66` SDK passes all 1,184 installed consumer routes, twelve PostgreSQL source/binary routes and installed debugger/checkpoints. At `9fb6c04`, all twenty selected platform CTests, four desktop DAP sessions and all 88 editor groups pass without exclusions. AppKit VM/native desktop/input checks pass separately at `5a98403`. | The `1efbe66` full source suite retains four failures and one ELF-only execution skip. Three fixture failures are resolved by later source checks; the tool-discovery fixture's first project run exceeds its 60-second child deadline and needs hosted follow-up. Release qualification remains open. | +| macOS 15 Apple Silicon arm64 | The relocated `1efbe66` SDK passes all 1,184 installed consumer routes, twelve PostgreSQL source/binary routes and installed debugger/checkpoints. At `9fb6c04`, all twenty selected platform CTests, four desktop DAP sessions and all 88 editor groups pass without exclusions. AppKit VM/native desktop/input checks pass separately at `5a98403`. | The `1efbe66` full source suite retains four failures and one ELF-only execution skip. Later source checks resolve its four fixture failures; they supplement the unchanged SDK rather than rewrite its original report. Release qualification remains open. | LLVM-OFF builds exercise frontend/interpreter behavior and rejection of disabled native operations. Their successful tests do not establish native execution. @@ -147,7 +147,7 @@ operations are rejected before capture; this is not arbitrary application rollback. See [NIL-0006](design/NIL-0006-ai-module.md) for commands, limits and the complete admission rules. -## Remaining macOS work +## macOS coverage and remaining work - Native incremental reuse passes on both Macs at `860750e`. The compiler resolves loaded images to ordinary files and exact shared-cache backing files for full @@ -155,7 +155,8 @@ the complete admission rules. The builder uses separate preparation and verification digest phases without caching source/manifest reads. Both actual linker-dependency probes collect 8,321 inputs, and both real VM/native reuse and invalidation campaigns pass. - A subsequent SDK archive-fallback refinement still needs hosted qualification. + The subsequent SDK archive-fallback refinement also passes both hosts at + `1efbe66`, together with all sixteen native, debugger and incremental contracts. Incomplete or unsupported dependency profiles deliberately rebuild. - A separate [development SDK workflow](../.github/workflows/macos-sdk.yml) now implements native Darwin consumer admission, root install layout, pinned diff --git a/docs/release/v1-profile.md b/docs/release/v1-profile.md index c13d049f..ef6f3c3b 100644 --- a/docs/release/v1-profile.md +++ b/docs/release/v1-profile.md @@ -93,15 +93,18 @@ not extend the v1 language or debugger guarantees. The LSP has a deliberately limited capability set. `nilbuild` supports the [local-project profile](../tools/projects.md), with process execution qualified on Linux. `nypm` and `nilaix` are not part of this release. -Sixteen logical standard-library modules are strictly compiled and packaged, -including `prelude`. Public imports use `nil.std.*`, with `ny.lang.*` retained -as aliases. Both frontends share the flat module-family layout. Runtime support +Current development SDKs strictly compile and package eighteen logical +standard-library modules, including `prelude`. Public imports use `nil.std.*`, +with `ny.lang.*` retained as aliases. Both frontends share the flat module-family +layout. Runtime support is API-specific, as described in the [library status](../standard-library/index.md). `nil.std.reflection` provides declaration/annotation snapshots, and `nil.std.db` provides bounded SQLite/PostgreSQL text/NULL operations through shared VM/native -providers. Complete semantic reference graphs, automatically hosted analyzer plugins, -typed/binary database values, network transports, cryptography, compression, -and general stream handles remain outside these implemented slices. +providers. Bounded TCP sockets and OS entropy are implemented for the interpreter; +their binary-text values do not yet have native/JIT support. Complete semantic +reference graphs, automatically hosted analyzer plugins, typed/binary database +values, TLS/general cryptography, compression and general stream handles remain +outside these implemented slices. Quarantined modules are not promoted by their presence in a design or by a module compiling successfully. diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 2ece137d..ce62309a 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -1,12 +1,12 @@ # Web runtime and platform parity — 2026-11-11 -The web handoff resumes at framework `dfb7ad7` on `nil-sockets-entropy-bytes` +The web handoff began at framework `dfb7ad7` on `nil-sockets-entropy-bytes` ([PR 38](https://github.com/exec-software/exec-nil-framework/pull/38)) and web `594579a` plus preserved local changes. The archived originals are under `atoma/archived-exec-ny-web` and `atoma/archived-exec-ny-webview`. Platform work comes first; the web changes are not discarded, reset or overwritten. -## Current baseline and remaining boundaries +## Starting baseline and boundaries Windows now uses Win32 and macOS uses AppKit; Linux uses GTK4. November 9 native AppKit provider and interpreter display results supersede the earlier private @@ -24,7 +24,7 @@ current native application/editor acceptance remain open. The platform/readme documentation now separates current source implementation from the recorded SDK and route baselines. -The latest source validation push run +At recovery, source validation push run [43095105713](https://github.com/exec-software/exec-nil-framework/actions/runs/43095105713) failed before any runner executed: GitHub reports an internal server error, correlation ID `0fbc4710-ecca-4198-a05c-f35c1d8d4bef`. It supplies no product @@ -512,6 +512,92 @@ CTests pass in 12.99 seconds; actual hosted follow-up remains required. Full SDK workflows also upload their source inventory immediately, before lengthy installed campaigns, so failed gates are visible without waiting for packaging. +## Source checkpoint at `9fb6c04` + +At `9fb6c04`, Linux's complete LLVM-ON suite passes all 590 registrations +without skips in 258.36 seconds; LLVM-OFF and extension jobs also pass +([source run 43135334607](https://github.com/exec-software/exec-nil-framework/actions/runs/43135334607)). +The raw log independently yields 590 unique passing test names, including the +new freestanding archive-import contract. The tested PR merge is +`94979957a693bf725211001cc1434c14c1fbef3c`, distinct from the feature head. + +The matching [web-runtime run 43135334608](https://github.com/exec-software/exec-nil-framework/actions/runs/43135334608) +passes the exact six interpreter contracts on Linux, Windows and both Macs, +with no failures or skips. All 148 retained evidence files were independently +hash-verified, together with unchanged source identity, exact JUnit inventories +and the retained timestamp command/cleanup reports. These focused checks do +not qualify native socket support or replace the installed SDK campaigns. + +Both architecture jobs in +[application run 43135334578](https://github.com/exec-software/exec-nil-framework/actions/runs/43135334578) +pass all twenty exact CTests, four owned desktop DAP sessions each and the complete +VS Code 1.146.0 host: 88 unique passing groups, all nine modules, no exclusions +and the full-suite completion sentinel on each host. All 620 evidence files +match their manifest hashes and the source remains unchanged. The formerly failing native +memory-profile groups now pass with the adjacent `nillinker` build target. +The final independent replay is `github-applications-43135334578/replay-report.json`, +SHA-256 `c362343a73c2adc1a567113a4a0a718306f72889fade367897625128775eb129`. + +## Windows executable replacement + +The fresh `9fb6c04` Windows suite completes all 589 registrations with +586 passes, two headless desktop skips and a different failure in +`ny.global-contract`: after running `control.exe`, the next successful native +link cannot replace it because `MoveFileExW` reports access denied. The fixture +subsequently cleans its directory successfully; the particular lock holder was +not captured, so it is not attributed to a specific process or scanner. + +The compiler already retries access/sharing/lock failures up to fifty times +with a 20 ms delay, but the standalone linker made only one attempt. Both now +use the same helper and preserve the existing output on persistent failure. +The helper neither removes the destination first nor changes its permissions. +This matches Windows' documented +[file-sharing contract](https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew): +an open handle without delete sharing prevents rename until that handle closes. +Four relevant Linux CTests pass after the refactor. The first local invocation +had one unbuilt process-fixture executable; that incomplete result is retained +alongside the complete rerun after building it. + +At `9fa354d`, strict MSVC (`/W4 /WX /O2`) passes all 46 checks against real +Windows handles: temporary and permanent sharing failures, read-only output, +missing source, exact binary payload preservation, error clearing and cleanup. +The single-attempt mutant fails precisely at the transient-release success +assertion. Eight raw streams and cleanup were independently verified under +`atomic-publication-v1/replay.json`; the original outer wrapper's +`released`/`release` diagnostic wording mismatch is preserved and reconciled +against those native exit results. The clean `9fa354d` compiler/linker build +then passes four exact focused CTests: atomic replacement, global contracts, +the native object pipeline and host-object linking. Their raw reports and +cleanup are verified in `sdk-9fa354d48575/replay-publication-focused.json`. +A corrected complete SDK still needs qualification. + +## Intel discovery deadline and SDK build limits + +The completed Intel `1efbe66` SDK also records 583 passes, four failures and +one ELF-only execution skip from 588 registrations. Its AppKit desktop tests +pass; the fourth failure instead comes from tool discovery. The first +adjacent-tool `nilbuild check` succeeds, then the first VM `nilbuild run` +exceeds the fixture's 60-second child deadline. This is not its 180-second +CTest limit or the workflow's 35-minute suite limit. The older failure did +not retain the child's partial output, so its internal active phase is unknown. + +Only the discovery fixture's two real Darwin project runs now receive the +same 180-second VM allowance as the project/editor fixtures. Other commands +retain 60 seconds. Timings and captured timeout output are retained, with a +600-second Darwin aggregate CTest guard. The Mac application workflow admits +discovery as its fifth SDK contract, for 21 total platform contracts, with a +bounded 30-minute combined step. The actual Linux discovery contract passes +all 21 checks; hosted Darwin follow-up remains required. + +The Linux SDK builder also passed a bare `--parallel` to CMake, overriding +`CMAKE_BUILD_PARALLEL_LEVEL=4`. A private empty-project control demonstrates +Make receives unbounded `-j` with that option and `-j4` without it. Removing +the bare option restores the documented caller-controlled build limit. +Both actual driver commands and raw results are retained under +`sdk-parallel-control`. Interrupted local build logs remain preserved; resumed +qualification uses a bounded transient user service and does not replace an +accepted SDK. + ## SQL provider environment scope The full Mac source/SDK workflows previously exported SQL's @@ -532,7 +618,7 @@ Actual Mac source and SDK campaigns must confirm the new provider scope. ## Preserved web handoff -The interrupted web tree contains the listener-to-engine replacement, template +The initial interrupted web snapshot contained the listener-to-engine replacement, template cache changes, static-file changes, status/main/test edits and four new template fixtures. Its existing 330 Matrix checks pass. Private, bounded loopback checks against copies of that source exposed gaps the current unit suite misses: @@ -545,6 +631,15 @@ against copies of that source exposed gaps the current unit suite misses: that directory. Root-constrained file access needs an actual filesystem boundary. All diagnostic servers were owned, stopped and reaped. The web source was not -changed. Its README still describes 272 checks and the deleted serial listener; -CI audits source paths but does not execute its runtime suite. These are the next -web tasks after platform parity work, not evidence of IIS/Apache equivalence. +changed by this audit. At that snapshot its README still described 272 checks +and the deleted serial listener; CI audited source paths without executing its +runtime suite. These are historical observations, not evidence of IIS/Apache +equivalence. + +A subsequent read-only check found separate work had advanced the web repository +to a clean `41eb756`: `9183727` commits the event loop, `c175582` adds logging and +HTTP dates, and `41eb756` adds configuration and virtual hosts. Its README now +reports 653 checks. Those changes are preserved; the older 330-check snapshot +and its reproductions must be checked against this new source before making +further web changes. The README's remaining-parity paragraph still lists logs, +configuration and virtual hosts as absent despite documenting them elsewhere. diff --git a/scripts/build-sdk.sh b/scripts/build-sdk.sh index a7a03e2e..e7778c58 100755 --- a/scripts/build-sdk.sh +++ b/scripts/build-sdk.sh @@ -104,7 +104,9 @@ cmake \ -DBUILD_TESTING="${run_tests}" \ -DCMAKE_INSTALL_BINDIR=. -cmake --build "${cmake_build_dir}" --parallel +# Let CMake honor CMAKE_BUILD_PARALLEL_LEVEL. A bare --parallel overrides that +# limit and asks Make for an unbounded -j build. +cmake --build "${cmake_build_dir}" evidence_path="${cmake_build_dir}/sdk-build-evidence.json" ctest_report="${cmake_build_dir}/sdk-ctest.xml" diff --git a/tests/tools/tool_discovery_contract.py b/tests/tools/tool_discovery_contract.py index e736254b..efeb259e 100644 --- a/tests/tools/tool_discovery_contract.py +++ b/tests/tools/tool_discovery_contract.py @@ -7,7 +7,9 @@ import os from pathlib import Path import subprocess +import sys import tempfile +import time def main() -> None: @@ -51,11 +53,29 @@ def main() -> None: path_environment = {**environment, 'PATH': os.pathsep.join( [str(work / 'missing tool directory'), str(compiler.parent), *retained_path])} - def run(command, *, env=adjacent_environment, status=0, executable=None): + def run(command, *, env=adjacent_environment, status=0, executable=None, timeout=60): nonlocal checks - result = subprocess.run(list(map(str, command)), cwd=work, env=env, - stdin=subprocess.DEVNULL, capture_output=True, - text=True, timeout=60, executable=executable) + argv = list(map(str, command)) + phase = f'{checks + 1}: {Path(argv[0]).name} {argv[1]}' + print(f'tool discovery phase {phase}: starting, timeout={timeout}s', flush=True) + started = time.monotonic() + try: + result = subprocess.run(argv, cwd=work, env=env, + stdin=subprocess.DEVNULL, capture_output=True, + text=True, timeout=timeout, executable=executable) + except subprocess.TimeoutExpired as error: + # TimeoutExpired may carry bytes even when text=True. Preserve + # captured diagnostics before the traceback loses those fields. + def captured(value): + return value.decode('utf-8', errors='replace') if isinstance(value, bytes) else value or '' + print(json.dumps({'phase': phase, 'command': argv, + 'elapsed_seconds': time.monotonic() - started, + 'timeout_seconds': timeout, + 'stdout': captured(error.stdout), + 'stderr': captured(error.stderr)}), file=sys.stderr, flush=True) + raise + print(f'tool discovery phase {phase}: exit={result.returncode}, ' + f'elapsed={time.monotonic() - started:.2f}s', flush=True) assert result.returncode == status, (command, result.returncode, result.stdout, result.stderr) checks += 1 @@ -68,7 +88,11 @@ def verify_build_tools(build_command, flags=(), *, env=adjacent_environment): checked = run([build_command, 'check', *build_options, '--format', 'json', *flags], env=env) assert json.loads(checked.stdout)['success'] - ran = run([build_command, 'run', *build_options, *flags], env=env) + # Real Darwin project runs fingerprint the complete toolchain before + # executing the VM. Use the measured project VM budget for only + # these two calls; discovery/check/error paths retain sixty seconds. + ran = run([build_command, 'run', *build_options, *flags], env=env, + timeout=180 if sys.platform == 'darwin' else 60) assert ran.stdout == 'discovered application\n', ran tested = run([build_command, 'test', *build_options, '--format', 'json', *flags], env=env) From 285aea25c74d95e12cd7cb124e834ad8f31a78d9 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 22:11:39 -0500 Subject: [PATCH 24/39] Qualify installed macOS UI and editor from sealed SDKs --- .github/workflows/macos-sdk-replay.yml | 88 ++++++++- scripts/ci/macos-desktop.py | 241 +++++++++++++++++++++++++ scripts/ci/macos-sdk-replay.py | 31 +++- scripts/ci/test_macos_desktop.py | 169 +++++++++++++++++ 4 files changed, 519 insertions(+), 10 deletions(-) create mode 100644 scripts/ci/macos-desktop.py create mode 100644 scripts/ci/test_macos_desktop.py diff --git a/.github/workflows/macos-sdk-replay.yml b/.github/workflows/macos-sdk-replay.yml index bf46252f..6be167fc 100644 --- a/.github/workflows/macos-sdk-replay.yml +++ b/.github/workflows/macos-sdk-replay.yml @@ -1,4 +1,4 @@ -name: macOS installed debugger replay +name: macOS installed SDK replay "on": workflow_dispatch: @@ -16,6 +16,10 @@ name: macOS installed debugger replay type: choice options: [both, x86_64, arm64] default: both + desktop: + description: Also accept installed nilstd/nillib UI, desktop DAP and the complete pinned editor + type: boolean + default: false permissions: contents: read @@ -29,9 +33,9 @@ concurrency: # This never rebuilds, activates, publishes or release-qualifies an SDK. jobs: replay: - name: macOS ${{ matrix.host.arch }} installed debugger + name: macOS ${{ matrix.host.arch }} installed SDK acceptance runs-on: ${{ matrix.host.runner }} - timeout-minutes: 60 + timeout-minutes: ${{ inputs.desktop && 115 || 60 }} strategy: fail-fast: false max-parallel: 2 @@ -58,6 +62,60 @@ jobs: set -o pipefail python3 -B scripts/ci/test_macos_sdk_replay.py -v 2>&1 | tee build/macos-replay-evidence/replay-controls.log python3 -B scripts/ci/test_macos_debugger.py -v 2>&1 | tee build/macos-replay-evidence/debugger-controls.log + python3 -B scripts/ci/test_macos_desktop.py -v 2>&1 | tee build/macos-replay-evidence/desktop-controls.log + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 + if: inputs.desktop + with: + node-version: '22' + - name: Build only the three test-only desktop helpers + if: inputs.desktop + timeout-minutes: 5 + run: | + set -o pipefail + cmake -S . -B build/desktop-helpers -DCMAKE_BUILD_TYPE=Release -DNIL_ENABLE_LLVM=OFF -DBUILD_TESTING=ON \ + 2>&1 | tee build/macos-replay-evidence/helpers-configure.log + cmake --build build/desktop-helpers --parallel 2 --target \ + nil_ui_appkit_input_driver nil_ui_appkit_compile_driver nil_vscode_host_runner \ + 2>&1 | tee build/macos-replay-evidence/helpers-build.log + cp build/desktop-helpers/CMakeCache.txt build/macos-replay-evidence/helpers-CMakeCache.txt + - name: Prepare pinned VS Code for installed SDK acceptance + if: inputs.desktop + timeout-minutes: 5 + run: | + mkdir -p build/editor-download + if [[ "$NIL_CI_ARCH" == arm64 ]]; then + export NIL_EDITOR_ARCHIVE=VSCode-darwin-arm64.zip + export NIL_EDITOR_SHA256=6e8d27ee0c5a8f2aa3c43e373407eecd11c270c3fe7f46100ef682aac14bb648 + else + export NIL_EDITOR_ARCHIVE=VSCode-darwin.zip + export NIL_EDITOR_SHA256=74b1deef1e33cc144d913d60c0e98cef6b1e29c1f2993d905c3b58576fa12202 + fi + curl --fail --location --max-time 120 --max-filesize 800000000 \ + --retry 2 --retry-all-errors --retry-delay 2 --retry-max-time 180 \ + "https://vscode.download.prss.microsoft.com/dbazure/download/stable/69047191fc427f2f92828279aae1d90d8eeda7ab/$NIL_EDITOR_ARCHIVE" \ + -o build/editor-download/editor.zip + python3 -B - <<'PY' + import hashlib, json, os + from pathlib import Path + archive = Path('build/editor-download/editor.zip') + digest = hashlib.sha256() + with archive.open('rb') as stream: + while block := stream.read(1024**2): digest.update(block) + assert digest.hexdigest() == os.environ['NIL_EDITOR_SHA256'] + Path('build/macos-replay-evidence/editor.json').write_text(json.dumps(dict( + version='1.146.0', revision='69047191fc427f2f92828279aae1d90d8eeda7ab', + archive=os.environ['NIL_EDITOR_ARCHIVE'], sha256=digest.hexdigest(), bytes=archive.stat().st_size), indent=2)+'\n') + PY + ditto -x -k build/editor-download/editor.zip build/editor-download + - name: Build and check current fixture extension + if: inputs.desktop + timeout-minutes: 5 + working-directory: vscode + run: | + set -o pipefail + npm ci 2>&1 | tee ../build/macos-replay-evidence/extension-install.log + npm test 2>&1 | tee ../build/macos-replay-evidence/extension-controls.log + npm run build 2>&1 | tee ../build/macos-replay-evidence/extension-build.log - name: Retrieve exact SDK artifact and authenticated metadata timeout-minutes: 7 env: @@ -121,15 +179,33 @@ jobs: (evidence / 'transport.json').write_text(json.dumps({'bytes': archive.stat().st_size, 'sha256': digest.hexdigest(), 'artifact_id': artifact['id'], 'source_run': int(source_run)}, indent=2) + '\n') PY - - name: Replay sealed installed debugger and execution checkpoints - timeout-minutes: 37 + - name: Replay sealed installed SDK acceptance + timeout-minutes: ${{ inputs.desktop && 79 || 37 }} + env: + NIL_REPLAY_DESKTOP: ${{ inputs.desktop }} run: | set -o pipefail + desktop_args=() + if [[ "$NIL_REPLAY_DESKTOP" == true ]]; then + editor="$(python3 -B - <<'PY' + import plistlib + from pathlib import Path + app = Path('build/editor-download/Visual Studio Code.app/Contents').resolve(strict=True) + with (app / 'Info.plist').open('rb') as stream: name = plistlib.load(stream)['CFBundleExecutable'] + assert name and Path(name).name == name + executable = app / 'MacOS' / name + assert executable.is_file() + print(executable) + PY + )" + desktop_args=(--desktop --desktop-helpers "$GITHUB_WORKSPACE/build/desktop-helpers" + --editor "$editor" --node "$(command -v node)") + fi python3 -B scripts/ci/macos-sdk-replay.py \ --artifact-zip "$GITHUB_WORKSPACE/build/macos-replay-evidence/source-artifact.zip" \ --artifact-metadata "$GITHUB_WORKSPACE/build/macos-replay-evidence/artifact-metadata.json" \ --sdk-revision "$NIL_SDK_REVISION" --helper-revision "$GITHUB_SHA" --architecture "$NIL_CI_ARCH" \ - --work-dir "$RUNNER_TEMP/nil-sdk-replay" \ + --work-dir "$RUNNER_TEMP/nil-sdk-replay" "${desktop_args[@]}" \ --output "$GITHUB_WORKSPACE/build/macos-replay-evidence/replay" \ 2>&1 | tee build/macos-replay-evidence/replay.log - name: Seal complete or partial replay evidence diff --git a/scripts/ci/macos-desktop.py b/scripts/ci/macos-desktop.py new file mode 100644 index 00000000..968ca474 --- /dev/null +++ b/scripts/ci/macos-desktop.py @@ -0,0 +1,241 @@ +#!/usr/bin/env python3 +"""Accept installed macOS UI, desktop DAP and the complete editor; no SDK build.""" +from __future__ import annotations + +import argparse +import hashlib +import importlib.util +import json +from pathlib import Path +import platform +import re +import shutil +import sys +import time + +ROOT = Path(__file__).resolve().parents[2] +TOTAL_SECONDS = 2400 +MODULES = ['analyzer-checks', 'project-checks', 'workspace-checks', 'semantic-checks', + 'completion-checks', 'member-signature-checks', 'language-values-checks', + 'memory-profile-checks', 'niltest-checks'] +EDITOR_NAMES = '9374544bc95fde9f58f9400444aa734ecdd0f4f16575a6042e39eef46ab2583d' +TOOL_SETTINGS = {'server': 'nillsp', 'compiler': 'nilcompiler', 'interpreter': 'nilinterpreter', + 'debugger': 'nildebugger', 'projectTool': 'nilbuild', 'testTool': 'niltest', 'aperture': 'aperture'} + + +def require(value, message): + if not value: + raise ValueError(message) + + +def load(relative, name): + spec = importlib.util.spec_from_file_location(name, ROOT / relative) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def ui_command(sdk, library, helpers, evidence): + require(library in ('nilstd', 'nillib'), 'Unknown installed library root') + return [sys.executable, '-B', str(ROOT / 'tests/tools/ui_display_contract.py'), + '--tester', str(sdk / 'niltest'), '--compiler', str(sdk / 'nilcompiler'), + '--interpreter', str(sdk / 'nilinterpreter'), '--sdk', str(sdk / library), + '--source', str(ROOT / 'tests/ui/gtk_desktop.matrix'), + '--interaction-source', str(ROOT / 'tests/ui/gtk_interaction.matrix'), '--llvm', 'ON', + '--appkit-input-driver', str(helpers / 'lib/libnil_ui_appkit_input_driver.dylib'), + '--appkit-compile-driver', str(helpers / 'bin/nil_ui_appkit_compile_driver'), + '--appkit-evidence-directory', str(evidence)] + + +def objects(text): + """Decode whole top-level JSON documents among the harness's human lines.""" + decoder, position = json.JSONDecoder(), 0 + while position < len(text): + if text[position] == '{' and (position == 0 or text[position - 1] == '\n'): + value, end = decoder.raw_decode(text, position) + yield value + position = end + else: + position += 1 + + +def audit_ui(stdout, directory, sdk, helpers, common, deadline): + ui = load('tests/tools/ui_display_contract.py', 'installed_ui_contract') + reports = [r for r in objects(stdout) if r.get('schema') == 'nil.test.results.v1'] + require(len(reports) == 3, 'Expected one desktop and two interaction reports') + ui.validate_desktop_report(reports[0], 'ON') + paths = list(directory.glob('nil-appkit-input-*/report.json')) + require(len(paths) == 1, 'Expected one owned AppKit interaction directory') + root = paths[0].parent + summary = common.read_json(paths[0]) + require(summary['schema'] == 'nil.appkit.interaction.v1' and summary['status'] == 'passed' and + summary['compiler'] == str(sdk / 'nilcompiler') and summary['evidenceDirectory'] == str(root) and + summary['source'] == str(ROOT / 'tests/ui/gtk_interaction.matrix') and + summary['sourceSha256'] == common.record(ROOT / 'tests/ui/gtk_interaction.matrix', deadline)['sha256'], + 'Installed UI compiler/source binding differs') + for key, path in (('inputDriver', helpers / 'lib/libnil_ui_appkit_input_driver.dylib'), + ('compileDriver', helpers / 'bin/nil_ui_appkit_compile_driver')): + require(summary[key] == str(path) and summary[key + 'Sha256'] == common.record(path, deadline)['sha256'], + 'AppKit test-only helper binding differs') + require([r['route'] for r in summary['routes']] == ['vm', 'native'], 'Incomplete UI route inventory') + for index, row in enumerate(summary['routes'], 1): + route = row['route'] + require(row['naturalCompletion'] is True and row['languageCases'] == 2 and len(row['inputs']) == 2, + 'UI route did not complete naturally') + raw = common.read_json(root / (route + '.stdout')) + require(raw == reports[index], 'Interaction stdout differs from retained report') + records = ui.validate_appkit_interaction(raw, route, root / route, row['inputs'][0]['token'], row['parentPid']) + require(records == row['inputs'], 'Interaction responder records differ') + return {'desktop_cases': 6, 'interaction_cases': 4, 'total_cases': 10, + 'report': common.record(paths[0], deadline)} + + +def audit_dap(report, sdk, common, deadline): + expected = [encoding + '-' + action for encoding in ('text', 'binary') for action in ('retention', 'disconnect')] + require(report['schema'] == 'nil.dap-desktop.v1' and report['status'] == 'PASS' and + [row['name'] for row in report['sessions']] == expected, 'Incomplete desktop DAP inventory') + require(report['tools'] == {str(sdk / name): common.record(sdk / name, deadline)['sha256'] + for name in ('nildebugger', 'nilcompiler')}, 'DAP used different installed tools') + for row in report['sessions']: + require(row['status'] == 'PASS' and row['natural_exit'] is True and row['stdin_open_at_exit'] is True and + row['cleanup_verified'] is True and row['exit_code'] == 0, 'Desktop DAP ownership/shutdown failed') + if row['name'].endswith('-retention'): + require(row['queued_event_retained'] is True, 'Desktop DAP lost its queued event') + require(len(report['commands']) == 2, 'Expected both desktop DAP encodings') + for index, row in enumerate(report['commands']): + argv = row['argv'] + require(row['exit_code'] == 0 and row['cleanup_verified'] is True and argv[0] == str(sdk / 'nilcompiler') and + argv[argv.index('--sdk-path') + 1] == str(sdk / 'nilstd') and + ('--emit-binary-nil' in argv) == bool(index), 'DAP compile SDK/encoding differs') + return {'sessions': 4, 'names': expected} + + +def audit_editor(report, launch, settings, sdk): + require(report['status'] == 'PASS' and report['platform'] == 'darwin' and report['vscode'] == '1.146.0' and + report['skippedChecks'] == [] and report['fullSuiteCompleted'] is True and + report['completedModules'] == MODULES, 'Incomplete pinned editor suite') + names = [row['name'] for row in report['checks']] + require(len(names) == len(set(names)) == 88 and all(row['status'] == 'PASS' for row in report['checks']) and + hashlib.sha256(json.dumps(sorted(names), separators=(',', ':'), ensure_ascii=False).encode()).hexdigest() == EDITOR_NAMES, + 'Editor named check inventory differs') + require(report['runId'] == launch['runId'] and report['workspace'] == launch['workspace'], 'Stale editor result') + require(settings['ny.sdkPath'] == str(sdk) and + all(settings['ny.' + key + '.path'] == str(sdk / tool) for key, tool in TOOL_SETTINGS.items()), + 'Editor configuration fell back to source-built tools/stdlib') + return {'groups': 88, 'modules': MODULES, 'version': '1.146.0', 'run_id': report['runId']} + + +def editor_launch(launch): + temporary = Path(launch['temporary']) + require(launch['platform'] == 'darwin' and temporary.parent == Path('/tmp') and + temporary.name.startswith('nil vscode host café 中文-') and not temporary.is_symlink() and + launch['profile'] == str(temporary / 'profile') and launch['workspace'] == str(temporary / 'workspace'), + 'Editor escaped its isolated profile/workspace') + require(re.fullmatch('[0-9a-f-]{36}', launch['runId']), 'Invalid editor run identifier') + return temporary + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + for name in ('sdk', 'origin', 'output', 'helpers', 'editor', 'node'): + parser.add_argument('--' + name, type=Path, required=True) + parser.add_argument('--expected-revision', required=True) + args = parser.parse_args() + require(sys.platform == 'darwin' and platform.machine() in ('arm64', 'x86_64'), 'Native macOS required') + common = load('scripts/ci/macos-debugger.py', 'installed_desktop_common') + sdk, origin, output = common.paths(args.sdk, args.origin, args.output) + helpers, editor, node = (path.resolve(strict=True) for path in (args.helpers, args.editor, args.node)) + output.mkdir(parents=True) + started = time.monotonic(); deadline = started + TOTAL_SECONDS + report = {'schema': 'nil.macos-installed-desktop.v1', 'status': 'FAIL', 'problems': [], 'phases': [], + 'sdk': str(sdk), 'expected_revision': args.expected_revision, + 'host': {'system': platform.system(), 'machine': platform.machine()}, + 'scope': 'Installed nilstd/nillib VM/native UI, four desktop DAP sessions and full editor; ' + 'source provider/lifecycle-depth tests remain separate; no Accessibility, SDK activation or release qualification.'} + before = seal = anchor = None + editor_reports = ROOT / 'build/vscode-host-smoke' + try: + require(not editor_reports.exists(), 'Editor reports must be fresh, never reused') + require((sdk / 'nilstd/ui/package.matrix').is_file(), 'Installed flat nilstd UI package is missing') + require((sdk / 'nillib/ui/package.nil').is_file(), 'Installed binary UI package is missing') + inputs = [node, editor, helpers / 'bin/nil_vscode_host_runner', + helpers / 'bin/nil_ui_appkit_compile_driver', helpers / 'lib/libnil_ui_appkit_input_driver.dylib', + ROOT / 'vscode/package.json', ROOT / 'vscode/dist/extension.js'] + before = {str(path): common.record(path, deadline) for path in inputs} + seal, anchor = common.inventory(sdk, deadline), common.record(origin, deadline) + verify = common.load(common.PROJECT / 'scripts/sdk_integrity.py', 'installed_desktop_integrity').verify + state = verify(sdk, origin) + common.verify_origin(state, sdk, args.expected_revision, platform.machine()) + report.update(inputs_before=before, sdk_before=seal, origin_before=anchor, sdk_integrity=state) + capture = load('tests/tools/ordinary_io_parity_contract.py', 'installed_desktop_capture') + owner = capture.load_ownership(ROOT) + environment = load('scripts/check-sdk-routes.py', 'installed_desktop_environment').consumer_environment() + for name in ('GH_TOKEN', 'GITHUB_TOKEN', 'ACTIONS_RUNTIME_TOKEN'): + environment.pop(name, None) + + def phase(name, argv, seconds, env=environment): + require(deadline - time.monotonic() > seconds + 20, 'Insufficient remaining desktop deadline') + row = capture.run_process(owner, list(map(str, argv)), output, env, output, name, seconds) + report['phases'].append(row) + require(row['status'] == 'PASS' and row['cleanup_verified'], 'Installed phase failed: ' + name) + return (output / row['stdout']).read_text(encoding='utf-8') + + report['ui'] = {} + for library in ('nilstd', 'nillib'): + directory = output / ('ui-' + library) + stdout = phase('ui-' + library, ui_command(sdk, library, helpers, directory), 450) + report['ui'][library] = audit_ui(stdout, directory, sdk, helpers, common, deadline) + directory = output / 'desktop-dap'; directory.mkdir() + phase('desktop-dap', [sys.executable, '-B', ROOT / 'nildebugger/tests/dap_desktop.py', + '--debugger', sdk / 'nildebugger', '--compiler', sdk / 'nilcompiler', + '--nilstd', sdk / 'nilstd', '--output-parent', directory], 300) + paths = list(directory.glob('nil-dap-desktop-*/report.json')) + require(len(paths) == 1, 'Expected one fresh desktop DAP result') + report['desktop_dap'] = audit_dap(common.read_json(paths[0]), sdk, common, deadline) + editor_environment = dict(environment, VSCODE_EXECUTABLE=str(editor), NIL_VSCODE_TOOL_DIRECTORY=str(sdk), + NIL_VSCODE_SDK_PATH=str(sdk), NIL_VSCODE_HOST_RUNNER=str(helpers / 'bin/nil_vscode_host_runner'), + NIL_VSCODE_HOST_TIMEOUT_MS='900000') + report['editor_environment'] = {key: value for key, value in editor_environment.items() + if key.startswith('NIL_VSCODE_') or key == 'VSCODE_EXECUTABLE'} + phase('editor', [node, ROOT / 'vscode/test/run-host.cjs'], 960, editor_environment) + launches = list(editor_reports.glob('*-launch.json')) + require(len(launches) == 1, 'Expected one fresh editor launch') + launch = common.read_json(launches[0]); temporary = editor_launch(launch) + settings = common.read_json(temporary / 'workspace/.vscode/settings.json') + result = common.read_json(editor_reports / (launch['runId'] + '.json')) + require(result == common.read_json(editor_reports / 'latest.json'), 'Editor latest report differs') + report['editor'] = audit_editor(result, launch, settings, sdk) + common.write_json(output / 'editor-settings.json', settings) + report['status'] = 'PASS' + except Exception as error: + report['problems'].append(type(error).__name__ + ': ' + str(error)) + finally: + try: + if editor_reports.is_dir(): + common.inventory(editor_reports, deadline) + shutil.copytree(editor_reports, output / 'editor-reports') + for path in editor_reports.glob('*-launch.json'): + launch = common.read_json(path); temporary = editor_launch(launch) + logs = temporary / 'profile/logs' + if logs.is_dir(): + common.inventory(logs, deadline) + shutil.copytree(logs, output / 'editor-logs' / launch['runId']) + require(before is not None and before == {path: common.record(Path(path), deadline) for path in before}, + 'Test helper/editor/node input changed') + require(seal is not None and common.inventory(sdk, deadline) == seal, 'Installed SDK changed') + require(anchor is not None and common.record(origin, deadline) == anchor, 'SDK origin changed') + report['inputs_unchanged'] = True + report['evidence_files'] = common.inventory(output, deadline) + except Exception as error: + report['problems'].append(type(error).__name__ + ': ' + str(error)) + report['cleanup_verified'] = len(report['phases']) == 4 and all(row['cleanup_verified'] for row in report['phases']) + report['elapsed_seconds'] = time.monotonic() - started + if report['problems'] or not report['cleanup_verified'] or report['elapsed_seconds'] > TOTAL_SECONDS: + report['status'] = 'FAIL' + common.write_json(output / 'report.json', report) + print(json.dumps({key: report[key] for key in ('status', 'expected_revision', 'problems')})) + return 0 if report['status'] == 'PASS' else 1 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/scripts/ci/macos-sdk-replay.py b/scripts/ci/macos-sdk-replay.py index c6007d6c..ba365ec5 100644 --- a/scripts/ci/macos-sdk-replay.py +++ b/scripts/ci/macos-sdk-replay.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Replay installed debugger/checkpoints from a sealed SDK; never build or publish it.""" +"""Replay installed acceptance from a sealed SDK; never build or publish it.""" from __future__ import annotations import argparse import hashlib @@ -323,18 +323,28 @@ def main(): for name in ('sdk-revision', 'helper-revision'): parser.add_argument('--' + name, required=True) parser.add_argument('--architecture', choices=('arm64', 'x86_64'), required=True) + parser.add_argument('--desktop', action='store_true', help='Also accept both installed UI roots, desktop DAP and full editor') + for name in ('desktop-helpers', 'editor', 'node'): + parser.add_argument('--' + name, type=Path) args = parser.parse_args() + require(all((args.desktop_helpers, args.editor, args.node)) == args.desktop and + (args.desktop or not any((args.desktop_helpers, args.editor, args.node))), + 'Desktop replay requires all three explicit helper/editor/node inputs') require(sys.platform == 'darwin' and platform.machine() == args.architecture, 'Native matching macOS host required') for value in (args.sdk_revision, args.helper_revision): require(re.fullmatch('[0-9a-f]{40}', value), 'Exact source revision required') require(not args.artifact_zip.is_symlink() and not args.artifact_metadata.is_symlink(), 'Artifact inputs may not be links') archive, metadata_path = args.artifact_zip.resolve(strict=True), args.artifact_metadata.resolve(strict=True) output, work = fresh_directories(args.output, args.work_dir, (archive, metadata_path)) - started = time.monotonic(); deadline = started + TOTAL_SECONDS + total_seconds = TOTAL_SECONDS + (2500 if args.desktop else 0) + started = time.monotonic(); deadline = started + total_seconds report = {'schema': 'nil.macos-sdk-debugger-replay.v1', 'status': 'FAIL', 'problems': [], 'phases': [], 'sdk_revision': args.sdk_revision, 'helper_revision': args.helper_revision, 'architecture': args.architecture, 'release_declaration': False, 'platform_parity': False, - 'scope': 'Installed debugger and same-process execution checkpoints only; no SDK build, activation or provider qualification'} + 'desktop_requested': args.desktop, 'whole_seconds': total_seconds, + 'scope': ('Installed debugger and same-process execution checkpoints' + + ('; installed nilstd/nillib UI, desktop DAP and full editor' if args.desktop else ' only') + + '; no SDK build, activation, source provider/lifecycle-depth or release qualification')} before = payload = original = dependencies = None sdk, copied = work / 'SDK replay λ', work / 'artifact' sdk.mkdir(); copied.mkdir() @@ -367,6 +377,19 @@ def main(): require(acceptance['status'] == 'PASS' and acceptance['expected_revision'] == args.sdk_revision, 'Debugger result is not bound to the selected SDK') report['debugger_report'] = record(output / 'debugger/report.json', deadline) + if args.desktop: + row = support.run_process(support.load_ownership(ROOT), [sys.executable, '-B', str(ROOT / 'scripts/ci/macos-desktop.py'), + '--sdk', str(sdk), '--origin', str(anchor), '--expected-revision', args.sdk_revision, + '--helpers', str(args.desktop_helpers.resolve(strict=True)), '--editor', str(args.editor.resolve(strict=True)), + '--node', str(args.node.resolve(strict=True)), '--output', str(output / 'desktop')], + work, environment, output, 'installed-desktop', remaining(deadline - 20, 2450)) + report['phases'].append(row) + require(row['status'] == 'PASS' and row['cleanup_verified'], 'Installed desktop/editor acceptance failed') + desktop = read_json(output / 'desktop/report.json', deadline) + require(desktop['status'] == 'PASS' and desktop['expected_revision'] == args.sdk_revision and + desktop['sdk'] == str(sdk) and desktop['inputs_unchanged'] is True and desktop['cleanup_verified'] is True, + 'Desktop result is not bound to the unchanged selected SDK') + report['desktop_report'] = record(output / 'desktop/report.json', deadline) report['status'] = 'PASS' except Exception as error: report['problems'].append(type(error).__name__ + ': ' + str(error)) @@ -388,7 +411,7 @@ def main(): report['after_checks'] = checks report['cleanup_verified'] = bool(report['phases']) and all(row['cleanup_verified'] for row in report['phases']) report['elapsed_seconds'] = time.monotonic() - started - if report['problems'] or not report['cleanup_verified'] or report['elapsed_seconds'] > TOTAL_SECONDS: + if report['problems'] or not report['cleanup_verified'] or report['elapsed_seconds'] > total_seconds: report['status'] = 'FAIL' save(output / 'report.json', report) print(json.dumps({key: report[key] for key in ('status', 'sdk_revision', 'helper_revision', 'problems')})) diff --git a/scripts/ci/test_macos_desktop.py b/scripts/ci/test_macos_desktop.py new file mode 100644 index 00000000..961bf08d --- /dev/null +++ b/scripts/ci/test_macos_desktop.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +"""Rejection controls for installed desktop evidence; these do not qualify macOS.""" +import copy +import hashlib +import importlib.util +import json +from pathlib import Path +import tempfile +import time +import unittest +from unittest import mock + +SPEC = importlib.util.spec_from_file_location('macos_desktop', Path(__file__).with_name('macos-desktop.py')) +gate = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(gate) + + +class DesktopTests(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name).resolve() + self.sdk = self.root / 'SDK λ'; self.sdk.mkdir() + self.helpers = self.root / 'helpers' + self.deadline = time.monotonic() + 30 + self.common = gate.load('scripts/ci/macos-debugger.py', 'desktop_control_common') + for tool in ('nilcompiler', 'nildebugger'): + (self.sdk / tool).write_bytes(tool.encode()) + + def editor(self): + names = ['check-' + str(n) for n in range(88)] + digest = hashlib.sha256(json.dumps(sorted(names), separators=(',', ':')).encode()).hexdigest() + patch = mock.patch.object(gate, 'EDITOR_NAMES', digest) + patch.start(); self.addCleanup(patch.stop) + report = dict(status='PASS', platform='darwin', vscode='1.146.0', skippedChecks=[], fullSuiteCompleted=True, + completedModules=gate.MODULES.copy(), checks=[dict(name=n, status='PASS') for n in names], + runId='run', workspace='/private/workspace') + launch = dict(runId='run', workspace='/private/workspace') + settings = {'ny.sdkPath': str(self.sdk), **{'ny.' + key + '.path': str(self.sdk / value) + for key, value in gate.TOOL_SETTINGS.items()}} + return report, launch, settings + + def test_complete_editor_and_exact_names(self): + report, launch, settings = self.editor() + self.assertEqual(gate.audit_editor(report, launch, settings, self.sdk)['groups'], 88) + for change in ('missing', 'duplicate', 'replacement', 'failure'): + altered = copy.deepcopy(report) + if change == 'missing': altered['checks'].pop() + if change == 'duplicate': altered['checks'][0] = altered['checks'][1] + if change == 'replacement': altered['checks'][0]['name'] = 'unknown' + if change == 'failure': altered['checks'][0]['status'] = 'FAIL' + with self.subTest(change=change), self.assertRaises(ValueError): + gate.audit_editor(altered, launch, settings, self.sdk) + + def test_editor_rejects_partial_or_stale_reports(self): + report, launch, settings = self.editor() + for key, value in [('platform', 'linux'), ('vscode', '1.145.0'), ('skippedChecks', ['omitted']), + ('fullSuiteCompleted', False), ('completedModules', gate.MODULES[:-1]), + ('runId', 'old'), ('workspace', '/other')]: + with self.subTest(key=key), self.assertRaises(ValueError): + gate.audit_editor({**report, key: value}, launch, settings, self.sdk) + + def test_editor_rejects_source_tool_and_library_fallback(self): + report, launch, settings = self.editor() + for key in settings: + with self.subTest(setting=key), self.assertRaises(ValueError): + gate.audit_editor(report, launch, {**settings, key: '/source/build/bin'}, self.sdk) + + def test_editor_owned_profile_paths(self): + root = Path('/tmp/nil vscode host café 中文-123') + launch = dict(temporary=str(root), platform='darwin', workspace=str(root / 'workspace'), + profile=str(root / 'profile'), runId='12345678-1234-1234-1234-123456789abc') + self.assertEqual(gate.editor_launch(launch), root) + for key, value in [('temporary', '/tmp/unrelated'), ('profile', '/tmp/user-profile'), + ('workspace', '/tmp/user-work'), ('runId', '../outside')]: + with self.subTest(key=key), self.assertRaises(ValueError): + gate.editor_launch({**launch, key: value}) + + def test_ui_commands_use_both_installed_roots_without_product_fixtures(self): + for library in ('nilstd', 'nillib'): + argv = gate.ui_command(self.sdk, library, self.helpers, self.root / library) + for option, expected in [('--sdk', self.sdk / library), ('--compiler', self.sdk / 'nilcompiler'), + ('--tester', self.sdk / 'niltest'), ('--interpreter', self.sdk / 'nilinterpreter')]: + self.assertEqual(argv[argv.index(option) + 1], str(expected)) + self.assertNotIn('--provider-test', argv) + self.assertNotIn('--lifecycle-test', argv) + self.assertEqual(argv[argv.index('--llvm') + 1], 'ON') + with self.assertRaises(ValueError): + gate.ui_command(self.sdk, 'nilstd/src', self.helpers, self.root) + + def test_json_stream_preserves_top_level_reports(self): + values = [dict(schema='one', tests=[dict(name='nested')]), dict(schema='two')] + text = 'human line\n' + json.dumps(values[0], indent=2) + '\nmore text\n' + json.dumps(values[1]) + self.assertEqual(list(gate.objects(text)), values) + with self.assertRaises(json.JSONDecodeError): + list(gate.objects('{invalid}\n')) + + def test_ui_audit_requires_all_ten_cases_and_bound_retained_inputs(self): + controls = gate.load('tests/tools/test_ui_appkit_display_contract.py', 'installed_ui_fixture_controls') + fixture = controls.AppKitHarnessTests(); fixture.setUp(); self.addCleanup(fixture.doCleanups) + parent = self.root / 'ui-nilstd'; root = parent / 'nil-appkit-input-fixture' + root.mkdir(parents=True) + source = gate.ROOT / 'tests/ui/gtk_interaction.matrix' + summary = dict(schema='nil.appkit.interaction.v1', status='passed', compiler=str(self.sdk / 'nilcompiler'), + evidenceDirectory=str(root), source=str(source), + sourceSha256=self.common.record(source, self.deadline)['sha256'], routes=[]) + for key, path in [('inputDriver', self.helpers / 'lib/libnil_ui_appkit_input_driver.dylib'), + ('compileDriver', self.helpers / 'bin/nil_ui_appkit_compile_driver')]: + path.parent.mkdir(parents=True, exist_ok=True); path.write_bytes(key.encode()) + summary[key], summary[key + 'Sha256'] = str(path), self.common.record(path, self.deadline)['sha256'] + desktop = dict(schema='nil.test.results.v1', errors=[], tests=[dict(name=name, + suite='nil.std.ui.gtk.desktop', route=route, status='passed', exitCode=77, signal=0, timedOut=False, cancelled=False) + for name in ('widgets_layout_and_model_binding', 'window_ownership_and_size_errors', + 'dynamic_subtrees_and_effective_state') for route in ('vm', 'native')]) + reports = [desktop] + for route in ('vm', 'native'): + directory = root / route; directory.mkdir() + report = fixture.evidence(directory, route=route); reports.append(report) + (root / (route + '.stdout')).write_text(json.dumps(report)) + records = [json.loads(row['stderr'].removeprefix('NIL_APPKIT_INPUT ')) for row in report['tests']] + summary['routes'].append(dict(route=route, naturalCompletion=True, languageCases=2, + parentPid=fixture.parent, inputs=records)) + path = root / 'report.json'; path.write_text(json.dumps(summary)) + stdout = '\n'.join(json.dumps(row, indent=2) for row in reports) + audit = lambda: gate.audit_ui(stdout, parent, self.sdk, self.helpers, self.common, self.deadline) + self.assertEqual(audit()['total_cases'], 10) + for mutate in (lambda value: value['routes'].pop(), + lambda value: value['routes'][0].update(naturalCompletion=False), + lambda value: value.update(compiler='/source/compiler')): + altered = copy.deepcopy(summary); mutate(altered); path.write_text(json.dumps(altered)) + with self.assertRaises(ValueError): audit() + path.write_text(json.dumps(summary)) + (root / 'vm.stdout').write_text('{}') + with self.assertRaises(ValueError): audit() + + def dap(self): + sessions = [dict(name=encoding + '-' + action, status='PASS', natural_exit=True, stdin_open_at_exit=True, + cleanup_verified=True, exit_code=0, queued_event_retained=True) + for encoding in ('text', 'binary') for action in ('retention', 'disconnect')] + commands = [dict(argv=[str(self.sdk / 'nilcompiler'), '--sdk-path', str(self.sdk / 'nilstd')] + + (['--emit-binary-nil'] if binary else []), exit_code=0, cleanup_verified=True) + for binary in (False, True)] + return dict(schema='nil.dap-desktop.v1', status='PASS', sessions=sessions, commands=commands, + tools={str(self.sdk / name): self.common.record(self.sdk / name, self.deadline)['sha256'] + for name in ('nildebugger', 'nilcompiler')}) + + def test_four_desktop_dap_sessions_require_natural_owned_completion(self): + report = self.dap() + self.assertEqual(gate.audit_dap(report, self.sdk, self.common, self.deadline)['sessions'], 4) + for key, value in [('natural_exit', False), ('stdin_open_at_exit', False), ('cleanup_verified', False), + ('exit_code', 1), ('queued_event_retained', False)]: + altered = copy.deepcopy(report); altered['sessions'][0][key] = value + with self.subTest(key=key), self.assertRaises(ValueError): + gate.audit_dap(altered, self.sdk, self.common, self.deadline) + report['sessions'].pop() + with self.assertRaises(ValueError): gate.audit_dap(report, self.sdk, self.common, self.deadline) + + def test_dap_rejects_wrong_sdk_bytes_or_source_layout(self): + report = self.dap() + changed = copy.deepcopy(report); changed['tools'][str(self.sdk / 'nilcompiler')] = '0' * 64 + with self.assertRaises(ValueError): gate.audit_dap(changed, self.sdk, self.common, self.deadline) + changed = copy.deepcopy(report); changed['commands'][0]['argv'][2] += '/src' + with self.assertRaises(ValueError): gate.audit_dap(changed, self.sdk, self.common, self.deadline) + changed = copy.deepcopy(report); changed['commands'][1]['argv'].remove('--emit-binary-nil') + with self.assertRaises(ValueError): gate.audit_dap(changed, self.sdk, self.common, self.deadline) + + +if __name__ == '__main__': + unittest.main() From 3cc2515b9c14a2073d53b1a0e6f077ed0b4390c8 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 22:32:47 -0500 Subject: [PATCH 25/39] Inherit destination access when publishing SDK libraries --- .../workflows/macos-application-parity.yml | 10 +- scripts/package-sdk.py | 26 ++++- tests/sdk_matrix_sources_tests.py | 103 ++++++++++++++++++ 3 files changed, 131 insertions(+), 8 deletions(-) diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index 36f6ea05..5477e533 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -30,8 +30,10 @@ name: macOS application platform checks - 'tests/tools/context_manifest.py' - 'nilassembler/tests/nilassembler_cli_tests.py' - 'scripts/sdk_freestanding_imports.py' + - 'scripts/package-sdk.py' - 'scripts/validate-sdk.py' - 'tests/sdk_freestanding_imports_tests.py' + - 'tests/sdk_matrix_sources_tests.py' - 'tests/freestanding_hook_contract.cpp' - 'vscode/**' workflow_dispatch: @@ -156,7 +158,7 @@ jobs: run: | set -o pipefail cmake --build build/application --parallel 2 --target \ - nilcompiler nilinterpreter nilassembler nillinker nillsp nildebugger nilbuild niltest aperture \ + nilcompiler nilinterpreter nilassembler nillinker nillsp nildebugger nilbuild niltest aperture nilruntime_shared \ nil_vscode_host_runner nil_file_time_support_tests nil_compiler_module_darwin_tests \ nil_darwin_toolchain_inputs_tests nil_freestanding_hook_default nil_freestanding_hook_override \ nildebugger_core_tests nildebugger_source_metadata_tests nildebugger_dap_owner_tests \ @@ -225,7 +227,7 @@ jobs: assert all(case.get('status') == 'run' and all(case.find(kind) is None for kind in ['failure','error','skipped']) for case in cases) PYTHON - - name: Verify project, discovery, assembler, context and archive contracts + - name: Verify project, discovery, assembler, context, archive and packaging contracts if: ${{ !cancelled() && steps.tools.outcome == 'success' }} timeout-minutes: 30 run: | @@ -233,8 +235,8 @@ jobs: import json, subprocess, xml.etree.ElementTree as ET from pathlib import Path expected = {'nilbuild.project-contract', 'nilbuild.tool-discovery-contract', 'nilassembler.cli', - 'ny.tools.context-manifest', 'nil.freestanding-archive-imports'} - pattern = r'^(nilbuild[.](project-contract|tool-discovery-contract)|nilassembler[.]cli|ny[.]tools[.]context-manifest|nil[.]freestanding-archive-imports)$' + 'ny.tools.context-manifest', 'nil.freestanding-archive-imports', 'ny.sdk-matrix-sources'} + pattern = r'^(nilbuild[.](project-contract|tool-discovery-contract)|nilassembler[.]cli|ny[.]tools[.]context-manifest|nil[.]freestanding-archive-imports|ny[.]sdk-matrix-sources)$' base = ['ctest', '--test-dir', 'build/application', '-R', pattern] root = Path('build/application-evidence').resolve() inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) diff --git a/scripts/package-sdk.py b/scripts/package-sdk.py index ff9fde61..2ceb46f9 100755 --- a/scripts/package-sdk.py +++ b/scripts/package-sdk.py @@ -241,6 +241,27 @@ def logical_module_name(source: Path, nilstd: Path) -> str: return ".".join(relative.parts) +def install_library_payload(temporary_root: Path, nillib: Path) -> None: + """Copy completed private build outputs with destination-inherited access.""" + files = regular_tree_files(temporary_root) + if nillib.exists() or nillib.is_symlink(): + raise SystemExit("refusing to replace an existing staged nillib directory") + # Renaming a TemporaryDirectory preserves its private mode/DACL. Recreate + # every published entry under the SDK stage instead, honoring its default + # ACL and the caller's umask; copy no private staging permission metadata. + # The enclosing SDK stage is not activated until packaging has succeeded. + nillib.mkdir() + try: + for source in files: + destination = nillib / source.relative_to(temporary_root) + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, destination) + except BaseException: + # mkdir succeeded above, so this subtree belongs to this attempt. + shutil.rmtree(nillib) + raise + + def compile_standard_library( compiler: Path, nilstd: Path, stage_dir: Path, runtime_source: Path, shared_source: Path | None = None, freestanding_source: Path | None = None @@ -374,10 +395,7 @@ def compile_standard_library( lines.append("\n[{}]\n{}".format(module_name, compact)) raise SystemExit("\n".join(lines)) - nillib = stage_dir / "nillib" - if nillib.exists() or nillib.is_symlink(): - raise SystemExit("refusing to replace an existing staged nillib directory") - os.replace(temporary_root, nillib) + install_library_payload(temporary_root, stage_dir / "nillib") runtime_source.unlink() if shared_source is not None: shared_source.unlink() diff --git a/tests/sdk_matrix_sources_tests.py b/tests/sdk_matrix_sources_tests.py index dcb4707b..6bdc8699 100644 --- a/tests/sdk_matrix_sources_tests.py +++ b/tests/sdk_matrix_sources_tests.py @@ -8,10 +8,12 @@ import os from pathlib import Path import shutil +import stat import subprocess import sys import tempfile import unittest +from unittest import mock ROOT = Path(__file__).resolve().parents[1] @@ -213,6 +215,107 @@ def test_colliding_source_aliases_are_rejected(self): self.assertFalse((stage / "nillib").exists()) self.assertTrue(runtime.exists()) + def test_published_library_inherits_access_without_exposing_private_build(self): + masks = (0o022, 0o027, 0o077) if os.name != "nt" else (None,) + for mask in masks: + with self.subTest(umask=mask), tempfile.TemporaryDirectory(prefix="nil-sdk-access-") as temporary: + root = Path(temporary) + destination_root = root / "stage/nillib" + copied = [] + original_copy = shutil.copyfile + + def observe_copy(source, destination, *args, **kwargs): + source, destination = Path(source), Path(destination) + publishing = destination.is_relative_to(destination_root) + if publishing: + private = next(parent for parent in source.parents + if parent.name.startswith(".nillib.stage.")) + self.assertEqual(private.parent, root / "stage") + if os.name != "nt": + self.assertEqual(stat.S_IMODE(private.stat().st_mode), 0o700) + copied.append(destination) + result = original_copy(source, destination, *args, **kwargs) + if publishing: + self.assertEqual(PACKAGER.sha256(source), PACKAGER.sha256(destination), destination) + return result + + previous = os.umask(mask) if mask is not None else None + try: + with mock.patch.object(PACKAGER.shutil, "copyfile", side_effect=observe_copy): + stage, unused = self.make_stage(root, { + "demo/nested/answer.matrix": "book demo.nested\npublic answer returns override integer begin paragraph\ngive 42\nend paragraph\n", + }) + finally: + if previous is not None: + os.umask(previous) + self.assertGreaterEqual(len(copied), 4) # module, alias and both runtimes + self.assertEqual(list(stage.glob(".nillib.stage.*")), []) + if mask is not None: + for path in (destination_root, *destination_root.rglob("*")): + expected = (0o777 if path.is_dir() else 0o666) & ~mask + self.assertEqual(stat.S_IMODE(path.stat().st_mode), expected, path) + + def test_library_copy_failure_removes_partial_output_and_preserves_inputs(self): + with tempfile.TemporaryDirectory(prefix="nil-sdk-copy-failure-") as temporary: + stage = Path(temporary) / "stage" + nilstd = stage / "nilstd" + nilstd.mkdir(parents=True) + (nilstd / "answer.matrix").write_text( + "book demo.answer\npublic answer returns override integer begin paragraph\ngive 42\nend paragraph\n", + encoding="utf-8", newline="\n") + originals = (OPTIONS.runtime, OPTIONS.shared_runtime) + inputs = [stage / source.name for source in originals] + for source, destination in zip(originals, inputs): + shutil.copyfile(source, destination) + expected = {path: PACKAGER.sha256(path) for path in inputs} + original_copy = shutil.copyfile + published = [] + + def fail_second_copy(source, destination, *args, **kwargs): + if Path(destination).is_relative_to(stage / "nillib"): + published.append(Path(destination)) + if len(published) == 2: + raise OSError("injected publication copy failure") + return original_copy(source, destination, *args, **kwargs) + + with mock.patch.object(PACKAGER.shutil, "copyfile", side_effect=fail_second_copy): + with self.assertRaisesRegex(OSError, "injected publication copy failure"): + PACKAGER.compile_standard_library(OPTIONS.compiler, nilstd, stage, *inputs) + self.assertEqual(len(published), 2) + self.assertFalse((stage / "nillib").exists()) + self.assertEqual(list(stage.glob(".nillib.stage.*")), []) + self.assertEqual({path: PACKAGER.sha256(path) for path in inputs}, expected) + + def test_library_install_preserves_existing_targets_and_rejects_links(self): + with tempfile.TemporaryDirectory(prefix="nil-sdk-copy-target-") as temporary: + root = Path(temporary) + private = root / "private" + private.mkdir(mode=0o700) + (private / "module.nil").write_bytes(b"completed output") + nillib = root / "nillib" + nillib.write_bytes(b"existing file") + with self.assertRaisesRegex(SystemExit, "existing staged nillib"): + PACKAGER.install_library_payload(private, nillib) + self.assertEqual(nillib.read_bytes(), b"existing file") + nillib.unlink() + nillib.mkdir() + (nillib / "keep").write_bytes(b"existing directory") + with self.assertRaisesRegex(SystemExit, "existing staged nillib"): + PACKAGER.install_library_payload(private, nillib) + self.assertEqual((nillib / "keep").read_bytes(), b"existing directory") + shutil.rmtree(nillib) + if os.name != "nt": + nillib.symlink_to(root / "absent", target_is_directory=True) + with self.assertRaisesRegex(SystemExit, "existing staged nillib"): + PACKAGER.install_library_payload(private, nillib) + self.assertTrue(nillib.is_symlink()) + self.assertEqual(nillib.readlink(), root / "absent") + nillib.unlink() + (private / "link.nil").symlink_to(private / "module.nil") + with self.assertRaisesRegex(SystemExit, "must not contain symlinks"): + PACKAGER.install_library_payload(private, nillib) + self.assertFalse(nillib.exists()) + if __name__ == "__main__": parser = argparse.ArgumentParser(description=__doc__) From 59ffa974632dc45c79c50fa04d7f7038db3f79eb Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 22:32:56 -0500 Subject: [PATCH 26/39] Refresh installed platform evidence and remaining Intel editor check --- README.md | 50 ++- docs/getting-started/index.md | 18 +- docs/platform-behavior.md | 10 +- docs/platform-support.md | 368 ++++++++-------------- docs/release/v1-profile.md | 5 +- docs/standard-library/index.md | 17 +- docs/standard-library/ui.md | 5 +- docs/tools/index.md | 11 +- docs/tools/projects.md | 16 +- notebook/2026-11-11-web-runtime-parity.md | 149 +++++++++ 10 files changed, 350 insertions(+), 299 deletions(-) diff --git a/README.md b/README.md index 55c67745..1e1743fb 100644 --- a/README.md +++ b/README.md @@ -246,32 +246,30 @@ such as `say formatted "value={5}"`; it works on interpreted and native routes. ## Project status -The current version is **1.0.0-rc.1**. Linux x86_64 is the reference development -platform; the `9fb6c04` LLVM-enabled source suite passes all 590 CTests without -skips. The Windows x86_64 development SDK has -[paired Linux acceptance and installed tooling coverage](notebook/2026-11-02-windows-port.md). -The newer Windows source campaign also passes the Win32 desktop contracts and -all 88 real VS Code host groups; a fresh SDK still needs separate qualification. -Intel and Apple Silicon macOS have passing native build baselines with LLVM -enabled and disabled, real interpreter/checkpoint execution, and representative -LLVM JIT/AOT execution. Both macOS architectures also pass bounded native inline -assembly and executable-context embedding, inspection and dead-strip retention -checks. Both macOS development SDKs also pass the full installed -standard-library campaign, isolated PostgreSQL application routes, installed -debugger checks and execution checkpoints. Both hosts now pass the underlying -AppKit provider and interpreter/native language-level desktop checks. Windows uses its -native Win32 provider; Linux uses GTK4. Both Mac architectures pass all 88 real -editor groups at `9fb6c04`. Current SDK qualification and an Intel project-tool -timeout follow-up remain open. The newer socket, entropy and text-byte -intrinsics pass the same interpreter contracts on Linux, Windows and both Macs. -File timestamps share an implementation across interpreter and native routes, -with text/binary NIL, JIT and AOT checks passing on all four hosts. -Both Macs now pass native allocation-retention contracts, desktop debugger -sessions and native incremental reuse. Full platform parity awaits those -remaining SDK and project-tool checks. -The [platform support matrix](docs/platform-support.md) separates verified -development behavior from remaining SDK, provider and release work. -Windows and macOS release qualification remains pending. The +The current version is **1.0.0-rc.1**. Development coverage spans Linux x86_64, +Windows x86_64, Intel macOS and Apple Silicon macOS. Relocated development SDKs +on all four hosts pass 1,184 installed standard-library routes, twelve PostgreSQL +source/binary routes, 146 debugger checks and same-process execution checkpoints. +Linux's current SDK also passes all 590 source tests, twenty installed GTK +desktop/input cases and all 88 real VS Code host groups without exclusions. + +Windows uses Win32 and macOS uses AppKit. Windows and Apple Silicon also pass +twenty installed desktop/input cases, four desktop debugger sessions and all +88 editor groups. Both Macs pass the expanded 21-contract application campaign, +native incremental reuse and all 88 build-tree editor groups; the earlier Intel +project-tool timeout is resolved. Intel's installed UI/debugger checks pass, +but its editor run timed out during a project build and remains under +investigation. A separate Windows account exposed a packaged-library permission +defect; the correction now passes real cross-account access tests. + +The newer socket, entropy and text-byte APIs pass the same interpreter contracts +on every host; these APIs currently require the interpreter on all platforms. +File timestamps also pass textual/binary NIL, LLVM JIT and AOT checks everywhere. +The [platform support matrix](docs/platform-support.md) records exact SDK/source +revisions, retained failures and remaining deployment boundaries. + +These are development results. Windows and macOS release qualification remains +pending. The [release profile](docs/release/v1-profile.md) defines supported configurations and release requirements. Known limitations and planned improvements are tracked in the [development findings](matrix/docs/next-version-findings.md). diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md index dc420558..33ebb599 100644 --- a/docs/getting-started/index.md +++ b/docs/getting-started/index.md @@ -203,16 +203,22 @@ developer VM is not proof of installation on a bare Windows machine. ## macOS development builds Native Intel and Apple Silicon builds are established with LLVM enabled and -disabled. The recorded `c123849` baseline passed compiler/interpreter/Matrix and -checkpoint execution on all four profiles, with representative JIT/AOT routes -also passing when LLVM is enabled. The full CTest inventory still has failures; -these are development builds, not qualified macOS SDK releases. +disabled. LLVM-enabled development SDKs pass the full installed standard-library +campaign, isolated PostgreSQL routes, debugger/checkpoints and AppKit +interpreter/native desktop/input checks. Original full CTest failures are +retained alongside passing current-source follow-ups; these are development +results, not qualified macOS SDK releases. The manually dispatched [macOS workflow](../../.github/workflows/macos-bringup.yml) contains the tested configure/build commands and records the runner, Apple SDK, LLVM and optional-provider inputs. It builds tools directly and retains the -complete test inventory. Its tool archives have not undergone installed SDK -extraction, relocation, provider, debugger/editor or release acceptance. +complete test inventory. The separate +[SDK workflow](../../.github/workflows/macos-sdk.yml) packages development SDKs +and checks extraction, relocation and installed consumers. The +[installed replay workflow](../../.github/workflows/macos-sdk-replay.yml) +tests authenticated, unchanged SDK artifacts with current fixtures, including +optional desktop/editor checks. This distinguishes passing source/build-tree +tests from installed-package acceptance. Start with the [platform support matrix](../platform-support.md) and the [macOS notebook](../../notebook/2026-11-04-macos-parity-plan.md) for the exact source baselines and remaining work. Disabling LLVM does not make native routes diff --git a/docs/platform-behavior.md b/docs/platform-behavior.md index 769b33e6..60854bbf 100644 --- a/docs/platform-behavior.md +++ b/docs/platform-behavior.md @@ -4,12 +4,12 @@ The same Ny or Matrix source is meant to produce the same result on every supported host. Where the framework had a choice, it chose the portable behavior rather than the host's local convention — so a program's output, a recorded transcript, and a byte offset in a diagnostic all mean the same thing -on Linux and on Windows. +on Linux, Windows and macOS. -Linux is the development and reference host. Windows development acceptance -compares against it. macOS builds and selected execution routes now pass, but -its full parity inventory still has failures. This page states the intended -portable behavior and explicit host differences; the +Linux is the development and reference host. Windows, Intel macOS and Apple +Silicon macOS have installed standard-library and tooling acceptance against +that reference. This page states the intended portable behavior and explicit +host differences; the [support matrix](platform-support.md) records what has actually been verified. ## Line endings diff --git a/docs/platform-support.md b/docs/platform-support.md index e2589af3..7a1a5c8f 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -1,249 +1,143 @@ # Platform support and validation -Implementation, development acceptance and release qualification are separate. -This page summarizes the recorded development baselines as of 2026-11-11; -the [v1 release profile](release/v1-profile.md) defines the reviewed binary -release boundary. A newer source checkout does not update an installed SDK. +Implementation, installed development acceptance and release qualification are +separate. This page records the current development evidence; the +[v1 release profile](release/v1-profile.md) defines the reviewed binary release +boundary. A source update does not update an installed SDK. -## Recorded platform coverage +## Current development coverage | Host | Verified development baseline | Remaining boundary | | --- | --- | --- | -| Linux x86_64 | Reference host. The clean `5630388` SDK passed installed consumer routes, shared runtime, debugger, Unicode/I/O, checkpoints and GTK checks. The newer `9fb6c04` LLVM-ON source suite passes all 590 CTests without skips. | The reviewed Ubuntu release toolchain is separate from this development SDK. New release binaries still require the release gates. | -| Windows x86_64 | The `5630388` installed SDK retains paired Linux acceptance. The newer private `f79b94b` source build passes 579 of 583 registered tests; separately pinned follow-ups resolve its fixture failure and execute the three skipped contracts. Real Win32 desktop and all 88 VS Code host groups pass. | The `8376252` debugger also passes six CTests and four real Win32 desktop sessions. Current-source SDK qualification remains open. Bare-host installation and Windows release qualification remain open. PostgreSQL requires an external runtime; the current UI provider is Win32. | -| macOS 15 Intel x86_64 | The relocated `1efbe66` SDK passes all 1,184 installed consumer routes, twelve PostgreSQL source/binary routes and installed debugger/checkpoints. At `9fb6c04`, all twenty selected platform CTests, four desktop DAP sessions and all 88 editor groups pass without exclusions. AppKit VM/native desktop/input checks pass separately at `5a98403`. | The `1efbe66` full source suite retains four failures and one ELF-only execution skip. Three fixture failures are resolved by later source checks; the tool-discovery fixture's first project run exceeds its 60-second child deadline and needs hosted follow-up. Release qualification remains open. | -| macOS 15 Apple Silicon arm64 | The relocated `1efbe66` SDK passes all 1,184 installed consumer routes, twelve PostgreSQL source/binary routes and installed debugger/checkpoints. At `9fb6c04`, all twenty selected platform CTests, four desktop DAP sessions and all 88 editor groups pass without exclusions. AppKit VM/native desktop/input checks pass separately at `5a98403`. | The `1efbe66` full source suite retains four failures and one ELF-only execution skip. Later source checks resolve its four fixture failures; they supplement the unchanged SDK rather than rewrite its original report. Release qualification remains open. | +| Linux x86_64 | Clean `9fa354d` SDK: all 590 source CTests pass without skips; 1,184 installed consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty GTK desktop/input cases and all 88 editor groups pass. | Qualified in a private consumer environment. The reviewed Ubuntu release toolchain and shared deployment permissions require their own gates. | +| Windows x86_64 | Clean `9fa354d` SDK: 588 of 590 source CTests pass, with two headless display skips subsequently exercised successfully in Nathan's desktop. Relocated SDK passes all 1,184 consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty Win32 desktop/input cases, four desktop DAP sessions and all 88 editor groups. | The candidate needed a recorded, scoped read/execute grant on `nillib`. The permanent packaging correction separately passes real cross-account access tests; older SDKs do not acquire it automatically. Bare-host installation and release qualification remain open. | +| macOS 15 Intel x86_64 | Relocated `1efbe66` SDK passes all 1,184 consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty installed AppKit desktop/input cases and four desktop DAP sessions. Current `eca43f1` source fixtures pass all 21 application/tooling CTests and all 88 build-tree editor groups. | Installed editor acceptance stops after 25 passing groups at a VM project-build deadline; diagnosis remains open. The original full source report retains four failures resolved by later fixtures and one ELF-only execution skip. Shared deployment and release qualification remain open. | +| macOS 15 Apple Silicon arm64 | Relocated `1efbe66` SDK passes all 1,184 consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty installed AppKit desktop/input cases, four desktop DAP sessions and all 88 editor groups. Current `eca43f1` fixtures also pass all 21 application/tooling CTests. | The original full source report retains four failures and one ELF-only execution skip; subsequent hosted checks resolve the four fixture failures. Shared deployment and release qualification remain open. | + +The SDKs contain eleven root commands, eighteen compiled standard-library +modules, their source, native runtimes and HTML documentation. The installed +consumer campaign covers both `nilstd` and `nillib`: 888 retained execution +results are independently replayed from raw reports, while 296 object/link +routes are checked by the pinned campaign driver. Each private candidate retains +267 payload files plus its manifest through relocation to a path containing +spaces and Unicode. Mac executables are checked as Mach-O for their expected +architecture. These candidates have not replaced accepted SDKs or activated a +release. + +The Linux editor result uses VS Code 1.144.1 in an isolated X11 session and empty +profile. Both current Mac application campaigns use VS Code 1.146.0. Each passes +88 distinct groups, all nine test modules, the full-suite completion sentinel +and no exclusions. Editor results using build-tree tools are distinguished from +results using an installed SDK. + +Exact revisions, evidence hashes, host preparation, interrupted attempts and +failure follow-ups are retained in the +[current parity notebook](../notebook/2026-11-11-web-runtime-parity.md). +Earlier baselines remain in the +[Windows/Linux closeout](../notebook/2026-11-02-windows-port.md) and +[macOS bring-up notebook](../notebook/2026-11-04-macos-parity-plan.md). + +## Source failures and subsequent checks + +The original full Mac LLVM-ON suites at `1efbe66` each record 583 passes, four +failures and one skip from 588 registrations. ARM failures are the test-only +AppKit input injection, two manual native-link fixtures missing framework +arguments, and a first project build exceeding its fixture deadline. Intel's +AppKit checks pass; its fourth failure is instead the first tool-discovery +project run exceeding a 60-second child deadline. + +Subsequent test fixtures isolate AppKit injection from host compilation, pass +the runtime's required framework arguments, and give real Darwin project runs +bounded allowances that reflect toolchain fingerprinting work. Explicit short +process/error deadlines remain unchanged. The complete 21-contract campaign at +`eca43f1` passes on both architectures, including project build/run and tool +discovery, with four desktop DAP sessions and all 88 editor groups +([run 43141540789](https://github.com/exec-software/exec-nil-framework/actions/runs/43141540789)). +These results supplement the unchanged SDK; the original failed reports remain +failed. The ELF-only freestanding executable fixture remains an explicit +format-specific skip. Actual Mach-O archives separately pass import validation. + +Windows's former transient native output-replacement failure is corrected at +`9fa354d`: compiler and linker share bounded Windows sharing/lock retries and +preserve existing output on persistent failure. Strict MSVC controls exercise +real held handles, read-only output, Unicode paths and exact byte preservation; +the former single-attempt implementation fails the transient-release control. +The complete source suite then has only the two headless display skips above. LLVM-OFF builds exercise frontend/interpreter behavior and rejection of disabled -native operations. Their successful tests do not establish native execution. -The macOS archives contain eleven executable tools per profile, verified as -Mach-O files for the expected architecture. The initial bring-up archives were -build products; the newer SDK workflow additionally stages, relocates and tests -development packages. Neither archive type is a release-qualified SDK. - -The exact counts, failures, skips, host provisioning and source identities are -retained in the [Windows/Linux closeout](../notebook/2026-11-02-windows-port.md) -and [macOS build baseline](../notebook/2026-11-04-macos-parity-plan.md). -At `e087964`, the full macOS inventories are complete and all build/backend gates -pass. Failures and skips retain their actual status: - -| Profile | Pass | Fail | Skip | Registered | -| --- | ---: | ---: | ---: | ---: | -| Intel, LLVM ON | 564 | 1 | 5 | 570 | -| Intel, LLVM OFF | 419 | 1 | 146 | 566 | -| Apple Silicon, LLVM ON | 563 | 2 | 5 | 570 | -| Apple Silicon, LLVM OFF | 419 | 1 | 146 | 566 | - -All four pass the filesystem, Unicode namespace, owned-process, parallel-process -and SDK-publication contracts. The new compiler-identity checks pass while the -positive cache-reuse test continues to fail on Darwin. Apple Silicon also fails -the x86_64 native-assembly fixture in that baseline. The subsequent bounded -`aarch64` implementation at `f96f237` passes 351 validator checks and eight -actual Ny/Matrix ARM executions through JIT, AOT and text/binary object linking. -Both bounded native assembly profiles are documented in the -[assembly contract](advanced/inline-assembly.md). -Both full LLVM-ON inventories at `f96f237` have 565 passes, one known cache-reuse -failure and five skips out of 571 registrations. Their catalog/registration -timeout mismatch was corrected subsequently. At `2e098f1`, the ARM SDK packages -with exact tested bytes and passes staged and relocated validation, including -an actual shared-runtime ABI/loader check. A diagnostic wording mismatch in the -malformed-library negative control stopped the installed consumer campaign; -full SDK acceptance is still pending. -These source checks do not replace accepted SDK bytes or establish full parity. - -The subsequent `2f5d695` SDKs on both architectures each complete all 1,184 reported -installed standard-library routes: 888 retained executions independently checked -from raw evidence, plus 296 object/link routes checked by the pinned campaign driver. -Each private PostgreSQL 16.15 provider passes build/install and canonical loader -checks, followed by six routes each against installed source and binary roots. -The full source inventory has 566 passes, two failures and three skips. The new -failure exposed Darwin accepting scoped IPv6 text as a plain numeric host address; -`8d1f65d` rejects scope suffixes before parsing. Native verification -passes on both architectures at `3414ce5`. The Intel result is retained in a -partial source-test run; it does not establish a complete corrected Intel SDK. -The installed debugger helper stopped before execution on a manifest-field -mismatch. Corrected acceptance at helper revision `12f70cb` subsequently passes -against both original, unchanged SDK artifacts: 146 named debugger checks, -15 JSONL/DAP sessions and the separate live execution-checkpoint contract per host. -This covers interpreter debugging; it does not establish native debugging or -editor GUI acceptance. - -The corrected Apple Silicon SDK at `3414ce5` passes 567 source tests, with only -the incremental-reuse failure and the three existing skips. All 1,184 installed -standard-library routes, twelve PostgreSQL routes, 146 debugger checks and the -live checkpoint contract pass again. The matching Intel source-test step reached -its twenty-minute workflow limit before producing a complete report. Packaging -therefore could not start; the corrected numeric-address test did pass beforehand. -The Intel retry at `0c88d2f` subsequently completes in 17 minutes 14 seconds: -567 source tests pass, with the same single incremental-reuse failure and three -skips as Apple Silicon. All 1,184 installed standard-library routes, twelve -PostgreSQL routes, 146 debugger checks and the live checkpoint contract pass. -This corrects the Intel development baseline without replacing accepted SDKs. - -A private Windows build at `e087964` also passes four focused compiler-identity, -source-protection and real VM/native incremental contracts: 181 checks, no skips. -This supplements the recorded Windows SDK baseline; it does not replace its -installed binaries or qualify a new complete SDK. A later private build at -`f96f237` passes five focused CTests, including 351 assembly validator checks, -eight native Ny/Matrix executions, the additional Matrix routes and the real -same-process execution-checkpoint contract. The accepted SDK remains unchanged. -A private Windows build at `8d1f65d` also passes both database runtime/client -tests after the numeric-address correction, including IPv4/IPv6 timeout and -socket-close checks. The original checkout, installed SDK and provider DLLs -remain unchanged. - -A private Windows build at `8aa60d2` records the first complete local suite since -the port closeout: 574 registered tests against 572 on the Linux reference. The -two inventories share 569 tests. Five are Windows-only: console UTF-8, -unmanifested UTF-8 file paths, the GTK loader, the native memory-limit self-test -and the SDK preflight. Three are Linux-only: toolchain inputs, POSIX process -semantics and a socket-backpressure check of the Python test client rather than -of a product surface. No language, frontend, type-checker, IR, interpreter, LLVM -or standard-library test fails. Four shared contracts fail only when the suite -runs as `SYSTEM`: that account reports `TEMP` cased `C:\WINDOWS\TEMP`, the -toolchain reports paths in their on-disk casing, and the affected assertions -compare the two strings exactly; the same four pass unchanged under an ordinary -user account. The Windows-only SDK preflight needs a machine execution -policy that permits local scripts, because it invokes `powershell.exe -File` -deliberately without a bypass; under the Windows client default of `Restricted` -every subtest observes a policy refusal instead of a preflight diagnostic, and -it passes once the policy is `RemoteSigned`. Two display tests skip for want of -an interactive desktop session and both PostgreSQL tests skip without a -server. This supplements the recorded Windows baseline; it neither -replaces the accepted SDK nor qualifies a new one. - -## Native executable context - -Thin Mach-O64 x86_64 and arm64 objects and executables support bounded inspection -and opt-in annotation embedding. Both LLVM-ON hosted profiles at `e087964` passed -3,592 context assertions and 60 CLI checks, including six native executions -and host-linker dead stripping. LLVM-OFF profiles passed the inspection tests; -they explicitly skip native embedding. See the -[container contract](advanced/executable-context.md) for supported forms and limits. - -## SDK publication mechanics - -The full `e087964` matrix passes the publication contract on all four profiles. -A separate native stress check also passes 5,000 replacements on each architecture. -Darwin retains old symlink inodes and preserves readable new links under -restrictive umasks. This verifies publication mechanics, not a complete installed -SDK. See [SDK publication](getting-started/index.md) for retention lifetime. - -## Runtime control and checkpoints +native operations. Their passing tests do not establish native execution. + +## Native code and tools + +Both Mac architectures pass native incremental reuse and invalidation. Compiler +identity hashes actual loaded files and exact shared-cache backing files; +native linker probes collect 8,321 inputs. Unmodeled dependency profiles rebuild +rather than reuse stale output. SQL runtime loader paths are scoped to execution +processes so they do not change ordinary compiler/linker identity. + +Bounded x86_64 and aarch64 native inline assembly are described in the +[assembly contract](advanced/inline-assembly.md). Thin Mach-O64 x86_64 and arm64 +objects/executables support bounded context inspection, opt-in annotation +embedding and dead-strip retention. See the +[container contract](advanced/executable-context.md) for supported forms. + +ELF, COFF and Mach-O freestanding runtime archives are checked for malformed or +mixed object profiles and unexpected imports. Actual MSVC no-CRT controls link +and execute with only the explicitly selected host exit import. This does not +claim a cross-platform bare-metal application profile. + +Managed and owned native lifetime tests pass on both Macs. The owned workloads +allocate and free 36,000 guarded buffers per memory mode; withholding frees +reaches the 96 MiB retained-payload bound. Darwin measures intercepted +malloc-family payloads, excluding allocator metadata and mappings. Linux's +address-space limit and Windows Job limits retain their platform-specific +metrics; these are not identical whole-process memory measurements. + +## Desktop and runtime control + +The UI provider is GTK4 on Linux, Win32 on Windows and AppKit on macOS. Linux +needs an external GTK4 runtime. Display/input tests need an interactive window +server; headless skips do not count as passes. Source AppKit campaigns pass +provider, lifecycle, layout, focus, sensitivity, Unicode entry and real +interpreter/native desktop/input contracts on both architectures. Test input +uses each owning application's responders, without claiming general desktop +automation or cross-process accessibility support. `nilinterpreter --ai` provides opt-in local control through Windows named pipes -or Unix sockets. Pause, resume, instruction stepping and bounded state inspection -use the real VM execution thread. `--ai-wait` starts paused before `main`. -The live checkpoint contract passed on both Windows/Linux development SDKs and -all four recorded macOS profiles. +or Unix sockets. Pause, resume, instruction stepping and bounded inspection use +the real VM execution thread. `--ai-wait` starts paused before `main`. `vm.snapshot` and `vm.restore` provide reusable **same-process** execution -checkpoints for a paused, deterministic VM program. They capture execution -position, frames, values, heap identities and admitted runtime state. The -standalone interpreter retains the latest successful checkpoint. External -effects, open resources, native/JIT execution, cross-process persistence and -code hot reload are outside this profile. Programs containing excluded -operations are rejected before capture; this is not arbitrary application -rollback. See [NIL-0006](design/NIL-0006-ai-module.md) for commands, limits and -the complete admission rules. - -## macOS coverage and remaining work - -- Native incremental reuse passes on both Macs at `860750e`. The compiler resolves - loaded images to ordinary files and exact shared-cache backing files for full - content hashing; its actual collector test passes on both Macs at `2632bc6`. - The builder uses separate preparation and verification digest phases without - caching source/manifest reads. Both actual linker-dependency probes collect - 8,321 inputs, and both real VM/native reuse and invalidation campaigns pass. - The subsequent SDK archive-fallback refinement also passes both hosts at - `1efbe66`, together with all sixteen native, debugger and incremental contracts. - Incomplete or unsupported dependency profiles deliberately rebuild. -- A separate [development SDK workflow](../.github/workflows/macos-sdk.yml) now - implements native Darwin consumer admission, root install layout, pinned - documentation tooling, and staged dependency/relocation checks. Both - candidates preserve all 260 payload files plus their manifests through relocation - and pass the installed standard-library and debugger/checkpoint campaigns. - The optional isolated PostgreSQL 16 profile now passes - its coherent client/server and installed source/binary routes on both architectures; - the scoped-address validation correction passes on both hosts. The corrected - Intel retry at `0c88d2f` completes all installed campaigns, as recorded above. -- Both Mac hosts pass the managed and owned native lifetime contracts at - `8376252`, including both owned memory modes. Each owned workload allocates - and frees 36,000 guarded buffers; mandatory withheld-free controls reach the - 96 MiB retained-payload limit. The allocator controls also pass more than 2 GiB - of cumulative allocation. This measures intercepted malloc-family payloads, - not total address space, allocator metadata or mappings. The old Darwin - virtual-address cap remains unusable and is not represented as passing. -- Desktop UI now uses one provider per platform: GTK on Linux and Win32 on - Windows. The Windows provider passes the full desktop contract on a real - interactive session -- provider contract, lifecycle release observation, the - Matrix desktop suite and the interaction suite that drives real keyboard - input -- while a session without an interactive window station still skips. - On macOS the provider is AppKit. Its published contract passes against a - real window server session on both hosted architectures; the newer `40017f1` - provider passes 156 display and 41 in-process keyboard checks per host, - and `nil.ui-display` now passes there too: the provider fixture, the - lifecycle fixture and the Matrix desktop suite. Real keyboard input is - covered in process by the AppKit provider fixture. At `5a98403`, both Macs - pass all seven selected UI CTests and all ten desktop/interaction Matrix - cases across interpreter/native routes. The test-only input driver uses - each owning application's responders; its compiler adapter keeps injection - out of host compilation. This does not qualify cross-process keyboard - injection or accessibility. Earlier - private GTK and AppKit prerequisite probes are retained as historical evidence, - but GTK is no longer the macOS product provider. The LLVM-ON campaign at - `965f159` passes all five selected CTests on both hosts, including all three - Matrix desktop cases on each interpreter/native route. That earlier six-case - baseline is superseded by the ten-case campaign above; editor acceptance - remains separate. -- Editor desktop integration, untested provider configurations and release-package - acceptance remain separate from the passing installed campaigns. Shipping a - Mach-O tool alone does not qualify these surfaces. - -The [macOS workflow](../.github/workflows/macos-bringup.yml) records the complete -inventory, including failures and skips. The [release gates](release/v1-profile.md#qualification-and-publication) -still apply before publishing a macOS SDK or declaring parity. - -## New web-runtime APIs - -The socket, secure-random and text-byte additions in the open web-runtime branch -are currently interpreter-backed. File timestamps share a stable native-metadata -implementation across interpreter and native routes: Linux text NIL, binary NIL, -JIT and AOT all pass unchanged-file and changed-metadata checks. Linux results -alone do not establish Windows or macOS support, and declarations do not establish -native/JIT execution. The actual native timestamp contract also passes on Windows -and both Mac architectures; full current-source packaging remains in progress. -Their evidence must join the older desktop, tooling and SDK baselines before -declaring current-source parity. - -The private Windows LLVM-ON build at `965f159` passes all six focused web-runtime -contracts with no skips, including VM socket cleanup and the native timestamp -routes. At `19bd5f1`, the hosted campaign passes the same six interpreter contracts -on all four hosts, with exact inventories, no skips and independent raw-result -replay. The tested PR merge and head revisions are retained separately. Current -macOS editor coverage is still in progress. The socket/byte/entropy -native limitation is common to Linux, Windows and macOS; those APIs currently -require the interpreter on all three hosts. - -## Current Windows source and editor follow-up - -At `f79b94b`, the full private Windows build records 579 passes, one portable -path-fixture failure and three skips across 583 registrations. The corrected -fixture passes 71 MSVC checks separately. The formerly ELF-only symbol contract -now passes all sixteen generic ABI/IR checks against actual COFF objects. -Nathan's interactive session passes all ten Matrix VM/native desktop and input -cases, 137 provider checks, 235 lifecycle checks and window discovery. The retained -route reports contain 1,138 successes and 78 expected native rejections; the -six PostgreSQL routes and owned-cluster shutdown also pass. - -The full real VS Code host passes 88 distinct groups with no exclusions, using -the `a518138` test-only physical-file alias correction with `f79b94b` products. -Extension unit coverage is 161 passes and one Windows-inapplicable POSIX FIFO -skip. An outer evidence wrapper incorrectly expected at least 100 host groups; -its raw failure is retained alongside independent replay of every artifact hash, -all six successful process phases and cleanup. This composed source evidence -does not replace the accepted SDK. The main-thread DAP execution change at -`8376252` subsequently passes all six debugger CTests, fourteen lifecycle sessions -and four actual desktop sessions on Windows and both Mac architectures. Linux -also passes all 586 source CTests at this revision. Details are in the -[web-runtime parity notebook](../notebook/2026-11-11-web-runtime-parity.md). +checkpoints for paused deterministic VM programs: execution position, frames, +values, heap identities and admitted runtime state. Open resources, external +effects, native/JIT execution, cross-process persistence and code hot reload +remain outside this profile. See [NIL-0006](design/NIL-0006-ai-module.md). +DAP keeps VM execution, inspection and UI cleanup on the process main thread. +Passing interpreter debugging does not establish native debugging. + +## Web-runtime APIs and deployment boundaries + +Sockets, secure randomness and text-byte APIs pass the same six interpreter +contracts on Linux, Windows and both Macs without skips. These additions require +the interpreter on every host because the native text ABI does not yet preserve +arbitrary embedded NUL bytes. File timestamps use native filesystem metadata +across interpreter and native routes; text/binary NIL, JIT and AOT checks pass +on all four hosts. PostgreSQL acceptance uses an isolated owned server and an +external client runtime; it does not bundle or qualify every provider version. + +The SDK builder honors `CMAKE_BUILD_PARALLEL_LEVEL`. SDK publication checks +cover staged layout, relocation, retained payload bytes and modes, and bounded +replacement. Cross-account Windows use additionally exposed a binary-library +access defect: the private temporary compilation directory's DACL survived its +move into the SDK. Packaging now copies validated, completed outputs into fresh +destination entries, inheriting normal SDK access without exposing private +compilation or changing machine ACLs. Linux tests cover restrictive umasks, +unchanged bytes and failed-copy rollback; a separate limited Windows account +reads all 102 published library files while remaining unable to read private +staging. These checks do not retroactively change older SDK permissions or +qualify arbitrary shared installations. + +The [release gates](release/v1-profile.md#qualification-and-publication) still +apply before publishing new binaries. Current development coverage does not +expand the reviewed Ubuntu release profile, establish a bare-host Windows or +macOS installation, or imply a stable v1 release. diff --git a/docs/release/v1-profile.md b/docs/release/v1-profile.md index ef6f3c3b..ba8bf455 100644 --- a/docs/release/v1-profile.md +++ b/docs/release/v1-profile.md @@ -6,8 +6,9 @@ release; it does not promote every design document to an implemented promise. No stable v1 tag or publication is implied by building a candidate. Development capability has advanced beyond the earlier reviewed RC binaries. -The [platform support matrix](../platform-support.md) records paired -Windows/Linux SDK acceptance and native macOS build/runtime baselines. Those +The [platform support matrix](../platform-support.md) records installed SDK, +desktop, debugger and editor acceptance across Linux, Windows and both Mac +architectures, including exact candidate revisions and remaining checks. Those results do not extend this document's pinned release qualification. ## Host and dependencies diff --git a/docs/standard-library/index.md b/docs/standard-library/index.md index f05b6c70..2f10ab4a 100644 --- a/docs/standard-library/index.md +++ b/docs/standard-library/index.md @@ -88,8 +88,8 @@ execution routes, and editor integration. `nil.std.ui` supplies checked windows, rows, columns, grids, labels, buttons, text entries, and checkboxes. Copied typed events and explicit text/flag models keep application logic in Matrix; the shared runtime owns native resources. -The provider uses GTK 4 on Linux and the Windows development target, returning -an explanatory error when the optional provider or a display is unavailable. +The provider uses GTK4 on Linux, Win32 on Windows and AppKit on macOS, returning +an explanatory error when the required provider or a display is unavailable. See the [UI reference](ui.md) and [NIL² counter application](../../examples/projects/nil2-counter). @@ -165,13 +165,12 @@ native named-struct-array lowering; do not infer enumeration support from the scalar environment operations. New source-tree verification is recorded in the Windows-port notebook; an older installed SDK does not acquire these additions. -The reviewed native release boundary is currently Linux. Windows has paired -development acceptance; Intel and Apple Silicon macOS now have native build -and selected VM/JIT/AOT execution evidence. Both macOS development SDKs -also pass the full installed standard-library campaign and isolated PostgreSQL -routes, plus installed debugger/checkpoint acceptance. AppKit window/event -prerequisites pass on both hosts, but GTK desktop acceptance, incremental reuse -and native memory-limit tests still prevent full macOS parity. See the +The reviewed native release boundary is currently Linux. Development SDKs on +Linux, Windows, Intel macOS and Apple Silicon macOS pass the full installed +standard-library campaign, isolated PostgreSQL routes and installed +debugger/checkpoint acceptance. AppKit desktop/input, native incremental reuse +and allocation-retention checks also pass on both Macs. macOS uses AppKit; +GTK is the Linux provider. See the [platform support matrix](../platform-support.md) for the recorded scope. Windows compiler AOT and standalone linking now launch argument vectors through `CreateProcessW` and embed UTF-8 process manifests in generated executables. diff --git a/docs/standard-library/ui.md b/docs/standard-library/ui.md index 5561994f..dfa88ca6 100644 --- a/docs/standard-library/ui.md +++ b/docs/standard-library/ui.md @@ -6,8 +6,9 @@ The VM and native compiler use one desktop provider per platform, each behind the same checked boundary: a GTK4 provider on Linux and a Win32 provider on Windows. Windows remains a development target, not a qualified release platform. The macOS provider is AppKit. Its provider, lifecycle and interpreter/native -language-level desktop contracts pass on Intel and Apple Silicon; current SDK -and editor acceptance remain separate. AppKit replaces the +language-level desktop contracts pass on Intel and Apple Silicon, including +twenty installed source/binary-library desktop/input cases on each architecture. +Editor and release qualification are recorded separately. AppKit replaces the earlier private GTK-on-macOS experiments. See [platform support](../platform-support.md). Examples display the framework generation as **NIL²**, with **NIL^2** as the plain-text spelling. This presentation does not change numeric release or diff --git a/docs/tools/index.md b/docs/tools/index.md index 500b1268..36299efc 100644 --- a/docs/tools/index.md +++ b/docs/tools/index.md @@ -62,12 +62,13 @@ The [project driver](projects.md) provides `init`, `convert`, `describe`, `check `build`, `run` and `test` over a `nil.project.nildata` file. It shares compiler analysis and project settings with the VS Code extension. VM/native targets share bounded parallel project scheduling. Incremental reuse requires complete -source and toolchain identities; Darwin currently rebuilds both targets while -loaded-image identity remains incomplete. See the +source and toolchain identities. Both Mac architectures now pass real VM/native +reuse and invalidation, including loaded images and shared-cache backing files; +unsupported dependency profiles rebuild. See the [incremental build contract](projects.md#incremental-and-parallel-builds). -Project process execution has recorded Linux and paired Windows `5630388` -development acceptance; [platform support](../platform-support.md) separates -that baseline from release qualification. +Project process execution has recorded Linux, Windows and both Mac architecture +development acceptance; [platform support](../platform-support.md) records exact +source and SDK revisions separately from release qualification. ## `niltest` diff --git a/docs/tools/projects.md b/docs/tools/projects.md index b80b8bb2..88745202 100644 --- a/docs/tools/projects.md +++ b/docs/tools/projects.md @@ -4,10 +4,11 @@ compiler. A `nil.project.nildata` manifest supplies the entrypoint, import roots, SDK selection and analyzer policy to the CLI and VS Code extension. Each project produces one executable; project references express the ordering -between projects. Project execution has paired Linux/Windows development -acceptance at the `5630388` SDK baseline; see the -[platform support matrix](../platform-support.md) for current scope and macOS -work. The native target requires an LLVM-enabled compiler and a host C++ linker. +between projects. Project execution has development acceptance on Linux, +Windows, Intel macOS and Apple Silicon macOS; see the +[platform support matrix](../platform-support.md) for exact source/SDK revisions +and remaining editor and deployment checks. The native target requires an +LLVM-enabled compiler and a host C++ linker. ## Create a project @@ -125,9 +126,10 @@ textual include and binary-library inputs, effective settings and toolchain, and the produced artifact. Native builds also account for the resolved runtime archive, linker driver, startup objects and system link-library files. Unknown native linker layouts or incomplete compiler loaded-file identity disable cache -reuse. Darwin currently rebuilds both VM and native outputs because loaded-image -identity is incomplete; native link-input discovery also remains open. Modifying -a source while preserving its timestamp, +reuse. Darwin includes actual loaded images and complete shared-cache backing +files, plus the admitted native link inputs. Real VM/native reuse and +invalidation pass on both Mac architectures; unknown profiles still rebuild. +Modifying a source while preserving its timestamp, changing import resolution, replacing a tool, or changing the existing artifact invalidates reuse. A missing or malformed state file is a cache miss. diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index ce62309a..2353feb4 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -643,3 +643,152 @@ reports 653 checks. Those changes are preserved; the older 330-check snapshot and its reproductions must be checked against this new source before making further web changes. The README's remaining-parity paragraph still lists logs, configuration and virtual hosts as absent despite documenting them elsewhere. + + +## Current installed Linux closeout and Mac source follow-up + +The private Linux SDK at exact `9fa354d48575378065f291c6208148c5e18ab61a` +completes all 590 source CTests with no failures or skips. After relocation to +`NIL SDK space λ`, all 1,184 installed consumer routes pass: 148 retained route +reports supply 888 independently replayed results and 1,628 command records; +296 native object/link executions are validated by the pinned driver. Installed +acceptance also passes 146 debugger checks/15 sessions, live execution +checkpoints, twelve PostgreSQL source/binary routes, sixteen timestamp route +observations with 100 repetitions each, twenty GTK desktop/input cases, +sixteen Unicode byte-file routes, six ordinary I/O observations and actual +relocated `libnil.so` loader/ABI identity. + +All 88 VS Code 1.144.1 host groups pass with all nine modules, no exclusions and +the full-suite completion sentinel. The launcher uses an empty private profile, +isolated X11 display and D-Bus, private XDG roots and a launcher-only +`--password-store=basic` option. All 78 source fixture pins remain unchanged; +no test assertions or product binaries were changed. The 267 payload files and +manifest preserve their bytes and modes, the original private install paths +are absent, and all six owned service groups are stopped with no remaining PID. +The composite report is `linux-sdk-9fa354d/linux-private-closeout.json`, SHA-256 +`a8d1a5dec3a5e96b0e5d0a717090723a72ccd878fa4a65ed3ca1abfcfd2b13d2`. + +Failed attempts remain recorded: PostgreSQL's long private Unix-socket path +exceeded its 107-byte bound; a remaining-only retry uses a short owned temporary +path. An editor launch inherited Wayland without a usable display. A later +attempt timed out during its first NILData open/show/language check, before any +group completed; this was not a demonstrated language-ID mismatch, and observed +portal/keyring activation is not established as its cause. The isolated X11 +launch then passes the unchanged assertions. Canonical successful-test cleanup +intentionally removes generated products; independent replay checks their +retained command/result metadata rather than claiming to hash absent files. + +Both hosted Mac architectures complete the expanded 21-contract application +campaign at feature head `eca43f1`, tested PR merge +`3c677f7145f083fc228d09371a6f92976ebf81b8` +([run 43141540789](https://github.com/exec-software/exec-nil-framework/actions/runs/43141540789)). +The exact inventory is five SDK/project contracts, six debugger contracts, +eight native contracts and two incremental contracts. Four real desktop DAP +sessions and all 88 VS Code 1.146.0 groups also pass per host, with all nine +modules and no exclusions. All 620 evidence hashes verify. Report +`github-applications-43141540789/replay-report.json` has SHA-256 +`1f14c9689ef8ec563854cb3f80a0b4b2d491411a028930973942b8b2285f7d10`. +This resolves the previously open Intel discovery deadline follow-up. Original +full-suite failures stay preserved; this is supplemental current-source +acceptance, not a rewritten full source report or a replacement SDK. + +## Windows installed desktop and library access + +The exact `9fa354d` Windows candidate completes 590 registered source tests: +588 pass, none fail, and `nilagent.window-discovery`/`nil.ui-display` skip in the +headless session. Eleven root tools, eighteen library modules and 69 compiled +documents are staged. The relocated package then passes all 1,184 consumer +routes, 146 debugger checks/15 sessions, five execution-checkpoint sessions, +twelve PostgreSQL source/binary routes, Unicode/ordinary I/O, timestamps and +runtime hooks. Independent replay verifies 442 installed evidence files and +28 raw phase streams. Installed report SHA-256: +`bc9d27c6f6c451f8ce4bcb5c1ff5b988a81c7fc577e688a3138952aba9071d23`. + +Nathan's interactive session initially stops at read-only preflight: it can +read 166 of the 268 SDK files, with all 102 missing files under `nillib`. +Nil/admin can read and hash all 268 unchanged; no readable payload differs. +The compiled library root retains the protected Owner Rights/System/Admin DACL +from Python 3.15's `TemporaryDirectory`, because packaging renamed that private +compilation root directly into `nillib`. Its descendants inherit the restriction. +This is a real cross-account packaging defect; same-user acceptance did not +exercise it. The analogous private POSIX root mode also needs correction. + +Only this private candidate's `nillib` received inheritable Nathan read/execute +access, with before/after ACLs and hashes retained under `desktop-access`. +All 268 payload bytes and stat modes remain unchanged; Windows DACLs did change +and are not described as unchanged by the payload seal. Both initial preflight +failures remain preserved. No accepted SDK, original checkout or machine-wide +ACL was changed. + +With that explicit access preparation, Nathan passes twenty VM/native UI cases +(ten per source/binary library root), 217 provider and 235 lifecycle checks per +root, window discovery, all 88 editor groups/all nine modules/no exclusions and +four naturally exiting desktop DAP sessions. Extension unit tests pass 161 +checks with one POSIX-only skip. All 65 desktop evidence files and sixteen raw +streams verify and the owned task is removed. Desktop report SHA-256: +`c50b32b8f7dd000d3c5159c19cf1c69c19c44057ad76c5305c40c3ca36efa51d`. +The previously headless display surfaces have now executed successfully; their +original source-report skips remain intact. The permanent packaging correction +requires its own tests before claiming default cross-account access. + + +## Published library access correction + +At `3cc2515`, `compile_standard_library` retains its private compilation directory and +copies the fully checked outputs into fresh destination entries. Each published +directory/file inherits the destination's normal access policy; it copies no +private mode/DACL metadata and does not call chmod or edit ACLs. Inventory +rejects symlinks and special files before creating `nillib`. A failed copy +removes only this attempt's new subtree; existing targets are preserved and +original runtime inputs remain until publication succeeds. + +The real Linux compiler/interpreter/native mixed-source suite passes thirteen +tests, including umasks 0022/0027/0077, private staging mode 0700, every copied +file's exact hash, an injected second-copy failure, original runtime preservation +and existing file/directory/dangling-symlink refusal. All 46 SDK qualification +controls also pass. The old Linux SDK's root really is 0700; its bytes and modes +remain unchanged. Evidence `package-access-linux-ucvmt82_/report.json`, SHA-256 +`c4f33be5b5ec3b3c8849c12eabfa549df0e1306bfd050d38afd459798890f362`. + +Actual Windows validation uses the same sealed `9fa354d` tools with only the +packaging/test scripts overlaid in a disposable tree. All thirteen mixed-source +and native tests pass. Limited Nathan reads and hash-matches all 102 published +library files, while access to the original private staging directory is still +denied (errno 13); all 121 published ACL entries inherit destination access. +The one-off task and private stage are removed. All 33 evidence files and eight +raw streams independently verify, and the original candidate remains unchanged. +The first probe expected a Windows-specific error field; actual denied reads +reported `PermissionError` with errno 13. That failed assertion remains retained +alongside the corrected denial check and successful bounded repeat. +Evidence `sdk-9fa354d48575/package-access-ed2765d377/probe-report.json`, SHA-256 +`da3fe80b73dd28f92caaf35dd75b87952c49dd417452d90929cd98f74d3538fd`. +The next Mac application campaign also includes the actual mixed-source/native +packaging test, increasing its exact selected CTest inventory from 21 to 22. + +## Installed Mac desktop follow-up + +[Run 43146638765](https://github.com/exec-software/exec-nil-framework/actions/runs/43146638765) +uses the exact authenticated `1efbe6623445cfcc486b8b7a7794dd112c357224` SDKs from +run `43129959317`, with separately pinned `285aea2` acceptance helpers. It builds +only three test-only desktop helpers; it does not rebuild or substitute product +tools or runtimes. The earlier dispatch `43146568751` had an incorrectly typed +SDK revision and was cancelled before acceptance; the correction is retained. + +ARM passes twenty installed UI cases, four desktop DAP sessions with retained +wire frames, 146 installed debugger checks/15 sessions, execution checkpoints +and all 88 VS Code 1.146.0 groups/all nine modules/no exclusions. Independent +replay verifies 213 new evidence files and all 920 original SDK evidence hashes, +alongside exact preservation of all 268 SDK file bytes/modes and helper/editor +inputs. Its desktop report SHA-256 is +`b1f1ba3e46dc0615b5088f27a5f939de1ec7e7617082662a0a14aed95f4c7692`. + +Intel passes the same installed UI, desktop DAP, debugger and checkpoint phases, +but the editor stops after 25 passing groups: the `ny: vm project check, build +and run tasks` group reaches its 180-second deadline. Project check completed +in 1,762 ms; project build started without a captured process-end event. The +suite's global task-end listener was already installed, so a late command-local +subscription is not established as the cause. No task-start PID/CPU snapshot +was retained, preventing attribution to compilation, fingerprinting or task +launch. All 211 evidence files verify and input seals remain unchanged. The +failure remains recorded; bounded task lifecycle diagnostics and a same-deadline +Intel follow-up are required before installed editor parity is complete. From fc782892e1655601830ae20273d0b638f3e93501 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 22:35:07 -0500 Subject: [PATCH 27/39] Retain editor task lifecycle evidence on bounded failures --- vscode/src/test/taskEvidence.test.ts | 2 + vscode/test/project-checks.cjs | 7 ++- vscode/test/suite.cjs | 17 +++++- vscode/test/task-evidence.cjs | 77 ++++++++++++++++++++++++++++ vscode/test/task-evidence.test.cjs | 70 +++++++++++++++++++++++++ 5 files changed, 170 insertions(+), 3 deletions(-) create mode 100644 vscode/src/test/taskEvidence.test.ts create mode 100644 vscode/test/task-evidence.cjs create mode 100644 vscode/test/task-evidence.test.cjs diff --git a/vscode/src/test/taskEvidence.test.ts b/vscode/src/test/taskEvidence.test.ts new file mode 100644 index 00000000..5d99af75 --- /dev/null +++ b/vscode/src/test/taskEvidence.test.ts @@ -0,0 +1,2 @@ +// Fixture diagnostics are pure Node controls; they do not qualify an editor host. +require("../../test/task-evidence.test.cjs"); diff --git a/vscode/test/project-checks.cjs b/vscode/test/project-checks.cjs index dbd048fd..9b73e2a0 100644 --- a/vscode/test/project-checks.cjs +++ b/vscode/test/project-checks.cjs @@ -26,7 +26,7 @@ async function replace(document, text) { edit.replace(document.uri, new vscode.Range(document.positionAt(0), document.positionAt(document.getText().length)), text); assert.equal(await vscode.workspace.applyEdit(edit), true); } -exports.run = async ({ check, workspace }) => { +exports.run = async ({ check, workspace, taskEvidence }) => { for (const language of ["ny", "matrix"]) { const directory = path.join(workspace, `project ${language}`); await fs.mkdir(path.join(directory, "src"), { recursive: true }); @@ -73,8 +73,11 @@ exports.run = async ({ check, workspace }) => { const phases = []; for (const command of [checkCommand, buildCommand, runCommand]) { const start = Date.now(); + const dispatch = taskEvidence?.dispatch(command); console.log(`Starting ${language} ${target} project command: ${command}`); - assert.equal(await vscode.commands.executeCommand(command), 0, command); + const exitCode = await vscode.commands.executeCommand(command); + if (dispatch) taskEvidence.complete(dispatch, exitCode); + assert.equal(exitCode, 0, command); const milliseconds = Date.now() - start; phases.push({ command, milliseconds }); console.log(`Completed ${language} ${target} project command: ${command} (${milliseconds} ms)`); diff --git a/vscode/test/suite.cjs b/vscode/test/suite.cjs index 098b3987..aad813f3 100644 --- a/vscode/test/suite.cjs +++ b/vscode/test/suite.cjs @@ -6,6 +6,7 @@ const { promisify } = require("node:util"); const vscode = require("vscode"); const executeFile = promisify(execFile); const { sameFile } = require("./host-support.cjs"); +const { createTaskEvidence } = require("./task-evidence.cjs"); const pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); async function bounded(promise, description, timeout = 25_000) { @@ -43,6 +44,8 @@ exports.run = async function run() { assert.ok(runId, "Isolated host run identity is required."); const report = { status: "RUNNING", runId, vscode: vscode.version, platform: process.platform, workspace, checks, skippedChecks: [], fullSuiteCompleted: false, completedModules: [] }; + const taskEvidence = createTaskEvidence(); + report.taskLifecycle = taskEvidence.report; const check = async (name, operation, timeout = 45_000) => { const start = Date.now(); const detail = await bounded(operation(), name, timeout); @@ -50,7 +53,7 @@ exports.run = async function run() { console.log(`PASS ${name}`); }; const runModule = async name => { - await require(`./${name}.cjs`).run({ check, workspace }); + await require(`./${name}.cjs`).run({ check, workspace, taskEvidence }); report.completedModules.push(name); }; const events = []; @@ -68,10 +71,16 @@ exports.run = async function run() { })().catch(error => { capture.error = String(error); }); }); const taskSubscription = vscode.tasks.onDidEndTaskProcess((event) => { + taskEvidence.event("process-end", event.execution, { exitCode: event.exitCode }); const execution = event.execution.task.execution; events.push({ task: event.execution.task.definition.task, source: event.execution.task.definition.source, exitCode: event.exitCode, command: execution?.command || execution?.process, args: execution?.args }); }); + const lifecycleSubscriptions = [ + vscode.tasks.onDidStartTask(event => taskEvidence.event("task-start", event.execution)), + vscode.tasks.onDidStartTaskProcess(event => taskEvidence.event("process-start", event.execution, { processId: event.processId })), + vscode.tasks.onDidEndTask(event => taskEvidence.event("task-end", event.execution)) + ]; try { assert.equal(vscode.workspace.isTrusted, true); const extension = vscode.extensions.getExtension("exec-software.ny-language-support"); @@ -385,11 +394,17 @@ exports.run = async function run() { } catch (error) { report.status = "FAIL"; report.error = error.stack || String(error); + try { + const directory = path.join(path.dirname(reportFile), `${runId}-task-diagnostics`); + await fs.mkdir(directory, { recursive: true }); + await taskEvidence.failure(report.error, vscode.tasks.taskExecutions, workspace, directory); + } catch (diagnosticError) { report.taskDiagnosticError = String(diagnosticError); } report.diagnostics = vscode.languages.getDiagnostics().map(([uri, items]) => ({ uri: uri.toString(), items: items.map(item => ({ source: item.source, message: item.message, line: item.range.start.line })) })); throw error; } finally { taskSubscription.dispose(); + lifecycleSubscriptions.forEach(subscription => subscription.dispose()); terminalSubscription?.dispose(); await fs.writeFile(reportFile, JSON.stringify(report, null, 2) + "\n"); } diff --git a/vscode/test/task-evidence.cjs b/vscode/test/task-evidence.cjs new file mode 100644 index 00000000..d5657230 --- /dev/null +++ b/vscode/test/task-evidence.cjs @@ -0,0 +1,77 @@ +// Test-only lifecycle evidence. This never starts, stops or retries a NIL task. +const path = require("node:path"); +const fs = require("node:fs/promises"); +const { execFile } = require("node:child_process"); +const { promisify } = require("node:util"); +const execute = promisify(execFile); + +async function inspectProcess(task, workspace, output, run = execute) { + const evidence = { processId: task.processId, expectedCommand: task.command, expectedArgs: task.args }; + if (!Number.isSafeInteger(task.processId) || task.processId <= 0 || !path.isAbsolute(task.command || "") || + !task.args?.some(value => typeof value === "string" && value.startsWith(workspace + path.sep))) { + return { ...evidence, sampled: false, reason: "No owned workspace task process declaration" }; + } + const options = { timeout: 2500, maxBuffer: 64 * 1024, encoding: "utf8" }; + try { + evidence.metrics = (await run("/bin/ps", ["-ww", "-p", String(task.processId), "-o", + "pid=,ppid=,pgid=,time=,%cpu=,stat=,etime="], options)).stdout; + evidence.command = (await run("/bin/ps", ["-ww", "-p", String(task.processId), "-o", "command="], options)).stdout.trim(); + // A task PID may already have exited. Reject a PID whose observed command + // no longer matches this run's executable and private workspace arguments. + // The command observation and subsequent sample are not atomic. + if (!evidence.command.startsWith(task.command + " ") || + !evidence.command.includes(workspace + path.sep) || !task.args.every(value => evidence.command.includes(value))) { + return { ...evidence, sampled: false, reason: "Observed process does not match the owned task" }; + } + try { await fs.access(output); return { ...evidence, sampled: false, reason: "Sample path already exists" }; } + catch (error) { if (error.code !== "ENOENT") throw error; } + const sample = await run("/usr/bin/sample", [String(task.processId), "1", "1", "-file", output], + { timeout: 6000, maxBuffer: 256 * 1024, encoding: "utf8" }); + return { ...evidence, sampled: true, sample: path.basename(output), stdout: sample.stdout, stderr: sample.stderr }; + } catch (error) { + return { ...evidence, sampled: false, error: String(error), stdout: error.stdout, stderr: error.stderr }; + } +} + +function createTaskEvidence() { + const events = [], tasks = [], dispatches = [], identities = new Map(); + const stamp = () => ({ at: new Date().toISOString(), milliseconds: Date.now() }); + const identify = execution => { + let task = identities.get(execution); + if (!task) { + const process = execution.task.execution; + task = { id: `task-${tasks.length + 1}`, definition: { ...execution.task.definition }, + command: process?.process || process?.command, args: process?.args, ended: false }; + identities.set(execution, task); tasks.push(task); + } + return task; + }; + const event = (kind, execution, extra = {}) => { + const task = identify(execution); + events.push({ kind, id: task.id, ...stamp(), ...extra }); + if (kind === "process-start") task.processId = extra.processId; + if (kind === "process-end") { task.processEnded = true; task.exitCode = extra.exitCode; } + if (kind === "task-end") task.ended = true; + }; + return { + report: { events, tasks, dispatches, failureSnapshots: [] }, + event, + dispatch(command) { + const row = { command, ...stamp(), completed: false }; dispatches.push(row); return row; + }, + complete(row, exitCode) { row.completed = true; row.exitCode = exitCode; row.elapsedMilliseconds = Date.now() - row.milliseconds; }, + async failure(reason, activeExecutions, workspace, directory, platform = process.platform, inspect = inspectProcess) { + const active = activeExecutions.map(identify); + const snapshot = { reason, ...stamp(), active: active.map(task => task.id), processes: [] }; + this.report.failureSnapshots.push(snapshot); + if (platform !== "darwin") return; + // Bounded to the three first observed active processes, never enumerate + // unrelated processes or diagnose a task from a completed execution. + for (const task of active.filter(task => !task.ended && !task.processEnded && task.processId).slice(0, 3)) { + snapshot.processes.push(await inspect(task, workspace, path.join(directory, `${task.id}-sample.txt`))); + } + } + }; +} + +module.exports = { createTaskEvidence, inspectProcess }; diff --git a/vscode/test/task-evidence.test.cjs b/vscode/test/task-evidence.test.cjs new file mode 100644 index 00000000..0b280127 --- /dev/null +++ b/vscode/test/task-evidence.test.cjs @@ -0,0 +1,70 @@ +const assert = require("node:assert/strict"); +const test = require("node:test"); +const { createTaskEvidence, inspectProcess } = require("./task-evidence.cjs"); +const fixture = () => ({ task: { definition: { task: "project-build" }, execution: { + process: "/SDK λ/nilbuild", args: ["build", "--project", "/private/work/project/nil.project.json"] } } }); + +test("task evidence associates dispatch, lifecycle and process events without changing completion", async () => { + const evidence = createTaskEvidence(), execution = fixture(); + const dispatch = evidence.dispatch("ny.buildProject"); + evidence.event("task-start", execution); + evidence.event("process-start", execution, { processId: 123 }); + evidence.event("process-end", execution, { exitCode: 0 }); + evidence.event("task-end", execution); evidence.complete(dispatch, 0); + assert.equal(evidence.report.tasks.length, 1); + assert.equal(evidence.report.tasks[0].processId, 123); + assert.equal(evidence.report.tasks[0].ended, true); + assert.ok(evidence.report.events.every(row => row.id === "task-1")); + assert.equal(dispatch.completed, true); + await evidence.failure("late", [execution], "/private/work", "/tmp", "darwin", async () => assert.fail("ended process sampled")); + assert.deepEqual(evidence.report.failureSnapshots[0].processes, []); +}); + +test("failure evidence distinguishes dispatch-only, task-created and child-started", async () => { + const evidence = createTaskEvidence(), unstarted = fixture(), started = fixture(); + evidence.dispatch("ny.buildProject"); + evidence.event("task-start", unstarted); + evidence.event("process-start", started, { processId: 321 }); + const inspected = []; + await evidence.failure("timeout", [unstarted, started], "/private/work", "/tmp", "darwin", async task => { + inspected.push(task.processId); return { processId: task.processId }; + }); + assert.deepEqual(inspected, [321]); + assert.deepEqual(evidence.report.failureSnapshots[0].active, ["task-1", "task-2"]); + assert.equal(evidence.report.dispatches[0].completed, false); +}); + +test("a process-end suppresses sampling before the task-end notification arrives", async () => { + const evidence = createTaskEvidence(), execution = fixture(); + evidence.event("process-start", execution, { processId: 123 }); + evidence.event("process-end", execution, { exitCode: 0 }); + assert.equal(evidence.report.tasks[0].ended, false); + assert.equal(evidence.report.tasks[0].processEnded, true); + await evidence.failure("waiting for task end", [execution], "/private/work", "/tmp", "darwin", + async () => assert.fail("completed child sampled")); + assert.deepEqual(evidence.report.failureSnapshots[0].processes, []); +}); + +test("diagnostics refuse an unowned PID or mismatched process before sampling", async () => { + let calls = 0; + const task = { processId: 123, command: "/SDK λ/nilbuild", args: ["build", "/private/work/project/file"] }; + const run = async (command, args) => { + calls++; assert.equal(command, "/bin/ps"); + return { stdout: args.at(-1) === "command=" ? "/other/tool /private/work/project/file" : "123 1 1 0:00 0 S 1:00" }; + }; + assert.equal((await inspectProcess(task, "/private/work", "/uncreated/output", run)).sampled, false); + assert.equal(calls, 2); + await inspectProcess({ ...task, processId: -1 }, "/private/work", "/uncreated/output", async () => assert.fail()); + await inspectProcess({ ...task, args: ["/other/work/file"] }, "/private/work", "/uncreated/output", async () => assert.fail()); +}); + +test("matching owned task receives only bounded read-only process diagnostics", async () => { + const task = { processId: 123, command: "/SDK λ/nilbuild", args: ["build", "/private/work/project/file"] }; + const calls = []; + const result = await inspectProcess(task, "/private/work", "/no-existing-task-evidence-123/sample.txt", async (command, args, options) => { + calls.push({ command, args }); assert.ok(options.timeout > 0 && options.timeout <= 6000); + return { stdout: args.at(-1) === "command=" ? "/SDK λ/nilbuild build /private/work/project/file" : "metrics", stderr: "" }; + }); + assert.equal(result.sampled, true); + assert.deepEqual(calls.at(-1), { command: "/usr/bin/sample", args: ["123", "1", "1", "-file", "/no-existing-task-evidence-123/sample.txt"] }); +}); From 7a47bf39ae7a219091f3a0137c1730fd92148b22 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 22:38:29 -0500 Subject: [PATCH 28/39] Update toolchain identity and native UI design status --- docs/analysis-api.md | 4 +-- .../NIL-0022-incremental-project-builds.md | 8 +++--- docs/design/NIL-0025-matrix-desktop-ui.md | 25 ++++++++++++++++--- docs/design/README.md | 2 +- docs/developer-tooling.md | 14 ++++++----- 5 files changed, 38 insertions(+), 15 deletions(-) diff --git a/docs/analysis-api.md b/docs/analysis-api.md index a9ad40e3..ee6e048f 100644 --- a/docs/analysis-api.md +++ b/docs/analysis-api.md @@ -177,8 +177,8 @@ Analysis reports also expose metadata for incremental build clients: | --- | --- | | `dependencies` | Sorted, unique `{path, digest}` entries for consumed native files. `path` is canonical and absolute; `digest` is lowercase hexadecimal SHA-256 of the exact consumed bytes. | | `dependenciesComplete` | True only after successful disk-based analysis with a complete, consistent dependency list containing the primary source. | -| `compilerFiles` | Sorted compiler toolchain file paths on Linux and Windows, including loaded shared libraries and known optional code inputs. Paths require content hashing; a partial list does not certify completeness. | -| `compilerFilesComplete` | Explicit boolean certification of the compiler file inventory, independent of source dependency completeness. Missing or false disables build reuse. Darwin currently reports false because loaded-image/shared-cache identity is incomplete. | +| `compilerFiles` | Sorted compiler toolchain file paths on Linux, Windows and macOS, including loaded shared libraries and known optional code inputs. Darwin includes the complete backing files for admitted dyld shared-cache images. Paths require content hashing; a partial list does not certify completeness. | +| `compilerFilesComplete` | Explicit boolean certification of the compiler file inventory, independent of source dependency completeness. Missing or false disables build reuse. Darwin certifies supported loaded-image/shared-cache profiles; unknown or unstable observations remain incomplete. | | `nativeRuntime` | Resolved native runtime archive path, or null when unavailable. | | `nativeLinker` | Resolved configured native linker driver path, or null when unavailable. | diff --git a/docs/design/NIL-0022-incremental-project-builds.md b/docs/design/NIL-0022-incremental-project-builds.md index 10d741f5..5c74ccc8 100644 --- a/docs/design/NIL-0022-incremental-project-builds.md +++ b/docs/design/NIL-0022-incremental-project-builds.md @@ -38,9 +38,11 @@ driver are identified separately for native toolchain fingerprints. `compilerFilesComplete` separately certifies the observed compiler loaded-file inventory. Missing or false certification disables reuse without discarding valid source dependency hashes or their output-collision and change protection. -Darwin currently leaves this flag false until loaded-image/shared-cache identity -is implemented. Its native link-input discovery is also incomplete. Successful -uncached builds do not establish incremental parity. +Darwin certifies supported loaded-image/shared-cache profiles, including full +backing-file content hashes. Its native link-input collector also covers the +admitted host toolchain and SDK dependencies. Real VM/native reuse and +invalidation pass on both Mac architectures; unsupported or unstable profiles +remain uncached. See [platform support](../platform-support.md) for evidence. The driver uses those inputs, effective project settings, tool identity and artifact content to decide whether compilation and linking can be skipped. diff --git a/docs/design/NIL-0025-matrix-desktop-ui.md b/docs/design/NIL-0025-matrix-desktop-ui.md index d84ae3b2..749bc294 100644 --- a/docs/design/NIL-0025-matrix-desktop-ui.md +++ b/docs/design/NIL-0025-matrix-desktop-ui.md @@ -1,9 +1,10 @@ # NIL-0025: Matrix desktop UI -- **Status:** Implemented Linux and Windows development GTK 4 profiles; macOS provider remains open +- **Status:** Implemented GTK4 on Linux, Win32 on Windows and AppKit on Intel/Apple Silicon macOS; release qualification remains separate -The original Linux design below is retained, followed by its Windows development -addendum. Current platform evidence and release limits are summarized in +The original Linux design below and dated Windows GTK addendum are retained as +design history. The current provider addendum supersedes that Windows deployment. +Current platform evidence and release limits are summarized in [platform support](../platform-support.md). ## Purpose @@ -94,3 +95,21 @@ evidence, not Windows release, bare-host or broader accessibility qualification. See the [current UI deployment and test contract](../standard-library/ui.md) and [Windows validation notebook](../../notebook/2026-11-02-windows-port.md) for the supported environment, evidence and remaining boundaries. + +## Current native providers — 2026-11-11 + +The checked `nil.std.ui` API now uses GTK4 only on Linux. Windows uses Win32 +directly, with no external GTK runtime or DLL-directory setting. macOS uses +AppKit/Cocoa and CoreGraphics, with creation and event processing on the process +main thread. The debugger keeps VM execution and UI cleanup on that same thread. +Win32 and AppKit implement row/column/grid layout behind the common boundary, +including hidden-child reflow, grid spans, expansion and fill. + +Both installed library roots pass ten interpreter/native desktop and input +cases on Windows and each Mac architecture. Tests use verified owned windows +on Windows and in-process AppKit responders on macOS. These checks supplement +the provider/lifecycle tests; they do not establish arbitrary desktop automation, +accessibility support, bare-host installation or release qualification. +See the [UI contract](../standard-library/ui.md) and +[platform matrix](../platform-support.md) for the current deployment requirements +and separately recorded editor acceptance. diff --git a/docs/design/README.md b/docs/design/README.md index 2f49752a..71e3aea3 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -51,7 +51,7 @@ not assigned architecture identifiers. | NIL-0022 | [Incremental project builds and NILData configuration](NIL-0022-incremental-project-builds.md) | Implemented initial local project graph and target scheduling profile | | NIL-0023 | [Dynamic values and Boolean vocabulary](NIL-0023-dynamic-values-and-booleans.md) | Implemented | | NIL-0024 | [Tests alongside source](NIL-0024-colocated-testing.md) | Implemented initial unit/integration profile | -| NIL-0025 | [Matrix desktop UI](NIL-0025-matrix-desktop-ui.md) | Implemented Linux and Windows development GTK 4 profiles; macOS provider remains open | +| NIL-0025 | [Matrix desktop UI](NIL-0025-matrix-desktop-ui.md) | Implemented GTK4 on Linux, Win32 on Windows and AppKit on Intel/Apple Silicon macOS; release qualification remains separate | | NIL-0026 | [Evolving the language without breaking what is written](NIL-0026-language-evolution.md) | Editions implemented in the project manifest | | NIL-0093 | [Executable context is artifact self-description](NIL-0093-executable-context.md) | Implemented within the documented inspection profile | diff --git a/docs/developer-tooling.md b/docs/developer-tooling.md index fb28d8c6..573f70db 100644 --- a/docs/developer-tooling.md +++ b/docs/developer-tooling.md @@ -28,18 +28,20 @@ dependency ordering, and bounded parallel jobs build independent projects and VM/native targets. SHA-256 input and artifact identities avoid redundant compilation only when the compiler certifies its source and loaded-code input graphs; native reuse also requires a complete linker/toolchain graph. Every -invocation still performs fresh analysis. Darwin currently rebuilds both VM -and native outputs while loaded-image/shared-cache identity is incomplete, -preserving source protection checks. See the +invocation still performs fresh analysis. Darwin now hashes admitted loaded +images, shared-cache backing files and native linker inputs; actual VM/native +reuse and invalidation pass on both Mac architectures. Unsupported profiles +rebuild, preserving source protection checks. See the [incremental build contract](tools/projects.md#incremental-and-parallel-builds). A single executable remains one compiler job over its complete import graph. Source debugging is implemented through `nildebugger --dap` and VS Code launch configurations for Ny, Matrix and debug-enabled NIL. Source breakpoints, stepping, frames, locals and streamed output use the interpreter's actual -execution state. The paired Windows/Linux `5630388` development baseline -includes installed debugger acceptance, with VS Code acceptance recorded for -Windows. See [platform support](platform-support.md) and the +execution state. Development SDKs on Linux, Windows and both Mac architectures +pass installed debugger and same-process execution-checkpoint acceptance. +See [platform support](platform-support.md) for separate installed editor +results and remaining checks, and the [source-debugging guide](tools/index.md#source-debugging-and-vs-code). Workspace source requests now let the compiler analyze open unsaved dependencies From 85ce6ce1b434fed66cec2905faa063de6c1d5c1e Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 23:04:05 -0500 Subject: [PATCH 29/39] Use Darwin path rules in portable task diagnostic controls --- vscode/test/task-evidence.cjs | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/vscode/test/task-evidence.cjs b/vscode/test/task-evidence.cjs index d5657230..b741dc88 100644 --- a/vscode/test/task-evidence.cjs +++ b/vscode/test/task-evidence.cjs @@ -7,8 +7,10 @@ const execute = promisify(execFile); async function inspectProcess(task, workspace, output, run = execute) { const evidence = { processId: task.processId, expectedCommand: task.command, expectedArgs: task.args }; - if (!Number.isSafeInteger(task.processId) || task.processId <= 0 || !path.isAbsolute(task.command || "") || - !task.args?.some(value => typeof value === "string" && value.startsWith(workspace + path.sep))) { + // These diagnostics run only on Darwin. Keep their path contract explicit + // so the same mocked controls also exercise it on Windows. + if (!Number.isSafeInteger(task.processId) || task.processId <= 0 || !path.posix.isAbsolute(task.command || "") || + !task.args?.some(value => typeof value === "string" && value.startsWith(workspace + path.posix.sep))) { return { ...evidence, sampled: false, reason: "No owned workspace task process declaration" }; } const options = { timeout: 2500, maxBuffer: 64 * 1024, encoding: "utf8" }; @@ -20,7 +22,7 @@ async function inspectProcess(task, workspace, output, run = execute) { // no longer matches this run's executable and private workspace arguments. // The command observation and subsequent sample are not atomic. if (!evidence.command.startsWith(task.command + " ") || - !evidence.command.includes(workspace + path.sep) || !task.args.every(value => evidence.command.includes(value))) { + !evidence.command.includes(workspace + path.posix.sep) || !task.args.every(value => evidence.command.includes(value))) { return { ...evidence, sampled: false, reason: "Observed process does not match the owned task" }; } try { await fs.access(output); return { ...evidence, sampled: false, reason: "Sample path already exists" }; } From b6ad3a5eaf4e5ff75d49a551a6bba083f925c32b Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 23:04:05 -0500 Subject: [PATCH 30/39] Retain bounded passing output for SDK packaging contracts --- .github/workflows/macos-application-parity.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index 5477e533..068dc611 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -244,6 +244,7 @@ jobs: assert {case['name'] for case in json.loads(inventory)['tests']} == expected with (root/'sdk-contracts.log').open('wb') as output: run = subprocess.run(base+['--no-tests=error', '--output-on-failure', + '--test-output-size-passed', '65536', '--output-junit', str(root/'sdk-contract-results.xml')], stdout=output, stderr=subprocess.STDOUT, timeout=1750) print((root/'sdk-contracts.log').read_text(errors='replace')) From 43fcaf911ceffc8d35e015a239961d6566524363 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Wed, 11 Nov 2026 23:07:29 -0500 Subject: [PATCH 31/39] Record completed four-host development SDK acceptance --- README.md | 16 +-- docs/platform-support.md | 21 +++- notebook/2026-11-11-web-runtime-parity.md | 115 ++++++++++++++++++++++ 3 files changed, 142 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 1e1743fb..34ecebce 100644 --- a/README.md +++ b/README.md @@ -253,14 +253,14 @@ source/binary routes, 146 debugger checks and same-process execution checkpoints Linux's current SDK also passes all 590 source tests, twenty installed GTK desktop/input cases and all 88 real VS Code host groups without exclusions. -Windows uses Win32 and macOS uses AppKit. Windows and Apple Silicon also pass -twenty installed desktop/input cases, four desktop debugger sessions and all -88 editor groups. Both Macs pass the expanded 21-contract application campaign, -native incremental reuse and all 88 build-tree editor groups; the earlier Intel -project-tool timeout is resolved. Intel's installed UI/debugger checks pass, -but its editor run timed out during a project build and remains under -investigation. A separate Windows account exposed a packaged-library permission -defect; the correction now passes real cross-account access tests. +Windows uses Win32 and macOS uses AppKit. Windows and both Mac architectures +also pass twenty installed desktop/input cases, four desktop debugger sessions +and all 88 editor groups. Both Macs pass the expanded application campaign and +native incremental reuse. An earlier Intel installed-editor timeout passed a +diagnostic rerun within the unchanged deadline; its cause remains unconfirmed +and the failed report is retained. A separate Windows account exposed a +packaged-library permission defect; the correction passes real cross-account +access tests. The newer socket, entropy and text-byte APIs pass the same interpreter contracts on every host; these APIs currently require the interpreter on all platforms. diff --git a/docs/platform-support.md b/docs/platform-support.md index 7a1a5c8f..6e6e1078 100644 --- a/docs/platform-support.md +++ b/docs/platform-support.md @@ -11,8 +11,8 @@ boundary. A source update does not update an installed SDK. | --- | --- | --- | | Linux x86_64 | Clean `9fa354d` SDK: all 590 source CTests pass without skips; 1,184 installed consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty GTK desktop/input cases and all 88 editor groups pass. | Qualified in a private consumer environment. The reviewed Ubuntu release toolchain and shared deployment permissions require their own gates. | | Windows x86_64 | Clean `9fa354d` SDK: 588 of 590 source CTests pass, with two headless display skips subsequently exercised successfully in Nathan's desktop. Relocated SDK passes all 1,184 consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty Win32 desktop/input cases, four desktop DAP sessions and all 88 editor groups. | The candidate needed a recorded, scoped read/execute grant on `nillib`. The permanent packaging correction separately passes real cross-account access tests; older SDKs do not acquire it automatically. Bare-host installation and release qualification remain open. | -| macOS 15 Intel x86_64 | Relocated `1efbe66` SDK passes all 1,184 consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty installed AppKit desktop/input cases and four desktop DAP sessions. Current `eca43f1` source fixtures pass all 21 application/tooling CTests and all 88 build-tree editor groups. | Installed editor acceptance stops after 25 passing groups at a VM project-build deadline; diagnosis remains open. The original full source report retains four failures resolved by later fixtures and one ELF-only execution skip. Shared deployment and release qualification remain open. | -| macOS 15 Apple Silicon arm64 | Relocated `1efbe66` SDK passes all 1,184 consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty installed AppKit desktop/input cases, four desktop DAP sessions and all 88 editor groups. Current `eca43f1` fixtures also pass all 21 application/tooling CTests. | The original full source report retains four failures and one ELF-only execution skip; subsequent hosted checks resolve the four fixture failures. Shared deployment and release qualification remain open. | +| macOS 15 Intel x86_64 | Relocated `1efbe66` SDK passes all 1,184 consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty installed AppKit desktop/input cases, four desktop DAP sessions and all 88 editor groups. Current `59ffa97` source fixtures pass all 22 application/tooling/packaging CTests. | The installed editor needed a diagnostic rerun after a project-build timeout; it passed unchanged deadlines, with the original cause unconfirmed. The original full source report retains four failures resolved by later fixtures and one ELF-only execution skip. Shared deployment and release qualification remain open. | +| macOS 15 Apple Silicon arm64 | Relocated `1efbe66` SDK passes all 1,184 consumer routes, twelve PostgreSQL routes, 146 debugger checks, execution checkpoints, twenty installed AppKit desktop/input cases, four desktop DAP sessions and all 88 editor groups. Current `59ffa97` source fixtures also pass all 22 application/tooling/packaging CTests. | The original full source report retains four failures and one ELF-only execution skip; subsequent hosted checks resolve the four fixture failures. Shared deployment and release qualification remain open. | The SDKs contain eleven root commands, eighteen compiled standard-library modules, their source, native runtimes and HTML documentation. The installed @@ -30,6 +30,15 @@ profile. Both current Mac application campaigns use VS Code 1.146.0. Each passes and no exclusions. Editor results using build-tree tools are distinguished from results using an installed SDK. +Installed desktop/editor coverage is complete on both Mac architectures: +[ARM run 43146638765](https://github.com/exec-software/exec-nil-framework/actions/runs/43146638765) +and [Intel run 43149252918](https://github.com/exec-software/exec-nil-framework/actions/runs/43149252918). +The first Intel editor attempt timed out after 25 groups. With task lifecycle +diagnostics added, all 88 groups pass using unchanged SDK bytes and deadlines; +the Ny VM project group takes 161.171 seconds of its 180-second allowance. +All 31 observed task lifecycles complete. This establishes successful installed +execution, not a root-cause fix for the earlier timeout or a performance promise. + Exact revisions, evidence hashes, host preparation, interrupted attempts and failure follow-ups are retained in the [current parity notebook](../notebook/2026-11-11-web-runtime-parity.md). @@ -57,6 +66,14 @@ These results supplement the unchanged SDK; the original failed reports remain failed. The ELF-only freestanding executable fixture remains an explicit format-specific skip. Actual Mach-O archives separately pass import validation. +The next campaign at `59ffa97` adds mixed-source/native SDK packaging, including +the destination-access regression: all 22 selected CTests, four desktop DAP +sessions and all 88 editor groups pass per Mac architecture +([run 43148956133](https://github.com/exec-software/exec-nil-framework/actions/runs/43148956133)). +The exact pinned thirteen-test packaging invocation passes; its original JUnit +output retains only eight complete result lines because CTest truncates passing +output at 1,024 bytes. Future runs retain a bounded 64 KiB per SDK/project test. + Windows's former transient native output-replacement failure is corrected at `9fa354d`: compiler and linker share bounded Windows sharing/lock retries and preserve existing output on persistent failure. Strict MSVC controls exercise diff --git a/notebook/2026-11-11-web-runtime-parity.md b/notebook/2026-11-11-web-runtime-parity.md index 2353feb4..1d2b2392 100644 --- a/notebook/2026-11-11-web-runtime-parity.md +++ b/notebook/2026-11-11-web-runtime-parity.md @@ -644,6 +644,23 @@ and its reproductions must be checked against this new source before making further web changes. The README's remaining-parity paragraph still lists logs, configuration and virtual hosts as absent despite documenting them elsewhere. +The clean `41eb756` handoff has now been checked against the sealed Linux +`9fa354d` interpreter: all 653 existing checks pass from a disposable source +copy, with four raw streams verified, all 49 repository files and all 268 SDK +files/modes unchanged. Current code still contains the four historical +mechanisms: permissive length/transfer framing, discarded pipelined surplus, +Range handling before conditionals without If-Range/HEAD gating, and lexical-only +static paths. These are code-level findings at the new revision, not fresh HTTP +network reproductions. The existing `read_regular_bytes` protects the final +path component only and cannot by itself establish root containment through +intermediate directories. No diagnostic server was started; the existing suite's +bounded loopback bind/poll/close case completed normally. +Evidence `web-readonly-41eb756/handoff.json`, SHA-256 +`092f5c6f1605e2e58620480c2d447f9221fcf94da3c0be2df3989b175a43b2f5`, +retains source lines and focused follow-ups. Runtime CI currently remains absent; +the web workflow runs source audits. Framework platform acceptance remains the +first priority before changing this consumer. + ## Current installed Linux closeout and Mac source follow-up @@ -792,3 +809,101 @@ was retained, preventing attribution to compilation, fingerprinting or task launch. All 211 evidence files verify and input seals remain unchanged. The failure remains recorded; bounded task lifecycle diagnostics and a same-deadline Intel follow-up are required before installed editor parity is complete. + + +The Intel-only diagnostic follow-up +[43149252918](https://github.com/exec-software/exec-nil-framework/actions/runs/43149252918) +passes with helper `fc782892e1655601830ae20273d0b638f3e93501` and the exact same +SDK, prerequisites and group deadlines. All twenty UI cases, four desktop DAP +sessions, 146 debugger checks/15 sessions, checkpoints and all 88 editor +groups/all nine modules/no exclusions pass. Independent replay verifies 213 +current evidence files, all 920 original evidence hashes, all 268 SDK files and +unchanged helper/editor inputs. Report +`github-sdk-desktop-43149252918/replay-report.json`, SHA-256 +`558c7714f5bc03ecd4ba367abd9cd0681b7755e192438f8ac39c5125b1011d9e`. + +The new test-only lifecycle collector records all 31 task executions with four +ordered events each (124 events), and all twelve project command dispatches +complete. Ny VM check/build/run takes 161.171 seconds within 180 seconds: +1.426 seconds checking, 128.389 building and 31.354 running. Ny native takes +241.450 seconds within 300; Matrix VM/native take 56.122/112.599 seconds. +The entire editor completes in 759.587 seconds within 900. The collector changes +no product behavior, test assertions, group inventory or deadline. Its five +controls and all 167 extension unit tests pass. It only diagnoses still-active, +matching workspace task processes on failure, with bounded read-only snapshots. + +The original timeout remains unexplained; no task-event-loss defect was +reproduced or fixed. These timings show that first Intel project builds can +consume much of the fixture budget. Future compiler/build-performance work +should measure the loaded-image/shared-cache hashing phases directly before +changing cache identity or timing policy. The passing diagnostic follow-up +completes same-user installed Mac desktop/editor coverage together with ARM's +original pass; it does not qualify shared deployment or a stable release. + + +## Final portability and evidence-retention checks + +The test-only Darwin task diagnostics originally used the current host's path +separator. Their POSIX mock inputs therefore failed two controls on Windows. +At `85ce6ce`, the Darwin-only inspector explicitly uses POSIX path validation; +its real Darwin behavior is unchanged. Actual Windows Node reproduces three +passes/two failures with the old helper, then passes all five controls with the +correction. The complete unit suite records 166 passes, no failures and its one +expected POSIX FIFO skip from 167 registrations. All eight raw streams and 103 +copied fixture source pins verify, with no SDK rebuild, real process sampling +or desktop/editor repeat. Report +`sdk-9fa354d48575/diagnostic-unit-6d55b84f08/report.json`, SHA-256 +`f47059c8c8d97f0de50a80ce2f367dd04041c8df177ad7884d973ac3b0cbaf13`. + +The packaging CTest's successful JUnit output was truncated at the default +1,024 bytes in the Mac campaign. Its exact pinned thirteen-test `--native` +invocation and success exit are retained, but only eight complete named result +lines survive in the raw stream. At `b6ad3a5`, future SDK/project contract runs +retain up to 64 KiB of passing output per test. A real bounded local CTest +control proves the default loses a trailing inventory marker beyond 4 KiB, +while the new bound retains it; neither changes the test verdict. Evidence is +`ctest-passing-output-n20zbgqp/report.json`. Existing truncated evidence is not +rewritten or represented as a full independent replay of thirteen result lines. + +Both Linux source workflows at feature `59ffa97` succeed: push +[43148948074](https://github.com/exec-software/exec-nil-framework/actions/runs/43148948074) +and PR [43148956132](https://github.com/exec-software/exec-nil-framework/actions/runs/43148956132), +tested merge `7c552ddcd2261891ce388d04d687d0bc0e5422d6`. Independent replay of all +six authenticated job logs finds identical results: LLVM ON 590 passes/no skips; +LLVM OFF 441 passes and 143 expected disabled-native skips from 584; extension +162 passes/no skips. The workflow emits logs rather than a JUnit artifact. +Report `source-59ffa97-monitor/linux-source-replay.json`, SHA-256 +`34d656cc4036d9b8814ad0ed69850cd94fa6146ca7cb1cb494531b1b16fa46dd`. +The corresponding [four-host web-runtime run 43148956115](https://github.com/exec-software/exec-nil-framework/actions/runs/43148956115) +also completes successfully on Linux, Windows, Intel macOS and Apple Silicon. + +The web README correction is committed and pushed as `2d3abee`, retaining the +unchanged `41eb756` implementation and its verified 653-check baseline. It +removes obsolete missing-feature claims, records implemented logs/configuration/ +virtual hosts, and states the four remaining HTTP/filesystem gaps. Its source +audit CI passes. GitHub accepted the direct master push but reported a bypass +of a PR-only rule; future web changes must use a branch and pull request. + + +The final expanded Mac source campaign +[43148956133](https://github.com/exec-software/exec-nil-framework/actions/runs/43148956133) +passes on both architectures at feature `59ffa974632dc45c79c50fa04d7f7038db3f79eb`, +tested merge `7c552ddcd2261891ce388d04d687d0bc0e5422d6`. Per host, the exact +inventory is six SDK/project/packaging contracts, six debugger contracts, +eight native contracts and two incremental contracts: all 22 pass without skips. +Four naturally exiting desktop DAP sessions and all 88 exact-name editor +groups/all nine modules/full-suite completion/no exclusions also pass on each +host. All 620 evidence hashes, official artifact ZIP digests/sizes and source/ +merge-parent bindings verify. Report +`github-applications-43148956133/replay-report.json`, SHA-256 +`39eaacc2159c8ed91a35f418636e862ec13d2a8dd56d4d698d7b449609a0aced`. +The thirteen-test packaging invocation passes with native consumers enabled on +both hosts, subject to the original raw-output retention limit recorded above. + +Installed development coverage is now complete across Linux, Windows, Intel +macOS and Apple Silicon for the supported common profiles, using the exact +candidate/helper identities in this notebook. The Intel timeout history and +platform-specific release/deployment limits remain explicit. No accepted SDK, +stable release tag, release service or Lich publication was changed. The web +handoff retains its current code, passing existing suite and four identified +behavior gaps for the next implementation work. From 8b042e94292a44d0de0ab25d6bac1d8962c291bc Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Fri, 13 Nov 2026 00:35:38 -0500 Subject: [PATCH 32/39] Document formatted text, queries and patterns in the Matrix guide The guide is the reference an agent writes Matrix from, and it described none of NIL-0014's formatted literals or NIL-0016's queries and text patterns, so generated code still builds text with plus ladders and hand-rolled loops. Each added example is compiled and run: the formatted literals join guide-constructs, and the query and pattern sections get a standard-library fixture of their own. Co-Authored-By: Claude Opus 5 --- matrix/MATRIX-GUIDE.md | 106 ++++++++++++++++++ .../execution/guide-constructs.expected | 3 + .../matrix/execution/guide-constructs.matrix | 11 ++ .../execution/guide-queries-patterns.expected | 10 ++ .../execution/guide-queries-patterns.matrix | 47 ++++++++ .../guide-queries-patterns.needs-stdlib | 0 6 files changed, 177 insertions(+) create mode 100644 tests/matrix/execution/guide-queries-patterns.expected create mode 100644 tests/matrix/execution/guide-queries-patterns.matrix create mode 100644 tests/matrix/execution/guide-queries-patterns.needs-stdlib diff --git a/matrix/MATRIX-GUIDE.md b/matrix/MATRIX-GUIDE.md index 6cf87724..5b64fdcc 100644 --- a/matrix/MATRIX-GUIDE.md +++ b/matrix/MATRIX-GUIDE.md @@ -127,6 +127,25 @@ end text The common indentation is removed, deeper indentation kept. The one limitation: a block cannot contain a line that is exactly `end text`. +**Interpolation is `formatted`.** The word in front of a quoted literal opts +that literal into holes. Each `{...}` is an ordinary Matrix expression, +evaluated once, in written order: + +```matrix +value summary is formatted "Count: {count plus 1}; who: {who}" +say formatted "Result: {choose with first being "one", second being "two"}" +say formatted "{{literal braces}} and {count}" +``` + +Scalars and text convert themselves; any other type needs an explicit +conversion first. There is no format-specifier language inside the braces — +precision and padding are library calls. `{{` and `}}` are literal braces in +this form only: an ordinary literal and a `begin text` block stay raw, and a +backslash stays a backslash in both. A formatted literal and its expressions +must fit on one physical source line, so interpolate `newline` for a line +break. `formatted` remains an ordinary name anywhere it is not followed by a +quote. + There are no character literals. A one-letter text value is `"a"`. --- @@ -746,6 +765,10 @@ contract method bodies, supertraits, associated types, `i16`/`u16`/`u32`/`u64`/ Reserved but unimplemented: `chapter` (a third namespace level), `forever` as a `static` modifier, and `requires` for declaring a book's authorities. +A `formatted` literal is one physical source line: multi-line interpolated text +is not available yet. Build it from `newline`, or concatenate a raw `begin text` +block with `plus`. + Matrix and Ny share the interpreter and native routes. Native aggregate-array and foreign-call limitations apply to both languages. @@ -805,3 +828,86 @@ out-of-range tuple-array read fails because there is no default tuple value. Native arrays of records, choices and nested arrays remain unsupported. See the shared [tuple reference](../docs/language-reference/tuples.md) for the exact contract. + +--- + +## 20. Queries + +`collect begin query` filters, orders, pages and reshapes an array in one +expression, and collects the result into a new array: + +```matrix +value top is collect begin query + from each score in scores + where score is greater than 10 + order by score descending + take 2 + give score times 2 +end query +``` + +Clauses appear at most once, in exactly this order: `from each`, `where`, +`order by`, `skip`, `take`, `give`. Only `from each` and `give` are required. +Out-of-order, duplicate and missing clauses are errors, and the clause words +stay ordinary names everywhere else. + +`order by` takes one scalar or text key, with `ascending` (the default) or +`descending`; equal keys keep their input order. The source expression is +evaluated once and snapshotted, so mutating the original array mid-query +cannot change which values are enumerated. The row name — `score` above — +is visible only inside the query's own clauses. + +Execution is **eager**: this is a loop the compiler writes for you, not a lazy +iterator chain and not a database query. The source must be an array of +scalars, text or tuples of those; records, choices and nested arrays are +outside this profile. + +`nil.std.collections` holds the terminals that inspect a collected result: +`query-count`, `query-any`, `query-first` and `query-last` (an `Option`), and +`query-single` (a `Result`, with distinct empty and multiple errors). They read +an array that already exists — they do not re-run the query — so use `take 1` +rather than collecting everything to look at the first row. + +--- + +## 21. Text patterns + +A pattern declaration spells a regular expression in words, so no page has to +carry escape-dense pattern source: + +```matrix +dollar-amount begin pattern + capture amount matching one or more ascii digits + require next to match literal " USD" +end pattern +``` + +The declaration is an immutable **text constant** holding the pattern source — +not a new type — and `nil.std.text` compiles and runs it: + +```matrix +read everything from nil.std.text + +value compiled is (compile_pattern with source being dollar-amount).unwrap with nothing +value found is ((first_match with pattern being compiled, + input being "price: 12 USD").unwrap with nothing).unwrap with nothing +say (capture_named with found being found, name being "amount").unwrap with nothing +``` + +Atoms are `literal "..."` (also `literal newline`, `literal tab`, +`literal quote`), `ascii digit`/`digits`/`letter`/`letters`/`whitespace`, +`any character`, `character from "0" through "9"`, `start of input`, +`end of input`, `capture matching `, +`require next [not] to match ` for lookahead, `either or ` +with further `or` arms, and a nested `begin pattern` ... `end pattern` block +that makes a whole sequence into one atom. Repetition goes in front of an atom: +`one or more`, `zero or more`, `optional`, `exactly 3`, `between 2 and 4`, and +`fewest` before any of those makes it lazy. Counts stop at 1000. Adjacent +phrases are a sequence. + +`compile_pattern` also accepts conventional notation, so +`"(?[0-9]+)(?= USD)"` is the same pattern. Matching is byte-oriented +and case-sensitive. Backreferences, lookbehind, flags, word boundaries and +Unicode properties are rejected rather than silently given another meaning. +Matrix lowercases capture labels and turns hyphens into underscores; look them +up by the resulting name. diff --git a/tests/matrix/execution/guide-constructs.expected b/tests/matrix/execution/guide-constructs.expected index ae82df4f..fb3a2d80 100644 --- a/tests/matrix/execution/guide-constructs.expected +++ b/tests/matrix/execution/guide-constructs.expected @@ -47,4 +47,7 @@ cleanup 255 255 3 +Count: 4; who: Nathan +Result: one +{literal braces} and 3 1 diff --git a/tests/matrix/execution/guide-constructs.matrix b/tests/matrix/execution/guide-constructs.matrix index ad2f1d53..ac510925 100644 --- a/tests/matrix/execution/guide-constructs.matrix +++ b/tests/matrix/execution/guide-constructs.matrix @@ -79,6 +79,11 @@ end paragraph answer returns number begin paragraph give 42 end paragraph +choose receives first as text, second as text returns text begin paragraph + when (the length of first) is greater than 0 then give first end when + give second +end paragraph + total-of receives more values as number returns number begin paragraph variable sum as number is 0 variable i as number is 0 @@ -235,6 +240,12 @@ main returns number begin paragraph value encoded is apply encode-sample across pixels say encoded at 0 + value who as text is "Nathan" + value tally as number is 3 + say formatted "Count: {tally plus 1}; who: {who}" + say formatted "Result: {choose with first being "one", second being "two"}" + say formatted "{{literal braces}} and {tally}" + value started as override long is now in milliseconds sleep for 1 milliseconds say (now in milliseconds) is at least started diff --git a/tests/matrix/execution/guide-queries-patterns.expected b/tests/matrix/execution/guide-queries-patterns.expected new file mode 100644 index 00000000..397376da --- /dev/null +++ b/tests/matrix/execution/guide-queries-patterns.expected @@ -0,0 +1,10 @@ +2 +110 +82 +4 +1 +110 +82 +1 +12 +12 diff --git a/tests/matrix/execution/guide-queries-patterns.matrix b/tests/matrix/execution/guide-queries-patterns.matrix new file mode 100644 index 00000000..f5f7574a --- /dev/null +++ b/tests/matrix/execution/guide-queries-patterns.matrix @@ -0,0 +1,47 @@ +!!! Verbatim constructs from MATRIX-GUIDE.md sections 20 and 21, which need the +!!! standard library: queries read nil.std.collections terminals and patterns +!!! are compiled by nil.std.text. +book tour.guide-queries + +read everything from nil.std.text +read everything from nil.std.collections + +dollar-amount begin pattern + capture amount matching one or more ascii digits + require next to match literal " USD" +end pattern + +main returns number begin paragraph + value scores as array of number is [41, 7, 19, 55] + value top is collect begin query + from each score in scores + where score is greater than 10 + order by score descending + take 2 + give score times 2 + end query + say the length of top + say top at 0 + say top at 1 + + value plain is collect begin query + from each score in scores + give score + end query + say query-count with items being plain + say query-any with items being top + say (query-first with items being top).unwrap with nothing + say (query-last with items being top).unwrap with nothing + say (query-single with items being top).is-error with nothing + + value compiled is (compile_pattern with source being dollar-amount).unwrap with nothing + value found is ((first_match with pattern being compiled, + input being "price: 12 USD").unwrap with nothing).unwrap with nothing + say (capture_named with found being found, name being "amount").unwrap with nothing + + value conventional is (compile_pattern with source being "(?[0-9]+)(?= USD)").unwrap with nothing + value same is ((first_match with pattern being conventional, + input being "price: 12 USD").unwrap with nothing).unwrap with nothing + say (capture_named with found being same, name being "amount").unwrap with nothing + give 0 +end paragraph diff --git a/tests/matrix/execution/guide-queries-patterns.needs-stdlib b/tests/matrix/execution/guide-queries-patterns.needs-stdlib new file mode 100644 index 00000000..e69de29b From d10f025060ef5bc8a2d6e8a850cdc7ae0a0f1f42 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Fri, 13 Nov 2026 00:35:38 -0500 Subject: [PATCH 33/39] Propose NIL-0027 structured markup literals A generator currently assembles a document out of fragments, which puts the control flow outside the markup, leaves the structure unchecked and makes escaping manual and context-blind. NIL-0027 stages the fix: a multiline formatted block first, then a markup literal that holds its own conditionals and loops and evaluates to a checked tree rather than text. The record settles the four decisions taken with it: raw-text element bodies stay literal, the parser carries no dialect, the value is a tree rendered by nil.std.markup, and the amortized text builder lands first. It also records that the obvious node-record representation cannot execute natively, so a markup value is an arena of scalar and text arrays. Co-Authored-By: Claude Opus 5 --- docs/design/NIL-0027-structured-markup.md | 282 ++++++++++++++++++++++ docs/design/README.md | 1 + 2 files changed, 283 insertions(+) create mode 100644 docs/design/NIL-0027-structured-markup.md diff --git a/docs/design/NIL-0027-structured-markup.md b/docs/design/NIL-0027-structured-markup.md new file mode 100644 index 00000000..bec26e4c --- /dev/null +++ b/docs/design/NIL-0027-structured-markup.md @@ -0,0 +1,282 @@ +# NIL-0027: Multiline formatted text and structured markup literals + +- **Status:** Proposed. Staged; nothing in this record ships yet. +- **Scope:** Ny and Matrix syntax, shared hole lexing and AST, checking, + lowering, `nil.std.markup`, text builder +- **Date:** 2026-11-13 + +## Context + +A program that generates a document today concatenates one. The shape recurs +in every generator written against this framework: + +```text +value nav is when signed-in + give "Dashboard | Projects | Logout" + otherwise give "Login | Register" +end when +value html is "" plus title plus "" +give html plus "" +``` + +Three separate defects, and only the first one is about typing: + +1. **The document is not editable as a document.** Its structure is spread + across bindings and `plus` chains, so a change to the markup is a change to + control flow. This is the failure mode of a generation of server-rendered + business applications, where one function is a ladder of conditionals each + writing fragments of tags. +2. **Nothing checks the structure.** `"
" plus x` cannot be known to be + balanced, so an unclosed element is discovered by a browser. +3. **Escaping is manual and context-blind.** The correct escape for a value + differs between a text node, an attribute value and a URL attribute. A + concatenated document applies none of them, and an interpolated string + cannot apply the right one either, because it does not know where the hole + landed. + +NIL-0014 introduced explicit holes but confined a formatted literal to one +physical source line. Raw `begin text` blocks are multiline but deliberately +inert. Neither carries structure. + +NIL-0016 is the precedent for the answer. A text pattern is not a string that +happens to be a regular expression, and a query is not a loop the author wrote +out; both are declared in the language's own words and lower to a checked +value. **A document deserves the same treatment,** and this record is the third +member of that family. + +## 1. One hole grammar, three levels of structure + +| Surface | Holes | Structure | Status | +| --- | --- | --- | --- | +| `formatted "..."` | yes | none, one line | NIL-0014, implemented | +| `formatted begin text ... end text` | yes | none, multiline | §2 of this record | +| `begin markup ... end markup` | yes | elements, attributes, control flow | §3 of this record | + +All three use the hole lexer NIL-0014 already built, which emits an expression's +tokens with their original byte offsets rather than re-lexing decoded text. +Everything in this record inherits that contract, and no construct here +introduces an expression language of its own: a hole holds an ordinary +expression in its frontend's grammar, and nothing else. + +## 2. Multiline formatted text + +```text +value page is formatted begin text +

Signed in as {who}

+

{count plus 1} projects

+end text +``` + +Ny spells the same construct `$"""` ... `"""`. + +The block follows the existing raw-block rules exactly — line-anchored +terminator, common leading indentation removed, deeper indentation kept, +leading and trailing blank lines dropped, no escape sequences — and adds +`{expression}` holes with `{{` and `}}` for literal braces. A single hole +stays on one physical line, which keeps the expression lexer unchanged. + +The one implementation constraint, named as follow-up work in NIL-0014: **the +dedent must not move the holes.** Indentation removal applies to literal +segments at lowering. The lexer records each segment's original span and the +number of stripped bytes per line, and hole tokens keep their original +coordinates, so definition, completion, signature help and local rename behave +as they do inside a single-line formatted literal. + +This is the smallest useful step and it is worth taking on its own: it is what +a C# raw interpolated string gives, and it removes most `plus` ladders from +ordinary code. It does not solve §3's problem, because the control flow still +lives outside the document. + +## 3. Markup literals + +```text +public page-shell receives title as text, signed-in as boolean, who as text + returns markup begin paragraph + give begin markup + + {title} + + + + + end markup +end paragraph +``` + +The conditional is *inside* the document, where the variation actually is. The +page remains one unit a person can read, edit and paste. + +### 3.1 What the block contains + +Elements `children`, self-closing ``, text, +comments, holes, and directives. Directives are spelled with the language's own +control words, so the block introduces no second vocabulary: + +```text +
    + {repeat for each p in projects} +
  • {p.name}
  • + {end repeat} +
+``` + +`{when ...}` / `{otherwise when ...}` / `{otherwise}` / `{end when}` and +`{repeat for each ... in ...}` / `{repeat for ... from ... to ...}` / +`{end repeat}` are the complete directive set. A directive may stand between +elements, between attributes, or inside a text node or attribute value. Its +arms must each be well-formed: a directive cannot open an element in one arm +and close it in another, because that reintroduces exactly the fragment +assembly this record exists to remove. + +**A hole cannot name an element or an attribute.** Both are fixed at compile +time, which is what makes the tree checkable. Dynamic attribute sets are +future work (§6). + +### 3.2 Hole typing, by position + +| Position | Accepted | Notes | +| --- | --- | --- | +| Text node | `text`, scalar, `markup`, `array of markup`, `nothing` | `markup` splices as nodes, never as escaped text | +| Attribute value | `text`, scalar, `nothing` | `nothing` omits the attribute entirely | +| URL attribute | `text` | scheme-checked; see §3.4 | +| Raw-text element body | `text` via `{raw ...}` only | see §3.3 | + +`{raw expression}` splices text without escaping. It is deliberately a word a +reviewer can grep for, and it is the only way to put unescaped text into a +document. + +### 3.3 Decision: raw-text elements are literal + +CSS and JavaScript are full of braces, and an author must be able to paste +them. Inside ` +``` + +The rule is a single lookahead over the element body and needs no escaping +convention, which keeps the no-escapes property of Matrix text intact in the +place where escaping would hurt most. + +### 3.4 Decision: the parser is dialect-free + +The block parses generic elements, attributes, text, comments and holes, and +checks balance. Everything specific to HTML lives in one table the **renderer** +owns: void elements, raw-text elements, boolean attributes, URL-bearing +attributes, and the escape sets. The parser accepts `
`; an HTML render +emits `
`, an XML render emits `
`. + +This is the difference between adding a construct and welding a document format +into the compiler. `begin markup as xml` then costs a table, not a grammar. + +Escaping is per context and belongs to the same table: + +- text node: `&`, `<`, `>` +- attribute value: `&`, `<`, `>`, `"`; output is always double-quoted +- URL attribute (`href`, `src`, `action`, `formaction`, `cite`, `poster`, + `data`, `srcset`): percent-encoded per component, with a scheme allowlist. + A hole producing a `javascript:` or `data:` URL is refused at run time unless + it came through `{raw}`. + +### 3.5 Decision: the value is a tree, not text + +`begin markup` evaluates to `markup`. `nil.std.markup::render with document +being page` produces `text`; `render ... as xml` and a `compact` option select +serialization. + +A string result would be simpler and would give up all four of the reasons for +doing this at all: + +1. Context-correct escaping requires knowing where a value sits. +2. Balance is checkable only while the structure still exists. +3. **Composition stays structural.** A paragraph returning `markup` + interpolates into another block as nodes — `
{page-header with title + being title}
` — so factoring a page into components does not route + back through text, and nothing is escaped twice. +4. A tree can be rendered to HTML, rendered to XML, or diffed. The last is + what a future webview host wants, and it is unavailable from a string + forever after. + +### 3.6 Representation: an arena, not a record graph + +The obvious implementation — a `node` record holding `array of node` children — +does not execute. Native arrays of records, choices and nested arrays are +unsupported (MATRIX-GUIDE §17), and an array of class handles compiles and then +fails at run time (§16). A tree of record handles would therefore be +interpreter-only, which is not acceptable for the one construct whose whole +purpose is to be used everywhere. + +A `markup` value is instead an **arena**: parallel arrays of scalars +(`kind`, `name-id`, `parent`, `first-child`, `next-sibling`, `text-id`) beside +a text pool `array of text`. Arrays of scalars and arrays of text are supported +on both routes today. Composition is an append of one arena into another with +an index offset. Rendering is one pre-order walk. + +This also gives the compile-time optimisation for free: a literal's static +subtrees have known shape, so the compiler emits their serialized bytes as +constant segments and the render becomes a sequence of appends with the holes +patched in. A page with twelve holes costs about twelve appends, not a tree +traversal. + +### 3.7 Decision: the text builder is a prerequisite + +The 2026-10-15 standard-library audit (item 7) records that the experimental +text builder concatenates on every append, so appending *n* segments is +quadratic. Rendering is the one workload that makes this visible on every +request. An amortized-growth builder in `nil.std.text` lands before the +renderer, and the renderer preflights its output size the way +`nil.std.text`'s pattern replacement already does. + +## 4. What this is not + +Not a template file format: there is no second language in a second file, and +holes are checked against the enclosing paragraph's parameters like any other +expression. Not a component system, a virtual DOM or a binding framework. No +precision or presentation grammar inside holes — those remain library calls, as +NIL-0014 decided. No automatic minification. No implicit database, network or +file access from a document. + +## 5. Staging + +1. §2 multiline formatted text. Self-contained, and useful the day it lands. +2. Amortized text builder (§3.7). +3. `begin markup` with the HTML table: parse, check balance, holes, `{when}`, + `{repeat}`, `{raw}`, render to text. +4. `nil.std.markup` constructors, so a document can also be assembled from + values rather than written literally, and the XML table. +5. Diffing two arenas. Out of scope here; named so that §3.6 is not mistaken + for speculation. + +## 6. Open questions + +- **Dynamic attribute sets.** `{spread attrs}` in attribute position, taking a + list of name/value pairs, would cover `data-*` generation. Deferred: it + weakens compile-time checking and is not needed by the first consumers. +- **Whitespace.** The initial rule is to preserve the block's text as written, + because HTML whitespace is sometimes significant, with collapsing offered by + `render ... compact`. Worth revisiting once there is real output to compare. +- **Editor support.** Holes keep original spans by construction (§2), but the + VS Code grammar needs markup-inside-Matrix highlighting or the construct will + look worse in an editor than the string it replaces. +- **Performance claims.** §3.6 asserts an append count, not a measurement. + Benchmark the rendered page against the concatenated equivalent before the + claim appears in any guide. +- **Ny spelling.** `$"""` for §2 is settled by analogy; the Ny surface for + §3 is not. `markup { ... }` is the obvious candidate and is not yet decided. + +## 7. Compatibility + +Every construct here is additive under NIL-0026 §1: no existing program +observes a difference, because `formatted` before `begin text` is currently a +parse error, `markup` is a new contextual word before `begin`, and no existing +text literal changes meaning. Ships on by default, no edition gate. diff --git a/docs/design/README.md b/docs/design/README.md index 71e3aea3..939f1e83 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -53,6 +53,7 @@ not assigned architecture identifiers. | NIL-0024 | [Tests alongside source](NIL-0024-colocated-testing.md) | Implemented initial unit/integration profile | | NIL-0025 | [Matrix desktop UI](NIL-0025-matrix-desktop-ui.md) | Implemented GTK4 on Linux, Win32 on Windows and AppKit on Intel/Apple Silicon macOS; release qualification remains separate | | NIL-0026 | [Evolving the language without breaking what is written](NIL-0026-language-evolution.md) | Editions implemented in the project manifest | +| NIL-0027 | [Multiline formatted text and structured markup literals](NIL-0027-structured-markup.md) | Proposed; staged, nothing shipped | | NIL-0093 | [Executable context is artifact self-description](NIL-0093-executable-context.md) | Implemented within the documented inspection profile | ## Retained Ny feature references From 20776a56d64a8852592993e307a311ca6349a8fd Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Fri, 13 Nov 2026 01:13:48 -0500 Subject: [PATCH 34/39] Interpolate a multiline Matrix text block NIL-0027 section 2. 'formatted' in front of 'begin text' opts a raw block into interpolation, so a generated document is written as one unit instead of a ladder of concatenations. Every raw-block rule still holds: the terminator is a line whose only content is 'end text', the common leading indentation is removed and deeper indentation kept, and nothing inside is escaped. A single hole still stays on one physical line. Indentation removal applies to literal segments only. The lexer records the source offset each dedented line starts at and maps every scanned piece back through it, so a hole's tokens keep their original file coordinates and NIL-0014's editing contract is unchanged. Because the block emits the token stream the quoted form already emits, the parser, checker, lowering, VM and native routes needed no change. Covered by the formatted-text-matrix parity fixture across the VM, binary and native routes, malformed blocks rejected on every route, a hole renamed inside a CRLF block while an identical word in doubled braces is left alone, and completion inside a block hole. Ny keeps only its single-line form. Co-Authored-By: Claude Opus 5 --- docs/design/NIL-0027-structured-markup.md | 21 ++- docs/language-reference/expressions.md | 25 ++- matrix/MATRIX-GUIDE.md | 28 ++- matrix/docs/grammar.ebnf | 14 +- matrix/docs/next-version-findings.md | 33 ++++ matrix/docs/types-and-text.md | 26 ++- nilcompiler/src/formatted_text.hpp | 65 +++++++ nilcompiler/src/matrix_lexer.cpp | 168 +++++++++++++++++- nilcompiler/src/matrix_lexer.hpp | 7 + tests/language/formatted-text-matrix.matrix | 17 ++ tests/language/formatted-text-matrix.stdout | 6 + .../execution/guide-constructs.expected | 4 + .../matrix/execution/guide-constructs.matrix | 9 + tests/tools/completion_contract.py | 13 ++ tests/tools/formatted_text_contract.py | 28 +++ tests/tools/semantic_contract.py | 24 +++ 16 files changed, 464 insertions(+), 24 deletions(-) diff --git a/docs/design/NIL-0027-structured-markup.md b/docs/design/NIL-0027-structured-markup.md index bec26e4c..b0a36b53 100644 --- a/docs/design/NIL-0027-structured-markup.md +++ b/docs/design/NIL-0027-structured-markup.md @@ -1,6 +1,7 @@ # NIL-0027: Multiline formatted text and structured markup literals -- **Status:** Proposed. Staged; nothing in this record ships yet. +- **Status:** §2 implemented for Matrix on 2026-11-13. §3 proposed; the Ny + spelling of §2 remains open. - **Scope:** Ny and Matrix syntax, shared hole lexing and AST, checking, lowering, `nil.std.markup`, text builder - **Date:** 2026-11-13 @@ -49,7 +50,7 @@ member of that family. | Surface | Holes | Structure | Status | | --- | --- | --- | --- | | `formatted "..."` | yes | none, one line | NIL-0014, implemented | -| `formatted begin text ... end text` | yes | none, multiline | §2 of this record | +| `formatted begin text ... end text` | yes | none, multiline | §2, implemented (Matrix) | | `begin markup ... end markup` | yes | elements, attributes, control flow | §3 of this record | All three use the hole lexer NIL-0014 already built, which emits an expression's @@ -87,6 +88,22 @@ a C# raw interpolated string gives, and it removes most `plus` ladders from ordinary code. It does not solve §3's problem, because the control flow still lives outside the document. +**Implemented for Matrix on 2026-11-13.** `scan_formatted_block` in +`nilcompiler/src/formatted_text.hpp` scans the dedented block content, and +`matrix::lexer::scan_formatted_text_block` maps every piece back through a +per-line offset table before emitting the existing `formatted_string_begin` / +`formatted_text` / `interpolation_*` tokens. The parser, checker, lowering and +both execution routes are untouched, because the token stream and therefore the +AST are the ones the quoted form already produced. Coverage: the +`formatted-text-matrix` parity fixture across the VM, binary-NIL and native +routes; malformed blocks rejected on every route in +`tests/tools/formatted_text_contract.py`; and a CRLF block in +`tests/tools/semantic_contract.py` whose hole is renamed while an identical +word inside doubled braces in the dedented literal is left alone, which is the +proof that indentation removal did not move the expression. + +Ny still has only `$"..."`. Its multiline spelling is §6's open question. + ## 3. Markup literals ```text diff --git a/docs/language-reference/expressions.md b/docs/language-reference/expressions.md index be581a84..a8e838f2 100644 --- a/docs/language-reference/expressions.md +++ b/docs/language-reference/expressions.md @@ -210,9 +210,10 @@ to the literal portions, so `\n` inserts a newline in the result. Embedded expressions keep their own string delimiters; no extra escaping is needed for `$"Result: {choose("first", "second")}"`. -The entire explicit string, including its expressions, must fit on one source -line. Empty holes, trailing tokens, unmatched braces and missing quotes are -errors. Formatting flags and precision specifiers are not part of this syntax; +A quoted explicit string, including its expressions, must fit on one source +line; the Matrix block form below spans lines, and a single hole still stays on +one of them. Empty holes, trailing tokens, unmatched braces and missing quotes +are errors. Formatting flags and precision specifiers are not part of this syntax; use an explicit formatting function inside a hole when needed. Legacy ordinary strings such as `"Hello {name}"` retain identifier-only @@ -234,6 +235,24 @@ formatted Matrix text, backslashes remain ordinary characters; use the named `formatted "First{newline}Second"`. Braces delimit ordinary Matrix expressions, including named calls such as `{choose with first being "one"}`. +`formatted` in front of `begin text` opts a raw block into interpolation: + +```text +value page is formatted begin text +

Signed in as {who}

+

{count plus 1} projects

+end text +``` + +The block keeps every raw-block rule — a line-anchored `end text` terminator, +removal of the common leading indentation, retained deeper indentation, dropped +leading and trailing blank lines, and no escape sequences — and adds the holes. +Indentation removal applies to literal segments only. Every hole retains its +original file coordinates, so definition, completion, signature help and local +rename behave inside a block exactly as they do inside a quoted formatted +literal. A block and a quoted literal produce the same interpolation AST, the +same shared conversion and the same lowering on both execution routes. + ## Lambda / Anonymous Functions ```ny diff --git a/matrix/MATRIX-GUIDE.md b/matrix/MATRIX-GUIDE.md index 5b64fdcc..aaffeffa 100644 --- a/matrix/MATRIX-GUIDE.md +++ b/matrix/MATRIX-GUIDE.md @@ -141,10 +141,26 @@ Scalars and text convert themselves; any other type needs an explicit conversion first. There is no format-specifier language inside the braces — precision and padding are library calls. `{{` and `}}` are literal braces in this form only: an ordinary literal and a `begin text` block stay raw, and a -backslash stays a backslash in both. A formatted literal and its expressions -must fit on one physical source line, so interpolate `newline` for a line -break. `formatted` remains an ordinary name anywhere it is not followed by a -quote. +backslash stays a backslash in both. A quoted formatted literal stays on one +physical source line, like every quoted literal. + +**A multi-line formatted literal is the raw block with holes.** `formatted` in +front of `begin text` opts that block into interpolation, and every other rule +of the block is unchanged — the terminator is a line, the common indentation is +removed, deeper indentation is kept, and nothing inside is escaped: + +```matrix +give formatted begin text + + {title} + + +end text +``` + +A single hole still stays on one physical line; the text around it spans as +many as it likes. `formatted` remains an ordinary name anywhere it is not +followed by a quote or by `begin text`. There are no character literals. A one-letter text value is `"a"`. @@ -765,10 +781,6 @@ contract method bodies, supertraits, associated types, `i16`/`u16`/`u32`/`u64`/ Reserved but unimplemented: `chapter` (a third namespace level), `forever` as a `static` modifier, and `requires` for declaring a book's authorities. -A `formatted` literal is one physical source line: multi-line interpolated text -is not available yet. Build it from `newline`, or concatenate a raw `begin text` -block with `plus`. - Matrix and Ny share the interpreter and native routes. Native aggregate-array and foreign-call limitations apply to both languages. diff --git a/matrix/docs/grammar.ebnf b/matrix/docs/grammar.ebnf index dc40947c..c1cae41b 100644 --- a/matrix/docs/grammar.ebnf +++ b/matrix/docs/grammar.ebnf @@ -454,11 +454,21 @@ text_literal = '"' { character_except_quote_or_newline } '"' | text_block ; named_character = 'quote' | 'newline' | 'tab' ; formatted_text = 'formatted' '"' { formatted_segment | '{{' | '}}' - | '{' expression '}' } '"' ; + | '{' expression '}' } '"' + | 'formatted' formatted_block ; (* formatted_segment is raw text excluding quotes, braces and newlines. * Expressions use the complete Matrix grammar, including their own quoted * literals. Doubled braces are literal text; backslashes remain ordinary. - * 'formatted' is contextual only before a quote, and remains a name elsewhere. *) + * 'formatted' is contextual before a quote and before 'begin text', and + * remains an ordinary name elsewhere. *) + +(* The block form is text_block with holes: identical terminator, dedent and + * blank-line rules, no escapes, and literal text that may span lines. A single + * hole still stays on one physical line. Indentation removal applies to the + * literal segments only, so a hole keeps its original source coordinates. *) +formatted_block = 'begin' 'text' newline + { any_line_with { '{{' | '}}' | '{' expression '}' } } + 'end' 'text' ; (* A raw block, following the universal 'begin ' / 'end ' shape. * It is terminated by a line whose only content is 'end text' -- a diff --git a/matrix/docs/next-version-findings.md b/matrix/docs/next-version-findings.md index 4a8b399f..e9a12c76 100644 --- a/matrix/docs/next-version-findings.md +++ b/matrix/docs/next-version-findings.md @@ -2480,3 +2480,36 @@ RHS lowering receives the destination context. Paired Ny/Matrix fixtures cover local and record-field assignments of `Option`, with side effects proving each payload is evaluated once. They run through textual/binary NIL, bundles, LLVM JIT and AOT. This does not extend native global storage to enums. + +### F88 — A generated document needs multiline interpolation before it needs a document type + +**Observed:** Every generator written against the framework builds its output +by concatenation, because a `formatted` literal was confined to one physical +source line (NIL-0014) and a raw `begin text` block carried no holes. The +result is the shape this framework exists to avoid: the document's structure +spread across bindings and `plus` chains, its control flow outside the markup, +and no check that the output is even balanced. Writing a page shell by hand +produced four bindings and six concatenations for eleven tags. + +A second, quieter cost: `MATRIX-GUIDE.md` documented none of `formatted`, +`collect begin query` or `begin pattern`, all of which had shipped. An agent +writing Matrix from the guide therefore could not have used them even where +they applied. + +**Fixed:** `formatted` in front of `begin text` opts a raw block into +interpolation, keeping every raw-block rule — line-anchored terminator, common +indentation removed, deeper indentation kept, no escapes — and adding holes. A +single hole still stays on one physical line. Indentation removal applies to +literal segments only: the lexer records the source offset each dedented line +starts at and maps every piece back through it, so hole tokens keep their +original coordinates and NIL-0014's editing contract holds unchanged. The +block emits the token stream the quoted form already emitted, so the parser, +checker, lowering, VM and native routes needed no change at all. The three +shipped constructs are now documented in the guide and asserted by +`tests/matrix/execution/guide-constructs.matrix` and +`guide-queries-patterns.matrix`. + +**Remaining:** Ny has no multiline spelling yet; `$"""` is the candidate. The +larger problem F88 only dents — control flow living inside the document, with +context-correct escaping and a checkable tree — is +[NIL-0027](../../docs/design/NIL-0027-structured-markup.md) §3. diff --git a/matrix/docs/types-and-text.md b/matrix/docs/types-and-text.md index a092dd44..4df58a9c 100644 --- a/matrix/docs/types-and-text.md +++ b/matrix/docs/types-and-text.md @@ -199,10 +199,28 @@ not make embedded NUL bytes representable by the native text ABI. Backslashes remain ordinary characters. `{{` and `}}` represent literal braces only in this explicitly formatted form. Empty holes, extra tokens, unmatched -braces and missing quotes are errors. The string and its expressions must stay -on one physical source line; interpolate `newline` to create a multiline -result. A raw `begin text` block inside a hole therefore is not supported. -Ordinary quoted text and `begin text` keep their previous raw behavior. +braces and missing quotes are errors. A quoted formatted literal and its +expressions stay on one physical source line. A raw `begin text` block inside a +hole therefore is not supported. Ordinary quoted text and `begin text` keep +their previous raw behavior. + +### Multiline formatted blocks + +`formatted` in front of `begin text` opts a raw block into interpolation: + +```text +value page as text is formatted begin text +

Signed in as {who}

+

{count plus 1} projects

+end text +``` + +Every raw-block rule still holds: the terminator is a line whose only content +is `end text`, the common leading indentation is removed and deeper indentation +kept, leading and trailing blank lines are dropped, and nothing inside is +escaped. A single hole stays on one physical line; the literal text around it +does not. Indentation is removed from literal segments only, so a hole keeps +its original source position and remains editable. `formatted` remains available as a name when it is not followed by a quote. Expression identifiers keep their original source ranges for editor checking, diff --git a/nilcompiler/src/formatted_text.hpp b/nilcompiler/src/formatted_text.hpp index 1aff2053..5b8a1f78 100644 --- a/nilcompiler/src/formatted_text.hpp +++ b/nilcompiler/src/formatted_text.hpp @@ -52,6 +52,17 @@ inline formatted_text_scan scan_formatted_text(const std::string &source, bool matrix, unsigned depth = 0); +// A multiline formatted block. The caller has already removed the block's +// common indentation and joined its lines, so 'content' is the logical text +// and every piece offset is an offset into it; the caller maps those back to +// original source coordinates, which is what keeps a hole's tokens editable. +// A block has no closing quote: its terminator is a line-anchored keyword the +// caller consumed, so a newline is ordinary literal text here. Literal bytes +// are never decoded, matching the raw block the form extends. +inline formatted_text_scan scan_formatted_block(const std::string &content, + bool matrix, + unsigned depth = 0); + inline std::size_t formatted_expression_end(const std::string &source, std::size_t begin, bool matrix, unsigned depth, @@ -224,5 +235,59 @@ inline formatted_text_scan scan_formatted_text(const std::string &source, return result; } +inline formatted_text_scan scan_formatted_block(const std::string &content, + bool matrix, unsigned depth) { + formatted_text_scan result; + result.end = content.size(); + const auto fail = [&](std::size_t at, const std::string &message) { + result.error_offset = at; + result.error = message; + }; + if (depth >= 64) { + fail(0, "interpolation nesting exceeds 64 levels"); + return result; + } + std::size_t cursor = 0; + std::size_t literal_start = 0; + std::string literal; + const auto flush = [&]() { + if (!literal.empty()) + result.pieces.push_back({false, std::move(literal), literal_start, + cursor - literal_start}); + literal.clear(); + }; + while (cursor < content.size()) { + const char c = content[cursor]; + if (c == '{') { + if (cursor + 1 < content.size() && content[cursor + 1] == '{') { + literal += '{'; + cursor += 2; + continue; + } + flush(); + const auto begin = cursor + 1; + const auto end = formatted_expression_end(content, begin, matrix, depth + 1, result); + if (!result.error.empty()) return result; + result.pieces.push_back({true, content.substr(begin, end - begin), begin, end - begin}); + cursor = end + 1; + literal_start = cursor; + continue; + } + if (c == '}') { + if (cursor + 1 < content.size() && content[cursor + 1] == '}') { + literal += '}'; + cursor += 2; + continue; + } + fail(cursor, "unmatched '}' in formatted text; write '}}' for a literal brace"); + return result; + } + literal += c; + ++cursor; + } + flush(); + return result; +} + } // namespace ny #endif diff --git a/nilcompiler/src/matrix_lexer.cpp b/nilcompiler/src/matrix_lexer.cpp index 2ac79d0e..e2c69c8b 100644 --- a/nilcompiler/src/matrix_lexer.cpp +++ b/nilcompiler/src/matrix_lexer.cpp @@ -433,11 +433,136 @@ void lexer::scan_formatted_string(std::vector &tokens, col_ = positions.column(pos_); } -// 'begin text' at end of line opens a raw block literal. -bool lexer::text_block_follows() const { - size_t probe = pos_; - while (probe < src_.size() && (src_[probe] == ' ' || src_[probe] == '\t')) - ++probe; +// A multi-line formatted block is the raw 'begin text' block with holes. The +// literal text is dedented exactly as a raw block's is, while every hole keeps +// its ORIGINAL source coordinates: indentation removal applies to literal +// segments only, never to an expression, so definition, completion, signature +// help and rename behave inside a block exactly as they do inside a +// single-line formatted literal. +void lexer::scan_formatted_text_block(std::vector &tokens, + const token &word) { + const int line = word.line, col = word.column; + while (!at_end() && peek() != '\n') advance(); // 'begin text' and its line + if (!at_end()) advance(); // the newline itself + const std::size_t header_end = pos_; + + struct block_line { + std::size_t start; + std::string text; + }; + std::vector lines; + bool terminated = false; + std::size_t terminator = pos_; + while (!at_end()) { + const std::size_t start = pos_; + std::string raw; + while (!at_end() && peek() != '\n') raw.push_back(advance()); + if (!at_end()) advance(); + if (!raw.empty() && raw.back() == '\r') raw.pop_back(); + if (normalise_line(raw) == "end text") { + terminated = true; + terminator = start; + break; + } + lines.push_back({start, raw}); + } + if (!terminated) { + error(line, col, "unterminated 'formatted begin text' block", + "close it with a line whose only content is 'end text'"); + tokens.push_back(make(token_kind::invalid, "", line, col)); + return; + } + + while (!lines.empty() && normalise_line(lines.front().text).empty()) + lines.erase(lines.begin()); + while (!lines.empty() && normalise_line(lines.back().text).empty()) + lines.pop_back(); + + size_t indent = std::string::npos; + for (const auto &entry : lines) { + const size_t first = entry.text.find_first_not_of(" \t"); + if (first == std::string::npos) continue; // blank lines do not set indent + indent = std::min(indent, first); + } + if (indent == std::string::npos) indent = 0; + + // The logical text, beside the source offset each logical line starts at. + // Dedent is therefore a per-line shift, and mapping a logical offset back to + // the file is exact rather than a second synthetic source. + std::string content; + std::vector logical_starts, source_starts; + for (size_t i = 0; i < lines.size(); ++i) { + if (i != 0) content.push_back('\n'); + const std::size_t strip = std::min(indent, lines[i].text.size()); + logical_starts.push_back(content.size()); + source_starts.push_back(lines[i].start + strip); + content += lines[i].text.substr(strip); + } + const auto source_offset = [&](std::size_t logical) -> std::size_t { + if (logical_starts.empty()) return header_end; + const auto row = static_cast( + std::upper_bound(logical_starts.begin(), logical_starts.end(), logical) - + logical_starts.begin() - 1); + return source_starts[row] + (logical - logical_starts[row]); + }; + + const auto scanned = + ny::scan_formatted_block(content, true, interpolation_depth_); + const ny::formatted_source_positions positions(src_, word.start, pos_, + word.line, word.column); + const auto emit = [&](token_kind kind, std::string value, std::size_t offset, + std::size_t length) { + token result; + result.kind = kind; + result.lexeme = std::move(value); + result.key = result.lexeme; + result.line = positions.line(offset); + result.column = positions.column(offset); + result.start = offset; + result.length = length; + tokens.push_back(std::move(result)); + }; + + if (!scanned.error.empty()) { + const auto at = source_offset(scanned.error_offset); + error(positions.line(at), positions.column(at), scanned.error); + emit(token_kind::invalid, scanned.error, at, 1); + return; + } + + emit(token_kind::formatted_string_begin, "formatted begin text", word.start, + header_end - word.start); + for (const auto &piece : scanned.pieces) { + const auto start = source_offset(piece.start); + const auto end = source_offset(piece.start + piece.length); + if (!piece.expression) { + emit(token_kind::formatted_text, piece.text, start, end - start); + continue; + } + emit(token_kind::interpolation_begin, "{", start - 1, 1); + lexer expression(piece.text, interpolation_depth_ + 1); + for (auto value : expression.scan()) { + if (value.kind == token_kind::end_of_file) continue; + if (value.line == 1) value.column += positions.column(start) - 1; + value.line += positions.line(start) - 1; + value.start += start; + tokens.push_back(std::move(value)); + } + for (const auto &failure : expression.errors()) { + const auto column = + failure.column + + (failure.line == 1 ? positions.column(start) - 1 : 0); + error(failure.line + positions.line(start) - 1, column, failure.message, + failure.hint); + } + emit(token_kind::interpolation_end, "}", end, 1); + } + emit(token_kind::formatted_string_end, "end text", terminator, 8); +} + +// The tail both block probes share: 'text' at 'probe', not running on into a +// longer identifier, and nothing but spacing left on the line. +bool lexer::text_word_ends_line(std::size_t probe) const { const char *word = "text"; for (size_t i = 0; i < 4; ++i) { if (probe + i >= src_.size()) return false; @@ -455,6 +580,34 @@ bool lexer::text_block_follows() const { return probe >= src_.size() || src_[probe] == '\n'; } +// 'begin text' at end of line opens a raw block literal. +bool lexer::text_block_follows() const { + size_t probe = pos_; + while (probe < src_.size() && (src_[probe] == ' ' || src_[probe] == '\t')) + ++probe; + return text_word_ends_line(probe); +} + +// 'formatted begin text' at end of line opens an interpolated block literal. +// 'formatted' has been consumed; 'begin' must follow on the same line. +bool lexer::formatted_block_follows() const { + size_t probe = pos_; + while (probe < src_.size() && (src_[probe] == ' ' || src_[probe] == '\t')) + ++probe; + const char *word = "begin"; + for (size_t i = 0; i < 5; ++i) { + if (probe + i >= src_.size()) return false; + if (ascii_lower(static_cast(src_[probe + i])) != word[i]) + return false; + } + probe += 5; + if (probe < src_.size() && name_part(static_cast(src_[probe]))) + return false; + while (probe < src_.size() && (src_[probe] == ' ' || src_[probe] == '\t')) + ++probe; + return text_word_ends_line(probe); +} + // Consume a raw block: every line after 'begin text' up to a line whose only // content is 'end text'. The common leading indentation is removed, and // leading and trailing blank lines are dropped. Nothing inside is escaped, @@ -552,6 +705,11 @@ std::vector lexer::scan() { scan_formatted_string(tokens, word, quote); continue; } + // 'formatted begin text' opens the same raw block with holes in it. + if (formatted_block_follows()) { + scan_formatted_text_block(tokens, word); + continue; + } } // 'begin text' at end of line opens a raw block literal rather than a // block of declarations. diff --git a/nilcompiler/src/matrix_lexer.hpp b/nilcompiler/src/matrix_lexer.hpp index 5378a123..17a15850 100644 --- a/nilcompiler/src/matrix_lexer.hpp +++ b/nilcompiler/src/matrix_lexer.hpp @@ -222,8 +222,15 @@ class lexer { // Raw block literal: consumes lines after 'begin text' until a line whose // only content is 'end text', then dedents by the common indentation. token scan_text_block(int line, int col); + // Interpolated block literal: the raw 'begin text' block with holes. The + // literal text is dedented; every hole keeps its original source position. + void scan_formatted_text_block(std::vector &tokens, const token &word); // True when 'begin' is immediately followed by 'text' and end of line. bool text_block_follows() const; + // True when 'formatted' is followed by 'begin text' and end of line. + bool formatted_block_follows() const; + // Shared tail of both probes: 'text' at probe, then nothing but spacing. + bool text_word_ends_line(std::size_t probe) const; token scan_backtick_identifier(); void error(int line, int col, std::string message, std::string hint = {}); }; diff --git a/tests/language/formatted-text-matrix.matrix b/tests/language/formatted-text-matrix.matrix index ac00c5ae..615179d2 100644 --- a/tests/language/formatted-text-matrix.matrix +++ b/tests/language/formatted-text-matrix.matrix @@ -29,5 +29,22 @@ main returns override integer begin paragraph say render of floating with input being 1.25 say render of text with input being "text" say "raw={amount}" + + !!! A multiline formatted block: the common indentation is removed, deeper + !!! indentation is kept, nothing is escaped and every hole is an ordinary + !!! expression evaluated once in written order. + value who as text is "world" + say formatted begin text + hello {who} and {items at 0} + kept indentation, C:\raw\path and a "quote" + {{literal braces}} plus {next with nothing}:{next with nothing} + end text + say tally + value empty is formatted begin text + end text + say the length of empty + say formatted begin text + nested {formatted "inner {items at 1}"} done + end text give 0 end paragraph diff --git a/tests/language/formatted-text-matrix.stdout b/tests/language/formatted-text-matrix.stdout index 76fba2e8..339f6a03 100644 --- a/tests/language/formatted-text-matrix.stdout +++ b/tests/language/formatted-text-matrix.stdout @@ -14,3 +14,9 @@ value=17 value=1.25 value=text raw={amount} +hello world and 7 + kept indentation, C:\raw\path and a "quote" +{literal braces} plus 4:5 +5 +0 +nested inner 11 done diff --git a/tests/matrix/execution/guide-constructs.expected b/tests/matrix/execution/guide-constructs.expected index fb3a2d80..f9cc8bd4 100644 --- a/tests/matrix/execution/guide-constructs.expected +++ b/tests/matrix/execution/guide-constructs.expected @@ -50,4 +50,8 @@ cleanup Count: 4; who: Nathan Result: one {literal braces} and 3 + + Home + + 1 diff --git a/tests/matrix/execution/guide-constructs.matrix b/tests/matrix/execution/guide-constructs.matrix index ac510925..7b3ab3b7 100644 --- a/tests/matrix/execution/guide-constructs.matrix +++ b/tests/matrix/execution/guide-constructs.matrix @@ -246,6 +246,15 @@ main returns number begin paragraph say formatted "Result: {choose with first being "one", second being "two"}" say formatted "{{literal braces}} and {tally}" + value page-title as text is "Home" + value page-nav as text is "Dashboard" + say formatted begin text + + {page-title} + + + end text + value started as override long is now in milliseconds sleep for 1 milliseconds say (now in milliseconds) is at least started diff --git a/tests/tools/completion_contract.py b/tests/tools/completion_contract.py index eb6a43d5..68164bbf 100644 --- a/tests/tools/completion_contract.py +++ b/tests/tools/completion_contract.py @@ -189,6 +189,19 @@ def applied(marked, completion, insertion): query(formatted_ny.replace('{sam|}', '{{sam|}}'), name='formatted_literal.ny', available=False, status=0) query(formatted_matrix.replace('{sam|}', '{{sam|}}'), name='formatted_literal.matrix', available=False, status=0) + # A hole inside a multiline block is an ordinary expression position + # even though the block's literal text has been dedented. + block_matrix = ('!!! \u00e9\U0001F600\r\nbook formatted-block-completion\r\n' + 'main returns number begin paragraph\r\n' + ' value sample-value is 7\r\n' + ' say formatted begin text\r\n' + ' {{literal}} {sam|}\r\n' + ' end text\r\n' + ' give 0\r\nend paragraph\r\n') + assert set(items(query(block_matrix, name='formatted-block.matrix'))) == {'sample-value'} + query(block_matrix.replace('{sam|}', '{{sam|}}'), name='formatted_block_literal.matrix', + available=False, status=0) + # Independent result budgets withhold the optional query rather than # return truncated candidates or omit contributing source snapshots. large = '# ' + 'x' * (1500 * 1024) + '\n' diff --git a/tests/tools/formatted_text_contract.py b/tests/tools/formatted_text_contract.py index 8f8ba250..e4a81ff9 100644 --- a/tests/tools/formatted_text_contract.py +++ b/tests/tools/formatted_text_contract.py @@ -43,6 +43,34 @@ def main(): assert output.read_bytes() == b'previous artifact', (path, flags) assert result.stderr.strip(), (path, flags) checks += 1 + # A multiline Matrix block rejects the same malformed holes, and an + # unterminated block is a lexer error rather than a silent literal. + blocks = [ + 'hello\n', # never closed + 'a } b\nend text\n', # unmatched brace + '{1 plus\n2}\nend text\n', # hole spans a line + '{}\nend text\n', # empty hole + '{unknown}\nend text\n', # undefined identifier + '{[1, 2]}\nend text\n', # no text contract + ] + for index, block in enumerate(blocks): + source = ('book invalid_formatted_block\n' + 'main returns override integer begin paragraph\n' + f'value result is formatted begin text\n{block}' + 'give 0\nend paragraph\n') + path = root / f'invalid-block-{index}.matrix' + path.write_text(source, newline='\n') + for flags in routes: + output = root / 'protected-output' + output.write_bytes(b'previous artifact') + result = subprocess.run([compiler, '--quiet', '--no-stdlib', + *flags, '--output', output, path], + capture_output=True, text=True, timeout=30) + assert result.returncode != 0, (path, flags, result.stdout) + assert output.read_bytes() == b'previous artifact', (path, flags) + assert result.stderr.strip(), (path, flags) + checks += 1 + # A function identifier has a return type, but is not a scalar value. path = root / 'function.ny' path.write_text('fn number() -> i32\nreturn 1\nend fn\n' diff --git a/tests/tools/semantic_contract.py b/tests/tools/semantic_contract.py index c5d91f25..7e619464 100644 --- a/tests/tools/semantic_contract.py +++ b/tests/tools/semantic_contract.py @@ -324,6 +324,30 @@ def rename_local(result, file, target, new_name): snapshot = next(item['text'] for item in renamed['semantic']['sources'] if item['path'] == str(root / name)) assert '{{' + spelling + '}}' in snapshot + # A multiline formatted block removes the block's common indentation + # from its literal text only. Every hole keeps its original file + # coordinates, so a rename edits the expression and leaves identical + # words in the dedented literal untouched. + block = ('!!! \u00e9\U0001F600\r\nbook formatted-block-editing\r\n' + 'message returns number begin paragraph\r\n' + ' value local-value is 7\r\n' + ' say formatted begin text\r\n' + ' {{local-value}} {local-value plus 1}\r\n' + ' end text\r\n' + ' give local-value\r\n' + 'end paragraph\r\n') + name = 'formatted-block.matrix' + result = analyze({name: block}, name) + target = binding(result, name, '{local-value plus', 'local-value') + assert target['rename_safe'] and target['references_complete'], target + assert len([use for use in result['semantic']['references'] + if use['symbol_id'] == target['id']]) == 2 + renamed = rename_local(result, name, target, 'renamed-value') + snapshot = next(item['text'] for item in renamed['semantic']['sources'] + if item['path'] == str(root / name)) + assert '{{local-value}}' in snapshot, snapshot + assert '{renamed-value plus 1}' in snapshot, snapshot + interpolation = ('fn message(input: i32) -> i32\nprintln("\\n{input}")\nreturn input\nend fn\n') result = analyze({'interpolation.ny': interpolation}, 'interpolation.ny') assert len(result['semantic']['references']) == 1 From a575592246eab0accb6193cde62363ffd0f8eebe Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Fri, 13 Nov 2026 01:51:23 -0500 Subject: [PATCH 35/39] Write a document as a document, not as concatenated fragments NIL-0027 section 3. 'begin markup' holds its own conditionals and loops, so the control flow lives inside the page instead of around it, and the block is parsed as markup at compile time: an unbalanced element, an unterminated directive, an arm that opens an element it does not close, and a hole that would name an element or an attribute are all compile errors. Escaping belongs to the position, never to the author. A hole learns during parsing whether it landed in element content, an attribute value, a whole URL attribute or a component inside one, and lowers to the matching nil.std.markup function; an executable scheme is neutralised rather than dropped. '{raw ...}' is the single greppable way to bypass that. The value is a nil.std.markup document rather than text, which is what lets a paragraph return a fragment and another block splice it instead of escaping it twice. NIL-0027 section 3.6 records why the record-graph tree that section originally specified was replaced by compile-time serialization: it cannot execute natively, and it bought only the one capability that was already deferred. Inside script and style a brace is an ordinary character, so CSS and JavaScript paste in unaltered; a body that is exactly one hole still interpolates through '{raw ...}'. 'markup' and 'raw' stay ordinary names. The literal reuses the block scanner from section 2, so holes keep their original source coordinates, and emit_markup produces ordinary NIL, so one emitter covers the VM and native routes. Covered by the markup-matrix parity fixture across the VM, binary, native and native-toolchain routes, and by a contract rejecting sixteen malformed documents on every route while proving hostile input is escaped once in each context. Co-Authored-By: Claude Opus 5 --- CMakeLists.txt | 7 + docs/design/NIL-0027-structured-markup.md | 109 +++-- docs/standard-library/index.md | 2 + docs/standard-library/markup.md | 63 +++ matrix/MATRIX-GUIDE.md | 99 +++++ matrix/docs/grammar.ebnf | 19 + matrix/docs/next-version-findings.md | 38 ++ nilcompiler/src/ast.hpp | 32 ++ nilcompiler/src/codegen.cpp | 145 +++++++ nilcompiler/src/formatted_text.hpp | 125 ++++++ nilcompiler/src/matrix_lexer.cpp | 55 ++- nilcompiler/src/matrix_lexer.hpp | 19 +- nilcompiler/src/matrix_parser.cpp | 380 ++++++++++++++++++ nilcompiler/src/matrix_parser.hpp | 1 + nilcompiler/src/type_checker.cpp | 82 ++++ nilcompiler/src/type_checker.hpp | 1 + nilstd/src/markup/escaping.ny | 153 +++++++ nilstd/src/markup/package.ny | 10 + tests/language/markup-matrix.matrix | 59 +++ tests/language/markup-matrix.stdout | 57 +++ .../execution/guide-queries-patterns.expected | 10 + .../execution/guide-queries-patterns.matrix | 27 +- tests/matrix/run_lexer_tests.py | 3 + tests/tools/markup_contract.py | 131 ++++++ 24 files changed, 1561 insertions(+), 66 deletions(-) create mode 100644 docs/standard-library/markup.md create mode 100644 nilstd/src/markup/escaping.ny create mode 100644 nilstd/src/markup/package.ny create mode 100644 tests/language/markup-matrix.matrix create mode 100644 tests/language/markup-matrix.stdout create mode 100644 tests/tools/markup_contract.py diff --git a/CMakeLists.txt b/CMakeLists.txt index e6927f1f..e613b842 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3150,6 +3150,7 @@ if(BUILD_TESTING) bugpass-constructor-context bugpass-call-context collected-queries collected-queries-matrix + markup-matrix signed-numbers-matrix integer-literal-widths integer-literal-widths-matrix checked-text-spans checked-text-spans-matrix @@ -3246,6 +3247,12 @@ if(BUILD_TESTING) --compiler $ --llvm "${NIL_ENABLE_LLVM}") set_tests_properties(nil.formatted-text-contract PROPERTIES LABELS "ny;matrix;analysis;interpolation" TIMEOUT 120) + add_test(NAME nil.markup-contract + COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/markup_contract.py" + --compiler $ --interpreter $ + --stdlib "${CMAKE_CURRENT_SOURCE_DIR}/nilstd/src" --llvm "${NIL_ENABLE_LLVM}") + set_tests_properties(nil.markup-contract PROPERTIES + LABELS "matrix;markup;analysis;native;interpreter;rc" TIMEOUT 300) if(TARGET nilassembler AND TARGET nillinker) add_test(NAME nil.formatted-nil-contract COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tests/tools/formatted_nil_contract.py" diff --git a/docs/design/NIL-0027-structured-markup.md b/docs/design/NIL-0027-structured-markup.md index b0a36b53..fc6b7ba3 100644 --- a/docs/design/NIL-0027-structured-markup.md +++ b/docs/design/NIL-0027-structured-markup.md @@ -1,7 +1,8 @@ # NIL-0027: Multiline formatted text and structured markup literals -- **Status:** §2 implemented for Matrix on 2026-11-13. §3 proposed; the Ny - spelling of §2 remains open. +- **Status:** §2 and §3 implemented for Matrix on 2026-11-13, with the + representation revised as recorded in §3.6. The Ny spellings of both remain + open. - **Scope:** Ny and Matrix syntax, shared hole lexing and AST, checking, lowering, `nil.std.markup`, text builder - **Date:** 2026-11-13 @@ -51,7 +52,7 @@ member of that family. | --- | --- | --- | --- | | `formatted "..."` | yes | none, one line | NIL-0014, implemented | | `formatted begin text ... end text` | yes | none, multiline | §2, implemented (Matrix) | -| `begin markup ... end markup` | yes | elements, attributes, control flow | §3 of this record | +| `begin markup ... end markup` | yes | elements, attributes, control flow | §3, implemented (Matrix) | All three use the hole lexer NIL-0014 already built, which emits an expression's tokens with their original byte offsets rather than re-lexing decoded text. @@ -224,35 +225,58 @@ doing this at all: what a future webview host wants, and it is unavailable from a string forever after. -### 3.6 Representation: an arena, not a record graph - -The obvious implementation — a `node` record holding `array of node` children — -does not execute. Native arrays of records, choices and nested arrays are -unsupported (MATRIX-GUIDE §17), and an array of class handles compiles and then -fails at run time (§16). A tree of record handles would therefore be -interpreter-only, which is not acceptable for the one construct whose whole -purpose is to be used everywhere. - -A `markup` value is instead an **arena**: parallel arrays of scalars -(`kind`, `name-id`, `parent`, `first-child`, `next-sibling`, `text-id`) beside -a text pool `array of text`. Arrays of scalars and arrays of text are supported -on both routes today. Composition is an append of one arena into another with -an index offset. Rendering is one pre-order walk. - -This also gives the compile-time optimisation for free: a literal's static -subtrees have known shape, so the compiler emits their serialized bytes as -constant segments and the render becomes a sequence of appends with the holes -patched in. A page with twelve holes costs about twelve appends, not a tree -traversal. - -### 3.7 Decision: the text builder is a prerequisite - -The 2026-10-15 standard-library audit (item 7) records that the experimental -text builder concatenates on every append, so appending *n* segments is -quadratic. Rendering is the one workload that makes this visible on every -request. An amortized-growth builder in `nil.std.text` lands before the -renderer, and the renderer preflights its output size the way -`nil.std.text`'s pattern replacement already does. +### 3.6 Representation: revised — serialized at compile time + +**This section replaced its original proposal on 2026-11-13, during +implementation.** It first specified a runtime arena: parallel scalar arrays +holding a node tree, because §3.5 argued for a tree over text. Building it +turned up two things. + +The first is that the obvious implementation does not execute. A `node` record +holding `array of node` children fails: native arrays of records, choices and +nested arrays are unsupported (MATRIX-GUIDE §17) and an array of class handles +compiles and then fails at run time (§16). An arena of parallel scalar and text +arrays does work on both routes, so the plan was sound. + +The second is that it buys almost nothing here. Of §3.5's four reasons for a +tree, three do not need one at run time: + +- **Context-correct escaping** is decided while the block is parsed, so it + costs nothing at run time and is *more* reliable than a tree walk. +- **Balance checking** happens at parse time, whatever the value becomes. +- **Composition without seams** needs only that a document be a distinct + *type*, not a distinct *shape*: an already-escaped fragment spliced into + another document is correct whether it is a node list or serialized bytes. + +Only the fourth — rendering one document to several dialects, or diffing two — +needs a runtime tree, and that was already deferred to §5. + +So a `markup` value is `nil.std.markup.document`, a record holding the +serialized, already-escaped text. The literal's static segments are compile- +time constants; only holes and control flow cost anything. The type is what +carries the guarantee, and the shape behind it stays an implementation detail: +moving to an arena later changes no source. + +What this costs, stated plainly: the dialect is fixed when the block is +compiled rather than when it is rendered, so `begin markup as xml` becomes a +second lowering rather than a second renderer, and diffing two documents will +need the arena after all. Both were out of scope for this version, and neither +is reachable from a program written against this one. + +### 3.7 Decision: the text builder — not yet needed + +The original plan made an amortized text builder a prerequisite, because the +2026-10-15 audit (item 7) records that the experimental builder concatenates on +every append. With §3.6's revision the renderer is not a tree walk at all: a +page is a sequence of `STR_CONCAT` appends the compiler emits directly, and the +builder never enters the path. The audit item stands on its own merits and is +still worth fixing; it is no longer blocking. + +What rendering does depend on is `nil.std.text`, and one gap surfaced there: +`to_lower` reaches the `_char_code` interop, which has no native ABI, so a page +using it would compile for the VM and fail natively. The URL scheme check folds +case with `replace` instead. Making `to_lower` native is separate follow-up +work. ## 4. What this is not @@ -265,14 +289,21 @@ file access from a document. ## 5. Staging -1. §2 multiline formatted text. Self-contained, and useful the day it lands. -2. Amortized text builder (§3.7). +1. §2 multiline formatted text. **Done 2026-11-13.** +2. ~~Amortized text builder~~ — not a prerequisite after §3.6; see §3.7. 3. `begin markup` with the HTML table: parse, check balance, holes, `{when}`, - `{repeat}`, `{raw}`, render to text. -4. `nil.std.markup` constructors, so a document can also be assembled from - values rather than written literally, and the XML table. -5. Diffing two arenas. Out of scope here; named so that §3.6 is not mistaken - for speculation. + `{repeat}`, `{raw}`, render to text. **Done 2026-11-13.** The literal is + lexed by the block scanner from §2, parsed as markup by + `parser::parse_markup_expression`, checked in `type_checker::visit` + (escape versus splice) and emitted by `emit_markup` as ordinary NIL, so both + execution routes are covered by one emitter. Coverage: the `markup-matrix` + parity fixture across the VM, binary, native and native-toolchain routes; + `tests/tools/markup_contract.py` rejecting sixteen malformed documents on + every route and proving hostile input is escaped in each context exactly + once. +4. `nil.std.markup` constructors beyond `raw` and `empty`, so a document can be + assembled from values rather than written literally, and the XML table. +5. Diffing two documents, which is what would bring §3.6's arena back. ## 6. Open questions diff --git a/docs/standard-library/index.md b/docs/standard-library/index.md index 2f10ab4a..ab94ddd2 100644 --- a/docs/standard-library/index.md +++ b/docs/standard-library/index.md @@ -16,6 +16,8 @@ bounded conventional regex, captures, byte spans, match iteration, whole-match collections, splitting and literal replacement. The [query reference](queries.md) defines eager filtering, stable ordering, pagination, projection and the Matrix-authored collection terminals. +The [markup reference](markup.md) defines the document type a markup literal +produces and the per-context escaping it lowers to. The [vector norm reference](math-vectors.md) describes stable two- and three-component lengths, finite-value checks, and checked normalization shared diff --git a/docs/standard-library/markup.md b/docs/standard-library/markup.md new file mode 100644 index 00000000..25ff7f6f --- /dev/null +++ b/docs/standard-library/markup.md @@ -0,0 +1,63 @@ +# Markup documents + +`read nil.std.markup` from Matrix, or import it from Ny. The module holds the +document type a `begin markup` literal produces ([NIL-0027](../design/NIL-0027-structured-markup.md)) +and the escaping that literal lowers to. The functions are ordinary public +functions, not compiler internals: the escaping a generated page depends on is +readable, testable source, and a document can be assembled by hand where a +literal does not fit. + +## The document type + +```text +pub struct document + content: text +end struct +``` + +A `document` is **serialized, already-escaped markup**. The distinction from +`text` is the point of the type: interpolating a document into another markup +block splices it, while interpolating text escapes it. Without the type, every +composition would either double-escape or silently trust its input. + +| Function | Result | +| --- | --- | +| `render(doc)` | the serialized text; the only supported way out of a document | +| `raw(content)` | adopt already-trusted markup as a document | +| `empty()` | a document that contributes nothing | +| `escape_text(content)` | element-content escaping: `&`, `<`, `>` | +| `escape_attribute(content)` | attribute escaping: `&`, `<`, `>`, `"`, `'` | +| `escape_url(content)` | a whole URL attribute value | +| `escape_url_component(content)` | one component inside a URL | + +A markup block calls these for you, choosing by the position the hole landed +in. Calling them directly is supported and is what hand-built documents do. + +## Escaping contract + +**Element content and attribute values** are escaped for the HTML of this +version: an attribute is always rendered double quoted, and both quote forms +are escaped so hand-written single-quoted markup stays correct. + +**A whole URL attribute value** (`href`, `src`, `action`, `formaction`, +`cite`, `poster`, `data`, `srcset`, `background`, `ping`, `manifest`) is scheme +filtered first. `javascript:`, `vbscript:` and `data:` are neutralised by +percent-encoding the scheme's colon, which leaves an inert relative reference: +the page still renders, nothing executes, and the original text stays visible +to anyone reading the output. A colon that appears after the first `/` belongs +to a path, not a scheme, and is left alone. Comparison folds ASCII case. + +**A component inside a URL** percent-encodes the reserved and unsafe ASCII set +and passes other bytes, including UTF-8, through for the browser to encode. +`%` is encoded first, so nothing is encoded twice. + +Escaping is byte-oriented and allocates a new text per pass. It is correct +before it is fast; a single-pass escaper is a later optimisation behind the +same contract. + +## What this module does not do + +No sanitizer, no allow-list filter, no DOM. `raw` adopts text without +inspecting it, so content from outside the program must be proven safe by the +caller or escaped by passing it through a hole instead. The module renders +HTML; XML serialization and whitespace-collapsing output are future work. diff --git a/matrix/MATRIX-GUIDE.md b/matrix/MATRIX-GUIDE.md index aaffeffa..0e5cf1fe 100644 --- a/matrix/MATRIX-GUIDE.md +++ b/matrix/MATRIX-GUIDE.md @@ -923,3 +923,102 @@ and case-sensitive. Backreferences, lookbehind, flags, word boundaries and Unicode properties are rejected rather than silently given another meaning. Matrix lowercases capture labels and turns hyphens into underscores; look them up by the resulting name. + +--- + +## 22. Markup documents + +`begin markup` writes a document as a document, with its conditionals and +loops inside it, instead of assembling one from concatenated fragments: + +```matrix +read everything from nil.std.markup + +page-shell receives title as text, signed-in as boolean, who as text + returns nil.std.markup.document begin paragraph + give begin markup + + {title} + + + + + end markup +end paragraph +``` + +The block's value is a **document**, not text. `nil.std.markup::render with +doc being page` serializes one, and a markup block needs +`read nil.std.markup` because that is where the document type and its escaping +live. + +**Escaping is the position's, not yours.** A hole is escaped for the context it +landed in, decided while the block is parsed: + +| Where the hole is | What happens | +| --- | --- | +| element content | `&`, `<`, `>` escaped | +| attribute value | `&`, `<`, `>`, `"`, `'` escaped | +| a whole URL attribute | an executable scheme is neutralised, then escaped | +| inside a longer URL value | percent-encoded as one component | +| `{raw expression}` | nothing; the caller owns the proof | + +A hole takes `text`, a scalar, or a document. `{raw ...}` is the only way to +put unescaped text into a page, which is what makes it the word to look for in +review. + +**Documents compose as documents.** A paragraph returning a document +interpolates into another block as markup, so it is spliced rather than escaped +a second time: + +```matrix +
{card with title being title, count being total}
+``` + +A document may only go where markup belongs. Interpolating one into an +attribute value is an error. + +**Control flow uses the language's own words.** `{when ...}`, +`{otherwise when ...}`, `{otherwise}`, `{end when}`, +`{repeat for each in }` and `{end repeat}` are the whole +set. A directive may sit between elements, between attributes, or inside text: + +```matrix +
    + {repeat for each name in names} +
  • {name}
  • + {end repeat} +
+link +``` + +Every arm must be a complete fragment. An arm that opens an element it does not +close is an error — which is the rule that stops a document from going back to +being assembled out of pieces. + +**What the compiler checks:** every element is closed, in order; every +directive is closed; and every hole has a type the renderer can use. An +unclosed `
` is a compile error, not something a browser discovers. + +**Braces inside ` +``` + +Elsewhere, `{{` and `}}` are literal braces, as in formatted text. + +**Known limits.** Element and attribute names are fixed: a hole cannot name +either, because that is what keeps the document checkable. An attribute value +must be quoted. Text is emitted as written, so a spliced document keeps the +indentation it was written with. The dialect is HTML; XML serialization and a +compact renderer are not available yet. diff --git a/matrix/docs/grammar.ebnf b/matrix/docs/grammar.ebnf index c1cae41b..0e0f8b4e 100644 --- a/matrix/docs/grammar.ebnf +++ b/matrix/docs/grammar.ebnf @@ -400,6 +400,7 @@ variadic_tail = 'more' expression { ',' expression } ; primary = literal | query_expr + | markup_block | identifier | 'my' identifier | nominal_type (* also carries explicit type arguments: @@ -470,6 +471,24 @@ formatted_block = 'begin' 'text' newline { any_line_with { '{{' | '}}' | '{' expression '}' } } 'end' 'text' ; +(* NIL-0027. A markup literal is the same block machinery reading markup: the + * literal segments are parsed as elements, attributes, comments and text so + * that balance is checked and each hole learns its context, then emitted + * verbatim. Holes carry the context's escaping, which the author never spells. + * Directives reuse the language's own control words and must each leave the + * element stack as they found it. Element and attribute names are fixed: a + * hole may not name either. Inside a raw-text element ('script', 'style') a + * brace is an ordinary character unless the body is exactly one hole. The + * value is a nil.std.markup document, not text. *) +markup_block = 'begin' 'markup' newline + { markup_content } 'end' 'markup' ; +markup_content = markup_literal | '{{' | '}}' + | '{' [ 'raw' ] expression '}' + | '{' 'when' expression '}' | '{' 'otherwise' [ 'when' expression ] '}' + | '{' 'end' 'when' '}' + | '{' 'repeat' 'for' 'each' identifier 'in' expression '}' + | '{' 'end' 'repeat' '}' ; + (* A raw block, following the universal 'begin ' / 'end ' shape. * It is terminated by a line whose only content is 'end text' -- a * line-anchored keyword rather than a character, so no content can diff --git a/matrix/docs/next-version-findings.md b/matrix/docs/next-version-findings.md index e9a12c76..1f1f6b01 100644 --- a/matrix/docs/next-version-findings.md +++ b/matrix/docs/next-version-findings.md @@ -2513,3 +2513,41 @@ shipped constructs are now documented in the guide and asserted by larger problem F88 only dents — control flow living inside the document, with context-correct escaping and a checkable tree — is [NIL-0027](../../docs/design/NIL-0027-structured-markup.md) §3. + +### F89 — A document assembled from fragments cannot be checked or escaped + +**Observed:** F88 gave a generator readable interpolation, and left the real +problem untouched: the control flow still lived outside the document. A page +shell was still four bindings and a conditional that produced two different +strings of tags, nothing checked that the result was balanced, and every +interpolated value carried whatever escaping its author remembered. The +escaping is the part that cannot be fixed by discipline — the correct escape +differs between element content, an attribute value and a URL attribute, and a +string does not know which one it landed in. + +**Fixed:** `begin markup` (NIL-0027 §3). The literal segments are parsed as +markup at compile time, so unbalanced elements and unterminated directives are +compile errors, and each hole is escaped for the position it occupies. The +block's value is a `nil.std.markup.document`, a distinct type whose whole job +is that an already-escaped fragment composed into another document is spliced +rather than escaped twice. `{raw ...}` is the single, greppable way to bypass +escaping. Control flow is written with the language's own words inside the +document, and every arm must leave the element stack as it found it. + +Three things fell out of building it, each recorded where it belongs: + +- The node tree the design record specified cannot execute natively, and would + have bought only the capability that was already deferred. The value is + serialized at compile time instead; NIL-0027 §3.6 records the reversal. +- `nil.std.text::to_lower` reaches `_char_code`, which has no native ABI, so a + page that used it compiled for the VM and failed natively. The URL scheme + check folds case with `replace`. Making `to_lower` native is open work. +- The checker spells `text` as `String` internally while the diagnostic + renderer prints `text`, so a comparison against `"text"` alone fails while + reporting that it got `text`. Any new check over text types must accept both + spellings. + +**Remaining:** Ny has no markup spelling. Whitespace is emitted as written, so +a composed document keeps its own indentation; a compact renderer and the XML +dialect are future work. Element and attribute names are fixed, so dynamic +attribute sets (`data-*` generation) still need a design. diff --git a/nilcompiler/src/ast.hpp b/nilcompiler/src/ast.hpp index b6f96621..c0102783 100644 --- a/nilcompiler/src/ast.hpp +++ b/nilcompiler/src/ast.hpp @@ -198,6 +198,7 @@ enum class ast_node_kind { hotreload_expr_node, // hotreload.method(args) - intrinsic hot reload if_expr_node, // if cond then expr else expr end if - conditional expression query_expr_node, // eager collect query over a checked array + markup_expr_node, // markup literal: a checked document with holes // Ny testing nodes test_block_node, @@ -336,6 +337,7 @@ struct ast_node::ast_visitor { virtual void visit(struct ast_hotreload_expr &e) {} virtual void visit(struct ast_if_expr &e) {} virtual void visit(struct ast_query_expr &e) {} + virtual void visit(struct ast_markup_expr &e) {} }; enum class visibility_kind { @@ -1205,6 +1207,36 @@ struct ast_query_expr : public ast_expr { void accept(ast_node::ast_visitor &v) override { v.visit(*this); } }; +// A markup literal (NIL-0027). The block is parsed as markup at compile time, +// which is what makes its structure checkable and lets each hole learn the +// context it landed in; what survives into the AST is a flat program of +// literal output, escaped holes and control flow, because that is all the +// serialized result needs. Escaping is chosen here, never by the author, and +// a hole that is already a document is spliced rather than escaped again. +struct ast_markup_item { + enum class item_kind { + literal, hole, when_begin, else_when, else_arm, end_when, repeat_begin, + end_repeat + }; + item_kind kind = item_kind::literal; + std::string text; // literal output, already serialized + ast_expr_ptr expr; // hole value, condition, or repeat source + std::string escape; // text | attribute | url | url_component | raw + std::string binding; // repeat row name + std::string element_type; // checked repeat element type + bool splice = false; // checked: the hole is already a document + int line = 0; +}; + +struct ast_markup_expr : public ast_expr { + std::vector items; + bool checked_markup = false; + + explicit ast_markup_expr(int line = 0) + : ast_expr(ast_node_kind::markup_expr_node, line) {} + void accept(ast_node::ast_visitor &v) override { v.visit(*this); } +}; + // Index expression: array[index] struct ast_index_expr : public ast_expr { ast_expr_ptr object; diff --git a/nilcompiler/src/codegen.cpp b/nilcompiler/src/codegen.cpp index 219ef77b..bde58c9e 100644 --- a/nilcompiler/src/codegen.cpp +++ b/nilcompiler/src/codegen.cpp @@ -3214,6 +3214,148 @@ class emitter { // Queries lower to ordinary typed NIL arrays and loops. Row expressions are // evaluated here in the enclosing frame, so captures require no closures. + // A markup literal builds its serialized document in one text local. The + // literal segments are constants decided at compile time; only the holes and + // the control flow cost anything at run time, and each hole carries the + // escape its position earned during parsing. + void emit_markup(const ast_markup_expr &markup) { + if (!markup.checked_markup) { + cg_.reject_codegen_construct(markup.line, + "a markup block requires checked holes and nil.std.markup", + "enable type checking and read nil.std.markup"); + return; + } + using item_kind = ast_markup_item::item_kind; + const int line = markup.line; + const auto push = [&](const std::string &op, std::vector args = {}, + int at = 0) { push_instr(op, args, at ? at : line); }; + const auto result = new_temp("markup.result"); + push("PUSH_CONST_STR", {get_or_add_const_string_name("")}); + push("STORE_LOCAL", {result}); + + struct when_frame { std::string next, end; }; + struct repeat_frame { std::string rows, index, count, top, done; }; + std::vector whens; + std::vector repeats; + std::vector> scopes; + + for (const auto &item : markup.items) { + switch (item.kind) { + case item_kind::literal: { + push("LOAD_LOCAL", {result}, item.line); + push("PUSH_CONST_STR", {get_or_add_const_string_name(item.text)}, item.line); + push("STR_CONCAT", {}, item.line); + push("STORE_LOCAL", {result}, item.line); + break; + } + case item_kind::hole: { + if (!item.expr) break; + push("LOAD_LOCAL", {result}, item.line); + emit_expr(*item.expr); + if (item.splice) { + push("GET_FIELD", {"content"}, item.line); + } else { + const auto expr_type = scalar_text_type(substitute_nil_type_name( + item.expr->inferred_type.value_or(""), substitutions)); + if (!supports_scalar_text(expr_type)) { + cg_.reject_codegen_construct(item.expr->line, + "a markup hole requires a checked scalar, text or document", + "enable type checking and call an explicit formatter for other types"); + break; + } + if (expr_type != "text") + push("TO_STRING", {expr_type}, item.line); + if (item.escape != "raw") + push("CALL", {"nil__std__markup__escape_" + item.escape, "1"}, item.line); + } + push("STR_CONCAT", {}, item.line); + push("STORE_LOCAL", {result}, item.line); + break; + } + case item_kind::when_begin: { + when_frame frame{new_label("L_markup_next"), new_label("L_markup_when_end")}; + if (item.expr) emit_expr(*item.expr); + push("JMP_IF_NOT", {frame.next}, item.line); + whens.push_back(frame); + break; + } + case item_kind::else_when: { + if (whens.empty()) break; + push("JMP", {whens.back().end}, item.line); + push("LABEL", {whens.back().next}, item.line); + whens.back().next = new_label("L_markup_next"); + if (item.expr) emit_expr(*item.expr); + push("JMP_IF_NOT", {whens.back().next}, item.line); + break; + } + case item_kind::else_arm: { + if (whens.empty()) break; + push("JMP", {whens.back().end}, item.line); + push("LABEL", {whens.back().next}, item.line); + whens.back().next.clear(); + break; + } + case item_kind::end_when: { + if (whens.empty()) break; + if (!whens.back().next.empty()) + push("LABEL", {whens.back().next}, item.line); + push("LABEL", {whens.back().end}, item.line); + whens.pop_back(); + break; + } + case item_kind::repeat_begin: { + repeat_frame frame{new_temp("markup.rows"), new_temp("markup.index"), + new_temp("markup.count"), new_label("L_markup_row"), + new_label("L_markup_rows_done")}; + if (item.expr) emit_expr(*item.expr); + push("STORE_LOCAL", {frame.rows}, item.line); + push("LOAD_LOCAL", {frame.rows}, item.line); + push("LEN", {}, item.line); + push("CAST", {"i64"}, item.line); + push("STORE_LOCAL", {frame.count}, item.line); + push("PUSH_CONST", {"0"}, item.line); + push("CAST", {"i64"}, item.line); + push("STORE_LOCAL", {frame.index}, item.line); + push("LABEL", {frame.top}, item.line); + push("LOAD_LOCAL", {frame.index}, item.line); + push("LOAD_LOCAL", {frame.count}, item.line); + push("CMP_LT", {}, item.line); + push("JMP_IF_NOT", {frame.done}, item.line); + scopes.push_back(std::make_unique(*this)); + declare_local(item.binding, item.element_type); + push("LOAD_LOCAL", {frame.rows}, item.line); + push("LOAD_LOCAL", {frame.index}, item.line); + push("GET_INDEX", {}, item.line); + push("STORE_LOCAL", {item.binding}, item.line); + repeats.push_back(frame); + break; + } + case item_kind::end_repeat: { + if (repeats.empty()) break; + const auto frame = repeats.back(); + repeats.pop_back(); + if (!scopes.empty()) scopes.pop_back(); + push("LOAD_LOCAL", {frame.index}, item.line); + push("PUSH_CONST", {"1"}, item.line); + push("CAST", {"i64"}, item.line); + push("ADD", {"i64"}, item.line); + push("STORE_LOCAL", {frame.index}, item.line); + push("JMP", {frame.top}, item.line); + push("LABEL", {frame.done}, item.line); + break; + } + } + } + // The result is a document, not text: the type is what keeps an already + // escaped fragment from being escaped a second time when it is composed. + const auto content = new_temp("markup.document"); + push("LOAD_LOCAL", {result}); + push("STORE_LOCAL", {content}); + push("NEW_STRUCT", {"nil::std::markup::document"}); + push("LOAD_LOCAL", {content}); + push("SET_FIELD", {"content"}); + } + void emit_query(const ast_query_expr &query) { if (!query.checked_query || !query.source || !query.projection) { cg_.reject_codegen_construct(query.line, @@ -3445,6 +3587,9 @@ class emitter { case ast_node_kind::query_expr_node: emit_query(static_cast(e)); break; + case ast_node_kind::markup_expr_node: + emit_markup(static_cast(e)); + break; case ast_node_kind::array_literal_expr_node: { const auto &arr = static_cast(e); for (const auto &elem : arr.elements) { diff --git a/nilcompiler/src/formatted_text.hpp b/nilcompiler/src/formatted_text.hpp index 5b8a1f78..ddceeafc 100644 --- a/nilcompiler/src/formatted_text.hpp +++ b/nilcompiler/src/formatted_text.hpp @@ -63,6 +63,15 @@ inline formatted_text_scan scan_formatted_block(const std::string &content, bool matrix, unsigned depth = 0); +// A markup block's content (NIL-0027). The hole rules are the block's, with +// one exception: inside a raw-text element a brace is an ordinary character, +// because CSS and JavaScript are full of them and an author must be able to +// paste either one unaltered. A raw-text element whose body is exactly one +// hole still interpolates, which is how a generated script or stylesheet is +// spliced in. +inline formatted_text_scan scan_markup_content(const std::string &content, + unsigned depth = 0); + inline std::size_t formatted_expression_end(const std::string &source, std::size_t begin, bool matrix, unsigned depth, @@ -289,5 +298,121 @@ inline formatted_text_scan scan_formatted_block(const std::string &content, return result; } +// The byte ranges of raw-text element bodies, in order. Only 'script' and +// 'style' are raw-text elements in HTML; an unterminated one is left to the +// markup parser to report, so scanning stops at the end of the content. +struct markup_raw_region { + std::size_t begin = 0, end = 0; + bool single_hole = false; +}; + +inline bool markup_name_follows(const std::string &content, std::size_t at, + const char *name) { + std::size_t index = 0; + for (; name[index]; ++index) { + if (at + index >= content.size()) return false; + char letter = content[at + index]; + if (letter >= 'A' && letter <= 'Z') letter += 'a' - 'A'; + if (letter != name[index]) return false; + } + const auto after = at + index; + if (after >= content.size()) return true; + const unsigned char next = static_cast(content[after]); + return !(std::isalnum(next) || next == '-' || next == '_'); +} + +inline std::vector markup_raw_regions(const std::string &content) { + std::vector regions; + for (std::size_t cursor = 0; cursor + 1 < content.size(); ++cursor) { + if (content[cursor] != '<') continue; + const char *name = nullptr; + if (markup_name_follows(content, cursor + 1, "script")) name = "script"; + else if (markup_name_follows(content, cursor + 1, "style")) name = "style"; + if (!name) continue; + const auto open_end = content.find('>', cursor); + if (open_end == std::string::npos) break; + if (open_end > cursor && content[open_end - 1] == '/') { cursor = open_end; continue; } + std::size_t close = open_end + 1; + for (; close < content.size(); ++close) { + if (content[close] != '<' || close + 1 >= content.size() || + content[close + 1] != '/') + continue; + if (markup_name_follows(content, close + 2, name)) break; + } + if (close > content.size()) close = content.size(); + markup_raw_region region{open_end + 1, std::min(close, content.size()), false}; + auto first = region.begin; + while (first < region.end && std::isspace(static_cast(content[first]))) ++first; + auto last = region.end; + while (last > first && std::isspace(static_cast(content[last - 1]))) --last; + region.single_hole = first < last && content[first] == '{' && content[last - 1] == '}'; + regions.push_back(region); + cursor = region.end; + } + return regions; +} + +inline formatted_text_scan scan_markup_content(const std::string &content, + unsigned depth) { + formatted_text_scan result; + result.end = content.size(); + const auto regions = markup_raw_regions(content); + const auto fail = [&](std::size_t at, const std::string &message) { + result.error_offset = at; + result.error = message; + }; + if (depth >= 64) { + fail(0, "interpolation nesting exceeds 64 levels"); + return result; + } + std::size_t cursor = 0, literal_start = 0, region = 0; + std::string literal; + const auto flush = [&]() { + if (!literal.empty()) + result.pieces.push_back({false, std::move(literal), literal_start, + cursor - literal_start}); + literal.clear(); + }; + while (cursor < content.size()) { + // A raw-text body that is not exactly one hole is literal, braces and all. + while (region < regions.size() && regions[region].end <= cursor) ++region; + if (region < regions.size() && !regions[region].single_hole && + cursor >= regions[region].begin && cursor < regions[region].end) { + literal.append(content, cursor, regions[region].end - cursor); + cursor = regions[region].end; + continue; + } + const char c = content[cursor]; + if (c == '{') { + if (cursor + 1 < content.size() && content[cursor + 1] == '{') { + literal += '{'; + cursor += 2; + continue; + } + flush(); + const auto begin = cursor + 1; + const auto end = formatted_expression_end(content, begin, true, depth + 1, result); + if (!result.error.empty()) return result; + result.pieces.push_back({true, content.substr(begin, end - begin), begin, end - begin}); + cursor = end + 1; + literal_start = cursor; + continue; + } + if (c == '}') { + if (cursor + 1 < content.size() && content[cursor + 1] == '}') { + literal += '}'; + cursor += 2; + continue; + } + fail(cursor, "unmatched '}' in markup; write '}}' for a literal brace"); + return result; + } + literal += c; + ++cursor; + } + flush(); + return result; +} + } // namespace ny #endif diff --git a/nilcompiler/src/matrix_lexer.cpp b/nilcompiler/src/matrix_lexer.cpp index e2c69c8b..8a40d1c0 100644 --- a/nilcompiler/src/matrix_lexer.cpp +++ b/nilcompiler/src/matrix_lexer.cpp @@ -433,16 +433,19 @@ void lexer::scan_formatted_string(std::vector &tokens, col_ = positions.column(pos_); } -// A multi-line formatted block is the raw 'begin text' block with holes. The +// A multi-line formatted block is the raw 'begin text' block with holes, and +// a markup block is the same machinery reading markup instead of text. The // literal text is dedented exactly as a raw block's is, while every hole keeps // its ORIGINAL source coordinates: indentation removal applies to literal // segments only, never to an expression, so definition, completion, signature // help and rename behave inside a block exactly as they do inside a // single-line formatted literal. -void lexer::scan_formatted_text_block(std::vector &tokens, - const token &word) { +void lexer::scan_block_literal(std::vector &tokens, const token &word, + bool markup) { const int line = word.line, col = word.column; - while (!at_end() && peek() != '\n') advance(); // 'begin text' and its line + const std::string closing = markup ? "end markup" : "end text"; + const std::string opening = markup ? "begin markup" : "formatted begin text"; + while (!at_end() && peek() != '\n') advance(); // the opening line if (!at_end()) advance(); // the newline itself const std::size_t header_end = pos_; @@ -459,7 +462,7 @@ void lexer::scan_formatted_text_block(std::vector &tokens, while (!at_end() && peek() != '\n') raw.push_back(advance()); if (!at_end()) advance(); if (!raw.empty() && raw.back() == '\r') raw.pop_back(); - if (normalise_line(raw) == "end text") { + if (normalise_line(raw) == closing) { terminated = true; terminator = start; break; @@ -467,8 +470,8 @@ void lexer::scan_formatted_text_block(std::vector &tokens, lines.push_back({start, raw}); } if (!terminated) { - error(line, col, "unterminated 'formatted begin text' block", - "close it with a line whose only content is 'end text'"); + error(line, col, "unterminated '" + opening + "' block", + "close it with a line whose only content is '" + closing + "'"); tokens.push_back(make(token_kind::invalid, "", line, col)); return; } @@ -507,7 +510,8 @@ void lexer::scan_formatted_text_block(std::vector &tokens, }; const auto scanned = - ny::scan_formatted_block(content, true, interpolation_depth_); + markup ? ny::scan_markup_content(content, interpolation_depth_) + : ny::scan_formatted_block(content, true, interpolation_depth_); const ny::formatted_source_positions positions(src_, word.start, pos_, word.line, word.column); const auto emit = [&](token_kind kind, std::string value, std::size_t offset, @@ -530,8 +534,8 @@ void lexer::scan_formatted_text_block(std::vector &tokens, return; } - emit(token_kind::formatted_string_begin, "formatted begin text", word.start, - header_end - word.start); + emit(markup ? token_kind::markup_begin : token_kind::formatted_string_begin, + opening, word.start, header_end - word.start); for (const auto &piece : scanned.pieces) { const auto start = source_offset(piece.start); const auto end = source_offset(piece.start + piece.length); @@ -557,19 +561,19 @@ void lexer::scan_formatted_text_block(std::vector &tokens, } emit(token_kind::interpolation_end, "}", end, 1); } - emit(token_kind::formatted_string_end, "end text", terminator, 8); + emit(markup ? token_kind::markup_end : token_kind::formatted_string_end, + closing, terminator, closing.size()); } -// The tail both block probes share: 'text' at 'probe', not running on into a +// The tail the block probes share: 'word' at 'probe', not running on into a // longer identifier, and nothing but spacing left on the line. -bool lexer::text_word_ends_line(std::size_t probe) const { - const char *word = "text"; - for (size_t i = 0; i < 4; ++i) { +bool lexer::word_ends_line(std::size_t probe, const char *word) const { + for (size_t i = 0; word[i]; ++i) { if (probe + i >= src_.size()) return false; if (ascii_lower(static_cast(src_[probe + i])) != word[i]) return false; } - probe += 4; + probe += std::char_traits::length(word); // The word must not run on into a longer identifier. if (probe < src_.size() && name_part(static_cast(src_[probe]))) @@ -585,7 +589,15 @@ bool lexer::text_block_follows() const { size_t probe = pos_; while (probe < src_.size() && (src_[probe] == ' ' || src_[probe] == '\t')) ++probe; - return text_word_ends_line(probe); + return word_ends_line(probe, "text"); +} + +// 'begin markup' at end of line opens a markup document literal. +bool lexer::markup_block_follows() const { + size_t probe = pos_; + while (probe < src_.size() && (src_[probe] == ' ' || src_[probe] == '\t')) + ++probe; + return word_ends_line(probe, "markup"); } // 'formatted begin text' at end of line opens an interpolated block literal. @@ -605,7 +617,7 @@ bool lexer::formatted_block_follows() const { return false; while (probe < src_.size() && (src_[probe] == ' ' || src_[probe] == '\t')) ++probe; - return text_word_ends_line(probe); + return word_ends_line(probe, "text"); } // Consume a raw block: every line after 'begin text' up to a line whose only @@ -707,7 +719,7 @@ std::vector lexer::scan() { } // 'formatted begin text' opens the same raw block with holes in it. if (formatted_block_follows()) { - scan_formatted_text_block(tokens, word); + scan_block_literal(tokens, word, false); continue; } } @@ -717,6 +729,11 @@ std::vector lexer::scan() { tokens.push_back(scan_text_block(word.line, word.column)); continue; } + // 'begin markup' opens a document literal: markup with holes in it. + if (word.is(kw::begin) && markup_block_follows()) { + scan_block_literal(tokens, word, true); + continue; + } tokens.push_back(word); continue; } diff --git a/nilcompiler/src/matrix_lexer.hpp b/nilcompiler/src/matrix_lexer.hpp index 17a15850..2dac89cc 100644 --- a/nilcompiler/src/matrix_lexer.hpp +++ b/nilcompiler/src/matrix_lexer.hpp @@ -131,6 +131,11 @@ enum class token_kind { interpolation_begin, interpolation_end, formatted_string_end, + // A markup literal reuses formatted_text and the interpolation tokens for + // its segments and holes; only the delimiters differ, because the parser + // reads the segments as markup rather than as text. + markup_begin, + markup_end, left_paren, right_paren, left_bracket, right_bracket, comma, @@ -222,15 +227,19 @@ class lexer { // Raw block literal: consumes lines after 'begin text' until a line whose // only content is 'end text', then dedents by the common indentation. token scan_text_block(int line, int col); - // Interpolated block literal: the raw 'begin text' block with holes. The - // literal text is dedented; every hole keeps its original source position. - void scan_formatted_text_block(std::vector &tokens, const token &word); + // Interpolated block literal: the raw 'begin text' block with holes, or the + // 'begin markup' document. The literal text is dedented; every hole keeps + // its original source position. + void scan_block_literal(std::vector &tokens, const token &word, + bool markup); // True when 'begin' is immediately followed by 'text' and end of line. bool text_block_follows() const; // True when 'formatted' is followed by 'begin text' and end of line. bool formatted_block_follows() const; - // Shared tail of both probes: 'text' at probe, then nothing but spacing. - bool text_word_ends_line(std::size_t probe) const; + // True when 'begin' is immediately followed by 'markup' and end of line. + bool markup_block_follows() const; + // Shared tail of the probes: the word at probe, then nothing but spacing. + bool word_ends_line(std::size_t probe, const char *word) const; token scan_backtick_identifier(); void error(int line, int col, std::string message, std::string hint = {}); }; diff --git a/nilcompiler/src/matrix_parser.cpp b/nilcompiler/src/matrix_parser.cpp index 2887e399..38863797 100644 --- a/nilcompiler/src/matrix_parser.cpp +++ b/nilcompiler/src/matrix_parser.cpp @@ -2998,6 +2998,8 @@ ny::ast_expr_ptr parser::parse_primary() { return number; } + if (check(matrix::token_kind::markup_begin)) return parse_markup_expression(); + if (match(matrix::token_kind::formatted_string_begin)) { std::vector parts; while (!check(matrix::token_kind::formatted_string_end) && !at_end()) { @@ -3229,6 +3231,384 @@ ny::ast_expr_ptr parser::parse_primary() { return std::make_shared("0", line); } +namespace { + +// HTML's closed sets. They sit beside the parser rather than inside its +// grammar on purpose (NIL-0027 section 3.4): the block parses generic +// elements, attributes and text, and only these tables know anything about +// HTML. A second dialect is another table, not another grammar. +bool markup_void_element(const std::string &name) { + static const char *const names[] = { + "area", "base", "br", "col", "embed", "hr", "img", "input", + "link", "meta", "param", "source", "track", "wbr"}; + for (const auto *candidate : names) + if (name == candidate) return true; + return false; +} + +bool markup_raw_text_element(const std::string &name) { + return name == "script" || name == "style"; +} + +// Attributes whose value is a URL. A hole that is the whole value is scheme +// filtered; a hole inside a longer value is percent encoded, because it is a +// component of someone else's URL rather than the URL itself. +bool markup_url_attribute(const std::string &name) { + static const char *const names[] = { + "href", "src", "action", "formaction", "cite", "poster", + "data", "srcset", "background", "ping", "manifest"}; + for (const auto *candidate : names) + if (name == candidate) return true; + return false; +} + +std::string markup_lower(std::string value) { + for (auto &letter : value) + if (letter >= 'A' && letter <= 'Z') letter += 'a' - 'A'; + return value; +} + +bool markup_name_byte(unsigned char c) { + return std::isalnum(c) || c == '-' || c == '_' || c == ':' || c >= 128; +} + +} // namespace + +// A markup literal (NIL-0027). The block's literal segments are parsed as +// markup so that the document's structure is checked and every hole learns the +// context it landed in; the segments are then emitted verbatim, because the +// author already wrote valid markup and this form does not reformat it. +ny::ast_expr_ptr parser::parse_markup_expression() { + const token start = advance(); // markup_begin + auto markup = std::make_shared(start.line); + using item_kind = ny::ast_markup_item::item_kind; + + enum class ctx { text, tag, attribute_value, raw_text, comment }; + ctx state = ctx::text; + std::vector open; + std::vector arm_base; + std::vector directive; // 1 when, 2 repeat + std::string attribute, raw_name; + char quote = 0; + bool value_started = false; + // A void element is never pushed, so '
' must not pop whatever is. + bool current_pushed = false; + std::size_t pending_url = std::string::npos; + bool reported_structure = false; + + const auto structure = [&](const token &at, const std::string &message, + const std::string &hint = {}) { + if (reported_structure) return; + reported_structure = true; + error(at, message, hint); + }; + const auto add = [&](item_kind kind, int line) -> ny::ast_markup_item & { + ny::ast_markup_item entry; + entry.kind = kind; + entry.line = line; + markup->items.push_back(std::move(entry)); + return markup->items.back(); + }; + const auto emit_literal = [&](const std::string &text, int line) { + if (text.empty()) return; + if (!markup->items.empty() && + markup->items.back().kind == item_kind::literal) { + markup->items.back().text += text; + return; + } + add(item_kind::literal, line).text = text; + }; + + // One literal segment, scanned as markup. Context carries across segments, + // because a hole may sit anywhere in a document, including inside a tag. + const auto scan_segment = [&](const token &piece) { + const std::string &text = piece.lexeme; + std::size_t cursor = 0; + // A hole in a URL attribute is only the whole value when the value ends + // immediately after it. + if (pending_url != std::string::npos) { + const bool whole = !text.empty() && text[0] == quote; + markup->items[pending_url].escape = whole ? "url" : "url_component"; + pending_url = std::string::npos; + } + while (cursor < text.size()) { + const char c = text[cursor]; + if (state == ctx::comment) { + const auto close = text.find("-->", cursor); + const auto stop = close == std::string::npos ? text.size() : close + 3; + emit_literal(text.substr(cursor, stop - cursor), piece.line); + if (close != std::string::npos) state = ctx::text; + cursor = stop; + continue; + } + if (state == ctx::raw_text) { + const auto close = markup_lower(text).find("') { + emit_literal(">", piece.line); + ++cursor; + if (!open.empty() && markup_raw_text_element(open.back())) { + raw_name = open.back(); + state = ctx::raw_text; + } else { + state = ctx::text; + } + continue; + } + if (c == '/' && cursor + 1 < text.size() && text[cursor + 1] == '>') { + emit_literal("/>", piece.line); + cursor += 2; + if (current_pushed && !open.empty()) open.pop_back(); + current_pushed = false; + state = ctx::text; + continue; + } + if (c == '=') { + emit_literal("=", piece.line); + ++cursor; + while (cursor < text.size() && (text[cursor] == ' ' || text[cursor] == '\t')) + emit_literal(std::string(1, text[cursor++]), piece.line); + if (cursor < text.size() && (text[cursor] == '"' || text[cursor] == '\'')) { + quote = text[cursor]; + value_started = false; + state = ctx::attribute_value; + emit_literal(std::string(1, text[cursor++]), piece.line); + } else { + structure(piece, "an attribute value must be quoted", + "write name=\"value\"; the renderer never adds quotes for you"); + } + continue; + } + if (markup_name_byte(static_cast(c))) { + const auto begin = cursor; + while (cursor < text.size() && + markup_name_byte(static_cast(text[cursor]))) + ++cursor; + attribute = markup_lower(text.substr(begin, cursor - begin)); + emit_literal(text.substr(begin, cursor - begin), piece.line); + continue; + } + emit_literal(std::string(1, c), piece.line); + ++cursor; + continue; + } + // ctx::text + if (c != '<') { + const auto stop = text.find('<', cursor); + const auto end = stop == std::string::npos ? text.size() : stop; + emit_literal(text.substr(cursor, end - cursor), piece.line); + cursor = end; + continue; + } + if (text.compare(cursor, 4, " + + +

{{literal braces}} and {raw "trusted"}

+ + + end markup +end paragraph + +main returns override integer begin paragraph + value names as array of text is ["alpha", "b", "gamma"] + value signed is page with signed-in being true, who being "Na", + names being names, query being "a b&c", target being "/dashboard" + say nil.std.markup::render with doc being signed + value guest is page with signed-in being false, who being "guest", + names being [], query being "x", target being "javascript:alert(1)" + say nil.std.markup::render with doc being guest + say nil.std.markup::render with doc being (nil.std.markup::empty with nothing) + give 0 +end paragraph diff --git a/tests/language/markup-matrix.stdout b/tests/language/markup-matrix.stdout new file mode 100644 index 00000000..a9a62b62 --- /dev/null +++ b/tests/language/markup-matrix.stdout @@ -0,0 +1,57 @@ + + Report & Review + +
+

Pro & Con

+

3 items

+
+ +
    + +
  • alpha
  • + +
  • b<ad>
  • + +
  • gamma
  • + +
+ target + search +
+ + + + +

{literal braces} and trusted

+ + + + Report & Review + +
+

Pro & Con

+

0 items

+
+ +
    + +
+ target + search +
+ + + + +

{literal braces} and trusted

+ + + diff --git a/tests/matrix/execution/guide-queries-patterns.expected b/tests/matrix/execution/guide-queries-patterns.expected index 397376da..be7e052a 100644 --- a/tests/matrix/execution/guide-queries-patterns.expected +++ b/tests/matrix/execution/guide-queries-patterns.expected @@ -8,3 +8,13 @@ 1 12 12 + + Home & Away + + + + diff --git a/tests/matrix/execution/guide-queries-patterns.matrix b/tests/matrix/execution/guide-queries-patterns.matrix index f5f7574a..93ff8c0e 100644 --- a/tests/matrix/execution/guide-queries-patterns.matrix +++ b/tests/matrix/execution/guide-queries-patterns.matrix @@ -1,16 +1,35 @@ -!!! Verbatim constructs from MATRIX-GUIDE.md sections 20 and 21, which need the -!!! standard library: queries read nil.std.collections terminals and patterns -!!! are compiled by nil.std.text. +!!! Verbatim constructs from MATRIX-GUIDE.md sections 20, 21 and 22, which need +!!! the standard library: queries read nil.std.collections terminals, patterns +!!! are compiled by nil.std.text, and a markup document is nil.std.markup's. book tour.guide-queries read everything from nil.std.text read everything from nil.std.collections +read everything from nil.std.markup dollar-amount begin pattern capture amount matching one or more ascii digits require next to match literal " USD" end pattern +page-shell receives title as text, signed-in as boolean, who as text + returns nil.std.markup.document begin paragraph + give begin markup + + {title} + + + + + end markup +end paragraph + main returns number begin paragraph value scores as array of number is [41, 7, 19, 55] value top is collect begin query @@ -43,5 +62,7 @@ main returns number begin paragraph value same is ((first_match with pattern being conventional, input being "price: 12 USD").unwrap with nothing).unwrap with nothing say (capture_named with found being same, name being "amount").unwrap with nothing + value page is page-shell with title being "Home & Away", signed-in being true, who being "" + say nil.std.markup::render with doc being page give 0 end paragraph diff --git a/tests/matrix/run_lexer_tests.py b/tests/matrix/run_lexer_tests.py index e179adb8..97104812 100755 --- a/tests/matrix/run_lexer_tests.py +++ b/tests/matrix/run_lexer_tests.py @@ -62,6 +62,9 @@ # NIL-0023 keeps ordinary variables and parameters named flag valid. "flag", "expect", + # NIL-0027: 'markup' is contextual after 'begin', and 'raw' only in front + # of an expression inside a markup hole. Both stay ordinary names. + "markup", "raw", } # Lowercase markers inside a numeric token are not standalone word terminals. diff --git a/tests/tools/markup_contract.py b/tests/tools/markup_contract.py new file mode 100644 index 00000000..315101e8 --- /dev/null +++ b/tests/tools/markup_contract.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""A markup block rejects malformed documents and escapes what it interpolates. + +The rejections are the point of the construct: a document whose elements do not +balance, a hole that would name an element or an attribute, and a conditional +arm that opens an element it does not close are all errors here rather than +broken HTML at run time. The positive checks prove the escaping that makes the +rejections worth having. +""" + +import argparse +from pathlib import Path +import subprocess +import tempfile + + +HEAD = 'book markup_rejection\nread everything from nil.std.markup\n' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--compiler', type=Path, required=True) + parser.add_argument('--interpreter', type=Path, required=True) + parser.add_argument('--stdlib', type=Path, required=True) + parser.add_argument('--llvm', choices=('ON', 'OFF'), required=True) + args = parser.parse_args() + compiler = args.compiler.resolve() + stdlib = args.stdlib.resolve() + routes = [('--no-bundle',), ('--no-bundle', '--emit-binary-nil')] + if args.llvm == 'ON': + routes += [('--native',), ('--native', '--jit')] + checks = 0 + + rejected = [ + # (name, body of main, whether nil.std.markup is read) + ('mismatched-close', '

x

', True), + ('unclosed-element', '

x

', True), + ('stray-close', '

', True), + ('arm-opens-only', '
{when flag}

a{end when}

', True), + ('arm-closes-foreign', '{repeat for each n in rows}
{end repeat}', True), + ('hole-names-attribute', '
y
', True), + ('hole-names-element', '<{label}>y', True), + ('unquoted-attribute', '
y
', True), + ('document-in-attribute', '
y
', True), + ('script-hole-without-raw', '', True), + ('comment-hole', '', True), + ('end-without-open', '

x

{end when}', True), + ('repeat-over-scalar', '{repeat for each n in flag}

x

{end repeat}', True), + ('condition-not-boolean', '{when label}

x

{end when}', True), + ('unsupported-hole-type', '

{rows}

', True), + ('missing-read', '

x

', False), + ] + with tempfile.TemporaryDirectory(prefix='nil-markup-contract-') as directory: + root = Path(directory) + for name, body, reads in rejected: + head = HEAD if reads else 'book markup_rejection\n' + source = (head + 'main returns override integer begin paragraph\n' + ' value flag is true\n' + ' value label as text is "cls"\n' + ' value rows as array of number is [1, 2]\n' + + (' value inner is begin markup\n x\n end markup\n' + if reads else '') + + ' value page is begin markup\n' + f' {body}\n' + ' end markup\n' + ' give 0\n' + 'end paragraph\n') + path = root / f'{name}.matrix' + path.write_text(source, newline='\n') + for flags in routes: + output = root / 'protected-output' + output.write_bytes(b'previous artifact') + result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), + *flags, '--output', output, path], + capture_output=True, text=True, timeout=60) + assert result.returncode != 0, (name, flags, result.stdout) + assert output.read_bytes() == b'previous artifact', (name, flags) + assert result.stderr.strip(), (name, flags) + checks += 1 + + # An unterminated block is a lexer error, before any markup parsing. + path = root / 'unterminated.matrix' + path.write_text(HEAD + 'main returns override integer begin paragraph\n' + ' value page is begin markup\n

x

\n' + ' give 0\nend paragraph\n', newline='\n') + result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), '--no-bundle', + '--output', root / 'unterminated.nil', path], + capture_output=True, text=True, timeout=60) + assert result.returncode != 0 and 'begin markup' in result.stderr, result.stderr + checks += 1 + + # Hostile values reach the page escaped, in every context, and a + # document composed into another is not escaped a second time. + hostile = (HEAD + + 'inner receives name as text returns nil.std.markup.document begin paragraph\n' + ' give begin markup\n {name}\n end markup\n' + 'end paragraph\n' + 'main returns override integer begin paragraph\n' + ' value attack as text is ""\n' + ' value page is begin markup\n' + '
{attack}
\n' + ' q\n' + ' {inner with name being attack}\n' + ' end markup\n' + ' say nil.std.markup::render with doc being page\n' + ' give 0\n' + 'end paragraph\n') + path = root / 'escaping.matrix' + path.write_text(hostile, newline='\n') + artifact = root / 'escaping.nil' + result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), '--no-bundle', + '--output', artifact, path], + capture_output=True, text=True, timeout=60) + assert result.returncode == 0, result.stderr + run = subprocess.run([args.interpreter.resolve(), artifact], + capture_output=True, text=True, timeout=60) + assert run.returncode == 0, run.stderr + page = run.stdout + # The only '<' that survive are the ones the document itself wrote. + assert '' not in page, page + assert '<script>alert(1)</script>' in page, page + assert 'class="<script>alert(1)</script>"' in page, page + assert '/q?v=%3cscript%3ealert(1)%3c%2fscript%3e' in page, page + assert '<script>alert(1)</script>' in page, page + assert '&lt;' not in page, page # composed documents escape once + checks += 6 + print(f'{checks} markup contract checks passed') + + +if __name__ == '__main__': + main() From c46dfa286bae98c0e1f90fd098a9a579ff863ef1 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Fri, 13 Nov 2026 02:04:11 -0500 Subject: [PATCH 36/39] Give Ny the multiline formatted block NIL-0027 section 2 for the other surface. '$"""' opens the block Matrix spells 'formatted begin text', sharing scan_formatted_block, so both frontends emit the same token stream and the same interpolation AST. One difference is forced by the surface rather than chosen: Matrix's 'end text' owns its line, while Ny closes on the first line whose content begins with '"""' and resumes lexing immediately after those quotes, because Ny is expression oriented and println($"""...""") has to work. A block is raw in both surfaces. Ny's backslash escapes apply to a quoted $"..." and not inside $""", which is what makes a path, a pattern or embedded JSON writable without doubling anything. Indentation removal applies to literal segments only, so holes keep their original file coordinates. An unterminated block now reports its own message rather than only 'Expected expression': the parser surfaces lexer messages for interpolation and formatted text, and formatted blocks join that list. Co-Authored-By: Claude Opus 5 --- docs/design/NIL-0027-structured-markup.md | 18 ++-- docs/language-reference/expressions.md | 24 ++++- nilcompiler/src/lexer.cpp | 126 ++++++++++++++++++++++ nilcompiler/src/lexer.hpp | 1 + nilcompiler/src/parser.cpp | 1 + tests/language/formatted-text.ny | 17 +++ tests/language/formatted-text.stdout | 6 ++ tests/tools/formatted_text_contract.py | 26 +++++ 8 files changed, 210 insertions(+), 9 deletions(-) diff --git a/docs/design/NIL-0027-structured-markup.md b/docs/design/NIL-0027-structured-markup.md index fc6b7ba3..a399fac2 100644 --- a/docs/design/NIL-0027-structured-markup.md +++ b/docs/design/NIL-0027-structured-markup.md @@ -1,8 +1,8 @@ # NIL-0027: Multiline formatted text and structured markup literals -- **Status:** §2 and §3 implemented for Matrix on 2026-11-13, with the - representation revised as recorded in §3.6. The Ny spellings of both remain - open. +- **Status:** §2 implemented for both surfaces on 2026-11-13. §3 implemented + for Matrix the same day, with the representation revised as recorded in + §3.6; the Ny spelling of §3 is settled in §6 and not yet built. - **Scope:** Ny and Matrix syntax, shared hole lexing and AST, checking, lowering, `nil.std.markup`, text builder - **Date:** 2026-11-13 @@ -51,7 +51,7 @@ member of that family. | Surface | Holes | Structure | Status | | --- | --- | --- | --- | | `formatted "..."` | yes | none, one line | NIL-0014, implemented | -| `formatted begin text ... end text` | yes | none, multiline | §2, implemented (Matrix) | +| `formatted begin text ... end text` / `$"""` | yes | none, multiline | §2, implemented | | `begin markup ... end markup` | yes | elements, attributes, control flow | §3, implemented (Matrix) | All three use the hole lexer NIL-0014 already built, which emits an expression's @@ -69,7 +69,10 @@ value page is formatted begin text end text ``` -Ny spells the same construct `$"""` ... `"""`. +Ny spells the same construct `$"""` ... `"""`. One difference is forced by the +surface: Matrix's `end text` owns its line, while Ny closes on the first line +whose content *begins* with `"""` and resumes lexing immediately after it, +because Ny is expression oriented and `println($"""...""")` has to work. The block follows the existing raw-block rules exactly — line-anchored terminator, common leading indentation removed, deeper indentation kept, @@ -103,7 +106,10 @@ routes; malformed blocks rejected on every route in word inside doubled braces in the dedented literal is left alone, which is the proof that indentation removal did not move the expression. -Ny still has only `$"..."`. Its multiline spelling is §6's open question. +**Ny followed on 2026-11-13.** `lexer::formatted_block` mirrors the Matrix +scanner and shares `scan_formatted_block`, so both surfaces produce the same +token stream and the same AST. A block is raw in both: Ny's backslash escapes +apply to a quoted `$"..."` and not inside `$"""`. ## 3. Markup literals diff --git a/docs/language-reference/expressions.md b/docs/language-reference/expressions.md index a8e838f2..34c8663d 100644 --- a/docs/language-reference/expressions.md +++ b/docs/language-reference/expressions.md @@ -205,14 +205,32 @@ and the same conversion is used by the interpreter and native compiler. Native text retains its existing NUL-free restriction; formatted text does not change the text storage ABI or permit embedded NUL constants natively. +Ny's multiline form opens with `$"""` and closes with the first line whose +content begins with `"""`, so a block can sit inside a call: + +```ny +println($""" +

Signed in as {who}

+

{count + 1} projects

+""") +``` + +A block is **raw**: unlike the quoted form, its literal segments carry no +backslash escapes, which is what makes a path, a regular expression or embedded +JSON writable without doubling anything. The block's common leading indentation +is removed and deeper indentation is kept, leading and trailing blank lines are +dropped, and indentation removal touches literal segments only, so every hole +keeps its original file coordinates. + `{{` and `}}` insert literal braces. Ny's normal backslash escapes still apply -to the literal portions, so `\n` inserts a newline in the result. Embedded +to the literal portions of a **quoted** formatted string, so `\n` inserts a +newline in the result. Embedded expressions keep their own string delimiters; no extra escaping is needed for `$"Result: {choose("first", "second")}"`. A quoted explicit string, including its expressions, must fit on one source -line; the Matrix block form below spans lines, and a single hole still stays on -one of them. Empty holes, trailing tokens, unmatched braces and missing quotes +line. Both surfaces have a block form that spans lines, and a single hole still +stays on one of them. Empty holes, trailing tokens, unmatched braces and missing quotes are errors. Formatting flags and precision specifiers are not part of this syntax; use an explicit formatting function inside a hole when needed. diff --git a/nilcompiler/src/lexer.cpp b/nilcompiler/src/lexer.cpp index 0ec46e52..5d1057cd 100644 --- a/nilcompiler/src/lexer.cpp +++ b/nilcompiler/src/lexer.cpp @@ -45,6 +45,11 @@ std::vector lexer::scan_tokens() { switch (c) { case '$': + if (peek() == '"' && peek_next() == '"' && current + 2 < src.size() && + src[current + 2] == '"') { + formatted_block(tokens); + continue; // Every emitted token already has its original range. + } if (peek() == '"') { formatted_string(tokens); continue; // Every emitted token already has its original range. @@ -551,6 +556,127 @@ token lexer::string() { return token{token_kind::string_literal, value, line}; } +// A multiline formatted block: $""" on its own line, content, then a line +// whose only content is """. It is the Ny spelling of Matrix's +// 'formatted begin text' (NIL-0027 section 2) and keeps the same rules: the +// block's common indentation is removed from the literal text, deeper +// indentation is kept, leading and trailing blank lines are dropped, and +// nothing inside is escaped -- a backslash in a block is a backslash, which is +// what makes a path, a regular expression or embedded JSON writable here. +// Indentation removal applies to the literal segments only, so a hole keeps +// its original file coordinates and stays editable. +void lexer::formatted_block(std::vector &tokens) { + const int opening_line = line; + const auto opening = start; + current += 3; // the three opening quotes + while (!is_at_end() && peek() != '\n') advance(); + if (!is_at_end()) advance(); + const auto header_end = current; + + const auto trimmed = [](const std::string &value) { + const auto first = value.find_first_not_of(" \t\r"); + if (first == std::string::npos) return std::string(); + const auto last = value.find_last_not_of(" \t\r"); + return value.substr(first, last - first + 1); + }; + + struct block_line { std::size_t start; std::string text; }; + std::vector lines; + bool terminated = false; + std::size_t terminator = current; + while (!is_at_end()) { + const auto begin = current; + std::string raw; + while (!is_at_end() && peek() != '\n') raw.push_back(advance()); + const bool had_newline = !is_at_end(); + if (had_newline) advance(); + if (!raw.empty() && raw.back() == '\r') raw.pop_back(); + // Ny is expression oriented, so the closing delimiter cannot own its line + // the way Matrix's 'end text' does: println($"""...""") has to work. + // The terminator is the first line whose content STARTS with three quotes, + // and lexing resumes immediately after them. + const auto first = raw.find_first_not_of(" \t"); + if (first != std::string::npos && raw.compare(first, 3, "\"\"\"") == 0) { + terminated = true; + terminator = begin + first; + current = terminator + 3; + break; + } + lines.push_back({begin, raw}); + if (!had_newline) break; + } + + const formatted_source_positions positions(src, opening, current, opening_line); + const auto emit = [&](token_kind kind, std::string value, std::size_t offset, + std::size_t length) { + tokens.push_back(token{kind, std::move(value), positions.line(offset), offset, length}); + }; + if (!terminated) { + emit(token_kind::invalid, + "unterminated formatted block; close it with a line whose only content is \"\"\"", + opening, 1); + line = positions.line(current); + return; + } + + while (!lines.empty() && trimmed(lines.front().text).empty()) + lines.erase(lines.begin()); + while (!lines.empty() && trimmed(lines.back().text).empty()) lines.pop_back(); + + std::size_t indent = std::string::npos; + for (const auto &entry : lines) { + const auto first = entry.text.find_first_not_of(" \t"); + if (first == std::string::npos) continue; // a blank line sets no indent + indent = std::min(indent, first); + } + if (indent == std::string::npos) indent = 0; + + std::string content; + std::vector logical_starts, source_starts; + for (std::size_t i = 0; i < lines.size(); ++i) { + if (i != 0) content.push_back('\n'); + const auto strip = std::min(indent, lines[i].text.size()); + logical_starts.push_back(content.size()); + source_starts.push_back(lines[i].start + strip); + content += lines[i].text.substr(strip); + } + const auto source_offset = [&](std::size_t logical) -> std::size_t { + if (logical_starts.empty()) return header_end; + const auto row = static_cast( + std::upper_bound(logical_starts.begin(), logical_starts.end(), logical) - + logical_starts.begin() - 1); + return source_starts[row] + (logical - logical_starts[row]); + }; + + const auto scanned = scan_formatted_block(content, false, interpolation_depth_); + if (!scanned.error.empty()) { + emit(token_kind::invalid, scanned.error, source_offset(scanned.error_offset), 1); + line = positions.line(current); + return; + } + tokens.push_back(token{token_kind::formatted_string_begin, "$\"\"\"", opening_line, + opening, header_end - opening}); + for (const auto &piece : scanned.pieces) { + const auto begin = source_offset(piece.start); + const auto end = source_offset(piece.start + piece.length); + if (!piece.expression) { + emit(token_kind::formatted_text, piece.text, begin, end - begin); + continue; + } + emit(token_kind::interpolation_begin, "{", begin - 1, 1); + lexer expression(piece.text, interpolation_depth_ + 1); + for (auto value : expression.scan_tokens()) { + if (value.kind == token_kind::eof) continue; + value.offset += begin; + value.line += positions.line(begin) - 1; + tokens.push_back(std::move(value)); + } + emit(token_kind::interpolation_end, "}", end, 1); + } + emit(token_kind::formatted_string_end, "\"\"\"", terminator, 3); + line = positions.line(current); +} + void lexer::formatted_string(std::vector &tokens) { const auto quote = current; const int opening_line = line; diff --git a/nilcompiler/src/lexer.hpp b/nilcompiler/src/lexer.hpp index 267f3fda..a77d471b 100644 --- a/nilcompiler/src/lexer.hpp +++ b/nilcompiler/src/lexer.hpp @@ -185,6 +185,7 @@ class lexer { token number(bool tuple_position = false); token string(); void formatted_string(std::vector &tokens); + void formatted_block(std::vector &tokens); token character(); // Lex a character literal 'c' token annotation(); // Lex an annotation token starting with '@' (e.g., @inline) diff --git a/nilcompiler/src/parser.cpp b/nilcompiler/src/parser.cpp index d3153305..4462590d 100644 --- a/nilcompiler/src/parser.cpp +++ b/nilcompiler/src/parser.cpp @@ -472,6 +472,7 @@ parser::parser(const std::string &source, const parser_options &opts) if (number.kind == token_kind::invalid && (number.lexeme.find("interpolation") != std::string::npos || number.lexeme.find("formatted text") != std::string::npos || + number.lexeme.find("formatted block") != std::string::npos || number.lexeme.rfind("Unterminated block comment", 0) == 0)) add_error(number, number.lexeme); if (number.kind == token_kind::number_literal && diff --git a/tests/language/formatted-text.ny b/tests/language/formatted-text.ny index 3ad1007d..0c0509b5 100644 --- a/tests/language/formatted-text.ny +++ b/tests/language/formatted-text.ny @@ -28,5 +28,22 @@ fn main() -> i32 println(render(1.25)) println(render("text")) println("legacy={value}") + + # A multiline formatted block: the common indentation is removed, deeper + # indentation is kept, nothing is escaped, and every hole is an ordinary + # expression evaluated once in written order. + let who = "world" + println($""" + hello {who} and {items[0]} + kept indentation, C:\raw\path and a "quote" + {{literal braces}} plus {next()}:{next()} + """) + println(tally) + let empty = $""" + """ + println(empty.len()) + println($""" + nested {$"inner {items[1]}"} done + """) return 0 end fn diff --git a/tests/language/formatted-text.stdout b/tests/language/formatted-text.stdout index aeba3e5f..99e8b1e8 100644 --- a/tests/language/formatted-text.stdout +++ b/tests/language/formatted-text.stdout @@ -14,3 +14,9 @@ value=17 value=1.25 value=text legacy=-1.6 +hello world and 7 + kept indentation, C:\raw\path and a "quote" +{literal braces} plus 4:5 +5 +0 +nested inner 11 done diff --git a/tests/tools/formatted_text_contract.py b/tests/tools/formatted_text_contract.py index e4a81ff9..1ad5daef 100644 --- a/tests/tools/formatted_text_contract.py +++ b/tests/tools/formatted_text_contract.py @@ -43,6 +43,32 @@ def main(): assert output.read_bytes() == b'previous artifact', (path, flags) assert result.stderr.strip(), (path, flags) checks += 1 + # Ny's multiline block rejects the same malformed holes. + ny_blocks = [ + 'hello\n', # never closed + 'a } b\n"\"\"\n', # unmatched brace + '{1 +\n2}\n"\"\"\n', # hole spans a line + '{}\n"\"\"\n', # empty hole + '{unknown}\n"\"\"\n', # undefined identifier + '{[1, 2]}\n"\"\"\n', # no text contract + ] + for index, block in enumerate(ny_blocks): + source = ('fn main() -> i32\n' + f'let result = $"\"\"\n{block}' + 'return 0\nend fn\n') + path = root / f'invalid-ny-block-{index}.ny' + path.write_text(source, newline='\n') + for flags in routes: + output = root / 'protected-output' + output.write_bytes(b'previous artifact') + result = subprocess.run([compiler, '--quiet', '--no-stdlib', + *flags, '--output', output, path], + capture_output=True, text=True, timeout=30) + assert result.returncode != 0, (path, flags, result.stdout) + assert output.read_bytes() == b'previous artifact', (path, flags) + assert result.stderr.strip(), (path, flags) + checks += 1 + # A multiline Matrix block rejects the same malformed holes, and an # unterminated block is a lexer error rather than a silent literal. blocks = [ From 010872548d1bd7b3c2900615306b1bf4af3c92ae Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Fri, 13 Nov 2026 02:24:28 -0500 Subject: [PATCH 37/39] Give Ny markup literals, from one shared document parser NIL-0027 section 3 for the other surface. The document parsing, the structural checks and the escape decisions move into markup_structure, which both frontends drive; each supplies only its own directive words and its own expression lexing, so the surfaces cannot drift in what a document means. Ny writes {if}, {elseif}, {else}, {end if}, {for x in xs} and {end for}; Matrix keeps {when} and {repeat for each}. The two parity fixtures render the same page byte for byte apart from their own indentation. Ny opens a block with '$markup' on its own line rather than the 'markup { ... }' the design record first guessed. A markup block is full of braces - holes, and the CSS or JavaScript inside a raw-text element - so a brace-delimited block would end at the first '}' in a stylesheet. The terminator has to be a line-anchored keyword, and the '$' is what keeps the opener unambiguous, since Ny has no 'begin' and a bare word before a newline could be an ordinary name. Two things only the Ny side could have found. The sigil skips six characters after the already-consumed '$', not seven, and getting that wrong swallowed the first content line into the header. Ny also names an imported type by the spelling the author wrote, so the document type is now recognised by the declaration it resolves to rather than by its name - a program's own struct called 'document' must not be spliced unescaped. Covered by a markup.ny parity fixture across the VM, binary, native and native-toolchain routes, and by ten more rejected documents plus an unterminated block in the markup contract. Co-Authored-By: Claude Opus 5 --- CMakeLists.txt | 2 +- docs/design/NIL-0027-structured-markup.md | 24 +- docs/language-reference/expressions.md | 55 ++++ docs/standard-library/markup.md | 3 +- nilcompiler/src/lexer.cpp | 40 ++- nilcompiler/src/lexer.hpp | 7 +- nilcompiler/src/markup_structure.hpp | 366 ++++++++++++++++++++++ nilcompiler/src/matrix_parser.cpp | 341 ++------------------ nilcompiler/src/parser.cpp | 81 +++++ nilcompiler/src/type_checker.cpp | 23 +- tests/language/markup.ny | 54 ++++ tests/language/markup.stdout | 57 ++++ tests/tools/markup_contract.py | 50 +++ 13 files changed, 765 insertions(+), 338 deletions(-) create mode 100644 nilcompiler/src/markup_structure.hpp create mode 100644 tests/language/markup.ny create mode 100644 tests/language/markup.stdout diff --git a/CMakeLists.txt b/CMakeLists.txt index e613b842..78f23a6e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3150,7 +3150,7 @@ if(BUILD_TESTING) bugpass-constructor-context bugpass-call-context collected-queries collected-queries-matrix - markup-matrix + markup markup-matrix signed-numbers-matrix integer-literal-widths integer-literal-widths-matrix checked-text-spans checked-text-spans-matrix diff --git a/docs/design/NIL-0027-structured-markup.md b/docs/design/NIL-0027-structured-markup.md index a399fac2..e3877569 100644 --- a/docs/design/NIL-0027-structured-markup.md +++ b/docs/design/NIL-0027-structured-markup.md @@ -1,8 +1,7 @@ # NIL-0027: Multiline formatted text and structured markup literals -- **Status:** §2 implemented for both surfaces on 2026-11-13. §3 implemented - for Matrix the same day, with the representation revised as recorded in - §3.6; the Ny spelling of §3 is settled in §6 and not yet built. +- **Status:** §2 and §3 implemented for both surfaces on 2026-11-13, with the + representation revised as recorded in §3.6. - **Scope:** Ny and Matrix syntax, shared hole lexing and AST, checking, lowering, `nil.std.markup`, text builder - **Date:** 2026-11-13 @@ -52,7 +51,7 @@ member of that family. | --- | --- | --- | --- | | `formatted "..."` | yes | none, one line | NIL-0014, implemented | | `formatted begin text ... end text` / `$"""` | yes | none, multiline | §2, implemented | -| `begin markup ... end markup` | yes | elements, attributes, control flow | §3, implemented (Matrix) | +| `begin markup ... end markup` / `$markup` | yes | elements, attributes, control flow | §3, implemented | All three use the hole lexer NIL-0014 already built, which emits an expression's tokens with their original byte offsets rather than re-lexing decoded text. @@ -309,6 +308,10 @@ file access from a document. once. 4. `nil.std.markup` constructors beyond `raw` and `empty`, so a document can be assembled from values rather than written literally, and the XML table. + Ny reached parity on 2026-11-13: `markup_structure` holds the document + parsing, the checks and the escape decisions, and each frontend supplies + only its own directive words and its own expression lexing, so the two + surfaces cannot drift in what a document means. 5. Diffing two documents, which is what would bring §3.6's arena back. ## 6. Open questions @@ -325,8 +328,17 @@ file access from a document. - **Performance claims.** §3.6 asserts an append count, not a measurement. Benchmark the rendered page against the concatenated equivalent before the claim appears in any guide. -- **Ny spelling.** `$"""` for §2 is settled by analogy; the Ny surface for - §3 is not. `markup { ... }` is the obvious candidate and is not yet decided. +- ~~**Ny spelling.**~~ Settled while implementing. `markup { ... }`, this + record's original candidate, does not survive contact with the content: a + markup block is full of braces — holes, and CSS or JavaScript inside a + raw-text element — so a brace-delimited block would end at the first `}` in a + stylesheet. The terminator has to be a line-anchored keyword, which is what + lets the block hold any bytes without escapes. Ny therefore opens with + `$markup` on its own line and closes with `end markup`. The `$` is what keeps + it unambiguous: Ny has no `begin`, so a bare word before a newline could be + an ordinary name, while `$` already means "a literal with holes in it" in + that surface. Matrix needs no sigil because `begin` supplies the same + disambiguation. ## 7. Compatibility diff --git a/docs/language-reference/expressions.md b/docs/language-reference/expressions.md index 34c8663d..b61fdba2 100644 --- a/docs/language-reference/expressions.md +++ b/docs/language-reference/expressions.md @@ -271,6 +271,61 @@ rename behave inside a block exactly as they do inside a quoted formatted literal. A block and a quoted literal produce the same interpolation AST, the same shared conversion and the same lowering on both execution routes. +## Markup literals + +A markup literal holds a document — elements, attributes, text and holes — and +its own control flow, so a generated page is written as one unit instead of +assembled from concatenated fragments. Matrix opens one with `begin markup`, +Ny with `$markup`, and both close with the block's own terminator: + +```ny +import nil.std.markup.escaping + +fn page(signed_in: bool, who: text, names: array) -> document + return $markup + +
    + {for name in names} +
  • {name}
  • + {end for} +
+ end markup +end fn +``` + +The directive words are each surface's own: Ny writes `{if}`, `{elseif}`, +`{else}`, `{end if}`, `{for x in xs}` and `{end for}`, while Matrix writes +`{when}`, `{otherwise when}`, `{otherwise}`, `{end when}`, +`{repeat for each x in xs}` and `{end repeat}`. Everything else is shared, +including the parsing of the document, the checks, the escaping and the value. + +The literal segments are parsed as markup at compile time. Unbalanced or +unclosed elements, an unterminated directive, an arm that opens an element it +does not close, and a hole that would name an element or an attribute are +compile errors. Element and attribute names are fixed; that is what keeps the +document checkable. + +Each hole is escaped for the context it landed in — element content, attribute +value, a whole URL attribute, or a component inside one — and `{raw ...}` is +the only way to interpolate without escaping. +[`nil.std.markup`](../standard-library/markup.md) holds those functions and the +`document` type the block produces, so a document interpolated into another +document is spliced rather than escaped a second time. Serialize one with +`render`. + +Inside ` +

{{literal braces}} and {raw "trusted"}

+ + + end markup +end fn + +fn main() -> i32 + let names = ["alpha", "b", "gamma"] + println(nil.std.markup::render(page(true, "Na", names, "a b&c", "/dashboard"))) + let empty: array = [] + println(nil.std.markup::render(page(false, "guest", empty, "x", "javascript:alert(1)"))) + println(nil.std.markup::render(nil.std.markup::empty())) + return 0 +end fn diff --git a/tests/language/markup.stdout b/tests/language/markup.stdout new file mode 100644 index 00000000..a9a62b62 --- /dev/null +++ b/tests/language/markup.stdout @@ -0,0 +1,57 @@ + + Report & Review + +
+

Pro & Con

+

3 items

+
+ +
    + +
  • alpha
  • + +
  • b<ad>
  • + +
  • gamma
  • + +
+ target + search +
+ + + + +

{literal braces} and trusted

+ + + + Report & Review + +
+

Pro & Con

+

0 items

+
+ +
    + +
+ target + search +
+ + + + +

{literal braces} and trusted

+ + + diff --git a/tests/tools/markup_contract.py b/tests/tools/markup_contract.py index 315101e8..2a154471 100644 --- a/tests/tools/markup_contract.py +++ b/tests/tools/markup_contract.py @@ -78,6 +78,56 @@ def main(): assert result.stderr.strip(), (name, flags) checks += 1 + # Ny rejects the same documents through the same shared structure, + # with its own control words. + ny_rejected = [ + ('ny-mismatched-close', '

x

'), + ('ny-unclosed-element', '

x

'), + ('ny-arm-opens-only', '
{if flag}

a{end if}

'), + ('ny-hole-names-attribute', '
y
'), + ('ny-unquoted-attribute', '
y
'), + ('ny-document-in-attribute', '
y
'), + ('ny-script-hole-without-raw', ''), + ('ny-end-without-open', '

x

{end if}'), + ('ny-condition-not-boolean', '{if label}

x

{end if}'), + ('ny-unsupported-hole-type', '

{rows}

'), + ] + for name, body in ny_rejected: + source = ('import nil.std.markup.escaping\n' + 'fn main() -> i32\n' + ' let flag = true\n' + ' let label = "cls"\n' + ' let rows = [1, 2]\n' + ' let inner = $markup\n x\n end markup\n' + ' let page = $markup\n' + f' {body}\n' + ' end markup\n' + ' return 0\n' + 'end fn\n') + path = root / f'{name}.ny' + path.write_text(source, newline='\n') + for flags in routes: + output = root / 'protected-output' + output.write_bytes(b'previous artifact') + result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), + *flags, '--output', output, path], + capture_output=True, text=True, timeout=60) + assert result.returncode != 0, (name, flags, result.stdout) + assert output.read_bytes() == b'previous artifact', (name, flags) + assert result.stderr.strip(), (name, flags) + checks += 1 + + # An unterminated Ny block is a lexer error too. + path = root / 'ny-unterminated.ny' + path.write_text('import nil.std.markup.escaping\n' + 'fn main() -> i32\n let page = $markup\n

x

\n' + ' return 0\nend fn\n', newline='\n') + result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), '--no-bundle', + '--output', root / 'ny-unterminated.nil', path], + capture_output=True, text=True, timeout=60) + assert result.returncode != 0 and 'markup block' in result.stderr, result.stderr + checks += 1 + # An unterminated block is a lexer error, before any markup parsing. path = root / 'unterminated.matrix' path.write_text(HEAD + 'main returns override integer begin paragraph\n' From e38145cb1d543320df7e7cba8deca1d9584cbffc Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Fri, 13 Nov 2026 11:22:18 -0500 Subject: [PATCH 38/39] Finish formatted text and markup validation for both language surfaces --- .../workflows/macos-application-parity.yml | 34 ++++ .github/workflows/web-runtime-parity.yml | 36 +++- README.md | 5 +- docs/design/NIL-0027-structured-markup.md | 157 ++++++++++----- docs/language-reference/expressions.md | 41 ++-- docs/standard-library/markup.md | 84 +++++--- matrix/MATRIX-GUIDE.md | 70 +++++-- matrix/docs/grammar.ebnf | 23 ++- matrix/docs/next-version-findings.md | 35 +++- nilcompiler/src/formatted_text.hpp | 70 +++++-- nilcompiler/src/lexer.cpp | 23 ++- nilcompiler/src/markup_structure.hpp | 148 ++++++++++++-- nilcompiler/src/matrix_lexer.cpp | 2 +- nilstd/src/markup/escaping.ny | 18 +- .../2026-11-13-formatted-markup-handoff.md | 88 +++++++++ notebook/README.md | 2 + tests/tools/completion_contract.py | 10 + tests/tools/formatted_text_contract.py | 19 ++ tests/tools/markup_contract.py | 184 ++++++++++++++++-- tests/tools/semantic_contract.py | 20 ++ 20 files changed, 897 insertions(+), 172 deletions(-) create mode 100644 notebook/2026-11-13-formatted-markup-handoff.md diff --git a/.github/workflows/macos-application-parity.yml b/.github/workflows/macos-application-parity.yml index 068dc611..5fae6c5e 100644 --- a/.github/workflows/macos-application-parity.yml +++ b/.github/workflows/macos-application-parity.yml @@ -25,6 +25,12 @@ name: macOS application platform checks - 'tests/tools/native_owned_lifetime.py' - 'tests/tools/native_symbol_surface.py' - 'tests/tools/file_modified_at_contract.py' + - 'tests/tools/formatted_text_contract.py' + - 'tests/tools/markup_contract.py' + - 'tests/language/formatted-text*' + - 'tests/language/markup*' + - 'tests/routes/ny_route_matrix.py' + - 'tests/tools/native_toolchain_pipeline.py' - 'tests/tools/project_build_contract.py' - 'tests/tools/tool_discovery_contract.py' - 'tests/tools/context_manifest.py' @@ -163,6 +169,34 @@ jobs: nil_darwin_toolchain_inputs_tests nil_freestanding_hook_default nil_freestanding_hook_override \ nildebugger_core_tests nildebugger_source_metadata_tests nildebugger_dap_owner_tests \ 2>&1 | tee build/application-evidence/build.log + - name: Verify formatted text and markup through interpreter and native routes + if: ${{ !cancelled() && steps.tools.outcome == 'success' }} + timeout-minutes: 8 + run: | + python3 - <<'PYTHON' + import json, subprocess, xml.etree.ElementTree as ET + from pathlib import Path + expected = {'nil.formatted-text-contract', 'nil.markup-contract'} + expected.update('ny.'+route+'.'+fixture for route in ['routes', 'tools'] + for fixture in ['formatted-text', 'formatted-text-matrix', 'markup', 'markup-matrix']) + pattern = r'^(nil[.](formatted-text-contract|markup-contract)|ny[.](routes|tools)[.](formatted-text|markup)(-matrix)?)$' + base = ['ctest', '--test-dir', 'build/application', '-R', pattern] + root = Path('build/application-evidence').resolve() + inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) + (root/'markup-inventory.json').write_bytes(inventory) + assert {case['name'] for case in json.loads(inventory)['tests']} == expected + with (root/'markup.log').open('wb') as output: + run = subprocess.run(base+['--no-tests=error', '--output-on-failure', + '--test-output-size-passed', '65536', + '--output-junit', str(root/'markup-results.xml')], stdout=output, + stderr=subprocess.STDOUT, timeout=450) + print((root/'markup.log').read_text(errors='replace')) + run.check_returncode() + cases = list(ET.parse(root/'markup-results.xml').getroot().iter('testcase')) + assert len(cases) == len(expected) and {case.get('name') for case in cases} == expected + assert all(case.get('status') == 'run' and all(case.find(kind) is None + for kind in ['failure', 'error', 'skipped']) for case in cases) + PYTHON - name: Verify debugger protocols and owned desktop sessions if: ${{ !cancelled() && steps.tools.outcome == 'success' }} timeout-minutes: 12 diff --git a/.github/workflows/web-runtime-parity.yml b/.github/workflows/web-runtime-parity.yml index ae4dba20..fd2b4f6e 100644 --- a/.github/workflows/web-runtime-parity.yml +++ b/.github/workflows/web-runtime-parity.yml @@ -11,6 +11,11 @@ name: Web runtime platform checks - 'tests/file_time_support_tests.cpp' - 'tests/tools/net_socket_contract.py' - 'tests/tools/file_modified_at_contract.py' + - 'tests/tools/formatted_text_contract.py' + - 'tests/tools/markup_contract.py' + - 'tests/language/formatted-text*' + - 'tests/language/markup*' + - 'tests/routes/ny_route_matrix.py' - 'tests/matrix/**' workflow_dispatch: @@ -89,13 +94,14 @@ jobs: "$NIL_CI_PYTHON" - <<'PY' import json, subprocess from pathlib import Path - pattern = r'^(net[.](support-contract|lifecycle-contract|socket-contract)|nil[.](file-time-support|file-modified-at-contract)|matrix[.]execution)$' + pattern = r'^(net[.](support-contract|lifecycle-contract|socket-contract)|nil[.](file-time-support|file-modified-at-contract|formatted-text-contract|markup-contract)|matrix[.]execution)$' base = ['ctest', '--test-dir', 'build/web-parity', '-C', 'Release', '-R', pattern] root = Path('build/web-parity-evidence').resolve() inventory = subprocess.check_output(base+['--show-only=json-v1'], timeout=30) (root/'inventory.json').write_bytes(inventory) expected = {'net.support-contract', 'net.lifecycle-contract', 'net.socket-contract', - 'nil.file-time-support', 'nil.file-modified-at-contract', 'matrix.execution'} + 'nil.file-time-support', 'nil.file-modified-at-contract', 'matrix.execution', + 'nil.formatted-text-contract', 'nil.markup-contract'} assert {test['name'] for test in json.loads(inventory)['tests']} == expected with (root/'ctest.log').open('wb') as output: result = subprocess.run(base+['--no-tests=error', '--output-on-failure', @@ -109,6 +115,32 @@ jobs: assert all(case.get('status') == 'run' and case.find('skipped') is None and case.find('failure') is None and case.find('error') is None for case in cases), 'Unexpected failure or skipped contract' PY + - name: Run formatted text and markup on every interpreted artifact route + timeout-minutes: 6 + run: | + "$NIL_CI_PYTHON" - <<'PYTHON' + import json, subprocess, sys + from pathlib import Path + root = Path('build/web-parity-evidence').resolve() + tests = json.loads((root/'inventory.json').read_text())['tests'] + command = next(test['command'] for test in tests if test['name'] == 'nil.markup-contract') + compiler = command[command.index('--compiler')+1] + interpreter = command[command.index('--interpreter')+1] + routes = ['text-interpreter', 'binary-interpreter', 'bundle-text', 'bundle-binary'] + for fixture in ['formatted-text', 'formatted-text-matrix', 'markup', 'markup-matrix']: + language = 'matrix' if fixture.endswith('-matrix') else 'ny' + report = root/(fixture+'-routes.json') + subprocess.run([sys.executable, 'tests/routes/ny_route_matrix.py', + '--compiler', compiler, '--interpreter', interpreter, + '--source', f'tests/language/{fixture}.{language}', + '--expected-stdout', f'tests/language/{fixture}.stdout', '--expected-exit', '0', + '--routes', ','.join(routes), '--strict', '--compiler-arg=--sdk-path', + '--compiler-arg='+str(Path('nilstd/src').resolve()), + '--work-dir', 'build/web-parity/language-routes', '--report-json', str(report)], + check=True, timeout=75) + data = json.loads(report.read_text()) + assert len(data['results']) == 4 and all(item['status'] == 'PASS' for item in data['results']) + PYTHON - name: Preserve source identity and evidence if: always() timeout-minutes: 2 diff --git a/README.md b/README.md index 34ecebce..7ee02c9e 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,10 @@ The framework includes: typed faults, and bounded parallel execution. - **Shared language features.** Structural tuples provide typed multiple results and local destructuring. Explicit string interpolation supports - expressions with consistent scalar formatting across execution routes. + expressions with consistent scalar formatting across execution routes, including + multiline blocks. [Structured markup](docs/standard-library/markup.md) keeps + generated HTML and its conditionals in one checked document, with context-specific + escaping and document composition in both Ny and Matrix. [Collected queries](docs/standard-library/queries.md) compose filtering, stable ordering and projection. [Text patterns](docs/standard-library/text-patterns.md) offer readable declarations, bounded regex matching with named captures, diff --git a/docs/design/NIL-0027-structured-markup.md b/docs/design/NIL-0027-structured-markup.md index e3877569..4f00abc5 100644 --- a/docs/design/NIL-0027-structured-markup.md +++ b/docs/design/NIL-0027-structured-markup.md @@ -1,7 +1,8 @@ # NIL-0027: Multiline formatted text and structured markup literals -- **Status:** §2 and §3 implemented for both surfaces on 2026-11-13, with the - representation revised as recorded in §3.6. +- **Status:** The initial §2 and §3 profile is implemented for both surfaces + on 2026-11-13. The current contract below and §3.6 supersede the original + runtime-tree, optional-hole and dialect-renderer proposals. - **Scope:** Ny and Matrix syntax, shared hole lexing and AST, checking, lowering, `nil.std.markup`, text builder - **Date:** 2026-11-13 @@ -53,6 +54,17 @@ member of that family. | `formatted begin text ... end text` / `$"""` | yes | none, multiline | §2, implemented | | `begin markup ... end markup` / `$markup` | yes | elements, attributes, control flow | §3, implemented | +**Current implementation contract.** A markup literal is checked as a document +at compile time, then evaluates to `nil.std.markup.document`, a record of +serialized text. Static markup stays literal; holes and directives assemble +the result at run time. `render` returns that text. There is no runtime DOM, +XML mode, compact renderer or automatic attribute omission. Ordinary holes +accept text and the formatted-text scalar types; a document splices only into +element content. Arrays and optional values need an explicit loop or +conversion. The original tree and renderer proposals are retained in §3.4–3.5 +as design history, not available APIs. The library reference records the +[current escaping policy](../standard-library/markup.md). + All three use the hole lexer NIL-0014 already built, which emits an expression's tokens with their original byte offsets rather than re-lexing decoded text. Everything in this record inherits that contract, and no construct here @@ -71,7 +83,9 @@ end text Ny spells the same construct `$"""` ... `"""`. One difference is forced by the surface: Matrix's `end text` owns its line, while Ny closes on the first line whose content *begins* with `"""` and resumes lexing immediately after it, -because Ny is expression oriented and `println($"""...""")` has to work. +because Ny is expression oriented and the closing `""")` must be allowed in +a call. The `$"""` opener must end its source line, apart from spaces or tabs; +content starts on the following line. The block follows the existing raw-block rules exactly — line-anchored terminator, common leading indentation removed, deeper indentation kept, @@ -79,12 +93,11 @@ leading and trailing blank lines dropped, no escape sequences — and adds `{expression}` holes with `{{` and `}}` for literal braces. A single hole stays on one physical line, which keeps the expression lexer unchanged. -The one implementation constraint, named as follow-up work in NIL-0014: **the -dedent must not move the holes.** Indentation removal applies to literal -segments at lowering. The lexer records each segment's original span and the -number of stripped bytes per line, and hole tokens keep their original -coordinates, so definition, completion, signature help and local rename behave -as they do inside a single-line formatted literal. +The implementation constraint, named as follow-up work in NIL-0014: **the +dedent must not move the holes.** The block scanner removes common indentation +and retains a per-line mapping back to the source. Hole tokens therefore keep +their original coordinates, so definition, completion, signature help and +local rename behave as they do inside a single-line formatted literal. This is the smallest useful step and it is worth taking on its own: it is what a C# raw interpolated string gives, and it removes most `plus` ladders from @@ -113,8 +126,10 @@ apply to a quoted `$"..."` and not inside `$"""`. ## 3. Markup literals ```text +read nil.std.markup + public page-shell receives title as text, signed-in as boolean, who as text - returns markup begin paragraph + returns nil.std.markup.document begin paragraph give begin markup {title} @@ -139,8 +154,10 @@ page remains one unit a person can read, edit and paste. ### 3.1 What the block contains -Elements `children`, self-closing ``, text, -comments, holes, and directives. Directives are spelled with the language's own +Elements `children`, HTML void elements such as `
`, +text, comments, holes, and directives. Self-closing syntax is accepted only for +HTML void elements; a non-void element needs its matching closing tag. This +profile uses HTML tables and does not model SVG or MathML namespace semantics. Directives are spelled with the language's own control words, so the block introduces no second vocabulary: ```text @@ -152,12 +169,19 @@ control words, so the block introduces no second vocabulary: ``` `{when ...}` / `{otherwise when ...}` / `{otherwise}` / `{end when}` and -`{repeat for each ... in ...}` / `{repeat for ... from ... to ...}` / -`{end repeat}` are the complete directive set. A directive may stand between -elements, between attributes, or inside a text node or attribute value. Its -arms must each be well-formed: a directive cannot open an element in one arm -and close it in another, because that reintroduces exactly the fragment -assembly this record exists to remove. +`{repeat for each ... in ...}` / `{end repeat}` are the implemented Matrix +directives. Ny uses `{if ...}`, `{elseif ...}`, `{else}`, `{end if}`, +`{for ... in ...}` and `{end for}`. Numeric-range directives from the original +proposal are not implemented. + +A directive may stand between elements, between attributes, or inside text or +an ordinary attribute value. Each arm must restore both the element stack and +the surrounding tag/attribute context: it cannot leave an element or quote +open for another arm to close. Directives inside URL attribute values are +rejected in this profile; calculate the URL beforehand or conditionally emit +the complete element. A bare boolean attribute cannot be an arm's entire +content: use a complete quoted attribute such as `disabled=""` inside that arm, +or conditionally emit the complete element. **A hole cannot name an element or an attribute.** Both are fixed at compile time, which is what makes the tree checkable. Dynamic attribute sets are @@ -165,16 +189,22 @@ future work (§6). ### 3.2 Hole typing, by position -| Position | Accepted | Notes | -| --- | --- | --- | -| Text node | `text`, scalar, `markup`, `array of markup`, `nothing` | `markup` splices as nodes, never as escaped text | -| Attribute value | `text`, scalar, `nothing` | `nothing` omits the attribute entirely | -| URL attribute | `text` | scheme-checked; see §3.4 | -| Raw-text element body | `text` via `{raw ...}` only | see §3.3 | +The initial proposal included arrays of documents and `nothing` holes. Those +are not implemented; this table describes the current checker. -`{raw expression}` splices text without escaping. It is deliberately a word a -reviewer can grep for, and it is the only way to put unescaped text into a -document. +| Position | Accepted | Behavior | +| --- | --- | --- | +| Element content | `text`, `bool`, `i32`, `i64`, `u8`, `f64`, `document` | Text/scalars escape; a document splices its serialized content | +| Ordinary attribute value | `text` and the same scalar types | Escape within the source's existing quotes; no implicit omission | +| URL attribute | `text` and the same scalar types | Whole-value scheme filter or fixed-prefix component encoding; see §3.4 | +| Code-bearing attribute or raw-text element body | Explicit `{raw ...}` only | Trusted bypass, not an escaping or sanitizing operation | + +Use a loop for an array and explicit control flow for an optional value. +`{raw expression}` is the explicit trusted bypass. It accepts text or a +document in content or an already quoted attribute value, and does not +validate or escape the inserted bytes. Even `raw` cannot name an element or +attribute or appear in a comment. A manually constructed document and `nil.std.markup::raw` likewise require trusted input; +the public record is not a security boundary. ### 3.3 Decision: raw-text elements are literal @@ -191,7 +221,13 @@ The rule is a single lookahead over the element body and needs no escaping convention, which keeps the no-escapes property of Matrix text intact in the place where escaping would hurt most. -### 3.4 Decision: the parser is dialect-free +### 3.4 Original proposal: a dialect-free renderer (superseded) + +**Design history.** The following renderer/table proposal preceded §3.6. +The implementation instead uses shared compiler-side HTML tables and library +escaping. Source quotes and self-closing spelling are preserved, and only HTML +is supported. The proposed scheme allowlist and runtime rejection below were +not implemented; see the current policy after the list. The block parses generic elements, attributes, text, comments and holes, and checks balance. Everything specific to HTML lives in one table the **renderer** @@ -211,7 +247,21 @@ Escaping is per context and belongs to the same table: A hole producing a `javascript:` or `data:` URL is refused at run time unless it came through `{raw}`. -### 3.5 Decision: the value is a tree, not text +**Current URL policy.** A whole-value hole is checked for `javascript:`, +`vbscript:` and `data:` after browser-relevant ASCII control/space normalization +and ASCII case folding. A match encodes the colons as `%3a`, then applies +attribute escaping; it does not throw or omit the attribute. Other schemes +are not allowlisted. A component hole requires a fixed safe prefix such as +`/search?q=`, `https://example.com/`, `?q=` or `#section-`; variable scheme +assembly is rejected. Ordinary holes in `on*`, `style`, `srcdoc`, `srcset`, +`imagesrcset` and `ping` attributes require an explicit trusted `raw` bypass. +This policy is not an HTML, JavaScript, CSS or URL-list sanitizer. + +### 3.5 Original proposal: a runtime tree (superseded) + +**Design history.** This argument motivated the first design. §3.6 replaced +its runtime representation and rendering API; `render ... as xml` and +`render ... compact` below are proposals, not implemented syntax. `begin markup` evaluates to `markup`. `nil.std.markup::render with document being page` produces `text`; `render ... as xml` and a `compact` option select @@ -230,7 +280,7 @@ doing this at all: what a future webview host wants, and it is unavailable from a string forever after. -### 3.6 Representation: revised — serialized at compile time +### 3.6 Representation: revised — compiled serialization with runtime holes **This section replaced its original proposal on 2026-11-13, during implementation.** It first specified a runtime arena: parallel scalar arrays @@ -246,8 +296,9 @@ arrays does work on both routes, so the plan was sound. The second is that it buys almost nothing here. Of §3.5's four reasons for a tree, three do not need one at run time: -- **Context-correct escaping** is decided while the block is parsed, so it - costs nothing at run time and is *more* reliable than a tree walk. +- **Context selection** happens while the block is parsed. The generated + code still calls the escaping functions at run time, but does not traverse + a runtime tree to decide which escaper to use. - **Balance checking** happens at parse time, whatever the value becomes. - **Composition without seams** needs only that a document be a distinct *type*, not a distinct *shape*: an already-escaped fragment spliced into @@ -256,11 +307,14 @@ tree, three do not need one at run time: Only the fourth — rendering one document to several dialects, or diffing two — needs a runtime tree, and that was already deferred to §5. -So a `markup` value is `nil.std.markup.document`, a record holding the -serialized, already-escaped text. The literal's static segments are compile- -time constants; only holes and control flow cost anything. The type is what -carries the guarantee, and the shape behind it stays an implementation detail: -moving to an arena later changes no source. +The implemented value type is `nil.std.markup.document`, a record holding +serialized text whose ordinary holes have already been escaped. Static +segments are compile-time constants; holes, control flow and result assembly +execute at run time. The type distinguishes +composition from ordinary text interpolation. Its public field and `raw` +constructor still allow trusted callers to supply arbitrary markup; the type +does not certify that manually constructed content is safe. A future arena +would need to preserve the public API or explicitly migrate its callers. What this costs, stated plainly: the dialect is fixed when the block is compiled rather than when it is rendered, so `begin markup as xml` becomes a @@ -302,26 +356,29 @@ file access from a document. `parser::parse_markup_expression`, checked in `type_checker::visit` (escape versus splice) and emitted by `emit_markup` as ordinary NIL, so both execution routes are covered by one emitter. Coverage: the `markup-matrix` - parity fixture across the VM, binary, native and native-toolchain routes; - `tests/tools/markup_contract.py` rejecting sixteen malformed documents on - every route and proving hostile input is escaped in each context exactly - once. + parity fixtures across the VM, binary, native and native-toolchain routes; + `tests/tools/markup_contract.py` checks malformed-document rejection and + context-specific escaping on the supported routes. 4. `nil.std.markup` constructors beyond `raw` and `empty`, so a document can be assembled from values rather than written literally, and the XML table. - Ny reached parity on 2026-11-13: `markup_structure` holds the document - parsing, the checks and the escape decisions, and each frontend supplies - only its own directive words and its own expression lexing, so the two - surfaces cannot drift in what a document means. + **Deferred.** 5. Diffing two documents, which is what would bring §3.6's arena back. + **Deferred.** + +Ny reached the initial profile on 2026-11-13: `markup_structure` holds document +parsing, structural checks and escape decisions. Each frontend supplies its +own directive words and expression lexing; parity fixtures exercise the shared +semantics. ## 6. Open questions - **Dynamic attribute sets.** `{spread attrs}` in attribute position, taking a list of name/value pairs, would cover `data-*` generation. Deferred: it weakens compile-time checking and is not needed by the first consumers. -- **Whitespace.** The initial rule is to preserve the block's text as written, - because HTML whitespace is sometimes significant, with collapsing offered by - `render ... compact`. Worth revisiting once there is real output to compare. +- **Whitespace.** The initial rule preserves text after the block's common + indentation and surrounding blank lines are removed. A composed document + keeps its own remaining indentation. `render ... compact` is a deferred + proposal; there is no whitespace-collapsing renderer in this version. - **Editor support.** Holes keep original spans by construction (§2), but the VS Code grammar needs markup-inside-Matrix highlighting or the construct will look worse in an editor than the string it replaces. @@ -334,7 +391,9 @@ file access from a document. raw-text element — so a brace-delimited block would end at the first `}` in a stylesheet. The terminator has to be a line-anchored keyword, which is what lets the block hold any bytes without escapes. Ny therefore opens with - `$markup` on its own line and closes with `end markup`. The `$` is what keeps + `$markup` at the end of its opening line and closes with line-leading + `end markup`. Ny resumes the surrounding expression after that terminator; + Matrix requires the terminator to occupy its line. The `$` is what keeps it unambiguous: Ny has no `begin`, so a bare word before a newline could be an ordinary name, while `$` already means "a literal with holes in it" in that surface. Matrix needs no sigil because `begin` supplies the same diff --git a/docs/language-reference/expressions.md b/docs/language-reference/expressions.md index b61fdba2..c3942c88 100644 --- a/docs/language-reference/expressions.md +++ b/docs/language-reference/expressions.md @@ -205,8 +205,10 @@ and the same conversion is used by the interpreter and native compiler. Native text retains its existing NUL-free restriction; formatted text does not change the text storage ABI or permit embedded NUL constants natively. -Ny's multiline form opens with `$"""` and closes with the first line whose -content begins with `"""`, so a block can sit inside a call: +Ny's multiline form opens with `$"""` at the end of its source line, apart +from spaces or tabs; content starts on the following line. It closes with the +first line whose non-indentation content begins with `"""`. Lexing resumes +after that delimiter, so a block can sit inside a call: ```ny println($""" @@ -309,22 +311,35 @@ The literal segments are parsed as markup at compile time. Unbalanced or unclosed elements, an unterminated directive, an arm that opens an element it does not close, and a hole that would name an element or an attribute are compile errors. Element and attribute names are fixed; that is what keeps the -document checkable. - -Each hole is escaped for the context it landed in — element content, attribute -value, a whole URL attribute, or a component inside one — and `{raw ...}` is -the only way to interpolate without escaping. -[`nil.std.markup`](../standard-library/markup.md) holds those functions and the -`document` type the block produces, so a document interpolated into another -document is spliced rather than escaped a second time. Serialize one with -`render`. +document checkable. Self-closing syntax is limited to HTML void elements such +as `
`; other elements need a closing tag. The initial profile uses HTML +tables, without SVG or MathML namespace semantics. + +Ordinary holes accept `text`, `bool`, `i32`, `i64`, `u8` and `f64`, escaping +them for their position. A `nil.std.markup.document` also composes into element +content without being escaped again. Arrays and optional values need explicit +loops or conversions; they do not splice or omit attributes automatically. + +A whole URL hole uses a limited executable-scheme filter, while a component +hole needs a fixed safe URL prefix. Directives inside URL attribute values and +ordinary holes in code-bearing or URL-list attributes are rejected. See the +[`nil.std.markup` escaping contract](../standard-library/markup.md) for the +exact policy. `{raw ...}` explicitly trusts its input and bypasses escaping; +it is not a sanitizer. + +The block evaluates to a document record containing serialized text. `render` +returns that text; it does not walk a runtime tree. XML serialization and a +compact renderer are not implemented. Inside ` -->'), + ] + for name, body in shared_rejected + contextual_rejected: + if language == 'ny': + source = ('import nil.std.markup.escaping\nfn main() -> i32\n' + 'let flag = true\nlet label = "marker"\n' + f'let page = $markup\n{body}\nend markup\nreturn 0\nend fn\n') + else: + source = (HEAD + 'main returns override integer begin paragraph\n' + 'value flag is true\nvalue label as text is "marker"\n' + f'value page is begin markup\n{body}\nend markup\ngive 0\nend paragraph\n') + path = root / f'context-{name}.{language}' + path.write_text(source, encoding='utf-8', newline='\n') + for flags in routes: + output = root / 'protected-context-output' + output.write_bytes(b'previous artifact') + result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), + *flags, '--output', output, path], + capture_output=True, text=True, encoding='utf-8', timeout=60) + assert result.returncode != 0, (name, language, flags, result.stdout) + assert output.read_bytes() == b'previous artifact', (name, language, flags) + assert result.stderr.strip(), (name, language, flags) + checks += 1 + + def render_routes(name, language, source, expected): + nonlocal checks + path = root / f'{name}.{language}' + path.write_text(source, encoding='utf-8', newline='\n') + for route, flags in enumerate(routes): + artifact = root / f'{name}-{language}-{route}' + result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), + *flags, '--output', artifact, path], + capture_output=True, text=True, encoding='utf-8', timeout=60) + assert result.returncode == 0, (name, flags, result.stderr) + if '--jit' in flags: + actual = result.stdout + else: + executable = artifact.with_suffix('.exe') if os.name == 'nt' else artifact + command = ([executable] if '--native' in flags else + [args.interpreter.resolve(), artifact]) + run = subprocess.run(command, capture_output=True, text=True, encoding='utf-8', timeout=60) + assert run.returncode == 0, (name, flags, run.stderr) + actual = run.stdout + assert actual == expected, (name, language, flags, actual, expected) + checks += 1 + + # Both surfaces use the same context decisions, and literal script + # lookalikes inside comments/quoted attributes never hide later holes. + for language in ('ny', 'matrix'): + opening, alternative, end = (('if', 'else', 'end if') if language == 'ny' + else ('when', 'otherwise', 'end when')) + body = ('path\n' + 'absolute\n' + 'query\n' + 'fragment\n' + 'whole\n' + 'svg\n' + '

{label}

\n' + '\n

{label}

\n' + '\n' + '\n' + '\n' + f'
x
\n' + f'

text

\n' + '') + expected = ('path\n' + 'absolute\n' + 'query\n' + 'fragment\n' + 'whole\n' + 'svg\n' + '

<marker>&

\n' + '\n

<marker>&

\n' + '\n' + '\n' + '\n' + '
x
\n' + '

text

\n' + '\n') + if language == 'ny': + source = ('import nil.std.markup.escaping\nfn main() -> i32\n' + 'let flag = true\nlet label = "&"\n' + 'let target = "JaVaScRiPt:marker"\nlet trusted = "marker()"\n' + f'let page = $markup\n{body}\nend markup\n' + 'println(nil.std.markup::render(page))\nreturn 0\nend fn\n') + else: + source = (HEAD + 'main returns override integer begin paragraph\n' + 'value flag is true\nvalue label as text is "&"\n' + 'value target as text is "JaVaScRiPt:marker"\nvalue trusted as text is "marker()"\n' + f'value page is begin markup\n{body}\nend markup\n' + 'say nil.std.markup::render with doc being page\ngive 0\nend paragraph\n') + render_routes('context-positive', language, source, expected) + + # The whole URL decision follows browser control normalization. These + # strings remain data; no browser or JavaScript execution is needed. + controls = [('java\\tscript:marker', 'java\tscript%3amarker'), + ('java\\nscript:marker', 'java\nscript%3amarker'), + ('java\\rscript:marker', 'java\nscript%3amarker'), + ('\\v\\f JaVaScRiPt:marker', '\v\f JaVaScRiPt%3amarker'), + ('\x01javascript:marker', '\x01javascript%3amarker')] + source = 'import nil.std.markup.escaping\nfn main() -> i32\n' + expected = '' + for index, (value, escaped) in enumerate(controls): + source += f'let value{index} = "{value}"\nlet page{index} = $markup\nx\nend markup\n' + source += f'println(nil.std.markup::render(page{index}))\n' + expected += f'x\n' + source += ('let page = $markup\n

{"a\\\"}b"}

\n

{"x" /* } */}

\nend markup\n' + 'println(nil.std.markup::render(page))\nreturn 0\nend fn\n') + expected += '

a"}b

\n

x

\n' + render_routes('url-controls-and-ny-holes', 'ny', source, expected) + + # The line-anchored terminator is a keyword, not the prefix of a + # longer body word, and Ny still permits expression continuation. + source = ('import nil.std.markup.escaping\nfn main() -> i32\n' + 'println(nil.std.markup::render($markup\n

\n' + 'end markupExtra\nend markup_extra\nend markupé\n

\n' + 'end markup))\nreturn 0\nend fn\n') + render_routes('ny-terminator-boundary', 'ny', source, + '

\nend markupExtra\nend markup_extra\nend markupé\n

\n') + # An unterminated Ny block is a lexer error too. path = root / 'ny-unterminated.ny' path.write_text('import nil.std.markup.escaping\n' 'fn main() -> i32\n let page = $markup\n

x

\n' - ' return 0\nend fn\n', newline='\n') + ' return 0\nend fn\n', encoding='utf-8', newline='\n') result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), '--no-bundle', '--output', root / 'ny-unterminated.nil', path], - capture_output=True, text=True, timeout=60) + capture_output=True, text=True, encoding='utf-8', timeout=60) assert result.returncode != 0 and 'markup block' in result.stderr, result.stderr checks += 1 @@ -132,10 +294,10 @@ def main(): path = root / 'unterminated.matrix' path.write_text(HEAD + 'main returns override integer begin paragraph\n' ' value page is begin markup\n

x

\n' - ' give 0\nend paragraph\n', newline='\n') + ' give 0\nend paragraph\n', encoding='utf-8', newline='\n') result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), '--no-bundle', '--output', root / 'unterminated.nil', path], - capture_output=True, text=True, timeout=60) + capture_output=True, text=True, encoding='utf-8', timeout=60) assert result.returncode != 0 and 'begin markup' in result.stderr, result.stderr checks += 1 @@ -156,14 +318,14 @@ def main(): ' give 0\n' 'end paragraph\n') path = root / 'escaping.matrix' - path.write_text(hostile, newline='\n') + path.write_text(hostile, encoding='utf-8', newline='\n') artifact = root / 'escaping.nil' result = subprocess.run([compiler, '--quiet', '-I', str(stdlib), '--no-bundle', '--output', artifact, path], - capture_output=True, text=True, timeout=60) + capture_output=True, text=True, encoding='utf-8', timeout=60) assert result.returncode == 0, result.stderr run = subprocess.run([args.interpreter.resolve(), artifact], - capture_output=True, text=True, timeout=60) + capture_output=True, text=True, encoding='utf-8', timeout=60) assert run.returncode == 0, run.stderr page = run.stdout # The only '<' that survive are the ones the document itself wrote. diff --git a/tests/tools/semantic_contract.py b/tests/tools/semantic_contract.py index 7e619464..44386b19 100644 --- a/tests/tools/semantic_contract.py +++ b/tests/tools/semantic_contract.py @@ -348,6 +348,26 @@ def rename_local(result, file, target, new_name): assert '{{local-value}}' in snapshot, snapshot assert '{renamed-value plus 1}' in snapshot, snapshot + # Ny closes the block inside the enclosing call. Its independently + # emitted tokens must preserve the same UTF-8/CRLF editing spans. + block_ny = ('# \u00e9\U0001F600\r\nfn message() -> i32\r\n' + ' let local_value = 7\r\n' + ' println($"\"\" \t\r\n' + ' {{local_value}} {local_value + 1}\r\n' + ' "\"\")\r\n' + ' return local_value\r\nend fn\r\n') + name = 'formatted-block.ny' + result = analyze({name: block_ny}, name) + target = binding(result, name, '{local_value +', 'local_value') + assert target['rename_safe'] and target['references_complete'], target + assert len([use for use in result['semantic']['references'] + if use['symbol_id'] == target['id']]) == 2 + renamed = rename_local(result, name, target, 'renamed_value') + snapshot = next(item['text'] for item in renamed['semantic']['sources'] + if item['path'] == str(root / name)) + assert '{{local_value}}' in snapshot, snapshot + assert '{renamed_value + 1}' in snapshot, snapshot + interpolation = ('fn message(input: i32) -> i32\nprintln("\\n{input}")\nreturn input\nend fn\n') result = analyze({'interpolation.ny': interpolation}, 'interpolation.ny') assert len(result['semantic']['references']) == 1 From a4ec74bc250d8517a869e764fbcd9bac309902e0 Mon Sep 17 00:00:00 2001 From: Nathan Young Date: Fri, 13 Nov 2026 12:18:24 -0500 Subject: [PATCH 39/39] Record completed four-platform formatted text and markup qualification --- .../2026-11-13-formatted-markup-handoff.md | 43 +++++++++++++++---- 1 file changed, 35 insertions(+), 8 deletions(-) diff --git a/notebook/2026-11-13-formatted-markup-handoff.md b/notebook/2026-11-13-formatted-markup-handoff.md index 85043428..4c031418 100644 --- a/notebook/2026-11-13-formatted-markup-handoff.md +++ b/notebook/2026-11-13-formatted-markup-handoff.md @@ -70,11 +70,33 @@ its isolated server shut down and its evidence is retained. The hosted source workflow installs those dependencies and requires zero LLVM-enabled skips. The changed-source Gitleaks pass found no credential findings in 20 files. -The existing four-host interpreter workflow now includes both language -contracts and four positive fixtures through text NIL, binary NIL and both -bundle forms. Both Mac native workflows additionally require ten exact -language/route/toolchain CTests, with no skipped contracts accepted. Hosted -results for this follow-up will be recorded after the push. +Hosted source validation passed on the exact pushed commit +`e38145cb1d543320df7e7cba8deca1d9584cbffc` and on PR merge revision +`7e0af8613e5762bf51fa0080490149325ee13fa4`: all 595 LLVM-enabled CTests +passed without failures or skips. The LLVM-disabled suite completed with its +expected unavailable-native skips; extension tests and builds also passed. +See the [push run](https://github.com/exec-software/exec-nil-framework/actions/runs/43462076837) +and [PR run](https://github.com/exec-software/exec-nil-framework/actions/runs/43462083069). + +Linux x86-64, Windows x86-64, Intel macOS and Apple Silicon macOS each passed +all eight focused runtime/language contracts, including 200 markup and 150 +formatted-text controls. Each also executed four Ny/Matrix fixtures through +text NIL, binary NIL and both bundle forms: 16 positive route observations +per host. All four downloaded artifacts passed manifest/hash and unchanged +source checks. See the [four-host interpreter run](https://github.com/exec-software/exec-nil-framework/actions/runs/43462083117). + +Intel and Apple Silicon macOS each completed the full native applications/editor +campaign. All ten language contracts passed without skips, including the 392 +markup and 216 formatted-text checks. Project, assembler, context, SDK +packaging and incremental-build checks passed. Each actual VS Code host +passed 88 checks across nine modules without skips. All 313 manifest entries +and the unchanged source seal were verified in each architecture's artifact. +See the [native/editor run](https://github.com/exec-software/exec-nil-framework/actions/runs/43462082930). + +The first ARM interpreter and native/editor attempts could not acquire hosted +runners and never started any checks. Their failure evidence is retained. +Both passed their single same-source retry. These allocation failures are +distinct from source test failures; no requested handoff campaign is pending. ## Decision and follow-up @@ -83,6 +105,11 @@ review gate. A consumer needs a compiler and matching `nilstd` source built from this branch; the old installed SDK is not sufficient for the new syntax. No accepted SDK, installer or Lich release is changed. -After the language handoff, resume the WebView provider work: finish Windows -owned-profile teardown, prove all four actual browser hosts, then publish the -new repository with the same public-repository audit and review gates. +The [Matrix WebView repository](https://github.com/exec-software/exec-nil-framework-webview) +is now public, with source secret/path audits, four required platform checks +and an independent review requirement matching the web server repository. +Linux and both Mac architectures have passed real browser consumers through +text/binary NIL, AOT and JIT. Windows has passed compilation, shared/framework +contracts and synthetic routes; its real browser initialization correction is +being qualified. The repository's platform notes record exact source evidence. +This follows the language handoff without activating an accepted SDK.