From 40baf99a27cba41480b551ae067df608b3bf561b Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sun, 1 Nov 2026 10:06:55 +0100 Subject: [PATCH 1/2] =?UTF-8?q?conformance:=20cmd/oracle=20=E2=80=94=20ask?= =?UTF-8?q?=20the=20reference=20interpreter=20without=20losing=20the=20ans?= =?UTF-8?q?wer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every parity fix in this repository starts with *"what does MRI actually do here?"*, and the obvious ways to ask it throw part of the answer away: - `ruby probe.rb > out 2>&1` discards the **exit status**, which for half these probes IS the answer — a SyntaxError is what they are looking for; - a pipe into `grep` or `tail` discards whichever part did not match the pattern the asker already had in mind, which is the part worth reading; - running the reference and rbgo as two separate commands invites comparing a fresh measurement against a remembered one. So this tool PARSES NOTHING. Both streams are copied through verbatim with their byte counts, the status is printed as a number, and the only judgement it makes is byte equality between engines. ```sh go run ./scripts/conformance/rubyspec/cmd/oracle \ -script probe.rb /usr/bin/ruby ./rbgo-main ./rbgo-branch ``` Three engines is the shape the measurement rules want: reference, control (`main`), treatment (the branch). A difference between the last two that the reference does not share is the defect, or the fix. ## ⛔ The row that justifies comparing BYTES and not lengths Measured while writing #856, on the warning probe: ```text === the engines DISAGREED === /usr/bin/ruby vs ./rbgo-main: stderr 1008 vs 407 bytes /usr/bin/ruby vs ./rbgo-branch: stderr 1008 vs 1008 bytes ``` The second row is **the same length and different content**: the branch had gained the warning but still differed from MRI in the line number and the quote style. A comparison by length — which is what reading two `wc -c` outputs amounts to, and what I had been doing by eye — would have called that parity. `-selftest` plants exactly that case, `'$x'` versus `'$y'`, equal length and one byte apart, and refuses to pass unless the comparator sees it. It runs in the BINARY as well as under `go test`, like `cmd/phantom`'s controls, because a comparator stuck on "agreed" would make every measurement taken with this tool read as parity — the most reassuring possible way for it to be broken. ## Exit status, and the distinction that is tested `0` agreed (or one engine named), `1` disagreed, `2` refused — no script, an unreadable script, no engine, or an engine that would not start. That last one is deliberate and has its own test: *"could not ask"* and *"answered differently"* call for opposite responses — fix the invocation, or go read the divergence — so a missing interpreter must never arrive as a disagreement. Same split, and same exit codes, as `cmd/ratchet`. A probe that never returns is killed at `-timeout` (default 1m) and REPORTED as `KILLED` rather than hanging the measurement. That is not hypothetical: #853 was an endless range that never came back, and it was first recorded as "failed to load" because a timeout had killed it silently. ## Tests The fake engines are `/bin/sh` scripts, not Ruby, on purpose: the tool knows nothing about Ruby, and a test needing an interpreter installed would measure the machine instead of the tool. They cover the exit status surviving, the two streams not being swapped (a single buffer could not tell you), the three refusals, the kill path, and — as the control for the disagreement cases — two engines that agree. ⛔ One of those tests failed first time and the tool was right: I had written `stderr 0 vs 26 bytes` where `warning: something\n` is 19. The byte counts being printed at all is what made that a one-line correction instead of a theory. README: a section under `cmd/guards`, with the disagreement above as the worked example. Co-authored-by: Claude Opus 5 --- scripts/conformance/rubyspec/README.md | 47 +++ .../conformance/rubyspec/cmd/oracle/main.go | 276 ++++++++++++++++++ .../rubyspec/cmd/oracle/main_test.go | 182 ++++++++++++ 3 files changed, 505 insertions(+) create mode 100644 scripts/conformance/rubyspec/cmd/oracle/main.go create mode 100644 scripts/conformance/rubyspec/cmd/oracle/main_test.go diff --git a/scripts/conformance/rubyspec/README.md b/scripts/conformance/rubyspec/README.md index d96a0046..14ed22ef 100644 --- a/scripts/conformance/rubyspec/README.md +++ b/scripts/conformance/rubyspec/README.md @@ -165,6 +165,53 @@ control ever stops reproducing, nothing below it is worth reading. It is an analysis tool, not a gate: nothing fails when it is not run. What it exists to prevent is a `UPDATE_BASELINE=1` that absorbs a loss silently. +## Asking the reference interpreter — `cmd/oracle` + +Every parity fix starts with *"what does MRI actually do here?"*, and the obvious +way to ask it loses the answer. `ruby probe.rb > out 2>&1` discards the **exit +status**; a pipe into `grep` or `tail` discards whichever part of the answer did +not match the pattern the asker already had in mind; and running the reference +and rbgo as two separate commands invites comparing a fresh measurement against +a remembered one. + +```sh +# the reference, the control, and the treatment — in one invocation +go run ./scripts/conformance/rubyspec/cmd/oracle \ + -script probe.rb /usr/bin/ruby ./rbgo-main ./rbgo-branch +``` + +It **parses nothing**. Both streams are copied through verbatim with their byte +counts, the status is printed as a number (or `KILLED`, for a probe that never +returns — one of the defects found this way was an endless range that hung), and +the only judgement made is byte equality between engines: + +```text +=== the engines DISAGREED === + /usr/bin/ruby vs ./rbgo-main: stderr 1008 vs 407 bytes + /usr/bin/ruby vs ./rbgo-branch: stderr 1008 vs 1008 bytes +``` + +That second row is the point. **Same length, different content** — the branch +had gained the warning but still differed from MRI in the line number and the +quote style. A comparison by length would have called it parity, which is why +`-selftest` plants exactly that case (`'$x'` vs `'$y'`, equal length, one byte +apart) and refuses to pass unless the comparator sees it: + +```console +$ go run ./scripts/conformance/rubyspec/cmd/oracle -selftest +selftest: the comparison is right on all 5 cases +``` + +Exit status is `0` when the engines agreed, `1` when they did not, and `2` for a +refusal — no script, an unreadable script, no engine named, or an engine that +would not start. The last distinction is deliberate and tested: *"could not +ask"* and *"answered differently"* call for opposite responses, so a missing +interpreter must never arrive as a disagreement. + +Three engines is the shape the measurement rules want: reference, control +(`main`), treatment (the branch). A difference between the last two that the +reference does not share is the defect, or the fix. + ## Caveats The shim is not full MSpec: a few matchers (`complain`, `output`, diff --git a/scripts/conformance/rubyspec/cmd/oracle/main.go b/scripts/conformance/rubyspec/cmd/oracle/main.go new file mode 100644 index 00000000..961b0d8c --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/oracle/main.go @@ -0,0 +1,276 @@ +// Copyright (c) the go-embedded-ruby/ruby authors +// +// SPDX-License-Identifier: BSD-3-Clause + +// Command oracle runs one Ruby script through several interpreters and prints +// everything each of them produced — exit status, stdout and stderr, byte counts +// and all — then says whether they agreed. +// +// It exists because the question "what does MRI actually do here?" is the first +// step of every parity fix in this repository, and the obvious way to ask it +// loses the answer. `ruby probe.rb > out 2>&1` discards the exit status; a pipe +// into `grep` or `tail` discards whichever part of the answer did not match the +// pattern the asker already had in mind; and running the reference and rbgo in +// two separate commands invites comparing a fresh measurement against a +// remembered one. +// +// So this tool PARSES NOTHING. It copies both streams through verbatim and +// reports the status as a number. The only judgement it makes is byte equality +// between engines, which is also the only judgement it can get wrong, and +// -selftest proves that comparison can see a difference. +// +// oracle -script probe.rb /usr/bin/ruby ./rbgo +// oracle -script probe.rb /usr/bin/ruby ./rbgo-main ./rbgo-branch +// +// Three engines is the shape the measurement rules ask for: the reference, the +// control (main) and the treatment (the branch). A difference between the last +// two that the reference does not share is the defect or the fix. +// +// Exit status: +// +// 0 every engine ran, and they agreed (or only one was named) +// 1 the engines disagreed +// 2 refused: no script, a script that cannot be read, or an engine that +// would not start — never reported as a disagreement, because "could not +// ask" and "answered differently" need opposite responses +package main + +import ( + "bytes" + "errors" + "flag" + "fmt" + "io" + "os" + "os/exec" + "time" +) + +// maxStream bounds what is held per stream. A probe that loops forever printing +// is a real possibility — one of the defects found with this tool was an endless +// range that never came back — and an unbounded buffer turns that into the +// machine swapping rather than a message. +const maxStream = 8 << 20 + +// run is one engine's whole answer. Nothing here is interpreted; the fields are +// the bytes as they arrived. +type run struct { + engine string + status int + stdout []byte + stderr []byte + err error // the engine would not start at all — not a verdict about Ruby +} + +func main() { + os.Exit(main1(os.Args[1:], os.Stdout, os.Stderr)) +} + +func main1(args []string, out, errOut io.Writer) int { + fs := flag.NewFlagSet("oracle", flag.ContinueOnError) + fs.SetOutput(errOut) + script := fs.String("script", "", "the Ruby script to run through every engine (required)") + timeout := fs.Duration("timeout", time.Minute, "kill an engine that has not finished by then") + selftest := fs.Bool("selftest", false, "prove the comparison can see a difference, and exit") + if err := fs.Parse(args); err != nil { + return 2 + } + if *selftest { + return runSelftest(out) + } + if *script == "" { + fmt.Fprintln(errOut, "oracle: -script is required") + return 2 + } + if _, err := os.ReadFile(*script); err != nil { + // Read it here rather than let each engine fail its own way: a missing + // script is one refusal, not N disagreements. + fmt.Fprintf(errOut, "oracle: cannot read %s: %v\n", *script, err) + return 2 + } + engines := fs.Args() + if len(engines) == 0 { + fmt.Fprintln(errOut, "oracle: name at least one interpreter") + return 2 + } + + runs := make([]run, 0, len(engines)) + for _, e := range engines { + runs = append(runs, ask(e, *script, *timeout)) + } + for _, r := range runs { + report(out, r) + } + refused := false + for _, r := range runs { + if r.err != nil { + fmt.Fprintf(errOut, "oracle: %s would not run: %v\n", r.engine, r.err) + refused = true + } + } + if refused { + return 2 + } + if len(runs) == 1 { + return 0 + } + if agreed(runs) { + fmt.Fprintf(out, "=== %d engines AGREED, byte for byte ===\n", len(runs)) + return 0 + } + fmt.Fprintf(out, "=== the engines DISAGREED ===\n") + for _, r := range runs[1:] { + fmt.Fprintf(out, " %s vs %s: %s\n", runs[0].engine, r.engine, differences(runs[0], r)) + } + return 1 +} + +// ask runs one engine. A non-zero exit is an ANSWER, not an error: a SyntaxError +// is exactly what half these probes are looking for. +func ask(engine, script string, timeout time.Duration) run { + r := run{engine: engine} + var out, errBuf bytes.Buffer + cmd := exec.Command(engine, script) + cmd.Stdin = nil // a probe that reads stdin must block on nothing + cmd.Stdout = &limitWriter{w: &out} + cmd.Stderr = &limitWriter{w: &errBuf} + cmd.WaitDelay = time.Second + + done := make(chan error, 1) + if err := cmd.Start(); err != nil { + r.err = err + return r + } + go func() { done <- cmd.Wait() }() + select { + case err := <-done: + var ee *exec.ExitError + if errors.As(err, &ee) { + r.status = ee.ExitCode() + } else if err != nil { + r.err = err + } + case <-time.After(timeout): + _ = cmd.Process.Kill() + <-done + r.status = -1 // distinct from any real exit status, and reported as such + } + r.stdout, r.stderr = out.Bytes(), errBuf.Bytes() + return r +} + +func report(w io.Writer, r run) { + fmt.Fprintf(w, "=== %s ===\n", r.engine) + if r.err != nil { + fmt.Fprintf(w, "did not run: %v\n", r.err) + return + } + if r.status < 0 { + fmt.Fprintln(w, "exit status: KILLED, it had not finished") + } else { + fmt.Fprintf(w, "exit status: %d\n", r.status) + } + fmt.Fprintf(w, "--- stdout (%d bytes) ---\n%s", len(r.stdout), r.stdout) + if n := len(r.stdout); n > 0 && r.stdout[n-1] != '\n' { + fmt.Fprintln(w) + } + fmt.Fprintf(w, "--- stderr (%d bytes) ---\n%s", len(r.stderr), r.stderr) + if n := len(r.stderr); n > 0 && r.stderr[n-1] != '\n' { + fmt.Fprintln(w) + } +} + +func agreed(runs []run) bool { + for _, r := range runs[1:] { + if r.status != runs[0].status || + !bytes.Equal(r.stdout, runs[0].stdout) || + !bytes.Equal(r.stderr, runs[0].stderr) { + return false + } + } + return true +} + +// differences names WHICH of the three channels diverged, because the three call +// for different reading: a status difference is usually an exception, a stderr +// difference usually a warning or a message, and a stdout difference the +// program's own answer. +func differences(a, b run) string { + var which []string + if a.status != b.status { + which = append(which, fmt.Sprintf("exit status %d vs %d", a.status, b.status)) + } + if !bytes.Equal(a.stdout, b.stdout) { + which = append(which, fmt.Sprintf("stdout %d vs %d bytes", len(a.stdout), len(b.stdout))) + } + if !bytes.Equal(a.stderr, b.stderr) { + which = append(which, fmt.Sprintf("stderr %d vs %d bytes", len(a.stderr), len(b.stderr))) + } + if len(which) == 0 { + return "nothing — which is a defect in this tool, not an agreement" + } + s := which[0] + for _, w := range which[1:] { + s += ", " + w + } + return s +} + +// runSelftest proves the comparison can see a difference without needing an +// interpreter. A comparator that answered "agreed" for everything would make +// every parity measurement taken with this tool read as parity, which is the +// most reassuring possible way for it to be broken — so it is checked, in the +// binary, by anyone who runs it. +func runSelftest(w io.Writer) int { + same := []run{{engine: "a", stdout: []byte("x"), stderr: []byte("y")}, {engine: "b", stdout: []byte("x"), stderr: []byte("y")}} + cases := []struct { + name string + runs []run + agree bool + }{ + {"identical", same, true}, + {"stdout differs", []run{same[0], {engine: "b", stdout: []byte("X"), stderr: []byte("y")}}, false}, + {"stderr differs", []run{same[0], {engine: "b", stdout: []byte("x"), stderr: []byte("Y")}}, false}, + {"status differs", []run{same[0], {engine: "b", status: 1, stdout: []byte("x"), stderr: []byte("y")}}, false}, + // The one a length-only comparison would miss, and warnings differing by + // one character is the commonest shape in this repository: `$x` vs `$y`. + {"same length, different bytes", []run{ + {engine: "a", stderr: []byte("global variable '$x' not initialized")}, + {engine: "b", stderr: []byte("global variable '$y' not initialized")}, + }, false}, + } + bad := 0 + for _, c := range cases { + if got := agreed(c.runs); got != c.agree { + fmt.Fprintf(w, "selftest FAILED: %s: agreed = %v, want %v\n", c.name, got, c.agree) + bad++ + } + } + if bad > 0 { + return 2 + } + fmt.Fprintf(w, "selftest: the comparison is right on all %d cases\n", len(cases)) + return 0 +} + +// limitWriter stops after maxStream bytes and keeps accepting writes, so a +// runaway probe is truncated rather than allowed to exhaust memory — and the +// byte count printed afterwards is the truncated one, which is why the cap is +// generous enough that no real probe reaches it. +type limitWriter struct { + w io.Writer + n int +} + +func (l *limitWriter) Write(p []byte) (int, error) { + if l.n >= maxStream { + return len(p), nil + } + room := maxStream - l.n + if len(p) > room { + p = p[:room] + } + n, err := l.w.Write(p) + l.n += n + return len(p), err +} diff --git a/scripts/conformance/rubyspec/cmd/oracle/main_test.go b/scripts/conformance/rubyspec/cmd/oracle/main_test.go new file mode 100644 index 00000000..5ffe1867 --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/oracle/main_test.go @@ -0,0 +1,182 @@ +// Copyright (c) the go-embedded-ruby/ruby authors +// +// SPDX-License-Identifier: BSD-3-Clause + +package main + +import ( + "bytes" + "os" + "path/filepath" + "runtime" + "strings" + "testing" +) + +// The tool's whole promise is that nothing is lost between the interpreter and +// the reader, so the tests are about what a redirection or a pipe would have +// dropped: the exit status, the stream a pattern did not match, and the +// difference between "could not ask" and "answered differently". +// +// The engines here are /bin/sh scripts rather than Ruby, deliberately: the tool +// knows nothing about Ruby, and a test that needed an interpreter installed +// would be measuring the machine instead of the tool. +func TestOracleKeepsWhatARedirectionWouldLose(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("the fake engines are POSIX shell scripts") + } + dir := t.TempDir() + // An "engine" that writes to BOTH streams and exits non-zero — the shape of + // a Ruby run that raised. Each ignores the script it is handed. + engine := fakeEngine(t, dir, "both", "echo out-line\necho err-line >&2\nexit 3\n") + script := filepath.Join(dir, "probe.rb") + if err := os.WriteFile(script, []byte("# ignored by the fake engine\n"), 0o644); err != nil { + t.Fatal(err) + } + + var out, errOut bytes.Buffer + if code := main1([]string{"-script", script, engine}, &out, &errOut); code != 0 { + t.Fatalf("exit = %d, want 0; stderr %q", code, errOut.String()) + } + got := out.String() + for _, want := range []string{ + "exit status: 3", // ⛔ what `cmd > file 2>&1` throws away + "out-line", + "err-line", + "--- stdout (9 bytes) ---", + "--- stderr (9 bytes) ---", + } { + if !strings.Contains(got, want) { + t.Errorf("report is missing %q:\n%s", want, got) + } + } + // The streams must not be swapped, which a single buffer could not tell you. + iOut := strings.Index(got, "--- stdout") + iErr := strings.Index(got, "--- stderr") + if !(iOut < strings.Index(got, "out-line") && strings.Index(got, "out-line") < iErr) { + t.Errorf("out-line is not under the stdout heading:\n%s", got) + } +} + +// Two engines that answer differently must exit 1 and NAME the channel that +// diverged; two that answer alike must exit 0. The second half is the control: +// a comparator stuck on "disagreed" would pass the first half alone. +func TestOracleComparesEngines(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("the fake engines are POSIX shell scripts") + } + dir := t.TempDir() + script := filepath.Join(dir, "probe.rb") + if err := os.WriteFile(script, []byte("#\n"), 0o644); err != nil { + t.Fatal(err) + } + quiet := fakeEngine(t, dir, "quiet", "echo same\n") + alsoQuiet := fakeEngine(t, dir, "also", "echo same\n") + noisy := fakeEngine(t, dir, "noisy", "echo same\necho warning: something >&2\n") + failing := fakeEngine(t, dir, "failing", "echo same\nexit 1\n") + + for _, tc := range []struct { + name string + engines []string + code int + wantIn string + }{ + {"agreeing", []string{quiet, alsoQuiet}, 0, "AGREED, byte for byte"}, + {"stderr diverges", []string{quiet, noisy}, 1, "stderr 0 vs 19 bytes"}, + {"status diverges", []string{quiet, failing}, 1, "exit status 0 vs 1"}, + {"one engine only", []string{quiet}, 0, "exit status: 0"}, + } { + t.Run(tc.name, func(t *testing.T) { + var out, errOut bytes.Buffer + args := append([]string{"-script", script}, tc.engines...) + if code := main1(args, &out, &errOut); code != tc.code { + t.Errorf("exit = %d, want %d; out %q err %q", code, tc.code, out.String(), errOut.String()) + } + if !strings.Contains(out.String(), tc.wantIn) { + t.Errorf("report is missing %q:\n%s", tc.wantIn, out.String()) + } + }) + } +} + +// ⛔ "Could not ask" must never read as "answered differently": the two call for +// opposite responses — fix the invocation, or go read the divergence — and a +// tool that returned 1 for a missing interpreter would send the reader to the +// wrong one. Every refusal is exit 2, as cmd/ratchet's is. +func TestOracleRefusalsAreNotDisagreements(t *testing.T) { + dir := t.TempDir() + script := filepath.Join(dir, "probe.rb") + if err := os.WriteFile(script, []byte("#\n"), 0o644); err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + name string + args []string + }{ + {"no script", []string{"/bin/echo"}}, + {"unreadable script", []string{"-script", filepath.Join(dir, "absent.rb"), "/bin/echo"}}, + {"no engine", []string{"-script", script}}, + {"an engine that will not start", []string{"-script", script, filepath.Join(dir, "not-a-program")}}, + } { + t.Run(tc.name, func(t *testing.T) { + var out, errOut bytes.Buffer + if code := main1(tc.args, &out, &errOut); code != 2 { + t.Errorf("exit = %d, want 2; out %q err %q", code, out.String(), errOut.String()) + } + }) + } +} + +// An engine that never returns is killed and REPORTED as killed, rather than +// hanging the measurement. One of the defects found with this tool was an +// endless range that never came back, so this case is not hypothetical. +func TestOracleKillsAnEngineThatNeverReturns(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("the fake engines are POSIX shell scripts") + } + dir := t.TempDir() + script := filepath.Join(dir, "probe.rb") + if err := os.WriteFile(script, []byte("#\n"), 0o644); err != nil { + t.Fatal(err) + } + hang := fakeEngine(t, dir, "hang", "sleep 60\n") + var out, errOut bytes.Buffer + code := main1([]string{"-script", script, "-timeout", "300ms", hang}, &out, &errOut) + if code != 0 { + t.Errorf("exit = %d, want 0 (one engine, killed is an answer); err %q", code, errOut.String()) + } + if !strings.Contains(out.String(), "KILLED") { + t.Errorf("report does not say it was killed:\n%s", out.String()) + } +} + +// The in-binary control must pass, and it must be able to fail: the last case +// inside it is two messages of the SAME LENGTH differing by one byte, which is +// the commonest shape here (`$x` vs `$y`) and the one a length comparison would +// call an agreement. +func TestSelftestPasses(t *testing.T) { + var out bytes.Buffer + if code := runSelftest(&out); code != 0 { + t.Fatalf("selftest exit = %d, want 0: %s", code, out.String()) + } + if !strings.Contains(out.String(), "right on all 5 cases") { + t.Errorf("selftest did not report its case count: %q", out.String()) + } + if agreed([]run{ + {stderr: []byte("global variable '$x' not initialized")}, + {stderr: []byte("global variable '$y' not initialized")}, + }) { + t.Error("two same-length, different-byte messages compared equal") + } +} + +// fakeEngine writes a shell script that behaves like an interpreter: it is +// handed a script path, ignores it, and produces fixed output. +func fakeEngine(t *testing.T, dir, name, body string) string { + t.Helper() + p := filepath.Join(dir, name) + if err := os.WriteFile(p, []byte("#!/bin/sh\n"+body), 0o755); err != nil { + t.Fatal(err) + } + return p +} From 4e8dd6834495e0aa2c5605c28cd0dd122ab16757 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sun, 1 Nov 2026 11:07:47 +0100 Subject: [PATCH 2/2] conformance: keep the oracle probes, with the answers written down MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four probes under `cmd/oracle/testdata/probes`, one question each, and a README holding what the reference answered and which upstream function owns the rule. The shapes take a minute to write and the reference run takes a second. What costs time is working out **which** function in MRI decides — `parse.y` does not decide constant-ness, `rb_sym_constant_char_p` does; `rb_gvar_get` does not warn, the undef getter it installs does — and which rows separate the plausible-but-wrong implementation from the right one. A probe thrown away is that work thrown away. ⛔ Three of these were written, lost to a reboot that emptied the scratchpad, and written again. They belong in the repository, which is where the rule about durable directories has been pointing all along. | probe | the row that earns its keep | |---|---| | `globals.rb` | silence after `$x = nil` — the subject is "never assigned", not "reads as nil" | | `constant_names.rb` | the **titlecase** digraphs `Dž`/`Lj` are constants and the **caseless** letters `日`/`א`/`ب` are not, so "not lowercase" is wrong in both directions | | `dynamic_constant.rb` | `Class.new { X = 1 }` inside a `def` is refused while `define_method(:m) { X = 1 }` at class level is not — the rule is syntactic, not about what will run as a method | | `basicobject_conversion.rb` | a BasicObject has **no** `#respond_to?`, which is not the same as having an overridden one | Replayed against `/usr/bin/ruby` before committing: 6 constant names of 12 characters, 16 `dynamic constant assignment` refusals (the seventeenth refusal is `class definition in method body`, a different message from a different check), 4 clean conversions, and the globals probe's six blocks. The README also records the two differences against the installed 2.6.10 that are **expected** and are not rbgo defects — the quote style (`` `$x' `` before 3.4, `'$x'` after, which is why ruby/spec matches on `` [`'] ``) and 2.6's one-line lag in a warning's position — and says plainly what that interpreter cannot answer at all: it predates numbered parameters, core `Set` and everything else 3.x–4.0 added, so a probe about those gets silence, which reads exactly like a negative. Co-authored-by: Claude Opus 5 --- .../cmd/oracle/testdata/probes/README.md | 53 +++++++++++++++++++ .../testdata/probes/basicobject_conversion.rb | 38 +++++++++++++ .../oracle/testdata/probes/constant_names.rb | 43 +++++++++++++++ .../testdata/probes/dynamic_constant.rb | 51 ++++++++++++++++++ .../cmd/oracle/testdata/probes/globals.rb | 49 +++++++++++++++++ 5 files changed, 234 insertions(+) create mode 100644 scripts/conformance/rubyspec/cmd/oracle/testdata/probes/README.md create mode 100644 scripts/conformance/rubyspec/cmd/oracle/testdata/probes/basicobject_conversion.rb create mode 100644 scripts/conformance/rubyspec/cmd/oracle/testdata/probes/constant_names.rb create mode 100644 scripts/conformance/rubyspec/cmd/oracle/testdata/probes/dynamic_constant.rb create mode 100644 scripts/conformance/rubyspec/cmd/oracle/testdata/probes/globals.rb diff --git a/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/README.md b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/README.md new file mode 100644 index 00000000..f1495366 --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/README.md @@ -0,0 +1,53 @@ +# Oracle probes + +Questions put to the reference interpreter, with its answers written down. + +```sh +go run ./scripts/conformance/rubyspec/cmd/oracle \ + -script scripts/conformance/rubyspec/cmd/oracle/testdata/probes/globals.rb \ + ./rbgo-main ./rbgo-branch /usr/bin/ruby +``` + +Each file is one question, each row inside it one shape, and the header comment +records what MRI answered and which upstream function decides. They are kept +because **the answer is the expensive part** — the shapes take a minute to write +and the reference run takes a second, but working out *which* function in MRI +owns the rule, and which rows distinguish the plausible-but-wrong +implementations from the right one, is where the time goes. A probe that is +thrown away is that work thrown away: three of these were written, lost to a +reboot, and written again. + +| probe | question | the row that matters | +|---|---|---| +| `globals.rb` | when does reading a global warn? | silence after `$x = nil` — the subject is "never assigned", not "reads as nil" | +| `constant_names.rb` | which first characters start a constant? | the **titlecase** digraphs are constants, the **caseless** letters are not, so "not lowercase" is wrong in both directions | +| `dynamic_constant.rb` | where is a constant assignment refused? | `Class.new { X = 1 }` in a `def` is refused but `define_method(:m) { X = 1 }` at class level is not — the rule is syntactic, not about execution | +| `basicobject_conversion.rb` | how is a conversion reached on an object outside Kernel? | a BasicObject has no `#respond_to?` at all, which is not the same as having an overridden one | + +## Reading a disagreement + +`oracle` compares **bytes**, not lengths, and the reason is in these probes' +history: a branch that had just gained the uninitialized-global warning reported +`stderr 1008 vs 1008 bytes` against MRI — the same length, differing only in the +line number and the quote style. A comparison by size would have called that +parity. + +Two differences against the 2.6.10 interpreter on this machine are **expected** +and are not rbgo defects: + +- **the quote style.** 2.6 writes ``global variable `$x' not initialized``; + 3.4 and later write `'$x'`. ruby/spec matches on `` [`'] `` for exactly this + reason. +- **the line number of a warning.** 2.6 reports the line BEFORE the read when + the read is a simple statement, and the right line when it is an argument to a + call. rbgo reports the true line in all three shapes. Replicating a stale line + is not parity worth having, and no spec asserts the position prefix. + +## What this reference interpreter cannot answer + +`/usr/bin/ruby` on macOS is **2.6.10**. It predates numbered parameters (2.7), +`Set` as a core class, `Range#overlap?` and everything else 3.x–4.0 added, so a +probe about those gets silence rather than an answer — which reads exactly like +a negative. Say which case you are in: a probe whose rule is older than 2.6 (the +constant and dynamic-constant rules here) has a valid oracle; one whose rule is +newer does not, and the spec file is then the authority. diff --git a/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/basicobject_conversion.rb b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/basicobject_conversion.rb new file mode 100644 index 00000000..1072c8bb --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/basicobject_conversion.rb @@ -0,0 +1,38 @@ +# A BasicObject does not include Kernel, so it has NO #respond_to? at all — +# and MRI's conversion path copes: vm_respond_to opens with `if (!cme) return +# -1`, where -1 means UNDECIDED rather than false, so rb_check_funcall falls +# through to a plain method lookup. +# +# ⛔ `findMethod(v, "respond_to?") != Kernel's entry` is TRUE for nil, which is +# how rbgo came to send #respond_to? to an object that has none. Two of these +# four shapes were broken on rbgo's main through the masgn (#to_ary) path with +# no spec covering them; the splat (#to_a) ones are asserted by +# language/array_spec.rb. +# +# MRI 2.6.10 answers: all four "no error" (188 bytes of stderr). +def shape(label) + yield + STDERR.puts "#{label.ljust(34)} -> no error" +rescue NoMethodError => e + STDERR.puts "#{label.ljust(34)} -> NoMethodError: #{e.message}" +rescue => e + STDERR.puts "#{label.ljust(34)} -> #{e.class}: #{e.message}" +end + +shape("splat a bare BasicObject") { a = *BasicObject.new } + +klass = Class.new(BasicObject) do + def to_a + [1, 2] + end +end +shape("splat a BasicObject with to_a") { a = *klass.new } + +shape("masgn a bare BasicObject") { a, b = BasicObject.new } + +aryklass = Class.new(BasicObject) do + def to_ary + [1, 2] + end +end +shape("masgn a BasicObject with to_ary") { a, b = aryklass.new } diff --git a/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/constant_names.rb b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/constant_names.rb new file mode 100644 index 00000000..4114d41c --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/constant_names.rb @@ -0,0 +1,43 @@ +# Which first characters start a CONSTANT? rb_sym_constant_char_p +# (symbol.c:508-542 at v4.0.9) decides, and it has FIVE branches: +# +# ASCII -> ISUPPER +# non-ASCII, rb_enc_isupper -> constant +# non-ASCII, rb_enc_islower -> local +# titlecase letter, Unicode -> constant +# anything else (caseless) -> local +# +# parse.y does NOT decide this: it assumes tCONSTANT provisionally and +# downgrades with `if (result == tCONSTANT && is_local_id(ident)) result = +# tIDENTIFIER` (parse.y:10453), so the ID system is the authority. +# +# Object.const_defined? raises NameError for a name that is not a constant +# name, so it exposes rb_is_const_name (symbol.c:1287) directly. +# +# MRI 2.6.10 answers: A, Ἅ, Д, İ, Dž, Lj are CONSTANT names; a, _, μ, 日, א, ب +# are not. ⛔ The titlecase rows and the caseless rows are the two that make +# "not lowercase" the wrong shortcut, in opposite directions. +[ + ["ASCII upper A", "A"], + ["ASCII lower a", "a"], + ["ASCII under _", "_"], + ["Lu GREEK Ἅ", "Ἅ"], + ["Lu CYRILLIC Д", "Д"], + ["Ll GREEK mu μ", "μ"], + ["Lt DZ digr Dž", "Dž"], + ["Lt DZE digr Lj", "Lj"], + ["Lo CJK 日", "日"], + ["Lo HEBREW א", "א"], + ["Lo ARABIC ب", "ب"], + ["Lu TURKISH I İ", "İ"], +].each do |label, ch| + name = "#{ch}Xyz" + begin + Object.const_defined?(name.to_sym) + STDERR.puts "#{label} -> CONSTANT name" + rescue NameError + STDERR.puts "#{label} -> not a constant name" + rescue => e + STDERR.puts "#{label} -> #{e.class}: #{e.message}" + end +end diff --git a/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/dynamic_constant.rb b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/dynamic_constant.rb new file mode 100644 index 00000000..a31547d7 --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/dynamic_constant.rb @@ -0,0 +1,51 @@ +# Where is a constant assignment refused? MRI's rule is purely SYNTACTIC: an +# enclosing `def` keyword, reached through any number of blocks, stopped by a +# class or module body or the top level. +# +# MRI 2.6.10 refuses the first ten with `dynamic constant assignment`, except +# `class I5` inside a def which is `class definition in method body` — a +# different message from a different check. It allows all six below `-- allowed`. +# +# ⛔ Two rows are the ones that show the rule is not "will this run as a +# method?": `Class.new { AA = 1 }` inside a def IS refused, while +# `define_method(:m) { AA = 1 }` at class level is NOT. Reasoning from execution +# gets both backwards. A third: a SCOPED assignment `C8::AA8 = 1` is refused +# inside a def too — being explicit about the namespace does not buy it. +[ + ["def body", "def d1; AA1 = 1; end"], + ["block inside a def", "def d2; [1].each { AA2 = 1 }; end"], + ["nested block inside a def", "def d3; [1].each { [2].each { AA3 = 1 } }; end"], + ["lambda inside a def", "def d4; -> { AA4 = 1 }; end"], + ["class body inside a def", "def d5; class I5; AA5 = 1; end; end"], + ["Class.new block inside a def", "def d6; Class.new { AA6 = 1 }; end"], + ["define_method at class level", "class C7; define_method(:m) { AA7 = 1 }; end"], + ["scoped assign inside a def", "class C8; def m; C8::AA8 = 1; end; end"], + ["op-assign inside a def", "def d9; AA9 ||= 1; end"], + ["masgn inside a def", "def d10; AA10, x = 1, 2; end"], + ["singleton def body", "class S1; def self.m; AC1 = 1; end; end"], + ["def inside class << self", "class S2; class << self; def m; AC2 = 1; end; end; end"], + ["const in class << self body", "class S3; class << self; AC3 = 1; end; end"], + ["def on an object", "o = Object.new; def o.m; AC4 = 1; end"], + ["module_eval block", "class S5; end; S5.module_eval { AC5 = 1 }"], + ["instance_eval block in def", "def d6b; Object.new.instance_eval { AC6 = 1 }; end"], + ["const in a def's rescue", "def d7; raise; rescue; AC7 = 1; end"], + ["const in a def's ensure", "def d8; ensure; AC8 = 1; end"], + ["const in a def default arg", "def d9b(x = (AC9 = 1)); end"], + ["nested def", "def d10b; def inner; AC10 = 1; end; end"], + ["-- allowed --", "nil"], + ["top level", "AB1 = 1"], + ["class body", "class C12; AB2 = 1; end"], + ["module body", "module M13; AB3 = 1; end"], + ["block at top level", "[1].each { AB4 = 1 }"], + ["block in a class body", "class C14; [1].each { AB5 = 1 }; end"], + ["singleton def body (no const)", "class C15; class << self; def m; end; end; end"], +].each do |label, src| + begin + eval(src) + STDERR.puts "#{label.ljust(30)} -> NO ERROR" + rescue SyntaxError => e + STDERR.puts "#{label.ljust(30)} -> SyntaxError: #{e.message.lines.first.to_s.strip}" + rescue => e + STDERR.puts "#{label.ljust(30)} -> #{e.class}: #{e.message.lines.first.to_s.strip}" + end +end diff --git a/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/globals.rb b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/globals.rb new file mode 100644 index 00000000..e145789b --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/oracle/testdata/probes/globals.rb @@ -0,0 +1,49 @@ +# Uninitialized-global warning: rb_gvar_undef_getter (variable.c:649 at v4.0.9). +# +# $VERBOSE is set here rather than through -w so the comparison does not depend +# on a flag either engine may spell differently -- and because +# `complain(verbose: true)` is how language/variables_spec.rb arranges it. +# Setting it at runtime also keeps MRI's compile-time void-context warnings out +# of the output. +# +# MRI 2.6.10 answers (1008 bytes of stderr): silence for every predefined name, +# ONE warning for the never-assigned name, silence after an explicit `= nil`, +# silence in a rescue body, TWO warnings for a name read twice, and silence +# while $VERBOSE is false. +$VERBOSE = true + +STDERR.puts "-- interpreter-defined names (expect SILENCE) --" +$; ; $\ ; $, ; $. ; $_ ; $* ; $" ; $: ; $? +$! ; $@ ; $~ ; $& ; $` ; $' ; $+ ; $1 +$0 ; $$ ; $stdout ; $stderr ; $stdin ; $> ; $< +$/ ; $-0 ; $DEBUG ; $VERBOSE ; $-W ; $-a ; $-l ; $-p +$LOAD_PATH ; $LOADED_FEATURES ; $PROGRAM_NAME ; $FILENAME +STDERR.puts "-- end --" + +STDERR.puts "-- never assigned (expect ONE warning naming it) --" +$specs_uninitialized_global_variable +STDERR.puts "-- end --" + +STDERR.puts "-- assigned nil, then read (expect SILENCE) --" +$assigned_to_nil = nil +$assigned_to_nil +STDERR.puts "-- end --" + +STDERR.puts "-- a rescue body (expect SILENCE: the errinfo cell is not a global) --" +begin + raise "x" +rescue => e + e.message +end +STDERR.puts "-- end --" + +STDERR.puts "-- read twice (MRI warns EACH time, the getter is still undef) --" +$read_twice +$read_twice +STDERR.puts "-- end --" + +STDERR.puts "-- silent when $VERBOSE is false (rb_warning, not rb_warn) --" +$VERBOSE = false +$never_assigned_while_quiet +$VERBOSE = true +STDERR.puts "-- end --"