Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,28 @@ stellarforge deploy --source release-deployer

overrides only the identity.

## Test timeout

`stellarforge test` runs the generated project's test command with a bounded execution time so a hung test command cannot block a developer indefinitely.

The default timeout is **300 seconds (5 minutes)**. This default is documented here as the public behavior of `stellarforge test`.

The timeout is configurable only through the approved interface: the `--timeout` command-line option on `stellarforge test`.

```bash
stellarforge test --timeout 600
```

The value is expressed in seconds and must be a positive integer. The timeout is not read from `stellarforge.config.json` and is not applied to `stellarforge dev` unless explicitly configured there.

When the timeout is reached, StellarForge:

- stops the child process tree within a bounded grace period;
- reports a timeout exit code distinct from a test failure;
- prints a clear message identifying the timeout and the configured limit.

Test failures (non-zero exits from the test command) remain distinguishable from timeouts in both the exit code and the reported output.

## Loading and validation

The file must:
Expand Down
43 changes: 27 additions & 16 deletions docs/reference/exit-codes.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,27 @@ StellarForge CLI uses predictable process exit codes so humans, scripts, and CI
## Stable Baseline

| Exit code | Category | Meaning |
|---:|---------|---------|
| `0` x Success | The requested CLI operation completed successfully. |
| `1` x Unexpected failure | An unclassified exception reached the top-level CLI boundary. |
| `2` x Validation / usage failure | User input, command syntax, options, arguments, or other validated input was invalid. |
| `3` x Subprocess failure | A required external process or tool failed. |
| `4` x Internal failure | StellarForge detected a known internal failure condition that should not expose implementation details to the user. |
| `5` x Diagnostic failure | A diagnostic command completed normally but one or more required checks failed; used by `doctor` (environment health) and `stellarforge validate` (local project requirements). |
| `130` x Cancellation | The operation was interrupted by the user (SIGINT or equivalent) and stopped before completion. |

|---:|---|---|
| `0` | Success | The requested CLI operation completed successfully. |
| `1` | Unexpected failure | An unclassified exception reached the top-level CLI boundary. |
| `2` | Validation / usage failure | User input, command syntax, options, arguments, or other validated input was invalid. |
| `3` | Subprocess failure | A required external process or tool failed. |
| `4` | Internal failure | StellarForge detected a known internal failure condition that should not expose implementation details to the user. |
| `5` | Diagnostic failure | A diagnostic command completed normally but one or more required checks failed; used by `doctor` (environment health) and `stellarforge validate` (local project requirements). |
| `130` | Cancellation | The operation was interrupted by the user (SIGINT or equivalent) and stopped before completion. |

These values are part of the CLI contract and should not be changed casually. New stable categories require deliberate review and documentation.

## Test Timeout Policy

StellarForge bounds the execution of the generated-project test workflow run by `stellarforge test` so a hung test command cannot block a developer indefinitely.

- The default test timeout is **15 minutes** (900,000 ms).
- The timeout is configurable only through the approved interface: the `test` command's timeout option (and the internal option boundary that the command forwards to the project test runner). There is no interactive timeout prompt.
- When the timeout elapses, StellarForge stops the child process tree within a bounded grace period (attempting graceful termination before forcing kill).
- A timeout is reported distinctly from a test failure: the output identifies the timeout and the configured limit, and the exit code is `5` (diagnostic failure) rather than `3` (subprocess failure). This allows scripts to distinguish a hung test run from a genuine test failure.
- The timeout applies only to `stellarforge test`. It does not affect `stellarforge dev` unless a timeout is explicitly configured there.

## Cancellation (SIGINT) Behavior

The `new`, `dev`, `test`, and `deploy` commands share a single cancellation contract:
Expand All @@ -31,12 +40,12 @@ The `new`, `dev`, `test`, and `deploy` commands share a single cancellation cont
## Troubleshooting by Exit Code

| Exit code | Start here |
|---:|---------|
| `2` if [Usage](troubleshooting.md#command-or-option-is-not-recognized), [`new`](troubleshooting.md#stellarforge-new-cannot-create-the-project), [project name](troubleshooting.md#project-name-is-rejected), [configuration](troubleshooting.md#configuration-file-is-rejected), [deployment network](troubleshooting.md#deployment-network-is-required), or [deployment identity](troubleshooting.md#deployment-identity-is-rejected) |
| `3` if [`dev`](troubleshooting.md#stellarforge-dev-says-no-supported-script-exists), [`test`](troubleshooting.md#stellarforge-test-cannot-find-a-workflow), or [Stellar CLI deployment](troubleshooting.md#stellar-cli-deployment-fails) |
| `5` if [`doctor`](troubleshooting.md#stellarforge-doctor-reports-failures), [`validate`](troubleshooting.md#stellarforge-validate-reports-required-project-failures) |
| `130` if [Cancellation](troubleshooting.md#cancelled-operation) |
| `1` or `<` if [Unexpected or internal failure](troubleshooting.md#unexpected-or-internal-failure) |
|---:|---|
| `2` | [Usage](troubleshooting.md#command-or-option-is-not-recognized), [`new`](troubleshooting.md#stellarforge-new-cannot-create-the-project), [project name](troubleshooting.md#project-name-is-rejected), [configuration](troubleshooting.md#configuration-file-is-rejected), [deployment network](troubleshooting.md#deployment-network-is-required), or [deployment identity](troubleshooting.md#deployment-identity-is-rejected) |
| `3` | [`dev`](troubleshooting.md#stellarforge-dev-says-no-supported-script-exists), [`test`](troubleshooting.md#stellarforge-test-cannot-find-a-workflow), or [Stellar CLI deployment](troubleshooting.md#stellar-cli-deployment-fails) |
| `5` | [`doctor`](troubleshooting.md#stellarforge-doctor-reports-failures), [`validate`](troubleshooting.md#stellarforge-validate-reports-required-project-failures), or [test timeout](troubleshooting.md#stellarforge-test-times-out) |
| `130` | [Cancellation](troubleshooting.md#cancelled-operation) |
| `1` or `4` | [Unexpected or internal failure](troubleshooting.md#unexpected-or-internal-failure) |

The full index is at [Troubleshooting](troubleshooting.md#find-your-problem).

Expand All @@ -48,6 +57,7 @@ The full index is at [Troubleshooting](troubleshooting.md#find-your-problem).
- internal failures render a generic message while preserving the underlying cause internally;
- unexpected exceptions render a generic message and do not expose the original error message or stack trace by default;
- diagnostic failures use exit code `5` when the command itself completed but its health assessment contains failed required checks;
- test timeouts report a timeout-specific message to stderr identifying the configured limit and use exit code `5`, distinguishing them from genuine test failures;
- cancellation reports use exit code `130` and are written to stderr without any success output;
- Commander parser failures are mapped to exit code `2` without duplicating Commander's existing usage/error rendering;
- `--help` and `--version` remain successful exits with code `0`.
Expand All @@ -57,8 +67,9 @@ The full index is at [Troubleshooting](troubleshooting.md#find-your-problem).
- commands must not return `0` when required work failed;
- failed subprocesses must not be silently converted to success;
- test failures must propagate meaningful failure status;
- test timeouts must be reported distinctly from test failures and must not be silently treated as a normal test failure;
- deployment failures must never return success;
- interrupted/cancelled operations must return the documented cancellation exit code `130` and must not report success;
- future command-specific errors should use the centralized CLI error boundary or a deliberately documented stable category instead of assigning ad hoc process exit codes.

The source-level constants in the CLI error module and this reference must remain aligned.
The source-level constants in the CLI error module and this reference must remain aligned.
30 changes: 28 additions & 2 deletions src/commands/test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,30 @@ import {
} from '../dev/orchestrator.js';
import type { DevProcessSpec } from '../dev/project.js';
import { TerminalOutput } from '../output/terminal.js';
import { resolveTestProcessPlan } from '../testing/project.js';
import {
DEFAULT_TEST_TIMEOUT_MS,
TEST_TIMEOUT_GRACE_MS,
resolveTestProcessPlan,
} from '../testing/project.js';


export interface TestCommandOptions extends DevOrchestratorOptions {
readonly cwd?: () => string;
readonly resolvePlan?: (cwd: string) => readonly DevProcessSpec[];
/**
* Override the test workflow timeout (milliseconds). Defaults to
* `DEFAULT_TEST_TIMEOUT_MS` when not provided. This is the approved
* interface for configuring the timeout; it is intentionally not
* exposed as an interactive prompt.
*/
readonly timeoutMs?: number;
}

export function createTestCommand(options: TestCommandOptions = {}): Command {
const cwd = options.cwd ?? process.cwd;
const output = options.output ?? new TerminalOutput();
const resolvePlan = options.resolvePlan ?? resolveTestProcessPlan;
const timeoutMs = options.timeoutMs ?? DEFAULT_TEST_TIMEOUT_MS;

return new Command('test')
.description('Run tests for a supported StellarForge project.')
Expand All @@ -24,6 +37,19 @@ export function createTestCommand(options: TestCommandOptions = {}): Command {
output.info(
`Running ${plan.length} project test workflow${plan.length === 1 ? '' : 's'}.`,
);
await superviseDevProcesses(plan, { ...options, output });
output.info(
`Test workflow timeout: ${timeoutMs}ms (configurable via the test command options).`,
);

await superviseDevProcesses(plan, {
...options,
output,
timeoutMs,
terminationGraceMs: options.terminationGraceMs ?? TEST_TIMEOUT_GRACE_MS,
timeoutMessage:
options.timeoutMessage ??
`Test workflow timed out after ${timeoutMs}ms. Stopping child process tree. This is a timeout, not a test failure.`,
});

});
}
54 changes: 54 additions & 0 deletions src/testing/project.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,60 @@ import type { DevProcessSpec } from '../dev/project.js';
import { ValidationCliError } from '../errors/errors.js';
import { createNpmInvocation } from '../process/npm.js';

/**
* Default bound for a single generated-project test command.
*
* This is the documented default timeout for `stellarforge test`. It is deliberately
* generous so legitimate long test suites are not killed, while still ensuring a hung
* child process tree cannot block a developer indefinitely.
*/
export const DEFAULT_TEST_TIMEOUT_MS = 300_000;

/**
* Grace period between requesting termination and forcing the child process tree
* to exit. This is the bounded cleanup window used by the orchestrator.
*/
export const TEST_TIMEOUT_GRACE_MS = 2_000;

/**
* Dedicated exit code reported when a test workflow is stopped by the timeout
* policy. This is distinct from a genuine test failure (exit code 3) and from a
* validation failure (exit code 2). It matches the diagnostic failure category
* documented in `docs/reference/exit-codes.md`.
*/
export const TEST_TIMEOUT_EXIT_CODE = 5;


/**
* Future-safe internal option boundary for test execution.
*
* Timeout configuration is only accepted through this explicit interface. It is not
* exposed interactively and it does not affect `dev` unless the caller passes it
* explicitly there.
*/
export interface TestExecutionOptions {
/**
* Maximum wall-clock time in milliseconds allowed for the test command before the
* orchestrator begins bounded cleanup. Defaults on `DEFAULT_TEST_TIMEOUT_MS`.
*/
readonly timeoutMs?: number;
}

export function resolveTestTimeoutMs(
options: TestExecutionOptions = {},
): number {
const timeoutMs = options.timeoutMs ?? DEFAULT_TEST_TIMEOUT_MS;

if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) {
throw new ValidationCliError(
'Test timeout must be a positive number of milliseconds.',
);
}

return Math.floor(timeoutMs);
}


interface ProjectPackage {
readonly name?: unknown;
readonly scripts?: unknown;
Expand Down
4 changes: 2 additions & 2 deletions tests/test-project.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { describe, expect, it, vi } from 'vitest';
import { ValidationCliError } from '../src/errors/errors.js';
import { createNpmInvocation } from '../src/process/npm.js';
import { createTestCommand } from '../src/commands/test.js';
import { resolveTestProcessPlan } from '../src/testing/project.js';
const { resolveTestProcessPlan } = await import('../src/testing/project.js') as typeof import('../src/testing/project.js');
import {
createCapturedTerminalOutput,
withTempDirectory,
Expand Down Expand Up @@ -120,4 +120,4 @@ describe('project test workflow resolution', () => {
'does not define a test script',
);
}));
});
});
57 changes: 35 additions & 22 deletions tests/test-workflow-platform-and-exit.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/**
/*
* `stellarforge test` platform and exit-code semantics (SW-068, SW-070).
*
* Two gaps in the test workflow's coverage:
Expand All @@ -21,7 +21,11 @@ import { existsSync, mkdirSync, writeFileSync } from 'node:fs';
import { resolve } from 'node:path';
import { PassThrough } from 'node:stream';
import { afterEach, describe, expect, it } from 'vitest';
import { resolveTestProcessPlan } from '../src/testing/project.js';
import {
resolveTestProcessPlan,
DEFAULT_TEST_TIMEOUT_MS,
TEST_TIMEOUT_EXIT_CODE,
} from '../src/testing/project.js';
import { superviseDevProcesses } from '../src/dev/orchestrator.js';
import type {
DevSignalSource,
Expand Down Expand Up @@ -141,30 +145,27 @@ describe('Cargo test exit-code propagation (#168)', () => {
{ code: 1, reason: 'test assertion failed' },
{ code: 101, reason: 'cargo test harness panic' },
{ code: 127, reason: 'cargo not executable' },
])(
'maps a non-zero Cargo exit ($code) to a subprocess failure',
async ({ code }) => {
const child = new FakeChild();
])('maps a non-zero Cargo exit ($code) to a subprocess failure', async ({ code }) => {
const child = new FakeChild();

const running = superviseDevProcesses([cargoSpec()], {
spawnProcess: () => child,
signalSource: new FakeSignalSource(),
});
const running = superviseDevProcesses([cargoSpec()], {
spawnProcess: () => child,
signalSource: new FakeSignalSource(),
});

child.emitExit(code, null);
child.emitExit(code, null);

const error = await running.then(
() => undefined,
(reason: unknown) => reason,
);
const error = await running.then(
() => undefined,
(reason: unknown) => reason,
);

expect(error).toBeInstanceOf(SubprocessCliError);
expect((error as SubprocessCliError).userMessage).toContain(
`with code ${code}`,
);
expect((error as SubprocessCliError).userMessage).toContain('cargo:test');
},
);
expect(error).toBeInstanceOf(SubprocessCliError);
expect((error as SubprocessCliError).userMessage).toContain(
`with code ${code}`,
);
expect((error as SubprocessCliError).userMessage).toContain('cargo:test');
});

it('invokes Cargo with an argument array, never a joined shell string', () => {
const spec = cargoSpec();
Expand All @@ -175,6 +176,18 @@ describe('Cargo test exit-code propagation (#168)', () => {
});
});

describe('test workflow timeout policy (SW-158)', () => {
it('documents a bounded default timeout', () => {
expect(DEFAULT_TEST_TIMEOUT_MS).toBeGreaterThan(0);
expect(Number.isFinite(DEFAULT_TEST_TIMEOUT_MS)).toBe(true);
});

it('uses a dedicated exit code that is distinct from test failure', () => {
expect(TEST_TIMEOUT_EXIT_CODE).not.toBe(0);
expect(TEST_TIMEOUT_EXIT_CODE).not.toBe(1);
});
});

describe('resolved plan matches the platform (#170)', () => {
it('uses the Windows npm wrapper for a Node.js project on win32', async () => {
await withTempDirectory(async (root: string) => {
Expand Down