From e0089c075de5bf2c113d46b963938f79d22eb509 Mon Sep 17 00:00:00 2001 From: Jochen Delabie Date: Sat, 31 Oct 2026 14:34:44 +0100 Subject: [PATCH] Add --priority: queue priority among your own runs waiting for a slot high, normal (default) or low for maestro, espresso and xcuitest, sent as `priority` on the run API. Higher priority starts first, then the oldest; it does not take slots from other accounts or raise the parallel limit. --- CHANGELOG.md | 2 ++ README.md | 3 +++ src/cli.ts | 15 ++++++++++++ src/models/espresso_options.ts | 9 ++++++++ src/models/maestro_options.ts | 28 +++++++++++++++++++++++ src/models/xcuitest_options.ts | 9 ++++++++ src/providers/espresso.ts | 2 ++ src/providers/maestro.ts | 2 ++ src/providers/xcuitest.ts | 2 ++ tests/models/maestro_options.test.ts | 23 +++++++++++++++++++ tests/providers/espresso.test.ts | 20 ++++++++++++++++ tests/providers/maestro.test.ts | 34 ++++++++++++++++++++++++++++ tests/providers/xcuitest.test.ts | 20 ++++++++++++++++ 13 files changed, 169 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 367441b..ec91d86 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,8 @@ All notable changes to `@testingbot/cli` are documented here. Releases are publi - `--timeout ` for `testingbot maestro`, `espresso` and `xcuitest`: how long to wait for results, counted from when the tests start and shared by every `--retry` attempt. Default 60. - `--fail-fast` for `testingbot maestro`, `espresso` and `xcuitest`: at the first failure that will not be retried, the runs still queued or running are cancelled. The command waits for them to stop, then reports as usual and fails with exit code 2. +- `--priority high|normal|low` for `testingbot maestro`, `espresso` and `xcuitest`: where the runs go in your own queue while they wait for a free parallel slot. High starts before normal, normal before low, then oldest first. It does not take slots from other accounts or raise your parallel limit. Needs the matching API change; older API versions ignore it. + ### Changed - When the wait for results times out (after `--timeout`, or the default 60 minutes), the CLI now cancels the runs that are still queued or running, prints the results so far and exits with code 2 (`outcome: failed` with an `error` in `--json`). Before, it exited with code 1 and left the runs going. `testingbot status --wait` still only stops watching and leaves the runs alone. diff --git a/README.md b/README.md index 9fe34b6..7c87317 100644 --- a/README.md +++ b/README.md @@ -189,6 +189,7 @@ Values passed with `-e` are stored in plain text: they are returned in the run's | `--retry ` | Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with `--async`. | | `--timeout ` | Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. | | `--fail-fast` | Stop at the first failure that will not be retried: the runs still queued or running are cancelled, and the summary, reports and artifacts cover what ran. With `--retry`, a test only counts as failed once its retries are used up. Cannot be combined with `--async`. | +| `--priority ` | Queue priority among your own runs waiting for a free parallel slot: `high`, `normal` (default) or `low`. Higher priority starts first, then the oldest run. It does not take slots from other accounts or raise your parallel limit. | | `--ignore-checksum-check` | Skip checksum verification and always upload the app | > **Note on `--retry`:** a failed flow/shard is retried immediately — as soon as it fails — without waiting for the other flows in the run to finish. Retry attempts appear live in the flow table marked with a `↻` icon. Each flow is retried independently up to N times, stopping as soon as that flow passes. Pass/fail uses the result of the **last** attempt per flow (last-attempt-wins), consistently across the CLI exit code, the TestingBot dashboard, and reports downloaded via `--report`. @@ -536,6 +537,7 @@ testingbot espresso [appFile] [testAppFile] [options] | `--retry ` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts | | `--timeout ` | Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. | | `--fail-fast` | Stop at the first failure that will not be retried: the runs still queued or running are cancelled, and the summary, reports and artifacts cover what ran. With `--retry`, a test only counts as failed once its retries are used up. Cannot be combined with `--async`. | +| `--priority ` | Queue priority among your own runs waiting for a free parallel slot: `high`, `normal` (default) or `low`. Higher priority starts first, then the oldest run. It does not take slots from other accounts or raise your parallel limit. | | `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all` (default) or `failed` | | `--artifacts-output-dir ` | Directory for the artifacts zip (defaults to the current directory) | @@ -645,6 +647,7 @@ testingbot xcuitest [appFile] [testAppFile] [options] | `--retry ` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts | | `--timeout ` | Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. | | `--fail-fast` | Stop at the first failure that will not be retried: the runs still queued or running are cancelled, and the summary, reports and artifacts cover what ran. With `--retry`, a test only counts as failed once its retries are used up. Cannot be combined with `--async`. | +| `--priority ` | Queue priority among your own runs waiting for a free parallel slot: `high`, `normal` (default) or `low`. Higher priority starts first, then the oldest run. It does not take slots from other accounts or raise your parallel limit. | | `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all` (default) or `failed` | | `--artifacts-output-dir ` | Directory for the artifacts zip (defaults to the current directory) | diff --git a/src/cli.ts b/src/cli.ts index d037e5f..012f346 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -550,6 +550,10 @@ program '--fail-fast', 'Stop at the first failure that will not be retried: cancel the runs still queued or running.', ) + .option( + '--priority ', + 'Queue priority among your own runs waiting for a free parallel slot: high, normal (default) or low.', + ) .option( '--download-artifacts [mode]', 'Download the video, logs and screenshots after completion. Mode: all (default) or failed.', @@ -660,6 +664,7 @@ program retry: args.retry, timeout: args.timeout, failFast: args.failFast, + priority: args.priority, downloadArtifacts: args.downloadArtifacts === true ? 'all' @@ -838,6 +843,10 @@ program '--fail-fast', 'Stop at the first failure that will not be retried: cancel the runs still queued or running.', ) + .option( + '--priority ', + 'Queue priority among your own runs waiting for a free parallel slot: high, normal (default) or low.', + ) .option( '--async', 'Start tests and exit immediately without waiting for results.', @@ -1079,6 +1088,7 @@ program retry: args.retry, timeout: args.timeout, failFast: args.failFast, + priority: args.priority, cancelPrevious: args.cancelPrevious, debug: args.debug, googlePlayStore: args.googlePlay, @@ -1189,6 +1199,10 @@ program '--fail-fast', 'Stop at the first failure that will not be retried: cancel the runs still queued or running.', ) + .option( + '--priority ', + 'Queue priority among your own runs waiting for a free parallel slot: high, normal (default) or low.', + ) .option( '--download-artifacts [mode]', 'Download the video, logs and screenshots after completion. Mode: all (default) or failed.', @@ -1292,6 +1306,7 @@ program retry: args.retry, timeout: args.timeout, failFast: args.failFast, + priority: args.priority, downloadArtifacts: args.downloadArtifacts === true ? 'all' diff --git a/src/models/espresso_options.ts b/src/models/espresso_options.ts index 595a971..b1b02ab 100644 --- a/src/models/espresso_options.ts +++ b/src/models/espresso_options.ts @@ -2,7 +2,9 @@ import { randomUUID } from 'node:crypto'; import TestingBotError from './testingbot_error'; import { MAX_RETRY, + parsePriority, validateTimeout, + type Priority, type ArtifactDownloadMode, type RunMetadata, } from './maestro_options'; @@ -92,6 +94,7 @@ export default class EspressoOptions { private _retry: number; private _timeout: number | undefined; private _failFast: boolean; + private _priority?: Priority; private _downloadArtifacts?: ArtifactDownloadMode; private _artifactsOutputDir?: string; // Metadata @@ -132,6 +135,7 @@ export default class EspressoOptions { retry?: number; timeout?: number; failFast?: boolean; + priority?: string; downloadArtifacts?: ArtifactDownloadMode; artifactsOutputDir?: string; metadata?: RunMetadata; @@ -192,6 +196,7 @@ export default class EspressoOptions { this._timeout = options?.timeout; validateTimeout(this._timeout); this._failFast = options?.failFast ?? false; + this._priority = parsePriority(options?.priority); this._downloadArtifacts = options?.downloadArtifacts; this._artifactsOutputDir = options?.artifactsOutputDir; this._metadata = options?.metadata; @@ -332,6 +337,10 @@ export default class EspressoOptions { return this._failFast; } + public get priority(): Priority | undefined { + return this._priority; + } + public get downloadArtifacts(): ArtifactDownloadMode | undefined { return this._downloadArtifacts; } diff --git a/src/models/maestro_options.ts b/src/models/maestro_options.ts index 09041c3..abf682f 100644 --- a/src/models/maestro_options.ts +++ b/src/models/maestro_options.ts @@ -93,6 +93,27 @@ export function validateTimeout(timeout: number | undefined): void { } } +/** + * --priority: where these runs go in your own queue while they wait for a free + * parallel slot. high starts before normal, normal before low, then oldest + * first. It does not take slots from other accounts or raise your limits. + */ +export const PRIORITIES = ['high', 'normal', 'low'] as const; +export type Priority = (typeof PRIORITIES)[number]; + +export function parsePriority( + priority: string | undefined, +): Priority | undefined { + if (priority === undefined) return undefined; + const level = priority.trim().toLowerCase(); + if (!(PRIORITIES as readonly string[]).includes(level)) { + throw new Error( + `Invalid --priority value (${priority}). Use one of: ${PRIORITIES.join(', ')}.`, + ); + } + return level as Priority; +} + export default class MaestroOptions { private static isIpaFile(app?: string): boolean { return app?.toLowerCase().endsWith('.ipa') ?? false; @@ -151,6 +172,7 @@ export default class MaestroOptions { private _retry: number; private _timeout: number | undefined; private _failFast: boolean; + private _priority?: Priority; private _cancelPrevious: boolean; private _debug: boolean; private _configFile?: string; @@ -195,6 +217,7 @@ export default class MaestroOptions { retry?: number; timeout?: number; failFast?: boolean; + priority?: string; cancelPrevious?: boolean; debug?: boolean; configFile?: string; @@ -277,6 +300,7 @@ export default class MaestroOptions { this._timeout = options?.timeout; validateTimeout(this._timeout); this._failFast = options?.failFast ?? false; + this._priority = parsePriority(options?.priority); this._debug = options?.debug ?? false; this._configFile = options?.configFile; this._groups = options?.groups; @@ -441,6 +465,10 @@ export default class MaestroOptions { return this._failFast; } + public get priority(): Priority | undefined { + return this._priority; + } + public get cancelPrevious(): boolean { return this._cancelPrevious; } diff --git a/src/models/xcuitest_options.ts b/src/models/xcuitest_options.ts index 8a721ff..97832dc 100644 --- a/src/models/xcuitest_options.ts +++ b/src/models/xcuitest_options.ts @@ -2,7 +2,9 @@ import { randomUUID } from 'node:crypto'; import TestingBotError from './testingbot_error'; import { MAX_RETRY, + parsePriority, validateTimeout, + type Priority, type ArtifactDownloadMode, type RunMetadata, } from './maestro_options'; @@ -79,6 +81,7 @@ export default class XCUITestOptions { private _retry: number; private _timeout: number | undefined; private _failFast: boolean; + private _priority?: Priority; private _downloadArtifacts?: ArtifactDownloadMode; private _artifactsOutputDir?: string; // Metadata @@ -112,6 +115,7 @@ export default class XCUITestOptions { retry?: number; timeout?: number; failFast?: boolean; + priority?: string; downloadArtifacts?: ArtifactDownloadMode; artifactsOutputDir?: string; metadata?: RunMetadata; @@ -165,6 +169,7 @@ export default class XCUITestOptions { this._timeout = options?.timeout; validateTimeout(this._timeout); this._failFast = options?.failFast ?? false; + this._priority = parsePriority(options?.priority); this._downloadArtifacts = options?.downloadArtifacts; this._artifactsOutputDir = options?.artifactsOutputDir; this._metadata = options?.metadata; @@ -277,6 +282,10 @@ export default class XCUITestOptions { return this._failFast; } + public get priority(): Priority | undefined { + return this._priority; + } + public get downloadArtifacts(): ArtifactDownloadMode | undefined { return this._downloadArtifacts; } diff --git a/src/providers/espresso.ts b/src/providers/espresso.ts index c191cf3..6371df7 100644 --- a/src/providers/espresso.ts +++ b/src/providers/espresso.ts @@ -169,6 +169,7 @@ export default class Espresso extends BaseProvider { capabilities: [capabilities], ...(espressoOptions && { espressoOptions }), ...(metadata && { metadata }), + ...(this.options.priority && { priority: this.options.priority }), }, }); @@ -314,6 +315,7 @@ export default class Espresso extends BaseProvider { capabilities: [capabilities], ...(espressoOptions && { espressoOptions }), ...(metadata && { metadata }), + ...(this.options.priority && { priority: this.options.priority }), }, { headers: { diff --git a/src/providers/maestro.ts b/src/providers/maestro.ts index 50ce79f..a4446c6 100644 --- a/src/providers/maestro.ts +++ b/src/providers/maestro.ts @@ -496,6 +496,7 @@ export default class Maestro extends BaseProvider { shardSplit: this.options.shardSplit, }), ...(metadata && { metadata }), + ...(this.options.priority && { priority: this.options.priority }), ...(this.options.cancelPrevious && { cancelPrevious: true }), ...(otherApps.length > 0 && { otherApps: previewOtherAppUrls, @@ -2486,6 +2487,7 @@ export default class Maestro extends BaseProvider { shardSplit: this.options.shardSplit, }), ...(metadata && { metadata }), + ...(this.options.priority && { priority: this.options.priority }), ...(this.options.cancelPrevious && { cancelPrevious: true }), ...(this.otherAppUrls.length > 0 && { otherApps: this.otherAppUrls, diff --git a/src/providers/xcuitest.ts b/src/providers/xcuitest.ts index 08623e1..5deadf2 100644 --- a/src/providers/xcuitest.ts +++ b/src/providers/xcuitest.ts @@ -165,6 +165,7 @@ export default class XCUITest extends BaseProvider { capabilities: [capabilities], ...(xcuitestOptions && { options: xcuitestOptions }), ...(metadata && { metadata }), + ...(this.options.priority && { priority: this.options.priority }), }, }); @@ -310,6 +311,7 @@ export default class XCUITest extends BaseProvider { capabilities: [capabilities], ...(xcuitestOptions && { options: xcuitestOptions }), ...(metadata && { metadata }), + ...(this.options.priority && { priority: this.options.priority }), }, { headers: { diff --git a/tests/models/maestro_options.test.ts b/tests/models/maestro_options.test.ts index c8a6699..8c97ef6 100644 --- a/tests/models/maestro_options.test.ts +++ b/tests/models/maestro_options.test.ts @@ -843,4 +843,27 @@ describe('MaestroOptions', () => { expect(caps).not.toHaveProperty('metadata'); }); }); + + describe('priority', () => { + it('accepts high, normal and low in any case', () => { + expect( + new MaestroOptions('app.apk', 'flows', 'Pixel 6', { priority: ' LOW ' }) + .priority, + ).toBe('low'); + expect(new MaestroOptions('app.apk', 'flows', 'Pixel 6').priority).toBe( + undefined, + ); + }); + + it('rejects anything else', () => { + expect( + () => + new MaestroOptions('app.apk', 'flows', 'Pixel 6', { + priority: 'urgent', + }), + ).toThrow( + 'Invalid --priority value (urgent). Use one of: high, normal, low.', + ); + }); + }); }); diff --git a/tests/providers/espresso.test.ts b/tests/providers/espresso.test.ts index f77658f..fbadec0 100644 --- a/tests/providers/espresso.test.ts +++ b/tests/providers/espresso.test.ts @@ -577,6 +577,26 @@ describe('Espresso', () => { ); }); + it('sends --priority with the run', async () => { + const prioritized = new Espresso( + mockCredentials, + new EspressoOptions( + 'path/to/app.apk', + 'path/to/testApp.apk', + 'Pixel 6', + { + priority: 'low', + }, + ), + ); + prioritized['appId'] = 1234; + axios.post = jest.fn().mockResolvedValueOnce({ data: { success: true } }); + + await prioritized['runTests'](); + + expect((axios.post as jest.Mock).mock.calls[0][1].priority).toBe('low'); + }); + it('should not include metadata when not provided', async () => { espresso['appId'] = 1234; diff --git a/tests/providers/maestro.test.ts b/tests/providers/maestro.test.ts index 1848cda..c0bc7b2 100644 --- a/tests/providers/maestro.test.ts +++ b/tests/providers/maestro.test.ts @@ -993,6 +993,40 @@ describe('Maestro', () => { }); }); + describe('--priority', () => { + it('sends the priority with the run', async () => { + const provider = new Maestro( + mockCredentials, + new MaestroOptions('path/to/app.apk', 'path/to/flows', 'Pixel 6', { + priority: 'High', + }), + ); + provider['appId'] = 1234; + axios.post = jest + .fn() + .mockResolvedValueOnce({ data: { success: true }, headers: {} }); + + await provider['runTests'](); + + expect((axios.post as jest.Mock).mock.calls[0][1].priority).toBe( + 'high', + ); + }); + + it('is not sent unless asked', async () => { + maestro['appId'] = 1234; + axios.post = jest + .fn() + .mockResolvedValueOnce({ data: { success: true }, headers: {} }); + + await maestro['runTests'](); + + expect((axios.post as jest.Mock).mock.calls[0][1]).not.toHaveProperty( + 'priority', + ); + }); + }); + it('should not include maestroOptions when none are set', async () => { maestro['appId'] = 1234; diff --git a/tests/providers/xcuitest.test.ts b/tests/providers/xcuitest.test.ts index 3a99b85..318789d 100644 --- a/tests/providers/xcuitest.test.ts +++ b/tests/providers/xcuitest.test.ts @@ -526,6 +526,26 @@ describe('XCUITest', () => { ); }); + it('sends --priority with the run', async () => { + const prioritized = new XCUITest( + mockCredentials, + new XCUITestOptions( + 'path/to/app.ipa', + 'path/to/testApp.zip', + 'iPhone 15', + { + priority: 'low', + }, + ), + ); + prioritized['appId'] = 1234; + axios.post = jest.fn().mockResolvedValueOnce({ data: { success: true } }); + + await prioritized['runTests'](); + + expect((axios.post as jest.Mock).mock.calls[0][1].priority).toBe('low'); + }); + it('should not include metadata when not provided', async () => { xcuiTest['appId'] = 1234;