Memory regression tests by alan-george-lk · Pull Request #275 · livekit/client-sdk-cpp

alan-george-lk · 2026-10-08T00:55:06Z

This PR introduces a new test binary called memory_tester for Linux, which has the following features:

Configurable common scenarios
- Examples: publishing tracks, room joining/closure
RSS and thread usage tracking
Configurable repeated iterations, acceptable memory and thread growth bounds

CI is also updated to run this tester and provide a nice markdown table report of the results, and fail jobs that introduce changes that fail this.

Documentation is also updated around testing in general, as it was a bit out of date.

Results

Here are two FFI release results with the following params:

Iterations: 100
Warmup: 20
Threshold: RSS growth ≤ 8 MiB; thread growth ≤ 0.

Prior release livekit-ffi/v0.12.81 results:

Scenario	RSS 0→20	RSS 20→100	Threads 0→20	Threads 20→100	Status
audio-track	+60.63 MiB	-2.64 MiB	+28	0	✅ PASS
video-track	+61.43 MiB	-2.62 MiB	+28	0	✅ PASS
data-track	+53.88 MiB	+0.03 MiB	+2	0	✅ PASS
room-reuse	+53.54 MiB	+3.47 MiB	+2	0	✅ PASS
room-client-leave	+53.66 MiB	+0.12 MiB	+2	0	✅ PASS
room-server-delete	+143.46 MiB	+108.94 MiB	+105	+400	❌ FAIL

Current FFI release results:

Scenario	RSS 0→20	RSS 20→100	Threads 0→20	Status
audio-track	+17.36 MiB	+0.19 MiB	+12	✅ PASS
video-track	+16.74 MiB	-0.05 MiB	+7	✅ PASS
data-track	+12.30 MiB	+0.03 MiB	+1	✅ PASS
room-reuse	+12.09 MiB	0.00 MiB	+1	✅ PASS
room-client-leave	+12.10 MiB	+0.05 MiB	+1	✅ PASS
room-server-delete	+12.11 MiB	+0.08 MiB	+1	✅ PASS

…ature-memory-regression-tests

devin-ai-integration

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

stephen-derosa

overall looks great, thanks for adding these!

stephen-derosa · 2026-10-16T16:16:31Z

+          matrix.e2e-testing &&
+          (inputs.integration_repeat > 0 ||
+           inputs.run_stress_tests ||
+           (matrix.name == 'ubuntu-24.04-x64' && inputs.build_type == 'release'))


we are only running these on ubuntu x64 release builds?

Fair callout, but some specific points:

The original customer report was on Linux/x64

The tools I built this with are linux-specific (the way RSS is tracked, and the MALLOC_ARENA stuff), given the original customer report

Running on multiple platforms will clog up the GitHub summary page/take more CI time for likely negligible coverage differences

So I felt having a single run on the most representative platform for robotics/physical AI (resource constrainted linux) would be most applicable

stephen-derosa · 2026-10-16T17:11:28Z

+  delegate.requireDisconnect(livekit::DisconnectReason::ClientInitiated);
+}
+
+// Exercises data-track publication, payload enqueueing, explicit unpublication,


these doc strings are great

…ature-memory-regression-tests

devin-ai-integration

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 1 new potential issue.

4 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-11-07T00:09:09Z

runCycle

+      capture-clock
+      capture-pattern


🟡 All-scenario runs fail on standard builds

After a default release-tests build, --scenario all runs capture scenarios that the SDK cannot execute. Both fail, making the all-scenario run fail even when every standard scenario passes.

Learn more

The build defaults to capture disabled, but still produces the Linux memory tester. The script now appends both capture scenarios to all regardless of that build configuration. In a standard build, runCycle rejects each capture scenario, and the script returns a failure even if every other scenario passes.

Example: Run ./build.sh release-tests followed by source scripts/set-test-tokens.sh and scripts/memory-tester.sh --scenario all. The six standard scenarios can pass, but both capture scenarios fail with a capture-enabled-build error.

Recommended fix: Make all select capture scenarios only when the built SDK supports their respective sources, or provide a standard-only group and retain the existing behavior of all for non-capture builds. Keep explicit capture scenario requests failing clearly when unsupported.

Was this helpful? React with 👍 or 👎 to provide feedback.

stephen-derosa · 2026-11-09T17:20:02Z

    "${CMAKE_CURRENT_SOURCE_DIR}/memory/memory_tester.cpp"
  )
  target_link_libraries(livekit_memory_tester PRIVATE livekit)
-  target_include_directories(livekit_memory_tester PRIVATE


hmu why is this no longer needed?

devin-ai-integration

Devin Review found 2 new potential issues.

5 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-11-10T16:29:52Z

+    if (( BASH_REMATCH[11] > advisory_max_thread_growth )); then
+      advisory_exceeded=true


🟡 Leading-zero thread limits hide advisories

With --advisory-max-thread-growth 08, a nine-thread increase passes the blocking gate but produces no advisory. Bash rejects 08 as octal inside the conditional, then continues to report PASS.

Learn more

The advisory thread limit accepts decimal digits, including strings with leading zeros. Bash arithmetic treats a leading zero as an octal literal, so 08 or 09 raises an arithmetic error in the conditional. Since the expression is in an if, the script continues without setting advisory_exceeded; a passing test can appear as PASS despite exceeding its advisory limit.

Example: Run with --max-thread-growth 12 --advisory-max-thread-growth 08. A final growth of nine threads passes the binary gate, while the shell prints an arithmetic error and skips the warning.

Recommended fix: Normalize advisory_max_thread_growth to decimal before using it in Bash arithmetic, or explicitly convert its digits as base 10. Cover leading-zero limits such as 08 in a shell test.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-11-10T16:29:53Z

main

+    if awk -v growth="${BASH_REMATCH[6]}" -v limit="${advisory_max_rss_growth_mib}" \
+        'BEGIN { exit !(growth > limit) }'; then


🟡 Rounded RSS growth hides threshold breaches

When RSS grows 8 MiB plus 4 KiB, awk compares the displayed 8.00 MiB against the advisory limit of 8. No warning appears, although measured growth exceeds the limit.

Learn more

The tester computes its RSS deltas in KiB, but main formats the report with two decimal places. The wrapper parses that rounded report for the advisory gate. A delta slightly above the configured limit can round down to the limit, leaving the scenario marked PASS instead of ADVISORY.

Example: The warmup-to-final delta is 8196 KiB, or 8.00390625 MiB. The report prints 8.00 MiB; against an 8 MiB advisory limit the wrapper emits no warning.

Recommended fix: Make the tester emit an unrounded machine-readable delta for the wrapper, or pass both limits into the tester and compare against its KiB samples before formatting.

Was this helpful? React with 👍 or 👎 to provide feedback.

alan-george-lk added 7 commits October 14, 2026 20:16

Fix "local track published for unknown sid" warning

2dab90b

Memory regression test

5adaeef

More scenarios

a89e5eb

Split audio/video track

a92fea4

Warm up stats vs. final

8ba2935

Cleanup docs, comments, add room reuse

682f6d6

Megabytes, emojis

d638cca

alan-george-lk force-pushed the alan/feature-memory-regression-tests branch from 2a21a8a to d638cca Compare October 15, 2026 03:06

alan-george-lk added 2 commits October 15, 2026 11:09

Merge branch 'main' of github.com:livekit/client-sdk-cpp into alan/fe…

da57840

…ature-memory-regression-tests

Rename and update docs

9af5d01

alan-george-lk marked this pull request as ready for review October 15, 2026 23:35

alan-george-lk requested review from a team and xianshijing-lk as code owners October 15, 2026 23:35

alan-george-lk changed the title ~~WIP: Memory regression tests~~ Memory regression tests Oct 15, 2026

devin-ai-integration Bot reviewed Oct 15, 2026

View reviewed changes

alan-george-lk added 4 commits October 15, 2026 17:42

Name alignment

c625207

Actually catch room server delete

4cf631b

Fix typo

b3381ed

Maybe fix instability

565be38

This comment was marked as resolved.

Sign in to view

Address devin.ai comments

205ed52

stephen-derosa approved these changes Oct 16, 2026

View reviewed changes

alan-george-lk added 4 commits October 19, 2026 11:20

PR feedback

2845a98

Merge branch 'main' of github.com:livekit/client-sdk-cpp into alan/fe…

295f1cf

…ature-memory-regression-tests

Latest rust

16d3f88

Add capture scenarios

1234d35

devin-ai-integration Bot reviewed Nov 7, 2026

View reviewed changes

Add gstreamer capture test

4e1ac0b

stephen-derosa approved these changes Nov 9, 2026

View reviewed changes

Change to advisory model to account for webrtc issues

ef935ef

devin-ai-integration Bot reviewed Nov 10, 2026

View reviewed changes

		if (( BASH_REMATCH[11] > advisory_max_thread_growth )); then
		advisory_exceeded=true

		if awk -v growth="${BASH_REMATCH[6]}" -v limit="${advisory_max_rss_growth_mib}" \
		'BEGIN { exit !(growth > limit) }'; then

Conversation

alan-george-lk commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Results

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

🔍 Devin Review: 2 flags

Uh oh!

This comment was marked as resolved.

Uh oh!

stephen-derosa left a comment

Choose a reason for hiding this comment

Uh oh!

stephen-derosa Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

alan-george-lk Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephen-derosa Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stephen-derosa Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alan-george-lk commented Oct 8, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading

alan-george-lk Oct 19, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading

devin-ai-integration Bot Nov 7, 2026 •

edited

Loading