Skip to content
OpenBMBPublic

About

EC2-Bench (EC²-Bench): benchmark for edge-cloud collaboration in long-horizon LLM agents — task performance, resource usage & privacy exposure. arXiv 2610.17816

Topics

Resources

Stars

24 stars

Watchers

0 watching

Forks

Repository files navigation

EC²-Bench

Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution

page arXiv dataset license

128 executable task definitions · task-level privacy notes · 4 collaboration strategies · Task Performance · Resource Usage · Privacy Exposure

EC²-Bench overview: a long-horizon edge-cloud collaborative trajectory, four collaboration strategies, and three research questions

EC²-Bench (EC2-Bench; Edge–Cloud Collaborative Agent Benchmark) is a benchmark for edge–cloud collaboration in long-horizon LLM agent execution. Edge-side models keep data local and cost little but are weak at demanding planning and tool use. Cloud-hosted frontier models are far more capable, but they are expensive and see everything they are sent. Prior edge–cloud systems coordinate the two once per query. In long-horizon agent execution, cloud participation is instead interleaved throughout the trajectory. That creates trajectory-level effects: each cloud call shapes later steps, accumulates cost, and decides how much workspace information leaves the device.

EC²-Bench provides 128 digital-workspace task definitions with inline evaluation checks and task-level privacy notes, together with a unified OpenClaw execution and evaluation environment. It scores every run on three axes at once: task performance, resource usage, and privacy exposure. We use it to study four representative collaboration strategies across edge models of increasing capability.

Inspect the 128-task index · Try one spreadsheet task · Compare against the paper results · Cite the paper

Key Findings

  1. Edge–cloud collaboration works for long-horizon agents (RQ1). Partial cloud participation improves performance over edge-only execution at a fraction of cloud-only cost and exposure. With Qwen3.5-2B, Per-step Routing lifts completion from 12.75 to 39.97 at 22.7% of cloud-only cost, and many configurations beat a linear edge↔cloud interpolation.
  2. Stronger edges close the performance gap, not the privacy gap (RQ2). The best collaborative configuration climbs from 39.97 (2B) to 76.84 (Qwen3.7-Plus), approaching cloud-only's 82.70. The completion gap across strategies narrows from 25.22 points (2B) to 3.67 (Qwen3.7-Plus), but privacy exposure does not converge: with Qwen3.7-Plus, strategies differ by only 3.67 points in completion but by 66.23 in privacy exposure.
  3. The best collaboration mode shifts with edge capability (RQ3). Weak edges benefit most from external routing and direct cloud execution; stronger edges benefit more from edge-controlled invocation and cloud advisory. Two edge-side abilities underlie the shift: deciding when to invoke the cloud (up to +4.4 over a matched random-invocation baseline) and following cloud guidance (instruction-following training raises advisory completion from 24.8 to 28.8).

Design Highlights

What we test Why it matters
🦞 OpenClaw-native The real OpenClaw agent loop (bash, browser, file ops, APIs, and reusable SKILL.md skills) driving a live local workspace Tasks need long-horizon planning, state tracking, and error recovery. Cloud calls land mid-trajectory over accumulated workspace context, not on a static prompt
🔀 Collaboration-centric 4 strategies spanning who decides when the cloud joins (external router vs. edge agent) and what the cloud does (execute vs. advise) Isolates how collaboration is organized from which models are used
🔐 Privacy exposure 100 tasks with 3,385 sensitivity units (personal + organizational, 12 categories); an LLM judge reaches 96.9% F1 against human labels Every cloud invocation is a potential leakage channel, so we audit what crosses the boundary over the whole trajectory
⚖️ Multi-dimensional Task performance · Resource usage · Privacy exposure, reported jointly No single number hides the trade-off
📦 Reproducible Each task runs in its own Docker container Graders are injected only after the agent finishes and are never visible during execution

Tasks

128 executable task definitions across 6 categories (Chinese & English). Each Markdown file under tasks/EC2Bench/ contains a prompt, an inline grade() verifier, a workspace path, and a descriptive privacy_note. Sensitive-unit evaluation reads gt/sensitive_items.json under each task workspace.

Category # Example tasks Core challenges
Office & Daily Tasks 36 ambiguous contact email, meeting notes, expense report, daily summary Multi-source aggregation, clarification, structured output
Information Search & Gathering 34 email search, competitive intelligence, paper affiliation lookup, CRM bug hunt Web + local data reconciliation, source verification
Safety & Security 21 leaked API-key detection, prompt injection, malicious skill, HIPAA/PHI referral Adversarial robustness, credential awareness, refusal
Development & Operations 14 system health check, automation-failure recovery, LLM API gateway skill Undocumented setups, debugging, skill creation
Data Analysis 13 order-profit analysis, month-end reconciliation, quarterly business insight Spreadsheet reasoning, state verification
Automation 10 flight booking, n8n workflow report, scheduled-briefing skill Long-horizon orchestration, recovery

Tasks are grouped by their original source labels: Claw-Eval (68), WildClawBench (22), QwenClawBench (21), LiveClawBench (4), PinchBench (3), and ClawBench (3). The ClawBench group includes two TrajectoryRL scenarios and one communication-audit task from an earlier task collection, with its original identifier preserved. Seven more tasks were curated to cover under-represented scenarios. Every task was standardized to one runtime and evaluation interface, then filtered so that edge-only, cloud-only, and collaborative execution can meaningfully differ.

Scoring. Every run is graded on three dimensions at once:

  • Task Performance: completion score (weighted, task-specific checks combining rule-based validation, environment-state auditing, and rubric-based LLM-as-a-judge) plus Pass³, the fraction of tasks with score > 0.7 in all 3 runs.
  • Resource Usage: cloud input / cache-read / output tokens with USD cost, plus edge-side FLOPs.
  • Privacy Exposure: the fraction of annotated sensitivity units that become visible to the cloud anywhere along the trajectory (lower is better).

Leaderboard

Edge = Qwen3.5-2B / 9B / 27B and Qwen3.7-Plus (27B and Plus serve as proxies for future, stronger edge models); Cloud = GLM-5.3; judge = GPT-5.4-mini; averaged over 3 runs. Cloud Tok. = raw / cache-read / output (millions); Cloud Cost in USD (OpenRouter pricing); Edge FLOPs in PetaFLOPs (not reported for Qwen3.7-Plus, whose parameter count is unavailable); Completion, Pass³, and Privacy Exposure in %.

EC²-Bench main results (paper Table 2, cloud = GLM-5.3)
Text version of the table above
Method Completion ↑ Pass³ ↑ Cloud Tok. (R/C/O) Cloud Cost ↓ Edge FLOPs ↓ Privacy Exp. ↓
Edge-only
Qwen3.5-2B 12.75 2.34 – – 142.98 0.00
Qwen3.5-9B 39.12 9.38 – – 121.64 0.00
Qwen3.5-27B 61.79 30.47 – – 564.49 0.00
Qwen3.7-Plus 71.34 53.12 – – – 0.00
Cloud-only
DeepSeek-v4-Pro 72.90 53.12 5.81 / 50.20 / 1.60 16.41 – 86.66
GLM-5.3 82.70 71.88 13.52 / 65.76 / 4.64 56.42 – 87.01
Edge-cloud collaboration (cloud = GLM-5.3)
Qwen3.5-2B
  Query Routing 23.98 12.50 1.81 / 10.49 / 0.79 8.76 156.30 12.59
  Per-step Routing 39.97 7.81 3.84 / 17.29 / 0.67 12.80 57.05 41.29
  Edge-Ctrl. Delegation 15.86 2.34 0.41 / 0.49 / 0.11 1.19 172.74 2.71
  Edge-Ctrl. Advisory 14.75 1.56 0.67 / 0.51 / 0.29 2.36 240.28 6.07
Qwen3.5-9B
  Query Routing 43.60 17.97 2.13 / 11.28 / 0.78 9.33 162.88 12.59
  Per-step Routing 52.71 21.88 1.98 / 7.46 / 0.31 6.07 29.96 27.52
  Edge-Ctrl. Delegation 52.88 17.19 0.83 / 1.03 / 0.36 3.01 184.98 16.45
  Edge-Ctrl. Advisory 60.46 26.56 2.02 / 2.61 / 1.19 8.73 332.43 45.46
Qwen3.5-27B
  Query Routing 65.53 38.28 2.69 / 14.29 / 0.75 10.76 649.92 12.53
  Per-step Routing 68.42 42.19 1.76 / 6.51 / 0.26 5.29 88.58 35.81
  Edge-Ctrl. Delegation 69.06 44.53 1.00 / 1.50 / 0.71 4.93 455.76 42.42
  Edge-Ctrl. Advisory 70.13 47.66 1.32 / 0.75 / 0.83 5.71 822.23 48.46
Qwen3.7-Plus
  Query Routing 73.17 60.16 4.39 / 15.60 / 0.88 14.03 – 12.51
  Per-step Routing 73.77 56.25 2.17 / 4.95 / 0.22 5.28 – 53.17
  Edge-Ctrl. Delegation 74.24 60.94 1.65 / 1.08 / 0.33 4.03 – 43.36
  Edge-Ctrl. Advisory 76.84 67.19 1.97 / 0.53 / 0.88 6.80 – 78.74
Alternative cloud model: DeepSeek-V4.1-Flash (paper Appendix A.8)
EC²-Bench results with DeepSeek-V4.1-Flash as the cloud model (paper Table 11)
Method Completion ↑ Pass³ ↑ Cloud Tok. (R/C/O) Cloud Cost ↓ Edge FLOPs ↓ Privacy Exp. ↓
Edge-only
Qwen3.5-2B 12.75 2.34 – – 142.98 0.00
Qwen3.5-9B 39.12 9.38 – – 121.64 0.00
Qwen3.5-27B 61.79 30.47 – – 564.49 0.00
Qwen3.7-Plus 71.34 53.12 – – – 0.00
Cloud-only
DeepSeek-v4-Pro 72.90 53.12 5.81 / 50.20 / 1.60 16.41 – 86.66
DeepSeek-v4.1-Flash 81.42 71.88 14.35 / 90.32 / 2.77 4.08 – 90.85
Edge-cloud collaboration (cloud = DeepSeek-v4.1-Flash)
Qwen3.5-2B
  Query Routing 23.46 13.28 2.60 / 8.96 / 0.38 0.64 139.06 13.08
  Per-step Routing 42.70 10.94 8.11 / 17.72 / 0.51 1.57 70.04 46.31
  Edge-Ctrl. Delegation 19.20 2.34 1.00 / 0.74 / 0.10 0.21 145.46 9.52
  Edge-Ctrl. Advisory 15.53 3.12 0.70 / 0.88 / 0.19 0.22 178.10 7.87
Qwen3.5-9B
  Query Routing 45.66 21.88 2.08 / 11.69 / 0.35 0.56 196.51 13.03
  Per-step Routing 50.36 19.53 2.57 / 4.15 / 0.14 0.48 31.20 28.74
  Edge-Ctrl. Delegation 51.79 17.19 1.66 / 1.17 / 0.29 0.43 204.07 21.18
  Edge-Ctrl. Advisory 56.00 24.22 2.82 / 2.50 / 0.62 0.80 340.08 39.25
Qwen3.5-27B
  Query Routing 63.51 39.84 3.68 / 28.34 / 0.52 0.95 601.08 13.07
  Per-step Routing 67.24 42.97 3.96 / 8.47 / 0.27 0.78 85.38 38.47
  Edge-Ctrl. Delegation 67.11 41.41 2.34 / 1.45 / 0.47 0.63 605.25 46.19
  Edge-Ctrl. Advisory 69.50 46.09 1.73 / 1.31 / 0.55 0.59 636.49 49.02
Qwen3.7-Plus
  Query Routing 73.88 60.16 4.07 / 14.97 / 0.48 0.94 – 12.67
  Per-step Routing 73.86 53.91 2.61 / 2.67 / 0.15 0.49 – 55.37
  Edge-Ctrl. Delegation 75.04 59.38 2.41 / 2.75 / 0.41 0.61 – 44.54
  Edge-Ctrl. Advisory 75.31 61.72 1.99 / 1.11 / 0.52 0.61 – 65.56

Quick Start

Use the Hugging Face task index to browse the task mix and annotations without running models. The leaderboard contains the reported results from Tables 2 and 11 of arXiv v2.

Check one task

The example below selects a single spreadsheet summarization task. Start in the repository root with Python 3.13 and an active virtual environment. If hf is unavailable after installation, activate the same environment before downloading the data. The dry run checks the runner's argument wiring without starting Docker or calling a model endpoint.

python -m pip install -r requirements.txt huggingface_hub
python scripts/run.py local-only --runtime docker \
  --task-filter '^EC2Bench_task_28_' --repeat 1 --parallel 1 --dry-run

To execute the same task, use Docker, an OpenAI-compatible edge-model endpoint reachable from the container, and a judge endpoint. Download just the example workspace and build the runtime from this checkout:

hf download chengpingan/EC2-Bench --repo-type dataset \
  --revision 12f9107ff7cb05969f667282438592b3687ed29b \
  --include 'workspace/ACE_Bench/task_28_spreadsheet_summary/**' \
  --local-dir ./ec2bench-data
docker build -f runtime/Dockerfile -t ec2-bench-runtime:review .

Set EDGE_BASE_URL, EDGE_API_KEY, JUDGE_BASE_URL, JUDGE_API_KEY, and JUDGE_MODEL in your local environment (see .env.example). Pass a models config with an edge provider and the matching model ID, following configs/models.example.json. The bundled example config also expands CLOUD_BASE_URL and CLOUD_API_KEY because it contains both providers. With only an edge endpoint, use a copy of the example containing just the edge provider. Keep the environment placeholders in the config.

python scripts/run.py local-only --runtime docker --data-root ./ec2bench-data \
  --task-filter '^EC2Bench_task_28_' --repeat 1 --parallel 1 \
  --edge-model edge/Qwen/Qwen3.5-9B --models-config /path/to/local-models.json \
  --output-dir output/spreadsheet-smoke

This single run checks your setup. The paper compares configurations over the full benchmark with three repeats.

Choose a comparison strategy

The first argument to scripts/run.py selects a strategy. The wrapper maps each strategy to a runner mode listed in configs/strategies.json.

Strategy scripts/run.py argument eval/run_batch.py --run-mode Cloud role
Edge-only local-only local-only No cloud collaborator
Cloud-only cloud-only cloud-only Runs the entire task
Query Routing query-routing query-router An external router selects cloud execution for the entire task
Per-step Routing per-step-routing step-router-execute-glimpse An external router sends selected steps to the cloud
Edge-controlled Delegation delegation agentic_executor_router The edge requests execution of a step
Edge-controlled Advisory advisory agentic-advisor The edge requests advice and continues to execute

Query Routing defaults to the bundled RouteLLM route table, scored with OPEA/routellm-e5-base-v2 and E5 mean pooling. It selects the cloud when score > threshold. Install requirements-routing.txt to score new prompts. The route builder also accepts saved scores through --scores-json. Per-step Routing uses the published configuration for supported model IDs. For an unlisted model, the wrapper requires GLIMPSE_ENTROPY_THRESHOLD. Collaborative modes require both model references. Pass --runtime docker to use the local image, or select --runtime e2b with your own compatible template as described in .env.example.

Check the Results

The runner writes task outputs (including scores and usage) and 0_summary_*.json under a timestamped directory inside --output-dir. For a paper comparison, report the task subset, runner commit, dataset revision, runtime, model and judge IDs, strategy, and repeat count. Read completion and Pass³ alongside the cloud-resource and privacy fields. Match the task set and repeat count when comparing against Table 2 or Table 11.

Cloud token prices used by the runner are declared in configs/metrics.json. The defaults are the snapshot from the paper's Table 12; MODEL_PRICING_FILE can supply prices for a new model. Record the pricing snapshot with any reported dollar estimates. Privacy exposure measures annotated data disclosed to the cloud collaborator along the trajectory. The score depends on the annotations and protocol used here. overall_std is the sample SD across task means, while repeat_overall_std is the sample SD across repeat means. Resource totals aggregate the retained execution attempt for each task and repeat.


Citation

If you use the task suite, compare against the reported results, or build on the evaluation protocol, cite the paper. When reporting a run, also record the code commit and the dataset revision so others can reproduce your setup.

@article{huang2026ec2bench,
  title   = {Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution},
  author  = {Huang, Pengcheng and Liu, Zhenghao and Ren, Shixin and Yu, Shi and Yao, Sijia and Yan, Yukun and Xiao, Chaojun and Sun, Maosong},
  journal = {arXiv preprint arXiv:2610.17816},
  year    = {2026}
}

Acknowledgements

EC²-Bench stands on the shoulders of a remarkable open-source agent community, and we are deeply grateful for it. The OpenClaw harness gives us a real, full-featured agent runtime (tools, skills, and a live workspace) to build on. Our tasks and evaluation design draw inspiration and adapted material from a series of outstanding agent benchmarks: Claw-Eval, WildClawBench, QwenClawBench, LiveClawBench, PinchBench, and ClawBench. Their task curation, grading, and reproducible harness design set the bar for trustworthy agent evaluation and made the privacy-aware, edge–cloud extension in EC²-Bench possible.

License

Released under the MIT License. Adapted tasks are used under the licenses of their source benchmarks: Claw-Eval, WildClawBench, QwenClawBench, LiveClawBench, and PinchBench are MIT-licensed, and the TrajectoryRL ClawBench scenarios (tasks 8 and 9) are MIT-licensed. Applicable upstream copyright and license notices are preserved in THIRD_PARTY_NOTICES.

About

EC2-Bench (EC²-Bench): benchmark for edge-cloud collaboration in long-horizon LLM agents — task performance, resource usage & privacy exposure. arXiv 2610.17816

Topics

Resources

Stars

24 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages