Autoresearch for GPU kernels. Give it any PyTorch model, go to sleep, wake up to optimized Triton or CUDA C++ kernels.
Inspired by @karpathy/autoresearch -- which demonstrated autonomous AI agents for LLM training research. AutoKernel applies the same philosophy to GPU kernel optimization: agent modifies one file, runs a fixed evaluation, keeps or reverts, repeats forever.
Give AutoKernel any PyTorch model. It will:
- Profile the model to find which GPU kernels are bottlenecks
- Extract each bottleneck as a standalone Triton or CUDA C++ kernel
- Optimize each kernel autonomously (edit, benchmark, keep/revert -- forever)
- Verify end-to-end correctness and report the total speedup
The agent reads program.md -- the "research org code" -- which contains comprehensive instructions for autonomous operation. It edits kernel.py one kernel at a time, runs bench.py (fixed benchmark with 5-stage correctness checks + roofline analysis), and either keeps or reverts the change. The orchestrator decides when to move to the next kernel using Amdahl's law.
Each experiment takes ~90 seconds. That's ~40 experiments/hour, ~320 overnight, across all kernels.
Requirements: NVIDIA GPU (tested on H100/A100/RTX 4090), Python 3.10+, uv.
# Install uv (if you don't have it)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Clone and setup
git clone https://github.com/RightNow-AI/autokernel.git
cd autokernel
uv sync
# One-time setup: test data + baselines
uv run prepare.py
# Profile a model (ships with GPT-2, LLaMA, BERT -- no transformers needed)
uv run profile.py --model models/llama_7b.py --class-name LlamaModel \
--input-shape 1,512 --dtype float16
# Extract top bottleneck kernels
uv run extract.py --top 5
# Verify benchmark works
uv run bench.pySpin up Claude, Codex, or any coding agent in this directory:
Read program.md and let's kick off a new experiment. Start with setup.
The agent will:
- Profile your model and present the optimization plan
- Create a branch (e.g.,
autokernel/mar10-llama7b) - Optimize each bottleneck kernel in priority order
- Verify end-to-end correctness and report total speedup
program.md is intentionally comprehensive so the agent can run 10+ hours without getting stuck. It includes a 6-tier optimization playbook, decision framework, crash handling, and Amdahl's law reasoning.
profile.py extract.py bench.py (loop) verify.py
Any PyTorch ──> Rank kernels ──> Generate baseline ──> Optimize each ──> End-to-end
model by GPU time Triton/CUDA kernels kernel (agent) verification
| Tool | What it does |
|---|---|
profile.py |
Profiles any PyTorch model with torch.profiler, ranks kernels by GPU time, classifies as compute/memory-bound |
extract.py |
Extracts top-N bottleneck kernels into standalone Triton or CUDA C++ kernel files (--backend triton|cuda) |
orchestrate.py |
Multi-kernel scheduler: decides which kernel to optimize next using Amdahl's law, tracks aggregate progress |
bench.py |
Fixed benchmark: 5-stage correctness (smoke, shape sweep, numerical stability, determinism, edge cases) + performance + roofline |
verify.py |
Plugs optimized kernels back into the model, checks end-to-end correctness, reports total speedup |
calibrate.py |
Measures achieved HBM bandwidth, FP16/BF16 GEMM throughput and a dense-attention peak on a new GPU, and compares them to the roofline peaks in the GPU database |
10 kernel types covering the core operations of modern deep learning:
| Kernel | Description | Key Metric |
|---|---|---|
| matmul | Dense matrix multiplication (M x K) @ (K x N) | TFLOPS |
| softmax | Row-parallel numerically stable softmax | GB/s |
| layernorm | Layer normalization with affine transform | GB/s |
| rmsnorm | RMS normalization (LLaMA-style) | GB/s |
| flash_attention | Scaled dot-product attention with causal masking | TFLOPS |
| gqa_prefill_attention | One causal prefill chunk against a long KV cache (up to 1M keys), grouped KV heads, bottom-right mask, contiguous or paged KV | TFLOPS |
| fused_mlp | SwiGLU-style fused MLP (gate + up + down) | TFLOPS |
| cross_entropy | Fused cross entropy loss | GB/s |
| rotary_embedding | Rotary position embeddings (RoPE) | GB/s |
| reduce | Parallel reduction (sum) | GB/s |
Each has a PyTorch reference in reference.py, a starter Triton kernel in kernels/, and a starter CUDA C++ kernel in kernels/cuda/ (gqa_prefill_attention has a Triton starter only). gqa_prefill_attention's reference is an exact float32 online softmax over KV tiles, and its timing baseline is PyTorch SDPA, because the oracle is far too slow to time at 1M keys.
gqa_prefill_attention counts FLOPs over the (query, key) pairs the causal mask keeps. A size with a page_size reads K and V the way a serving engine stores them: page pools [num_pages, page_size, heads_kv, head_dim], an int32 block_table and kv_len. The pages are shuffled, and the reserved page 0 and each last page's unused slots hold large values, so a kernel that ignores the table or gets the length wrong fails. --page-size N runs every size at the engine's page size (0 = contiguous). --calibration adds pct_calibrated_peak: throughput as a percentage of the dense-attention peak calibrate.py measured on this GPU at the same head_dim, instead of only the datasheet's tensor-core peak.
uv run calibrate.py --attention 8192x131072:16/2:256 --attention 8192x1048576:16/2:256 --out results/calibrate.json
cp kernels/gqa_prefill_attention.py kernel.py
uv run bench.py --page-size 16 --calibration results/calibrate.jsonSelf-contained model definitions ship with AutoKernel (no transformers library needed):
| Model | File | Params | Usage |
|---|---|---|---|
| GPT-2 Small | models/gpt2.py |
124M | --class-name GPT2 --input-shape 1,1024 |
| LLaMA (compact) | models/llama_7b.py |
160M | --class-name LlamaModel --input-shape 1,512 |
| LLaMA 7B | models/llama_7b.py |
7B | --class-name LlamaModel7B --input-shape 1,2048 |
| BERT-base | models/bert_base.py |
110M | --class-name BertModel --input-shape 8,512 |
| Custom | models/custom.py |
-- | Template for your own model |
For HuggingFace models (uv sync --extra models):
uv run profile.py --module transformers --class-name AutoModelForCausalLM \
--pretrained meta-llama/Llama-2-7b-hf --input-shape 1,2048 --dtype float16AutoKernel integrates with KernelBench, the standard benchmark for evaluating AI-generated GPU kernels (250+ problems across 4 difficulty levels). While most KernelBench evaluations use one-shot LLM generation, AutoKernel runs 50-300+ iterative refinement experiments per problem -- systematically exploring the optimization space instead of guessing.
# Install KernelBench dependencies
uv sync --extra kernelbench
# Fetch Level 1 problems from HuggingFace
uv run kernelbench/bridge.py fetch --source hf --level 1
# Set up a specific problem for optimization
uv run kernelbench/bridge.py setup --level 1 --problem 1 --source hf
# Evaluate (correctness + speedup vs PyTorch reference)
uv run kernelbench/bench_kb.py
# Batch score an entire level (computes fast_p metric)
uv run kernelbench/scorer.py --level 1The agent reads kernelbench/program_kb.md for KernelBench-specific optimization instructions:
how to write ModelNew classes, when to use CUDA C++ vs Triton, fusion strategies per problem
level, and the edit-bench-keep/revert loop adapted for the KernelBench fast_p metric.
| Tool | What it does |
|---|---|
kernelbench/bridge.py |
Loads problems from HuggingFace or local repo, caches them, generates starter kernel.py |
kernelbench/bench_kb.py |
Evaluates ModelNew vs Model: 5-trial correctness + CUDA event timing + stability + determinism |
kernelbench/scorer.py |
Batch evaluation across a level, computes fast_p at thresholds (1.0x, 1.5x, 2.0x, 3.0x, 5.0x) |
kernelbench/program_kb.md |
Agent instructions for KernelBench mode |
engine/prefill_ttft.py times prefill on a running server: single-token streaming requests whose prompts
are exactly ISL random token IDs (for example 128K to 1M), one request at a time.
--mode cold: every run sends one fresh prompt, so the prefix cache cannot help.--mode warm: every run sends the same prefix twice and times both requests, each exactly ISL tokens: cold (a fresh prefix plus a fresh--suffix-tokenstail), then warm (the same prefix plus another fresh tail: a follow-up question over an already-read document;--suffix-tokens 0resends the same prompt). One run reports cold and warm TTFT and their ratio per ISL. When the server reportscached_tokens(vLLM--enable-prompt-tokens-details), a cold request with cached tokens or a warm one without them is invalid; when it does not, warm cells are marked as unverified.
Token counts come from usage (prompt tokens must equal ISL, one completion token). It writes the raw JSON
and a per-request log, and uses the same identity file and comparison gates as decode_step.py: compare
and table refuse a profile mismatch (exit 4). The server's max_model_len must be at least ISL + 1.
uv run engine/prefill_ttft.py run --base-url http://127.0.0.1:8000 --model <served-name> --mode warm \
--isl 131072,262144,524288,1048575 --runs 3 --label stock --identity stock-identity.json --out results/ttft-stock.json
uv run engine/prefill_ttft.py table results/ttft-stock.json # cold vs warm for one run
uv run engine/prefill_ttft.py compare results/ttft-stock.json results/ttft-ours.json --arm-delta image.digest,arm.labelcalibrate.py also measures an attention peak: dense attention of an 8K-query chunk over 128K keys at
head_dim 128 and 256, per SDPA backend (flash, cuDNN, memory-efficient) and FlashInfer when installed, each
checked against the float32 oracle before it is timed. The best one is attention_tflops, next to the
bandwidth and GEMM peaks. Pass --attention QxKV:HQ/HKV:D for other shapes and --out to write the JSON
that bench.py --calibration reads.
Kernel speedups only matter if the serving engine's decode step gets faster. engine/decode_step.py
records SSE event-gap latency and usage-based throughput in a common decode window at requested
batch B. vLLM's output collector can merge outputs when a consumer falls behind: an SSE event
is not necessarily one engine step. /metrics supplies sampled occupancy, not scheduler-level
proof that exactly B requests ran at every instant.
Supply a local identity JSON for each arm. All fields below are required; values are
illustrative placeholders to replace with the actual measurement configuration.
sm_count is per GPU; gpu_count is the allocated GPU count. Use method: "none",
k: 0, and draft_revision: null when speculation is disabled. Additional fields
are preserved and compared. The harness records declarations; it does not attest them.
{
"hardware": {"gpu_name": "NVIDIA H100", "gpu_count": 2, "sm_count": 132, "driver": "<driver version>"},
"image": {"digest": "sha256:<actual image digest>"},
"engine": {"version": "<engine version or commit>"},
"model": {"id": "Qwen/Qwen3-8B", "revision": "<immutable revision>"},
"parallelism": {"tp": 2, "dcp": 1, "ep": 1},
"speculation": {"method": "none", "k": 0, "draft_revision": null},
"kv_dtype": "auto",
"arm": {"label": "stock"}
}uv sync --extra engine
# Baseline twice (the A/A ratio is your noise floor), then the candidate
uv run engine/decode_step.py run --base-url http://127.0.0.1:8000 --model Qwen/Qwen3-8B \
--batch 1,2,4,8,16,32,64 --isl 1 --osl 1024 --label stock \
--identity stock-identity.json --out results/stock.json
# Repeat the same workload for stock-repeat.json, then ours.json with its own identity.
uv run engine/decode_step.py compare results/stock.json results/stock-repeat.json
uv run engine/decode_step.py compare results/stock.json results/ours.json --arm-delta image.digest,arm.label
uv run engine/decode_step.py table results/stock.json results/ours.json --arm-delta image.digest,arm.label- Prompts are random token IDs, so ISL is exact and the prefix cache cannot help.
- Tokens come from continuous
usage.completion_tokens, including multi-token speculative chunks. Missing, non-increasing, or inconsistent usage invalidates the stream; chunk counts never substitute. A successful stream also requires[DONE], astoporlengthfinish reason, the exact requested completion count, and observed prompt usage equal to the requested ISL. Truncated streams, missing terminal evidence and token-count mismatches invalidate the point with reasons. /metricssums all matchingvllm:num_requests_runningseries across engines, with a 250 ms pause between polls. Every attempt is retained inmetric_polls, including failures and request/response timestamps. Theoccupancyrecord states the coverage bounds and reasons: at leastmax(2, ceil(window_s / 1.5) + 1)distinct fresh samples; no sample-to-sample or window-boundary gap above 1.5 s; a fresh sample's scrape completes within 1.0 s; and the engine-series set remains present and stable. Failed or slow polls never count as fresh and are reported, but a lone one voids nothing while the gap bound holds, since/metricscompetes with B streams on a busy API server. Missing, duplicate, negative or non-finite metric values are failed polls. Insufficient coverage makesexact_b: null/UNVERIFIED; adequate coverage with an observed count different from B givesfalse. Eventruemeans only that this sampled occupancy check passed.compareandtablerequireexact_b_all: true, coverage evidence, and usable event statistics, checking raw repetitions when present. Failures refuse with exit 5.--allow-inexactpermits diagnostic output with affected rows and ratios explicitly marked.- Compatibility keys
step_ms_p50,step_ms_p90andtokens_per_stepremain, withstep_basis: "sse-event-gap"; tables say event gap and tokens/event. When identity declaresspeculation.k == 0, any event carrying more than one usage token (including the first event) is counted inmerged_events. A positive count marks that point'sstep_statusUNVERIFIEDand blocks event-gap ratios unless--allow-inexactis passed.EVENT_GAPmeans only an event statistic, never a verified engine step. With speculation or unknown identity, merging cannot be inferred from token counts:merged_eventsis null and the limitation is recorded instep_reason. Cross-check latency with graph-replay time from a profiler. - Both commands refuse different workload profiles, primary rate formulas, or undeclared identity
differences with exit 4, naming the differing keys.
--arm-deltalists exact dotted leaf keys, such asimage.digest,arm.labelorspeculation.method,speculation.k,speculation.draft_revision. It grants no workload exceptions. Parent keys and unknown keys refuse; no differences are implicit. - Without
--identity,runwarns and writesidentity: null. Comparisons refuse missing identity with exit 4 unless--allow-missing-identityis supplied; output then saysUNQUALIFIED. An incomplete or malformed identity cannot use that override.tableapplies the same identity gate. run --dry-runprints the resolved measurement plan without HTTP calls or output writes. Each real warmup/timed run logs progress to<out>.log(ordecode-step.logwithout--out);--logselects another log path. Keep credentials out of identity files, labels, URLs and notes.
New output uses schema autokernel-decode-step.v2. Each repetition includes streams, even
without --keep-timelines: t_first_output_token, t_last_output_token (client monotonic
seconds), usage token counts, and both tok_s_n_over_dt and tok_s_n_minus_1_over_dt.
Here N is the full-stream usage count and dt is last output arrival minus first output
arrival; usage-only trailers do not extend dt. Single-token or nonpositive intervals have null rates.
Each repetition's full-stream rates are the mean across its streams; summaries take the median
across valid repetitions. rate_formula_primary defaults to window: usage-token deltas inside
the common decode window divided by its duration (window_decode_tok_s), per user as
tok_s_window_per_user. The window starts at the latest first-token arrival and ends at the
earliest last-token arrival. For each stream, subtract the last cumulative usage at or before
the start from the last cumulative usage at or before the end; interpolate nothing. This
accounting is independent of the fully-contained-gap filter used for event latency and tokens/event.
window_tokens and window_boundary_usage retain the numerator and per-stream boundary counts;
window_token_basis: "cumulative-at-boundaries" identifies this corrected accounting.
Select --rate-formula-primary '(N-1)/dt' (production decode telemetry's
formula) or 'N/dt' explicitly for the full-stream rates; these can include other requests'
prefill overlap. Tables name the selected formula; tok_s_per_user uses it and decode_tok_s is
B times it.
Old JSON still loads. Its missing identity needs the explicit override, and missing occupancy or
event-statistic evidence also requires --allow-inexact. Old rates retain the label
legacy window delta/dt; summaries cannot reconstruct first/last stream timestamps or missing
boundary tokens. Older files counted only tokens in fully-contained event gaps, so their window
rates cannot compare with corrected boundary-based rates (exit 4, window_token_basis).
Different primary formulas also refuse with exit 4: rerun both arms on the same accounting basis
before using their ratios. Exit 2 denotes bad run arguments or no valid runs;
exit 3 denotes an unreadable result or log. Overrides are diagnostic, not qualified evidence.
Offline regression gates (no server or GPU; includes refusal cases):
python -m pytest engine/test_decode_step.py
# If pytest is unavailable:
python -m unittest engine.test_decode_step -vRun the client on the server's host, and cross-check event-gap measurements against nsys graph-replay time.
Export optimized kernels to the HuggingFace Hub for easy distribution. Users can then load your kernels with a single line:
from kernels import get_kernel
module = get_kernel("your-username/kernel-name")# Export an optimized CUDA kernel
uv run export_hf.py --name my_matmul
# Upload to Hub (requires `pip install kernels` and `huggingface-cli login`)
cd workspace/hf_export/my_matmul
kernels upload . --repo_id your-username/my_matmulautokernel/
kernel.py the file the agent modifies (one kernel at a time)
program.md agent instructions -- the "research org code"
bench.py fixed benchmark + 5-stage correctness harness
reference.py PyTorch reference implementations (ground truth)
prepare.py one-time setup: test data, baselines
profile.py profile any PyTorch model, rank kernels by GPU time
extract.py extract bottleneck kernels into workspace/
orchestrate.py multi-kernel scheduler (Amdahl's law)
verify.py end-to-end model verification + speedup report
export_hf.py export optimized kernels to HuggingFace Kernels format
analysis.py experiment visualization (generates progress.png)
kernels/ starter Triton kernels (9 types)
kernels/cuda/ starter CUDA C++ kernels (9 types, tensor core accelerated)
kernelbench/ KernelBench integration (bridge, eval harness, scorer)
models/ self-contained model definitions (GPT-2, LLaMA, BERT)
workspace/ runtime artifacts (gitignored)
Dual backend: Triton + CUDA C++. Triton for fast iteration (Python-like syntax, compiles in seconds). CUDA C++ for maximum performance (direct access to tensor cores via wmma, PTX intrinsics, shared memory bank-conflict-free layouts). Triton regularly reaches 80-95% of cuBLAS; CUDA C++ can match or exceed it. Both backends share the same kernel_fn() interface -- bench.py runs identically on either.
Correctness first. The benchmark checks kernel output against PyTorch before measuring performance. A fast but wrong kernel is immediately reverted. This prevents the agent from "optimizing" by producing garbage.
Amdahl's law orchestration. The orchestrator prioritizes by impact. A 1.5x speedup on a 60% kernel (1.25x end-to-end) beats a 3x speedup on a 5% kernel (1.03x end-to-end). It moves on when diminishing returns set in.
Single file to modify. The agent only touches kernel.py. Scope stays manageable, diffs reviewable, reverts clean.
TSV logging. Results go to a plain results.tsv file. Human-readable, git-friendly, trivially parseable, no infrastructure.
Every experiment is logged to results.tsv (tab-separated):
| Column | Description |
|---|---|
experiment |
Sequential experiment number (0 = baseline) |
tag |
Short identifier |
kernel_type |
Which kernel (e.g., matmul) |
throughput_tflops |
Measured throughput (higher is better) |
latency_us |
Execution time in microseconds |
pct_peak |
Percentage of GPU theoretical peak |
speedup_vs_pytorch |
Speedup vs PyTorch/cuBLAS |
correctness |
PASS, FAIL, TIMEOUT, or CRASH |
peak_vram_mb |
Peak GPU memory usage |
description |
What was tried |
This project is autoresearch for GPU kernels -- directly inspired by Andrej Karpathy's autoresearch, the original experiment in autonomous AI research agents for LLM training. Karpathy showed that an AI agent can run hundreds of experiments overnight, methodically exploring a search space and logging every result. AutoKernel applies that same loop -- agent edits one file, runs a fixed evaluation, keeps or reverts -- to the domain of GPU kernel optimization with Triton and native CUDA C++.
KernelBench integration is based on the work of Simon Guo, Sean Resta, et al. at Stanford's Scaling Intelligence Lab. Their paper "KernelBench: Can LLMs Write GPU Kernels?" (2025) established the standard benchmark for evaluating AI-generated GPU kernels. AutoKernel extends this by applying iterative optimization (300+ experiments per problem) instead of one-shot generation. KernelBench dataset and evaluation protocol: ScalingIntelligence/KernelBench.
Built by RightNow AI.
- AMD ROCm GPU support: MI300X, MI325X, MI350X, MI355X detection and specs (thanks @andyluo7)
- Fixed
verify.pySyntaxError on Python 3.13+ - Fixed CUDA flash_attention ignoring
sm_scaleparameter - Fixed CUDA cross_entropy returning wrong dtype
- Fixed Triton rotary_embedding broadcasting truncation
- Fixed Triton reduce output shape for non-last-dim reductions
- Enhanced profiler:
--export-trace,--memory-snapshot,--torch-compile-logflags - HuggingFace Kernels export via
export_hf.py
- Native CUDA C++ backend with 9 starter kernels (tensor cores, warp intrinsics, shared memory tiling)
- KernelBench integration (250+ standardized GPU kernel problems)
--backend triton|cudaflag forextract.py
- Initial release: Triton kernel optimization pipeline with 5-stage correctness harness
See CHANGELOG.md for full details.
MIT
