flash-decoding

Star

Here are 5 public repositories matching this topic...

Yolande1994 / CUDA_GPT2_Inference

Star

纯 CUDA 实现 GPT2 推理，逐步优化算子与架构，最终在单请求下推理速度领先 vLLM 约 1.5 倍。含详细 NCU 分析与 README 优化报告。

cpp cuda transformer cuda-kernels gpt-2 kv-cache gemv llm-inference flash-attention flash-decoding cuda-graphs

Updated Nov 13, 2026
Cuda

8b-is / transformers-ultra

Sponsor

Star

Ultra-High-Throughput Transformers Engine: NVIDIA FP8/TMA, Zero-Allocation Slotted Static KV Cache, Sub-5µs CUDA Graphs, Fused Logits Sampling, Tree Speculative Decoding, Split-KV Flash-Decoding & Apple Silicon Metal (MSL) Simdgroup Kernels

transformers pytorch zero-allocation tma apple-silicon fp8 llm-inference speculative-decoding flash-decoding cuda-graphs metal-msl

Updated Nov 13, 2026
Python

iris465 / llm-decode-cuda

Star

From-scratch C++/CUDA LLM inference engine (SmolLM2-360M) on a GTX 1650 Ti: custom GEMV at 97% of bandwidth, split-KV attention, INT8 weights, cuBLAS autotuning, verified against HuggingFace

performance gpu cuda inference cublas llama quantization int8 llm flash-decoding

Updated Oct 15, 2026
Cuda

JohnScheuer / flash-decoding-bench

Star

Benchmark comparing Flash-Decoding-style chunked KV-parallel attention against standard SDPA at decode time (q_len=1) on RTX 2070, measuring the crossover point where KV-parallelism outperforms fused attention kernels.

benchmarking cuda pytorch transformer decode attention kv-cache long-context llm-inference flash-attention flash-decoding mlsystems

Updated Jul 19, 2026
Python

Sergasgr / CodeAlign-Runtime

Star

C++/CUDA inference engine for Qwen2.5-Coder-0.5B, written from scratch: INT4 GEMV/GEMM kernels, Flash-Decoding, a 24-layer decode engine and lossless speculative decoding. Verified against Hugging Face; benchmarked against bandwidth ceilings, PyTorch and llama.cpp.

benchmarking cpp cuda low-latency cuda-kernels quantization gpu-programming pytorch-extension gemv int4 llm-inference qwen speculative-decoding flash-decoding

Updated Oct 11, 2026
Python

Add this topic to your repo

To associate your repository with the flash-decoding topic, visit your repo's landing page and select "manage topics."

Learn more