gqa

Train Scene Graph Generation for Visual Genome and GQA in PyTorch >= 1.2 with improved zero and few-shot generalization [BMVC 2020, ICCV 2021]

Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.

249M-param MoE transformer built from scratch in PyTorch. GQA, RoPE, SwiGLU, sparse MoE with 3 aux losses, AMP training loop no Trainer abstractions. Architecture mirrors LLaMA/Mistral/Mixtral decisions, fully inspectable.

PyTorch DataLoader for many VQA datasets

A PyTorch implementation of the Hyena-Y model, a convolution-based multi-hybrid architecture optimized for edge devices.

Predicting a subgraph alongside the answer in a graph based VQA model

MHA, MQA, GQA, MLA 相关原理及简要实现

Vision-Language, Solve GQA(Visual Reasoning in the Real World) dataset.

World's First 250M Small Language Model with a Native NVMe Hardware Memory Bus. 100% Factual Retrieval, 100% Math Precision, #1 Composite Efficiency (0.40). Architect: Yadlapalli Avinash Ricky.

Build an LLM inference engine from scratch in pure C

A RAG-based question-answering system that processes user queries using local documents. It extracts relevant information to answer questions, falling back to a large language model when local sources are insufficient, ensuring accurate and contextual responses.

[GCPR 2023] Zero-shot Translation of Attention Patterns in VQA Models to Natural Language

CPU-first long-context KV memory optimization with deterministic page regeneration and exact folded attention.

Modular, step-by-step implementation of GPT & Modern LLMs from scratch in PyTorch. Featuring RoPE, RMSNorm, SwiGLU, GQA, KV-Cache, Unit Tests, and an Interactive Web Playground.

A character-level GPT model built from scratch in PyTorch, exclusively trained to generate text in the style of Shakespeare. Features a modular RAG pipeline.

A code deep-dive on one of the key innovations from Deepseek - Multihead Latent Attention (MLA)

Custom CUDA kernels for KV-cache eviction + INT8 quantized paged attention (vLLM-oriented), PyTorch C++ extensions with Python API: eviction @16k blocks 1707.75us host -> 562us fused (~3x); INT8 attention p50 1.76-14.18ms; ~49.9% KV memory vs FP16; 26/26 pytest passing on T1000.

Reproduction study of KV-cache eviction methods on small GQA models. Retaining a fact is not sufficient for using it.

LLM pretraining from scratch on FineWeb dataset (architecture and all components explained), plus optimal use of GPU on SLURM cluster

This repository contains an implementation of Group Query Attention (GQA), an efficient variant of multi-head attention used in modern transformer models like LLaMA.