Skip to content
OEvortexPublic

About

Codebase indexing and semantic search engine

Resources

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

vortexa   🧠

Codebase indexing and semantic search engine

Dense + sparse hybrid retrieval · AST-aware chunking · LMDB persistence · MCP server

License Python PyPI version PyPI downloads


Table of Contents


Overview

vortexa is a standalone codebase indexing and semantic search engine designed for AI agents and developers. It builds a persistent, hybrid search index over source code using:

  • Dense retrieval via VortexEmbedderV4 (native LF2/LF4 dequant, SIF+PC, Matryoshka) or static embeddings (Model2Vec / SentenceTransformers)
  • Sparse retrieval via BM25 keyword scoring
  • AST-aware chunking that respects function and class boundaries via tree-sitter
  • LMDB-backed storage for fast, persistent vector and chunk storage

The result: natural language code search that understands intent, not just keywords.

results = indexer.search("authentication middleware that validates JWT tokens", top_k=5)
# → Finds the right files even if they use "auth", "verify", "token" instead of "authentication"

vortexa can run as a standalone Python library, be embedded into any agent, or serve as an MCP server for LLM tools.


Features

Semantic search Find code by describing what it does in natural language — no exact-string matching needed.
Hybrid retrieval Combines dense embeddings (semantic meaning) with BM25 (keyword precision) using adaptive alpha weighting.
AST-aware chunking Splits source code at function/class/block boundaries using tree-sitter when available, falls back to line-based splitting.
Incremental indexing Content-hash memoization means only changed files are re-indexed. Full re-index avoids redundant embedding computations.
Persistent storage LMDB-backed vector store survives restarts. Embedding cache avoids recomputing identical content.
Live watch mode Background thread polls for file changes and auto-re-indexes with configurable debounce. The watcher uses the same extension set as the index, so it never loops on files it does not index.
Realtime progress A live progress bar reports file scanning, chunking, model loading, embedding, keyword indexing, and saving — on stderr, so JSON and MCP output stay clean.
MCP server Expose search, resolve, explain, status, and reindex tools for MCP-compatible agents (Claude Code, Cursor, etc.)
Zero mandatory heavy deps Core requires only numpy, lmdb, and pathspec. Model2Vec and tree-sitter are optional extras.

Quick Start

Installation

# Core (BM25 + line-based chunking)
pip install vortexa

# Full (Model2Vec embeddings + tree-sitter AST chunking)
pip install "vortexa[full]"

# With MCP server support
pip install "vortexa[mcp]"

Index a codebase

from vortexa.core.indexer import CodebaseIndexer

indexer = CodebaseIndexer(root=".")
stats = indexer.index()

print(f"Indexed {stats.indexed_files} files, {stats.total_chunks} chunks")
print(f"Languages detected: {stats.languages}")

Search with natural language

results = indexer.search("CSV parser implementation", top_k=5)

for r in results:
    print(f"{r.chunk.file_path}:{r.chunk.start_line}  score={r.score:.3f}")
    print(f"  {r.chunk.content[:150].strip()}")
    print()

Output:

src/parsers/csv_parser.py:42  score=0.892
  def parse_csv(filepath: str, delimiter: str = ",") -> list[dict]:
      """Parse a CSV file into a list of dictionaries."""
      with open(filepath, "r") as f:

tests/test_csv_parser.py:15  score=0.756
  def test_parse_csv_with_header():
      result = parse_csv("test.csv")
      assert len(result) == 3

Python API

Indexing

from vortexa.core.indexer import CodebaseIndexer
from vortexa.core.types import ChunkConfig

# Default chunking (aim for 50-line chunks, 5-line overlap)
indexer = CodebaseIndexer(root="/path/to/project")
stats = indexer.index()
# → IndexStats(indexed_files=127, total_chunks=843, languages={"python": 45, "typescript": 32, ...})

# Custom chunk configuration
indexer = CodebaseIndexer(
    root=".",
    chunk_config=ChunkConfig(chunk_size=100, chunk_overlap=10),
)
stats = indexer.index(force=False, include_text_files=True)

# Force full re-index
stats = indexer.index(force=True)

Track indexing progress in your own UI with the progress callback:

from vortexa.core.progress import IndexProgress

def on_progress(event: IndexProgress) -> None:
    print(f"{event.phase}: {event.current}/{event.total} {event.message}")

stats = indexer.index(progress=on_progress)

Searching

# Hybrid search (auto-weighted semantic + BM25)
results = indexer.search("error handling", top_k=10)

# Pure semantic search
results = indexer.search("database connection pool", top_k=5, alpha=1.0)

# Pure BM25 keyword search
results = indexer.search("parse csv", top_k=5, alpha=0.0)

# Symbol lookup (find definitions by name)
results = indexer.find_symbol("ConnectionPool", top_k=5)

# Related chunks (find chunks similar to a given chunk index)
results = indexer.find_related(chunk_idx=3, top_k=5)

Each result is a SearchResult with:

Field Type Description
chunk.file_path str Relative file path
chunk.start_line int Start line number
chunk.end_line int End line number
chunk.content str Code snippet (up to 500 chars)
chunk.language str Detected programming language
chunk.lineage Lineage Source path + byte offsets
chunk.chunk_hash str Content hash for memoization
score float Relevance score (0–1)
source str "semantic", "bm25", or "hybrid"

Watch Mode

from vortexa.interfaces.watcher import IndexWatcher

watcher = IndexWatcher(indexer, poll_interval=3.0)
watcher.start()   # Background thread, polls every 3s, debounces 2s
# ... files change on disk, auto-re-index happens ...
watcher.stop()

Management

# Index statistics
stats = indexer.stats()
# → {indexed_files: 127, total_chunks: 843, languages: {...}, memo_hits: 42, memo_misses: 15}

# Reset
indexer.clear()   # Delete the persistent index

CLI Indexing and Status

Build or update the index without running a search. A realtime progress bar is drawn on stderr; stdout stays machine-readable JSON:

# Build or incrementally update the index (default: current directory)
vortexa index

# Index a specific repository and include Markdown/JSON/YAML
vortexa index /path/to/project --include-text

# Rebuild from scratch
vortexa index --force

# Check whether the index is ready (does not build anything)
vortexa status

# Human-readable status instead of JSON
vortexa status --plain

Example progress output:

[vortexa] Indexing /path/to/project
[vortexa] Scanning files: 128/210 (60%) ETA 1s — src/auth/middleware.py
[vortexa] Embedding chunks: 512/940 (54%) ETA 4s
[vortexa] ✓ 210 files · 940 chunks · 64 reused · 876 embedded (6.42s)

vortexa index prints JSON statistics by default:

{
  "root": "/path/to/project",
  "model": "VTXAI/vtx-embed-7M-lf2",
  "indexed_files": 210,
  "total_chunks": 940,
  "languages": {"python": 142, "typescript": 68},
  "changed_files": 12,
  "removed_files": 1,
  "memo_hits": 64,
  "embedded_chunks": 876,
  "up_to_date": false,
  "index_time_ms": 6418.2,
  "errors": []
}

Progress is suppressed with --quiet (both progress and summaries) or --no-progress (bar only). Both flags are also available on search, resolve, and explain.


CLI Search

The installed vortexa command can also search a codebase directly:

# Search the current working directory
vortexa -q "authentication middleware that validates JWT tokens"

# Search a specific codebase root
vortexa -q "CSV parser implementation" /path/to/project

# Pass Kilo-style environment details; `Working directory` is used as the root
vortexa -q "error handling" "Working directory: /path/to/project
Workspace root folder: /"

Useful flags:

Flag Description
-q, --query Search query. Quote multi-word queries.
--root Codebase root to index and search. Overrides environment_details.
--top-k Maximum number of results to return. Default: 10.
--alpha Semantic weight from 0.0 to 1.0; defaults to adaptive weighting.
--min-score Drop results below this score. Zero-score noise is always removed.
--include-text Include text files such as .md, .json, and .yaml in the index.
--force Force a full re-index before searching.
--no-index Search the existing index only.
--quiet Suppress progress and index status messages on stderr.
--no-progress Hide the realtime indexing progress bar.
--plain Print human-readable results instead of JSON.
--model Embedding model ID or alias (mini, nano; both use LF2 by default). Use mini-full or nano-full for LF4.
-f, --fast Explicitly select LF2 for a full-model alias. The mini and nano aliases already use LF2.

By default CLI output is JSON:

[
  {
    "file": "src/auth/middleware.py",
    "lines": "12-48",
    "score": 0.892,
    "source": "hybrid",
    "content": "def validate_jwt(token: str) -> User: ..."
  }
]

The vortexa command still starts the MCP server when no query is provided. You can also start the server explicitly:

vortexa serve
# or
vortexa-serve

Model Configuration

vortexa supports configurable embedding models. The default is mini (VTXAI/vtx-embed-7M-lf2), using the native LF2 2-bit checkpoint. Use mini-full or nano-full to select the original LF4 checkpoints.

Available Models

Alias Model ID Description
mini VTXAI/vtx-embed-7M-lf2 Default native 2-bit 7M model. 256-dimensional embeddings.
nano VTXAI/vtx-embed-1M-lf2 Default native 2-bit 1M model. 64-dimensional embeddings.
mini-full VTXAI/vtx-embed-7M Original LF4 checkpoint for maximum quality.
nano-full VTXAI/vtx-embed-1M Original LF4 lightweight checkpoint.

CLI Usage

# Use the nano model (smaller, faster)
vortexa -q "authentication" --model nano /path/to/project

# Use mini explicitly (same as default)
vortexa -q "authentication" --model mini /path/to/project

# Use a custom HuggingFace model ID
vortexa -q "authentication" --model VTXAI/vtx-embed-1M /path/to/project

Python API

from vortexa.core.indexer import CodebaseIndexer

# Use nano model (LF2 by default)
indexer = CodebaseIndexer(root="/path/to/project", model_id="nano")

# Use mini model (LF2 by default)
indexer = CodebaseIndexer(root="/path/to/project", model_id="mini")

# Opt into the original LF4 checkpoint
indexer = CodebaseIndexer(root="/path/to/project", model_id="mini-full")

Alternative Embedders

The embedding.py module also provides SentenceTransformerEmbedder for transformer-based models (requires the sentence-transformers package).


Inference API

vortexa can be used as a standalone embedding inference engine for VTXAI models, similar to how sentence-transformers works but purpose-built for Vortex-Embed models.

Python API

from vortexa.core.inference import embed

# Encode a single string (default: mini model)
vec = embed("India is a diverse country")

# Encode multiple strings with the nano model
vecs = embed(["Indian cricket team is strong", "Chennai is a major city"], model="nano")

# Use any HuggingFace model ID
vecs = embed(["Indian agriculture output"], model="VTXAI/vtx-embed-7M")

# Shape: (1, D) for single text, (N, D) for batch
print(vec.shape)

CLI

# Encode text with the mini model (default)
vortexa embed "India is a diverse country"

# Encode with the nano model
vortexa embed "Indian cricket team is strong" --model nano

# Encode multiple strings
vortexa embed "India has 28 states" "Chennai is in Tamil Nadu" --model nano

# Truncate embeddings with Matryoshka
vortexa embed "India has 28 states" --model nano --dim 64

# Read one text string per line from a file
vortexa embed --file queries.txt --model mini

# Use a custom model ID
vortexa embed "Indian monsoon patterns" --model VTXAI/vtx-embed-7M

MCP Server

# Auto-indexes current directory, serves on stdio
python -m vortexa.interfaces.mcp_server

# Or via the installed entry point
vortexa serve

On startup it indexes the current working directory and reports realtime progress on stderr:

[vortexa] Indexing /projects/my-app
[vortexa] Scanning files: 127/127 (100%) — src/main.py
[vortexa] Embedding chunks: 843/843 (100%) ETA 0ms
[vortexa] ✓ 127 files · 843 chunks · 0 reused · 843 embedded (2.41s)
[vortexa] Ready: 127 files, 843 chunks in 2410ms
[vortexa] Auto-reindex watcher started (polling every 3s)

The server exposes five tools:

Tool Description Arguments
search Semantic + BM25 hybrid code search query (str), top_k (int, default 10), min_score (float, default 0.0)
resolve Feature-level search with graph context, tests, imports, callers, and callees query (str), top_k (int, default 5)
explain Explain a file, file:line location, or symbol location (str)
status Check whether the index is ready, still indexing, empty, or failed none
reindex Incrementally update or force-rebuild the index with MCP progress notifications force (bool), include_text (bool or null)

Use search for discovery, resolve when you need feature context, and explain when the file or symbol is already known. Call status before relying on results while startup indexing may still be running, and use reindex only after large external changes. The server indexes the current directory and auto-reindexes it while running.

You can also start the server for a specific root without changing directory:

vortexa serve --root /path/to/project --include-text
vortexa serve --root /path/to/project --no-watch

Usage with Claude Code / Cursor

Add to your MCP configuration file (~/.cursor/mcp.json or Claude Code's mcp_servers config):

{
  "mcpServers": {
    "vortexa": {
      "command": "python",
      "args": ["-m", "vortexa.interfaces.mcp_server"],
      "cwd": "/path/to/your/project"
    }
  }
}

The agent will now have access to semantic code search — it can find functions, classes, and patterns by describing them in natural language. This is significantly more effective than grep or rg for exploratory queries.


Agent Skill

Vortexa ships with an Agent Skills-compatible skill at skills/vortexa/SKILL.md. It teaches compatible agents when to prefer Vortexa, how to choose between search, resolve, and explain, and how to interpret returned file paths, line ranges, scores, and graph context.

The skill follows the standard SKILL.md structure: YAML frontmatter with a lowercase hyphenated name and trigger-oriented description, followed by workflow instructions. Copy or link the skills/vortexa directory into the skills directory used by your agent environment.

---
name: vortexa
description: Use Vortexa to index and semantically search local codebases...
---

Use the installed MCP server when available; otherwise the same workflow is available through the vortexa search, vortexa resolve, and vortexa explain commands.


Architecture

Directory Layout

vortexa/
├── core/
│   ├── indexer.py       # CodebaseIndexer — main orchestrator
│   ├── chunking.py      # AST-aware (tree-sitter) + line-based chunking
│   ├── embedding.py     # Embedding model wrappers (Model2Vec, SentenceTransformers, LF4)
│   ├── language.py      # Language detection & file extension mapping
│   ├── progress.py      # UI-agnostic indexing progress events
│   └── types.py         # Shared types (Chunk, ChunkConfig, IndexStats, SearchResult, ...)
├── storage/
│   ├── vector_store.py  # LMDB-backed persistent vector store
│   ├── bm25.py          # BM25 keyword index with persistent storage
│   └── walker.py        # File system walker with .gitignore support
├── search/
│   ├── search.py        # Hybrid search orchestrator (dense + sparse)
│   ├── ranking.py       # Result ranking & symbol query detection
│   └── tokens.py        # Identifier tokenization (camelCase, snake_case)
└── interfaces/
    ├── cli.py           # Command-line index/search/status entrypoint
    ├── mcp_server.py    # MCP server (stdio transport)
    ├── progress.py      # Realtime terminal/MCP progress bar renderer
    └── watcher.py       # Live file poller with debounced auto-reindex

Data Flow

sequenceDiagram
    participant User as User Code
    participant Indexer as CodebaseIndexer
    participant Walker as File Walker
    participant Chunker as Chunking Engine
    participant Embedder as Embedding Model
    participant Store as LMDB Vector Store
    participant BM25 as BM25 Index
    participant Search as Search Engine

    User->>Indexer: index()
    Indexer->>Walker: walk_files(root, extensions)
    Walker-->>Indexer: file_paths
    loop Each file
        Indexer->>Chunker: chunk_source(source, language)
        Chunker-->>Indexer: list[Chunk]
        Indexer->>Embedder: embed(chunks)
        Embedder-->>Indexer: vectors
        Indexer->>Store: store(vectors, chunks)
        Indexer->>BM25: index(chunks)
    end
    Indexer-->>User: IndexStats

    User->>Search: search(query)
    Search->>Store: query(vector)
    Search->>BM25: query(tokens)
    Search->>Search: hybrid_fusion(results)
    Search-->>User: list[SearchResult]
Loading

Indexing Pipeline

graph LR
    A[Source Files] --> B[File Walker<br/>.gitignore aware]
    B --> C[Language Detector]
    C --> D{AST Available?}
    D -->|Yes| E[Tree-sitter Parser<br/>Function/class boundaries]
    D -->|No| F[Line-based Splitter<br/>Configurable size/overlap]
    E --> G[Chunk Set]
    F --> G
    G --> H[Embedding Model<br/>VortexEmbedderV4 (LF4) / Model2Vec / SentenceTransformer]
    G --> I[BM25 Tokenizer]
    H --> J[(LMDB Vector Store)]
    I --> K[(BM25 Index)]
    J --> L[Content Hash Memo]
    K --> L
    L --> M[Skip unchanged files]
Loading

Module Dependencies

graph TD
    subgraph "Public API"
        Indexer["core.indexer<br/>CodebaseIndexer"]
        Search["search.search<br/>search_hybrid()"]
    end

    subgraph "Core"
        Chunking["core.chunking<br/>chunk_source()"]
        Embedding["core.embedding<br/>Embedder"]
        Language["core.language<br/>detect_language()"]
        Types["core.types<br/>Chunk, ChunkConfig, ..."]
    end

    subgraph "Storage"
        VectorStore["storage.vector_store<br/>LMDB Vector Store"]
        BM25["storage.bm25<br/>BM25 Index"]
        Walker["storage.walker<br/>walk_files()"]
    end

    subgraph "Interfaces"
        CLI["interfaces.cli<br/>Command-line search"]
        MCP["interfaces.mcp_server<br/>FastMCP server"]
        Watcher["interfaces.watcher<br/>IndexWatcher"]
    end

    Indexer --> Chunking
    Indexer --> Embedding
    Indexer --> Language
    Indexer --> Types
    Indexer --> VectorStore
    Indexer --> BM25
    Indexer --> Walker
    Indexer --> Search

    Search --> Embedding
    Search --> VectorStore
    Search --> BM25
    Search --> Types

    CLI --> Indexer
    MCP --> Indexer
    MCP --> Watcher
    Watcher --> Walker
Loading

Dependencies

Package Required Used For
numpy Yes Vector operations, embedding inference
lmdb Yes Persistent vector and chunk metadata storage
bm25s Yes Fast BM25 keyword index and persistence
pathspec Yes .gitignore pattern matching in file walker
model2vec Optional Alternative static embeddings
huggingface-hub Yes (default model) Loading LF2 defaults (VTXAI/vtx-embed-7M-lf2, VTXAI/vtx-embed-1M-lf2)
tokenizers Yes (default model) HF tokenizer for embedding model
safetensors Yes (default model) Safe tensor loading for 4-bit weights
sentence-transformers Optional Transformer-based dense embeddings
model2vec Optional Alternative static embeddings
tree-sitter-language-pack Optional AST-aware code chunking
fastmcp Optional MCP server for LLM tool integration

Install optional groups:

pip install "vortexa[full]"     # model2vec + sentence-transformers + tree-sitter
pip install "vortexa[full, mcp]" # everything including MCP server

License

Copyright 2025 VortexAI

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

About

Codebase indexing and semantic search engine

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages