LLM Billing Proxy

HTTP proxy that sits between end users and LLM servers and bills them per token.

Requirements

Python >=3.11
Python packages from pyproject.toml (dependencies)

Quick start

python3 -m venv venv
source venv/bin/activate

python3 -m llmproxy

The SQLite schema is created automatically on first run (and by llmproxyctl). You can pre-create it with sqlite3 db.sqlite < llmproxy/schema.sql, but that step is optional; the schema script is idempotent.

Configuration

See llmproxy/config.toml for an example configuration file. The program will update the list of configured backends, the rate limits (including the [files] poll limits) and the [provenance] settings (see AI Act provenance marking) from the config file on SIGHUP. SIGHUP also flushes the authentication cache (see Authentication).

Each backend must define type, which decides the endpoints it is served on:

`type`	Endpoints
`chat`	`/v1/chat/completions`, `/v1/completions`, `/v1/messages`, `/v1/responses`
`embedding`	`/v1/embeddings`
`transcription`	`/v1/audio/transcriptions`
`conversion`	`/v1/files/convert`
`decision`	`/v1/systemone` (native), `/v1/decisions` (OpenAI Decisions subset)

Decision backends

/v1/systemone: native Basal format; types noul, choice, score, multi, act, and tool.
/v1/decisions: supported OpenAI Decisions beta format; types predicate, choice, and score.

Configure type = "decision" and decision_adapter = "basal" (the default). Both endpoints use the configured Basal backend and share authentication, rate limits, billing and metrics.

Use a Python client exposing client.decisions (tested with openai==3.26.0):

from openai import OpenAI

client = OpenAI(
    base_url="YOUR_PROXY_BASE_URL/v1",
    api_key="YOUR_PROXY_API_KEY",
)
result = client.decisions.create(
    model="basal-20-max",
    input="The customer received a damaged product.",
    questions=[{
        "type": "predicate",
        "name": "complaint",
        "instructions": "Is this a complaint?",
    }],
)
print(result.answers[0].probability)

/v1/decisions accepts model, input and questions. Input is a string or user messages with text content or input_text parts. Text parts are joined with newlines, messages with blank lines. Tables can be supplied as text. Images, audio, files, other roles, streaming and empty input are rejected with 400. A state over the model's context gets 422 context_length_exceeded.

Answers preserve question order: predicate returns a probability, choice returns one string/boolean option, and score returns the expected zero-based level index. Choice and score questions require 2-255 unique options/labels. Question and branch limits default to 16 and 64; each supported question uses one branch. Other request parameters are rejected.

Billing uses native input/output token counts. Cache and reasoning detail counters are zero. Invalid backend responses return 502 before billing.

Both endpoints answer with the model alias the client asked for, not the backend's model id. On /v1/systemone a backend 400/413/422/429 (the backend validates native semantics) keeps its status and body; any other backend 4xx (e.g. 401/403 from a wrong backend token) is the proxy's fault and becomes a 502 without the upstream body. /v1/models exposes supported endpoints and decision_capabilities.

A request naming a model on an endpoint of another type is rejected with 404 and code: model_not_supported before inference, rate limiting and billing. An unknown model receives 404 and code: model_not_found. A request without model (absent or null) is malformed: 400 with param: model.

Upgrading: the proxy refuses to start while any backend lacks a valid type (a SIGHUP reload keeps the previous backends; llmproxyctl, which loads the same config, refuses to run as well). Versions without this check ignore the key, so add type to every backend in the deployed config before rolling out the new version.

Each backend may define max_model_len, the real context limit of the deployment in tokens (prompt plus completion). This value is exposed through /v1/models so clients can avoid sending requests that exceed the backend limit. It should match the backend deployment setting (for example vLLM --max-model-len), not just the public model card.

A backend may also define model, the real model id (repo name, e.g. meta-llama/Meta-Llama-3-8B-Instruct) the backend expects in place of the client's model field. This is exposed through /v1/models as model_repo -- (null when unset) -- so clients can inspect which actual model is behind each served model name.

pkill -f -HUP 'python3? .*llmproxy'

# or in Docker
docker kill -s=SIGHUP CONTAINER

File conversion (PDF to Markdown)

POST /v1/files/convert (multipart: file, model, optional output_format) converts a PDF with a conversion backend (the marker microservice) and bills MODEL/DEVICE/conversion per page. A conversion backend must have a non-empty token and device:

[backends.pdf-to-markdown]
url = "https://marker-microservice.example.com"
token = "APP_TOKEN"      # the microservice's APP_TOKEN
device = "a5000"         # billed as pdf-to-markdown/a5000/conversion
type = "conversion"
timeout = 1800           # a synchronous conversion takes minutes
health_path = "healthz"  # the microservice serves /healthz, not /health

By default the request is synchronous: the response arrives when the conversion is done, which can take minutes. With background=true the proxy answers as soon as the job is queued and the client polls for the result, so an agent does not have to hold a connection open for the whole conversion:

curl -s -i https://llm.example.com/v1/files/convert \
  -H "Authorization: Bearer $KEY" \
  -F model=pdf-to-markdown -F background=true -F file=@document.pdf
# HTTP/1.1 202 Accepted
# Location: /v1/files/convert/fc_WyJwZGYt...
# Retry-After: 5
# {"id": "fc_WyJwZGYt...", "object": "file.conversion",
#  "model": "pdf-to-markdown", "status": "queued"}

curl -s https://llm.example.com/v1/files/convert/fc_WyJwZGYt... \
  -H "Authorization: Bearer $KEY"

import time
import requests

API = "https://llm.example.com/v1/files/convert"
AUTH = {"Authorization": "Bearer YOUR-API-SECRET"}

with open("document.pdf", "rb") as f:
    res = requests.post(API, headers=AUTH,
        files={"file": ("document.pdf", f, "application/pdf")},
        data={"model": "pdf-to-markdown", "background": "true"})
res.raise_for_status()
job = res.json()

while job["status"] in ("queued", "in_progress"):
    time.sleep(int(res.headers.get("Retry-After", "5")))
    res = requests.get("%s/%s" % (API, job["id"]), headers=AUTH)
    res.raise_for_status()
    job = res.json()

if job["status"] == "completed":
    print(job["output"])  # same fields as the synchronous response
else:
    print("failed:", job["error"]["message"])

GET /v1/files/convert/{id} answers 200 with status in_progress (with Retry-After), completed (output, output_format, page_count, images, provenance when the backend sends it) or failed (error, not billed). background accepts true/false (or 1/0); anything else is a 400.

Billing: the pages are billed when a completed result is picked up for the first time; later polls return the same result unbilled. Nothing is billed at submission or for a failed conversion. Once the proxy holds a complete, valid result it claims the delivery from the backend, which grants the claim exactly once per job, so this holds across repeated or concurrent polls, proxy restarts and replicas; a fetch that fails or is refused does not use up the first pickup. A result the proxy cannot bill (no valid page_count, a backend without delivery claims) is never handed out: the poll is a 502.
Access: the id is bound to the API key that submitted the conversion; any other key, a modified id, or a job the backend no longer keeps (results expire with the backend's retention, 7 days by default) gets 404 with code: conversion_not_found. The proxy keeps no state: the id carries the backend and its job id, signed with a key derived from the backend's token. Changing that token invalidates the ids of open conversions.
Rate limits: polls do not count against the model's rpm or concurrency limits; only the submission does. For a background submission concurrency bounds only the upload, not the jobs waiting on the backend after the 202: the number of jobs a key can queue is bounded by rpm. Polls have a limit of their own: each API key may have at most 4 polls in flight (more get 429 with Retry-After), and a backend answer over 128 MiB is a 502. Both are set in [files] (max_polls_in_flight, max_poll_response_bytes) and reloaded on SIGHUP. A client that stops reading a result for timeout_read loses the connection (the result was billed; the next poll returns it unbilled).
Upgrading: the backend must support background conversions (file-processor with background and delivery claims). Deploy it before the proxy. With an older file-processor synchronous requests keep working and background=true is a 502 at submission: it either converts synchronously (the work is not billed) or, with MARKER_DATALAB_COMPAT_SYNC=False, queues the job under a content-hash id the proxy refuses.

Monitoring (Prometheus metrics)

The proxy exposes Prometheus-compatible metrics at /metrics by default. This endpoint can be scraped by Prometheus to monitor request volume, error rates, backend latency, and token usage — enabling alerts when traffic spikes or error rates increase.

Configuration

Metrics are controlled by the [metrics] section in config.toml:

[metrics]
enabled = true        # set to false to disable the /metrics endpoint
path = "/metrics"     # path under which metrics are served

Available metrics

Metric	Type	Labels	Description
`llmproxy_requests_total`	Counter	`method`, `path`, `status`	Total HTTP requests received
`llmproxy_request_duration_seconds`	Histogram	`method`, `path`, `status`	End-to-end request latency
`llmproxy_active_requests`	Gauge	—	In-flight requests
`llmproxy_backend_requests_total`	Counter	`model`, `status`	Requests forwarded to backends
`llmproxy_backend_duration_seconds`	Histogram	`model`	Backend response latency
`llmproxy_backend_errors_total`	Counter	`model`, `error_type`	Backend errors (timeout, connection, client_error)
`llmproxy_tokens_total`	Counter	`model`, `type`	Tokens processed (prompt, completion, embedding)
`llmproxy_audio_seconds_total`	Counter	`model`	Seconds of audio transcribed
`llmproxy_file_conversions_total`	Counter	`model`	PDF pages converted to Markdown
`llmproxy_user_requests_total`	Counter	`user`, `model`	Admitted requests per user (api-key comment, else key id) and model
`llmproxy_user_active_requests`	Gauge	`user`, `model`	In-flight requests per user and model — non-zero series are who is connected right now

Quick start (Kubernetes + Prometheus Operator)

If you run Prometheus Operator (kube-prometheus-stack), the full path from deploy to Grafana is:

Deploy the new llmproxy version — the /metrics endpoint is active on port 8080 by default.

Create a ServiceMonitor so Prometheus Operator automatically starts scraping:

# Check your Prometheus selector first:
kubectl get prometheus -A -o jsonpath='{.items[*].spec.serviceMonitorSelector}'

# Apply the ServiceMonitor (adjust the release label to match):
kubectl apply -f k8s/servicemonitor.yaml

If llmproxy has no Kubernetes Service, use the PodMonitor instead:

kubectl apply -f k8s/podmonitor.yaml

Verify Prometheus sees the target:

# Target should appear as "up":
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090
# Open http://localhost:9090/targets → look for "llmproxy"

Import the Grafana dashboard:
- Grafana → Dashboards → New → Import → Upload JSON file
- Select grafana/dashboard.json
- Choose your Prometheus datasource
- The dashboard has 13 panels: request rate, error rate, status codes, active requests, latency percentiles, backend latency/errors, token usage, plus per-user visibility — connected right now (user × model, in-flight), request rate by user, and user × model activity over the last 15 minutes

Prometheus scrape config (standalone, non-Operator)

If you run Prometheus without the Operator, add a scrape job to your prometheus.yml instead:

scrape_configs:
  - job_name: "llmproxy"
    metrics_path: /metrics
    static_configs:
      - targets: ["llm-billing-proxy:8080"]

See examples/prometheus-scrape.yml for a complete example.

Alert rules

Example alert rules for high request volume and high error rates are provided in examples/prometheus-alerts.yml. Key alerts:

LLMProxyHighRequestRate — warning when request rate exceeds 100 req/s
LLMProxyVeryHighRequestRate — critical when request rate exceeds 500 req/s
LLMProxyHighErrorRate — warning when 5xx error rate exceeds 5%
LLMProxyCriticalErrorRate — critical when 5xx error rate exceeds 20%
LLMProxyBackendErrors — per-model backend errors
LLMProxyHighBackendLatency — p95 backend latency above 30s

Grafana dashboard

A pre-built Grafana dashboard is available at grafana/dashboard.json. Import it via Dashboards → New → Import → Upload JSON file.

Output budgets

The optional budget contract exposes forwarded output limits, backend stop signals and reasoning usage through the model catalog, HTTP headers, non-streaming JSON and Prometheus. It is off by default. mode = "observe" leaves budgets untouched; mode = "enforce" independently enables configured defaults and clamping. JSON metadata and headers have separate switches, and config reloads on SIGHUP.

Native reasoning budgeting is verified only for Chat Completions; the tested Qwen/vLLM 0.26.0 backend ignores that parameter on Responses and Messages. The contract documents this limitation and the live E2E evidence.

Testing

python3 -m unittest

Live budget tests are opt-in: set BUDGET_VLLM_URL, BUDGET_VLLM_TOKEN and BUDGET_VLLM_MODEL, then run python3 -m unittest tests.test_e2e_budget -v. The contract documentation describes the backend requirements and bounded generation limits.

Building the image

To build the Docker image, run the following command:

docker build -t ghcr.io/comtegra/llmproxy:master .

Deployment

The following instructions assume that you already have a Docker compose repository and file.

SQLite

Create an SQLite database according to llmproxy/schema.sql. This is optional: the proxy creates the schema automatically on first run. To pre-create it (the script is idempotent):

sqlite3 db.sqlite < llmproxy/schema.sql

Copy llmproxy/config.toml to your repository. A good relative path would be secrets/llm-billing-proxy-config.toml.
Add appropriate entries to your compose file's services and secrets sections. There's a template compose.yml in this repository.
Bring the new service up and verify it started correctly (e.g docker compose up llm-billing-proxy).

MongoDB

Create a MongoDB user with the following privileges (see section Database below for JS snippets):

{resource: {db: "cgc", collection: "api_keys"}, actions: ["find"]}
{resource: {db: "billing", collection: "events_oneoff"}, actions: ["insert"]}

Copy llmproxy/config.toml to your repository. A good relative path would be secrets/llm-billing-proxy-config.toml.
In the config edit uri in section db so that it includes credentials for the Mongo user (e.g. mongodb://myuser:mypass@host:27017/?authSource=cgc).
Add appropriate entries to your compose file's services and secrets sections. There's a template compose.yml in this repository.
Bring the new service up and verify it started correctly (e.g docker compose up llm-billing-proxy).

Testing

Sample data can be added to an SQLite database by using sample.sql. Ditto for MongoDB in sample.js.

After loading sample data you may run a query like this:

curl -v -H'Content-Type: application/json' -H'Authorization: Bearer token2' \
    -d'{"messages": [{"role": "system", "content": "You are an assistant."}, {"role": "user", "content": "Write a limerick about python exceptions"}], "model": "llama31-70b", "stream": true}' \
    'http://localhost:8080/v1/chat/completions'

Database

If using SQLite see llmproxy/schema.sql. Otherwise read on.

This program uses MongoDB for authentication and completion logging. The schema is compatible with Comtergra GPU Core. The MongoDB user needs the following privileges:

db.getSiblingDB("cgc").createRole({
  role: "apiKeysReader",
  privileges: [
    {
      resource: {db: "cgc", collection: "api_keys"},
      actions: ["find"],
    },
  ],
  roles: [],
});
db.getSiblingDB("billing").createRole({
  role: "completionBillingWriter",
  privileges: [
    {
      resource: {db: "billing", collection: "events_oneoff"},
      actions: ["insert"],
    },
  ],
  roles: [],
});

db.getSiblingDB("cgc").createUser({
  user: "llm-billing-proxy",
  pwd: "mypass",
  roles: [
    { role: "completionBillingWriter", db: "cgc" },
    { role: "apiKeysReader", db: "billing" },
  ],
});

Authentication

Users authenticate via bearer tokens. Tokens are stored in the database as SHA256 hashes. They can be generated by the following command:

python3 -c 'import hashlib, secrets; print("Token:", t:=secrets.token_urlsafe(64)); print("Hash:", hashlib.sha256(t.encode()).hexdigest())'

If using SQLite see llmproxy/schema.sql. Otherwise read on.

When a user attempts to authenticate, the following query is performed to the cgc.api_keys collection.

{
    "access_level": "LLM",
    "secret": TOKEN-HASH,
    "$or": [
        {"date_expiry": {"$gt": new Date()}},
        {"date_expiry": null},
    ],
}

TOKEN-HASH is replaced with the SHA256 hash of the bearer token.

The documents are required to have an additional field: user_id.

Authentication cache

Successful key lookups are cached in memory for auth_cache_ttl seconds (default 5, set 0 to disable) so that repeated requests — including ones the proxy is about to reject — do not each cost a database round trip.

Key expiry is not affected: the expiry timestamp is re-checked on every request, so an expired key stops working immediately. What the cache does delay is revocation — deleting or disabling a key can take up to auth_cache_ttl seconds to take effect. Send SIGHUP to flush the cache and make a revocation apply at once.

The llmproxy_auth_cache_hits_total and llmproxy_auth_cache_misses_total counters show whether the cache is doing anything.

Completion logging

If using SQLite see llmproxy/schema.sql. Otherwise read on.

When a user performs a completion, two events (prompt token and completion token counts) are inserted into the collection billing.events_oneoff. These documents have the following fields:

date_created -- date and time when the request finished processing
user_id -- user_id of the API key
api_key_id -- id of the API key
product -- a string in the following format: MODEL/DEVICE/TYPE, where
- MODEL is the name of the backend
- DEVICE is the name of the GPU where the model runs
- TYPE is prompt, completion, embedding, transcription, or conversion (PDF pages for /v1/files/convert)
quantity -- token count
request_id -- request ID to correlate prompt/completion counts

EU AI Act provenance marking (Art. 50(2))

Regulation (EU) 2024/1689 Art. 50(2) requires providers of generative AI systems to mark outputs as artificially generated in a machine-readable format, applicable from 2026-08-02. (The AI Omnibus package, provisionally agreed in May 2026, extends the marking deadline to 2026-12-02 for generative systems already on the market before 2026-08-02 -- verify the final enacted text before relying on the extension.) The proxy marks its generative responses at the point of generation:

Every generative response (streaming and non-streaming) carries the X-AI-Generated: true header.
Non-streaming responses of /v1/chat/completions, /v1/completions, /v1/messages and /v1/responses additionally carry a machine-readable top-level provenance object:

"provenance": {
    "ai_generated": true,
    "digital_source_type": "http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia",
    "legal_basis": "Regulation (EU) 2024/1689, Article 50(2)",
    "generator": "comtegra-llmproxy",
    "generator_version": "1.4.0",
    "model_id": "llama3-8b",
    "request_id": "same UUID as the X-Request-ID header",
    "timestamp": "2026-08-03T12:00:00Z"
}

digital_source_type is the IPTC Digital Source Type vocabulary term for content fully generated by a trained model (the same term C2PA, OpenAI and Meta use), model_id is the public backend alias the client selected and request_id correlates the content with the billing events and logs. Downstream deployers can rely on the field for their own Art. 50(4) disclosure duties and audits; the body field (not the header) is the durable channel -- it flows through SDK response objects and survives persistence. Cross-origin browser clients can read the header too: every response carries Access-Control-Expose-Headers: X-Request-ID, X-AI-Generated.

Endpoint coverage:

/v1/chat/completions, /v1/completions, /v1/messages, /v1/responses -- body field + header. SSE streams get the HEADER ONLY; stream bodies are forwarded byte-for-byte, because injecting synthetic events breaks strict SDK stream parsers.
/v1/audio/transcriptions -- header only. A transcript's semantics come from the input audio (the Art. 50(2) carve-out for output that does not substantially alter the input), and the verbose_json body is forwarded byte-for-byte.
/v1/files/convert -- header only. Converted Markdown is derived from the uploaded PDF; Marker's JSON body (including any provenance object) is forwarded unchanged. The same holds for a completed background conversion picked up through GET /v1/files/convert/{id} (Marker's provenance object is passed through as the provenance field).
/v1/embeddings, /v1/models, error responses -- never marked: vectors, metadata and errors are not synthetic content.

Non-streaming marked bodies are re-serialized (values unchanged; numbers are bit-identical for any double-precision JSON consumer). With logprobs enabled this re-serialization touches every float, which costs some CPU at high request rates.

Official OpenAI/Anthropic SDKs in all languages tolerate and preserve the extra field (e.g. completion.model_extra["provenance"] in openai-python). Should a client with a strict, unknown-field-rejecting parser break, disable the marking without a restart -- the emergency kill switch is:

[provenance]
enabled = false  # then send SIGHUP

The generator string is configurable in the same table. Marking is enabled by default when the [provenance] table is absent.

Name		Name	Last commit message	Last commit date
.agents/skills/deploy		.agents/skills/deploy
.github/workflows		.github/workflows
docs		docs
examples		examples
grafana		grafana
k8s		k8s
llmproxy		llmproxy
scripts		scripts
tests		tests
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
Dockerfile		Dockerfile
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
compose.yaml		compose.yaml
pyproject.toml		pyproject.toml
sample.js		sample.js
sample.sql		sample.sql

Navigation Menu

Repository navigation

About

Repository files navigation

LLM Billing Proxy

Requirements

Quick start

Configuration

Decision backends

File conversion (PDF to Markdown)

Monitoring (Prometheus metrics)

Configuration

Available metrics

Quick start (Kubernetes + Prometheus Operator)

Prometheus scrape config (standalone, non-Operator)

Alert rules

Grafana dashboard

Output budgets

Testing

Building the image

Deployment

SQLite

MongoDB

Testing

Database

Authentication

Authentication cache

Completion logging

EU AI Act provenance marking (Art. 50(2))

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

LLM Billing Proxy

Requirements

Quick start

Configuration

Decision backends

File conversion (PDF to Markdown)

Monitoring (Prometheus metrics)

Configuration

Available metrics

Quick start (Kubernetes + Prometheus Operator)

Prometheus scrape config (standalone, non-Operator)

Alert rules

Grafana dashboard

Output budgets

Testing

Building the image

Deployment

SQLite

MongoDB

Testing

Database

Authentication

Authentication cache

Completion logging

EU AI Act provenance marking (Art. 50(2))

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages