lumen.
Hardware

00Overview

faisalmumtaz89 on GitHub faisalmumtaz89/Lumen 8

Lumen, the inference engine for your own GPU.

Lumen is a single binary that downloads a model and streams text. Written in Rust, native on Apple Silicon and NVIDIA. No Python, no setup.

Install
curl -fsSL https://servelumen.com/install.sh | bash

01Quick start

Three commands, start to serving.

  1. Install

    One command sets up Lumen for your machine: Metal on Apple Silicon, CUDA on NVIDIA.

    Install
    curl -fsSL https://servelumen.com/install.sh | bash

  2. Run

    Name a model and a prompt. Lumen fetches the model on first use and streams the answer.

    Run
    lumen run qwen3.5-9b "Write a haiku about light"

  3. Serve

    Start the server and point any OpenAI- or Anthropic-compatible tool at it. It just works.

    Serve
    lumen-server qwen3.5-9b

02Why Lumen

Built to owe nothing to anyone's stack.

  • Zero setup

    No Python, no drivers to build against, no config files. The binary handles the model, the conversion, and your GPU by itself.

  • Truly native

    Written from scratch in Rust with its own GPU code for Apple Silicon and NVIDIA, not a wrapper around someone else's engine.

  • Works with your tools

    The built-in server speaks the OpenAI and Anthropic APIs, so anything that talks to them talks to Lumen.

03Benchmarks

Kernel against kernel, nothing else.

Decode throughput on the same model, batch 1, no speculative decoding, every engine measured on the same card in one sitting. Untagged rows run the same weight bytes; a row tagged with a format runs a different quantization of the same model.

      Method: one stream, greedy decode, no speculative decoding, every engine back to back on the same card, five runs after a warmup; speed is decode tokens per second. CUDA cells: Lumen v0.31.0 at defaults, 1,024-token prompt, 128 generated tokens (RTX 5090 with its clock pinned). Rows without a tag run the same weight bytes; a tagged row runs a different quantization of the same model.

      04Platform

      One CUDA build, every modern NVIDIA.

      • Compute capability 8.0+

        Ampere and Hopper: A100, H100, RTX 30- and 40-series. One release binary covers all of them.

      • Driver-only install

        Kernels JIT-compile at runtime via NVRTC. No CUDA toolkit on the machine, the driver is enough.

      • Fits your card

        Peak VRAM: the 9B runs in ~5–18 GB depending on quant; the 30B MoE in ~24–72 GB. Every MoE quant fits a single A100-80GB.

      Native Metal, resident in unified memory.

      • Apple M-series

        The Metal backend is hand-written MSL shaders for Apple GPUs, validated end-to-end on an M3 Ultra.

      • Zero-copy load

        Weights memory-map from disk straight into unified memory. mmap is the default load path on Metal.

      • Convert once

        The same converted model file runs on both backends: pull on your Mac, deploy the identical file to a CUDA box.

      05Server

      An API your tools already speak.

      Boot lumen-server and point any OpenAI or Anthropic client at port 8000.

      POST /v1/chat/completions   OpenAI · SSE
POST /v1/completions        OpenAI
POST /v1/messages           Anthropic · SSE
GET  /v1/models

      • Streaming and tool calls

        SSE streaming on both wire formats, with template-driven tool-call parsing that survives split markers mid-stream.

      • Extended thinking

        A per-request reasoning toggle with its own token budget, separate from max_tokens, so a long trace never starves the answer.

      • Reproducible on demand

        Kernels are deterministic at a fixed seed. Pass seed, or temperature 0, and the same request returns the same tokens.

      06Format

      LBC, our own format.

      Models run from .lbc, Lumen's Layer-Blob Container, built for how inference actually reads weights.

      headermagic · CRC32
      layer index × L
      expert indexMoE only
      layer blob × L128 KiB-aligned
      tokenizerCRC32
      .lbc · v4 · little-endian

      • Self-contained

        Weights, hyperparameters, and the full tokenizer (token table, merges, special tokens, chat template) in one file. No sidecar files to lose.

      • One blob per layer

        Each layer is a contiguous blob at a 128 KiB-aligned offset, so weights memory-map zero-copy and load GPU-resident or stream layer by layer.

      • Per-tensor quantization

        A file-level scheme with per-tensor overrides, so one file can mix BF16, Q8_0, Q4_0, Q4_1 and the K-quants. Conversion can keep every tensor in its source format, so the file streams the same weight bytes the source provides.

      • Reads other engines' checkpoints

        lumen convert --from-hf imports compressed-tensors INT4 checkpoints with values preserved exactly, so Lumen runs the same file vLLM and SGLang serve.

      • Versioned, checksummed

        Header and tokenizer sections carry CRC32s. Readers accept versions up to 4 and reject newer files outright rather than misread them.

      07Model registry

      Name a model. Lumen does the rest.

      • qwen3.5-9b

        9B

        Fast and light. The best place to start.

      • qwen3.5-moe

        30B · MoE

        Mixture-of-experts: big-model answers, small-model speed.

      • qwen3.8-27b

        27B

        The newest generation, supported from day zero.