Laya VRAM Requirements: Every Checkpoint, File and Runtime

Can a 16 GB or 24 GB GPU, or a CPU, run Laya?

Yes, all three. The largest published file is 850 MB, and with the default input length the peak is 0.76 GB–1.40 GB for every file, so a 16 GB or 24 GB card uses less than a tenth of its memory, and a 4 GB card is enough. With no GPU the model runs from system RAM, and 3 GB free is enough even for the llama.cpp setup, which loads the model twice. What changes with the hardware is speed, not whether it runs: tens of milliseconds per request on a GPU, a few hundred on a CPU (see the measurements below).

Every Laya file: size and memory

Each checkpoint at its default input length (512 tokens for English, 1,024 for typed-decisions and multilingual), one row per pass. Size is the published file in MB as Hugging Face lists it; memory is in GB (GiB).

Longer inputs and bigger batches

Memory grows with rows × tokens, and the attention scores with the square of the length. Even the heaviest case here, laya-multilingual reading one 8,192-token document, stays under 5 GB.

Three ways to run Laya locally

1. PyTorch: the laya Python package

The reference way, and the only one that runs all three checkpoints and routes each request to the right one. It downloads the safetensors files on first use and runs on CUDA, Apple GPUs or the CPU. PyPI, GitHub.

pip install laya

from laya import Router

router = Router()  # Router(preload=True) loads all three checkpoints up front
result = router.predict(
    "We were billed twice for March. Please refund the duplicate.",
    {"churn_risk": {"type": "noul", "instructions": "Does the user threaten to leave?"}},
)
print(result["answers"]["churn_risk"]["noul"])  # probability of yes

2. ggmlc: the standalone laya binary

A single binary from the ggmlc releases runs the mys/laya-*-GGUF files with no Python: CUDA or Metal when present, else the CPU. It has a CLI, an HTTP server with a web page and a JSON-RPC daemon. These GGUF files are ggmlc's own format: llama.cpp cannot load them.

huggingface-cli download mys/laya-GGUF laya_english_f16.gguf --local-dir .

laya decide laya_english_f16.gguf --preset email --device auto

3. llama.cpp: encoder in llama-server, head in Python

The fr0stbit3/laya-*-gguf files hold only the ModernBERT encoder, which llama-server serves as per-token embeddings. The decision head is the sibling -head.safetensors file and runs in Python with the laya package, which still downloads the original weights once. The model card calls these files lightly tested; prefer F16 or Q8_0.

llama-server -m laya-F16.gguf --embeddings --pooling none -c 2048 -ub 2048 -b 2048 --port 8080

Ollama

Laya is not in the official Ollama library. A community upload, s1gnature/laya, is an embedding model: ollama embed returns the encoder's vectors, not Laya's answers, because the decision head is not part of it. ollama run does not chat with it: Laya does not generate text.

Laya is not LLaVA

The names are close, the models are not. LLaVA is a vision-language chat model (a 7B–34B LLM with an image encoder) that writes text about pictures; its default Ollama downloads are 4.7–20 GB. Laya is a 0.3–0.4B text encoder with a decision head: you give it a text or JSON "state" and typed questions (choice, score, yes/no), and it returns calibrated probabilities in one forward pass. It never generates tokens, reads no images and has no KV cache, which is why none of the LLM calculators on this site apply to it.

Published speed measurements

File sizes read from the Hugging Face API on 2026-09-29; layer counts from each checkpoint's encoder/config.json. The weights are exact; activation memory is an estimate from the architecture (one encoder layer's tensors at a time, in BF16 for PyTorch and F32 for ggml) plus a fixed 0.5 GB for the runtime. Repositories: convaiinnovations/laya, mys/laya-GGUF, fr0stbit3/laya-gguf, convaiinnovations/laya-typed-decisions, mys/laya-typed-decisions-GGUF, fr0stbit3/laya-typed-decisions-gguf, convaiinnovations/laya-multilingual, mys/laya-multilingual-GGUF, fr0stbit3/laya-multilingual-gguf

For chat LLMs, use the LLM VRAM calculator.