A model on your own machine¶

Free, private, and offline once the weights are downloaded. mimem talks to local runtimes through the same adapter it uses for hosted APIs, because Ollama, llama.cpp, LM Studio, vLLM and Jan all serve the OpenAI /chat/completions shape. The only difference is the URL.

Ollama¶

ollama serve
ollama pull llama3.1:8b

Then check mimem can see it:

uv run mimem doctor

  local      configured    ollama on 11434
             -> mimem build paper.pdf --llm --provider local --base-url http://localhost:11434/v1

And run it:

uv run mimem build paper.pdf --out out/paper --llm --provider local --model llama3.1:8b

--base-url is optional when the server is on a default port: --provider local finds it.

The others¶

mimem doctor probes all four ports, so if yours is running it will be found and named.

For anything on a non-standard port, or a server on another machine:

uv run mimem build paper.pdf --llm --provider local --base-url http://192.168.1.9:8080/v1

Which models actually work¶

This is the real question with local models, and it is not about quality. mimem needs typed answers — a GlossOut with a short_def and a long_def, a FigureOut with seven fields and a confidence — and small models are much worse at holding a schema than at writing the sentence inside it.

The adapter negotiates, once, down a ladder:

1. json_schema — the server validates the shape itself. Recent Ollama, vLLM, OpenAI.
2. json_object — the server guarantees valid JSON and nothing more; the schema goes in the prompt. Most local servers.
3. prompted — no server-side support at all; the reply is scanned for the first JSON object.

Below that, it retries twice, handing the model its own validation error. If it still cannot produce the shape, the task degrades — it does not half-parse. A smaller programme is something you can notice; a wrongly-filled one is not.

Rules of thumb, offered as starting points rather than measurements:

• 7–8B instruct models generally manage gloss and compress. They are unreliable on figure, which has seven fields.
• Below 7B, expect frequent degradation. Check the elaboration report rather than assuming.
• Vision needs a multimodal model (llama3.2-vision, qwen2-vl, gemma3). Without one, figures fall back to their captions, which is a documented degradation and not a failure.

mimem doctor --live reports which rung your server settled on and how many schema retries the probe needed. A server needing retries on a one-field ping will struggle with the real tasks.

Expect it to be slow¶

A 98-page review produces dozens of requests, each carrying the source document as context. On a laptop CPU that is minutes to hours; on a GPU with an 8B model, a few minutes. Two things help, and both are on by default:

• the cache — answers are stored by content, so fixing one paragraph re-asks about that paragraph and nothing else;
• --dry-run — see how many calls a document will make before making any.

When it goes wrong¶