lab-bench results
What open models do on hardware we control: one RTX 4090 with 24 GB. BKS-Lab measures this before we recommend a local model to a customer. Each number links to its report, where the method and the limits are written down. Data: bks-lab/lab-bench-results, code and raw rows: github.com/bks-lab/lab-bench.
Capabilities of the machine
Best local model per capability. Where two models are not separated by the paired test, the one with less peak VRAM is shown.
| Capability | Best local model | Quality | Baseline | Speed | Peak VRAM | Report |
|---|
Tool families per task
A hosted decision API (TypeSafe Jev), GLiNER encoders, open LLMs through Ollama and a classic trained classifier on the same tasks. The best arm of each family is picked after the fact on the same data. A cell says nothing about another task. The classic family is trained on the train split of the same dataset, the other three get no training data. Green marks the best of those three zero-shot families only.