Skip to content

Redrob Eval

Available on GitHub

Compare models
on your own test sets.

An open LLM evaluation workbench in three modules: Compare, Evolve, Deploy. Frontier APIs, OpenRouter, and your own vLLM endpoints sit in one list. Relative cost, never absolute currency.

Clone on GitHub

Compare

The front door. Run any model from any source live on a catalog dataset or your own prompts, on text or image, then settle the unscored tasks with a blind preference bracket.

  • Curated, OpenRouter, frontier, and self-hosted vLLM in one list
  • Quality only where references exist; latency, TTFT, throughput always measured
  • One blind single-elimination bracket per prompt, votes become routing labels
Compare

Evolve

GEPA search over instruction, demos, model, and policies under an explicit quality floor. Catalog datasets, or your own goal and rubric scored by an LLM judge or checklist QWK.

  • Baseline vs evolved report with a Pareto chart you can regenerate offline
  • Train and val are fair game; test is reported once and refused if reused
  • Tokenizer fertility caps demos: watch demos_requested vs demos_fitted
Evolve

Deploy

Serve self-hosted models on your own GPU host over SSH: measure, start, health, benchmark, resumable terminal. Then put the served endpoint back in Compare against the frontier.

  • Self-hosted relative cost comes from measured throughput, not published pricing
  • FP8 vs bf16 always appears in result caveats
  • GPU host config stays in a gitignored root .env
Deploy
Compare

And much more

Every module, and the rules we hold ourselves to, in one list.

Compare

Any model from any source, live on your own prompts

Evolve

GEPA search under an explicit quality floor

Deploy

Serve self-hosted models on your own GPU over SSH

Blind preference

One single-elimination bracket per prompt, names hidden

Preference-derived routes

Human votes feed the same routing corpus as metrics

Text and image

Modalities in one run path, audio slots in as one more

Relative cost

Baseline percentages only, never absolute currency

One-shot test sets

Report once; no silent reuse of the same slice

Server-side keys

Provider keys stay on your own server

Apache-2.0

Clone, remix, and ship without asking permission

Web workbench

UI and harness in one open repo

Published limits

What failed is written down next to what worked

Public clone path

GitHub first: this hub exists to get you into the repo

Get started

Clone the public repo. Run it locally. Inspect the method.

01

Clone and configure

git clone the public repo, copy .env.example to .env, and set a provider key.

02

Install & run

yarn install && yarn dev, then open localhost:3939. Keys stay on the server.

03

Compare, evolve, deploy

Pick a model in Compare, evolve it under a quality floor, then deploy what you chose.

Clone on GitHub
git clone https://github.com/redrob-labs/redrob-eval.git && cd redrob-eval && yarn install && yarn dev

Common questions

License, limits, and how the path actually runs. Claims travel with method and caveat.