About this template
Serve a model. Measure it. Stop guessing at latency.
If you need an OpenAI-compatible serve shape and honest p50/p99 numbers — not a screenshot of somebody else's benchmark — this OpenScience lab is already laid out for serve and bench work.
Numbers, written to disk.
What the vLLM-style serve and bench loop covers
Two skills, one workflow.
ml-inference-serve designs the serving stack: the OpenAI-compatible endpoint shape, the client config, the runbook. ml-inference-bench measures it — latency and throughput, reported as p50 and p99 instead of "feels fast." Results land under a fixed layout, which is what makes two runs comparable a week apart.
Design first. Then measure.
Worth saying up front: the fork ships those two skills and the directory layout, not a running server. ml-inference-serve designs the stack — installing vLLM, or whatever you serve with, is still your step.
What lands in your fork
- The OpenScience ml agent as the default for inference work
ml-inference-serve— designs inference serving stacksml-inference-bench— latency and throughput benches- Layout:
~/ml-inference-lab/{serve,bench,clients,runbooks}/
What you supply (BYOK)
A model key, plus a Hugging Face token if you pull weights.
export ANTHROPIC_API_KEY=sk-ant-...
# export HF_TOKEN=... # optional, for Hugging Face pulls
OPENAI_API_KEY, GEMINI_API_KEY and OPENROUTER_API_KEY are read too. Keys never ship in the snapshot.
Your first five minutes
Fork, then click Open OpenScience.
Ask the ml agent for the whole thing at once:
Serve this model with an OpenAI-compatible API and run a latency/throughput bench. Report p50 and p99.
You should land configs and results under the lab layout — a file you can diff next month, not a wall of unmeasured advice.
Honest limits
Read this part before you fork.
No GPU is attached to the VM, so you bench the endpoints and hardware you point it at, and GPU cost stays yours if you attach any. This is a lab on your own VM, not a managed multi-region inference product. If what you want is literature review and experiment scaffolding, take the ML co-scientist OpenScience template instead.
30-day money-back on your first jurniti purchase.
Fork it. Serve. Bench. Read the numbers.