LLM Eval Harness
Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.
Servidor MCP · de razilzul
Resumo
LLM Eval Harness é um servidor MCP publicado por razilzul. Marketplaces que lemos e que o listam: 1 (Official MCP Registry). Vimos pela primeira vez em 2026-10-18 e não registramos mudanças desde então. Conforme nossa passagem pelos marketplaces em 2026-10-22.
O que pede
- Roda na sua máquina
- Segredos: 2
- Configurações: 2
Exige: ANTHROPIC_API_KEY (secreta), GEMINI_API_KEY (secreta)
O que declara, pede e executa, por completo
Instale apenas plugins de desenvolvedores em quem você confia. Mostramos o que os marketplaces publicam e não testamos plugins.
Funciona com
- Claude Code
- Codex
- Cursor
- VS Code
- Copilot CLI
Pelos marketplaces que o listam e pelo transporte MCP que declara, conforme cada cliente documenta. Não testamos plugins.
Conectar
Comandos e configurações gerados apenas a partir do que os marketplaces declaram. Não os executamos por você e não testamos plugins.
Como servidor MCP local: imagem Docker
Gerado a partir do que Official MCP Registry declara.
claude mcp add --env "ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY>" --env "GEMINI_API_KEY=<YOUR_GEMINI_API_KEY>" --transport stdio llm-eval-harness -- docker run -i --rm -e ANTHROPIC_API_KEY -e GEMINI_API_KEY ghcr.io/razilzul/llm-eval-harness:1.0.0[mcp_servers.llm-eval-harness]
command = "docker"
args = ["run", "-i", "--rm", "-e", "ANTHROPIC_API_KEY", "-e", "GEMINI_API_KEY", "ghcr.io/razilzul/llm-eval-harness:1.0.0"]
[mcp_servers.llm-eval-harness.env]
ANTHROPIC_API_KEY = "<YOUR_ANTHROPIC_API_KEY>"
GEMINI_API_KEY = "<YOUR_GEMINI_API_KEY>"{
"mcpServers": {
"llm-eval-harness": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"-e",
"ANTHROPIC_API_KEY",
"-e",
"GEMINI_API_KEY",
"ghcr.io/razilzul/llm-eval-harness:1.0.0"
],
"env": {
"ANTHROPIC_API_KEY": "<YOUR_ANTHROPIC_API_KEY>",
"GEMINI_API_KEY": "<YOUR_GEMINI_API_KEY>"
}
}
}
}docker run -i --rm -e "ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY>" -e "GEMINI_API_KEY=<YOUR_GEMINI_API_KEY>" ghcr.io/razilzul/llm-eval-harness:1.0.0Substitua os valores entre sinais de menor e maior pelos seus. Mostramos os nomes das configurações, nunca valores.
Onde está publicado
| Marketplace | Status | Versão | Números | Verificado | Visto pela primeira vez |
|---|---|---|---|---|---|
| Official MCP Registry (Abrir o registro) | Listado | 1.0.0 | Forks (GitHub): 0 Estrelas (GitHub): 0 |
Sem marcação |
Cada número é do próprio marketplace e nunca é somado entre marketplaces. Um marketplace que não publica números não mostra nenhum: sem número não é zero.
O que mudou
- Passou a ser um produto que acompanhamos
- Official MCP Registry: apareceu
Plugins semelhantes
Produtos cujos próprios nomes, descrições, ferramentas e skills são os mais próximos deste pelo significado, em todos os marketplaces que lemos. A proximidade é ordenada apenas dentro desta lista.
-
Read, measure and explain LLM eval runs of a digline suite. Cannot promote a baseline.
-
Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
-
llmtest-mcp io.github.tjacquesson/llmtest-mcp
Benchmark AI models on real prompts. Find cheaper, faster alternatives across 340+ models.
-
Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading.
-
Decide, design, gate, judge, and defend evaluations for LLM, agent, and retrieval systems. Not ordinary unit tests.
-
Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom…
-
metrillm io.github.MetriLLM/metrillm
Benchmark local LLM models — speed, quality & hardware fitness verdict from any MCP client
-
Read and score Opik traces, experiments, datasets and prompts from your AI assistant.
-
llm-observability-orchestration-agent-langchain-community ai.getvda/llm-observability-orchestration-agent-langchain-community
LLM Observability & Orchestration Agent (LangChain Community)
-
LLM serving benchmarks, SGLang model Day-0 support, profiling, code review, prod incident triage, PR-history dossiers, and SGLang explainer videos.
Plugins semelhantes por marketplace
A mesma pergunta feita a cada marketplace separadamente: existe lá um equivalente deste plugin?
Listado em Claude Code: claude-plugins-community (5)
-
AI agent evaluation sandbox for Claude Code. Holdout scenario testing with Doer/Judge/Adversary/Observer roles, probabilistic satisfaction scoring, and append-only JSONL audit…
-
Synthetic monitoring for LLM inference endpoints. Probe TTFT, latency, throughput, and errors across OpenAI, Anthropic, Google, Azure, Bedrock, and any OpenAI-compatible server.
-
Multi-LLM orchestration MCP server. Delegate tasks to Claude Code, Codex, and Gemini CLIs with session management, retry logic, async job orchestration, and token optimization.
-
Claude model selection advisor — recommends optimal model for each task via shell hook. Zero attack surface.
-
Skills for working with Langfuse, the open-source LLM engineering platform for tracing, prompt management, and evaluation.
Listado em Claude Code: claude-plugins-official (5)
-
Skills for adding DeepEval evaluations, tracing, datasets, Confident AI reports, and iterative improvement loops to AI applications.
-
Skills for working with Langfuse, the open-source LLM engineering platform for tracing, prompt management, and evaluation.
-
Skills for tracing, evaluating, and improving AI agents with MLflow. Supports the full agent improvement loop: instrument → trace → evaluate → iterate → validate.
-
OpenTelemetry observability for Claude Code sessions. Captures tool calls, LLM invocations, token usage, and errors as OTel traces. Send telemetry to Dash0 or any…
-
DataRobot skills for AI/ML workflows — model training, deployment, predictions, feature engineering, monitoring, explainability, data preparation, App Framework CI/CD, and…
Listado em Claude: knowledge-work-plugins (5)
-
Skills for working with Langfuse, the open-source LLM engineering platform for tracing, prompt management, and evaluation.
-
DataRobot skills for AI/ML workflows — model training, deployment, predictions, feature engineering, monitoring, explainability, data preparation, App Framework CI/CD, and…
-
AI-first project auditor and re-engineer based on the 9 design principles and 7 design patterns from the TechWolf AI-First Bootcamp
-
Build multimodal AI applications with Pixeltable -- tables, computed columns, embedding search, UDFs, tool-calling agents, and 25+ AI provider integrations.
-
Agent skills for Qdrant vector search covering scaling, performance optimization, search quality, monitoring, deployment, model migration, version upgrades, and SDK usage across…
Listado em Codex: openai-curated (openai/plugins) (5)
-
Evaluate Codex skills and plugins from chat with a beginner-friendly start command, local-first reports, token budget explanations, and guided benchmarking.
-
Skills for NVIDIAs ecosystem spans GPU acceleration, CUDA, AI agents, inference, robotics, Physical AI, Omniverse, and simulation. This plugin helps you understand the pieces…
-
Build with OpenAI APIs, Agents SDK, and ChatGPT Apps, and create and save OpenAI API keys from Codex.
-
Available for US1 customers only. Analyze, investigate, and act on your Datadog telemetry directly from ChatGPT using natural language. Ask questions about your production…
-
CrowdStrike Falcon Foundry development skills for building cybersecurity applications on the Falcon platform. Includes UI development, collections, functions, workflows, API…
Listado em Community GitHub marketplaces (5)
-
Decide, design, gate, judge, and defend evaluations for LLM, agent, and retrieval systems. Not ordinary unit tests.
-
Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom…
-
LLM serving benchmarks, SGLang model Day-0 support, profiling, code review, prod incident triage, PR-history dossiers, and SGLang explainer videos.
-
Skills for adding DeepEval evaluations, tracing, datasets, Confident AI reports, and iterative improvement loops to AI applications.
-
Scaffolding, running, documenting, and publishing AI evaluations. Ships skills and commands for setting up eval workspaces, creating custom evals (or adapting existing…
Listado em Copilot CLI: github/awesome-copilot (5)
-
Run a lightweight, role-separated AI development team with flexible tools, developer-selected models, proportionate planning, and optional QA.
-
See your local Copilot token usage by day, model, and session, and get tips to use it more efficiently.
-
Three-layer verification pipeline for AI output. Extracts claims, finds sources, and flags hallucination risks so humans can verify before acting.
-
Arize AX platform skills for LLM observability, evaluation, and optimization. Includes trace export, instrumentation, datasets, experiments, evaluators, AI provider integrations…
-
Visualizes live session AI-credit usage as a Pac-Man board with pellets, ghosts, fruit milestones, and game-over limits.
Listado em Copilot CLI: github/copilot-plugins (5)
-
Toolkit for building Microsoft 365 Copilot declarative agents — scaffolding, JSON manifest development, and capability configuration.
-
Build and manage Microsoft Dataverse solutions using natural language. Includes table/column creation, solution lifecycle, data operations, and MCP server configuration.
-
model-apps
Build and deploy Power Apps generative pages for model-driven apps with specialist agents for planning, entity creation, and parallel code generation. Requires PAC CLI >= 2.7.0…
-
workiq
Query Microsoft 365 data with natural language — emails, meetings, documents, Teams messages, and more.
-
workiq-productivity
WorkIQ productivity skills powered by the local WorkIQ CLI — read-only insights across email, meetings, calendar, Teams, SharePoint, projects, and people.
Listado em Cursor: cursor/plugins (5)
-
Fan large tasks out across parallel cloud agents with planners, workers, verifiers, and structured handoffs.
-
Iterative self-referential AI loops using the Ralph Wiggum technique.
-
Inspect and manage feature gates, experiments, dynamic configs, and metrics.
-
Incremental transcript-driven memory updates for AGENTS.md using high-signal bullet points only.
-
Build apps, scripts, and automations with the TypeScript SDK.
Listado em Docker MCP Catalog (5)
-
Read and score Opik traces, experiments, datasets and prompts from your AI assistant.
-
Bridges multiple AI models and CLIs, enabling orchestrated workflows across Claude Code, Gemini CLI, Codex CLI, and other AI development tools.
-
Wayfound’s MCP server allows business users to govern, supervise, and improve AI Agents.
-
Analyze documents, images, videos, and audio.
-
The Official Model Context Protocol (MCP) server for Kagi Search & other tools.
Listado em Official MCP Registry (5)
-
Read, measure and explain LLM eval runs of a digline suite. Cannot promote a baseline.
-
Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
-
llmtest-mcp io.github.tjacquesson/llmtest-mcp
Benchmark AI models on real prompts. Find cheaper, faster alternatives across 340+ models.
-
Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading.
-
metrillm io.github.MetriLLM/metrillm
Benchmark local LLM models — speed, quality & hardware fitness verdict from any MCP client
Listado em npm registry (MCP servers) (5)
-
One workspace for every AI coding assistant. Shared tasks, memory, and governance.
-
Read-only MCP server exposing the dsh-plugin-certification registry for DeepSeek Harness plugins.
-
easytable-mcp io.github.chrischall/easytable-mcp
easyTable restaurant reservations for Claude — availability, and create/modify/cancel bookings
-
Human-in-the-loop for AI agents over MCP elicitation: confirm/form dialogs plus a capability doctor
-
Browser-local and CLI static evidence for deployed AI model artifacts.
Encontrados pelo significado, não por código ou repositório em comum: plugins semelhantes continuam sendo produtos distintos, e isto não é uma avaliação nem uma recomendação.
Quer esta marca no seu plugin em nosso catálogo?
Prove que o mantém com um arquivo no repositório, um arquivo no seu domínio ou um registro DNS, e você poderá mostrar esta marca na ficha, com banners ao vivo para o seu site e o README. Como funciona a verificação
Você também recebe alertas quando um marketplace o publica, o remove, muda a marca de verificado ou o que ele pede, e o endpoint remoto passa a ser monitorado como um serviço web.
Dados de Official MCP Registry. Os números vêm das nossas próprias passagens regulares pelos marketplaces, última leitura em . Não publicamos dados de contato dos editores de plugins.