I need to test and evaluate agents
Trace, test and score LLM apps and agents before and after they ship.
29 tools · 13 official
Picked from listings that mention evals, evaluation, llm observability, tracing or prompt testing in their name, description, tags or skills; those in Infrastructure & Ops, Code & DevTools or Orchestration, or with API access, rank higher.
Best fits
Official
The ones published by the company behind the product.
Langfuse
Microsoft PromptFlow
OpenLLMetry
Datadog LLM Observability
Arize Phoenix
Google Agents CLI
LangChain / LangSmith
AgentScope
Related tasks
- monitor apps in production44 tools
Query logs, errors, metrics and incidents from your monitoring tools. - run code in a sandbox43 tools
Run code an agent writes in an isolated container or microVM, away from your machine. - deploy and host an app52 tools
Deploy and host apps, APIs, models and MCP servers from an agent or the command line. - query a database46 tools
Ask Postgres, MySQL, MongoDB and other databases questions, run SQL and manage schemas from an agent.
In stacks
Missing a tool that does this? Suggest a listing.
/tasks/evaluate-agents
I need to test and evaluate agents · 29 fit · 13 official · keywords: evals, evaluation, llm observability, tracing, prompt testing · categories: infrastructure, code-devtools, orchestration · access: api
Trace, test and score LLM apps and agents before and after they ship.
best fits
langfuseapi+cliScore traces with custom evaluation rubrics to track quality over timefuture-agiapi+cliEvaluation Suite: Scores outputs for hallucination, relevance, and task success using configurable evaluators across datasets.promptflowapi+cliBatch Evaluation: Executes a flow across a dataset and scores results with groundedness, relevance, and safety evaluators.openllmetryapiWorkflow Tracing: Groups multi-step agent runs into one trace so latency and cost attribute per step.latitudeapi+cli+mcpAutomated Evaluations: Scores sessions with condition sets, LLM judges, or custom JavaScript scripts, and replays regression datasets against fixed…mcpjamapi+cli+mcpCross-Model Evals: Runs test cases with expected tool calls across multiple LLMs and tracks accuracy over time to catch MCP server regressions.llm-spaceAgent Run Tracing: Shows every model call and tool run inside the agent loop as it happens.opikapi+mcp+cliPyTest CI Evaluation: Tests LLM pipelines on every commit through the PyTest integration so evaluation results are recorded per CI run.datadog-llm-observabilityapi+mcp+cliEnd-to-end monitoring for LLM applications — trace prompts, evaluate responses, track costs, and debug AI pipelines in production.heliconeapiSession Tracing: Groups multi-step agent calls into sessions so a whole run can be replayed and debugged as one unit.arize-phoenixapi+cliEvaluation Runs: Runs LLM-as-judge evaluations over captured traces and datasets to quantify quality regressions across versions.google-agents-clicliEvaluation Runs: Runs evaluation suites against an agent and reports quality metrics before it is deployed to production.official
langfuse, promptflow, openllmetry, datadog-llm-observability, arize-phoenix, google-agents-cli, langchain-langsmith, agentscope, genkit, cursor-agent-sdk, nvidia-nemo-agent-toolkit, cloudflare-ai-gateway, vercel-ai-gateway
related tasks
/tasks/monitor-apps-in-production44 fitmonitor apps in production/tasks/run-code-in-a-sandbox43 fitrun code in a sandbox/tasks/deploy-an-app52 fitdeploy and host an app/tasks/query-a-database46 fitquery a databasein stacks
/stacks/agents-in-productionarize-phoenix+heliconeWatch your agents in productionmore
/browse?task=evaluate-agentsall 29, with filters/tasksevery task/api/tasks/evaluate-agentsthis task as JSONGET · no auth