run 2026-10-17 · 2026-10-17

MCP tool-routing benchmark

Name: MCP tool-routing benchmark
Creator: Glimind
License: https://creativecommons.org/licenses/by/4.0/

On 50 everyday agent tasks, Glimind picks the right MCP tool 47 times (94% exact, 94% weighted), versus 43 for embedding nearest neighbour and 32 for keyword search over the same live catalog, graded by the same LLM judge.

Glimind route_task

47/50

94% weighted · p50 928 ms

Embedding nearest neighbour

43/50

87% weighted · p50 498 ms

Keyword search

32/50

74% weighted · p50 761 ms

Methods

Glimind route_task — Hybrid semantic + keyword retrieval over live tools, operation/entity-aware reranking, first-party preference, reliability and readiness tie-breaks.
Embedding nearest neighbour — The single nearest tool to the task embedding over the same live catalog — typical 'semantic tool search'.
Keyword search — Postgres full-text rank of the task words over tool names and descriptions — typical registry search.

Every task and pick

Judge

You grade a tool router. Given a TASK and the TOOL it chose (name, server, description), answer how well the tool accomplishes the task FOR A GENERAL USER.
2 = it directly does the task in general (official or general-purpose tool for exactly this).
1 = partially/indirectly (related tool, narrower scope, needs extra steps, or scoped to one vendor's product/data).
0 = wrong tool or does not do the task.
Respond ONLY with JSON {"score":0|1|2,"why":"<=12 words"}.

Reproduce: node --import tsx scripts/benchmark.mts <runId> — routes each task three ways over the live catalog and grades every pick with the same judge prompt (temperature 0). Data: https://glimind.com/v1/routing-benchmark (CC-BY 4.0).

Task	Glimind route_task	Embedding nearest neighbour	Keyword search
send a slack message to a channel	2 `io.github.pipeworx-io/slack_connect/slack_send_message` Directly sends Slack messages to channels.	2 `io.github.pipeworx-io/slack_connect/slack_send_message` Directly sends Slack messages to channels.	1 `com.youspot/youspot/draft_slack_message` Drafts message but doesn't send; needs extra step.
post a message in a Slack channel	2 `com.youspot/youspot/send_slack_message` Directly posts messages to Slack channels.	2 `com.youspot/youspot/send_slack_message` Directly posts messages to Slack channels.	1 `com.youspot/youspot/draft_slack_message` Drafts message but doesn't post; user must manually send.
create a GitHub issue in a repository	2 `io.github.github/github-mcp-server/issue_write` Directly creates GitHub issues as described.	2 `in.vynix/vynix-mcp/create_github_issue` Directly creates GitHub issues as described.	2 `com.ecomdly/ecomdly/github_create_issue` Directly creates GitHub issues as described.
list open pull requests on github	2 `io.github.github/github-mcp-server/list_pull_requests` Directly lists GitHub pull requests.	2 `com.gluecron/gluecron/gluecron_list_prs` Directly lists open PRs on GitHub repos.	0 `com.astranl/crossing/find_bounties` Tool finds bounties, not general PRs.
create a jira ticket	2 `com.atlassian/atlassian-mcp-server/jira_create_issue` Directly creates Jira tickets as intended.	2 `io.github.tlkc888-jenkins/joy/create_ticket` Directly creates a ticket for issue tracking.	0 `ai.zerowidth/compass/compass_changes_upsert` Creates changes, not Jira tickets.
search Confluence pages	2 `io.github.pipeworx-io/confluence/confluence_search` Directly searches Confluence pages by keyword.	2 `io.github.pipeworx-io/confluence/confluence_search` Directly searches Confluence pages by keyword.	2 `io.github.pipeworx-io/confluence/confluence_search` Directly searches Confluence pages by keyword.
list my linear issues	2 `io.github.pipeworx-io/linear/linear_list_issues` Directly lists Linear issues for the user.	2 `io.github.pipeworx-io/linear/linear_list_issues` Directly lists Linear issues for the user.	2 `io.github.pipeworx-io/linear/linear_list_issues` Directly lists Linear issues for the user.
create a stripe payment link	2 `io.github.mifactory-bot/mifactory-payments/create_payment_link` Directly creates Stripe payment links.	0 `app.frankki/letters/wallet_topup_link` Creates top-up link, not payment link.	0 `me.faxer/faxer/create_fax_payment` Creates fax payment link, not general Stripe payment link
refund a stripe charge	2 `io.github.sella-network/sella/charge_refund` Directly refunds a Stripe charge.	1 `io.agentlot/marketplace/cancel_reserved_payment` Cancels uncaptured auth, not general refund.	1 `ai.bankee/inferventis-mcp/stripe_payment_records` Retrieves records but doesn't process refunds.
deploy my app to vercel	2 `com.tldmath/tldmath/plan_app_move` Directly plans deploying apps to Vercel.	2 `io.github.sirarchiemccarthy-oss/agentbuilders/deploy_app` Directly deploys apps, including to Vercel-like platforms.	0 `ai.facesign/facesign-mcp/export_app` FaceSign-specific tool, not for general Vercel deployment
get Cloudflare DNS records	2 `com.datanexusmcp/mcp-server/domain_fetch_dns_records` Directly fetches DNS records via Cloudflare DNS.	2 `holdings.proof/mcp-server/connect_cloudflare` Directly manages Cloudflare DNS records.	0 `com.inboxguardstest.api/email/email_fix_plan` Wrong tool; email fix plan, not DNS records.
query a supabase table	0 `io.github.mcp-dir/supabase-mcp/supabase_search_docs` Searches docs, not the actual database table.	0 `io.github.mcp-dir/supabase-mcp/supabase_search_docs` Searches docs, not the actual database table.	1 `io.github.pipeworx-io/stat-gl/query_table` Similar concept but scoped to PxWeb/Statistics Greenland, not general SQL.
list figma files	0 `io.github.pipeworx-io/figma/get_file` Lists file details, not files.	0 `io.github.wolfe-jam/rust-faf-mcp/faf_read` Tool parses .faf files, not lists Figma files.	0 `io.github.pipeworx-io/figma/get_file` Lists file details, not files.
search my Notion workspace	2 `io.github.pipeworx-io/notion_connect/notion_search` Directly searches Notion workspace by keyword.	2 `io.github.pipeworx-io/notion_connect/notion_search` Directly searches Notion workspace by keyword.	2 `io.corpusiq/multi-source-mcp/notion_connector` Directly searches Notion workspace.
create a notion page	2 `io.github.kfuras/notipo/create_post` Directly creates a Notion page as described.	0 `io.github.pipeworx-io/notion_connect/notion_get_page` Tool reads pages, not creates them.	2 `io.github.kfuras/notipo/create_post` Directly creates a Notion page as described.
send an email	2 `io.github.hail-hq/hail-mcp/email_send` Directly sends emails, general-purpose tool.	2 `io.github.hail-hq/hail-mcp/email_send` Directly sends emails, general-purpose tool.	2 `com.youspot/youspot/send_email` Directly sends email to user's own address.
read my gmail inbox	2 `ai.duvera/gateway/gmail__mail_read` Directly reads Gmail inbox as requested.	2 `ai.duvera/gateway/gmail__mail_read` Directly reads Gmail inbox as requested.	2 `ai.duvera/gateway/gmail__mail_read` Directly reads Gmail inbox as requested.
create a calendar event	2 `app.openkrill/send-out/create_calendar_event` Directly creates calendar events for scheduling.	2 `ai.chronary/mcp/create_calendar` Directly creates calendar, prerequisite for events.	2 `io.github.wellapp-ai/well-mcp/well_create_calendar_event` Directly creates calendar events on Google Calendar.
read rows from a google sheet	2 `io.github.pipeworx-io/google_sheets/sheets_read` Directly reads rows from Google Sheets.	2 `io.github.pipeworx-io/google_sheets/sheets_read` Directly reads rows from Google Sheets.	2 `io.github.pipeworx-io/google_sheets/sheets_read` Directly reads rows from Google Sheets.
upload a file to google drive	0 `io.github.enritarta/flask/upload_video` Uploads to Flask, not Google Drive.	0 `io.github.enritarta/flask/upload_video` Uploads to Flask, not Google Drive.	2 `io.github.hermoso-ai/hermoso/save_to_drive` Directly uploads any file to Google Drive.
search the web for recent news	2 `io.github.wzltmp/mcp-automations/daily_digest` Directly searches web for recent news.	2 `mu.micro/mu/web_search` Directly searches web for news.	2 `io.agentsvc/services/search_read` Directly searches recent news articles.
search the web	2 `mu.micro/mu/web_search` Directly performs web search for general users.	2 `ai.shumi/mcp/search_web` Directly searches the web for general use.	2 `io.github.despiteryan/agentwholesale/web-search` Directly performs general web search.
fetch a web page and convert it to markdown	2 `ai.usesuperflow/tools/page_to_markdown` Directly fetches and converts web pages to markdown.	2 `dev.weblens/weblens/fetch_webpage` Directly fetches and converts webpages to markdown.	2 `io.github.djrobson5/agentmd/convert_url_to_markdown` Directly fetches and converts web pages to markdown.
scrape product prices from amazon	2 `io.oxylabs/oxylabs-mcp/amazon_product_scraper` Directly scrapes Amazon product prices.	2 `io.oxylabs/oxylabs-mcp/amazon_product_scraper` Directly scrapes Amazon product prices.	2 `io.github.pipeworx-io/decodo/decodo_amazon_product` Directly scrapes Amazon product prices and data.
take a screenshot of a website	2 `com.geekflare/mcp/screenshot` Directly captures website screenshots.	2 `com.geekflare/mcp/screenshot` Directly captures website screenshots.	2 `io.github.vee3io/vee3/website_screenshot` Directly takes website screenshots.
run a SQL query against a postgres database	2 `ai.duvera/gateway/postgres__sql_read` Directly executes read-only SQL queries on Postgres.	2 `ai.duvera/gateway/postgres__sql_read` Directly executes read-only SQL queries on Postgres.	2 `com.floot/floot/query_database` Directly runs SQL queries on Postgres databases.
query a mongodb collection	2 `io.github.mongodb-js/mongodb-mcp-server/find` Directly queries MongoDB collections.	0 `com.docimprint/api/ask_collection` RAG over docs, not MongoDB query	1 `ai.smry.r/smry-product/create_news_query` Narrow scope to news, not general MongoDB
get the current weather in Paris	2 `fr.scorelook/capucine/get_weather_context` Directly provides weather for a given city.	2 `fr.scorelook/capucine/weather_looks` Directly provides local weather for any city including Paris.	0 `io.github.truefixr/atlascast-truefixr/get_current_weather` Only works for US locations, not Paris
get a 7 day weather forecast	2 `com.gadgethumans.api/api-hub/get_forecast` Directly provides 7-day weather forecast.	2 `com.gadgethumans.api/api-hub/get_forecast` Directly provides 7-day weather forecast.	2 `io.github.outdoorithm/outdoorithm/get_weather` Directly provides 7-day weather forecast with camping details.
get the latest stock price for AAPL	2 `io.github.pipeworx-io/alphavantage/av_quote` Directly gets real-time stock price for AAPL.	2 `io.github.blackboxfoundry/livedatalink/stock_quote` Directly provides latest stock price for any ticker.	2 `io.github.blackboxfoundry/livedatalink/stock_quote` Directly provides latest stock price for any ticker.
get crypto prices	2 `com.gadgethumans/api-hub/get_crypto_prices` Directly fetches real-time crypto prices.	2 `com.gadgethumans/api-hub/get_crypto_prices` Directly fetches real-time crypto prices.	2 `io.github.connerlambden/helium-mcp/get_ticker` Directly gets crypto prices via ticker.
convert 100 USD to EUR	2 `dev.cz-agents/cnb/convert` Directly converts currencies using official rates.	2 `dev.cz-agents/cnb/convert` Directly converts currencies using official rates.	2 `io.github.lulu-the-narwhal/fx-converter-mcp/convert_currency` Directly converts currencies using real rates.
geocode an address to coordinates	2 `com.gadgethumans.api/api-hub/geocode` Directly geocodes addresses to coordinates.	2 `com.gadgethumans.api/api-hub/geocode` Directly geocodes addresses to coordinates.	2 `com.eckari/capabilities/location_us_geocode` Directly geocodes US addresses to coordinates.
get driving directions between two cities	2 `io.github.whiteknightonhorse/apibase/geo.navigation.route` Directly provides driving directions between cities.	2 `io.github.whiteknightonhorse/apibase/geo.navigation.route` Directly provides driving directions between cities.	1 `com.haulest/haulest/get_route_facts` Provides driving miles/hours but focused on moving, not general directions.
translate text from english to spanish	2 `io.github.krakonjac300-pixel/agent-media-toolkit/translate` Directly translates text to specified language.	2 `io.github.i02202/automan/traduccion` Directly translates text between languages, including English to Spanish.	1 `com.docimprint/api/translate_url` Translates URLs, not direct text input.
transcribe an audio file to text	2 `app.openkrill/transcript-kit/transcribe_media` Directly transcribes audio to text with captions.	2 `ai.com.mcp/openai-tools/createTranscription` Directly transcribes audio to text.	2 `io.github.mikeypetrillo/agent402/audio.transcribe` Directly transcribes audio to text via URL.
generate an image from a text prompt	2 `mu.micro/mu/images_generate` Directly generates images from text prompts.	2 `io.github.cnghockey/sats4ai/image` Directly generates images from text prompts.	2 `io.clueso/video/generate_media` Directly generates images from text prompts.
convert text to speech	2 `io.github.deesmo/arch-tools-mcp/text-to-speech` Directly converts text to speech via ElevenLabs.	2 `io.github.bytevirts/viewmax/generate_speech` Directly converts text to speech as described.	2 `io.github.its-fortunatefolly/hubvibe/hubvibe_speech_synthesize` Directly converts text to speech via TTS.
look up a company's SEC filings	2 `com.apiacre/api-acre/research.sec-filing-signals` Directly retrieves and analyzes SEC filings for companies.	2 `com.apiacre/api-acre/research.sec-filing-signals` Directly retrieves and analyzes SEC filings for companies.	1 `io.github.jethrokirk-sudo/filingapi/get_guidance` Related but narrower; extracts guidance, not general filings.
search academic papers on arxiv	2 `com.tomesphere/tomesphere/search_papers` Directly searches academic papers on arxiv and related repositories.	2 `org.memoryapi/research/search_arxiv` Directly searches arXiv for academic papers.	2 `xyz.kadec0/academic-research/papers` Directly searches arxiv and other academic sources.
get wikipedia summary of a topic	2 `io.github.seekdaseek/agentfeed/get_wiki` Directly provides Wikipedia summaries for any topic.	2 `app.cutgpt/research/wikipedia_lookup` Directly fetches Wikipedia summaries for any topic.	2 `art.maddegen.agent/synapse-web/wiki` Directly provides Wikipedia summaries for any topic.
search youtube videos	2 `io.apidirect/api-direct/search_youtube` Directly searches YouTube videos by keyword.	2 `io.apidirect/api-direct/search_youtube` Directly searches YouTube videos by keyword.	2 `io.github.its-fortunatefolly/hubvibe/hubvibe_video_youtube` Directly searches YouTube videos as described.
get the time in Tokyo	2 `io.github.sadri-dridi/bcp47-ok/tz-asia-tokyo` Directly returns current time in Tokyo.	2 `io.github.sadri-dridi/bcp47-ok/tz-asia-tokyo` Directly returns current time in Tokyo.	0 `ai.gondola/gondola/get_upcoming_trips` Unrelated to time; it's about trip reservations.
send an SMS message	2 `mu.micro/mu/sms_send` Directly sends SMS messages.	2 `io.github.wuzenghai616-lang/goldbean/sms_send` Directly sends SMS messages.	1 `com.innergcomplete/shearquery/ghl_send_sms` Sends SMS via GoHighLevel, not a general SMS tool.
shorten a URL	2 `io.github.wuzenghai616-lang/goldbean/url_shorten` Directly shortens URLs as described.	2 `io.github.wuzenghai616-lang/goldbean/url_shorten` Directly shortens URLs as described.	2 `ink.kool/url-shortener/shorten_url` Directly shortens URLs as described.
validate an email address	2 `io.tinyfn/tinyfn/validate_email` Directly validates email addresses.	2 `io.tinyfn/tinyfn/validate_email` Directly validates email addresses.	2 `ai.startuphub/startuphub/validate_email` Directly validates email deliverability.
look up DNS records for a domain	2 `io.github.venomseven/nslookup/dns_lookup` Directly performs DNS record lookups for any domain.	2 `com.geekflare/mcp/dnsRecord` Directly retrieves DNS records for a domain.	1 `art.maddegen.agent/synapse-web/domain_whois` Provides nameservers but not full DNS records like A/AAAA/MX.
check if a website is up	2 `dev.domainee/domainee/tools_website_status_checker` Directly checks website uptime.	2 `com.geekflare/mcp/siteStatus` Directly checks website availability.	2 `ai.rexity/website-check/website_check` Directly checks if a website is up.
run python code in a sandbox	2 `world.scalix/cloud/scalix_sandbox_run` Directly runs Python code in an isolated sandbox.	2 `io.github.poiuyhje/eqvps/run_code` Directly runs Python code in a sandbox.	2 `io.github.poiuyhje/eqvps/run_code` Directly runs Python code in a sandbox.
search npm packages	2 `io.github.whiteknightonhorse/apibase/npm.packages.search` Official npm search tool for packages.	2 `io.github.whiteknightonhorse/apibase/npm.packages.search` Official npm search tool for packages.	2 `io.github.isaiahdupree/devstack-mcp/search_packages` Directly searches npm packages via registry=all.