047175dbed
Make the OpenAPI-driven MCP surface more usable by agents, following the mcp-builder guidance. - Enrich proto descriptions (single source of truth, flows to OpenAPI + MCP tool descriptions): document the memo `filter` CEL grammar with fields and examples (replacing the dangling "Refer to Shortcut.filter"), clarify the created_ts/updated_ts vs create_time/update_time naming, the visibility enum, the declarative replace semantics of Set* ops, and steer tag filters to `"x" in tags` (not the unsupported `tag == "x"`). - Mark SetMemoAttachments / SetMemoRelations idempotent via a per-operation override the HTTP-method heuristic can't express. - Curate two read-only orientation tools: shortcut_list_shortcuts (surfaces reusable CEL filters) and auth_get_current_user (the single allowed auth/identity op, for resolving the current user); guard test updated to keep the rest of the auth/user surface excluded. - Add a task-level evaluation suite (server/router/mcp/evals) with 10 verified questions, pinned to the deterministic demo seed.
62 lines
2.7 KiB
XML
62 lines
2.7 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
Task-level evaluation for the memos MCP server.
|
|
|
|
These questions test whether an LLM can accomplish realistic tasks by
|
|
composing the curated MCP tools (memo_list_memos with CEL filters,
|
|
memo_list_memo_comments, memo_list_memo_reactions, shortcut_list_shortcuts,
|
|
auth_get_current_user, ...). They are NOT unit tests of the server plumbing —
|
|
see *_test.go for that.
|
|
|
|
Answers are pinned to the deterministic seed in
|
|
store/seed/sqlite/01__dump.sql and were verified by querying a fresh
|
|
demo-mode instance. Run against a freshly seeded instance ONLY (the public
|
|
demo at demo.usememos.com is a shared, mutable account and will not match).
|
|
See README.md in this directory for how to launch one.
|
|
|
|
Questions deliberately avoid absolute dates: the seed uses relative
|
|
timestamps (strftime('now','-N days')), so only relative ordering, counts,
|
|
and content are stable.
|
|
-->
|
|
<evaluation>
|
|
<qa_pair>
|
|
<question>Look at every memo in the instance and add up all the emoji reactions across them. What is the total number of reactions?</question>
|
|
<answer>12</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>How many memos are pinned?</question>
|
|
<answer>2</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Considering only the memos that are NOT pinned, one of them has more reactions than any other. That memo is about a book the author started reading. What is the title of that book?</question>
|
|
<answer>Deep Work</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Two memos are pinned. One contains external hyperlinks and the other does not. Give the single tag of the pinned memo that contains hyperlinks.</question>
|
|
<answer>sponsors</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>How many memos contain at least one fenced code block?</question>
|
|
<answer>2</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>How many distinct users have authored the memos in this instance?</question>
|
|
<answer>2</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Find the memo that is a cheat sheet of git commands. How many comments does it have?</question>
|
|
<answer>2</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Find the memo containing a movie watchlist. What is the username of the user who created it?</question>
|
|
<answer>alice</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Exactly one memo is a travel bucket list. It names a Nordic country the author plans to visit in winter to see the Northern Lights. Which country?</question>
|
|
<answer>Iceland</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Using the authenticated session, what is the username (login name) of the account these memos belong to?</question>
|
|
<answer>demo</answer>
|
|
</qa_pair>
|
|
</evaluation>
|