{ "timestamp": "2026-09-13T20:02:29.542Z", "api": "http://localhost:3001", "ragUp": true, "totalCases": 67, "passedCases": 67, "failedCases": 0, "skippedCases": 0, "totalAssertions": 542, "passedAssertions": 542, "byCategory": { "routing": { "total": 6, "passed": 6, "failed": 0, "skipped": 0 }, "access": { "total": 6, "passed": 6, "failed": 0, "skipped": 0 }, "library": { "total": 2, "passed": 2, "failed": 0, "skipped": 0 }, "agent": { "total": 12, "passed": 12, "failed": 0, "skipped": 0 }, "leave": { "total": 7, "passed": 7, "failed": 0, "skipped": 0 }, "safety": { "total": 3, "passed": 3, "failed": 0, "skipped": 0 }, "flow": { "total": 3, "passed": 3, "failed": 0, "skipped": 0 }, "manager": { "total": 5, "passed": 5, "failed": 0, "skipped": 0 }, "hr": { "total": 2, "passed": 2, "failed": 0, "skipped": 0 }, "help": { "total": 12, "passed": 12, "failed": 0, "skipped": 0 }, "sensitive": { "total": 9, "passed": 9, "failed": 0, "skipped": 0 } }, "cases": [ { "caseId": "routing-01", "caseName": "\"equity plan\" is a policy question, not a plan to book", "category": "routing", "status": "passed", "steps": [ { "label": "alex says \"What does the executive equity plan grant?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" }, { "description": "nothing was drafted", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 2, "passedAssertions": 2, "durationMs": 7 }, { "caseId": "routing-02", "caseName": "Vacation ask goes to the agent with three options", "category": "routing", "status": "passed", "steps": [ { "label": "alex says \"I want to book a vacation\"", "output": "Alex, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Dec 28 to Dec 30** (3 PTO days, 11 days off)\n- 3 PTO days gets you 11 days off, Dec 24 through Jan 3.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25) and New Year's Eve (Dec 31) and New Year's Day (Jan 1), which don't count against PTO.\n- Priya Sharma (approved, Dec 28 to Dec 30) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Dec 21 to Dec 23** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Dec 19 through Dec 27.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Nov 23 to Nov 25** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Nov 21 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (pending, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "offers 3 date options", "pass": true, "reason": "3 options" }, { "description": "trace includes search_policies", "pass": true, "reason": "orchestrator > get_leave_balances > search_policies > get_company_holidays > get_team_calendar > suggest_vacation_dates" }, { "description": "trace includes suggest_vacation_dates", "pass": true, "reason": "orchestrator > get_leave_balances > search_policies > get_company_holidays > get_team_calendar > suggest_vacation_dates" } ] } ], "totalAssertions": 4, "passedAssertions": 4, "durationMs": 526 }, { "caseId": "routing-03", "caseName": "Policy phrasings route to the library", "category": "routing", "status": "passed", "steps": [ { "label": "alex says \"How do I report a conflict of interest?\"", "output": "## Conflicts of interest\n\nDisclose any outside activity, investment, or relationship that could conflict\nwith your responsibilities. When in doubt, disclose and ask HR.\n\n[Code of Conduct]", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" } ] }, { "label": "alex says \"Can I carry over unused PTO?\"", "output": "## Carryover\n\nYou may carry over up to **5 unused PTO days** into the next calendar year. Days\nbeyond the cap are forfeited on December 31 and are not paid out.\n\n[PTO Policy]", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" } ] }, { "label": "alex says \"How many sick days do I get?\"", "output": "Sick leave is tracked separately from PTO. Employees receive 10 sick days per year. Sick days do not carry over and are not paid out on departure.\n\n[PTO Policy / Sick leave]", "assertions": [ { "description": "contains \"10 sick days\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] }, { "label": "alex says \"How long is parental leave?\"", "output": "Alex, here is what applies to you as an employee based in the United States:\n- Entitlement: 16 weeks of fully paid leave for a birthing parent, 12 weeks for a non-birthing parent [US Leave Supplement / Company paid leave].\n- Job protection: up to 12 weeks under FMLA, running concurrently with the paid leave [US Leave Supplement / Job protection under FMLA]. State paid family leave, where it exists, is coordinated by HR so pay stays at 100% [US Leave Supplement / State programs].\n- Eligibility: You need 90 days of service. You joined on Tue, Mar 15, 2022, so you qualify [Parental Leave Policy / Eligibility].\n- To apply: at least 30 days before the leave starts, with medical certification form from your healthcare provider with the expected due date. The request goes to Jordan Park to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected due date, or the date you'd like the leave to begin.", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "trace includes reconcile", "pass": true, "reason": "orchestrator > resolve_asker > search_policies > search_policies > reconcile > check_eligibility > search_policies" } ] }, { "label": "alex says \"What is the code of conduct?\"", "output": "# Code of Conduct\n\n**Applies to:** All employees, contractors, and interns. This document is public within the company.\n\n[Code of Conduct]", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" } ] } ], "totalAssertions": 7, "passedAssertions": 7, "durationMs": 1208 }, { "caseId": "routing-04", "caseName": "Time-off phrasings route to the agent", "category": "routing", "status": "passed", "steps": [ { "label": "alex says \"Plan a trip for me in December\"", "output": "Alex, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Dec 28 to Dec 30** (3 PTO days, 11 days off)\n- 3 PTO days gets you 11 days off, Dec 24 through Jan 3.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25) and New Year's Eve (Dec 31) and New Year's Day (Jan 1), which don't count against PTO.\n- Priya Sharma (approved, Dec 28 to Dec 30) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Dec 21 to Dec 23** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Dec 19 through Dec 27.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Nov 23 to Nov 25** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Nov 21 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (pending, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "offers 3 date options", "pass": true, "reason": "3 options" } ] }, { "label": "alex says \"5 days around Thanksgiving\"", "output": "Alex, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Nov 30 to Dec 4** (5 PTO days, 11 days off)\n- 5 PTO days gets you 11 days off, Nov 26 through Dec 6.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Nov 19 to Nov 25** (5 PTO days, 11 days off)\n- 5 PTO days gets you 11 days off, Nov 19 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (pending, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Nov 2 to Nov 6** (5 PTO days, 9 days off)\n- 5 PTO days gets you 9 days off, Oct 31 through Nov 8.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 5 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "offers 3 date options", "pass": true, "reason": "3 options" } ] }, { "label": "alex says \"What are my balances?\"", "output": "Here's where you stand, Alex:\n- **Paid Time Off:** 12 days (96 hrs) available, 24 hrs pending approval\n- **Sick Leave:** 8 days (64 hrs) available\n- **Floating Holiday:** 2 days (16 hrs) available\n\nPending: Paid Time Off Sep 21 to Sep 23 (3 days)\n\nWant me to find the best dates for a vacation?", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] }, { "label": "alex says \"Who's out on my team?\"", "output": "**Out in the next 90 days:**\n- Sarah Kim: Oct 12 to Oct 16 (approved)\n- David Lee: Nov 23 to Nov 25 (pending)", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] }, { "label": "alex says \"How much notice do I need?\"", "output": "Submit PTO requests at least two weeks (14 calendar days) in advance for planned absences. Your manager approves requests based on team coverage. Requests longer than 10 consecutive workdays require director approval in addition to your manager.\n\n[PTO Policy / Requesting time off]", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] } ], "totalAssertions": 7, "passedAssertions": 7, "durationMs": 2929 }, { "caseId": "routing-05", "caseName": "A stray \"yes\" with nothing drafted cannot submit", "category": "routing", "status": "passed", "steps": [ { "label": "alex says \"Yes, send it\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-001\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-09-21\",\"endDate\":\"2026-09-23\",\"totalDays\":3,\"totalHours\":24,\"status\":\"pending\",\"comment\":\"Family trip\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-02T10:30:00Z\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "still one request on file", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 4, "passedAssertions": 4, "durationMs": 11 }, { "caseId": "routing-06", "caseName": "Policy question and time-off ask share one thread", "category": "routing", "status": "passed", "steps": [ { "label": "alex says \"Can I carry over unused PTO?\"", "output": "## Carryover\n\nYou may carry over up to **5 unused PTO days** into the next calendar year. Days\nbeyond the cap are forfeited on December 31 and are not paid out.\n\n[PTO Policy]", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" } ] }, { "label": "alex says \"I want to book a vacation\"", "output": "Alex, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Dec 28 to Dec 30** (3 PTO days, 11 days off)\n- 3 PTO days gets you 11 days off, Dec 24 through Jan 3.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25) and New Year's Eve (Dec 31) and New Year's Day (Jan 1), which don't count against PTO.\n- Priya Sharma (approved, Dec 28 to Dec 30) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Dec 21 to Dec 23** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Dec 19 through Dec 27.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Nov 23 to Nov 25** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Nov 21 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (pending, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "offers 3 date options", "pass": true, "reason": "3 options" } ] }, { "label": "alex says \"Request option 1\"", "output": "**Request summary**\n**Type:** Paid Time Off\n**Dates:** Mon, Dec 28, 2026 to Wed, Dec 30, 2026\n**Work days:** 3 (24 hours)\n**Balance after:** 9 days\n**Approver:** Jordan Park\n\nHeads up: Priya Sharma (approved, Dec 28 to Dec 30) already out in this window.\n\nReady to send this to Jordan Park for approval? Say yes to confirm.", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-12-28 to 2026-12-30, 3 days" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"Can I carry over unused PTO?\"", "output": "## Carryover\n\nYou may carry over up to **5 unused PTO days** into the next calendar year. Days\nbeyond the cap are forfeited on December 31 and are not paid out.\n\n[PTO Policy]", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"Yes, send it\"", "output": "Sent. Request **REQ-2026-006** is pending with Jordan Park, who has been notified at jordan.park@acme.com. You'll see the decision here as soon as it's made.", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "request submitted to the HR system", "pass": true, "reason": "REQ-2026-006 pending" } ] } ], "totalAssertions": 10, "passedAssertions": 10, "durationMs": 1753 }, { "caseId": "access-01", "caseName": "Alex (engineering) sees on-call, not the equity plan or comp bands", "category": "access", "status": "passed", "steps": [ { "label": "alex says \"What is the engineering on-call policy?\"", "output": "## Rotation\n\nEngineers participate in a weekly on-call rotation. Each rotation runs Monday\n10:00 to the following Monday 10:00. No engineer is scheduled for more than one\nweek in any four-week period.\n\n[Engineering On-Call Policy]", "assertions": [ { "description": "contains \"On-Call\"", "pass": true, "reason": "found" }, { "description": "answered from \"Engineering On-Call Policy\"", "pass": true, "reason": "Engineering On-Call Policy" }, { "description": "searchable shelf is exactly: Code of Conduct, PTO Policy, Parental Leave Policy, Leave of Absence Procedure, US Leave Supplement, Engineering On-Call Policy", "pass": true, "reason": "Code of Conduct | Engineering On-Call Policy | Leave of Absence Procedure | PTO Policy | Parental Leave Policy | US Leave Supplement" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] }, { "label": "alex says \"What does the executive equity plan grant?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"four years\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"equity plan\"", "pass": true, "reason": "absent" }, { "description": "no accessible source, so abstain", "pass": true, "reason": "abstained" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] } ], "totalAssertions": 8, "passedAssertions": 8, "durationMs": 17 }, { "caseId": "access-02", "caseName": "Jordan (manager, exec group) sees the equity plan, not comp bands", "category": "access", "status": "passed", "steps": [ { "label": "jordan says \"How do executive equity grants vest?\"", "output": "## Vesting\n\nRefresh grants vest over **four years** with a one-year cliff, then quarterly.\nUnvested units are forfeited on departure.\n\n[Executive Equity Plan]", "assertions": [ { "description": "contains \"four years\"", "pass": true, "reason": "found" }, { "description": "answered from \"Executive Equity Plan\"", "pass": true, "reason": "Executive Equity Plan" }, { "description": "searchable shelf is exactly: Code of Conduct, PTO Policy, Parental Leave Policy, Leave of Absence Procedure, US Leave Supplement, Executive Equity Plan", "pass": true, "reason": "Code of Conduct | Executive Equity Plan | Leave of Absence Procedure | PTO Policy | Parental Leave Policy | US Leave Supplement" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] }, { "label": "jordan says \"What is the salary band for a senior engineer?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"150,000\"", "pass": true, "reason": "absent" }, { "description": "no accessible source, so abstain", "pass": true, "reason": "abstained" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] } ], "totalAssertions": 7, "passedAssertions": 7, "durationMs": 10 }, { "caseId": "access-03", "caseName": "Bob (HR admin) sees comp bands, not the equity plan or on-call", "category": "access", "status": "passed", "steps": [ { "label": "bob says \"What is the salary band for a senior engineer?\"", "output": "## Engineering bands (annual base, USD)\n\n- L3 Software Engineer: 120,000 – 150,000\n- L4 Senior Engineer: 150,000 – 195,000\n- L5 Staff Engineer: 195,000 – 245,000\n- L6 Principal Engineer: 245,000 – 310,000\n\n[Compensation Bands]", "assertions": [ { "description": "contains \"150,000\"", "pass": true, "reason": "found" }, { "description": "answered from \"Compensation Bands\"", "pass": true, "reason": "Compensation Bands" }, { "description": "searchable shelf is exactly: Code of Conduct, PTO Policy, Parental Leave Policy, Leave of Absence Procedure, US Leave Supplement, Compensation Bands, Executive Compensation Bands, India Leave Supplement", "pass": true, "reason": "Code of Conduct | Compensation Bands | Executive Compensation Bands | India Leave Supplement | Leave of Absence Procedure | PTO Policy | Parental Leave Policy | US Leave Supplement" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] }, { "label": "bob says \"What are the compensation bands for executives?\"", "output": "## Compensation bands for executives (annual base, USD)\n\nThe compensation bands for executive roles are:\n\n- Director: 275,000 – 340,000\n- Vice President: 340,000 – 430,000\n- SVP: 430,000 – 550,000\n- C-level: set by the Compensation Committee and not published in this document.\n\nCanary: CANARY-EXEC-9B2D\n\n[Executive Compensation Bands]", "assertions": [ { "description": "contains \"340,000\"", "pass": true, "reason": "found" }, { "description": "answered from \"Executive Compensation Bands\"", "pass": true, "reason": "Executive Compensation Bands" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] }, { "label": "bob says \"What does the executive equity plan grant?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "no accessible source, so abstain", "pass": true, "reason": "abstained" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] } ], "totalAssertions": 9, "passedAssertions": 9, "durationMs": 12 }, { "caseId": "access-04", "caseName": "Marcus (contractor) sees only the code of conduct", "category": "access", "status": "passed", "steps": [ { "label": "marcus says \"What is the code of conduct?\"", "output": "# Code of Conduct\n\n**Applies to:** All employees, contractors, and interns. This document is public within the company.\n\n[Code of Conduct]", "assertions": [ { "description": "answered from \"Code of Conduct\"", "pass": true, "reason": "Code of Conduct" }, { "description": "searchable shelf is exactly: Code of Conduct", "pass": true, "reason": "Code of Conduct" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] }, { "label": "marcus says \"Can I carry over unused PTO?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"5 unused\"", "pass": true, "reason": "absent" }, { "description": "no accessible source, so abstain", "pass": true, "reason": "abstained" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 4 }, { "caseId": "access-05", "caseName": "Priya (part-time, no department group) sees employee docs only", "category": "access", "status": "passed", "steps": [ { "label": "priya says \"Can I carry over unused PTO?\"", "output": "## Carryover\n\nYou may carry over up to **5 unused PTO days** into the next calendar year. Days\nbeyond the cap are forfeited on December 31 and are not paid out.\n\n[PTO Policy]", "assertions": [ { "description": "answered from \"PTO Policy\"", "pass": true, "reason": "PTO Policy" }, { "description": "searchable shelf is exactly: Code of Conduct, PTO Policy, Parental Leave Policy, Leave of Absence Procedure, US Leave Supplement", "pass": true, "reason": "Code of Conduct | Leave of Absence Procedure | PTO Policy | Parental Leave Policy | US Leave Supplement" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] }, { "label": "priya says \"What is the engineering on-call policy?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "no accessible source, so abstain", "pass": true, "reason": "abstained" }, { "description": "no out-of-scope document is named, flagged, or counted", "pass": true, "reason": "clean" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 5 }, { "caseId": "access-06", "caseName": "Canary strings never reach the wrong persona", "category": "access", "status": "passed", "steps": [ { "label": "alex says \"What is the salary band for a senior engineer?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"CANARY-COMP\"", "pass": true, "reason": "absent" } ] }, { "label": "alex says \"How do executive equity grants vest?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"CANARY-EQ\"", "pass": true, "reason": "absent" } ] }, { "label": "bob says \"How do executive equity grants vest?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"CANARY-EQ\"", "pass": true, "reason": "absent" } ] }, { "label": "jordan says \"What is the salary band for a senior engineer?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"CANARY-COMP\"", "pass": true, "reason": "absent" } ] }, { "label": "alex says \"What are the compensation bands for executives?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"CANARY-EXEC\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"340,000\"", "pass": true, "reason": "absent" }, { "description": "no accessible source, so abstain", "pass": true, "reason": "abstained" } ] }, { "label": "jordan says \"What are the compensation bands for executives?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"CANARY-EXEC\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"340,000\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"four years\"", "pass": true, "reason": "absent" }, { "description": "no accessible source, so abstain", "pass": true, "reason": "abstained" } ] } ], "totalAssertions": 11, "passedAssertions": 11, "durationMs": 16 }, { "caseId": "library-01", "caseName": "Library lists only readable documents, with text", "category": "library", "status": "passed", "steps": [ { "label": "anon GET http://localhost:8930/api/docs?user=alice@corp.com", "output": "[{\"docId\": \"code_of_conduct.md\", \"title\": \"Code of Conduct\", \"allowed\": true, \"public\": true, \"allowedGroups\": [], \"allowedUsers\": [], \"words\": 129, \"text\": \"# Code of Conduct\\n\\n**Applies to:** All employees, contractors, and interns. This document is public within the company.\\n\\n## Our principles\\n\\nWe treat each other with respect. Harassment, discrimination, and retaliation\\nare never tolerat", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "six documents for Alex", "pass": true, "reason": "ok" }, { "description": "all have full text", "pass": true, "reason": "ok" }, { "description": "none flagged as locked", "pass": true, "reason": "ok" }, { "description": "on-call and US supplement included, comp bands and India supplement absent", "pass": true, "reason": "ok" } ] }, { "label": "anon GET http://localhost:8930/api/docs?user=contractor@ext.com", "output": "[{\"docId\": \"code_of_conduct.md\", \"title\": \"Code of Conduct\", \"allowed\": true, \"public\": true, \"allowedGroups\": [], \"allowedUsers\": [], \"words\": 129, \"text\": \"# Code of Conduct\\n\\n**Applies to:** All employees, contractors, and interns. This document is public within the company.\\n\\n## Our principles\\n\\nWe treat each other with respect. Harassment, discrimination, and retaliation\\nare never tolerat", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "one document for the contractor", "pass": true, "reason": "ok" } ] }, { "label": "anon GET http://localhost:8930/api/docs?user=ananya@corp.com", "output": "[{\"docId\": \"code_of_conduct.md\", \"title\": \"Code of Conduct\", \"allowed\": true, \"public\": true, \"allowedGroups\": [], \"allowedUsers\": [], \"words\": 129, \"text\": \"# Code of Conduct\\n\\n**Applies to:** All employees, contractors, and interns. This document is public within the company.\\n\\n## Our principles\\n\\nWe treat each other with respect. Harassment, discrimination, and retaliation\\nare never tolerat", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "India supplement for Ananya, no US supplement", "pass": true, "reason": "ok" } ] }, { "label": "anon GET http://localhost:8930/api/docs?user=bob@corp.com", "output": "[{\"docId\": \"code_of_conduct.md\", \"title\": \"Code of Conduct\", \"allowed\": true, \"public\": true, \"allowedGroups\": [], \"allowedUsers\": [], \"words\": 129, \"text\": \"# Code of Conduct\\n\\n**Applies to:** All employees, contractors, and interns. This document is public within the company.\\n\\n## Our principles\\n\\nWe treat each other with respect. Harassment, discrimination, and retaliation\\nare never tolerat", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "comp bands readable by HR", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 11, "passedAssertions": 11, "durationMs": 6 }, { "caseId": "library-02", "caseName": "Local fallback library obeys the same rule", "category": "library", "status": "passed", "steps": [ { "label": "alex GET /api/policy/docs", "output": "[{\"docId\":\"blackout_periods\",\"title\":\"Engineering Blackout Periods\",\"allowed\":true,\"public\":false,\"allowedGroups\":[\"all-employees\",\"contractors\"],\"allowedUsers\":[],\"words\":75,\"text\":\"# Engineering Blackout Periods\\n\\n## What a blackout period is\\nDuring a blackout period, discretionary time off is not approved for Engineering staff without director sign-off. Sick leave and emergencies are always a", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "no locked entries", "pass": true, "reason": "ok" }, { "description": "manager guide absent for an employee", "pass": true, "reason": "ok" } ] }, { "label": "jordan GET /api/policy/docs", "output": "[{\"docId\":\"contractor_time_off\",\"title\":\"Contractor Time Off\",\"allowed\":true,\"public\":false,\"allowedGroups\":[\"contractors\",\"managers\"],\"allowedUsers\":[],\"words\":47,\"text\":\"# Contractor Time Off\\n\\n## Eligibility\\nContractors do not accrue paid leave. Contractors may request unpaid time off, which requires the engagement manager's approval.\\n\\n## Notice\\nGive at least one week (7 calendar days) of ", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "manager guide present for a manager", "pass": true, "reason": "ok" } ] }, { "label": "marcus GET /api/policy/docs", "output": "[{\"docId\":\"contractor_time_off\",\"title\":\"Contractor Time Off\",\"allowed\":true,\"public\":false,\"allowedGroups\":[\"contractors\",\"managers\"],\"allowedUsers\":[],\"words\":47,\"text\":\"# Contractor Time Off\\n\\n## Eligibility\\nContractors do not accrue paid leave. Contractors may request unpaid time off, which requires the engagement manager's approval.\\n\\n## Notice\\nGive at least one week (7 calendar days) of ", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "contractor doc present, PTO policy absent", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 7, "passedAssertions": 7, "durationMs": 3 }, { "caseId": "agent-01", "caseName": "Full-time balances", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"What are my balances?\"", "output": "Here's where you stand, Alex:\n- **Paid Time Off:** 12 days (96 hrs) available, 24 hrs pending approval\n- **Sick Leave:** 8 days (64 hrs) available\n- **Floating Holiday:** 2 days (16 hrs) available\n\nPending: Paid Time Off Sep 21 to Sep 23 (3 days)\n\nWant me to find the best dates for a vacation?", "assertions": [ { "description": "contains \"Paid Time Off\"", "pass": true, "reason": "found" }, { "description": "contains \"12 days\"", "pass": true, "reason": "found" }, { "description": "contains \"24 hrs pending\"", "pass": true, "reason": "found" }, { "description": "contains \"Sick Leave\"", "pass": true, "reason": "found" }, { "description": "contains \"8 days\"", "pass": true, "reason": "found" }, { "description": "contains \"Floating Holiday\"", "pass": true, "reason": "found" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 533 }, { "caseId": "agent-02", "caseName": "Part-time balances are prorated", "category": "agent", "status": "passed", "steps": [ { "label": "priya says \"What are my balances?\"", "output": "Here's where you stand, Priya:\n- **Paid Time Off (Prorated):** 8 days (48 hrs) available\n- **Sick Leave (Prorated):** 4 days (24 hrs) available\n\nWant me to find the best dates for a vacation?", "assertions": [ { "description": "contains \"Prorated\"", "pass": true, "reason": "found" }, { "description": "contains \"8 days\"", "pass": true, "reason": "found" }, { "description": "contains \"4 days\"", "pass": true, "reason": "found" }, { "description": "does not contain \"Floating Holiday\"", "pass": true, "reason": "absent" } ] } ], "totalAssertions": 4, "passedAssertions": 4, "durationMs": 388 }, { "caseId": "agent-03", "caseName": "Contractor has unpaid time off only", "category": "agent", "status": "passed", "steps": [ { "label": "marcus says \"What are my balances?\"", "output": "Marcus, as a contractor you don't accrue paid leave. You can request unpaid time off, which still needs Jordan Park's approval. Want me to find good dates?", "assertions": [ { "description": "contains \"contractor\"", "pass": true, "reason": "found" }, { "description": "contains \"unpaid\"", "pass": true, "reason": "found" }, { "description": "does not contain \"12 days\"", "pass": true, "reason": "absent" } ] } ], "totalAssertions": 3, "passedAssertions": 3, "durationMs": 698 }, { "caseId": "agent-04", "caseName": "Vacation planner: best window, coverage, policy citations", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"I want to book a vacation\"", "output": "Alex, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Dec 28 to Dec 30** (3 PTO days, 11 days off)\n- 3 PTO days gets you 11 days off, Dec 24 through Jan 3.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25) and New Year's Eve (Dec 31) and New Year's Day (Jan 1), which don't count against PTO.\n- Priya Sharma (approved, Dec 28 to Dec 30) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Dec 21 to Dec 23** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Dec 19 through Dec 27.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Nov 23 to Nov 25** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Nov 21 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (pending, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "contains \"12 PTO days\"", "pass": true, "reason": "found" }, { "description": "contains \"14-day notice\"", "pass": true, "reason": "found" }, { "description": "contains \"carry only 5 days\"", "pass": true, "reason": "found" }, { "description": "contains \"PTO Policy\"", "pass": true, "reason": "found" }, { "description": "contains \"11 days off\"", "pass": true, "reason": "found" }, { "description": "contains \"Christmas\"", "pass": true, "reason": "found" }, { "description": "offers 3 date options", "pass": true, "reason": "3 options" }, { "description": "option 1 is 2026-12-28 to 2026-12-30", "pass": true, "reason": "2026-12-28 to 2026-12-30" }, { "description": "every option keeps coverage above the minimum", "pass": true, "reason": "2026-12-28:true 2026-12-21:true 2026-11-23:true" } ] } ], "totalAssertions": 9, "passedAssertions": 9, "durationMs": 620 }, { "caseId": "agent-05", "caseName": "Planner honours a requested month and length", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"5 days in November\"", "output": "Alex, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Nov 30 to Dec 4** (5 PTO days, 11 days off)\n- 5 PTO days gets you 11 days off, Nov 26 through Dec 6.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Nov 19 to Nov 25** (5 PTO days, 11 days off)\n- 5 PTO days gets you 11 days off, Nov 19 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (pending, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Nov 2 to Nov 6** (5 PTO days, 9 days off)\n- 5 PTO days gets you 9 days off, Oct 31 through Nov 8.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 5 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "offers 3 date options", "pass": true, "reason": "3 options" }, { "description": "every option is 5 PTO days", "pass": true, "reason": "ok" }, { "description": "options start in November", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 3, "passedAssertions": 3, "durationMs": 765 }, { "caseId": "agent-06", "caseName": "Contractor vacation uses the 7-day notice and unpaid framing", "category": "agent", "status": "passed", "steps": [ { "label": "marcus says \"I want to book a vacation\"", "output": "Marcus, as a contractor you don't accrue paid leave, so these would be unpaid days with Jordan Park's approval.\nEarliest start under the 7-day notice rule is Sun, Sep 20, 2026 (Contractor Time Off / Notice).\nPolicy check: During a blackout period, discretionary time off is not approved for Engineering staff without director sign-off. Sick leave and emergencies are always allowed. [Engineering Blackout Periods / What a blackout period is]\n\nHere are the best windows I found:\n\n**Option 1: Dec 28 to Dec 30** (3 PTO days, 11 days off)\n- 3 PTO days gets you 11 days off, Dec 24 through Jan 3.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25) and New Year's Eve (Dec 31) and New Year's Day (Jan 1), which don't count against PTO.\n- Priya Sharma (approved, Dec 28 to Dec 30) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n\n**Option 2: Dec 21 to Dec 23** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Dec 19 through Dec 27.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25), which don't count against PTO.\n- Nobody else on the team is out that week.\n\n**Option 3: Nov 23 to Nov 25** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Nov 21 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (pending, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n\nMy pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "contains \"contractor\"", "pass": true, "reason": "found" }, { "description": "contains \"unpaid\"", "pass": true, "reason": "found" }, { "description": "contains \"7-day notice\"", "pass": true, "reason": "found" }, { "description": "contains \"Contractor Time Off\"", "pass": true, "reason": "found" }, { "description": "offers 3 date options", "pass": true, "reason": "3 options" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 413 }, { "caseId": "agent-07", "caseName": "Blackout dates are refused with the citation", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"Can I take Oct 27 to Oct 29 off?\"", "output": "Q4 release freeze (2026-10-26 to 2026-10-30) is a blackout period for Engineering. Want me to suggest the nearest clear week instead?\n\n[Engineering Blackout Periods / Planning around blackouts]", "assertions": [ { "description": "contains \"blackout\"", "pass": true, "reason": "found" }, { "description": "contains \"Engineering Blackout Periods\"", "pass": true, "reason": "found" }, { "description": "contains \"nearest clear week\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "nothing was drafted", "pass": true, "reason": "ok" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 634 }, { "caseId": "agent-08", "caseName": "Insufficient balance is explained with alternatives", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"I want to take Nov 2 to Nov 25 off\"", "output": "You'd need 18 days (144 hrs) of Paid Time Off but only have 12 days (96 hrs) available. Want a shorter range, or to split it across Paid Time Off and another leave type?\n\n[PTO Policy / Requesting time off]", "assertions": [ { "description": "contains \"only have\"", "pass": true, "reason": "found" }, { "description": "suggests an alternative", "pass": true, "reason": "found \"shorter\"" }, { "description": "nothing was drafted", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 3, "passedAssertions": 3, "durationMs": 551 }, { "caseId": "agent-09", "caseName": "Holidays inside a range are excluded from the PTO count", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"I want to take Nov 23 to Nov 27 off\"", "output": "Thanksgiving is a company holiday, no PTO needed. Day After Thanksgiving is a company holiday, no PTO needed.\n\n**Request summary**\n**Type:** Paid Time Off\n**Dates:** Mon, Nov 23, 2026 to Fri, Nov 27, 2026\n**Work days:** 3 (24 hours), 2 holidays excluded\n**Balance after:** 9 days\n**Approver:** Jordan Park\n\nHeads up: David Lee (pending, Nov 23 to Nov 25) already out in this window.\n\nReady to send this to Jordan Park for approval? Say yes to confirm.", "assertions": [ { "description": "contains \"Thanksgiving\"", "pass": true, "reason": "found" }, { "description": "contains \"2 holidays excluded\"", "pass": true, "reason": "found" }, { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-11-23 to 2026-11-27, 3 days" }, { "description": "draft counts 3 work days", "pass": true, "reason": "3" } ] } ], "totalAssertions": 4, "passedAssertions": 4, "durationMs": 364 }, { "caseId": "agent-10", "caseName": "Short-notice request gets a heads-up, not a refusal", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"I want to take next Friday off\"", "output": "**Request summary**\n**Type:** Paid Time Off\n**Dates:** Fri, Sep 18, 2026 to Fri, Sep 18, 2026\n**Work days:** 1 (8 hours)\n**Balance after:** 11 days\n**Approver:** Jordan Park\n\nHeads up: this starts in 5 days, inside the 14-day notice window [PTO Policy / Requesting time off]. Jordan Park can still approve it.\n\nReady to send this to Jordan Park for approval? Say yes to confirm.", "assertions": [ { "description": "contains \"notice window\"", "pass": true, "reason": "found" }, { "description": "contains \"8 hours\"", "pass": true, "reason": "found" }, { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-09-18 to 2026-09-18, 1 days" }, { "description": "draft counts 1 work days", "pass": true, "reason": "1" } ] } ], "totalAssertions": 4, "passedAssertions": 4, "durationMs": 426 }, { "caseId": "agent-11", "caseName": "Company holidays and team calendar", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"When are the company holidays?\"", "output": "**Remaining 2026 company holidays:**\n- Thanksgiving: Thu, Nov 26, 2026\n- Day After Thanksgiving: Fri, Nov 27, 2026\n- Christmas Eve: Thu, Dec 24, 2026\n- Christmas Day: Fri, Dec 25, 2026\n- New Year's Eve: Thu, Dec 31, 2026\n\nWant me to plan a vacation around one of these?", "assertions": [ { "description": "contains \"Thanksgiving\"", "pass": true, "reason": "found" }, { "description": "contains \"Christmas Day\"", "pass": true, "reason": "found" } ] }, { "label": "alex says \"Who's out on my team?\"", "output": "**Out in the next 90 days:**\n- Sarah Kim: Oct 12 to Oct 16 (approved)\n- David Lee: Nov 23 to Nov 25 (pending)", "assertions": [ { "description": "contains \"Sarah Kim\"", "pass": true, "reason": "found" }, { "description": "contains \"David Lee\"", "pass": true, "reason": "found" }, { "description": "does not contain \"Alex Chen\"", "pass": true, "reason": "absent" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 1368 }, { "caseId": "agent-12", "caseName": "Notice question answered from the time-off rules, no leak", "category": "agent", "status": "passed", "steps": [ { "label": "alex says \"How much notice do I need?\"", "output": "Submit PTO requests at least two weeks (14 calendar days) in advance for planned absences. Your manager approves requests based on team coverage. Requests longer than 10 consecutive workdays require director approval in addition to your manager.\n\n[PTO Policy / Requesting time off]", "assertions": [ { "description": "states the notice period", "pass": true, "reason": "found \"two weeks\"" }, { "description": "contains \"PTO Policy\"", "pass": true, "reason": "found" }, { "description": "does not contain \"locked\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"Manager Guide\"", "pass": true, "reason": "absent" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 718 }, { "caseId": "leave-01", "caseName": "Ananya (India): 26 weeks, reconciled from global policy and India supplement", "category": "leave", "status": "passed", "steps": [ { "label": "ananya says \"How much maternity leave do I get?\"", "output": "Ananya, here is what applies to you as an employee based in India:\n- Entitlement: 26 weeks of paid maternity leave, up to 8 weeks of it before the expected delivery date [India Leave Supplement / Maternity leave entitlement]. The global policy's 16 weeks is the floor and the India entitlement is higher, so it applies [Parental Leave Policy / Country supplements].\n- Pay: 100% of base for the full 26 weeks [India Leave Supplement / Pay during leave].\n- Eligibility: You need 80 days worked in the 12 months before the expected delivery date. You joined on Mon, Jun 10, 2024, so you qualify [India Leave Supplement / Eligibility].\n- To apply: at least 30 days before the leave starts, with maternity certificate from a registered medical practitioner stating the expected delivery date. The request goes to Rohan Mehta to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected delivery date, or the date you'd like the leave to begin.", "assertions": [ { "description": "contains \"26 weeks\"", "pass": true, "reason": "found" }, { "description": "contains \"8 weeks\"", "pass": true, "reason": "found" }, { "description": "contains \"India Leave Supplement / Maternity leave entitlement\"", "pass": true, "reason": "found" }, { "description": "contains \"Parental Leave Policy / Country supplements\"", "pass": true, "reason": "found" }, { "description": "contains \"you qualify\"", "pass": true, "reason": "found" }, { "description": "contains \"Leave of Absence Procedure\"", "pass": true, "reason": "found" }, { "description": "contains \"maternity certificate\"", "pass": true, "reason": "found" }, { "description": "contains \"Rohan Mehta\"", "pass": true, "reason": "found" }, { "description": "does not contain \"Jordan Park\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"FMLA\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"US Leave Supplement\"", "pass": true, "reason": "absent" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "trace includes resolve_asker", "pass": true, "reason": "orchestrator > resolve_asker > search_policies > search_policies > reconcile > check_eligibility > search_policies" }, { "description": "trace includes reconcile", "pass": true, "reason": "orchestrator > resolve_asker > search_policies > search_policies > reconcile > check_eligibility > search_policies" }, { "description": "trace includes check_eligibility", "pass": true, "reason": "orchestrator > resolve_asker > search_policies > search_policies > reconcile > check_eligibility > search_policies" }, { "description": "searched more than once", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 16, "passedAssertions": 16, "durationMs": 622 }, { "caseId": "leave-02", "caseName": "Sarah (US): 16 weeks paid, FMLA concurrent, no India rules", "category": "leave", "status": "passed", "steps": [ { "label": "sarah says \"How much maternity leave do I get?\"", "output": "Sarah, here is what applies to you as an employee based in the United States:\n- Entitlement: 16 weeks of fully paid leave for a birthing parent, 12 weeks for a non-birthing parent [US Leave Supplement / Company paid leave].\n- Job protection: up to 12 weeks under FMLA, running concurrently with the paid leave [US Leave Supplement / Job protection under FMLA]. State paid family leave, where it exists, is coordinated by HR so pay stays at 100% [US Leave Supplement / State programs].\n- Eligibility: You need 90 days of service. You joined on Mon, Nov 8, 2021, so you qualify [Parental Leave Policy / Eligibility].\n- To apply: at least 30 days before the leave starts, with medical certification form from your healthcare provider with the expected due date. The request goes to Jordan Park to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected due date, or the date you'd like the leave to begin.", "assertions": [ { "description": "contains \"16 weeks\"", "pass": true, "reason": "found" }, { "description": "contains \"12 weeks under FMLA\"", "pass": true, "reason": "found" }, { "description": "contains \"US Leave Supplement\"", "pass": true, "reason": "found" }, { "description": "contains \"you qualify\"", "pass": true, "reason": "found" }, { "description": "does not contain \"26 weeks\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"India Leave Supplement\"", "pass": true, "reason": "absent" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "trace includes reconcile", "pass": true, "reason": "orchestrator > resolve_asker > search_policies > search_policies > reconcile > check_eligibility > search_policies" } ] } ], "totalAssertions": 8, "passedAssertions": 8, "durationMs": 581 }, { "caseId": "leave-03", "caseName": "Same question, two countries, two answers, no cross-leak", "category": "leave", "status": "passed", "steps": [ { "label": "ananya says \"How much parental leave do I get?\"", "output": "Ananya, here is what applies to you as an employee based in India:\n- Entitlement: 26 weeks of paid maternity leave, up to 8 weeks of it before the expected delivery date [India Leave Supplement / Maternity leave entitlement]. The global policy's 16 weeks is the floor and the India entitlement is higher, so it applies [Parental Leave Policy / Country supplements].\n- Pay: 100% of base for the full 26 weeks [India Leave Supplement / Pay during leave].\n- Eligibility: You need 80 days worked in the 12 months before the expected delivery date. You joined on Mon, Jun 10, 2024, so you qualify [India Leave Supplement / Eligibility].\n- To apply: at least 30 days before the leave starts, with maternity certificate from a registered medical practitioner stating the expected delivery date. The request goes to Rohan Mehta to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected delivery date, or the date you'd like the leave to begin.", "assertions": [ { "description": "contains \"26 weeks\"", "pass": true, "reason": "found" }, { "description": "does not contain \"16 weeks of fully paid\"", "pass": true, "reason": "absent" } ] }, { "label": "sarah says \"How much parental leave do I get?\"", "output": "Sarah, here is what applies to you as an employee based in the United States:\n- Entitlement: 16 weeks of fully paid leave for a birthing parent, 12 weeks for a non-birthing parent [US Leave Supplement / Company paid leave].\n- Job protection: up to 12 weeks under FMLA, running concurrently with the paid leave [US Leave Supplement / Job protection under FMLA]. State paid family leave, where it exists, is coordinated by HR so pay stays at 100% [US Leave Supplement / State programs].\n- Eligibility: You need 90 days of service. You joined on Mon, Nov 8, 2021, so you qualify [Parental Leave Policy / Eligibility].\n- To apply: at least 30 days before the leave starts, with medical certification form from your healthcare provider with the expected due date. The request goes to Jordan Park to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected due date, or the date you'd like the leave to begin.", "assertions": [ { "description": "contains \"16 weeks\"", "pass": true, "reason": "found" }, { "description": "does not contain \"26 weeks\"", "pass": true, "reason": "absent" } ] }, { "label": "ananya says \"What is in the US Leave Supplement?\"", "output": "## Country supplements\n\nWhere local law provides more than this policy, the local entitlement applies\nand the company pays the difference. Employees based in India: see the India\nLeave Supplement. Employees based in the United States: see the US Leave\nSupplement. Contractors are not eligible for parental leave.\n\n[Parental Leave Policy]", "assertions": [ { "description": "does not contain \"1,250 hours\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"FMLA\"", "pass": true, "reason": "absent" }, { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" }, { "description": "\"US Leave Supplement\" is neither a source nor on the shelf", "pass": true, "reason": "excluded" } ] }, { "label": "sarah says \"What is in the India Leave Supplement?\"", "output": "## Country supplements\n\nWhere local law provides more than this policy, the local entitlement applies\nand the company pays the difference. Employees based in India: see the India\nLeave Supplement. Employees based in the United States: see the US Leave\nSupplement. Contractors are not eligible for parental leave.\n\n[Parental Leave Policy]", "assertions": [ { "description": "does not contain \"80 days\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"26 weeks\"", "pass": true, "reason": "absent" }, { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" }, { "description": "\"India Leave Supplement\" is neither a source nor on the shelf", "pass": true, "reason": "excluded" } ] } ], "totalAssertions": 12, "passedAssertions": 12, "durationMs": 1049 }, { "caseId": "leave-04", "caseName": "Contractor is told they are not eligible, nothing drafted", "category": "leave", "status": "passed", "steps": [ { "label": "marcus says \"How much parental leave do I get?\"", "output": "Marcus, parental leave applies to regular employees. Contractors are not eligible for parental leave [Parental Leave Policy / Country supplements]. If your contract has its own leave terms, Jordan Park or HR can confirm them.", "assertions": [ { "description": "contains \"not eligible\"", "pass": true, "reason": "found" }, { "description": "does not contain \"Want me to start\"", "pass": true, "reason": "absent" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "nothing was drafted", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 4, "passedAssertions": 4, "durationMs": 641 }, { "caseId": "leave-05", "caseName": "India: due date opens the 8-week window, start date drafts, confirm submits to manager and HR", "category": "leave", "status": "passed", "steps": [ { "label": "ananya says \"How much maternity leave do I get?\"", "output": "Ananya, here is what applies to you as an employee based in India:\n- Entitlement: 26 weeks of paid maternity leave, up to 8 weeks of it before the expected delivery date [India Leave Supplement / Maternity leave entitlement]. The global policy's 16 weeks is the floor and the India entitlement is higher, so it applies [Parental Leave Policy / Country supplements].\n- Pay: 100% of base for the full 26 weeks [India Leave Supplement / Pay during leave].\n- Eligibility: You need 80 days worked in the 12 months before the expected delivery date. You joined on Mon, Jun 10, 2024, so you qualify [India Leave Supplement / Eligibility].\n- To apply: at least 30 days before the leave starts, with maternity certificate from a registered medical practitioner stating the expected delivery date. The request goes to Rohan Mehta to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected delivery date, or the date you'd like the leave to begin.", "assertions": [ { "description": "contains \"Want me to start\"", "pass": true, "reason": "found" }, { "description": "nothing was drafted", "pass": true, "reason": "ok" } ] }, { "label": "ananya says \"My due date is Nov 20\"", "output": "Your leave can begin as early as Fri, Sep 25, 2026 (8 weeks before Fri, Nov 20, 2026) or any day up to the delivery date. Which start date would you like?", "assertions": [ { "description": "contains \"Sep 25\"", "pass": true, "reason": "found" }, { "description": "contains \"8 weeks before\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "nothing was drafted", "pass": true, "reason": "ok" }, { "description": "trace includes apply_rule", "pass": true, "reason": "orchestrator > apply_rule" } ] }, { "label": "ananya says \"Nov 2\"", "output": "**Leave of absence summary**\n**Type:** Parental leave (India)\n**Entitlement:** 26 weeks, 26 paid at 100%\n**Dates:** Mon, Nov 2, 2026 to Sun, May 2, 2027\n**Expected return:** Mon, May 3, 2027\n**Expected date:** Fri, Nov 20, 2026\n**Attach:** Maternity certificate from a registered medical practitioner stating the expected delivery date\n**Routed to:** Rohan Mehta (coverage) and HR (eligibility)\n\nNo PTO is deducted. Ready to send this to Rohan Mehta and HR? Say yes to confirm.", "assertions": [ { "description": "contains \"Expected return\"", "pass": true, "reason": "found" }, { "description": "contains \"Rohan Mehta (coverage) and HR (eligibility)\"", "pass": true, "reason": "found" }, { "description": "contains \"No PTO is deducted\"", "pass": true, "reason": "found" }, { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-11-02 to 2027-05-02, 130 days" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" }, { "description": "draft is a 26-week leave of absence", "pass": true, "reason": "ok" } ] }, { "label": "ananya says \"yes\"", "output": "Sent. Request **REQ-2026-006** is with Rohan Mehta to acknowledge coverage and with HR to confirm eligibility; HR replies within 5 business days. Please attach: Maternity certificate from a registered medical practitioner stating the expected delivery date.", "assertions": [ { "description": "request submitted to the HR system", "pass": true, "reason": "REQ-2026-006 pending" }, { "description": "submitted as parental leave with details", "pass": true, "reason": "ok" } ] }, { "label": "bob GET /api/notifications", "output": "[{\"id\":\"NOTE-2\",\"createdAt\":\"2026-09-13T20:01:56.755Z\",\"read\":true,\"toEmployeeId\":\"WD-10007\",\"kind\":\"approval_request\",\"subject\":\"Eligibility check: Ananya Iyer, parental leave 2026-11-02 to 2027-05-02\",\"body\":\"REQ-2026-006: 26 weeks (IN). Basis: India Leave Supplement / Maternity leave entitlement; Parental Leave Policy / Country supplements; India Leave Supplement / Eligibility; Leave of Absence", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "HR asked to confirm eligibility", "pass": true, "reason": "ok" } ] }, { "label": "rohan GET /api/notifications", "output": "[{\"id\":\"NOTE-1\",\"createdAt\":\"2026-09-13T20:01:56.755Z\",\"read\":true,\"toEmployeeId\":\"WD-10020\",\"kind\":\"approval_request\",\"subject\":\"Ananya Iyer requests parental leave: 2026-11-02 to 2027-05-02\",\"body\":\"Ananya Iyer submitted REQ-2026-006, 26 weeks of parental leave (2026-11-02 to 2027-05-02). HR is confirming eligibility; please acknowledge coverage.\",\"requestId\":\"REQ-2026-006\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "her India manager is asked to acknowledge coverage", "pass": true, "reason": "ok" } ] }, { "label": "jordan GET /api/notifications", "output": "[]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "the US manager is not", "pass": true, "reason": "ok" } ] }, { "label": "rohan GET /api/manager/dashboard", "output": "{\"manager\":{\"employeeId\":\"WD-10020\",\"name\":\"Rohan Mehta\",\"email\":\"rohan.mehta@acme.com\",\"employeeType\":\"full-time\",\"role\":\"manager\",\"department\":\"Engineering\",\"title\":\"Engineering Manager, Bengaluru\",\"hireDate\":\"2020-09-14\",\"location\":\"IN-KA\",\"workSchedule\":{\"days\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\"],\"hoursPerDay\":8},\"manager\":{\"id\":\"WD-10002\",\"name\":\"Dana Whitfield\",\"email\":\"dana", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "leave sits in the India manager's queue", "pass": true, "reason": "ok" }, { "description": "India team of three", "pass": true, "reason": "ok" } ] }, { "label": "jordan GET /api/manager/dashboard", "output": "{\"manager\":{\"employeeId\":\"WD-10015\",\"name\":\"Jordan Park\",\"email\":\"jordan.park@acme.com\",\"employeeType\":\"full-time\",\"role\":\"manager\",\"department\":\"Engineering\",\"title\":\"Engineering Manager\",\"hireDate\":\"2019-06-03\",\"location\":\"US-CA\",\"workSchedule\":{\"days\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\"],\"hoursPerDay\":8},\"manager\":{\"id\":\"WD-10002\",\"name\":\"Dana Whitfield\",\"email\":\"dana.whitfield@", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "not in the US manager's queue", "pass": true, "reason": "ok" } ] }, { "label": "ananya says \"What are my balances?\"", "output": "Here's where you stand, Ananya:\n- **Paid Time Off:** 13 days (104 hrs) available\n- **Sick Leave:** 8 days (64 hrs) available\n\nPending: Parental Leave Nov 2 to May 2 (130 days)\n\nWant me to find the best dates for a vacation?", "assertions": [ { "description": "contains \"13 days\"", "pass": true, "reason": "found" } ] } ], "totalAssertions": 27, "passedAssertions": 27, "durationMs": 2520 }, { "caseId": "leave-06", "caseName": "US: start date drafts 16 weeks; approval does not touch PTO", "category": "leave", "status": "passed", "steps": [ { "label": "sarah says \"How much maternity leave do I get?\"", "output": "Sarah, here is what applies to you as an employee based in the United States:\n- Entitlement: 16 weeks of fully paid leave for a birthing parent, 12 weeks for a non-birthing parent [US Leave Supplement / Company paid leave].\n- Job protection: up to 12 weeks under FMLA, running concurrently with the paid leave [US Leave Supplement / Job protection under FMLA]. State paid family leave, where it exists, is coordinated by HR so pay stays at 100% [US Leave Supplement / State programs].\n- Eligibility: You need 90 days of service. You joined on Mon, Nov 8, 2021, so you qualify [Parental Leave Policy / Eligibility].\n- To apply: at least 30 days before the leave starts, with medical certification form from your healthcare provider with the expected due date. The request goes to Jordan Park to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected due date, or the date you'd like the leave to begin.", "assertions": [ { "description": "nothing was drafted", "pass": true, "reason": "ok" } ] }, { "label": "sarah says \"Start it on Dec 1\"", "output": "**Leave of absence summary**\n**Type:** Parental leave (US)\n**Entitlement:** 16 weeks, 16 paid at 100%, 12 weeks FMLA job protection concurrent\n**Dates:** Tue, Dec 1, 2026 to Mon, Mar 22, 2027\n**Expected return:** Tue, Mar 23, 2027\n**Attach:** Medical certification form from your healthcare provider with the expected due date\n**Routed to:** Jordan Park (coverage) and HR (eligibility)\n\nNo PTO is deducted. Ready to send this to Jordan Park and HR? Say yes to confirm.", "assertions": [ { "description": "contains \"12 weeks FMLA job protection concurrent\"", "pass": true, "reason": "found" }, { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-12-01 to 2027-03-22, 80 days" }, { "description": "16-week draft ending Mar 22", "pass": true, "reason": "ok" } ] }, { "label": "sarah says \"confirm\"", "output": "Sent. Request **REQ-2026-006** is with Jordan Park to acknowledge coverage and with HR to confirm eligibility; HR replies within 5 business days. Please attach: Medical certification form from your healthcare provider with the expected due date.", "assertions": [ { "description": "request submitted to the HR system", "pass": true, "reason": "REQ-2026-006 pending" } ] }, { "label": "jordan POST /api/manager/requests/REQ-2026-006/decision", "output": "{\"requestId\":\"REQ-2026-006\",\"employeeId\":\"WD-10051\",\"employeeName\":\"Sarah Kim\",\"leaveTypeId\":\"PARENTAL\",\"leaveTypeName\":\"Parental Leave\",\"startDate\":\"2026-12-01\",\"endDate\":\"2027-03-22\",\"totalDays\":80,\"totalHours\":640,\"status\":\"approved\",\"comment\":\"Submitted via Time Off Assistant\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-13T20:01:59.050Z\",\"details\":{\"kind\":\"parental\",\"country\":\"US\",\"enti", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "approved", "pass": true, "reason": "ok" } ] }, { "label": "sarah says \"What are my balances?\"", "output": "Here's where you stand, Sarah:\n- **Paid Time Off:** 9 days (72 hrs) available\n- **Sick Leave:** 6 days (48 hrs) available\n- **Floating Holiday:** 1 days (8 hrs) available\n\nWant me to find the best dates for a vacation?", "assertions": [ { "description": "contains \"9 days\"", "pass": true, "reason": "found" } ] } ], "totalAssertions": 8, "passedAssertions": 8, "durationMs": 2409 }, { "caseId": "leave-07", "caseName": "A second date before confirming moves the start, not a new PTO request", "category": "leave", "status": "passed", "steps": [ { "label": "ananya says \"I am expecting, how much leave do I get?\"", "output": "Ananya, here is what applies to you as an employee based in India:\n- Entitlement: 26 weeks of paid maternity leave, up to 8 weeks of it before the expected delivery date [India Leave Supplement / Maternity leave entitlement]. The global policy's 16 weeks is the floor and the India entitlement is higher, so it applies [Parental Leave Policy / Country supplements].\n- Pay: 100% of base for the full 26 weeks [India Leave Supplement / Pay during leave].\n- Eligibility: You need 80 days worked in the 12 months before the expected delivery date. You joined on Mon, Jun 10, 2024, so you qualify [India Leave Supplement / Eligibility].\n- To apply: at least 30 days before the leave starts, with maternity certificate from a registered medical practitioner stating the expected delivery date. The request goes to Rohan Mehta to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected delivery date, or the date you'd like the leave to begin.", "assertions": [ { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] }, { "label": "ananya says \"Start on Nov 2\"", "output": "**Leave of absence summary**\n**Type:** Parental leave (India)\n**Entitlement:** 26 weeks, 26 paid at 100%\n**Dates:** Mon, Nov 2, 2026 to Sun, May 2, 2027\n**Expected return:** Mon, May 3, 2027\n**Attach:** Maternity certificate from a registered medical practitioner stating the expected delivery date\n**Routed to:** Rohan Mehta (coverage) and HR (eligibility)\n\nNo PTO is deducted. Ready to send this to Rohan Mehta and HR? Say yes to confirm.", "assertions": [ { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-11-02 to 2027-05-02, 130 days" } ] }, { "label": "ananya says \"Actually Nov 9\"", "output": "**Leave of absence summary**\n**Type:** Parental leave (India)\n**Entitlement:** 26 weeks, 26 paid at 100%\n**Dates:** Mon, Nov 9, 2026 to Sun, May 9, 2027\n**Expected return:** Mon, May 10, 2027\n**Attach:** Maternity certificate from a registered medical practitioner stating the expected delivery date\n**Routed to:** Rohan Mehta (coverage) and HR (eligibility)\n\nNo PTO is deducted. Ready to send this to Rohan Mehta and HR? Say yes to confirm.", "assertions": [ { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-11-09 to 2027-05-09, 130 days" }, { "description": "draft moved to Nov 9 and is still parental leave", "pass": true, "reason": "ok" } ] }, { "label": "ananya says \"never mind\"", "output": "No problem, I've cancelled that draft. Tell me when you'd like to go and I'll start again.", "assertions": [ { "description": "contains \"cancelled\"", "pass": true, "reason": "found" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "ananya GET /api/requests", "output": "[]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "nothing on file", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 8, "passedAssertions": 8, "durationMs": 2137 }, { "caseId": "safety-01", "caseName": "Nothing is submitted without an explicit yes", "category": "safety", "status": "passed", "steps": [ { "label": "alex says \"Submit PTO for Dec 21 to Dec 23\"", "output": "**Request summary**\n**Type:** Paid Time Off\n**Dates:** Mon, Dec 21, 2026 to Wed, Dec 23, 2026\n**Work days:** 3 (24 hours)\n**Balance after:** 9 days\n**Approver:** Jordan Park\n\nReady to send this to Jordan Park for approval? Say yes to confirm.", "assertions": [ { "description": "contains \"confirm\"", "pass": true, "reason": "found" }, { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-12-21 to 2026-12-23, 3 days" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"hmm, maybe\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-001\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-09-21\",\"endDate\":\"2026-09-23\",\"totalDays\":3,\"totalHours\":24,\"status\":\"pending\",\"comment\":\"Family trip\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-02T10:30:00Z\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "still one request on file", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 724 }, { "caseId": "safety-02", "caseName": "Cancel discards the draft", "category": "safety", "status": "passed", "steps": [ { "label": "alex says \"I want to take Dec 21 to Dec 23 off\"", "output": "**Request summary**\n**Type:** Paid Time Off\n**Dates:** Mon, Dec 21, 2026 to Wed, Dec 23, 2026\n**Work days:** 3 (24 hours)\n**Balance after:** 9 days\n**Approver:** Jordan Park\n\nReady to send this to Jordan Park for approval? Say yes to confirm.", "assertions": [ { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-12-21 to 2026-12-23, 3 days" } ] }, { "label": "alex says \"Actually, never mind\"", "output": "No problem, I've cancelled that draft. Tell me when you'd like to go and I'll start again.", "assertions": [ { "description": "contains \"cancelled\"", "pass": true, "reason": "found" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"yes\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-001\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-09-21\",\"endDate\":\"2026-09-23\",\"totalDays\":3,\"totalHours\":24,\"status\":\"pending\",\"comment\":\"Family trip\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-02T10:30:00Z\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "still one request on file", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 1159 }, { "caseId": "safety-03", "caseName": "Overlapping an existing request is refused", "category": "safety", "status": "passed", "steps": [ { "label": "alex says \"I want to take Sep 22 to Sep 24 off\"", "output": "You already have a Paid Time Off request (pending) from 2026-09-21 to 2026-09-23.\n\n[PTO Policy / Requesting time off]", "assertions": [ { "description": "contains \"already have\"", "pass": true, "reason": "found" }, { "description": "nothing was drafted", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 2, "passedAssertions": 2, "durationMs": 633 }, { "caseId": "flow-01", "caseName": "Alex requests, Jordan approves, Alex is told, balances move", "category": "flow", "status": "passed", "steps": [ { "label": "alex says \"I want to book a vacation\"", "output": "Alex, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Dec 28 to Dec 30** (3 PTO days, 11 days off)\n- 3 PTO days gets you 11 days off, Dec 24 through Jan 3.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25) and New Year's Eve (Dec 31) and New Year's Day (Jan 1), which don't count against PTO.\n- Priya Sharma (approved, Dec 28 to Dec 30) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Dec 21 to Dec 23** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Dec 19 through Dec 27.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Nov 23 to Nov 25** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Nov 21 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (pending, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "offers 3 date options", "pass": true, "reason": "3 options" } ] }, { "label": "alex says \"Request option 1\"", "output": "**Request summary**\n**Type:** Paid Time Off\n**Dates:** Mon, Dec 28, 2026 to Wed, Dec 30, 2026\n**Work days:** 3 (24 hours)\n**Balance after:** 9 days\n**Approver:** Jordan Park\n\nHeads up: Priya Sharma (approved, Dec 28 to Dec 30) already out in this window.\n\nReady to send this to Jordan Park for approval? Say yes to confirm.", "assertions": [ { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-12-28 to 2026-12-30, 3 days" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"Yes, send it\"", "output": "Sent. Request **REQ-2026-006** is pending with Jordan Park, who has been notified at jordan.park@acme.com. You'll see the decision here as soon as it's made.", "assertions": [ { "description": "contains \"REQ-2026-006\"", "pass": true, "reason": "found" }, { "description": "contains \"Jordan Park\"", "pass": true, "reason": "found" }, { "description": "request submitted to the HR system", "pass": true, "reason": "REQ-2026-006 pending" } ] }, { "label": "alex GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-001\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-09-21\",\"endDate\":\"2026-09-23\",\"totalDays\":3,\"totalHours\":24,\"status\":\"pending\",\"comment\":\"Family trip\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-02T10:30:00Z\"},{\"requestId\":\"REQ-2026-006\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"l", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "two requests, new one pending", "pass": true, "reason": "ok" } ] }, { "label": "jordan GET /api/notifications", "output": "[{\"id\":\"NOTE-1\",\"createdAt\":\"2026-09-13T20:02:06.031Z\",\"read\":true,\"toEmployeeId\":\"WD-10015\",\"kind\":\"approval_request\",\"subject\":\"Alex Chen requests Paid Time Off: 2026-12-28 to 2026-12-30\",\"body\":\"Alex Chen submitted REQ-2026-006 for 3 days (2026-12-28 to 2026-12-30). Note: \\\"Submitted via Time Off Assistant\\\". Approve or deny in the manager view.\",\"requestId\":\"REQ-2026-006\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "manager was notified", "pass": true, "reason": "ok" } ] }, { "label": "jordan GET /api/manager/dashboard", "output": "{\"manager\":{\"employeeId\":\"WD-10015\",\"name\":\"Jordan Park\",\"email\":\"jordan.park@acme.com\",\"employeeType\":\"full-time\",\"role\":\"manager\",\"department\":\"Engineering\",\"title\":\"Engineering Manager\",\"hireDate\":\"2019-06-03\",\"location\":\"US-CA\",\"workSchedule\":{\"days\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\"],\"hoursPerDay\":8},\"manager\":{\"id\":\"WD-10002\",\"name\":\"Dana Whitfield\",\"email\":\"dana.whitfield@", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "request waits in the queue", "pass": true, "reason": "ok" } ] }, { "label": "jordan POST /api/manager/requests/REQ-2026-006/decision", "output": "{\"requestId\":\"REQ-2026-006\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-12-28\",\"endDate\":\"2026-12-30\",\"totalDays\":3,\"totalHours\":24,\"status\":\"approved\",\"comment\":\"Submitted via Time Off Assistant\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-13T20:02:06.031Z\",\"decidedAt\":\"2026-09-13T20:02:06.040Z\"}", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "approved", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/notifications", "output": "[{\"id\":\"NOTE-2\",\"createdAt\":\"2026-09-13T20:02:06.040Z\",\"read\":true,\"toEmployeeId\":\"WD-10042\",\"kind\":\"decision\",\"subject\":\"Jordan Park approved your request REQ-2026-006\",\"body\":\"Jordan Park approved Paid Time Off for 2026-12-28 to 2026-12-30.\",\"requestId\":\"REQ-2026-006\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "employee sees the approval", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-001\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-09-21\",\"endDate\":\"2026-09-23\",\"totalDays\":3,\"totalHours\":24,\"status\":\"pending\",\"comment\":\"Family trip\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-02T10:30:00Z\"},{\"requestId\":\"REQ-2026-006\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"l", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "request now approved", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"What are my balances?\"", "output": "Here's where you stand, Alex:\n- **Paid Time Off:** 9 days (72 hrs) available, 24 hrs pending approval\n- **Sick Leave:** 8 days (64 hrs) available\n- **Floating Holiday:** 2 days (16 hrs) available\n\nPending: Paid Time Off Sep 21 to Sep 23 (3 days)\n\nWant me to find the best dates for a vacation?", "assertions": [ { "description": "contains \"9 days\"", "pass": true, "reason": "found" } ] } ], "totalAssertions": 19, "passedAssertions": 19, "durationMs": 2136 }, { "caseId": "flow-02", "caseName": "Deny with a reason returns the days and tells the employee why", "category": "flow", "status": "passed", "steps": [ { "label": "alex says \"I want to take Dec 21 to Dec 23 off\"", "output": "**Request summary**\n**Type:** Paid Time Off\n**Dates:** Mon, Dec 21, 2026 to Wed, Dec 23, 2026\n**Work days:** 3 (24 hours)\n**Balance after:** 9 days\n**Approver:** Jordan Park\n\nReady to send this to Jordan Park for approval? Say yes to confirm.", "assertions": [ { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-12-21 to 2026-12-23, 3 days" } ] }, { "label": "alex says \"yes\"", "output": "Sent. Request **REQ-2026-006** is pending with Jordan Park, who has been notified at jordan.park@acme.com. You'll see the decision here as soon as it's made.", "assertions": [ { "description": "request submitted to the HR system", "pass": true, "reason": "REQ-2026-006 pending" } ] }, { "label": "jordan POST /api/manager/requests/REQ-2026-006/decision", "output": "{\"requestId\":\"REQ-2026-006\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-12-21\",\"endDate\":\"2026-12-23\",\"totalDays\":3,\"totalHours\":24,\"status\":\"denied\",\"comment\":\"Submitted via Time Off Assistant\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-13T20:02:07.609Z\",\"decidedAt\":\"2026-09-13T20:02:07.614Z\",\"decisionNote\":\"Rele", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "denied", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/notifications", "output": "[{\"id\":\"NOTE-2\",\"createdAt\":\"2026-09-13T20:02:07.614Z\",\"read\":true,\"toEmployeeId\":\"WD-10042\",\"kind\":\"decision\",\"subject\":\"Jordan Park denied your request REQ-2026-006\",\"body\":\"Jordan Park denied Paid Time Off for 2026-12-21 to 2026-12-23. Note: \\\"Release wrap-up that week\\\".\",\"requestId\":\"REQ-2026-006\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "employee sees the reason", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-001\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-09-21\",\"endDate\":\"2026-09-23\",\"totalDays\":3,\"totalHours\":24,\"status\":\"pending\",\"comment\":\"Family trip\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-02T10:30:00Z\"},{\"requestId\":\"REQ-2026-006\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"l", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "denied with the note attached", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"What are my balances?\"", "output": "Here's where you stand, Alex:\n- **Paid Time Off:** 12 days (96 hrs) available, 24 hrs pending approval\n- **Sick Leave:** 8 days (64 hrs) available\n- **Floating Holiday:** 2 days (16 hrs) available\n\nPending: Paid Time Off Sep 21 to Sep 23 (3 days)\n\nWant me to find the best dates for a vacation?", "assertions": [ { "description": "contains \"12 days\"", "pass": true, "reason": "found" } ] } ], "totalAssertions": 9, "passedAssertions": 9, "durationMs": 1703 }, { "caseId": "flow-03", "caseName": "Contractor request reaches the manager as unpaid", "category": "flow", "status": "passed", "steps": [ { "label": "marcus says \"I want to take Nov 9 to Nov 11 off\"", "output": "As a contractor, this would be unpaid time off.\n**Request summary**\n**Type:** Unpaid Time Off\n**Dates:** Mon, Nov 9, 2026 to Wed, Nov 11, 2026\n**Work days:** 3 (24 hours)\n**Pay:** unpaid\n**Approver:** Jordan Park\n\nReady to send this to Jordan Park for approval? Say yes to confirm.", "assertions": [ { "description": "contains \"unpaid\"", "pass": true, "reason": "found" }, { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-11-09 to 2026-11-11, 3 days" } ] }, { "label": "marcus says \"confirm\"", "output": "Sent. Request **REQ-2026-006** is pending with Jordan Park, who has been notified at jordan.park@acme.com. You'll see the decision here as soon as it's made.", "assertions": [ { "description": "request submitted to the HR system", "pass": true, "reason": "REQ-2026-006 pending" } ] }, { "label": "jordan GET /api/manager/dashboard", "output": "{\"manager\":{\"employeeId\":\"WD-10015\",\"name\":\"Jordan Park\",\"email\":\"jordan.park@acme.com\",\"employeeType\":\"full-time\",\"role\":\"manager\",\"department\":\"Engineering\",\"title\":\"Engineering Manager\",\"hireDate\":\"2019-06-03\",\"location\":\"US-CA\",\"workSchedule\":{\"days\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\"],\"hoursPerDay\":8},\"manager\":{\"id\":\"WD-10002\",\"name\":\"Dana Whitfield\",\"email\":\"dana.whitfield@", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "unpaid request in the queue", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 1099 }, { "caseId": "manager-01", "caseName": "Dashboard shape for a team of five", "category": "manager", "status": "passed", "steps": [ { "label": "jordan GET /api/manager/dashboard", "output": "{\"manager\":{\"employeeId\":\"WD-10015\",\"name\":\"Jordan Park\",\"email\":\"jordan.park@acme.com\",\"employeeType\":\"full-time\",\"role\":\"manager\",\"department\":\"Engineering\",\"title\":\"Engineering Manager\",\"hireDate\":\"2019-06-03\",\"location\":\"US-CA\",\"workSchedule\":{\"days\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\"],\"hoursPerDay\":8},\"manager\":{\"id\":\"WD-10002\",\"name\":\"Dana Whitfield\",\"email\":\"dana.whitfield@", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "five direct reports", "pass": true, "reason": "ok" }, { "description": "two requests pending at start", "pass": true, "reason": "ok" }, { "description": "approved leave on the calendar", "pass": true, "reason": "ok" }, { "description": "denied request in history with its note", "pass": true, "reason": "ok" }, { "description": "holidays in the horizon", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 3 }, { "caseId": "manager-02", "caseName": "Only the manager can decide, and only once", "category": "manager", "status": "passed", "steps": [ { "label": "alex POST /api/manager/requests/REQ-2026-003/decision", "output": "{\"error\":\"Manager access only\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] }, { "label": "bob POST /api/manager/requests/REQ-2026-003/decision", "output": "{\"error\":\"Manager access only\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] }, { "label": "jordan POST /api/manager/requests/REQ-2026-003/decision", "output": "{\"requestId\":\"REQ-2026-003\",\"employeeId\":\"WD-10063\",\"employeeName\":\"David Lee\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-11-23\",\"endDate\":\"2026-11-25\",\"totalDays\":3,\"totalHours\":24,\"status\":\"approved\",\"comment\":\"Thanksgiving with in-laws\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-04T18:40:00Z\",\"decidedAt\":\"2026-09-13T20:02:09.282Z\"}", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" } ] }, { "label": "jordan POST /api/manager/requests/REQ-2026-003/decision", "output": "{\"error\":\"Request is already approved\"}", "assertions": [ { "description": "HTTP 409", "pass": true, "reason": "got 409" } ] }, { "label": "jordan POST /api/manager/requests/REQ-2026-003/decision", "output": "{\"error\":\"decision must be approved or denied\"}", "assertions": [ { "description": "HTTP 400", "pass": true, "reason": "got 400" } ] }, { "label": "alex GET /api/manager/dashboard", "output": "{\"error\":\"Manager or HR access only\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 9 }, { "caseId": "manager-03", "caseName": "Coverage math in the planner matches the team calendar", "category": "manager", "status": "passed", "steps": [ { "label": "jordan POST /api/manager/requests/REQ-2026-003/decision", "output": "{\"requestId\":\"REQ-2026-003\",\"employeeId\":\"WD-10063\",\"employeeName\":\"David Lee\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-11-23\",\"endDate\":\"2026-11-25\",\"totalDays\":3,\"totalHours\":24,\"status\":\"approved\",\"comment\":\"Thanksgiving with in-laws\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-04T18:40:00Z\",\"decidedAt\":\"2026-09-13T20:02:09.287Z\"}", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" } ] }, { "label": "alex says \"3 days around Thanksgiving\"", "output": "Alex, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Dec 28 to Dec 30** (3 PTO days, 11 days off)\n- 3 PTO days gets you 11 days off, Dec 24 through Jan 3.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25) and New Year's Eve (Dec 31) and New Year's Day (Jan 1), which don't count against PTO.\n- Priya Sharma (approved, Dec 28 to Dec 30) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Nov 23 to Nov 25** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Nov 21 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- David Lee (approved, Nov 23 to Nov 25) is out. Team would be at 3 of 5, which meets the 60% coverage minimum.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Dec 21 to Dec 23** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Dec 19 through Dec 27.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "contains \"David Lee (approved\"", "pass": true, "reason": "found" }, { "description": "offers 3 date options", "pass": true, "reason": "3 options" } ] } ], "totalAssertions": 3, "passedAssertions": 3, "durationMs": 573 }, { "caseId": "manager-04", "caseName": "Manager works the queue from the Ask thread: coverage, approve, notify", "category": "manager", "status": "passed", "steps": [ { "label": "ananya says \"How much maternity leave do I get?\"", "output": "Ananya, here is what applies to you as an employee based in India:\n- Entitlement: 26 weeks of paid maternity leave, up to 8 weeks of it before the expected delivery date [India Leave Supplement / Maternity leave entitlement]. The global policy's 16 weeks is the floor and the India entitlement is higher, so it applies [Parental Leave Policy / Country supplements].\n- Pay: 100% of base for the full 26 weeks [India Leave Supplement / Pay during leave].\n- Eligibility: You need 80 days worked in the 12 months before the expected delivery date. You joined on Mon, Jun 10, 2024, so you qualify [India Leave Supplement / Eligibility].\n- To apply: at least 30 days before the leave starts, with maternity certificate from a registered medical practitioner stating the expected delivery date. The request goes to Rohan Mehta to acknowledge coverage and to HR to confirm eligibility [Leave of Absence Procedure / How to apply].\n\nWant me to start the request? Tell me your expected delivery date, or the date you'd like the leave to begin.", "assertions": [] }, { "label": "ananya says \"Sep 30\"", "output": "**Leave of absence summary**\n**Type:** Parental leave (India)\n**Entitlement:** 26 weeks, 26 paid at 100%\n**Dates:** Wed, Sep 30, 2026 to Tue, Mar 30, 2027\n**Expected return:** Wed, Mar 31, 2027\n**Attach:** Maternity certificate from a registered medical practitioner stating the expected delivery date\n**Routed to:** Rohan Mehta (coverage) and HR (eligibility)\n\nHeads up: this starts in 17 days, inside the 30-day notice window [Leave of Absence Procedure / How to apply]. It can still be submitted as soon as practicable.\n\nNo PTO is deducted. Ready to send this to Rohan Mehta and HR? Say yes to confirm.", "assertions": [ { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-09-30 to 2027-03-30, 130 days" } ] }, { "label": "ananya says \"yes\"", "output": "Sent. Request **REQ-2026-006** is with Rohan Mehta to acknowledge coverage and with HR to confirm eligibility; HR replies within 5 business days. Please attach: Maternity certificate from a registered medical practitioner stating the expected delivery date.", "assertions": [ { "description": "request submitted to the HR system", "pass": true, "reason": "REQ-2026-006 pending" } ] }, { "label": "rohan says \"show me coverage\"", "output": "Coverage for your team (3 on the roster):\n- Ananya Iyer: 26 weeks of parental leave, Sep 30 to Mar 30 (pending, REQ-2026-006). Also out: Arjun Rao (approved, Nov 9 to Nov 13). Team would be at 1 of 3 on Nov 9 to Nov 13, below the 60% minimum.\n\nAlready approved: Arjun Rao Nov 9 to Nov 13.\n\nParental leave is an entitlement, so a thin week is a planning flag, not a reason to deny. HR confirms eligibility separately [Manager Guide: Team Coverage / Deciding between overlapping requests].\n\nSay \"approve Ananya\" or \"deny Ananya because ...\" and I'll draft the decision for you to confirm.", "assertions": [ { "description": "contains \"3 on the roster\"", "pass": true, "reason": "found" }, { "description": "contains \"Ananya Iyer\"", "pass": true, "reason": "found" }, { "description": "contains \"Arjun Rao\"", "pass": true, "reason": "found" }, { "description": "contains \"1 of 3\"", "pass": true, "reason": "found" }, { "description": "contains \"below\"", "pass": true, "reason": "found" }, { "description": "contains \"planning flag\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "trace includes team_coverage", "pass": true, "reason": "orchestrator > get_pending_approvals > team_coverage" } ] }, { "label": "rohan says \"what is waiting on me\"", "output": "**Waiting on you (1):**\n- Ananya Iyer: 26 weeks of parental leave, Sep 30 to Mar 30 (pending, REQ-2026-006). Also out: Arjun Rao (approved, Nov 9 to Nov 13). Team would be at 1 of 3 on Nov 9 to Nov 13, below the 60% minimum.\n\nSay \"approve Ananya\" or \"deny Ananya because ...\" to draft a decision.", "assertions": [ { "description": "contains \"Waiting on you (1)\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] }, { "label": "rohan says \"approve Ananya\"", "output": "**Decision to confirm**\n**Approve:** REQ-2026-006, Ananya Iyer\n**What:** 26 weeks of parental leave, Wed, Sep 30, 2026 to Tue, Mar 30, 2027\n**Coverage:** 1 of 3 present at the thinnest point, below the minimum\n**Note:** you are acknowledging coverage; HR confirms eligibility.\n\nAnanya will be told right away. Say yes to confirm.", "assertions": [ { "description": "contains \"Decision to confirm\"", "pass": true, "reason": "found" }, { "description": "contains \"Approve:\"", "pass": true, "reason": "found" }, { "description": "contains \"Say yes to confirm\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "ananya GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-006\",\"employeeId\":\"WD-10077\",\"employeeName\":\"Ananya Iyer\",\"leaveTypeId\":\"PARENTAL\",\"leaveTypeName\":\"Parental Leave\",\"startDate\":\"2026-09-30\",\"endDate\":\"2027-03-30\",\"totalDays\":130,\"totalHours\":1040,\"status\":\"pending\",\"comment\":\"Submitted via Time Off Assistant\",\"managerName\":\"Rohan Mehta\",\"submittedAt\":\"2026-09-13T20:02:11.432Z\",\"details\":{\"kind\":\"parental\",\"country\":\"IN\",\"", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "still pending before the yes", "pass": true, "reason": "ok" } ] }, { "label": "rohan says \"yes\"", "output": "Approved. Ananya Iyer has been told. HR will confirm eligibility and reply within 5 business days.", "assertions": [ { "description": "contains \"Approved\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" }, { "description": "trace includes decide_request", "pass": true, "reason": "orchestrator > decide_request" } ] }, { "label": "ananya GET /api/notifications", "output": "[{\"id\":\"NOTE-3\",\"createdAt\":\"2026-09-13T20:02:13.567Z\",\"read\":true,\"toEmployeeId\":\"WD-10077\",\"kind\":\"decision\",\"subject\":\"Rohan Mehta approved your request REQ-2026-006\",\"body\":\"Rohan Mehta approved Parental Leave for 2026-09-30 to 2027-03-30.\",\"requestId\":\"REQ-2026-006\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "Ananya is told", "pass": true, "reason": "ok" } ] }, { "label": "ananya GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-006\",\"employeeId\":\"WD-10077\",\"employeeName\":\"Ananya Iyer\",\"leaveTypeId\":\"PARENTAL\",\"leaveTypeName\":\"Parental Leave\",\"startDate\":\"2026-09-30\",\"endDate\":\"2027-03-30\",\"totalDays\":130,\"totalHours\":1040,\"status\":\"approved\",\"comment\":\"Submitted via Time Off Assistant\",\"managerName\":\"Rohan Mehta\",\"submittedAt\":\"2026-09-13T20:02:11.432Z\",\"details\":{\"kind\":\"parental\",\"country\":\"IN\",", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "approved on file", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 26, "passedAssertions": 26, "durationMs": 3715 }, { "caseId": "manager-05", "caseName": "Deny needs a reason; cancel discards; HR cannot decide", "category": "manager", "status": "passed", "steps": [ { "label": "jordan says \"deny David\"", "output": "What should David be told? Say \"deny David because ...\" and I'll draft it.", "assertions": [ { "description": "contains \"What should David be told\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] }, { "label": "jordan says \"deny David because release wrap-up that week\"", "output": "**Decision to confirm**\n**Deny:** REQ-2026-003, David Lee\n**What:** Paid Time Off, 3 days, Mon, Nov 23, 2026 to Wed, Nov 25, 2026\n**Coverage:** 4 of 5 present\n**Reason to the employee:** \"release wrap-up that week\"\n\nDavid will be told right away. Say yes to confirm.", "assertions": [ { "description": "contains \"Deny:\"", "pass": true, "reason": "found" }, { "description": "contains \"release wrap-up that week\"", "pass": true, "reason": "found" } ] }, { "label": "jordan says \"never mind\"", "output": "Discarded. Nothing was decided.", "assertions": [ { "description": "contains \"Discarded\"", "pass": true, "reason": "found" } ] }, { "label": "jordan GET /api/manager/dashboard", "output": "{\"manager\":{\"employeeId\":\"WD-10015\",\"name\":\"Jordan Park\",\"email\":\"jordan.park@acme.com\",\"employeeType\":\"full-time\",\"role\":\"manager\",\"department\":\"Engineering\",\"title\":\"Engineering Manager\",\"hireDate\":\"2019-06-03\",\"location\":\"US-CA\",\"workSchedule\":{\"days\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\"],\"hoursPerDay\":8},\"manager\":{\"id\":\"WD-10002\",\"name\":\"Dana Whitfield\",\"email\":\"dana.whitfield@", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "David still pending", "pass": true, "reason": "ok" } ] }, { "label": "jordan says \"approve REQ-2026-003\"", "output": "**Decision to confirm**\n**Approve:** REQ-2026-003, David Lee\n**What:** Paid Time Off, 3 days, Mon, Nov 23, 2026 to Wed, Nov 25, 2026\n**Coverage:** 4 of 5 present\n\nDavid will be told right away. Say yes to confirm.", "assertions": [ { "description": "contains \"David Lee\"", "pass": true, "reason": "found" } ] }, { "label": "jordan says \"yes\"", "output": "Approved. David Lee has been told.", "assertions": [ { "description": "contains \"Approved\"", "pass": true, "reason": "found" } ] }, { "label": "bob says \"approve Alex\"", "output": "Only the employee's manager can approve or deny. HR confirms eligibility; you can see every request under Org overview.", "assertions": [ { "description": "contains \"Only the employee\"", "pass": true, "reason": "found" }, { "description": "contains \"manager\"", "pass": true, "reason": "found" }, { "description": "routed to the time-off agent", "pass": true, "reason": "agent (offline)" } ] }, { "label": "alex GET /api/requests", "output": "[{\"requestId\":\"REQ-2026-001\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"leaveTypeId\":\"PTO\",\"leaveTypeName\":\"Paid Time Off\",\"startDate\":\"2026-09-21\",\"endDate\":\"2026-09-23\",\"totalDays\":3,\"totalHours\":24,\"status\":\"pending\",\"comment\":\"Family trip\",\"managerName\":\"Jordan Park\",\"submittedAt\":\"2026-09-02T10:30:00Z\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "Alex untouched", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 14, "passedAssertions": 14, "durationMs": 3467 }, { "caseId": "hr-01", "caseName": "HR sees the whole org, read-only", "category": "hr", "status": "passed", "steps": [ { "label": "bob GET /api/manager/dashboard", "output": "{\"manager\":{\"employeeId\":\"WD-10007\",\"name\":\"Bob Rivera\",\"email\":\"bob.rivera@acme.com\",\"employeeType\":\"full-time\",\"role\":\"hr_admin\",\"department\":\"People\",\"title\":\"HR Business Partner\",\"hireDate\":\"2018-02-12\",\"location\":\"US-CA\",\"workSchedule\":{\"days\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\"],\"hoursPerDay\":8},\"manager\":{\"id\":\"WD-10002\",\"name\":\"Dana Whitfield\",\"email\":\"dana.whitfield@acme.c", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "everyone but Bob on the roster (Sam Okafor joined in use case 04)", "pass": true, "reason": "ok" }, { "description": "both pending requests visible", "pass": true, "reason": "ok" }, { "description": "roster spans departments", "pass": true, "reason": "ok" } ] }, { "label": "bob POST /api/manager/requests/REQ-2026-001/decision", "output": "{\"error\":\"Manager access only\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 3 }, { "caseId": "hr-02", "caseName": "HR admin can still request time off like anyone", "category": "hr", "status": "passed", "steps": [ { "label": "bob says \"I want to book a vacation\"", "output": "Bob, you have 12 PTO days available.\nEarliest start under the 14-day notice rule is Sun, Sep 27, 2026 (PTO Policy / Requesting time off).\nYou can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 (PTO Policy / Carryover).\nPolicy check: You may carry over up to 5 unused PTO days into the next calendar year. Days beyond the cap are forfeited on December 31 and are not paid out. [PTO Policy / Carryover]\n\nHere are the best windows I found:\n\n**Option 1: Dec 28 to Dec 30** (3 PTO days, 11 days off)\n- 3 PTO days gets you 11 days off, Dec 24 through Jan 3.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25) and New Year's Eve (Dec 31) and New Year's Day (Jan 1), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 2: Nov 23 to Nov 25** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Nov 21 through Nov 29.\n- Includes Thanksgiving (Nov 26) and Day After Thanksgiving (Nov 27), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\n**Option 3: Dec 21 to Dec 23** (3 PTO days, 9 days off)\n- 3 PTO days gets you 9 days off, Dec 19 through Dec 27.\n- Includes Christmas Eve (Dec 24) and Christmas Day (Dec 25), which don't count against PTO.\n- Nobody else on the team is out that week.\n- Falls before Dec 31, so it uses days you would otherwise forfeit.\n\nMy pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.", "assertions": [ { "description": "contains \"12 PTO days\"", "pass": true, "reason": "found" }, { "description": "offers 3 date options", "pass": true, "reason": "3 options" } ] }, { "label": "bob says \"Request option 2\"", "output": "**Request summary**\n**Type:** Paid Time Off\n**Dates:** Mon, Nov 23, 2026 to Wed, Nov 25, 2026\n**Work days:** 3 (24 hours)\n**Balance after:** 9 days\n**Approver:** Dana Whitfield\n\nReady to send this to Dana Whitfield for approval? Say yes to confirm.", "assertions": [ { "description": "contains \"Dana Whitfield\"", "pass": true, "reason": "found" }, { "description": "a request is drafted and awaiting confirmation", "pass": true, "reason": "2026-11-23 to 2026-11-25, 3 days" } ] } ], "totalAssertions": 4, "passedAssertions": 4, "durationMs": 938 }, { "caseId": "help-01", "caseName": "\"my laptop is not working\" routes to the help agent and offers the article first", "category": "help", "status": "passed", "steps": [ { "label": "alex says \"My laptop is not working, it keeps freezing\"", "output": "The knowledge base has an article that matches: **Laptop freezes or stops responding** [KB-102]. Two steps, about five minutes. Try them and tell me whether it worked. If it does not, I will log a ticket with what you already tried, so IT does not ask you to do it twice.", "assertions": [ { "description": "contains \"Laptop freezes\"", "pass": true, "reason": "found" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "offers article KB-102", "pass": true, "reason": "KB-102" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" }, { "description": "trace includes search_kb", "pass": true, "reason": "orchestrator > search_kb" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 555 }, { "caseId": "help-02", "caseName": "Policy phrasings about devices stay with policy search", "category": "help", "status": "passed", "steps": [ { "label": "alex says \"What is the policy on personal devices?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" } ] } ], "totalAssertions": 1, "passedAssertions": 1, "durationMs": 10 }, { "caseId": "help-03", "caseName": "The contractor gets no employees-only badge article and the abstain wording", "category": "help", "status": "passed", "steps": [ { "label": "marcus says \"My badge is not working at the door\"", "output": "Nothing in the knowledge base you can see covers that. I can log a ticket instead:\n**Subject:** My badge is not working at the door\n**Category:** Other · **Priority:** Medium\n**Already tried:** nothing yet\nLog it?", "assertions": [ { "description": "contains \"Nothing in the knowledge base you can see\"", "pass": true, "reason": "found" }, { "description": "does not contain \"KB-106\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"Badge access\"", "pass": true, "reason": "absent" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "no article offered", "pass": true, "reason": "none" }, { "description": "a ticket is drafted and waits for a yes", "pass": true, "reason": "Other, tried 0" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 518 }, { "caseId": "help-04", "caseName": "\"Still broken\" after the article drafts a ticket carrying the tried steps; a stray \"maybe\" logs nothing", "category": "help", "status": "passed", "steps": [ { "label": "alex says \"My laptop keeps freezing\"", "output": "The knowledge base has an article that matches: **Laptop freezes or stops responding** [KB-102]. Two steps, about five minutes. Try them and tell me whether it worked. If it does not, I will log a ticket with what you already tried, so IT does not ask you to do it twice.", "assertions": [ { "description": "offers article KB-102", "pass": true, "reason": "KB-102" } ] }, { "label": "alex says \"I installed the update, still freezing\"", "output": "Sorry about that. Here is the ticket I would log:\n**Subject:** Laptop freezes after restart and endpoint agent update 7.4.1\n**Category:** Hardware, laptop · **Priority:** Medium\n**Already tried:** Restart; Endpoint agent update 7.4.1 (from KB-102)\nLog it?", "assertions": [ { "description": "contains \"Already tried\"", "pass": true, "reason": "found" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "a ticket is drafted and waits for a yes", "pass": true, "reason": "Hardware, laptop, tried 2" }, { "description": "draft carries 2 tried steps", "pass": true, "reason": "2" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" }, { "description": "trace includes propose_ticket", "pass": true, "reason": "orchestrator > propose_ticket" } ] }, { "label": "alex says \"sounds good, maybe later\"", "output": "Sorry about that. Here is the ticket I would log:\n**Subject:** Laptop freezes after restart and endpoint agent update 7.4.1\n**Category:** Hardware, laptop · **Priority:** Medium\n**Already tried:** Restart; Endpoint agent update 7.4.1 (from KB-102)\nLog it?\n\n(Say yes to log it, or no to drop it.)", "assertions": [ { "description": "no ticket was logged", "pass": true, "reason": "ok" }, { "description": "a ticket is drafted and waits for a yes", "pass": true, "reason": "Hardware, laptop, tried 2" } ] }, { "label": "alex GET /api/tickets", "output": "[]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "nothing logged for Alex", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 11, "passedAssertions": 11, "durationMs": 950 }, { "caseId": "help-05", "caseName": "\"Yes\" logs IT-1047 owned by Sam with the tried steps attached", "category": "help", "status": "passed", "steps": [ { "label": "alex says \"My laptop keeps freezing\"", "output": "The knowledge base has an article that matches: **Laptop freezes or stops responding** [KB-102]. Two steps, about five minutes. Try them and tell me whether it worked. If it does not, I will log a ticket with what you already tried, so IT does not ask you to do it twice.", "assertions": [ { "description": "offers article KB-102", "pass": true, "reason": "KB-102" } ] }, { "label": "alex says \"Still broken, I tried the steps\"", "output": "Sorry about that. Here is the ticket I would log:\n**Subject:** Laptop freezes after restart and endpoint agent update 7.4.1\n**Category:** Hardware, laptop · **Priority:** Medium\n**Already tried:** Restart; Endpoint agent update 7.4.1 (from KB-102)\nLog it?", "assertions": [ { "description": "a ticket is drafted and waits for a yes", "pass": true, "reason": "Hardware, laptop, tried 2" } ] }, { "label": "alex says \"Yes, log it\"", "output": "Logged. Ticket **IT-1047** is in the IT queue, owned by Sam Okafor, with the steps you tried attached. Ask me \"where is my ticket\" any time, or say \"talk to a person\" if you cannot wait.", "assertions": [ { "description": "contains \"IT-1047\"", "pass": true, "reason": "found" }, { "description": "contains \"Sam Okafor\"", "pass": true, "reason": "found" }, { "description": "ticket IT-1047 open", "pass": true, "reason": "IT-1047 open" }, { "description": "trace includes submit_ticket", "pass": true, "reason": "orchestrator > submit_ticket" } ] }, { "label": "alex GET /api/tickets", "output": "[{\"ticketId\":\"IT-1047\",\"requesterId\":\"WD-10042\",\"requesterName\":\"Alex Chen\",\"subject\":\"Laptop freezes after restart and endpoint agent update 7.4.1\",\"category\":\"Hardware, laptop\",\"priority\":\"Medium\",\"status\":\"open\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[\"Restart\",\"Endpoint agent update 7.4.1\"],\"offeredArticle\":\"KB-102\",\"tags\":[\"laptop\",\"freeze\",\"endpoint-agent-7.4.1\"],\"replies\":[{\"", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "IT-1047 open with 2 tried steps and KB-102 offered", "pass": true, "reason": "ok" } ] }, { "label": "sam GET /api/notifications", "output": "[{\"id\":\"NOTE-1\",\"createdAt\":\"2026-09-13T20:02:21.449Z\",\"read\":true,\"toEmployeeId\":\"WD-10099\",\"kind\":\"ticket_update\",\"subject\":\"New ticket IT-1047 from Alex Chen\",\"body\":\"Laptop freezes after restart and endpoint agent update 7.4.1 (Hardware, laptop, Medium). Already tried: Restart; Endpoint agent update 7.4.1.\",\"requestId\":\"IT-1047\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "Sam was told", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"Where is my ticket?\"", "output": "1 open:\n**IT-1047** · Laptop freezes after restart and endpoint agent update 7.4.1 · open · owned by Sam Okafor", "assertions": [ { "description": "contains \"IT-1047\"", "pass": true, "reason": "found" }, { "description": "contains \"open\"", "pass": true, "reason": "found" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" } ] } ], "totalAssertions": 13, "passedAssertions": 13, "durationMs": 1815 }, { "caseId": "help-06", "caseName": "Sam opens IT-1045 and five cases like it are named, with the cause from the resolved ones", "category": "help", "status": "passed", "steps": [ { "label": "sam says \"Open IT-1045\"", "output": "**IT-1045**, Arjun Rao, \"Laptop not responding, tried the KB steps\", Medium, opened 1 day ago, open. Already tried: Restart; Endpoint agent update 7.4.1 (from the article). Offered KB-102 first.\n\n**4 other tickets in the last 14 days look like this** (same category, same symptom, every one mentions the endpoint agent 7.4 update): 2 resolved, 2 still open.\n- IT-1039 Priya Sharma, \"Laptop freezing since the endpoint update\", resolved\n- IT-1040 David Lee, \"Machine hangs every hour after installing 7.4.1\", resolved\n- IT-1042 Sarah Kim, \"Laptop keeps freezing, restart does not help\", open\n- IT-1044 Kavya Nair, \"Cursor stuck and laptop unresponsive after update\", open\n\nThe article these employees were offered (KB-102) tells them to install 7.4.1, which is the version that freezes.\n\nLikely cause, from the resolved tickets: Rolled the endpoint agent back to 7.3.9 from the self-service portal and restarted; the freeze stopped. 7.4.1 is the cause on laptops with the 2024 firmware.\n\nWant me to draft a help article so the next employee can do this themselves?", "assertions": [ { "description": "contains \"IT-1039\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1040\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1042\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1044\"", "pass": true, "reason": "found" }, { "description": "contains \"7.3.9\"", "pass": true, "reason": "found" }, { "description": "contains \"draft a help article\"", "pass": true, "reason": "found" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "4 similar cases named", "pass": true, "reason": "4: IT-1039 IT-1040 IT-1042 IT-1044" }, { "description": "trace includes find_similar_tickets", "pass": true, "reason": "orchestrator > open_ticket > find_similar_tickets" } ] } ], "totalAssertions": 9, "passedAssertions": 9, "durationMs": 383 }, { "caseId": "help-07", "caseName": "An employee cannot open another employee's ticket", "category": "help", "status": "passed", "steps": [ { "label": "alex says \"Open IT-1039\"", "output": "I can only show tickets you logged. IT-1039 is not one of yours.", "assertions": [ { "description": "contains \"not one of yours\"", "pass": true, "reason": "found" }, { "description": "does not contain \"Priya\"", "pass": true, "reason": "absent" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/support/queue", "output": "{\"error\":\"IT support or HR access only\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 507 }, { "caseId": "help-08", "caseName": "Publish: Sam's yes publishes KB-108, replies on the open tickets, and the next laptop question gets KB-108 not KB-102", "category": "help", "status": "passed", "steps": [ { "label": "alex says \"My laptop keeps freezing\"", "output": "The knowledge base has an article that matches: **Laptop freezes or stops responding** [KB-102]. Two steps, about five minutes. Try them and tell me whether it worked. If it does not, I will log a ticket with what you already tried, so IT does not ask you to do it twice.", "assertions": [ { "description": "offers article KB-102", "pass": true, "reason": "KB-102" } ] }, { "label": "alex says \"Still broken\"", "output": "Sorry about that. Here is the ticket I would log:\n**Subject:** Laptop freezes after restart and endpoint agent update 7.4.1\n**Category:** Hardware, laptop · **Priority:** Medium\n**Already tried:** Restart; Endpoint agent update 7.4.1 (from KB-102)\nLog it?", "assertions": [ { "description": "a ticket is drafted and waits for a yes", "pass": true, "reason": "Hardware, laptop, tried 2" } ] }, { "label": "alex says \"Yes\"", "output": "Logged. Ticket **IT-1047** is in the IT queue, owned by Sam Okafor, with the steps you tried attached. Ask me \"where is my ticket\" any time, or say \"talk to a person\" if you cannot wait.", "assertions": [ { "description": "ticket IT-1047", "pass": true, "reason": "IT-1047 open" } ] }, { "label": "sam says \"Open IT-1047\"", "output": "**IT-1047**, Alex Chen, \"Laptop freezes after restart and endpoint agent update 7.4.1\", Medium, opened just now, open. Already tried: Restart; Endpoint agent update 7.4.1. Offered KB-102 first.\n\n**5 other tickets in the last 14 days look like this** (same category, same symptom, every one mentions the endpoint agent 7.4 update): 2 resolved, 3 still open.\n- IT-1039 Priya Sharma, \"Laptop freezing since the endpoint update\", resolved\n- IT-1040 David Lee, \"Machine hangs every hour after installing 7.4.1\", resolved\n- IT-1042 Sarah Kim, \"Laptop keeps freezing, restart does not help\", open\n- IT-1044 Kavya Nair, \"Cursor stuck and laptop unresponsive after update\", open\n- IT-1045 Arjun Rao, \"Laptop not responding, tried the KB steps\", open\n\nThe article these employees were offered (KB-102) tells them to install 7.4.1, which is the version that freezes.\n\nLikely cause, from the resolved tickets: Rolled the endpoint agent back to 7.3.9 from the self-service portal and restarted; the freeze stopped. 7.4.1 is the cause on laptops with the 2024 firmware.\n\nWant me to draft a help article so the next employee can do this themselves?", "assertions": [ { "description": "contains \"5 other tickets\"", "pass": true, "reason": "found" }, { "description": "5 similar cases named", "pass": true, "reason": "5: IT-1039 IT-1040 IT-1042 IT-1044 IT-1045" } ] }, { "label": "sam says \"Yes, draft the article\"", "output": "Draft article:\n**Title:** Laptop freezes after endpoint agent 7.4.1: roll back to 7.3.9\n**Applies to:** everyone · **Symptom:** laptop freezes or stops responding after the 7.4.1 update\n**Steps:**\n1. Open the self-service portal and choose Endpoint agent, then Versions.\n2. Select 7.3.9 and install it. This takes about two minutes.\n3. Restart the laptop. If it freezes again after the restart, log a ticket and say \"rollback done\".\n**Replaces:** KB-102\nPublish it? It goes to the knowledge base, and the 4 open tickets on this get a reply with the link.", "assertions": [ { "description": "contains \"roll back to 7.3.9\"", "pass": true, "reason": "found" }, { "description": "contains \"Publish it?\"", "pass": true, "reason": "found" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "trace includes propose_article", "pass": true, "reason": "orchestrator > propose_article" } ] }, { "label": "sam says \"hmm, let me think\"", "output": "Draft article:\n**Title:** Laptop freezes after endpoint agent 7.4.1: roll back to 7.3.9\n**Applies to:** everyone · **Symptom:** laptop freezes or stops responding after the 7.4.1 update\n**Steps:**\n1. Open the self-service portal and choose Endpoint agent, then Versions.\n2. Select 7.3.9 and install it. This takes about two minutes.\n3. Restart the laptop. If it freezes again after the restart, log a ticket and say \"rollback done\".\n**Replaces:** KB-102\nPublish it? It goes to the knowledge base, and the 4 open tickets on this get a reply with the link.\n\n(Say publish to put it in the knowledge base, or no to drop it.)", "assertions": [ { "description": "does not contain \"Published\"", "pass": true, "reason": "absent" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" } ] }, { "label": "sam says \"Publish\"", "output": "Published **KB-108**. Replied on IT-1047, IT-1042, IT-1044, IT-1045 with the article; each requester was notified. KB-102 now points to it. The tickets stay open until each employee confirms.", "assertions": [ { "description": "contains \"IT-1047\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1042\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1044\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1045\"", "pass": true, "reason": "found" }, { "description": "contains \"KB-102 now points to it\"", "pass": true, "reason": "found" }, { "description": "published KB-108", "pass": true, "reason": "KB-108" }, { "description": "trace includes publish_article", "pass": true, "reason": "orchestrator > publish_article > reply_on_tickets" }, { "description": "trace includes reply_on_tickets", "pass": true, "reason": "orchestrator > publish_article > reply_on_tickets" } ] }, { "label": "alex GET /api/notifications", "output": "[{\"id\":\"NOTE-2\",\"createdAt\":\"2026-09-13T20:02:25.773Z\",\"read\":true,\"toEmployeeId\":\"WD-10042\",\"kind\":\"ticket_update\",\"subject\":\"Sam Okafor replied on IT-1047\",\"body\":\"A fix is published: KB-108 \\\"Laptop freezes after endpoint agent 7.4.1: roll back to 7.3.9\\\". Follow the steps and reply \\\"it worked\\\" to close this ticket.\",\"requestId\":\"IT-1047\"}]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "Alex was told about the fix", "pass": true, "reason": "ok" } ] }, { "label": "sarah GET /api/tickets", "output": "[{\"ticketId\":\"IT-1042\",\"requesterId\":\"WD-10051\",\"requesterName\":\"Sarah Kim\",\"subject\":\"Laptop keeps freezing, restart does not help\",\"category\":\"Hardware, laptop\",\"priority\":\"Medium\",\"status\":\"in_progress\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[\"Restart\",\"Endpoint agent update 7.4.1 (from the article)\"],\"offeredArticle\":\"KB-102\",\"tags\":[\"laptop\",\"freeze\",\"endpoint-agent-7.4.1\"],\"re", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "IT-1042 carries the templated reply", "pass": true, "reason": "ok" } ] }, { "label": "priya says \"My laptop is freezing again\"", "output": "There is a new article for exactly this, published by Sam Okafor just now: **Laptop freezes after endpoint agent 7.4.1: roll back to 7.3.9** [KB-108]. 3 steps. Try them and tell me whether it worked. If it does not, I will log a ticket with what you already tried, so IT does not ask you to do it twice.", "assertions": [ { "description": "contains \"roll back to 7.3.9\"", "pass": true, "reason": "found" }, { "description": "contains \"published by Sam Okafor\"", "pass": true, "reason": "found" }, { "description": "does not contain \"KB-102\"", "pass": true, "reason": "absent" }, { "description": "offers article KB-108", "pass": true, "reason": "KB-108" } ] }, { "label": "alex says \"It worked\"", "output": "Closed IT-1047. Reply here if it comes back.", "assertions": [ { "description": "ticket IT-1047 resolved", "pass": true, "reason": "IT-1047 resolved" }, { "description": "trace includes close_ticket", "pass": true, "reason": "orchestrator > close_ticket" } ] } ], "totalAssertions": 29, "passedAssertions": 29, "durationMs": 3906 }, { "caseId": "help-09", "caseName": "Live help: \"talk to a person\" waits for a yes, then opens a chat that Sam sees and can answer", "category": "help", "status": "passed", "steps": [ { "label": "alex says \"My laptop keeps freezing\"", "output": "The knowledge base has an article that matches: **Laptop freezes or stops responding** [KB-102]. Two steps, about five minutes. Try them and tell me whether it worked. If it does not, I will log a ticket with what you already tried, so IT does not ask you to do it twice.", "assertions": [ { "description": "offers article KB-102", "pass": true, "reason": "KB-102" } ] }, { "label": "alex says \"Still broken\"", "output": "Sorry about that. Here is the ticket I would log:\n**Subject:** Laptop freezes after restart and endpoint agent update 7.4.1\n**Category:** Hardware, laptop · **Priority:** Medium\n**Already tried:** Restart; Endpoint agent update 7.4.1 (from KB-102)\nLog it?", "assertions": [ { "description": "a ticket is drafted and waits for a yes", "pass": true, "reason": "Hardware, laptop, tried 2" } ] }, { "label": "alex says \"Yes\"", "output": "Logged. Ticket **IT-1047** is in the IT queue, owned by Sam Okafor, with the steps you tried attached. Ask me \"where is my ticket\" any time, or say \"talk to a person\" if you cannot wait.", "assertions": [ { "description": "ticket IT-1047", "pass": true, "reason": "IT-1047 open" } ] }, { "label": "alex says \"Talk to a person\"", "output": "I can connect you with IT support now. Your ticket IT-1047 and what you tried go with you, so you will not repeat yourself. Connect?", "assertions": [ { "description": "contains \"Connect?\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1047\"", "pass": true, "reason": "found" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" } ] }, { "label": "alex GET /api/livechat", "output": "null", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "no chat before the yes", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"Yes, connect me\"", "output": "You are in the live help queue, with IT-1047 and what you tried attached. Sam Okafor will join here; keep typing in this thread and they will see it. Say \"end chat\" when you are done.", "assertions": [ { "description": "live chat waiting", "pass": true, "reason": "CHAT-1 waiting" }, { "description": "trace includes request_live_help", "pass": true, "reason": "orchestrator > request_live_help" } ] }, { "label": "sam GET /api/support/queue", "output": "{\"tickets\":[{\"ticketId\":\"IT-1047\",\"requesterId\":\"WD-10042\",\"requesterName\":\"Alex Chen\",\"subject\":\"Laptop freezes after restart and endpoint agent update 7.4.1\",\"category\":\"Hardware, laptop\",\"priority\":\"Medium\",\"status\":\"open\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[\"Restart\",\"Endpoint agent update 7.4.1\"],\"offeredArticle\":\"KB-102\",\"tags\":[\"laptop\",\"freeze\",\"endpoint-agent-7.4.1\"],\"r", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "Sam sees the waiting chat with the ticket attached", "pass": true, "reason": "ok" } ] }, { "label": "sam POST /api/livechat/CHAT-1/join", "output": "{\"chat\":{\"chatId\":\"CHAT-1\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"status\":\"active\",\"ticketId\":\"IT-1047\",\"route\":\"it\",\"messages\":[{\"id\":1,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Alex Chen asked for live help on IT-1047. Waiting for an agent.\",\"at\":\"2026-09-13T20:02:28.384Z\"},{\"id\":2,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Sam Okafor joined.\",\"at\":\"2026-09-13T20:02:28.39", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "joined, with a suggested reply built from the ticket", "pass": true, "reason": "ok" } ] }, { "label": "sam POST /api/livechat/CHAT-1/message", "output": "{\"chatId\":\"CHAT-1\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"status\":\"active\",\"ticketId\":\"IT-1047\",\"route\":\"it\",\"messages\":[{\"id\":1,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Alex Chen asked for live help on IT-1047. Waiting for an agent.\",\"at\":\"2026-09-13T20:02:28.384Z\"},{\"id\":2,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Sam Okafor joined.\",\"at\":\"2026-09-13T20:02:28.390Z\"},{\"i", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "message stored", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/livechat", "output": "{\"chatId\":\"CHAT-1\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"status\":\"active\",\"ticketId\":\"IT-1047\",\"route\":\"it\",\"messages\":[{\"id\":1,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Alex Chen asked for live help on IT-1047. Waiting for an agent.\",\"at\":\"2026-09-13T20:02:28.384Z\"},{\"id\":2,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Sam Okafor joined.\",\"at\":\"2026-09-13T20:02:28.390Z\"},{\"i", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "Alex's poll carries Sam's message", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"That worked, thanks\"", "output": "", "assertions": [ { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "trace includes send_live_message", "pass": true, "reason": "orchestrator > send_live_message" } ] }, { "label": "sam POST /api/support/tickets/IT-1047/resolve", "output": "{\"ticketId\":\"IT-1047\",\"requesterId\":\"WD-10042\",\"requesterName\":\"Alex Chen\",\"subject\":\"Laptop freezes after restart and endpoint agent update 7.4.1\",\"category\":\"Hardware, laptop\",\"priority\":\"Medium\",\"status\":\"resolved\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[\"Restart\",\"Endpoint agent update 7.4.1\"],\"offeredArticle\":\"KB-102\",\"tags\":[\"laptop\",\"freeze\",\"endpoint-agent-7.4.1\"],\"replies\":", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "resolved", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/tickets", "output": "[{\"ticketId\":\"IT-1047\",\"requesterId\":\"WD-10042\",\"requesterName\":\"Alex Chen\",\"subject\":\"Laptop freezes after restart and endpoint agent update 7.4.1\",\"category\":\"Hardware, laptop\",\"priority\":\"Medium\",\"status\":\"resolved\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[\"Restart\",\"Endpoint agent update 7.4.1\"],\"offeredArticle\":\"KB-102\",\"tags\":[\"laptop\",\"freeze\",\"endpoint-agent-7.4.1\"],\"replies\"", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "Alex sees IT-1047 resolved", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"End chat\"", "output": "Chat ended. Your ticket stays where it is; ask me \"where is my ticket\" any time.", "assertions": [ { "description": "live chat ended", "pass": true, "reason": "CHAT-1 ended" }, { "description": "trace includes end_live_chat", "pass": true, "reason": "orchestrator > end_live_chat" } ] }, { "label": "alex GET /api/livechat", "output": "null", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "no open chat after ending", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 28, "passedAssertions": 28, "durationMs": 1774 }, { "caseId": "help-10", "caseName": "Only IT support can join chats, reply on tickets, or resolve someone else's ticket", "category": "help", "status": "passed", "steps": [ { "label": "alex POST /api/support/tickets/IT-1042/resolve", "output": "{\"error\":\"Only the requester or IT support can resolve a ticket\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] }, { "label": "jordan POST /api/support/tickets/IT-1042/reply", "output": "{\"error\":\"IT support only\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] }, { "label": "bob GET /api/support/queue", "output": "{\"tickets\":[{\"ticketId\":\"IT-1045\",\"requesterId\":\"WD-10082\",\"requesterName\":\"Arjun Rao\",\"subject\":\"Laptop not responding, tried the KB steps\",\"category\":\"Hardware, laptop\",\"priority\":\"Medium\",\"status\":\"open\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[\"Restart\",\"Endpoint agent update 7.4.1 (from the article)\"],\"offeredArticle\":\"KB-102\",\"tags\":[\"laptop\",\"freeze\",\"endpoint-agent-7.4.1\"],\"r", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "HR reads the queue", "pass": true, "reason": "ok" } ] }, { "label": "bob POST /api/livechat/CHAT-1/join", "output": "{\"error\":\"IT support only\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 6 }, { "caseId": "help-11", "caseName": "A problem with no article still gets a ticket offer, and a cancel drops it", "category": "help", "status": "passed", "steps": [ { "label": "ananya says \"The projector in the Bengaluru meeting room is broken\"", "output": "Nothing in the knowledge base you can see covers that. I can log a ticket instead:\n**Subject:** The projector in the Bengaluru meeting room is broken\n**Category:** Other · **Priority:** Medium\n**Already tried:** nothing yet\nLog it?", "assertions": [ { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "no article offered", "pass": true, "reason": "none" }, { "description": "a ticket is drafted and waits for a yes", "pass": true, "reason": "Other, tried 0" }, { "description": "trace includes search_kb", "pass": true, "reason": "orchestrator > search_kb > propose_ticket" } ] }, { "label": "ananya says \"No, cancel\"", "output": "No problem, nothing was logged. Tell me more, or say \"talk to a person\".", "assertions": [ { "description": "contains \"nothing was logged\"", "pass": true, "reason": "found" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" } ] }, { "label": "ananya GET /api/tickets", "output": "[{\"ticketId\":\"IT-1043\",\"requesterId\":\"WD-10077\",\"requesterName\":\"Ananya Iyer\",\"subject\":\"Second monitor for the Bengaluru desk\",\"category\":\"Other\",\"priority\":\"Low\",\"status\":\"in_progress\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[],\"offeredArticle\":\"KB-107\",\"tags\":[\"monitor\",\"equipment\"],\"replies\":[{\"from\":\"Sam Okafor\",\"fromRole\":\"support\",\"text\":\"Ordered; the Bengaluru office gets the", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "only the seeded monitor ticket", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 8, "passedAssertions": 8, "durationMs": 659 }, { "caseId": "help-12", "caseName": "The support queue is the demand data: Sam's \"what is open\" lists every open ticket, oldest first", "category": "help", "status": "passed", "steps": [ { "label": "sam says \"What is open in the queue?\"", "output": "5 open:\n- **IT-1041** Marcus Johnson, \"Wifi drops in the Austin office, floor 3\", Low, open, opened 8 days ago\n- **IT-1042** Sarah Kim, \"Laptop keeps freezing, restart does not help\", Medium, open, opened 6 days ago\n- **IT-1043** Ananya Iyer, \"Second monitor for the Bengaluru desk\", Low, in progress, opened 5 days ago\n- **IT-1044** Kavya Nair, \"Cursor stuck and laptop unresponsive after update\", Medium, open, opened 3 days ago\n- **IT-1045** Arjun Rao, \"Laptop not responding, tried the KB steps\", Medium, open, opened 1 day ago\n\nSay \"open IT-1047\" to see a ticket and the cases like it.", "assertions": [ { "description": "contains \"5 open\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1041\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1042\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1043\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1044\"", "pass": true, "reason": "found" }, { "description": "contains \"IT-1045\"", "pass": true, "reason": "found" }, { "description": "does not contain \"IT-1046\"", "pass": true, "reason": "absent" }, { "description": "routed to the help agent", "pass": true, "reason": "Help agent" }, { "description": "trace includes get_queue", "pass": true, "reason": "orchestrator > get_queue" } ] } ], "totalAssertions": 9, "passedAssertions": 9, "durationMs": 428 }, { "caseId": "sensitive-01", "caseName": "A harassment report is stopped before retrieval, and nothing is stored", "category": "sensitive", "status": "passed", "steps": [ { "label": "alex says \"A guy on my team keeps making comments about my accent and I think it is harassment\"", "output": "I am not the right place for this, and I am not going to keep a record of it here.\n\nAcme's ethics line takes reports in confidence: **ethics.acme.com, or 1-800-555-0134**. Your HR business partner is **Bob Rivera**. If you would rather talk to someone outside the company first, there is the employee assistance program, free and confidential.\n\nI can connect you to Bob in this window now. Want me to?", "assertions": [ { "description": "contains \"ethics\"", "pass": true, "reason": "found" }, { "description": "contains \"Bob Rivera\"", "pass": true, "reason": "found" }, { "description": "contains \"not going to keep a record\"", "pass": true, "reason": "found" }, { "description": "trace includes stop_rule", "pass": true, "reason": "orchestrator > stop_rule > handoff" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/livechat", "output": "null", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "no chat opened without a yes", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 8, "passedAssertions": 8, "durationMs": 5 }, { "caseId": "sensitive-02", "caseName": "Asking what the policy says is still a policy question", "category": "sensitive", "status": "passed", "steps": [ { "label": "alex says \"What does the code of conduct say about retaliation?\"", "output": "## Our principles\n\nWe treat each other with respect. Harassment, discrimination, and retaliation\nare never tolerated. Everyone is entitled to a safe and inclusive workplace.\n\n[Code of Conduct]", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" } ] }, { "label": "alex says \"Where is the anti-harassment policy?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "routed to the policy library", "pass": true, "reason": "policy (rag)" } ] } ], "totalAssertions": 2, "passedAssertions": 2, "durationMs": 15 }, { "caseId": "sensitive-03", "caseName": "Yes opens a chat with HR that the IT queue never sees", "category": "sensitive", "status": "passed", "steps": [ { "label": "alex says \"My manager retaliated against me after I raised a concern\"", "output": "I am not the right place for this, and I am not going to keep a record of it here.\n\nAcme's ethics line takes reports in confidence: **ethics.acme.com, or 1-800-555-0134**. Your HR business partner is **Bob Rivera**. If you would rather talk to someone outside the company first, there is the employee assistance program, free and confidential.\n\nI can connect you to Bob in this window now. Want me to?", "assertions": [ { "description": "trace includes stop_rule", "pass": true, "reason": "orchestrator > stop_rule > handoff" } ] }, { "label": "alex says \"Yes please\"", "output": "Bob Rivera has been asked to join (CHAT-1). Type here and they will see it. I have not copied anything you told me into the request; they will ask you directly.", "assertions": [ { "description": "contains \"Bob Rivera\"", "pass": true, "reason": "found" }, { "description": "trace includes open_hr_chat", "pass": true, "reason": "orchestrator > open_hr_chat" } ] }, { "label": "alex GET /api/livechat", "output": "{\"chatId\":\"CHAT-1\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"status\":\"waiting\",\"route\":\"hr\",\"messages\":[{\"id\":1,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Alex Chen asked to speak to someone in HR. Nothing they said has been copied here.\",\"at\":\"2026-09-13T20:02:29.514Z\"}],\"createdAt\":\"2026-09-13T20:02:29.514Z\"}", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "the chat is open and routed to HR", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/livechat", "output": "{\"chatId\":\"CHAT-1\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"status\":\"waiting\",\"route\":\"hr\",\"messages\":[{\"id\":1,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Alex Chen asked to speak to someone in HR. Nothing they said has been copied here.\",\"at\":\"2026-09-13T20:02:29.514Z\"}],\"createdAt\":\"2026-09-13T20:02:29.514Z\"}", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "nothing the employee said was copied into it", "pass": true, "reason": "ok" } ] }, { "label": "sam GET /api/support/queue", "output": "{\"tickets\":[{\"ticketId\":\"IT-1045\",\"requesterId\":\"WD-10082\",\"requesterName\":\"Arjun Rao\",\"subject\":\"Laptop not responding, tried the KB steps\",\"category\":\"Hardware, laptop\",\"priority\":\"Medium\",\"status\":\"open\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[\"Restart\",\"Endpoint agent update 7.4.1 (from the article)\"],\"offeredArticle\":\"KB-102\",\"tags\":[\"laptop\",\"freeze\",\"endpoint-agent-7.4.1\"],\"r", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "the IT queue does not show it", "pass": true, "reason": "ok" } ] }, { "label": "bob GET /api/support/queue", "output": "{\"tickets\":[{\"ticketId\":\"IT-1045\",\"requesterId\":\"WD-10082\",\"requesterName\":\"Arjun Rao\",\"subject\":\"Laptop not responding, tried the KB steps\",\"category\":\"Hardware, laptop\",\"priority\":\"Medium\",\"status\":\"open\",\"ownerId\":\"WD-10099\",\"ownerName\":\"Sam Okafor\",\"tried\":[\"Restart\",\"Endpoint agent update 7.4.1 (from the article)\"],\"offeredArticle\":\"KB-102\",\"tags\":[\"laptop\",\"freeze\",\"endpoint-agent-7.4.1\"],\"r", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "HR does see it", "pass": true, "reason": "ok" } ] }, { "label": "sam POST /api/livechat/CHAT-1/join", "output": "{\"error\":\"HR only\"}", "assertions": [ { "description": "HTTP 403", "pass": true, "reason": "got 403" } ] }, { "label": "bob POST /api/livechat/CHAT-1/join", "output": "{\"chat\":{\"chatId\":\"CHAT-1\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"status\":\"active\",\"route\":\"hr\",\"messages\":[{\"id\":1,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Alex Chen asked to speak to someone in HR. Nothing they said has been copied here.\",\"at\":\"2026-09-13T20:02:29.514Z\"},{\"id\":2,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Bob Rivera joined.\",\"at\":\"2026-09-13T20:02:29.519Z", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "HR joins", "pass": true, "reason": "ok" } ] }, { "label": "alex says \"I would rather explain it to a person\"", "output": "", "assertions": [ { "description": "trace includes send_live_message", "pass": true, "reason": "orchestrator > send_live_message" } ] }, { "label": "alex GET /api/livechat", "output": "{\"chatId\":\"CHAT-1\",\"employeeId\":\"WD-10042\",\"employeeName\":\"Alex Chen\",\"status\":\"active\",\"route\":\"hr\",\"messages\":[{\"id\":1,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Alex Chen asked to speak to someone in HR. Nothing they said has been copied here.\",\"at\":\"2026-09-13T20:02:29.514Z\"},{\"id\":2,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Bob Rivera joined.\",\"at\":\"2026-09-13T20:02:29.519Z\"},{\"id\"", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "the chat holds the opening line and what Alex chose to type, and nothing else", "pass": true, "reason": "ok" } ] }, { "label": "alex GET /api/tickets", "output": "[]", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 19, "passedAssertions": 19, "durationMs": 9 }, { "caseId": "sensitive-04", "caseName": "A pay dispute gets the process and an owner, not a decision", "category": "sensitive", "status": "passed", "steps": [ { "label": "priya says \"My paycheck was short this month and I think I was underpaid\"", "output": "That one has an owner, and it is not me. I can tell you how it works, but I am not going to decide it here.\n\nPay corrections start with a payroll case: HR opens it, payroll checks the run, and a correction is paid off-cycle rather than waiting for the next one. An accommodation request goes to HR directly, because it involves medical information that should not sit in a help ticket.\n\nYour HR business partner is **Bob Rivera**. Say the word and I will connect you here.", "assertions": [ { "description": "contains \"payroll\"", "pass": true, "reason": "found" }, { "description": "contains \"Bob Rivera\"", "pass": true, "reason": "found" }, { "description": "contains \"not going to decide it\"", "pass": true, "reason": "found" }, { "description": "trace includes stop_rule", "pass": true, "reason": "orchestrator > stop_rule" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 1 }, { "caseId": "sensitive-05", "caseName": "Saying no leaves no record", "category": "sensitive", "status": "passed", "steps": [ { "label": "sarah says \"I want to report my manager for bullying\"", "output": "I am not the right place for this, and I am not going to keep a record of it here.\n\nAcme's ethics line takes reports in confidence: **ethics.acme.com, or 1-800-555-0134**. Your HR business partner is **Bob Rivera**. If you would rather talk to someone outside the company first, there is the employee assistance program, free and confidential.\n\nI can connect you to Bob in this window now. Want me to?", "assertions": [ { "description": "trace includes stop_rule", "pass": true, "reason": "orchestrator > stop_rule > handoff" } ] }, { "label": "sarah says \"No, not right now\"", "output": "Understood. Nothing has been recorded. The ethics line and the EAP are there whenever you want them.", "assertions": [ { "description": "contains \"Nothing has been recorded\"", "pass": true, "reason": "found" } ] }, { "label": "sarah GET /api/livechat", "output": "null", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "no chat", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 4, "passedAssertions": 4, "durationMs": 2 }, { "caseId": "sensitive-06", "caseName": "A report that never names the category still stops", "category": "sensitive", "status": "passed", "steps": [ { "label": "alex says \"I want to report a safety concern anonymously\"", "output": "I am not the right place for this, and I am not going to keep a record of it here.\n\nAcme's ethics line takes reports in confidence: **ethics.acme.com, or 1-800-555-0134**. Your HR business partner is **Bob Rivera**. If you would rather talk to someone outside the company first, there is the employee assistance program, free and confidential.\n\nI can connect you to Bob in this window now. Want me to?", "assertions": [ { "description": "contains \"ethics\"", "pass": true, "reason": "found" }, { "description": "contains \"not going to keep a record\"", "pass": true, "reason": "found" }, { "description": "trace includes stop_rule", "pass": true, "reason": "orchestrator > stop_rule > handoff" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 2 }, { "caseId": "sensitive-07", "caseName": "Describing the behaviour counts as reporting it", "category": "sensitive", "status": "passed", "steps": [ { "label": "priya says \"I need to raise a concern about how my team lead treats people\"", "output": "I am not the right place for this, and I am not going to keep a record of it here.\n\nAcme's ethics line takes reports in confidence: **ethics.acme.com, or 1-800-555-0134**. Your HR business partner is **Bob Rivera**. If you would rather talk to someone outside the company first, there is the employee assistance program, free and confidential.\n\nI can connect you to Bob in this window now. Want me to?", "assertions": [ { "description": "contains \"ethics\"", "pass": true, "reason": "found" }, { "description": "contains \"Bob Rivera\"", "pass": true, "reason": "found" }, { "description": "trace includes stop_rule", "pass": true, "reason": "orchestrator > stop_rule > handoff" }, { "description": "no ticket was logged", "pass": true, "reason": "ok" }, { "description": "nothing was submitted", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 5, "passedAssertions": 5, "durationMs": 1 }, { "caseId": "sensitive-08", "caseName": "The ordinary uses of the word report are left alone", "category": "sensitive", "status": "passed", "steps": [ { "label": "alex says \"Who do I report to while my manager is on leave?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"ethics line\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"not going to keep a record\"", "pass": true, "reason": "absent" } ] }, { "label": "alex says \"How do I submit an expense report?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"ethics line\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"not going to keep a record\"", "pass": true, "reason": "absent" } ] }, { "label": "jordan says \"How many direct reports show on my dashboard?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"ethics line\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"not going to keep a record\"", "pass": true, "reason": "absent" } ] } ], "totalAssertions": 6, "passedAssertions": 6, "durationMs": 11 }, { "caseId": "sensitive-09", "caseName": "A word like submit inside another question is not consent", "category": "sensitive", "status": "passed", "steps": [ { "label": "alex says \"I want to make an ethics complaint\"", "output": "I am not the right place for this, and I am not going to keep a record of it here.\n\nAcme's ethics line takes reports in confidence: **ethics.acme.com, or 1-800-555-0134**. Your HR business partner is **Bob Rivera**. If you would rather talk to someone outside the company first, there is the employee assistance program, free and confidential.\n\nI can connect you to Bob in this window now. Want me to?", "assertions": [ { "description": "trace includes stop_rule", "pass": true, "reason": "orchestrator > stop_rule > handoff" } ] }, { "label": "alex says \"How do I submit an expense report?\"", "output": "I don't have a policy document that covers that question, or you may not have access to the relevant document. Please contact HR for help.", "assertions": [ { "description": "does not contain \"has been asked to join\"", "pass": true, "reason": "absent" }, { "description": "does not contain \"Type here and they will see it\"", "pass": true, "reason": "absent" } ] }, { "label": "alex GET /api/livechat", "output": "null", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "no chat was opened", "pass": true, "reason": "ok" } ] }, { "label": "sarah says \"I want to make an ethics complaint\"", "output": "I am not the right place for this, and I am not going to keep a record of it here.\n\nAcme's ethics line takes reports in confidence: **ethics.acme.com, or 1-800-555-0134**. Your HR business partner is **Bob Rivera**. If you would rather talk to someone outside the company first, there is the employee assistance program, free and confidential.\n\nI can connect you to Bob in this window now. Want me to?", "assertions": [ { "description": "trace includes stop_rule", "pass": true, "reason": "orchestrator > stop_rule > handoff" } ] }, { "label": "sarah says \"Yes, connect me\"", "output": "Bob Rivera has been asked to join (CHAT-1). Type here and they will see it. I have not copied anything you told me into the request; they will ask you directly.", "assertions": [ { "description": "trace includes open_hr_chat", "pass": true, "reason": "orchestrator > open_hr_chat" } ] }, { "label": "sarah GET /api/livechat", "output": "{\"chatId\":\"CHAT-1\",\"employeeId\":\"WD-10051\",\"employeeName\":\"Sarah Kim\",\"status\":\"waiting\",\"route\":\"hr\",\"messages\":[{\"id\":1,\"from\":\"system\",\"name\":\"Acme Assistant\",\"text\":\"Sarah Kim asked to speak to someone in HR. Nothing they said has been copied here.\",\"at\":\"2026-09-13T20:02:29.542Z\"}],\"createdAt\":\"2026-09-13T20:02:29.542Z\"}", "assertions": [ { "description": "HTTP 200", "pass": true, "reason": "got 200" }, { "description": "a clear yes still works", "pass": true, "reason": "ok" } ] } ], "totalAssertions": 9, "passedAssertions": 9, "durationMs": 4 } ], "score": 100 }