Time off: an agent that plans

Problem

The form asks for two dates. The real question is which two.

How many days do you have? Which ones will you lose at year end? Where do the holidays fall, so three PTO days become eleven off? Who on the team is already out that week? Is the week inside a release freeze? Employees do that math in their heads, badly, and managers redo it when the request lands. The form captures the answer and none of the reasoning.

An employee should be able to say "I want a vacation" and get a plan. So the agent does the reasoning, out loud, and the form disappears into a confirmation.

Solution

Say you want a vacation and get a plan.

Type "I want to book a vacation" and the assistant checks your balance, reads the notice and carryover rules and cites them, pulls the holidays and the team calendar, scores every window in the next six months, and shows three as cards: dates, PTO days used, total days off, holidays included, who else is out, and a coverage chip. Pick one and it drafts a summary. Say yes and it submits to the HR system and notifies your manager. Nothing is submitted until you say yes, and the tool refuses otherwise.

The demo as Alex: three windows as cards, each with its PTO cost, days off, holidays, who else is out, and a coverage check. Option 1 is the pick.

I want to book a vacation

Alex, you have 12 PTO days available. Earliest start under the 14-day notice rule is Tue, Sep 22, 2026 PTO Policy / Requesting time off. You can carry only 5 days into next year, so 7 of your 12 days are use-it-or-lose-it before Dec 31 PTO Policy / Carryover.

Option 1: Dec 28 to Dec 30 (3 PTO days, 11 days off). 3 PTO days gets you 11 days off, Dec 24 through Jan 3. Includes Christmas Eve, Christmas Day, New Year's Eve, and New Year's Day, which don't count against PTO. Priya Sharma (approved, Dec 28 to Dec 30) is out. Team would be at 3 of 5, which meets the 60% coverage minimum. Falls before Dec 31, so it uses days you would otherwise forfeit.

Option 2: Dec 21 to Dec 23 (3 PTO days, 9 days off) and Option 3: Nov 23 to Nov 25 (3 PTO days, 9 days off) follow, each with the same four lines.

My pick is Option 1: 11 days off for 3 PTO days with coverage intact. Reply with an option number, or tell me when you'd rather go.

orchestratorIntent: time off (rule: matched "book"). Routed to the time-off agent, 9 tools.
get_leave_balancesChecked HR-system balances: Paid Time Off 12 days (24 hrs pending), Sick Leave 8 days, Floating Holiday 2 days
search_policiesSearched the 5 policies that apply to Alex. Top match: PTO Policy / Carryover
get_company_holidaysPulled 2026 holidays for US-CA: 5 still ahead (Thanksgiving Nov 26, Day After Thanksgiving Nov 27, Christmas Eve Dec 24, Christmas Day Dec 25, ...)
get_team_calendarChecked team calendar Sep 8 to Mar 27: Sarah Kim approved Oct 12 to Oct 16; David Lee pending Nov 23 to Nov 25; Priya Sharma approved Dec 28 to Dec 30
suggest_vacation_datesScored candidate windows from Sep 22 onward; kept 3: Dec 28 to Dec 30 (3 PTO, 11 off); Dec 21 to Dec 23 (3 PTO, 9 off); Nov 23 to Nov 25 (3 PTO, 9 off)

Captured from the running demo. The earliest start, the holiday count, and the calendar window move with the date; the three options do not until the year turns.

Architecture and flow

Same front door. An agent that reads, plans, and waits for a yes.

This is the one use case that changes something in a system of record, so it is where the model, the tools, and the gate all show at once. The model drives. The tools are code. The gate is one check that nothing can talk its way past.

Employee

One conversationThe same assistant as use case 01. "I want to book a vacation" in plain words.

Front door

Same sign-in and orchestrator as use case 01Who you are, your team, and your manager from the sign-in; an orchestrator that hears vacation, balance, dates, or who is out and hands the turn here with nine tools. How that works.

Time-off agent

Claude, in a loopDecides which tools to call and in what order, reads each result, and stops when it has a plan or a question. Up to twelve rounds.

Nine tools. Eight read or draft, one writes, and it waits for your yes.

get_leave_balanceswhat you have left, pending and accrued, per leave type
search_policiesthe rules you are allowed to see, with the section cited
get_company_holidayscompany holidays for your location and year
get_team_calendarwho on your team is already out in a date range
check_date_conflictsoverlaps, holidays, blackout windows and coverage on specific dates
suggest_vacation_datesranks windows on balance, rules, holidays and coverage, returns three
get_time_off_requestsyour existing requests
propose_time_off_requestdrafts the request for you to confirm. Submits nothing
submit_time_off_requestwritethe only tool that changes the HR system. Refuses without an explicit yes

Inside Suggest dates

The planner, plain codeScores every window in the next six months: days off per PTO day, teammates already out, blackouts skipped, coverage checked, days you would forfeit. Same inputs, same three options, every time.

Systems

HR systemBalances, requests, team calendar, holidays. Read by eight tools, written only by Submit.

Policy libraryNotice period, carryover cap, blackout windows, with the section cited.

Manager's queueThe request with a coverage note. Approve from the panel or in chat; the decision lands in your thread.

Always on

The yes gateDraft first. Submit refuses without an explicit yes.

The traceEvery tool call, its input, and a one-line result.

The evals23 cases for this agent and the approval flow behind it.

Two layers, both true at once. The loop is the model deciding what to do next. The tools are code, so the numbers it reasons over cannot drift. The only write is Submit, and it sits behind the yes gate.

The guardrails, and the test that proves each one

Each rule below says where it lives in the code, what you see when it stops something, and the test that would fail if it broke.

The rule	Where it lives	What you see	Proof
Nothing submits without an explicit yes	`confirm_gate` in `server/agents/orchestrator.ts`, checked against the latest message, not the conversation so far	The draft stays on screen and the assistant asks again. The trace shows the refusal and why.	safety-01 safety-02
A stray yes with nothing drafted writes nothing	the same gate, denial reason `nothing drafted`	“Nothing is drafted yet. Propose a request first.”	routing-05
The rules are code	notice period, blackout windows and carryover are worked out in the planner and read through `search_policies`, so the model cannot bend them	A request inside the release freeze is refused, with the policy section quoted back.	agent-07
No double booking	`check_date_conflicts` against existing requests before a draft is offered	The overlap is named and nothing is submitted.	safety-03
Eligibility is read from the HR record	balances, employment type and country are read at the start of the turn, before the model sees anything	A part-time employee sees prorated balances. A contractor is offered unpaid time off, whatever they type.	agent-02 agent-03
Only the manager decides, and only once	approval scoped to the requester’s manager; a decided request cannot be decided again	HR cannot approve. A second decision on the same request is refused.	manager-02 manager-05

The flow, in six steps.

You ask, and the orchestrator sends it to the time-off agent.

"I want to book a vacation." The agent gets the message, who you are, and its nine tools. Nothing tells it which to call first.
It reads four things.

Your balance, the rules that shape the choice, the holidays, and who on your team is already out. Each is a tool, and each leaves a line in the trace.
It asks the planner for the best windows.

The planner scores every window and returns three: dates, PTO days used, total days off, holidays included, who else is out, and whether coverage holds. The model presents them; it cannot change them.
You pick one, and it drafts.

Type, dates, work days with holidays excluded, balance after, and who approves. Then it asks: ready to send this to your manager?
Nothing is sent without your yes.

Submit checks two things: a draft exists, and your latest message is an explicit yes. "Maybe" does nothing.
Your manager decides, and you are told.

The request lands in their queue with a coverage note. They approve on the panel or in chat, and the decision shows up in your thread within seconds, with the reason if it was a no.

The demo as Jordan, the manager: what is waiting on him, a coverage note on each request, and the words that draft a decision.

Architecture · production

Swap the backend. Keep the contract.

The nine tools are the integration boundary. In production each one is backed by a connector to the system of record instead of mock data, and nothing above that line changes: not the orchestrator, not the gate, not the trace, not the evals. This is what that looks like against the three HR systems most enterprises actually run.

Tool	In the demo	Workday	SAP SuccessFactors	Oracle HCM
get_leave_balances	Mock balances	Absence Management REST, `balances` per worker and plan	OData `TimeAccount` and `TimeAccountDetail`	REST `planBalances`, as of today or projected
get_time_off_requests	Mock request list	`workers/{id}/timeOffDetails`, filtered by status and dates	`EmployeeTime` filtered by `userId` and `approvalStatus`	REST `absences` for the person
get_team_calendar	Mock team requests	`timeOffDetails` for each direct report	`EmployeeTime` for the manager's reports	`absences` for the team, approved and pending
get_company_holidays	A fixed list	Holiday calendar by location	Holiday calendar assigned to the employee	Calendar events for the worker's location
submit_time_off_request	Mock write, manager notified	POST `workers/{id}/requestTimeOff`, which starts the Request Time Off business process	Create an `EmployeeTime` entry with the `timeType`; approval runs in the SAP workflow	POST `absences`; approval runs in the Oracle workflow
Manager decision	Approve in chat or panel	Approval belongs to the vendor's own business process. The assistant surfaces the request and the coverage note, records the decision through the same connector, and reads the status back. It does not keep a second copy of the truth.

search_policies stays where it is: the policy library is its own connector to SharePoint, Drive, or the handbook, with permissions read at index time. suggest_vacation_dates, the two draft tools, and the confirmation gate are pure code and never touch a vendor.

Two ways to connect. For a system of record, one MCP server per system, exposing these same nine tool names with the vendor API behind them. The agent code is identical across customers, a connector can be tested on its own against a sandbox tenant, and swapping Workday for SuccessFactors is a configuration change. For a vendor that ships its own agent, say absence planning inside the HR suite, A2A is the other case: the orchestrator delegates the whole intent to that agent, gets a result back, and the gate and the trace still sit on this side.

Architecture · identity

Every call is made as the employee.

In most enterprises the directory is Microsoft Entra ID, formerly Azure AD, or Okta. Here is the Entra version. The employee signs in with OIDC, and the groups in their token decide what they have access to, the same rule the demo uses. When the agent needs the HR system, it does not use a service account with broad rights. It exchanges the employee's token, using the on-behalf-of flow, for a token scoped to that employee and to delegated permissions only, so the connector can never read or write beyond what the person could do in the vendor's own UI. Entra's agent identity model gives the agent its own registration for this, with a federated credential rather than a stored secret.

Sign in · Entra ID

Employee signs in with OIDCThe ID token carries the identity and group claims: all employees, engineering, managers, HR admins, country

Assistant resolves the askerGroups from the token become the access model for search and the scope for every tool

Act · on behalf of

Agent identity, on-behalf-of exchangeThe agent presents the employee's token plus its own federated credential and receives a downstream token for the HR system's API, delegated scopes only, no application roles

MCP connector calls the HR systemAs the employee. Their balance, their requests, their team if they manage one. The vendor enforces its own security on top

Audit

Two logs, one storyThe assistant's trace says what the agent did and why. Entra's sign-in and token logs say who it did it as. A write with no yes in the transcript cannot exist

The gate does not change in production. It gets a second enforcer: even if the model reached the submit tool without a yes, the token in hand can only do what that employee could do, and every exchange is logged by the directory.

What stays exactly as it is in the demo: the orchestrator and its three layers, the nine-tool contract, the deterministic planner, the confirmation gate, the trace, and the 67 evals, which now run against a sandbox tenant instead of mock data.

Architecture · edge cases

Where the agent earns its keep.

Blackout

"Can I take Oct 27 to 29 off?" is refused with the citation and an offer to find the nearest clear week. Nothing is drafted.

Not enough days

A four-week ask gets "you'd need 20 days and have 12," plus the two ways out: a shorter range or a split across leave types.

Holiday inside the range

Nov 23 to 27 costs three PTO days, not five. The draft says so and shows which two days were excluded.

Short notice

"Next Friday" is inside the 14-day window. The agent drafts it anyway with a heads-up and the citation, because the manager can still approve it.

A stray yes

"Yes, send it" with nothing drafted routes to search and submits nothing. The eval checks this on every run.

Evaluation framework

One right answer, checked on every change.

Twenty-three of the suite's 67 cases cover this agent and the approval flow behind it: twelve for the planner, three for safety, three for the request-to-decision flow, five for the manager. They answer questions with one right answer. Does "I want to book a vacation" return three options with Dec 28 to 30 first and coverage intact on all three? Are part-time balances prorated, and does a contractor get the unpaid path with the 7-day notice? Is a blackout week refused with the citation, and a short-notice request warned rather than refused? Are holidays inside a range left out of the PTO count? Does "hmm, maybe" after a draft submit nothing, and does cancel discard it? When Alex requests and Jordan approves, is Alex told and do the balances move? Does a deny need a reason, can only the manager decide, and only once? Does the coverage math in the planner match the team calendar? The latest run passes all 23, and all 67 in the suite. The method is in the evals chapter.

Read the suite, case by case. Or the source: test-cases.ts

Group	Cases	What they hold the agent to
Planner	12	Balances for full-time, part-time, and contractors. Three options with the right one first. A requested month and length honored. Blackouts refused with the citation. Holidays left out of the count. Short notice warned, not refused. Not enough days explained.
Safety	3	Nothing submitted without an explicit yes. Cancel discards the draft. An overlap with an existing request is refused.
Flow	3	Alex requests, Jordan approves, Alex is told, and the balance moves. A deny carries its reason back. A contractor's request reaches the manager as unpaid.
Manager	5	The queue for a team of five. Only the manager decides, and only once. Coverage math matches the calendar. The whole queue can be worked from chat. Deny needs a reason; HR cannot decide.

Outcome

Requests that are right the first time.

The time-off agent owns the share of requests approved as filed. The waste it removes is the loop. A request lands in a blackout week, or misses the notice window, or leaves the team with nobody on call. The manager denies it, the employee picks new dates, and the whole thing goes around again, costing the employee twice and the manager once.

The planner reads the balance, the holiday calendar, the team calendar, the blackout list and the notice rule before it proposes anything, so the three windows it offers are already legal and the holidays inside them are already out of the day count. That is the work the loop was doing by hand.

A complete booking, three turns from "I want to book a vacation" to a filed request, cost 30 cents of model spend in a traced run. That is one run, so read it as an order of magnitude. Set it against the manager minutes on the other side of the loop.

The number this can never buy is a request filed without a yes. The agent drafts and the employee submits. A stray "yes, send it" with nothing drafted submits nothing, and an eval checks that on every change.

Owns · the number this answers for

Approved as filedRequests approved without a denial and a second attempt↑ over time

Moves · what follows if it works

Manager minutes per decisionQueue, coverage and a drafted decision arrive together in the manager's own thread↓ over time

Time to fileFrom "I want to book a vacation" to a request in the queueone conversation

Filed through the assistantShare of requests that never touch the form↑ over time

Guardrail · never traded for the number

Writes without an explicit yesAnything submitted, approved or denied that the person did not confirm0

Balance over-drawnA request filed against days the employee does not have0

The arithmetic, with your headcount in it, is on the business case.

What I learned

The planner is the product, the chat is the wrapper.

Deterministic first, model second.
The scoring, the coverage math, and the confirmation gate are plain code. With an API key, a model drives the tool loop and chooses the order; without one, a fixed orchestrator runs the same tools. The trace looks the same and the guarantees are the same. The model adds judgment on top of rules, not instead of them.
Show the work in the thread.
"Worked through 5 steps" under each answer is the feature people ask about. It is also what made the routing bug described on the evals page obvious in five seconds.
A human gate is cheap and non-negotiable.
One check in the submit tool. It cost nothing to build and it is the difference between a demo and something HR would sign off on.

How you can build this

How to build this yourself.

The order matters. Build the tools and the gate before the model ever sees a message, then let the model drive. Eight steps.

Write the one-page spec first.

Who the users are, which tools only read and which write, what each write needs before it runs, and what the assistant must never do. Write the acceptance transcript too: the conversation that has to work, word for word. I built this one first, deciding the rules as I went, and wrote the spec afterward. The spec is where those rules turned into decisions I could see and defend. Write it before the second use case at the latest.

Prompt
Here is what I want the assistant to do and for whom. Draft a one-page spec: users and roles, tools split into read, propose, and write, the guard each write needs, what must never happen, and one acceptance transcript. Mark every product decision you had to guess as DECIDE.
Write the tools as plain functions first.

Balances, rules, holidays, team calendar, requests. Each takes the signed-in employee, returns data, and returns a one-line summary for the trace. Mock the HR system behind them so you can run everything offline.

Prompt
Design read-only tools for a time-off assistant over a mock HR system: balances, rules, holidays, team calendar, and the employee's requests. Each returns structured data plus a one-line summary for a trace.
Make the planner deterministic.

Score every window in the horizon: calendar days off per PTO day, teammates already out, blackouts skipped, coverage checked, year-end forfeit. Same inputs, same three options, every time.

Prompt
Write a pure function that scores every candidate vacation window over six months and returns the top three with a reason each: skip blackouts, flag windows that break 60% coverage, prefer holidays, never call a model.
Put one gate on the only write.

A draft tool that produces a summary, and a submit tool that refuses unless a draft exists and the latest message is an explicit yes. This is the difference between a demo and something HR would sign off on.

Prompt
Add a draft tool and a submit tool that refuses unless a draft exists and the latest message is an explicit yes. Write the test that proves a stray yes with nothing drafted submits nothing.
Let the model drive the loop, with a bound.

Give Claude the tools, a system prompt with who the employee is, and a cap on rounds. It decides which tools to call and in what order. Keep an offline orchestrator that runs the same tools in a fixed order, so the demo and the tests work without a key.

Prompt
Wire the tools into a Claude tool-use loop capped at twelve rounds, with the employee's name, type, and manager in the system prompt, and an offline fallback that runs the same tools in a fixed order when there is no API key.
Put an orchestrator in front.

Intent first, then one use case. Session state, then readable rules, then a small model only when the rules are silent. Each agent gets only its own tools, and the decision becomes the first line of the trace.

Prompt
Design an orchestrator that classifies each message by session state, then rules, then a small model only when nothing matched, routes to an agent with only its tool subset, and records the decision as the first trace step.
Wrap the real HR system in a connector.

When the mock has to become Workday, SAP, or Oracle, nothing above the tools changes. Put an MCP server in front of the vendor's API that exposes the same nine tool names, and make every call with a token exchanged for the signed-in employee, never a service account. Point the evals at a sandbox tenant and run the same 67 cases.

Prompt
Build an MCP server that exposes the same nine tools over a vendor absence API, every call made with an on-behalf-of token for the signed-in employee, so the agent and the eval suite run unchanged against a sandbox tenant.
Write the evals before you trust it.

Every case is a sequence of steps as a named persona, and the checks run on structure: which path ran, which option came first, whether anything was submitted, whether a locked document was ever named. Run them on every change.

Prompt
Build an eval runner where each case is a list of persona steps and the checks assert on structure: which path ran, which option came first, whether a draft is pending, whether anything was submitted, and that no locked document is ever named.

Key terms

Definitions, in plain language.

Tool-use loop

The model calls a tool, reads the result, decides what to do next, and repeats until it has enough to answer. The tools are code; the choosing is the model's.

Deterministic planner

A function with no model in it. Same inputs, same three options, every time. What makes the numbers testable.

Yes gate

The two conditions the submit tool checks: a draft exists, and the latest message is an explicit yes. The only write in the system, behind one door.

MCP server

A standard way to expose tools to an agent. In production, one per HR system, presenting the same nine tool names with the vendor's API behind them.

A2A

Agent-to-agent. One agent hands a task to another and gets a result back. The case for a vendor that ships its own agent.

On-behalf-of flow

The token exchange that lets the agent call the HR system as you, with only the permissions you have. Never a service account with broad rights.

Resources

Worth your time.

Building Effective Agents
Anthropic. The difference between a workflow and an agent, and when each is the right call.
Tool use with Claude
Anthropic docs. How the tool-use loop works, what a tool definition looks like, and how results go back to the model.
Model Context Protocol
The spec and the reasoning behind it. Read this before writing the first connector.
Agent2Agent protocol
The open protocol for one agent to hand a task to another. The case where a vendor ships its own agent.
Microsoft Entra: on-behalf-of flow
Microsoft Learn. The token exchange that lets the agent act as the employee with delegated scopes only.
Entra Agent ID: agent OAuth flows
Microsoft Learn. Agent identities, blueprints, and why a federated credential beats a stored secret.
Evaluating it
The full method: what good had to mean, the layers measured separately, and the grader checked against hand labels.

The form asks for two dates. The real question is which two.

Say you want a vacation and get a plan.

Same front door. An agent that reads, plans, and waits for a yes.

The guardrails, and the test that proves each one

You ask, and the orchestrator sends it to the time-off agent.

It reads four things.

It asks the planner for the best windows.

You pick one, and it drafts.

Nothing is sent without your yes.

Your manager decides, and you are told.

Swap the backend. Keep the contract.

Every call is made as the employee.

Where the agent earns its keep.

One right answer, checked on every change.

Requests that are right the first time.

The planner is the product, the chat is the wrapper.

Deterministic first, model second.

Show the work in the thread.

A human gate is cheap and non-negotiable.

How to build this yourself.

Write the one-page spec first.

Write the tools as plain functions first.

Make the planner deterministic.

Put one gate on the only write.

Let the model drive the loop, with a bound.

Put an orchestrator in front.

Wrap the real HR system in a connector.

Write the evals before you trust it.

Definitions, in plain language.

Worth your time.