Lab 05: Your first custom agent: question-tester, before
At a glance
Section titled “At a glance”Why this matters
Section titled “Why this matters”You need a baseline before an improvement means anything. The question-tester agent acts like an analyst’s AI assistant: it reads the model, writes DAX, records assumptions, and saves answers in a schema the grader can score. The baseline model is intentionally weak, so a low score is not failure. It’s evidence you can compare after the model becomes AI-ready.
Concepts
Section titled “Concepts”- Custom agent: a
.github/agents/*.agent.mdfile that defines an agent’s purpose, tools, boundaries, and handoffs. See Agents. - Prompt file: a reusable chat task.
/answer-business-questionspasses alabelinput to the agent and tells it where to save the run. See Prompts. - Evaluation run:
evaluation/runs/<label>.json, with one answer object per question and agradeblock written by the grader. - Expected-answer isolation: the agent is told not to read expected answers, and the hook denies those reads for CLI and cloud-agent surfaces.
- Grader:
scripts/Grade-QuestionRun.ps1scores each answer 0, 1, or 2 against the deterministic expected answers.
Before you begin
Section titled “Before you begin”- You finished Lab 04 and Power BI Modeling MCP can connect to the open Desktop model.
- On Windows,
fabric\ContosoPharmacy.pbipis open and refreshed in Power BI Desktop. - You are still on the baseline model. Do not run the improver agent yet.
- If you do not have Desktop, use the catch-up path in step 6.
Step 1. Read the question-tester contract
Section titled “Step 1. Read the question-tester contract”Open the custom agent file and read the frontmatter, tools, boundaries, and steps:
---name: question-testerdescription: Answers the five Contoso Pharmacy business questions using only the semantic model (metadata and DAX through Power BI Modeling MCP), saves the answers as evaluation/runs/<label>.json, and grades the run. Use to create a baseline or after run, or to test whether a model change improved answers.argument-hint: Run label, for example local-baseline or local-ai-readytools: ['read', 'search', 'execute', 'edit', 'todo', 'powerbi-modeling-mcp/*']handoffs: - label: Compare with the baseline agent: question-tester prompt: Compare this run with local-baseline using Compare-QuestionRuns.ps1 and explain each question that changed. send: false - label: Review the change agent: model-reviewer prompt: Review the semantic model and report changes on this branch against main, and include the question comparison in out/question-comparison.md if it exists. send: false---# Question tester
You play the part of an analyst's AI assistant. You answer business questions from the semantic model as it is right now, so the lab can measure how model changes affect answers.
Load the `business-question-evaluation` skill and follow it exactly.
## Boundaries
- Get questions only from `pwsh ./scripts/Get-BusinessQuestions.ps1`.- Never read `evaluation/expected/`, `evaluation/questions.yaml`, `tools/python/pharmacy_demo/modelgen/`, or `tools/python/pharmacy_demo/samples.py`, and never run `pharmacy_demo dax-queries`. They hold the answers. The hooks deny them.- Never change the model. The only files you write are `evaluation/runs/<label>.json` and, when comparing, `out/question-comparison.md`.- Answer from the model you find. Don't improve it first, and don't guess what an improved model would say.- If Power BI Modeling MCP isn't connected or Power BI Desktop isn't running, stop and say so. Never invent result rows.
## Steps
1. Confirm the label with the user if they didn't give one. Use `local-baseline` on the baseline model and `local-ai-ready` after Lab 09.2. Connect to the model open in Desktop through Power BI Modeling MCP and list its tables and measures.3. For each question, write and run one DAX query, then record the answer, DAX, assumptions, and rows.4. Save `evaluation/runs/<label>.json`.5. Grade it: ```powershell pwsh ./scripts/Grade-QuestionRun.ps1 -Run <label> ```6. Report the score per question and, for anything below 2, the model gap you think caused it.The important part is the boundary: the tester answers from the model as it is. It must not improve the model, and it must not read expected answers.
You should see: question-tester has Power BI Modeling MCP tools, can write only the run file and comparison output, and must stop instead of inventing rows when Desktop or MCP is unavailable.
Step 2. Confirm what keeps the test honest
Section titled “Step 2. Confirm what keeps the test honest”The skill repeats the rule, and the hook makes it deterministic outside VS Code’s instruction-only path:
## Rules that keep the test honest
- Get the questions only from `pwsh ./scripts/Get-BusinessQuestions.ps1` (add `-Json` for machine output). It prints the ID, the question, paraphrases, and the `answer_columns` you must use.- Never read `evaluation/expected/`, `evaluation/questions.yaml`, `tools/python/pharmacy_demo/modelgen/`, or `tools/python/pharmacy_demo/samples.py`, and never run `pharmacy_demo dax-queries`. They contain the answers. The repo's hooks deny these.- Answer from the model as it is now. On the baseline model, use the names you find (for example `fact_rx_fill`), and don't fix the model first. A low baseline score is the expected result.- Don't write the `grade` block. Only the grader writes it.
## Step 1. Answer each question
For each question:
1. Decide the interpretation. If the question is ambiguous (which month is "latest", what "unusually high" means), write your choice in `assumptions`.2. Write one DAX query that returns exactly the `answer_columns`, one row per expected key. Use `EVALUATE` with `SUMMARIZECOLUMNS` or `ADDCOLUMNS`, and add `ORDER BY`.3. Run it with Power BI Modeling MCP (`dax_query_operations`) against the model open in Power BI Desktop.4. Copy the result rows into the run file. Rename the keys to the `answer_columns` names.
If Power BI Desktop isn't available (macOS, Linux, or the cloud agent), stop and say so. Don't invent numbers. The learner can use the sample runs in `evaluation/runs/sample-*.json` instead.View .github/skills/business-question-evaluation/SKILL.md on GitHub
MAINTAINER_ONLY = [ ("evaluation/expected/", "Expected answers are hidden from agents so question runs stay honest."), ( "evaluation/questions.yaml", "questions.yaml holds reference answers. Use: python -m pharmacy_demo questions.", ), ("tools/python/pharmacy_demo/modelgen/", "The checkpoint generator contains the lab answers."), ("tools/python/pharmacy_demo/samples.py", "Sample runs are generated from the answers."), ( "pharmacy_demo dax-queries", "Reference DAX is hidden from agents. Run scripts/Invoke-DaxQuestionTests.ps1 for pass or fail only.", ), if not maintainer: for target in targets: reason = _match(target, MAINTAINER_ONLY) if reason: return "deny", f"{target.lstrip('/')}: {reason}" if is_shell: for command in _collect(args, COMMAND_KEYS): for pattern, reason in SHELL_DENY: if pattern.search(command): return "deny", reason if not maintainer: normalized = command.replace("\\", "/") for fragment, reason in MAINTAINER_ONLY: if fragment in normalized: return "deny", reasonIn VS Code, the main protection is the agent instruction. In Copilot CLI and the cloud agent, the pre-tool-use hook denies file reads or commands that try to access evaluation/expected/, evaluation/questions.yaml, the sample generator, or pharmacy_demo dax-queries.
You should see: the same forbidden paths in both the agent instructions and the hook. This is why the tester uses the public script instead of opening the YAML file directly.
Step 3. List the public questions, not the answers
Section titled “Step 3. List the public questions, not the answers”Run the script the agent is allowed to use:
pwsh ./scripts/Get-BusinessQuestions.ps1pwsh ./scripts/Get-BusinessQuestions.ps1The script’s help says it prints the five questions, paraphrases, and answer columns without reference answers:
#Requires -Version 7.2<#.SYNOPSIS Prints the five business questions, their paraphrases, and the answer columns, without answers..DESCRIPTION This is the only way agents and learners should read the question set. The full file, evaluation/questions.yaml, also holds reference answers and is hidden from agents by the hooks..PARAMETER Json Print the questions as JSON..EXAMPLE pwsh ./scripts/Get-BusinessQuestions.ps1#>[CmdletBinding(PositionalBinding = $false)]param([switch] $Json)
. (Join-Path $PSScriptRoot '_Common.ps1')$arguments = @('questions')if ($Json) { $arguments += '--json' }exit (Invoke-PharmacyDemo @arguments)You should see: Q01 to Q05, each with paraphrases and answer_columns. You should not see the expected rows, reference SQL, or reference DAX.
Step 4. Run the prompt with the local-baseline label
Section titled “Step 4. Run the prompt with the local-baseline label”- Open Copilot Chat: Ctrl+Alt+I (macOS: Ctrl+Cmd+I), or View > Chat.
- Switch to Agent mode.
- Pick question-tester from the agent dropdown.
- Type this prompt and press Enter:
/answer-business-questions label=local-baselineVS Code prompt files can also show an input box for label. This prompt declares the input as ${input:label:local-baseline or local-ai-ready}:
---name: answer-business-questionsdescription: Answer the five business questions against the current model, save the run file with the given label, and grade it.agent: question-testerargument-hint: Run label, for example local-baseline---Answer the Contoso Pharmacy business questions against the model open in Power BI Desktop.
Run label: `${input:label:local-baseline or local-ai-ready}`
1. List the questions with `pwsh ./scripts/Get-BusinessQuestions.ps1`.2. Answer each one with a DAX query through Power BI Modeling MCP. Record your assumptions.3. Save `evaluation/runs/${input:label}.json` in the format from the `business-question-evaluation` skill.4. Grade it with `pwsh ./scripts/Grade-QuestionRun.ps1 -Run ${input:label}`.5. Report the score per question.
If Power BI Desktop or the MCP server isn't available, stop and tell me. Don't make up numbers.View .github/prompts/answer-business-questions.prompt.md on GitHub
Approve read and DAX tool calls after you inspect them. Do not approve any model edit.
You should see: the agent calls Get-BusinessQuestions.ps1, connects through Power BI Modeling MCP, runs one DAX query per question, and writes evaluation/runs/local-baseline.json. It then runs Grade-QuestionRun.ps1 -Run local-baseline.
Step 5. Review and grade the run file
Section titled “Step 5. Review and grade the run file”Open evaluation/runs/local-baseline.json and compare it with the schema:
{ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/sabajamalian/fabric-powerbi-ai-devops-demo/evaluation/run.schema.json", "title": "Question run", "description": "One run of the question-tester agent (or a person) answering evaluation/questions.yaml against one model state.", "type": "object", "required": ["label", "model_state", "created", "answers"], "additionalProperties": false, "properties": { "label": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" }, "model_state": { "enum": ["baseline", "lab07", "lab08", "ai-ready", "other"] }, "illustrative": { "type": "boolean", "description": "True for committed sample runs that illustrate typical results." }, "created": { "type": "string" }, "agent": { "type": "string" }, "surface": { "enum": ["vscode", "copilot-cli", "cloud-agent", "manual", "generated"] }, "model": { "type": "string", "description": "LLM used, if known." }, "notes": { "type": "string" }, "answers": { "type": "array", "items": { "type": "object", "required": ["question_id", "rows"], "additionalProperties": false, "properties": { "question_id": { "type": "string", "pattern": "^Q\\d{2}$" }, "answer_text": { "type": "string" }, "dax": { "type": "string" }, "assumptions": { "type": "array", "items": { "type": "string" } }, "rows": { "type": "array", "items": { "type": "object" } }, "grade_override": { "type": "object", "required": ["score", "reason"], "properties": { "score": { "enum": [0, 1, 2] }, "reason": { "type": "string", "minLength": 5 } } } } } }, "grade": { "type": "object", "description": "Written by Grade-QuestionRun.ps1. Don't edit by hand.", "required": ["total", "max", "by_question"], "properties": { "total": { "type": "integer" }, "max": { "type": "integer" }, "graded": { "type": "string" }, "by_question": { "type": "object" } } } }}The skill reference explains the same shape in learner terms:
# Run file format
A run file records one set of answers to the five questions. Schema: `evaluation/run.schema.json`.
```json{ "label": "local-baseline", "model_state": "baseline", "created": "2026-01-20T15:04:05Z", "agent": "question-tester", "surface": "vscode", "model": "the chat model you ran with", "notes": "Baseline model, raw names.", "illustrative": false, "answers": [ { "question_id": "Q03", "answer_text": "Average processing time per store, all of 2024 and 2025. Riverbend is slowest.", "dax": "EVALUATE\nSUMMARIZECOLUMNS(dim_store[store_nm], \"avg\", AVERAGE(fact_rx_fill[proc_mins]))", "assumptions": ["Whole data range, because no period was named."], "rows": [ { "store": "Riverbend", "avg_processing_minutes": 14.2 } ] } ]}```
## Fields
| Field | Required | Notes ||---|---|---|| `label` | yes | Lowercase letters, digits, hyphens. Use the file name without `.json`; comparisons show it as the column heading. || `model_state` | yes | `baseline`, `lab07`, `lab08`, `ai-ready`, or `other` || `created` | yes | UTC time, ISO 8601 || `agent` | no | The agent name, or `manual` || `surface` | no | `vscode`, `copilot-cli`, `cloud-agent`, `manual`, or `generated` || `model` | no | The chat model used || `notes` | no | Anything a reviewer should know || `illustrative` | no | `true` only for the generated sample runs || `answers` | yes | One entry per question you answered |View .github/skills/business-question-evaluation/references/run-file.md on GitHub
Run the grader yourself, even if the agent already did:
pwsh ./scripts/Grade-QuestionRun.ps1 -Run local-baselinepwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 05pwsh ./scripts/Grade-QuestionRun.ps1 -Run local-baselinepwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 05The grader owns the grade block:
#Requires -Version 7.2<#.SYNOPSIS Grades a question run against the expected answers and writes the grade into the run file..DESCRIPTION A run file (evaluation/runs/<label>.json) holds the answers an agent or person gave to the five business questions. Each question scores 0, 1, or 2 using the rubric in evaluation/rubric.md..PARAMETER Run A run label such as local-baseline, or a path to a run file..PARAMETER NoWrite Print the grade without updating the run file..PARAMETER Json Print the grade as JSON..EXAMPLE pwsh ./scripts/Grade-QuestionRun.ps1 -Run local-baseline#>[CmdletBinding(PositionalBinding = $false)]param( [Parameter(Mandatory)] [string] $Run, [switch] $NoWrite, [switch] $Json)
. (Join-Path $PSScriptRoot '_Common.ps1')$arguments = @('grade', $Run)if ($NoWrite) { $arguments += '--no-write' }if ($Json) { $arguments += '--json' }exit (Invoke-PharmacyDemo @arguments)You should see: a total score out of 10, a grade.by_question object in the JSON file, and PASS L05-baseline-run. Typical baseline results are low on Q02, Q04, and Q05 because the model lacks the Medication relationship, explicit time-comparison measures, and business definitions.
Step 6. Use the no-Desktop path when needed
Section titled “Step 6. Use the no-Desktop path when needed”If you cannot run Desktop, restore the completed Lab 05 checkpoint. It seeds evaluation/runs/local-baseline.json from the committed sample run, then you grade it locally.
pwsh ./scripts/lab/Restore-LabCheckpoint.ps1 -Checkpoint lab05-completepwsh ./scripts/Grade-QuestionRun.ps1 -Run local-baselinepwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 05pwsh ./scripts/lab/Restore-LabCheckpoint.ps1 -Checkpoint lab05-completepwsh ./scripts/Grade-QuestionRun.ps1 -Run local-baselinepwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 05You should see: the checkpoint creates or refreshes evaluation/runs/local-baseline.json, the grader writes the grade block, and the lab check passes. This path proves the evaluation pipeline without live DAX.
Step 7. Compare your baseline with the committed sample
Section titled “Step 7. Compare your baseline with the committed sample”Open the expected-results page for the sample scores, then read the rubric:
# Answer quality rubric
Each question in `questions.yaml` is scored 0, 1, or 2 by `scripts/Grade-QuestionRun.ps1`. A run's total is the sum across questions (10 points for the five questions).
| Score | Meaning | How the grader decides ||---|---|---|| 2 | Correct | Every expected row is present, keys match exactly (case and spacing ignored), every value is within the question's tolerance, and there are no extra rows. For `ranking` questions the order also matches. || 1 | Partially correct | The full check fails, but the question's `partial` rule passes (see below). || 0 | Wrong or missing | Neither check passes, or the answer is missing. |
## Partial credit rules
| Rule | Passes when ||---|---|| `row_fraction` | At least `min` of the expected rows are present with correct values. || `top_key` | The first row's key matches the expected first row (for example, the right slowest store). || `any_key` | At least one expected key is present and no more than one unexpected key is returned. || `column_match` | The named column matches the expected value (for example, the right December total with the wrong comparison). |
## Matching rules
- Column names are matched after lowercasing and replacing spaces and punctuation with `_`. The agent should still use the `answer_columns` names.- Percent and rate columns (`*_pct`, `*_rate`, `*_index`) accept either a fraction (0.25) or a percentage (25). The grader normalizes.- Numbers within tolerance count as equal. Tolerances live in `questions.yaml`.
## Human override
A reviewer can add `grade_override: { score, reason }` to an answer in a run file when the grader is too strict or too lenient, for example when the agent stated a reasonable alternative interpretation. The grader reports overrides separately so they're visible in the comparison table.
## What this rubric doesn't measure
It measures whether the answer matches the default interpretation in the question set. It doesn't score explanation quality, DAX style, or speed. Agent output varies between runs, so compare at least two runs per model state before drawing conclusions.The committed sample baseline scores 3 of 10, but your live run may differ by a point. What matters is the pattern: Q01 is usually easiest, Q02 fails because category filters do not reach the fact table, Q03 may sum minutes, Q04 lacks a definition of unusual refill activity, and Q05 lacks a previous-period measure.
You should see: a concrete before score and recorded assumptions. Keep them. Labs 08 to 10 use them to prove the model improved.
Verify your work
Section titled “Verify your work”Run this from the repo root in a PowerShell 7 terminal (Terminal > New Terminal in VS Code):
pwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 05pwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 05Checks marked Windows only print SKIP on macOS. SKIP doesn't count as a failure.
When everything is done you should see:
PASS L05-baseline-run evaluation/runs/local-baseline.json exists, matches the schema, and is graded
Lab 05: 1 of 1 checks passed.Each FAIL line prints a hint and a link back to the step on this page that fixes it. The command exits with code 0 when every required check passes.
| Check | What it looks at | If it fails |
|---|---|---|
L05-baseline-run | evaluation/runs/local-baseline.json exists, matches the schema, and is graded | Run the question-tester agent with label local-baseline, then pwsh ./scripts/Grade-QuestionRun.ps1 -Run local-baseline. Go to step 5 |
Troubleshooting
Section titled “Troubleshooting”The question-tester agent is missing from the picker
Run Developer: Reload Window from View > Command Palette. If it still does not appear, open .github/agents/question-tester.agent.md and check for frontmatter errors. VS Code usually reports custom-agent parse errors in View > Output, then the GitHub Copilot Chat channel.
The prompt asks me for a label even though I typed one inline
Use local-baseline in the input box and continue. VS Code prompt input handling can vary by version. The prompt file’s label input still resolves to the same run path.
The run file fails schema validation
Check that the top-level fields include label, model_state, created, and answers. Each answer needs question_id and rows. Do not hand-write the grade block. Run pwsh ./scripts/Grade-QuestionRun.ps1 -Run local-baseline after the shape is valid.
The agent tries to read evaluation/questions.yaml or expected answers
Stop the run and remind it: “Use only pwsh ./scripts/Get-BusinessQuestions.ps1 for the questions. Do not read expected answers.” In VS Code this is an instruction violation. In CLI and cloud-agent runs, the hook denies it.
DAX tool calls fail or return no rows
Go back to Lab 04. Power BI Desktop must have fabric\ContosoPharmacy.pbip open and refreshed, the MCP server must be running, and the Power BI tools must be selected in Configure Tools.
Knowledge check
Section titled “Knowledge check”Recap and next
Section titled “Recap and next”You ran your first repo custom agent, used a prompt-file input, saved a baseline evaluation run, and graded it. The run file now captures how the raw model performs before any AI-readiness fixes. Next you install a shared semantic-model authoring skill and ask a read-only assessor to explain what should change.
Go further
Section titled “Go further”- Run the question-tester a second time with
label=local-baseline-2and compare the assumptions. Do not overwritelocal-baseline. - Open Business questions after the run and compare the public contract with the hidden reference answers. Do this as a learner, not inside the tester agent.
- Add a note to
evaluation/runs/local-baseline.jsonexplaining the model state and the Copilot model you used, then rerun the grader.
Something wrong or unclear on this page? Open a lab feedback issue. The form is prefilled with this page's name. Nothing is sent until you submit it, and please don't include screenshots of real accounts.