Lab 10: question-tester, after

At a glance

Time

About 30 minutes

Level

intermediate

Platforms

Windows 11

Copilot surfaces

VS Code

You'll use

custom-agent, prompt-file, mcp

Starts from

lab09-complete: Adds AI metadata: Q&A on, default labels, and the fabric/ai-prep drafts.

Ends at

None needed

Last verified

Not yet run end to end on Windows 11

Windows only: needs Power BI Desktop

Nobody has run this lab end to end on Windows 11 yet. The steps follow the current docs for each tool, but menu labels may differ. The verify command at the end is the source of truth, and feedback is welcome.

Why this matters

You improved names, relationships, measures, and AI metadata. Now you measure whether those changes helped. The question-tester agent answers the same five questions from Lab 05, writes another run file, and lets the grader score it the same way. This is still an agent run, so wording and sometimes a point can vary, but the AI-ready model should score clearly higher than the baseline.

Concepts

• After run: a second graded question run against the AI-ready model. The label in this lab is local-ai-ready.
• Rubric: deterministic scoring against expected rows. It checks answer data, not explanation style. See Evaluation methodology.
• Run variance: agent answers can differ between runs. The comparison is useful because the same grader and same questions are used before and after.
• Deterministic DAX tests: the Lab 08 DAX tests exist because agent runs vary. They run fixed DAX queries against Desktop and don’t rely on agent judgment.

Before you begin

• You finished Lab 09, or restored lab09-complete.
• Power BI Desktop is open with fabric\ContosoPharmacy.pbip and the model is refreshed.
• The Power BI Modeling MCP server is running in VS Code.
• evaluation/runs/local-baseline.json exists from Lab 05 and has a grade block.

Skipped a lab or want a clean start?

Restore the lab09-complete checkpoint (Adds AI metadata: Q&A on, default labels, and the fabric/ai-prep drafts). The script refuses to run if you have uncommitted changes, creates a new lab/lab09-complete-... branch, and restores only fabric/ and lab/samples/. Close Power BI Desktop first.

pwsh ./scripts/lab/Restore-LabCheckpoint.ps1 -Checkpoint lab09-complete

It needs the upstream remote from Lab 01, step 3.

Steps

Step 1. Read the tester and comparison prompts

Open the two prompt files so you know what the agent is allowed to do.

1. Open Quick Open: Ctrl+P, or Go > Go to File.
2. Open .github/prompts/answer-business-questions.prompt.md.
3. Open .github/prompts/compare-question-runs.prompt.md.
4. Open evaluation/rubric.md.

The answer prompt requires Desktop and MCP, and it tells the agent not to make up numbers:

.github/prompts/answer-business-questions.prompt.md (lines 7-17)

Answer the Contoso Pharmacy business questions against the model open in Power BI Desktop.

Run label: `${input:label:local-baseline or local-ai-ready}`

1. List the questions with `pwsh ./scripts/Get-BusinessQuestions.ps1`.
2. Answer each one with a DAX query through Power BI Modeling MCP. Record your assumptions.
3. Save `evaluation/runs/${input:label}.json` in the format from the `business-question-evaluation` skill.
4. Grade it with `pwsh ./scripts/Grade-QuestionRun.ps1 -Run ${input:label}`.
5. Report the score per question.

If Power BI Desktop or the MCP server isn't available, stop and tell me. Don't make up numbers.

View .github/prompts/answer-business-questions.prompt.md on GitHub

The comparison prompt has no paraphrase flag. It compares two graded run files:

.github/prompts/compare-question-runs.prompt.md (lines 7-17)

Compare two question runs.

- Before: `${input:before:local-baseline}`
- After: `${input:after:local-ai-ready}`

1. Run `pwsh ./scripts/Compare-QuestionRuns.ps1 -Before ${input:before} -After ${input:after}`.
2. Read `out/question-comparison.md` and both run files.
3. For each question whose score changed, name the model change that most likely caused it (a measure, a description, an AI instruction, a relationship). Point to the file.
4. For any question still below 2, suggest the next change to try.

Agent answers vary between runs, so say when a one-point change could be noise.

View .github/prompts/compare-question-runs.prompt.md on GitHub

You should see: the prompt saves evaluation/runs/<label>.json, then runs Grade-QuestionRun.ps1.

I finished step 1

Step 2. Run question-tester on the AI-ready model

Windows only

1. In Power BI Desktop, select Home > Refresh and wait for refresh to finish.
2. In VS Code, open Copilot Chat: Ctrl+Alt+I, or View > Chat.
3. Pick question-tester from the agent dropdown.
4. Type /answer-business-questions label=local-ai-ready and press Enter.
5. Approve DAX query tool calls after you read them.
6. If the agent asks to read evaluation/expected/ or evaluation/questions.yaml, deny it and remind it to use pwsh ./scripts/Get-BusinessQuestions.ps1.

The evaluation skill explains the honest path:

.github/skills/business-question-evaluation/SKILL.md (lines 9-25)

## Rules that keep the test honest

- Get the questions only from `pwsh ./scripts/Get-BusinessQuestions.ps1` (add `-Json` for machine output). It prints the ID, the question, paraphrases, and the `answer_columns` you must use.
- Never read `evaluation/expected/`, `evaluation/questions.yaml`, `tools/python/pharmacy_demo/modelgen/`, or `tools/python/pharmacy_demo/samples.py`, and never run `pharmacy_demo dax-queries`. They contain the answers. The repo's hooks deny these.
- Answer from the model as it is now. On the baseline model, use the names you find (for example `fact_rx_fill`), and don't fix the model first. A low baseline score is the expected result.
- Don't write the `grade` block. Only the grader writes it.

## Step 1. Answer each question

For each question:

1. Decide the interpretation. If the question is ambiguous (which month is "latest", what "unusually high" means), write your choice in `assumptions`.
2. Write one DAX query that returns exactly the `answer_columns`, one row per expected key. Use `EVALUATE` with `SUMMARIZECOLUMNS` or `ADDCOLUMNS`, and add `ORDER BY`.
3. Run it with Power BI Modeling MCP (`dax_query_operations`) against the model open in Power BI Desktop.
4. Copy the result rows into the run file. Rename the keys to the `answer_columns` names.

If Power BI Desktop isn't available (macOS, Linux, or the cloud agent), stop and say so. Don't invent numbers. The learner can use the sample runs in `evaluation/runs/sample-*.json` instead.

View .github/skills/business-question-evaluation/SKILL.md on GitHub

You should see: evaluation/runs/local-ai-ready.json with five answers, DAX, assumptions, rows, and a grade block. The agent reports a score per question after running pwsh ./scripts/Grade-QuestionRun.ps1 -Run local-ai-ready.

What varies between runs

Copilot's wording, the order it works in, and the exact DAX or TMDL it writes will differ from what you see here and from your neighbor's run. That's expected.

The DAX and wording may differ from the sample run. A one-point difference on one question can be normal.

These must match:

• The run label is local-ai-ready
• The file has a grade block
• The agent does not read expected answers

I finished step 2

Step 3. Compare baseline and AI-ready runs

Run the comparison prompt or the script directly. The lab check looks for both a higher AI-ready score and out/question-comparison.md.

Windows

pwsh ./scripts/Compare-QuestionRuns.ps1 -Before local-baseline -After local-ai-ready

The script writes to out/question-comparison.md by default:

scripts/Compare-QuestionRuns.ps1

#Requires -Version 7.2
<#
.SYNOPSIS
    Compares two graded question runs and writes a Markdown before-and-after table.
.PARAMETER Before
    Run label or path for the earlier run, for example local-baseline.
.PARAMETER After
    Run label or path for the later run, for example local-ai-ready.
.PARAMETER OutFile
    Where to write the table. Default: out/question-comparison.md.
.EXAMPLE
    pwsh ./scripts/Compare-QuestionRuns.ps1 -Before local-baseline -After local-ai-ready
.EXAMPLE
    pwsh ./scripts/Compare-QuestionRuns.ps1 -Before sample-baseline -After sample-ai-ready
#>
[CmdletBinding(PositionalBinding = $false)]
param(
    [Parameter(Mandatory)] [string] $Before,
    [Parameter(Mandatory)] [string] $After,
    [string] $OutFile
)

. (Join-Path $PSScriptRoot '_Common.ps1')
if (-not $OutFile) { $OutFile = Join-Path (Get-RepoRoot) 'out' 'question-comparison.md' }
$target = [System.IO.Path]::GetFullPath($OutFile, (Get-Location).Path)
exit (Invoke-PharmacyDemo compare --before $Before --after $After --out $target)

View scripts/Compare-QuestionRuns.ps1 on GitHub

You should see: out/question-comparison.md with a before and after table. The total score for local-ai-ready should be higher than local-baseline. Q02, Q04, and Q05 should show the clearest improvement.

I finished step 3

Step 4. Read the score, not just the prose

Open the comparison and both run files.

1. Open out/question-comparison.md.
2. Open evaluation/runs/local-baseline.json.
3. Open evaluation/runs/local-ai-ready.json.
4. Compare the grade.by_question blocks before you compare answer text.

The rubric is strict about rows and values:

evaluation/rubric.md

# Answer quality rubric

Each question in `questions.yaml` is scored 0, 1, or 2 by `scripts/Grade-QuestionRun.ps1`. A run's total is the sum across questions (10 points for the five questions).

| Score | Meaning | How the grader decides |
|---|---|---|
| 2 | Correct | Every expected row is present, keys match exactly (case and spacing ignored), every value is within the question's tolerance, and there are no extra rows. For `ranking` questions the order also matches. |
| 1 | Partially correct | The full check fails, but the question's `partial` rule passes (see below). |
| 0 | Wrong or missing | Neither check passes, or the answer is missing. |

## Partial credit rules

| Rule | Passes when |
|---|---|
| `row_fraction` | At least `min` of the expected rows are present with correct values. |
| `top_key` | The first row's key matches the expected first row (for example, the right slowest store). |
| `any_key` | At least one expected key is present and no more than one unexpected key is returned. |
| `column_match` | The named column matches the expected value (for example, the right December total with the wrong comparison). |

## Matching rules

- Column names are matched after lowercasing and replacing spaces and punctuation with `_`. The agent should still use the `answer_columns` names.
- Percent and rate columns (`*_pct`, `*_rate`, `*_index`) accept either a fraction (0.25) or a percentage (25). The grader normalizes.
- Numbers within tolerance count as equal. Tolerances live in `questions.yaml`.

## Human override

A reviewer can add `grade_override: { score, reason }` to an answer in a run file when the grader is too strict or too lenient, for example when the agent stated a reasonable alternative interpretation. The grader reports overrides separately so they're visible in the comparison table.

## What this rubric doesn't measure

It measures whether the answer matches the default interpretation in the question set. It doesn't score explanation quality, DAX style, or speed. Agent output varies between runs, so compare at least two runs per model state before drawing conclusions.

View evaluation/rubric.md on GitHub

The committed sample results show the pattern you should expect:

Question	Typical baseline	Typical AI-ready
Q01	2	2
Q02	0	2
Q03	1	2
Q04	0	2
Q05	0	2

You should see: Q02 improves after the Medication relationship and time-comparison measures, Q03 improves after average processing time, Q04 improves after the refill index definition, and Q05 improves after previous period is defined.

I finished step 4

Step 5. Account for run variance

Run variance is expected because the agent chooses DAX and interpretations. Don’t overreact to a one-point wobble.

1. Ask question-tester: Explain any question in local-ai-ready that scored below 2. Point to the model change that would help.
2. If one score looks surprising, run the after test a second time with a different local label.

Windows

# Optional second run. The prompt supports labels, not a paraphrase flag.
# In chat: /answer-business-questions label=local-ai-ready-2
pwsh ./scripts/Grade-QuestionRun.ps1 -Run local-ai-ready-2
pwsh ./scripts/Compare-QuestionRuns.ps1 -Before local-ai-ready -After local-ai-ready-2 -OutFile out/question-variance.md

You should see: two AI-ready runs that are close. If they differ, the assumptions field usually explains why. The deterministic proof still comes from the Lab 08 DAX tests, not from repeated agent runs.

I finished step 5

Step 6. Keep the local outputs local

The files this lab creates are for your machine. Don’t commit them unless a maintainer asks for a sample.

Windows

git status --short evaluation/runs out

You should see: local-ai-ready.json and out/question-comparison.md may appear as ignored or untracked local output, depending on your Git settings. Your model files should be unchanged after this lab.

I finished step 6

Verify your work

Run this from the repo root in a PowerShell 7 terminal (Terminal > New Terminal in VS Code):

Windows

pwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 10

macOS

pwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 10

Checks marked Windows only print SKIP on macOS. SKIP doesn't count as a failure.

When everything is done you should see:

Expected output

PASS  L10-after-run   evaluation/runs/local-ai-ready.json exists and is graded
PASS  L10-improved    local-ai-ready scores higher than local-baseline
PASS  L10-comparison  out/question-comparison.md exists

Lab 10: 3 of 3 checks passed.

Each FAIL line prints a hint and a link back to the step on this page that fixes it. The command exits with code 0 when every required check passes.

Check	What it looks at	If it fails
L10-after-run	evaluation/runs/local-ai-ready.json exists and is graded	Run question-tester with label local-ai-ready, then Grade-QuestionRun.ps1. Go to step 2
L10-improved	local-ai-ready scores higher than local-baseline	Compare the runs with pwsh ./scripts/Compare-QuestionRuns.ps1 and review the failed questions. Go to step 3
L10-comparison	out/question-comparison.md exists	Run pwsh ./scripts/Compare-QuestionRuns.ps1 -Before local-baseline -After local-ai-ready. Go to step 3

Troubleshooting

The agent says Desktop or MCP isn’t available

Open fabric\ContosoPharmacy.pbip in Power BI Desktop, refresh it, and start the Power BI Modeling MCP server from Ctrl+Shift+P > MCP: List Servers. Then rerun /answer-business-questions label=local-ai-ready.

The run file has no grade block

Run pwsh ./scripts/Grade-QuestionRun.ps1 -Run local-ai-ready. The evaluation instructions say the grader writes the grade block, not the agent by hand.

The comparison says AI-ready did not improve

Open both run files and check labels first. If the labels are right, read each failed question’s assumptions. A wrong latest month, summed processing minutes, or missing refill threshold means the agent didn’t use the AI-ready metadata. Ask it to answer again and cite the measure or instruction it used.

I want to test paraphrases

Each question in evaluation/questions.yaml lists paraphrases (see Business questions). Ask the question-tester one of them in chat, for example “Which categories grew or shrank the most last month?”, and compare its DAX with the graded run. The grader and the Lab 10 check only score the main wording, so treat this as exploration.

Knowledge check

Why does Lab 08 have deterministic DAX tests if Lab 10 already uses an agent?

Agent answers can vary, so fixed DAX tests prove the model returns the expected numbers.The DAX tests train Copilot before the run.The DAX tests replace Power BI Desktop.

Show the answer

Agent answers can vary, so fixed DAX tests prove the model returns the expected numbers.

The agent run measures AI behavior. The DAX tests measure the model with fixed queries, so they don't vary by prompt wording or agent choices.

What does a score of 2 mean?

The explanation is polished and includes a chart.Every expected row and value matches within tolerance, with no extra rows.The agent used the shortest DAX query.

Show the answer

Every expected row and value matches within tolerance, with no extra rows.

The rubric scores the answer rows. It doesn't score explanation quality, DAX style, or speed.

What should you do before committing after Lab 10?

Commit local-ai-ready.json and out/question-comparison.md.Check that only local outputs changed, then leave them uncommitted.Delete the model changes from Labs 07 to 09.

Show the answer

Check that only local outputs changed, then leave them uncommitted.

Local run files and comparison output are evidence for your machine. They are usually gitignored and not part of the model PR.

Recap and next

You reran the same business questions against the AI-ready model, graded the after run, and compared it with the Lab 05 baseline. You also saw why deterministic tests and agent-based evaluation answer different questions. Next you run the same customization stack in Copilot CLI and watch hooks enforce guardrails.

Next: Lab 11: Copilot CLI and hooks

Go further

• Run one exploratory after test with paraphrased prompts, then compare its assumptions with local-ai-ready.
• Ask model-reviewer to explain any remaining score below 2 and classify it as a model gap, instruction gap, or agent variance.
• Open Expected results and compare your score pattern with the committed sample.

Something wrong or unclear on this page? Open a lab feedback issue. The form is prefilled with this page's name. Nothing is sent until you submit it, and please don't include screenshots of real accounts.