Lab 11: Copilot CLI and hooks

At a glance

Time

About 35 minutes

Level

advanced

Platforms

Windows 11, macOS

Copilot surfaces

Copilot CLI

You'll use

cli, custom-agent, skill, hook

Starts from

lab09-complete: Adds AI metadata: Q&A on, default labels, and the fabric/ai-prep drafts.

Ends at

None needed

Last verified

Not yet run end to end on Windows 11

Works on macOS

Nobody has run this lab end to end on Windows 11 yet. The steps follow the current docs for each tool, but menu labels may differ. The verify command at the end is the source of truth, and feedback is welcome.

Why this matters

Instructions are guidance. Hooks are code. In this lab you run Copilot CLI from the repo, pick the same custom agents and skills, and watch a deterministic hook deny an unsafe edit before the agent can make it. Then you make a TMDL edit and see the post-tool hook run the local linter automatically.

Concepts

• Copilot CLI: the terminal Copilot surface. GitHub docs say you install it with npm install -g @github/copilot, start it with copilot, trust the folder, and use slash commands such as /agent and /skills.
• Repository hook: a JSON file under .github/hooks/*.json that runs commands on lifecycle events. See Hooks.
• preToolUse: a hook event that can return permissionDecision: "deny" with a reason before a tool runs.
• postToolUse: a hook event that can add context after a successful tool call. This repo uses it to run the TMDL lint after .tmdl edits.
• VS Code agent hooks: VS Code documents agent hooks as preview. VS Code can load hooks from .github/hooks, but event names and OS-specific command fields can differ by version. Use Copilot CLI for this lab’s hook checks.

Before you begin

• You finished Lab 10 or restored lab09-complete.
• You ran pwsh ./scripts/Initialize-DevEnvironment.ps1 in Lab 01, so .venv exists. The hook wrappers do nothing if .venv is missing.
• Copilot CLI is installed. If it isn’t, follow GitHub’s current install command: npm install -g @github/copilot.
• Close Power BI Desktop if it is open. This lab may touch TMDL text and then restore the demo edit.

Skipped a lab or want a clean start?

Restore the lab09-complete checkpoint (Adds AI metadata: Q&A on, default labels, and the fabric/ai-prep drafts). The script refuses to run if you have uncommitted changes, creates a new lab/lab09-complete-... branch, and restores only fabric/ and lab/samples/. Close Power BI Desktop first.

pwsh ./scripts/lab/Restore-LabCheckpoint.ps1 -Checkpoint lab09-complete

It needs the upstream remote from Lab 01, step 3.

Steps

Step 1. Inspect the hook configuration

Open the hook files before you start the CLI.

1. Open Quick Open: Ctrl+P (macOS: Cmd+P), or Go > Go to File.
2. Open .github/hooks/guardrails.json.
3. Open scripts/hooks/run-hook.ps1 and scripts/hooks/run-hook.sh.
4. Open tools/python/pharmacy_demo/hooks.py.

The JSON wires each lifecycle event to a PowerShell or bash wrapper:

.github/hooks/guardrails.json

{
  "version": 1,
  "hooks": {
    "sessionStart": [
      {
        "type": "command",
        "bash": "./scripts/hooks/run-hook.sh session-start",
        "powershell": "pwsh -NoProfile -File ./scripts/hooks/run-hook.ps1 -HookEvent session-start",
        "cwd": ".",
        "timeoutSec": 20
      }
    ],
    "preToolUse": [
      {
        "type": "command",
        "bash": "./scripts/hooks/run-hook.sh pre-tool-use",
        "powershell": "pwsh -NoProfile -File ./scripts/hooks/run-hook.ps1 -HookEvent pre-tool-use",
        "cwd": ".",
        "timeoutSec": 20
      }
    ],
    "postToolUse": [
      {
        "type": "command",
        "bash": "./scripts/hooks/run-hook.sh post-tool-use",
        "powershell": "pwsh -NoProfile -File ./scripts/hooks/run-hook.ps1 -HookEvent post-tool-use",
        "cwd": ".",
        "timeoutSec": 30
      }
    ],
    "sessionEnd": [
      {
        "type": "command",
        "bash": "./scripts/hooks/run-hook.sh session-end",
        "powershell": "pwsh -NoProfile -File ./scripts/hooks/run-hook.ps1 -HookEvent session-end",
        "cwd": ".",
        "timeoutSec": 10
      }
    ]
  }
}

View .github/hooks/guardrails.json on GitHub

The Python implementation writes JSON lines to out/hooks/session.log:

tools/python/pharmacy_demo/hooks.py (lines 92-103)

def _log(root: Path, entry: dict) -> None:
    try:
        folder = root / "out" / "hooks"
        folder.mkdir(parents=True, exist_ok=True)
        with (folder / "session.log").open("a", encoding="utf-8", newline="\n") as fh:
            fh.write(json.dumps({"time": _now(), **entry}) + "\n")
    except OSError:
        pass


def _emit(event_name: str, **fields: str) -> None:
    print(json.dumps({**fields, "hookSpecificOutput": {"hookEventName": event_name, **fields}}))

View tools/python/pharmacy_demo/hooks.py on GitHub

You should see: sessionStart, preToolUse, postToolUse, and sessionEnd configured. On Windows the wrapper calls scripts/hooks/run-hook.ps1 -HookEvent <event>. On macOS it calls scripts/hooks/run-hook.sh <event>.

I finished step 1

Step 2. Start Copilot CLI and trust the folder

Windows

copilot

macOS

copilot

1. When Copilot asks whether you trust this folder, choose Yes, proceed for this session, or Yes, and remember this folder for future sessions if this is your own clone.
2. If the CLI asks you to sign in, type /login and follow the browser flow.
3. Wait for the first prompt to appear.

You should see: the sessionStart hook injects a message like “Contoso Pharmacy lab. All data is synthetic” and reports the model state. The hook also appends a sessionStart entry to out/hooks/session.log.

I finished step 2

Step 3. List agents and skills in the CLI

Use the CLI slash commands to see the repo customizations.

/agent
/skills list
/skills info prep-data-for-ai

1. In /agent, choose model-improver for the next two steps.
2. In /skills list, confirm the repo skills appear.
3. In /skills info prep-data-for-ai, confirm the skill location is under this repo.

You should see: the same custom agents from .github/agents/ and skills from .github/skills/ that you used in VS Code. GitHub docs also support /skills reload if you add a skill during a running CLI session.

I finished step 3

Step 4. Trigger a denied tool call

Ask the selected agent to make an edit that the repo never allows:

Change data/generated/dim_store.csv so Riverbend is in the North region.

Approve only enough for the agent to attempt the edit. Don’t approve a workaround that changes generator code or writes somewhere else.

The denial rules are in Python:

tools/python/pharmacy_demo/hooks.py (lines 40-68)

NEVER_EDIT = [
    (
        "data/generated/",
        "Generated data. Change tools/python/pharmacy_demo/datagen.py and run scripts/New-SyntheticData.ps1.",
    ),
    (
        "evaluation/expected/",
        "Expected answers come from the DuckDB oracle. Run scripts/Update-ExpectedResults.ps1.",
    ),
    ("/.pbi/cache.abf", "Power BI Desktop cache file."),
    ("/.pbi/localSettings.json", "Power BI Desktop local settings."),
    (
        ".SemanticModel/Copilot/",
        "Power BI Desktop writes the Copilot folder. Draft in fabric/ai-prep/ and apply through Prep data for AI.",
    ),
]
MAINTAINER_ONLY = [
    ("evaluation/expected/", "Expected answers are hidden from agents so question runs stay honest."),
    (
        "evaluation/questions.yaml",
        "questions.yaml holds reference answers. Use: python -m pharmacy_demo questions.",
    ),
    ("tools/python/pharmacy_demo/modelgen/", "The checkpoint generator contains the lab answers."),
    ("tools/python/pharmacy_demo/samples.py", "Sample runs are generated from the answers."),
    (
        "pharmacy_demo dax-queries",
        "Reference DAX is hidden from agents. Run scripts/Invoke-DaxQuestionTests.ps1 for pass or fail only.",
    ),
]

View tools/python/pharmacy_demo/hooks.py on GitHub

The hook returns a deny decision and reason in the format the Copilot hooks reference documents:

tools/python/pharmacy_demo/hooks.py (lines 230-235)

    if event == "pre-tool-use":
        decision, reason = decide(payload, root, maintainer)
        _log(root, {"event": "preToolUse", "tool": name, "decision": decision or "none", "reason": reason})
        if decision:
            _emit("PreToolUse", permissionDecision=decision, permissionDecisionReason=reason)
        return 0

View tools/python/pharmacy_demo/hooks.py on GitHub

You should see: the CLI tells the agent the tool call was denied. The reason names data/generated/dim_store.csv and says generated data must come from the generator. out/hooks/session.log gets a JSON line with "event":"preToolUse" and "decision":"deny".

What varies between runs

Copilot's wording, the order it works in, and the exact DAX or TMDL it writes will differ from what you see here and from your neighbor's run. That's expected.

The exact tool name depends on how the agent tries to edit the file. The hook checks paths, so the denial should be the same.

These must match:

• out/hooks/session.log has a preToolUse entry
• The latest denied entry has decision deny
• No file under data/generated/ changes

I finished step 4

Step 5. Trigger the postToolUse TMDL lint

Now ask for a small TMDL edit that you can restore afterward:

Make a harmless description wording improvement in fabric/ContosoPharmacy.SemanticModel/definition/tables/Date.tmdl, then stop after saving the file.

Approve the edit after you inspect it. If the agent proposes a model-wide change, tell it: “Only make one description wording change in Date.tmdl.”

The post-tool hook runs the TMDL lint only when a .tmdl file was edited:

tools/python/pharmacy_demo/hooks.py (lines 206-252)

def _edited_tmdl(payload: dict, root: Path) -> bool:
    _, args = _tool(payload)
    targets = _collect(args, PATH_KEYS) + _patch_paths(args)
    return any(t.lower().endswith(".tmdl") for t in targets)


def main(event: str, root: Path) -> int:
    payload = _payload()
    maintainer = os.environ.get("CPDEMO_MAINTAINER") == "1"
    name, _ = _tool(payload)

    if event == "session-start":
        state = "unknown"
        sm = root / "fabric" / "ContosoPharmacy.SemanticModel" / "definition" / "tables"
        if sm.is_dir():
            state = "baseline names" if (sm / "fact_rx_fill.tmdl").exists() else "renamed model"
        message = (
            "Contoso Pharmacy lab. All data is synthetic; never add real patient, customer, or tenant data. "
            f"Model state: {state}."
        )
        _log(root, {"event": "sessionStart", "state": state})
        _emit("SessionStart", additionalContext=message)
        return 0

    if event == "pre-tool-use":
        decision, reason = decide(payload, root, maintainer)
        _log(root, {"event": "preToolUse", "tool": name, "decision": decision or "none", "reason": reason})
        if decision:
            _emit("PreToolUse", permissionDecision=decision, permissionDecisionReason=reason)
        return 0

    if event == "post-tool-use":
        if not _edited_tmdl(payload, root):
            return 0
        from . import tmdl

        findings = tmdl.errors(tmdl.lint(root / "fabric" / "ContosoPharmacy.SemanticModel"))
        summary = "pass" if not findings else f"{len(findings)} error(s)"
        _log(root, {"event": "postToolUse", "tool": name, "lint": summary})
        if findings:
            lines = [f"{f.file}:{f.line} {f.rule} {f.message}" for f in findings[:10]]
            _emit(
                "PostToolUse",
                additionalContext="TMDL lint found errors after your edit. Fix them before continuing:\n"
                + "\n".join(lines),
            )
        return 0

View tools/python/pharmacy_demo/hooks.py on GitHub

You should see: the CLI continues after the edit, and out/hooks/session.log gets a JSON line with "event":"postToolUse" and a "lint" value such as "pass" or "1 error(s)". If lint errors appear, ask the agent to fix them before continuing.

I finished step 5

Step 6. Read the hook log and restore the demo edit

Open the log from the terminal and then restore the harmless TMDL edit.

Windows

Get-Content out/hooks/session.log -Tail 20
Select-String -Path out/hooks/session.log -Pattern '"decision":"deny"','"lint"'
git restore -- fabric/ContosoPharmacy.SemanticModel/definition/tables/Date.tmdl

macOS

tail -20 out/hooks/session.log
grep -E '"decision":"deny"|"lint"' out/hooks/session.log
git restore -- fabric/ContosoPharmacy.SemanticModel/definition/tables/Date.tmdl

You should see: at least one denied preToolUse event and at least one postToolUse event with lint. git status --short fabric should be clean after the restore.

I finished step 6

Step 7. End the CLI session cleanly

In Copilot CLI, press Ctrl+C once and confirm that you want to exit. If your version lists a quit slash command in ?, you can use that instead. Then read the last log entry.

Windows

Get-Content out/hooks/session.log -Tail 5

macOS

tail -5 out/hooks/session.log

You should see: a sessionEnd entry if the CLI emitted the event. Some exits can terminate before a long-running end hook writes, so the Lab 11 checks focus on the deny and lint entries.

I finished step 7

Step 8. Compare hooks with instructions

Read the guardrail side by side with the evaluation instructions.

.github/instructions/evaluation.instructions.md (lines 7-13)

- `evaluation/expected/*.json` are generated by the DuckDB oracle from the synthetic CSVs. Never edit them. Maintainers regenerate them with `pwsh ./scripts/Update-ExpectedResults.ps1`.
- When you answer questions as the `question-tester` agent, don't open `evaluation/questions.yaml` or anything under `evaluation/expected/`. Use `pwsh ./scripts/Get-BusinessQuestions.ps1`, which leaves out the reference answers.
- A run file lives at `evaluation/runs/<label>.json` and must match `evaluation/run.schema.json`. Labels are lowercase with hyphens, for example `local-baseline`.
- Each answer's `rows` use the `answer_columns` names from the questions output, in that order. Put every assumption you made in `assumptions`.
- Never write the `grade` block yourself. `pwsh ./scripts/Grade-QuestionRun.ps1 -Run <label>` writes it.
- `sample-*.json` runs are illustrative and committed. `local-*.json` runs are yours and gitignored.
- Scoring rules are in `evaluation/rubric.md`.

View .github/instructions/evaluation.instructions.md on GitHub

The instruction says not to read or edit answers. The hook enforces the dangerous parts with code. That means a prompt injection can tell the agent to ignore instructions, but it still can’t edit data/generated/ or evaluation/expected/ through a normal tool call when the hook runs.

VS Code preview status

VS Code’s agent hooks documentation says hooks are preview and can be disabled by organization policy. It also documents PascalCase event names such as PreToolUse and PostToolUse. This repo’s lab check is based on the Copilot CLI log format from tools/python/pharmacy_demo/hooks.py, so use CLI for the required evidence.

You should see: the difference between advisory guidance and deterministic checks. The same policy appears in instructions for model behavior and hooks for enforcement.

I finished step 8

Verify your work

Run this from the repo root in a PowerShell 7 terminal (Terminal > New Terminal in VS Code):

Windows

pwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 11

macOS

pwsh ./scripts/lab/Test-LabProgress.ps1 -Lab 11

Checks marked Windows only print SKIP on macOS. SKIP doesn't count as a failure.

When everything is done you should see:

Expected output

PASS  L11-hook-deny  The hook log shows at least one denied tool call
PASS  L11-hook-lint  The hook log shows a postToolUse TMDL lint result

Lab 11: 2 of 2 checks passed.

Each FAIL line prints a hint and a link back to the step on this page that fixes it. The command exits with code 0 when every required check passes.

Check	What it looks at	If it fails
L11-hook-deny	The hook log shows at least one denied tool call	Ask the agent in Copilot CLI to edit data/generated/dim_store.csv and watch the hook deny it. Go to step 4
L11-hook-lint	The hook log shows a postToolUse TMDL lint result	Ask the agent to edit a .tmdl file; the post-edit hook runs the lint. Go to step 5

Troubleshooting

Copilot CLI isn’t installed

Install it with the current GitHub Docs command: npm install -g @github/copilot. Node.js 22 or later is required for the npm install path. You can also use WinGet on Windows or Homebrew on macOS if those are allowed on your machine.

The hook says .venv is missing

Run pwsh ./scripts/Initialize-DevEnvironment.ps1 from the repo root. The wrappers intentionally exit 0 when .venv is missing so a fresh clone still works, but the guardrails won’t run until the Python tools are installed.

No denied event appears in session.log

Make sure you asked for an edit under data/generated/, evaluation/expected/, .pbi/, or *.SemanticModel/Copilot/. A read-only request doesn’t trigger the deny. If the agent refused before using a tool, ask it to apply the edit so the hook can block the tool call.

No lint event appears after the TMDL step

The post-tool hook only logs lint when the tool input names a path ending in .tmdl. Ask for a direct edit to fabric/ContosoPharmacy.SemanticModel/definition/tables/Date.tmdl, not a general review or a command that doesn’t pass the file path to the hook.

VS Code hooks behave differently

VS Code agent hooks are preview. Check Developer: Show Agent Debug Logs and your organization’s policy settings. For this lab, use Copilot CLI because Test-LabProgress.ps1 -Lab 11 reads out/hooks/session.log written by the repo’s hook implementation.

Knowledge check

Why use a hook instead of only an instruction?

A hook runs deterministic code at tool boundaries, while an instruction only guides model behavior.A hook improves DAX performance.A hook lets Power BI Desktop refresh faster.

Show the answer

A hook runs deterministic code at tool boundaries, while an instruction only guides model behavior.

Instructions shape the agent's choices. Hooks inspect actual tool calls and can deny or add context regardless of prompt wording.

Which log entry satisfies the Lab 11 deny check?

A sessionStart entry with model state.A preToolUse entry with decision deny.A postToolUse entry with lint pass.

Show the answer

A preToolUse entry with decision deny.

The check reads out/hooks/session.log and looks for event preToolUse plus decision deny.

When does this repo's postToolUse lint run?

After any prompt.After a successful tool call that edited a path ending in .tmdl.Only when Power BI Desktop is open.

Show the answer

After a successful tool call that edited a path ending in .tmdl.

The hook collects tool paths and runs the TMDL lint only when a .tmdl target was edited.

Recap and next

You started Copilot CLI, loaded the repo’s custom agents and skills, and captured two hook proofs: a denied unsafe edit and an automatic TMDL lint after a model edit. Next you push your feature branch, open a pull request, let CI validate it, and request a Copilot code review.

Next: Lab 12: Pull request, CI, and Copilot code review

Go further

• Add a user-level hook in ~/.copilot/hooks/ that logs session starts, then remove it after testing.
• Ask Copilot CLI to explain the latest denied log entry and suggest the safe path.
• Compare the CLI hook docs with VS Code’s preview hook docs and note the event-name differences.

Something wrong or unclear on this page? Open a lab feedback issue. The form is prefilled with this page's name. Nothing is sent until you submit it, and please don't include screenshots of real accounts.