Add Copilot Review Analyst skill — automated review effectiveness analysis pipeline, Fixes AB#3557002

[shahzaibj]

Summary

Adds the copilot-review-analyst skill -- an end-to-end pipeline for measuring GitHub Copilot code review effectiveness across our Android Auth repositories (Common, MSAL, Broker).

AB#3557002

What This Does

When invoked, the skill runs a 5-phase analysis pipeline:

1. Data Collection (analyze.ps1) -- Fetches all Copilot inline review comments from human-authored PRs via the GitHub API, records which comments received human replies
2. Diff Verification (precise.ps1) -- For unreplied comments, checks if the engineer silently acted on the feedback by comparing commit diffs
3. AI Reply Classification -- The agent reads every replied comment in context and classifies it as helpful or not-helpful based on the engineer's reply
4. Final Classification (final-classification.ps1) -- Merges all evidence into a single dataset with per-comment verdicts, per-engineer stats, and per-repo breakdowns
5. Report Generation -- Produces Markdown and Outlook-compatible HTML reports (both team-internal with real names and org-wide anonymized versions)

Key Features

• Automated auth handling -- Script auto-detects the EMU (*_microsoft) account from gh auth status, switches to it for private repo access, and restores the original account on completion
• History tracking (append-history.ps1) -- Appends each run's snapshot to a persistent history file for trend analysis across periods
• Three-way classification -- Comments are classified as Confirmed Helpful, Confirmed Not Helpful, or Unresolved (no evidence either way) -- never assumed
• Account merging -- Engineers with separate personal/EMU GitHub accounts are merged via account-map.json

Files Added

Path	Purpose
SKILL.md	Skill instructions -- invocation guide, pipeline phases, report generation rules
scripts/analyze.ps1	Phase 1: Data collection with auth auto-switching
scripts/precise.ps1	Phase 2: Diff-level verification of unreplied comments
scripts/final-classification.ps1	Phase 4: Merge all evidence into final dataset
scripts/append-history.ps1	Phase 4.5: Append run snapshot to history for trends
references/classification-rules.md	Phase 3 guide: how to classify replied comments
references/report-formatting.md	Phase 5 guide: report structure and Outlook HTML rules
references/account-map.json	GitHub login to display name mapping
references/manual-audit-template.json	Schema for Phase 3 output files
assets/*.md, *.html	Report templates (from first analysis) used as structural references

Results So Far

This skill has been used to produce two effectiveness reports:

• Jan 24-Mar 25, 2026 (60 days, 557 comments) -- Baseline: 44% response rate, 39% helpful, 44% unresolved
• Mar 24-Apr 19, 2026 (26 days, 170 comments) -- After team engagement push: 79% response rate (+34pp), 63% helpful (+24pp), 16% unresolved (-28pp)

Testing

• Ran the full pipeline end-to-end for the Mar 24-Apr 19 period
• Verified auth auto-switching works (personal -> EMU -> personal restore)
• Generated all 4 report variants (team/anonymous x markdown/HTML)
• Validated trend data with 2 history entries

[github-actions]

❌ Work item link check failed. Description does not contain AB#{ID}.

Click here to Learn more.

[github-actions]

✅ Work item link check complete. Description contains link AB#3557002 to an Azure Boards work item.

[github-actions]

❌ Invalid work item number: AB#3557002**

##. Work item number must be a valid integer.

Click here to learn more.

[github-actions]

✅ Work item link check complete. Description contains link AB#3557002 to an Azure Boards work item.