Template preview (excerpt)
Scroll to see all columns
Section 4: Performance Metrics
4.1 Technical Performance Metrics
| Metric | Definition | Threshold | Alert Level |
|---|---|---|---|
| [Accuracy] | [% correct predictions] | [>95%] | [Critical if <90%] |
4.2 Drift Metrics
| Metric | Calculation Method | Threshold | Check Frequency |
|---|---|---|---|
| [Feature drift] | [PSI or KL Divergence] | [PSI <0.1] | [Daily] |
Section 5: Alerting and Escalation
5.1 Alert Severity Levels
| Level | Criteria | Response Time |
|---|---|---|
| Critical | [Immediate risk, compliance violation] | [15 minutes] |
| High | [Significant degradation] | [4 hours] |
View the worked example
Alert Severity Levels:
| Level | Criteria | Response Time |
|---|---|---|
| Critical | System down, accuracy <90%, discriminatory outcome | 15 minutes |
| High | Accuracy <95%, drift threshold exceeded | 4 hours |
| Medium | Approaching threshold, unusual pattern | 24 hours |
| Low | Minor metric movement, informational | Next business day |
Before you begin
For compliance, risk, product, and ML ops teams shipping agentic Processes into regulated environments.
The EU AI Act Article 72 requires providers of high-risk AI systems to establish and document a post-market monitoring system. This is not optional. It is a compliance requirement with specific mandates for how you monitor AI systems after deployment.
This template provides a structured 8-section approach covering system identification, monitoring objectives, data collection, performance metrics, alerting, review procedures, incident response, and documentation.
When to use this resource
- You are deploying an agent into a regulated Process (credit, claims, KYC/AML, HR).
- You need to prove ongoing quality, safety, and policy compliance after go-live.
- You are preparing an Annex IV dossier or an audit readiness review.
Information to gather
- System identification with regulatory classification and Annex IV references.
- Monitoring objectives connected to risk register with success criteria.
- Data sources, collection frequency, quality requirements, and privacy considerations.
- Performance thresholds (technical, drift, fairness) with severity levels.
- Alerting configuration with escalation matrix and after-hours coverage.
- Review procedures (continuous, sampling, periodic) with governance integration.
- Incident response procedures including Article 73 regulatory reporting.
Review checklist
Use these checks in the review with your system owner. Confirm the applicable requirements and attach evidence for the decisions your team makes.
- System identification links to regulatory classification and Annex IV documentation.
- Monitoring objectives connect to identified risks with clear success criteria.
- Data collection covers all sources with quality requirements and privacy considerations.
- Performance metrics include technical, drift, and fairness thresholds with severity levels.
- Alerting defines severity levels, notification matrix, and escalation procedures.
- Review procedures include continuous monitoring, sampling, and periodic governance reviews.
- Incident response covers definition, immediate response, investigation, and Article 73 reporting.
- Documentation and evidence storage ensures tamper-evident retention with audit readiness.
Operating controls and evidence
- Govern
Policy-as-code checkpoints that block or require review for high-risk actions.
Versioned change control for model/prompt/policy/Process updates.
- Assure
Risk-tiered sampling reviews (baseline + burst during incidents or after changes).
Near-miss tracking (blocked / nearly blocked steps) as a measurable control signal.
- Prove
Configurable retention schedules, integrity verification, and an append-only Audit Trail.
Evidence Room export bundles (manifest + checksums) so auditors can verify independently.
Questions about this resource
What is "post-market monitoring" in plain language?
It is how you prove the system stays safe and fit-for-purpose after go-live: what you measure, how you review samples, and how you respond to incidents and changes.
What does Article 72 require?
Article 72 requires active data collection on system performance, identification of needs for corrective action, and documentation of monitoring activities. This template covers all three requirements.
What is the difference between alerting and incident response?
Alerting detects issues and notifies the right people. Incident response defines what happens after detection: assessment, containment, investigation, resolution, and documentation.
What triggers Article 73 regulatory reporting?
Serious incidents involving high-risk AI systems must be reported to market surveillance authorities immediately upon becoming aware. The template includes a regulatory reporting section.
How do we ensure evidence integrity?
Use tamper-evident storage (append-only ledger with cryptographic integrity), complete capture at decision points, access logging, and regular integrity audits.
What do auditors reject most often?
Plans that are generic. Auditors want named owners, specific thresholds, clear escalation procedures, and evidence that can be exported and verified independently.
Download file language: English