Skip to content

FUND-2451 ZGW Audittrail documentation #123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
heuvea wants to merge 1 commit into
base: main
Choose a base branch
from
Open

FUND-2451 ZGW Audittrail documentation #123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
209 changes: 209 additions & 0 deletions docs/audittrail/audittrail.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
---
id: audittrail
title: AuditTrail Service
---
Comment on lines +1 to +4
Copy link
Copy Markdown
Member

@Grabauskas Grabauskas May 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing more metadata. Look into other pages.


# AuditTrail Service

The AuditTrail system provides a comprehensive mechanism for tracking all CRUD operations performed on ZGW (Zaakgericht Werken) resources. It captures who did what, when, and records the before/after state of affected entities.


## Two Storage Strategies

### 1. Legacy — Full Snapshot (`AuditTrailService`)

Each audit entry stores the complete old (`Oud`) and new (`Nieuw`) JSON representation of the entity. Simple but storage-intensive.

**Database table:** `audittrail` (entity: `AuditTrailRegel`)

| Column | Description |
|--------------------|--------------------------------------------------|
| `oud` | Full JSON of the entity **before** the operation |
| `nieuw` | Full JSON of the entity **after** the operation |
| `hoofdobject` | URL of the main object |
| `hoofdobject_id` | GUID of the main object |
| `actie` | Action type (create, update, etc.) |
| `resultaat` | HTTP status code of the operation |
| `request_id` | Correlation ID of the HTTP request |

### 2. Delta-Based — Compressed (`DeltaBasedAuditTrail`)

Stores only the differences (deltas) between versions, with periodic full snapshots for efficient reconstruction. Significantly reduces storage for frequently updated entities.

**Database table:** `audittrail_deltas` (entity: `AuditTrailDelta`)

| Column | Description |
|--------------------|-----------------------------------------------------------|
| `delta_json` | JSON delta — only the changed properties |
| `snapshot_json` | Full JSON snapshot (stored periodically or on create) |
| `versie` | Sequential version number per resource |
| `resource_id` | GUID identifying the specific sub-resource |

#### Snapshot Strategy

- **Version 1** (`create`): Always stores a full snapshot.
- **Every N versions** (default: 25): A full snapshot is stored instead of a delta, configurable via `SnapshotInterval`.
- **Forced snapshots**: When `ForceSnapshotVersionWhenResourceChanged` is enabled, a snapshot is forced.
- **Updates with no changes**: Silently skipped — no audit entry is created.

Note:
The `ForceSnapshotVersionWhenResourceChanged` option is enabled for hoofdobject Zaak only. This ensures that during the next update of hoofdobject Zaak, the full snapshot is saved instead of a delta, which would cause the sub-resources to be included in hoofdobject Zaak.

#### Delta Format

Deltas are generated by `AuditDeltaGenerator` and use JSON diff with special markers:

| Marker | Meaning |
|--------------|------------------------------------------------------------------|
| `__removed` | Property existed in old version but was removed |
| `__replace` | Entire value should replace existing (not merge), used for complex properties listed in `PropertiesUsingCurrentValue` |

**Array deltas** use an object with `added`, `removed`, and `updated` keys:
```json
{
"myArray": {
"added": [{ "Id": "...", "name": "new item" }],
"removed": [{ "Id": "...", "name": "old item" }],
"updated": [{ "Id": "...", "changedField": "new value" }]
}
}
```

Array elements are matched by `Id` property if present, otherwise by full JSON equality.

#### State Reconstruction

When reading audit trail entries, the `DeltaBasedAuditTrail` reconstructs the full old/new state for each entry by:

1. Loading all entries for a resource, ordered by version.
2. Starting from the initial snapshot (or the nearest periodic snapshot).
3. Sequentially applying each delta to rebuild the entity state at each version.
4. Returning `AuditTrailRegel` objects with populated `Oud`/`Nieuw` fields — making the output identical to the legacy format.

### Enums

| Enum | Values |
|--------------|-------------------------------------------------------------------|
| `AuditActie` | `create`, `destroy`, `update`, `partial_update`, `retrieve` |

### Configuration

`AuditTrailOptions` carries per-request configuration:

| Property | Type | Description |
|--------------|----------------------------|--------------------------------------------------------|
| `Bron` | `string` | Source system identifier (default: `"n/a"`) |
| `Resource` | `string` | Resource type name (default: `"n/a"`) |
| `Properties` | `Dictionary<string, object>` | Extensible property bag for implementation-specific settings |

**Supported properties in `Properties`:**

| Key | Type | Used By | Description |
|----------------------------------------|-----------------|------------------------|-------------|
| `MaskFields` | `List<string>` | All implementations | Property names to mask with `"******"` in audit output |
| `PropertiesUsingCurrentValue` | `List<string>` | `DeltaBasedAuditTrail` | Properties where the full current value is stored instead of a delta (wrapped in `__replace`) |
| `ForceSnapshotVersionWhenResourceChanged` | `bool` | `DeltaBasedAuditTrail` | Force a full snapshot when the latest entry belongs to a different resource |
| `SnapshotInterval` | `int` | `DeltaBasedAuditTrail` | Number of versions between periodic snapshots (default: 25) |

Note:
The option `PropertiesUsingCurrentValue` is configured on the field zaakgeometrie of hoofdobject Zaak. This is a complex Geometry type whose deltas we do not want to store.

## Tracked Actions

Each API operation maps to an audit action with a Dutch display description:

| Method | `AuditActie` | `ActieWeergave` | HTTP Status |
|--------------------|------------------|--------------------------------------------------------|-------------|
| `CreatedAsync` | `create` | `"Object aangemaakt"` | 201 |
| `UpdatedAsync` | `update` | `"Object bijgewerkt"` | 200 |
| `PatchedAsync` | `partial_update` | `"Object bijgewerkt"` | 200 |
| `DestroyedAsync` | `destroy` | `"Object verwijderd"` | 204 |
| `GetAsync` | `retrieve` | `"Object gelezen"` (or custom) | 200 |
| `GetListAsync` | `retrieve` | `"Lijst van objecten gelezen"` (with filter/page info) | 200 |

## Captured Context

Each audit entry automatically captures the following from the HTTP context:

| Field | Source |
|----------------------|---------------------------------|
| `ApplicatieId` | Client ID from the request |
| `ApplicatieWeergave` | Application label |
| `GebruikersId` | User ID from authentication |
| `GebruikersWeergave` | User display representation |
| `RequestId` | HTTP request correlation ID |
| `AanmaakDatum` | UTC timestamp of the operation |

## Field Masking

Sensitive fields can be masked by configuring `MaskFields` in `AuditTrailOptions.Properties`. Masking:

- Replaces string property values with `"******"`.
- Recurses into nested objects and collections.
- Is applied to both old and new DTO snapshots before serialization.

Note: `MaskFields` is enabled for field inpBsn in table zaakrollen_natuurlijk_personen which contains sensitive content.

## Usage Pattern

The delta-based audit trail was recently added to the ZGW. The reason for this is that the legacy audit trail does not store the data efficiently.

How is the choice now determined as to which audit trail legacy or delta-based to use?

Existing hoofdobjecten such as Zaak, Besluit, and EnkelvoudigInformationObject (including underlying resource objects) continue to use the legacy audit trail because the old history is stored therein. If an audit trail is requested, it therefore contains the entire history of all mutations of the hoofdobject and underlying resources.

Newly created hoofdobjecten Zaak, Besluit, and EnkelvoudigInformationObject (including underlying resource objects) are written to the delta-based audit trail.



## Migration (Legacy → Delta) / not applied yet

`AuditTrailMigrator` converts existing legacy audit trail entries to the delta-based format:

1. Reads all `AuditTrailRegel` entries for a given `hoofdObjectId`.
2. For each entry, uses `DeltaBasedAuditTrailWithImporter.ImportAsync` to re-process the old/new JSON into delta/snapshot format.
3. Imported entries are prefixed with `"Migrator. "` in the `Toelichting` field.

`AuditTrailExporter` (DEBUG-only) can export both legacy and migrated entries to JSON files for comparison/validation.

Note:
One option is not to apply the migration at all. At some point, the old legacy can be completely cleaned up after archiving.

## Database Schema

### Table: `audittrail`

| Column | Type | Constraints |
|---------------------|---------------|--------------------|
| `id` | `uuid` | PK |
| `bron` | `varchar(50)` | NOT NULL |
| `applicatie_id` | `varchar(100)`| NOT NULL |
| `applicatieweergave`| `varchar(200)`| NOT NULL |
| `gebruikers_id` | `varchar(255)`| NOT NULL |
| `gebruikersweergave`| `varchar(255)`| NOT NULL |
| `actie` | `varchar(50)` | NOT NULL |
| `actieweergave` | `varchar(200)`| NOT NULL |
| `resultaat` | `int` | |
| `hoofdobject` | `varchar(1000)`| NOT NULL |
| `resource` | `varchar(50)` | NOT NULL |
| `resourceurl` | `varchar(1000)`| NOT NULL |
| `toelichting` | `text` | NOT NULL |
| `resourceweergave` | `varchar(200)`| NOT NULL |
| `aanmaakdatum` | `timestamp` | |
| `oud` | `jsonb` | |
| `nieuw` | `jsonb` | |
| `request_id` | `varchar(255)`| |
| `hoofdobject_id` | `uuid` | |

### Table: `audittrail_deltas`

Same columns as `audittrail` plus:

| Column | Type | Description |
|---------------------|---------------|--------------------------------|
| `delta_json` | `jsonb` | JSON delta (changes only) |
| `snapshot_json` | `jsonb` | Full JSON snapshot |
| `versie` | `int` | Sequential version number |
| `resource_id` | `uuid` | Identifies the sub-resource |

> **Note:** `audittrail_deltas` does **not** have the `oud`/`nieuw` columns. The old/new state is reconstructed at read time from snapshots and deltas.
5 changes: 5 additions & 0 deletions sidebars.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,11 @@ const sidebars: SidebarsConfig = {
label: "Use of BSN"
}
]
},
{
type: "doc",
id: "audittrail/audittrail",
label: "AuditTrail"
Copy link
Copy Markdown
Member

@Grabauskas Grabauskas May 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put it under General category.

}
]
};
Expand Down
Loading