Operate the installation as an owned system#
Operational readiness includes the target topology, accepted release, service allocations, monitoring, credential custody and recovery procedures. A successful build or container health response proves only part of that system. Record who responds to failures, which recovery objectives apply and how the observed installation matches its reviewed configuration.
Infrastructure owns shared service lifecycle, certificate and ingress operations. Application owners retain migrations, business authorization, message semantics and application-specific recovery. Coordinate work across these boundaries without giving application runtime identities infrastructure administration rights.
Separate diagnostics, receipts and business audit#
| Evidence | What it establishes |
|---|---|
| Health and diagnostic telemetry | Observed service state and clues for investigation |
| Deployment/setup receipt | The recorded operation, selected artifacts and verified steps |
| Messages acceptance receipt | Durable acceptance of a specific immutable publication |
| Application outcome receipt | The owning application's committed operation result |
| Business audit record | Application-defined actor/action/outcome evidence under its retention policy |
A deployment receipt is not a business audit trail, and message acceptance is not proof that a human completed work. Define the durable business-audit destination, access and retention separately. Keep secrets, protected payloads, preference values and raw cache keys out of general diagnostic telemetry. Correlate through bounded identifiers and retain the matching release/schema context needed for investigation.
Monitoring and certificate lifecycle#
Assign an alert owner, notification destination, private collection path and thresholds. Monitor authenticated readiness, authentication failures, certificate expiry, resource pressure, storage headroom and backup age. Include database connections and locks, broker queue depth/unacknowledged delivery/dead letters, and cache eviction/persistence behavior where those services are used.
The v1 fresh-installer profile documents short-lived data-service certificates and an explicit due-only renewal operation. It does not install an unattended scheduler automatically. Scheduler ownership, alert delivery, reconnect behavior and maintenance windows need acceptance. Managed v2 installations have a different composition and documented maintenance gates; use the supported path for the actual installation.
Keep private administration and telemetry endpoints on approved private paths. Frontend SSO does not automatically authenticate telemetry ingestion. Verify both sides if an optional operations dashboard is deployed.
Back up state that can be restored#
Backups use an approved encrypted off-host repository with protected authentication. Define retention, backup interval, recovery point objective and recovery time objective. A same-host copy is a rehearsal, not independent custody. A backup receipt is not proof of a successful restore.
| Service | Recovery material and limit |
|---|---|
| PostgreSQL | Matching dump/migration/grant material; the documented v1 logical-dump path does not provide WAL/PITR |
| RabbitMQ | Definitions plus the complete message store and stable node identity; definitions alone omit queued messages |
| Redis | Complete AOF directory and RDB/ACL material under the documented persistence mode; cache recovery must not replace authoritative business storage |
| Identity and certificate foundation | Database/configuration, protected CA/ACME state, keys and the required custody procedure; realm export alone is incomplete |
Restore into a separate isolated destination using compatible tools, images and migrations. Verify content, runtime permissions and denial paths before changing live connections. For the broker, test confirmed persistent messages, acknowledgements, retries and dead-letter behavior. Measure recovery time and the actual recoverable point rather than assuming configured values were achieved.
Change and failure procedure#
- Stop repeating uncertain writes and identify the exact owning component.
- Preserve volumes, journals and sanitized image/mount/configuration observations.
- Check certificate validity, authentication, capacity, storage and backup freshness.
- Select a reviewed recovery or forward-repair action and verify it in isolation where required.
- Switch the affected consumer only after authenticated readiness and business recovery conditions have passed.
Credential rotation is a separate scoped change: create a replacement identity with exact rights, validate it, switch the consumer, drain old connections and revoke the old identity in the agreed window. Re-running initialization with different passwords is not rotation.
Image updates require compatibility review and a restoration/upgrade rehearsal. A new database major version may require new storage and a supported migration; an old image may not read the new persisted format. Retain previous artifacts but use rollback only when state compatibility allows it. Single-node services and multiple containers on one host do not establish high availability across independent failure domains.