Early-stage, for-fun project. Expect things to be incomplete and to change.
betriebshandbuch-docs

Operating Manual Model ​

Author Comment

Some of the aspects of the operating manual only hold metadata, some describe the current state, and some prescripe the desired state.

Author Comment

This page needs a feedback loop with developers, managers, and operators from the industry.

This page describes, abstractly, what our operating manual model captures. It does not cover how the model is actually structured; that comes later in the implementation. Some of what we describe here need not be modeled explicitly at all, as it can be discovered from the Kubernetes resources. The model itself may also be composed of several other, potentially reusable, models.

Each section is tagged to show which facet of operating the app it covers:

  • context information to understand the app
  • guarantees the promises it makes and what it needs to run well
  • observability how the app reports its state
  • operations what to do, day-2 and during incidents
  • runtime its live, reconciled state

Application ​

context

The metadata of the application, such as its name, description, version, and license (type and expiry).

Business Criticality ​

context

The importance of the application is to the business, from the business point of view. On-call reads it alongside the service levels to judge how much a problem matters.

System Architecture ​

context

A high-level component-based view of the application to capture the relationships between applications.

Use Cases ​

context

How the application is used, activity by activity. A use case usually spans several applications. Each use case also describes its workload against the app and its dependencies: the type (e.g. periodic) and the intensity. This gives a left-to-right view of the system, complementing the top-down system architecture.

Contacts ​

context

Technical, business, budget-owner, and on-call contacts, with name, email, phone, and more.

Service Levels ​

guarantees

What the application promises and how well it must run: service level agreements and objectives (SLA/SLO) such as an availability target, and service hours (Servicezeiten) that state when it must be available.

Scalability ​

guarantees

How the application can scale and be updated. Can it scale vertically, by giving it more resources? Is it stateless, so it can scale horizontally by running more replicas? And during a rolling update, must the single pod be recreated (causing downtime), or can a new pod come up alongside the old one so multiple run at once?

Resources ​

guarantees

What the application needs to run: resource requests and limits. It also captures expected growth, so capacity planning can look ahead.

Data ​

context

What the application stores and how to handle it:

  • criticality: in the worst case, can the data simply be recreated, or must it be backed up?
  • backups: how the data is backed up and how well the backups are protected (e.g. following the 3-2-1 rule).
  • integrity: the conditions under which data integrity is compromised, e.g. when a dependency is unhealthy or too many pods are stopped at once.
  • confidentiality: how sensitive the data is and whether it holds personal data (PII).

Health ​

observability

The startup, liveness, and readiness probes of the application.

Metrics ​

observability

The metrics that can be scraped along with scrape jobs.

Dashboards ​

observability

The dashboards to visualize the metrics.

Alerts ​

observability

Alerts on the app's own metrics and on the metrics of other apps in the system architecture.

Management Operations ​

operations

Day-2 operations such as restart, stop, scale, backup, restore, and custom logic (e.g. disabling a feature while a dependency is unhealthy). Following the idea we take from TOSCA, an operation abstracts away the underlying technology. Hence, the implementation of the operation may differ widely, e.g., a Shell script, a Python programm, or any Kubernetes resource. Operations can declare preconditions and are triggered by events:

  • Manual: run on demand.
  • Scheduled: recurring or at a specific timestamp.
  • Alerts: e.g. restart on a firing alert.
  • System events: e.g. a dependency becomes unhealthy.

Failure Catalog ​

operations

Known failure modes described as symptom, likely cause, and remediation. A remediation links to a natural-language instruction and/or a management operation, so the catalog ties an alert to the thing that fixes it. In the best case the remediation is automated, but in reality there is always some human handling to describe. Disaster recovery and failover live here as example entries.

Escalation ​

operations

Escalation chains describe who to reach and in what order, e.g., on-call engineers, devs, end users, and how, e.g., wake-up call, email, and so on.

Maintenance ​

operations

Maintenance windows and who to notify. Dependent apps can model, on their side, which management operations should run when this app enters maintenance, for example, stopping themselves.

context

Pointers to the resources around the application: code repository, docs, logs, dashboards, and ticketing system (so we can open a ticket to report a bug).

Status ​

runtime

The live state of the operating manual as a Kubernetes resource:

  • Resources: all Kubernetes resources of the app, discovered via a tracking annotation, owner references, and labels.
  • State: an aggregated view of the resources' health that represents the app's health, plus flags such as whether the app is in maintenance. Following the Gateway API, we use distinct states and conditions so it is clear why an app is (un)healthy.