Early-stage, for-fun project. Expect things to be incomplete and to change.
betriebshandbuch-docs

Introduction ​

Author Comment

Early-stage, for-fun project. Expect things to be incomplete and to change.

Domain ​

Modern software runs as many applications deployed and operated on top of platforms such as Kubernetes. On-call operators are responsible for running applications, even outside of business hours. When something breaks, the on-call engineer has to handle an incident for an application whose internals they may not know and whose developers are not available. Therefore, the engineer gathers the required knowledge in the form of an operating manual, usually a PDF or a documentation page. This manual contains, e.g., the application's architecture, use cases, instructions for actions, and contacts.

Problems ​

However, there are several problems with such operating manuals. Operating manuals ...

  • must exist
  • must be up to date
  • must be quickly available
  • must not be ambiguous (natural language can be interpreted differently)
  • differ in their form, e.g., just a README in a repo or a wiki page
  • differ in the contained knowledge, e.g., missing architecture descriptions
  • must be tailored to a specific business context, e.g., SLAs
  • are not executable
  • and thier implementation drift over time

Further,

  • interfaces to start management operations differ widely per application
  • operators must identify situations and execute the correct management operations

State of the Art ​

The state of the art offers no normalized, executable format for the operating manual itself. It stays an ad-hoc document, such as a README, a wiki page, or a PDF, written in natural language and disconnected from what actually runs. As a result, manuals differ from application to application in both their form and the knowledge they contain, they are ambiguous and open to interpretation, and they cannot be executed.

The most relevant practice in this space is DevOps engineering, which narrows the gap between developers and operators, and which we build on by authoring the manual as code. However, DevOps targets the resources and configuration that run an application, not the operating manual, so it neither standardizes the operational knowledge. It also cannot supply the business context, because that depends on how the application is used within a specific company. This is especially clear for an open-source application, whose developers do not tailor the documentation to one specific company.

Solution ​

To tackle the mentioned problems and improve the role of the on-call operator, we propose the model-driven approach Betriebshandbuch (bhb), which is the German word for operating manual. The core of the approach is the Operating Manual Model to formalize, release, and execute the operating manual along with the applications. The model contains different aspects, e.g., a generic description of the application, its business criticality, and how the application is used.

The approach also gives you an interactive operating manual. Because we model the system architecture, you get a top-down view of the whole system and how its parts depend on each other. Because we also model use cases, you get a left-to-right view of how the system is actually used.

A primary design principle is not to reinvent the wheel. Instead, we rely on linking already existing technologies and their configurations with our model. With that, our Operating Manual Model is also a normalized view of operationally relevant aspects.

Implementation ​

We implement the solution for Kubernetes as an operator. The Operating Manual Model consists of (i) various Custom Resource Definitions and (ii) of references to existing Resource Definitions of Kubernetes itself but also of any operator. Different plugins enable the semantic understanding of such resource definitions. With that, our implementation is not limited to specific CRDs but extensible. Further, the Operating Manual Model is part of the Helm Chart of the application. A plugin for kubectl enables interacting with the model to operate the application.

Outlook ​

Going forward, we want to evaluate the approach together with the industry, through interviews, case studies, and user studies.

We also want to utilize the model as follows. First, we may check that a resource matches the model, e.g. that critical data has a backup that follows the 3-2-1 rule. Also, we may generate a resource, e.g. metric scraping jobs or alerts. Last, we may enrich existing resources with missing configurations, e.g., an alert so it reaches the right people.