There are some useful pieces of information from a recent twitter thread on 'figuring out how systems are functioning'.
- "Three Pillars" implies some sort of definition. But "metrics, logs and traces" are a random grab bag! Two primitive data storage types (one of which should never be used anymore) and one visualization layer (for third type which isn't named!) -- Charity Majors
- I think the "Three Pillars of Observability" is an odd way to look a things. Pillars, by their nature are related only by what they hold up. Metrics nor tracing, should be pillars. All events should be traced and have metrics attached to them. -- Michael Wilde
- There are no three pillars of observability -- Charity Majors
- You want to know why observability is playing catch up? Because the EVENT is the execution path of your code through the system. It is the correct lens through which to understand your systems from the inside out. Without it, you are toast -- Charity Majors
- The event is the connective tissue for metrics, logs and tracing, because you can derive the first two from events and you can visualize the third from events. But it doesn't go in reverse: you can never work backwards from metrics, logs, and/or traces to get your event back -- Charity Majors
- Context is EVERYTHING in debugging, yet people have been happily discarding it all irretrievably at write time because they were so stuck in the mindset of metrics, logs and traces. This shit matters, man. How we frame it in our brain informs what we build going forward.-- Charity Majors
- A system is observable iff you can determine the behaviour of the system based on its outputs -- Greg Poirier
- Monitoring is the action of observing and checking the behaviior and outputs of a system and its components over time -- Greg Poirier
What hasn't been mentioned is 'high cardinality is king' and helps with context (if not, is the context).
2021/08/24 - Going off on a tangent: Observability: A New Theory Based on the Group of Invariance:
[Reviewed by Bill Satzer, on 10/31/2020 ]
Observability is a critical concept in control theory. Loosely speaking, the state of a system is observable if knowledge of its inputs and outputs during a given time interval permit determination of that state. For linear time-invariant systems, there is a simple criterion for observability: that the observability matrix (conceptually straightforward and easy to compute) should have full rank. For more general systems with all the inputs known there are similar criteria. But for nonlinear systems with unknown inputs, the problem is a good deal harder. This monograph offers a new approach to observability in nonlinear systems, one that was developed by the author.
The primary problem that the theory of observability studies is determining whether an input-output system provides the information necessary to estimate the state. Such information comes from sensors that provide measurements of system inputs and outputs. One way to formulate this problem quantitatively is to ask whether it is possible to determine the initial state of a system given system inputs and outputs over a given time interval.