I’d like to discuss a common problem we often encounter when creating a product without an experienced team: the gap that arises when we rush to deliver quickly and meet market demands. The pressure to prioritize other demands leads many teams to leave observability aside, resulting in solutions that are difficult to maintain and manage. First, consider as an example questions such as:

How many requests are we receiving per minute?

How many errors are occurring?

What is the average CPU and memory usage of this application?

For Customer ‘X’, what is our average response time?

These questions should not be secondary; Answer them from the beginning of the application. They are essential and help steer the solution architecture in the right direction, both technically and commercially. Therefore, the inability to measure or understand what is happening with our software should always be a priority and treated as a primary need.

What does Observability involve?

Logs: Records of events or actions that occur within the system. Logs provide a detailed history of what happened, so they are crucial for diagnosing problems or understanding system behavior.

Metrics: Quantitative measures that provide insights into the state and performance of the system. This includes both business and technical metrics, such as counters (e.g., requests per second, products sold, platform hits) and meters (e.g., CPU and memory usage).

Traces: Tracing a request or transaction through various services and system components. Tracing helps you understand the flow of an operation and identify bottlenecks or failures in call chains. But with the help of OpenTelemetry, creating and accessing this information has become much easier.

OpenTelemetry

OpenTelemetry (OTel) is an open-source convention that allows development teams to generate, process, and transmit telemetry data in a unified format. Developed by the Cloud Native Computing Foundation (CNCF), it offers standardized protocols and tools for collecting and directing metrics, logs, and traces.

Suggestions and Examples:

Elastic Stack

Elastic Stack, as the name suggests, refers to the Elastic Cloud suite of products. Open-source tools can be used at no cost when deployed on-premises or self-managed. When using Elastic Cloud, a subscription fee is applied. Next, see what the main components are:

Elasticsearch: A distributed search and analytics engine based on Lucene. It is used to index and store data, allowing for real-time searches and analysis of large data sets.

Logstash: A server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch.

Kibana: A web-based user interface for visualizing Elasticsearch data. Kibana allows users to create and share dashboards that display real-time data visualizations.

Beats: A lightweight agent platform for sending data, where each “beat” is an agent designed to collect specific types of data from remote machines and send it to Elasticsearch or Logstash.

Grafana Stack

Similarly, Grafana Stack consists of open-source tools that can be used at no cost on-premise or with paid managed services. Key components include:

Grafana: A data analysis and visualization platform that allows users to create, explore, and share dashboards by visualizing data in real-time. Grafana is known for its ability to integrate with various data sources such as Prometheus, InfluxDB, Elasticsearch, among others.

Prometheus: An open-source monitoring and alerting system that collects and stores metrics such as time-series data, enabling queries and alerts based on specific queries.

Mimir: Formerly known as Thanos, Mimir is a highly scalable, multi-tenant, long-term storage solution for Prometheus metrics. It was renamed after Grafana Labs, the company behind Grafana, acquired the project.

Pyroscope: An open-source continuous profiling database that provides efficient storage and querying, helping to understand resource usage in applications down to the line-of-code level.

Loki: A horizontally scalable log aggregation system, inspired by Prometheus, designed for efficient storage and search. Loki focuses on providing a storage and querying solution for logs, integrating seamlessly with Grafana for visualization.

Time: A large-scale, distributed tracing system that stores and queries tracing data similar to Jaeger and Zipkin. Tempo is designed as a tracing backend that integrates with Grafana for trace visualization.

Graphite: A time-series database for storing real-time performance data. Grafana can serve as a visualization interface for data stored in Graphite.

InfluxDB: A time-series database optimized for high-availability storage and querying of time-series, event, and metric data.

Conclusion

Leaving our application or product unobservable is like being in a boat without an oar—we can’t influence the direction we’re heading; We can only hope that it is the right one. So, before you deprioritize observability in your product, think twice.

As they say, “If you can’t measure it, you can’t manage it.” Fun fact: it is usually said that this phrase is by Peter Drucker, but there are controversies; It seems he never said that.

Adriano Torini is a specialist in software development, with a postgraduate degree in Software Engineering. He is passionate about finding new solutions and solving problems. In his free time, he dedicates himself to physical activities, reading and continuous learning, always seeking to be 1% better every day.