Estafet consultants occasionally produce short, practical tech notes designed to help the broader software development community and colleagues.
If you would like to have a more detailed discussion about any of these areas and/or how Estafet can help your organisation and teams with best practice in improving your SDLC, you are very welcome to contact us at enquiries@estafet.com
Java Observability with Open Telemetry
In the realm of modern computer systems where complex applications frequently span multiple diverse infrastructures, Observability is a key principle of IT Operations allowing daily activities to be monitored and unexpected behaviours which can emerge from the interactions of numerous variables to be understood. By offering real-time insights into the internal states of systems, observability transforms the opaque computer systems into a comprehensible landscape.
Many commercial products such as Dynatrace, Data Dog and New Relic as well as Azure Cloud Monitor and AWS CloudWatch for specific cloud vendors are available, but there are open source alternatives, one of which is OpenTelemetry …
“a collection of APIs, SDKs, and tools. Use it to instrument, generate,
collect, and export telemetry data (metrics, logs, and traces) to help you
analyze your software’s performance and behavior.”
OpenTelemetry is a Cloud Native Computing Foundation project which was formed from the merger of the earlier OpenTracing and OpenCensus projects. It also forms part of version 6.0 of the MicroProfile specification – an open-source community specification for Enterprise Java microservices.
It includes an API for sending telemetry data to an observability backend and a set of libraries that developers can use to either automatically or manually instrument the code for HTTP requests, database calls, etc. Additional attributes can be included in the information transmitted to the backend using the OpenTelemetry SDK.
Although OpenTelemetry can send data directly to an observability backend, there is an OpenTelemetry Collector available. This is a separate application which receives, processes and exports telemetry data to an observability backend. Using the Collector is recommended as it allows data to be offloaded quickly and simply, reducing the overhead on the application. The collector can also take care of other overheads such as retries, batching, encryption and filtering of sensitive data.
The storage, presentation and monitoring of the telemetry data is the responsibility of the observability backend. OpenTelementry can export data to many different backends such as Grafana, ZipKin and Jaeger (which will be used in the example below). Most of the commercial products mentioned above also accept data from OpenTelemetry.
There are several methods for enabling observability in a Java application. It can be set up manually, for a Spring Boot application an OpenTelementry tracer for Spring Cloud Sleuth is available, however, the simplest approach is to use automatic instrumentation which uses a Java agent jar to dynamically inject bytecode to capture telemetry data. This also allows tracing to be set up without rebuilding an application. Simply set the following options when starting the application …
-javaagent:/app/opentelemetry/opentelemetry-javaagent-1.29.0.jar
-Dotel.service.name=blueprint
-Dotel.metrics.exporter=prometheus
… where /app/opentelementry is the folder containing the OpenTelementry jar file from OpenTelemetry Java Instrumentation Releases and the otel.service.name is a label which will be attached to the telemetry data allowing data for individual services to be more easily located.
To start the Jaeger observability backend from a Docker image, run the following command …
The Jaeger UI will then be accessible at http://localhost:16686/search …
The sample data below was generated using a demo ChatGPT application called ‘blueprint’ which has a GET endpoint that issues a request to ChatGPT for the prompt “Create an analogy for the phrase …”.
Here we can see the service selected matches the otel.service.name set in the application and a list of traces for the requests sent in the last hour. Drilling into the most recent request gives a breakdown of the time spent in each of the stages of the call …
Expanding further the final stage shows details of the request sent to ChatGPT inducing the HTTP method, content length, status code and the URL called …
Finally, to add a custom attribute to the data set to the backend such as an internal request id, first add the following dependency to the pom file …
… then include the code below …
This adds an attribute called customAttribute to the telemetry data …
Note: OpenTelemetry also generates monitoring data, however Jaeger is primarily a tool for capturing and analysing requests within a distributed system. For monitoring and alerting a backend tool such as Prometheus can be utilised.
Observability offers IT Operations a detailed, real-time window into the workings of complex computer systems allowing issues to be promptly detected and addressed, and trends to be identified. It also gives Development teams insights into where proactive enhancements would give the most benefit. As systems continue to grow in complexity, employing observability has become more and more of a necessity rather than a luxury.
By Jeremy Gosling, Principal Consultant at Estafet
