4.8. Integration with Tracing Tools #
PPEM supports distributed tracing for requests. Tracing data is exported from PPEM using OTLP (OpenTelemetry protocol) and sent to a receiver that supports OTLP, such as pgpro-otel-collector, OpenTelemetry Collector, Grafana Tempo, or Jaeger.
Note
Distributed tracing applies to incoming requests to the manager and agents, as well as to their databases. It also applies to requests required for internal manager and agent operations.
However, distributed tracing does not apply to requests to other services.
4.8.1. Tracing Architecture #
Below is an example of a distributed tracing architecture.
Figure 4.1. Distributed tracing architecture
The user interacts with PPEM through the web application. To perform the required user actions, the web application sends requests to the manager.
Upon receiving a request from the web-application or API, the manager creates a root span, generates the trace context consisting of the trace and span ID, and saves the span start timestamp. The trace context is then associated with the request and used throughout the entire time this request is processed. After processing is finished, the span end timestamp is saved.
For each request to the repository or agents, the manager creates a child span. The child span has a unique ID and start/end timestamp, but it reuses the trace ID of the root span. The child span also stores the parent span ID. Once the child span starts or ends, its trace context is updated.
With each request to an agent, the manager passes the trace context. Similarly to the manager, upon receiving a request, the agent creates a child span reusing the received trace ID. The agent also creates child spans when sending requests to databases.
The manager and agents periodically send tracing data to the receiver, which saves it to the storage. Tracing data with the same trace ID represents a single trace that you can view using Jaeger or Grafana.
4.8.2. Integration with Grafana Tempo #
This section explains how to provide tracing by integrating PPEM with Grafana Tempo.
Important
Grafana Tempo is used as an example to demonstrate the integration process. You can select any other supported trace storage based on your organization's requirements.
Within this integration, PPEM sends tracing data using OTLP via HTTP to Grafana Tempo for storing and processing. If required, tracing data is then sent to the Grafana visualization system to be represented in the form of graphs.
Before performing this instruction, install and set up Grafana and Grafana Tempo. For more information, refer to the official Grafana and Grafana Tempo documentation.
To integrate PPEM with Grafana Tempo:
Configure Grafana Tempo to receive tracing data.
In the Grafana Tempo configuration file, specify:
distributor: receivers: otlp: protocols: http: endpoint: "receiver_endpoint_URL"Where
endpointis the endpoint URL of the tracing data receiver, for example,0.0.0.0:4318.Configure the manager and agents to export tracing data.
In the
ppem-manager.ymlmanager andppem-agent.ymlagent configuration files, specify:otlp: traces: exporter: protocol: "http" endpoint_url: "exporter_endpoint_URL"Where
endpoint_urlis the endpoint URL of the tracing data exporter, for example,http://tempo.example.org:4318/v1/traces.Configure Grafana to work with Grafana Tempo:
Go to Home → Connections → Data sources.
Click Add new data source.
From Data source type, select Tempo.
Enter the data source name, for example,
tempo-1.In Connection URL, enter the endpoint URL of the tracing data exporter, for example,
http://tempo.example.org.(Optional) To view the list of traces:
Go to Home → Explore.
Select the previously created data source.