OpenTelemetryのSignalsをOpenTelemetry Collectorを利用してGrafana Cloudに送る方法について

OpenTelemetry の Signals を Grafana Cloud に送る ymgyt · 2024‑10‑10

自己紹介 [speaker] name = "Yamaguchi Yuta" github = "ymgyt" [company] name = "FRAIM Inc."

概要 OpenTelemetry の Logs, Traces, Metrics を OpenTelemetry Collector を利用して Grafana Cloud に送る方法について

OpenTelemetry とは • CNCF Project • Components ‣ Data model ‣ Instrumentation (API, SDK) ‣ Procotol (OTLP) ‣ Collector ‣ Schema

OpenTelemetry とは • CNCF Project • Components ‣ Data model ‣ Instrumentation (API, SDK) ‣ Procotol (OTLP) ‣ Collector ‣ Schema 永続化、Query, Visualize についてはスコープ外

Vendor 中立性 と big tent philosophyと の親和性 > One of the biggest advantages of the OpenTelemetry project is its vendor neutrality. > Vendor neutrality also happens to be a core element of our big tent philosophy here at Grafana Labs. OpenTelemetry and vendor neutrality: how to build an observability strategy with maximum flexibility

• https://grafana.com/blog/2024/09/12/opentelemetry-and-vendor-neutrality-how-to-build-an-observability-strategy-with-maximum-flexibility/

OpenTelemetry Collector とは > The OpenTelemetry Collector offers a vendor‑ agnostic implementation of how to receive, process and export telemetry data. opentelemetry.io/docs/collector/

• https://opentelemetry.io/docs/collector/

Grafana Cloud の認証 OpenTelemetry Collector から Grafana Cloud に export するには以下の情報が必要 • OTLP Endpoint • Grafana Cloud instance ID • API Token

• OTLP Endpoint Grafana Cloud の認証 https://grafana.com/orgs/{org}/stacks/{stack}/ otlp‑info

• Grafana Cloud instance ID Grafana Cloud の認証 https://grafana.com/orgs/{org}/stacks/{stack}/ otlp‑info

• API Token Grafana Cloud の認証 https://grafana.com/orgs/{org}/stacks/{stack}/ otlp‑info

Grafana Cloud の認証 Collector の設 定 extensions: basicauth/grafanacloud: client_auth: username: ${env:GC_INSTANCE_ID} password: ${env:GC_API_KEY} receivers: otlp: protocols: { grpc: { endpoint: "127.0.0.1:4317"}} exporters: otlphttp/grafanacloud: auth: authenticator: basicauth/grafanacloud endpoint: ${env:GC_OTLP_ENDPOINT} service: extensions: [basicauth/grafanacloud] pipelines: traces: &pipeline receivers: [otlp] exporters: [otlphttp/grafanacloud] metrics: *pipeline logs: *pipeline

• label の"."は"_"に変換される • 特定の Resource は index label として扱われる ‣ deployment.environment, cloud.region, service.{namespace,name,id}, k8s. Logs の変換 {cluster.name, container.name, ...} ‣ 変更するには、support に連絡が必要 ‣ index label として扱われなかったものは strucuted metadata として扱われる OTLP: OpenTelemetry Protocol format considerations

• https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/
• https://grafana.com/docs/grafana-cloud/send-data/otlp/otlp-format-considerations/#logs

2024 年 3 月 25 日から 90 日前の間に otlp で log を export していた場合、grafana cloud 側の内部 で loki exporter による変換がなされているので、 該当ユーザは注意が必要(該当するかは support に問い合わせる) Logs の変換 service.namespace/service.name が job label に 変換される等 Why and How to Adopt the Native OTLP Log Format?

• https://grafana.com/docs/grafana-cloud/send-data/otlp/adopt-new-logs-format/

Metrics は Mimir(Prometheus compatible database)に保存される Metrics の変換 OpenTelemetry と Promehteus の metrics がど のように変換されるかは Prometheus and OpenMetrics Compatibility という仕様に定めら れている (Status は Development なので Breaking change はありえる)

• https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#otlp-metric-points-to-prometheus

• Metrics の名前や attributes(label) の"."や"-"は"_"に変換される http.response.status.code => Metrics の変換 http_response_status_code • unit と type が suffix に追加される graphql.duration => graphql_duration_seconds_bucket

promehteus の以下の label は otel の service resource から mapping される Metrics の変換 • job は service.namespace/service.name • instance は service.instance.id 仕様で service.name, service.namespace, service.instance.id の triplet がユニークである ことが求められている

• https://opentelemetry.io/docs/specs/semconv/resource/#service

Resource は target_info metrics に変換される Metrics の変換 target_info{ deployment_environment_name="production", instance="myhost", job="myservice/api", service_version="0.2.5"}

collector で resource を明示的に attribute に set しておくと metrics に resource の情報をのせる ことができる Metrics の変換 processor: transform: metric_statements: - context: metric statements: - set(attributes["namespace"], resource.attributes["k8s_namespace_name"]) - set(attributes["container"], resource.attributes["k8s.container.name"]) - set(attributes["pod"], resource.attributes["k8s.pod.name"]) - set(attributes["cluster"], resource.attributes["k8s.cluster.name"])

Prometheus の OpenTelemetry への取り組み • UTF‑8 metric and label name ‣ .を promehteus でも使えるようになる • Native support for resource attributes ‣ attribute と resource が promethues では flat に表現されている • Delta temporality How the Prometheus community is investing in OpenTelemetry

• https://grafana.com/blog/2024/04/03/how-the-prometheus-community-is-investing-in-opentelemetry/

Traces の変換 Tempo は OpenTelemetry の Attribute と Resource をサポートしている OpenTelemetry 上の Trace がそのまま Tempo の trace として表現されている ("."が"_"に変換さ れたりしない) Best practices for traces

• https://grafana.com/docs/tempo/latest/operations/best-practices/#best-practices-for-traces

まとめ • OTLP を利用して Grafana Cloud に logs, metrics, traces を簡単に export できる • OpenTelemetry のデータ構造がどのようにそ れぞれのサービスでマッピングされるかを意 識する必要がある • OpenTelemetry を利用した際の experience の 改善が日々進んでいる

参考 1 • OpenTelemetry mission, vision, and values • OTLP: The OpenTelemetry Protocol • OTLP: OpenTelemetry Procotol format considerations • OpenTelemetry and vendor neutrality • How the Prometheus community is investing in OpenTelemetry • Proper support for OTEL resource attributes

• https://opentelemetry.io/community/mission/
• https://grafana.com/docs/grafana-cloud/send-data/otlp/
• https://grafana.com/docs/grafana-cloud/send-data/otlp/otlp-format-considerations/
• https://grafana.com/blog/2024/09/12/opentelemetry-and-vendor-neutrality-how-to-build-an-observability-strategy-with-maximum-flexibility/
• https://grafana.com/blog/2024/04/03/how-the-prometheus-community-is-investing-in-opentelemetry/
• https://docs.google.com/document/d/1FgHxOzCQ1Rom-PjHXsgujK8x5Xx3GTiwyG__U3Gd9Tw/edit?pli=1#heading=h.unv3m5m27vuc

参考 2 • OTel to Prometheus Usage Issues • PromCon 2023 ‑ Towrds making Prometheus OpenTelemetry native • Ux of using target_info • Prometheus as an OTel native metrics backend • Prometheus data source update: Redefining our big tent philosophy • OpenTelemetry best practices: A user’s guide to getting started with OpenTelemetry

• https://docs.google.com/document/d/1epvoO_R7JhmHYsII-GJ6Yw99Ky91dKOqOtZGqX7Bk0g/edit#heading=h.5sybau7waq2q
• https://www.youtube.com/watch?v=mcabOH70FqU
• https://docs.google.com/document/d/1gG-eTQ4SxmfbGwkrblnUk97fWQA93umvXHEzQn2Nv7E/edit#heading=h.b79ugwqjcgse
• https://docs.google.com/document/d/1NGdKqcmDExynRXgC_u1CDtotz9IUdMrq2yyIq95hl70/edit
• https://grafana.com/blog/2024/08/06/prometheus-data-source-update-redefining-our-big-tent-philosophy/
• https://grafana.com/blog/2023/12/18/opentelemetry-best-practices-a-users-guide-to-getting-started-with-opentelemetry/

参考 3 • Prometheus and OpenMetrics Compatibility • Ingesting logs to Loki using OpenTelemetry Collector • Introduction to Ingesting OpenTelemetry Logs with Loki | Zero to Hero: Loki | Grafana • Why and How to Adopt the Native OTLP Log Format? • Best practices for traces

• https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/
• https://grafana.com/docs/loki/latest/send-data/otel/
• https://www.youtube.com/watch?v=snXhe1fDDa8
• https://grafana.com/docs/grafana-cloud/send-data/otlp/adopt-new-logs-format/
• https://grafana.com/docs/tempo/latest/operations/best-practices/#best-practices-for-traces