Skip to content

Conversation

@goakley
Copy link
Contributor

@goakley goakley commented Dec 4, 2025

Description

Datadog's metric model treats tags on data points as arbitrary strings. However, the product has special support for tags formatted as a key:value string. key:value tags allow slicing metrics in powerful ways and are core to many of Datadog's offerings.

Since Datadog tags are arbitrary strings, there's nothing to stop users from recording tags with the same key: prefix on the same data point. In fact, Datadog's own integrations can do this: The kube_service tag from the kubelet integration will have one entry per Kubernetes service that fronts a pod. (An example set of tags might look something like [...,"kube_service:datadog-cluster-agent-metrics-api","kube_service:datadog-cluster-agent-admission-controller",...]).

Currently, when the datadogreceiver receives multiple tags for a data point that share the same key, it only retains the last key:value pair as an OpenTelemetry attribe. In the above example, the attribute map would have an entry kube_service = datadog-cluster-agent-admission-controller - "datadog-cluster-agent-metrics-api" would have been discarded.

This change introduces a feature gate that changes the tag parsing behaviour to support tags that share the same key. Instead of overwriting the previous Str entry in the attribute map, it converts the attribute map to a Slice and stores all values for that key. This conversion only happens if more than one value is encountered for the same key, so even with the feature flag enabled, there will not be any behaviour change for users who do not send tags that share the same key.

I chose to implement this as a feature flag instead of a config setting because I believe this is technically correct behaviour, and should be the default in some future release of the OpenTelemetry Collector. Feedback on this decision is appreciated.

Testing

Unit tests have been added against tagsToAttributes to make sure attribute conversion works as expected both with and without the feature gate. The collector has been built and run locally, confirming that exporters receive the expected (slice) attributes when multiple tags with the same key are received.

Documentation

The README for the receiver now describes how tags are processed, both with and without the feature gate.

Datadog's metric model treats tags on data points as arbitrary strings.
However, the product has special support for tags formatted as a `key:value` string.
`key:value` tags allow slicing metrics in powerful ways and are core to many of Datadog's offerings.

Since Datadog tags are arbitrary strings, there's nothing to stop users from recording tags with the same `key:` prefix on the same data point.
In fact, Datadog's own integrations can do this: The `kube_service` tag from [the `kubelet` integration](https://docs.datadoghq.com/containers/kubernetes/tag/) will have one entry per Kubernetes service that fronts a pod.
(An example set of tags might look something like `[...,"kube_service:datadog-cluster-agent-metrics-api","kube_service:datadog-cluster-agent-admission-controller",...]`).

Currently, when the `datadogreceiver` receives multiple tags for a data point that share the same key, it only retains the last `key:value` pair as an OpenTelemetry attribe.
In the above example, the attribute map would have an entry `kube_service = datadog-cluster-agent-admission-controller` - "datadog-cluster-agent-metrics-api" would have been discarded.

This change introduces a feature gate that changes the tag parsing behaviour to support tags that share the same key.
Instead of overwriting the previous `Str` entry in the attribute map, it converts the attribute map to a `Slice` and stores all values for that key.
This conversion only happens if more than one value is encountered for the same key, so even with the feature flag enabled, there will not be any behaviour change for users who do not send tags that share the same key.

I chose to implement this as a feature flag instead of a config setting because I believe this is technically correct behaviour, and should be the default in some future release of the OpenTelemetry Collector.
Feedback on this decision is appreciated.
@MovieStoreGuy
Copy link
Contributor

This makes sense to me, and I don't see any issue with this approach.

Thank you for putting it behind a feature gate to begin with :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants