Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus scrape endpoint #110

Merged
merged 6 commits into from
May 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions docs/agent-vs-direct/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ The eBPF auto-instrumenter can work in two operation modes:
to Grafana. In this scenario, the Agent takes care of the authentication in the Grafana endpoint.
It also integrates better with some Grafana exclusive features, such as the span-to-metrics and
span-to-service graph converters (**TODO**: add links).
* **Direct mode**: the auto-instrumenter will send the metrics and traces directly to a remote endpoint
(currently only supporting the OpenTelemetry protocol). In this mode, the auto-instrumenter needs to be configured
with the authentication data.
* **Direct mode**: the auto-instrumenter can **push** the metrics and/or traces directly to a remote endpoint
(using the OpenTelemetry, OTEL, protocol) or just expose a Prometheus HTTP endpoint ready to be scraped (**pull** mode).
In direct OTEL push mode, the auto-instrumenter needs to be configured with the authentication data.

```mermaid
flowchart TD
EA1(Auto-instrumenter) --> |OTLP metrics & traces| GA1(Grafana Agent)
EA1(Auto-instrumenter) --> |Push OTLP metrics & traces| GA1(Grafana Agent)

subgraph Grafana
M(Mimir)
Expand All @@ -28,20 +28,25 @@ flowchart TD
GA1 --> |Metrics| M
GA1 --> |Traces| T

EA2(Auto-instrumenter) --> |OTLP metrics & traces| OTLP
EA2(Auto-instrumenter) --> |Push OTLP metrics & traces| OTLP

P(Prometheus) --> |Pull /metrics HTTP endpoint| EA2
```
<center><i>eBPF auto-instrumenters running in Agent mode (left) and Direct mode (right)</i></center>

## Running in Direct mode

You can follow our [quickstart tutorial](../tutorial/README.md) for a quick introduction
to the eBPF auto-instrumenter running in Direct mode. You will find there how to provide
to the eBPF auto-instrumenter running in Direct mode using OpenTelemetry. You will find there how to provide
an OTLP endpoint and authentication credentials by means of the following environment
variables:

* `OTEL_EXPORTER_OTLP_ENDPOINT`
* `OTEL_EXPORTER_OTLP_HEADERS`

To run in direct mode using the Prometheus scrape endpoint, please refer to the
[configuration documentation](../config.md).

## Running in Agent mode

> ℹ️ This tutorial assumes that both the Agent and the Auto-instrumenter are installed
Expand Down
59 changes: 56 additions & 3 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,11 @@ flowchart TD

ROUT --> OTELM(OTEL<br/> metrics<br/> exporter)
ROUT --> OTELT(OTEL<br/> traces<br/> exporter)

ROUT --> PROM(Prometheus<br/>HTTP<br/>endpoint)
style ROUT stroke-dasharray: 3 3;
style OTELM stroke-dasharray: 3 3;
style OTELT stroke-dasharray: 3 3;
style PROM stroke-dasharray: 3 3;
```

A quick description of the components:
Expand All @@ -39,8 +40,10 @@ A quick description of the components:
the incoming data will be directly forwarded to the next stage.
* [OTEL metrics exporter](#otel_metrics) exports metrics data to an external
[OpenTelemetry](https://opentelemetry.io/) metrics collector.
* [OTEL traces exporter](#otel_metrics) exports span data to an external
* [OTEL traces exporter](#otel_traces) exports span data to an external
[OpenTelemetry](https://opentelemetry.io/) traces collector.
* [Prometheus HTTP endpoint](#prom) sets an HTTP endpoint in the auto-instrumenter
that allows any external scraper to pull metrics in [Prometheus](https://prometheus.io/) format.

Following sections explain both the global configuration properties, as well as
the options for each component.
Expand Down Expand Up @@ -248,6 +251,52 @@ so the metrics exporter won't be activated unless explicitly specified.
Specifies the name of the instrumented service to be reported by the traces exporter.
If unset, it will be the path of the instrumented service (e.g. `/usr/local/bin/service`).

## Prometheus HTTP endpoint (YAML section: `prometheus_export`)<a id="prom"></a>

This component opens an HTTP endpoint in the auto-instrumenter
that allows any external scraper to pull metrics in [Prometheus](https://prometheus.io/)
format. It will be enabled if the `port` property is set.

| YAML | Env var | Type | Default |
|--------|-------------------|------|---------|
| `port` | `PROMETHEUS_PORT` | int | (unset) |

Specifies the HTTP port to open a Prometheus scrape endpoint. If unset or 0,
no Prometheus endpoint will be open.

| YAML | Env var | Type | Default |
|----------------|---------------------------|--------|-----------------|
| `service_name` | `PROMETHEUS_SERVICE_NAME` | string | executable path |

Specifies the name of the instrumented service to be reported by the metrics exporter.
If unset, it will be the path of the instrumented service (e.g. `/usr/local/bin/service`).

| YAML | Env var | Type | Default |
|--------|-------------------|--------|------------|
| `path` | `PROMETHEUS_PATH` | string | `/metrics` |

Specifies the HTTP query path to acquire the list of Prometheus metrics.

| YAML | Env var | Type | Default |
|-----------------|-------------------------|---------|---------|
| `report_target` | `METRICS_REPORT_TARGET` | boolean | `false` |

Specifies whether the exporter must submit `http_target` as a metric attribute.

To be consistent with the OpenTelemetry specification, `http_target` is the full HTTP request
path and query arguments.

It is disabled by default to avoid cardinality explosion in paths with IDs. As an alternative,
it is recommended to group these requests in the [routes node](#routes).

| YAML | Env var | Type | Default |
|---------------|-----------------------|---------|---------|
| `report_peer` | `METRICS_REPORT_PEER` | boolean | `false` |

Specifies whether the exporter must submit the caller peer address as a metric attribute.

It is disabled by default to avoid cardinality explosion.

## YAML file example

```yaml
Expand All @@ -257,7 +306,11 @@ ebpf:
open_port: 443
wakeup_len: 100

otel_metrics:
otel_traces:
service_name: my-instrumented-service
endpoint: https://otlp-gateway-prod-eu-west-0.grafana.net/otlp

prometheus_export:
port: 8999
path: /metrics
```
11 changes: 9 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ require (
github.com/hashicorp/golang-lru/v2 v2.0.2
github.com/mariomac/guara v0.0.0-20221222112709-f95b15506aee
github.com/mariomac/pipes v0.7.0
github.com/prometheus/client_golang v1.15.1
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/stretchr/testify v1.8.2
go.opentelemetry.io/collector/pdata v0.66.0
Expand All @@ -27,13 +28,15 @@ require (
golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb
golang.org/x/sys v0.6.0
google.golang.org/grpc v1.53.0
google.golang.org/protobuf v1.28.1
google.golang.org/protobuf v1.30.0
gopkg.in/yaml.v3 v3.0.1
)

require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/bytedance/sonic v1.8.0 // indirect
github.com/cenkalti/backoff/v4 v4.2.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
Expand All @@ -52,10 +55,14 @@ require (
github.com/klauspost/cpuid/v2 v2.0.9 // indirect
github.com/leodido/go-urn v1.2.1 // indirect
github.com/mattn/go-isatty v0.0.17 // indirect
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pelletier/go-toml/v2 v2.0.6 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/common v0.42.0 // indirect
github.com/prometheus/procfs v0.9.0 // indirect
github.com/tklauser/go-sysconf v0.3.11 // indirect
github.com/tklauser/numcpus v0.6.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
Expand Down
21 changes: 18 additions & 3 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bytedance/sonic v1.5.0/go.mod h1:ED5hyg4y6t3/9Ku1R6dU/4KyJ48DZ4jPhfY1O2AihPM=
github.com/bytedance/sonic v1.8.0 h1:ea0Xadu+sHlu7x5O3gKhRpQ1IKiMrSiHttPF0ybECuA=
github.com/bytedance/sonic v1.8.0/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U=
Expand All @@ -45,6 +47,8 @@ github.com/cenkalti/backoff/v4 v4.2.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyY
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 h1:qSGYFH7+jGhDF8vLC+iwCD4WpbV1EBDSzWkJODFLams=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311/go.mod h1:b583jCggY9gE99b6G5LEC39OIiVsWj+R97kbl5odCEk=
Expand Down Expand Up @@ -192,15 +196,26 @@ github.com/mariomac/pipes v0.7.0 h1:J7exY2HSiDD/ANy0PB7+PCHomSlC0tG1R1je+zhBYhQ=
github.com/mariomac/pipes v0.7.0/go.mod h1:jQjrTTJxshdixw/Gh4NbnilbL0pVyq2Ssp/v2YE3A1c=
github.com/mattn/go-isatty v0.0.17 h1:BTarxUcIeDqL27Mc+vyvdWYSL28zpIhv3RoTdsLMPng=
github.com/mattn/go-isatty v0.0.17/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 h1:ZqeYNhU3OHLH3mGKHDcjJRFFRrJa6eAM5H+CtDdOsPc=
github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo=
github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/pelletier/go-toml/v2 v2.0.6 h1:nrzqCb7j9cDFj2coyLNLaZuJTLjWjlaz6nvTvIwycIU=
github.com/pelletier/go-toml/v2 v2.0.6/go.mod h1:eumQOmlWiOPt5WriQQqoM5y18pDHwha2N+QD+EUNTek=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.15.1 h1:8tXpTmJbyH5lydzFPoxSIJ0J46jdh3tylbvM1xCv0LI=
github.com/prometheus/client_golang v1.15.1/go.mod h1:e9yaBhRPU2pPNsZwE+JdQl0KEt1N9XgF6zxWmaC0xOk=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.3.0 h1:UBgGFHqYdG/TPFD1B1ogZywDqEkwp3fBMvqdiQ7Xew4=
github.com/prometheus/client_model v0.3.0/go.mod h1:LDGWKZIo7rky3hgvBe+caln+Dr3dPggB5dvjtD7w9+w=
github.com/prometheus/common v0.42.0 h1:EKsfXEYo4JpWMHH5cg+KOUWeuJSov1Id8zGR8eeI1YM=
github.com/prometheus/common v0.42.0/go.mod h1:xBwqVerjNdUDjgODMpudtOMwlOwf2SaTr1yjz4b7Zbc=
github.com/prometheus/procfs v0.9.0 h1:wzCHvIvM5SxWqYvwgVL7yJY8Lz3PKn49KQtpgMYJfhI=
github.com/prometheus/procfs v0.9.0/go.mod h1:+pB4zwohETzFnmlpe6yd2lSc+0/46IYZRB/chUwxUZY=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
Expand Down Expand Up @@ -532,8 +547,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=
google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
Expand Down
9 changes: 0 additions & 9 deletions pkg/ebpf/common/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,6 @@ type TracerConfig struct {
// BpfBaseDir specifies the base directory where the BPF pinned maps will be mounted.
// By default, it will be /var/run/otelauto
BpfBaseDir string `yaml:"bpf_fs_base_dir" env:"BPF_FS_BASE_DIR"`

// Below this line, there are some configuration options that need to be setup by
// the invoker code.
// TODO: add to the Pipes library the possibility of receiving a context.Context
// from the node constructors, so we can pass this info without reusing the configuration.

// OnOffsets will be called when the offsets are discovered (if not nil).
// It is useful to make executable information visible to other parts of the code
OnOffsets func(offsets *goexec.Offsets) `yaml:"-" json:"-"`
}

// Probe holds the information of the instrumentation points of a given function: its start and end offsets and
Expand Down
9 changes: 6 additions & 3 deletions pkg/ebpf/tracer.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"strconv"
"strings"

"github.com/grafana/ebpf-autoinstrument/pkg/pipe/global"

"golang.org/x/sys/unix"

"github.com/grafana/ebpf-autoinstrument/pkg/ebpf/goruntime"
Expand Down Expand Up @@ -232,9 +234,10 @@ func inspect(ctx context.Context, cfg *ebpfcommon.TracerConfig, functions []stri
if err != nil {
return nil, fmt.Errorf("error analysing target executable: %w", err)
}
if cfg.OnOffsets != nil {
cfg.OnOffsets(offsets)
}

parts := strings.Split(offsets.FileInfo.CmdExePath, "/")
global.Context(ctx).ServiceName = parts[len(parts)-1]

return offsets, nil
}

Expand Down
36 changes: 28 additions & 8 deletions pkg/export/otel/metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import (
"strings"
"time"

"github.com/grafana/ebpf-autoinstrument/pkg/pipe/global"

"golang.org/x/exp/slog"

"github.com/grafana/ebpf-autoinstrument/pkg/transform"
Expand All @@ -19,6 +21,15 @@ import (
semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
)

const (
HTTPServerDuration = "http.server.duration"
HTTPClientDuration = "http.client.duration"
RPCServerDuration = "rpc.server.duration"
RPCClientDuration = "rpc.client.duration"
HTTPServerRequestSize = "http.server.request.size"
HTTPClientRequestSize = "http.client.request.size"
)

type MetricsConfig struct {
ServiceName string `yaml:"service_name" env:"OTEL_SERVICE_NAME"`
Interval time.Duration `yaml:"interval" env:"METRICS_INTERVAL"`
Expand Down Expand Up @@ -67,10 +78,16 @@ func newMetricsReporter(ctx context.Context, cfg *MetricsConfig) (*MetricsReport
reportPeer: cfg.ReportPeerInfo,
}

// If service name is not explicitly set, we take the service name as set by the
// executable inspector
svcName := cfg.ServiceName
if svcName == "" {
svcName = global.Context(ctx).ServiceName
}
// TODO: make configurable
resources := resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceNameKey.String(cfg.ServiceName),
semconv.ServiceNameKey.String(svcName),
)
opts, err := getMetricEndpointOptions(cfg)
if err != nil {
Expand All @@ -85,33 +102,36 @@ func newMetricsReporter(ctx context.Context, cfg *MetricsConfig) (*MetricsReport
metric.WithReader(metric.NewPeriodicReader(mr.exporter,
metric.WithInterval(cfg.Interval))),
)
// time units for HTTP and GRPC durations are in seconds, according to the OTEL specification:
// https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/metrics/semantic_conventions
// TODO: set ExplicitBucketBoundaries here and in prometheus from the previous specification
mr.httpDuration, err = mr.provider.Meter(reporterName).
Float64Histogram("http.server.duration", instrument.WithUnit("ms"))
Float64Histogram(HTTPServerDuration, instrument.WithUnit("s"))
if err != nil {
return nil, fmt.Errorf("creating http duration histogram metric: %w", err)
}
mr.httpClientDuration, err = mr.provider.Meter(reporterName).
Float64Histogram("http.client.duration", instrument.WithUnit("ms"))
Float64Histogram(HTTPClientDuration, instrument.WithUnit("s"))
if err != nil {
return nil, fmt.Errorf("creating http duration histogram metric: %w", err)
}
mr.grpcDuration, err = mr.provider.Meter(reporterName).
Float64Histogram("rpc.server.duration", instrument.WithUnit("ms"))
Float64Histogram(RPCServerDuration, instrument.WithUnit("s"))
if err != nil {
return nil, fmt.Errorf("creating grpc duration histogram metric: %w", err)
}
mr.grpcClientDuration, err = mr.provider.Meter(reporterName).
Float64Histogram("rpc.client.duration", instrument.WithUnit("ms"))
Float64Histogram(RPCClientDuration, instrument.WithUnit("s"))
if err != nil {
return nil, fmt.Errorf("creating grpc duration histogram metric: %w", err)
}
mr.httpRequestSize, err = mr.provider.Meter(reporterName).
Float64Histogram("http.server.request.size", instrument.WithUnit("By"))
Float64Histogram(HTTPServerRequestSize, instrument.WithUnit("By"))
if err != nil {
return nil, fmt.Errorf("creating http size histogram metric: %w", err)
}
mr.httpClientRequestSize, err = mr.provider.Meter(reporterName).
Float64Histogram("http.client.request.size", instrument.WithUnit("By"))
Float64Histogram(HTTPClientRequestSize, instrument.WithUnit("By"))
if err != nil {
return nil, fmt.Errorf("creating http size histogram metric: %w", err)
}
Expand Down Expand Up @@ -171,7 +191,7 @@ func (r *MetricsReporter) metricAttributes(span *transform.HTTPRequestSpan) []at

func (r *MetricsReporter) record(span *transform.HTTPRequestSpan, attrs []attribute.KeyValue) {
t := span.Timings()
duration := t.End.Sub(t.RequestStart).Seconds() * 1000
duration := t.End.Sub(t.RequestStart).Seconds()
switch span.Type {
case transform.EventTypeHTTP:
// TODO: for more accuracy, there must be a way to set the metric time from the actual span end time
Expand Down
10 changes: 9 additions & 1 deletion pkg/export/otel/traces.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ import (
"strings"
"time"

"github.com/grafana/ebpf-autoinstrument/pkg/pipe/global"

"golang.org/x/exp/slog"

"github.com/grafana/ebpf-autoinstrument/pkg/transform"
Expand Down Expand Up @@ -71,10 +73,16 @@ func TracesReporterProvider(ctx context.Context, cfg TracesConfig) (node.Termina
func newTracesReporter(ctx context.Context, cfg *TracesConfig) (*TracesReporter, error) {
r := TracesReporter{ctx: ctx}

// If service name is not explicitly set, we take the service name as set by the
// executable inspector
svcName := cfg.ServiceName
if svcName == "" {
svcName = global.Context(ctx).ServiceName
}
// TODO: make configurable
resources := resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceNameKey.String(cfg.ServiceName),
semconv.ServiceNameKey.String(svcName),
)

// Instantiate the OTLP HTTP traceExporter
Expand Down
Loading