Smoke OTEL - 30432846470

[github-actions]

Executive Summary

Overall: FAIL

The current workflow run (30432846470) is fully visible end-to-end for the Local OTLP mirror and Sentry. Grafana and Datadog both fail: Grafana's MCP surface lacks a trace/span query tool and is blocked by a datasource permission error; Datadog's OTLP export is rejected (404) and its MCP surface exposes zero tools.

Results Matrix

Backend	Write Config Present	Write Export Succeeded	Read Config Present	Read Query Succeeded	Overall
Local OTLP	✅	✅	—	—	✅
Sentry	✅	✅	✅	✅	✅
Grafana	✅	✅	✅	❌	❌
Datadog	✅	❌	❌	❌	❌

Evidence

Local OTLP (send_status = pass)

• GH_AW_OTLP_ENDPOINTS set, configured hosts: o205451.ingest.us.sentry.io, otlp-gateway-prod-eu-west-2.grafana.net, otlp-intake.datadoghq.eu.
• /tmp/gh-aw/otel.jsonl contains span gh-aw.agent.setup, trace ID 66940c8b816a69e4adbfef5f61b4ed6b, with resource attributes gh-aw.run.id=30432846470 and github.run_id=30432846470 matching this run.
• otlp-export-errors.count = 1; the single recorded error is attributed to Datadog (see below), not Sentry or the local mirror.

Sentry (sentry_status = pass)

• whoami succeeded: authenticated as Mara Kiefer, org github (region us.sentry.io).
• find_projects located project gh-aw in org github.
• list_events (dataset spans, project gh-aw, statsPeriod=24h, query trace:66940c8b816a69e4adbfef5f61b4ed6b) returned spans for exactly the same trace ID emitted locally (66940c8b816a69e4adbfef5f61b4ed6b), including POST /mcp/sentry, POST /mcp/datadog, POST /mcp/tavily, POST /mcp/agenticworkflows, POST /mcp/mcpscripts at 2026-07-29T07:47:57Z/07:48:33Z, consistent with this run's MCP gateway activity.
• errors and logs datasets for project gh-aw (24h) returned "No results found" — explicit empty-dataset finding, not a query failure (query executed successfully, Sentry link returned).
• Trace URL: https://github.sentry.io/explore/traces/trace/66940c8b816a69e4adbfef5f61b4ed6b
• No OTLP export errors attributable to Sentry host (o205451.ingest.us.sentry.io) in otlp-export-errors.jsonl.

Grafana (grafana_status = fail)

• MCP connection succeeded; only 2 tools exposed: list_datasources, get_datasource — no trace/span/query tool exists in this MCP surface, so current-run or fallback trace queries cannot be executed at all.
• list_datasources --type tempo confirmed a tracing datasource exists: grafanacloud-ghaw-traces (uid grafanacloud-traces, type tempo).
• get_datasource --uid grafanacloud-ghaw-traces failed: 403 getDataSourceByUidForbidden — Permissions needed: datasources:read.
• No OTLP export errors attributable to the Grafana host (otlp-gateway-prod-eu-west-2.grafana.net) in otlp-export-errors.jsonl, so write path appears to have succeeded, but this cannot be independently confirmed via read access.

Datadog (datadog_status = fail)

• MCP connection technically succeeded (initialize returned 200), but datadog --help reports 0 tools available, and calling whoami returned Error [-32602]: unknown tool "whoami". No trace/span query tool exists.
• otlp-export-errors.jsonl contains: {"host":"otlp-intake.datadoghq.eu","status":404,"reason":"Not Found"} — the OTLP write to Datadog was rejected.
• No current-run or fallback gh-aw spans could be checked since no query tool is exposed.

Failure Analysis

Grafana — Read Query Succeeded: ❌

• Failing step: get_datasource on the grafanacloud-ghaw-traces (Tempo) datasource.
• Evidence: 403 getDataSourceByUidForbidden {"message":"You'll need additional permissions to perform this action. Permissions needed: datasources:read"}. Additionally, the Grafana MCP tool surface only exposes list_datasources/get_datasource — no span/trace search tool (e.g. Tempo query) is available at all, so even with correct permissions no trace-level query could be run.
• Likely root cause: (a) Read path/auth — the configured GRAFANA_SERVICE_ACCOUNT_TOKEN lacks the datasources:read permission; (b) configuration/tooling gap — the Grafana MCP server build in use does not register a Tempo/trace-query tool, so read verification of current-run spans is structurally impossible via this MCP surface regardless of permissions.
• Next step: Grant the service account datasources:read (and ideally traces:read/Tempo query scope), and confirm the Grafana MCP server configuration includes a trace-search tool (e.g. query_traces or equivalent) — the currently enabled toolset needs to be expanded before this check can pass.

Datadog — Write Export Succeeded: ❌, Read Config Present: ❌, Read Query Succeeded: ❌

• Failing step: OTLP export to otlp-intake.datadoghq.eu and MCP tool invocation (whoami).
• Evidence: Export error {"host":"otlp-intake.datadoghq.eu","status":404,"reason":"Not Found"}; MCP --help lists 0 tools; whoami call returns Error [-32602]: unknown tool "whoami".
• Likely root cause: Write path — a 404 from the Datadog OTLP intake endpoint typically indicates a wrong/misconfigured ingestion path or an invalid/missing API key rather than a transient network issue (GH_AW_OTEL_DATADOG_ENDPOINT/GH_AW_OTEL_DATADOG_API_KEY should be verified). Read path — the Datadog MCP server appears to be misconfigured or not fully initialized in this workflow (DD_API_KEY/DD_APPLICATION_KEY/DD_APP_KEY secrets should be verified), since it registers no callable tools at all.
• Next step: Verify GH_AW_OTEL_DATADOG_ENDPOINT matches Datadog's documented OTLP intake path for the configured site (.eu) and that GH_AW_OTEL_DATADOG_API_KEY is valid; separately verify DD_API_KEY/DD_APPLICATION_KEY are correctly populated for the Datadog MCP server so it can register its tool set.

Run

https://github.com/github/gh-aw/actions/runs/30432846470

Generated by 🧪 Smoke OTEL · sonnet50 · 34 AIC · ⌖ 4.93 AIC · ⊞ 10.5K · ◷
_{Comment /smoke-otel-backends to run again
Add label smoke to run again}

• expires on Jul 29, 2026, 1:52 AM UTC-08:00

[github-actions]

This issue was automatically closed because it expired on 2026-07-29T09:52:35.511Z.

Closed by Workflow