grafana / grafana/killercoda

Grafana "Fundamentals Environment" not Ingesting Data

Open
#280 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30
Forks
16
PR merge metrics
No merged PRs in 30d

Description

When following the interactive tutorial I can't complete more than a few steps because no data seems to be ingested by Prometheus.

### Steps to Reproduce
1. Log in and get a new environment at: https://killercoda.com/grafana-labs/course/grafana/grafana-fundamentals
2. Complete all the docker setup steps:
```sh
git clone https://github.com/grafana/tutorial-environment.git
cd tutorial-environment
docker-compose up -d
docker-compose ps

ubuntu:~/tutorial-environment$ docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------------------------------
tutorial-environment_app_1 /bin/sh -c /app http://db ... Up 0.0.0.0:8081->80/tcp,:::8081->80/tcp
tutorial-environment_db_1 /db Up 0.0.0.0:8082->80/tcp,:::8082->80/tcp
tutorial-environment_grafana_1 /run.sh Up 0.0.0.0:3000->3000/tcp,:::3000->3000/tcp
tutorial-environment_loki_1 /usr/bin/loki -config.file ... Up 0.0.0.0:3100->3100/tcp,:::3100->3100/tcp
tutorial-environment_prometheus_1 /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp,:::9090->9090/tcp
tutorial-environment_promtail_1 /usr/bin/promtail -config. ... Up
```
3. Open the app at: http://localhost:8081 and confirm it's working
4. Open Grafana at: http://localhost:3000 and confirm it's working
5. Click on "Explore"
6. Switch to "Code" in the top right toggle
7. Write `tns_request_duration_seconds_count` in the query box and press Shift+Enter

At this point according to the tutorial I would expect to see a graph with some data, however the tab below shows "No data"

Image

The same thing happens with the next queries such as
- `rate(tns_request_duration_seconds_count[5m])`
- `sum(rate(tns_request_duration_seconds_count[5m])) by(route)`
- `sum(rate(tns_request_duration_seconds_count[5m])) by(status_code)`

### Prometeus Logs
```
ubuntu:~/tutorial-environment$ docker logs tutorial-environment_prometheus_1
ts=2026-01-29T23:18:45.455Z caller=main.go:544 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2026-01-29T23:18:45.456Z caller=main.go:588 level=info msg="Starting Prometheus Server" mode=server version="(version=2.49.0, branch=HEAD, revision=683a46cc9999dedf66dd0a4c77644b8d0031d063)"
ts=2026-01-29T23:18:45.457Z caller=main.go:593 level=info build_context="(go=go1.21.6, platform=linux/amd64, user=root@a83669167414, date=20240115-10:51:05, tags=netgo,builtinassets,stringlabels)"
ts=2026-01-29T23:18:45.457Z caller=main.go:594 level=info host_details="(Linux 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 02ab5fd82e79 (none))"
ts=2026-01-29T23:18:45.457Z caller=main.go:595 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2026-01-29T23:18:45.457Z caller=main.go:596 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2026-01-29T23:18:45.465Z caller=web.go:565 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2026-01-29T23:18:45.515Z caller=main.go:1039 level=info msg="Starting TSDB ..."
ts=2026-01-29T23:18:45.542Z caller=head.go:606 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2026-01-29T23:18:45.543Z caller=head.go:687 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=6.731µs
ts=2026-01-29T23:18:45.543Z caller=head.go:695 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2026-01-29T23:18:45.554Z caller=tls_config.go:274 level=info component=web msg="Listening on" address=[::]:9090
ts=2026-01-29T23:18:45.555Z caller=tls_config.go:277 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
ts=2026-01-29T23:18:45.555Z caller=head.go:766 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
ts=2026-01-29T23:18:45.555Z caller=head.go:803 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=77.228µs wal_replay_duration=11.439003ms wbl_replay_duration=199ns total_replay_duration=11.737057ms
ts=2026-01-29T23:18:45.560Z caller=main.go:1060 level=info fs_type=EXT4_SUPER_MAGIC
ts=2026-01-29T23:18:45.560Z caller=main.go:1063 level=info msg="TSDB started"
ts=2026-01-29T23:18:45.562Z caller=main.go:1245 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2026-01-29T23:18:45.566Z caller=main.go:1282 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=3.443226ms db_storage=2.018µs remote_storage=1.854µs web_handler=658ns query_engine=1.32µs scrape=531.494µs scrape_sd=37.535µs notify=834ns notify_sd=1.506µs rules=1.44µs tracing=10.634µs
ts=2026-01-29T23:18:45.566Z caller=main.go:1024 level=info msg="Server is ready to receive web requests."
ts=2026-01-29T23:18:45.566Z caller=manager.go:146 level=info component="rule manager" msg="Starting rule manager..."
```
### Grafana Logs
They are a bit verbose but I grepped "error" and maybe these are relevant logs
```
ubuntu:~/tutorial-environment$ docker logs tutorial-environment_grafana_1 | grep error
logger=migrator t=2026-01-29T23:18:46.214197866Z level=debug msg="Executing sql migration" id="create migration_log table" sql="CREATE TABLE IF NOT EXISTS `migration_log` (\n`id` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL\n, `migration_id` TEXT NOT NULL\n, `sql` TEXT NOT NULL\n, `success` INTEGER NOT NULL\n, `error` TEXT NOT NULL\n, `timestamp` DATETIME NOT NULL\n);"
logger=migrator t=2026-01-29T23:18:47.416134822Z level=debug msg="Executing sql migration" id="create alert table v1" sql="CREATE TABLE IF NOT EXISTS `alert` (\n`id` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL\n, `version` INTEGER NOT NULL\n, `dashboard_id` INTEGER NOT NULL\n, `panel_id` INTEGER NOT NULL\n, `org_id` INTEGER NOT NULL\n, `name` TEXT NOT NULL\n, `message` TEXT NOT NULL\n, `state` TEXT NOT NULL\n, `settings` TEXT NOT NULL\n, `frequency` INTEGER NOT NULL\n, `handler` INTEGER NOT NULL\n, `severity` TEXT NOT NULL\n, `silenced` INTEGER NOT NULL\n, `execution_error` TEXT NOT NULL\n, `eval_data` TEXT NULL\n, `eval_date` DATETIME NULL\n, `new_state_date` DATETIME NOT NULL\n, `state_changes` INTEGER NOT NULL\n, `created` DATETIME NOT NULL\n, `updated` DATETIME NOT NULL\n);"
logger=migrator t=2026-01-29T23:18:50.259502719Z level=info msg="Executing migration" id="add snapshot error_string column"
logger=migrator t=2026-01-29T23:18:50.259921906Z level=debug msg="Executing sql migration" id="add snapshot error_string column" sql="alter table `cloud_migration_snapshot` ADD COLUMN `error_string` TEXT NULL "
logger=migrator t=2026-01-29T23:18:50.273927476Z level=info msg="Migration successfully executed" id="add snapshot error_string column" duration=14.422343ms
logger=migrator t=2026-01-29T23:18:50.277875425Z level=debug msg="Executing sql migration" id="create cloud_migration_resource table v1" sql="CREATE TABLE IF NOT EXISTS `cloud_migration_resource` (\n`id` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL\n, `uid` TEXT NOT NULL\n, `resource_type` TEXT NOT NULL\n, `resource_uid` TEXT NOT NULL\n, `status` TEXT NOT NULL\n, `error_string` TEXT NULL\n, `snapshot_uid` TEXT NOT NULL\n);"
logger=migrator t=2026-01-29T23:18:50.385751488Z level=info msg="Executing migration" id="add cloud_migration_resource.error_code column"
logger=migrator t=2026-01-29T23:18:50.385894313Z level=debug msg="Executing sql migration" id="add cloud_migration_resource.error_code column" sql="alter table `cloud_migration_resource` ADD COLUMN `error_code` TEXT NULL "
logger=migrator t=2026-01-29T23:18:50.396827499Z level=info msg="Migration successfully executed" id="add cloud_migration_resource.error_code column" duration=11.07366ms
logger=resource-migrator t=2026-01-29T23:18:50.883470188Z level=debug msg="Executing sql migration" id="create resource_migration_log table" sql="CREATE TABLE IF NOT EXISTS `resource_migration_log` (\n`id` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL\n, `migration_id` TEXT NOT NULL\n, `sql` TEXT NOT NULL\n, `success` INTEGER NOT NULL\n, `error` TEXT NOT NULL\n, `timestamp` DATETIME NOT NULL\n);"
logger=plugins.registration t=2026-01-29T23:18:51.067348651Z level=error msg="Could not register plugin" pluginId=table error="plugin table is already registered"
logger=plugins.initialization t=2026-01-29T23:18:51.067357441Z level=error msg="Could not initialize plugin" pluginId=table error="plugin table is already registered"
logger=sqlstore.transactions t=2026-01-29T23:18:51.158334258Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0
logger=sqlstore.transactions t=2026-01-29T23:18:51.204832825Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0
logger=sqlstore.transactions t=2026-01-29T23:18:51.298686799Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1
logger=sqlstore.transactions t=2026-01-29T23:18:51.319142406Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2
logger=sqlstore.transactions t=2026-01-29T23:18:51.339725932Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3
logger=sqlstore.transactions t=2026-01-29T23:18:51.359867131Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4
logger=serviceaccounts t=2026-01-29T23:18:51.385012038Z level=error msg="Failed to lock and execute the migration of API keys to service accounts" error="database is locked"
logger=context userId=0 orgId=1 uname= t=2026-01-29T23:19:09.186714527Z level=debug msg="Failed to authenticate user in global scope" error="[anonymous.invalid-org] anonymous user cannot authenticate in org 1"
ubuntu:~/tutorial-environment$
```

**Browsers used:**
- Brave v1.86.146
- Edge v144.0.3719.92

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the tutorial-environment repository using docker-compose up -d, then inspect the Prometheus setup and the running app after querying tns_request_duration_seconds_count in Grafana Explore. Check whether the app exposes the metric and whether Prometheus is scraping it. Done means the tutorial queries return data in Grafana.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, prometheus
Domain
devops, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.