Remove host from Grafana

nfn

Hi

I'm using Grafana + Prometheus + Node Exporter with Node Exporter Full Dashboard and would like to remove an old host from Grafana Dashboard.

Whenever I delete a series, the data is removed from Prometheus but in the Grafana Dashboard, the host remains there.

These are the commands I use:

curl -s -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="my.instance.tld:9100",job="job_name"}'
curl -s -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'

How can I remove the host from Grafana Dashboard?

Thanks

rcy026

Without knowing your config, I'm guessing the dashboard populates the host field with a variable.
Check the settings for the dashboard to see how it generates those variables, my guess is that it's trough a Prometheus query and that the host still exists in the database.
Or you simply have not updated the variable after removing the host from Prometheus, there is a button for it in the dashboard variables settings.

jmgcaguicla

Unless you hardcoded hosts in the Grafana dashboard, it's most likely still living somewhere in Prometheus.

Perhaps you forgot to disable the job for that host and it's still scraping?

nfn

I'm testing Grafana, just added a new target and after an hour, I removed from Prometheus configuration.

I'm using the default Prometheus configuration.

nfn

if someone is interested, it looks like the only way to do this is to wait up to the retention period (15 days by default)
After removing the instance, I forced two hours and the host was purged from Prometheus.

If you need to configure retention to a value other than 15d, you'll need to set --storage.tsdb.retention.time in the system's service on ExecStart

https://prometheus.io/docs/prometheus/latest/storage/#compaction