Remove host from Grafana

Hi

I'm using Grafana + Prometheus + Node Exporter with Node Exporter Full Dashboard and would like to remove an old host from Grafana Dashboard.

Whenever I delete a series, the data is removed from Prometheus but in the Grafana Dashboard, the host remains there.

These are the commands I use:

curl -s -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="my.instance.tld:9100",job="job_name"}'
curl -s -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'

How can I remove the host from Grafana Dashboard?

Thanks

Comments

  • edited October 2022

    Without knowing your config, I'm guessing the dashboard populates the host field with a variable.
    Check the settings for the dashboard to see how it generates those variables, my guess is that it's trough a Prometheus query and that the host still exists in the database.
    Or you simply have not updated the variable after removing the host from Prometheus, there is a button for it in the dashboard variables settings.

    Thanked by (1)nfn
  • edited October 2022

    Unless you hardcoded hosts in the Grafana dashboard, it's most likely still living somewhere in Prometheus.

    Perhaps you forgot to disable the job for that host and it's still scraping?

    Thanked by (1)nfn
  • nfnnfn
    edited October 2022

    I'm testing Grafana, just added a new target and after an hour, I removed from Prometheus configuration.

    I'm using the default Prometheus configuration.

  • if someone is interested, it looks like the only way to do this is to wait up to the retention period (15 days by default)
    After removing the instance, I forced two hours and the host was purged from Prometheus.

    If you need to configure retention to a value other than 15d, you'll need to set --storage.tsdb.retention.time in the system's service on ExecStart

    https://prometheus.io/docs/prometheus/latest/storage/#compaction

Sign In or Register to comment.