Motivation

I recently read a post about how someone’s homelab got hacked. Since I recently created this website, it bothered me that a forgejo vulnerability caused them to get hacked since I’m running forgejo. I made sure I wasn’t vulnerable (they were running a rather old version of forgejo). I then thought to myself how could I detect this, since at that moment, I didn’t have any kind of monitoring in place on my VPS. So I decided that I would add monitoring and alerting to my instance.

I chose to use Grafana or as I later learned about it, they now call it the LGTM stack. My decision basically boiled down to wanting to self host an opensource solution and I previously setup Grafana a long time ago.

Installation

1. Setup the package repository on debian by following Grafana’s instructions.
2. Make sure your repos are up to date: apt update
3. I installed Grafana (dashboards + alerting), Mimir (metrics db), Loki (logs db), and Alloy (collector): apt install grafana mimir loki alloy

Configuring

Mimir

The default configuration requires a replication factor of 3, I didn’t need any redundancy since I was installing this on one machine, so I had to update the config at /etc/mimir/config.yml:

ingester:
  ring:
    replication_factor: 1

Alloy

You need to configure alloy for your use case, but my /etc/alloy/config.alloy looks like:

logging {
  level = "warn"
}

prometheus.remote_write "default" {
  // Send metrics to a Mimir instance
  endpoint {
    url = "http://localhost:8080/api/v1/push"

    headers = {
      "X-Scope-OrgID" = "anonymous",
    }
  }
}

prometheus.exporter.unix "default" {
  // include_exporter_metrics = true
  // disable_collectors       = ["mdadm"]
}

prometheus.scrape "default" {
  targets = array.concat(
    prometheus.exporter.unix.default.targets,
    [{
      // Self-collect metrics
      job         = "alloy",
      __address__ = "127.0.0.1:12345",
    }],
  )

  forward_to = [prometheus.remote_write.default.receiver]
}


//////////////// logs

local.file_match "local_files" {
    path_targets = [
      {"__path__" = "/var/log/caddy/blog_access.log", "job" = "caddy_blog", "hostname" = constants.hostname},
      {"__path__" = "/var/log/caddy/git_access.log", "job" = "caddy_git", "hostname" = constants.hostname},
    ]
    sync_period  = "5s"
}

loki.source.file "log_scrape" {
    targets    = local.file_match.local_files.targets
    forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://localhost:3100/loki/api/v1/push"
  }
}

There’s some additional things I could do like extract timestamps and log levels, but just to get started this is good enough.

Ensuring Services are Started and Enabled

Don’t forget to systemctl start and systemctl enable each of the above services to ensure they are running.

Conclusion

By following the above high level steps, I was able to achieve a basic working LGTM stack minus the T since I didn’t need distributed tracing at the time.

One frustration I had while installing and configuring all the above is that the docs for installing the LGTM stack are very verbose. I wish the docs had a single document on setting up a single, selfhosted machine running grafana. Well I guess that’s why I wrote this doc. Hope this helps someone!

I now have basic monitoring in place to notify me of abnormal conditions on my server.