Grafana monitoring and more importantly alerting!
So the first docker containers I fired up on my new Raspbery Pi 4 with 8 GB of RAM when I received it back in July of 2022 were Telegraf, Influx, and Grafana to monitor the new little doggy. And I got a really pretty dashboard working and then kinda forgot about it:

Until this week when I realized that I had a process pegged at 100% since way back in december:

So the first thing to do was to figure out what was pegging the CPU which turned out to be a nightly log rotation that went bad. Unfortunately can’t put too many details because I did not take any notes while investigating and solving, only can comment that it had created thousands of files and every time it tried to run hung. Trying to clean up the files was really fun as doing an rm with a wildcard even gave me and error that there were too many files. Google to the rescue and I got that under control but realized that having a really pretty dashboard did nothing if I did not setup up alerting when things were abnormal.