We’re about to embark on an exciting journey into the realm of log aggregation using Grafana, Promtail, and Loki—all orchestrated through Docker Compose. This trio not only empowers you to take complete control over your logging but also extends a user-friendly graphical interface to explore your data with ease.
Prerequisites
Before we dive in, ensure that Docker and Docker Compose are installed on your home server. If you’re new to Docker, think of it as a toolkit that simplifies the process of developing, shipping, and running applications by leveraging containerization. For installation guidance, I recommend checking out the official Docker documentation.
Setting Up Loki and Promtail
At the heart of our setup is Loki, a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. Unlike traditional logging systems, Loki is designed for efficiency—it doesn’t index the content of logs but rather utilizes a set of labels for each log stream. This makes it both cost-effective and easy to operate.
Promtail acts as our log shipper, collecting log paths and metadata before pushing them to Loki. Let’s get started by crafting a docker-compose.yml file that will define our services:
version: "3"
services:
loki:
image: grafana/loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
volume:
- ./loki-local-config.yaml:/etc/loki/local-config.yaml
promtail:
image: grafana/promtail
volumes:
- ./promtail-config.yaml:/etc/promtail/config.yaml
- /var/log:/var/log
command: -config.file=/etc/promtail/config.yaml
Create Configuration Files
In the same directory as your docker-compose.yml, create two files: loki-local-config.yaml and promtail-config.yaml.
For loki-local-config.yaml, use the following configuration:
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 3m
max_chunk_age: 1h
chunk_target_size: 1048576
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
And for promtail-config.yaml, input:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
Run the Docker Compose
With all the configuration files in place, it’s time to bring our setup to life. Navigate to the directory housing your docker-compose.yml file and execute the command:
docker-compose up
Setting Up Grafana
Now we turn our attention to Grafana, the multi-platform visualization tool that transforms your data into stunning graphs, charts, and alerts.
Create a new Docker Compose file for Grafana with the following:
version: "3"
services:
grafana:
image: grafana/grafana
ports:
- 3000:3000
volumes:
- grafana_storage:/var/lib/grafana
volumes:
grafana_storage:
Run this Docker Compose file just as you did for Loki and Promtail. Once it’s up, navigate to Grafana by entering http://localhost:3000 in your browser.
To integrate Grafana with Loki:
- Click on the gear icon in the side menu.
- Select “Data Sources.”
- Click on “Add data source.”
- Choose “Loki.”
- In the URL field, input:
http://localhost:3100.
Congratulations! If all has gone according to plan, you now have a fully functional logging solution right at your fingertips. Happy logging!