Using Loki for Logging

In this Writer's Room blog, Andela Community member Isaac Kiptanui explains how you can use Loki – an open-source log aggregation system built on Prometheus – for logging, and creating fast queries and indexing, even at a large scale!

Loki is an open-source log aggregation system that is designed to be easy to operate, highly available, and cost-effective. It was created by Grafana Labs and is part of the CNCF (Cloud Native Computing Foundation) ecosystem.

Loki is built on top of Prometheus, another popular CNCF project, and is designed to be highly scalable and performant. It is intended to be used as a backend for logging in cloud-native environments, where traditional log aggregation tools may not be well-suited.

Loki uses a unique indexing approach that is designed to be highly efficient and scalable. Instead of indexing logs by their content, Loki indexes logs by their labels, which are essentially key-value pairs that describe the log entry. This allows for very fast queries and indexing, even at very large scale.

Loki supports a range of different log formats and can be integrated with a wide variety of logging tools and platforms. It can be run on-premises or in the cloud, and can be integrated with other cloud-native tools like Kubernetes and Istio.

To use Loki for logging in a Golang API, you can use a third-party library that supports Loki as a logging backend. One such library is logrus with the lokihook.

Here's an example of how to set up logrus with the loki hook to send logs to Loki:

package mainimport ( "github.com/sirupsen/logrus" "github.com/sirupsen/logrus/hooks/loki")func main() { // Create a new logrus logger log := logrus.New()// Create a new loki hook hook, err := loki.NewHook("http://loki:3100/loki/api/v1/push", "my-app", &loki.Config{}) if err != nil { log.Fatalf("error creating loki hook: %v", err) }// Add the hook to the logger log.Hooks.Add(hook)// Use the logger to log messages log.Info("hello, world!")}

In this example, we create a new logrus logger and a new loki hook. The loki hook is configured with the URL of the Loki server and the name of the application. We then add the hook to the logger using the Add method.

Finally, we use the logger to log a message with the Info method. This message will be sent to the Loki server via the loki hook.

With this setup, you can use the Loki server to view logs from your Golang API and analyze them using Loki's powerful querying and visualization features.

To check logs that have been sent to Loki, you can use the Loki web UI or the command-line interface.

To use the Loki web UI:

1. Open a web browser and go to the URL of your Loki server, e.g. http://localhost:3100.
2. In the top navigation bar, click the “Explore” button.
3. In the query editor, enter a query that matches the logs you want to view, e.g. {app="my-app"}.
4. Click the “Run” button to execute the query.
5. The logs that match the query will be displayed in the log viewer.

To use the Loki command-line interface:

1. Open a terminal or command prompt.
2. Use the loki command-line tool to query logs, e.g. loki query '{app="my-app"}'.
3. The logs that match the query will be displayed in the terminal.

You can also integrate Loki with other tools, such as Grafana, to create custom dashboards and alerts based on your logs.

You can also use Docker to run Loki and other related tools, such as Promtail and Grafana. Here's an example docker-compose.yml file that sets up a simple Loki stack with Promtail and Grafana:

In this example, we define three services:

• loki: the main Loki server that stores and indexes logs
• promtail: a log collector that sends logs from files on the host machine to Loki
• grafana: a data visualization and exploration tool that can be used to view logs stored in Loki

We map the loki service to port 3100 on the host machine and the grafanaservice to port 3000 on the host machine, so we can access them via a web browser.

We also define volumes for the loki and promtail services, which allow us to mount local configuration files into the containers. These configuration files define the specific behavior and settings of the services.

With this docker-compose.yml file, you can run the entire Loki stack by running the following command in the same directory as the docker-compose.yml file:

This will start all three services and display their logs in the terminal.

You can then access Loki at http://localhost:3100, Grafana at http://localhost:3000, and configure Promtail to collect logs from specific files on the host machine by editing the promtail-config.yaml file.

Want to be part of a vibrant tech community?

Then join the Andela Talent Network!

Find Work

If you found this blog useful, check out our other blog posts for more essential insights!