Example blog

How to import logs with Humio Log Collector

[ad_1]

Begin

For this handy guide, we will explain how to use the Humio log collector to collect and send log events to your CrowdStrike Falcon® LogScale repository. Although Log Sender supports multiple types of log sources (see list here), we will cover the use case of collecting log events from journald.

To get started, just make sure you already have a working Falcon LogScale instance, formerly known as Humio, either self-hosted or via Falcon LogScale Cloud. We will be using Falcon LogScale Cloud with a Falcon LogScale Community Edition (LCE) account. If you don’t have a free LCE account yet, you can create one here. Typically, your account request will be approved within two business days.

Step 1: Add a new repository

Once connected to Falcon LogScale, the first thing to do is to create a repository. This will be the “bucket” where we will ship our log data. Move towards Repositories and views and click + Add new.

Fill in the required fields, then save the repository configuration. We’ll keep it basic for now, but more information on setting up the repository can be found here.

Step 2: Create an ingest token

Now that the repository is created, we need an ingestion token for Humio Log Collector to use when authenticating. In your repository, navigate to Settingsthen to Ingest Tokens. Click on + Add token.

Hostname Humio Ingest

For our demo, we will name our token ingestion test and use the system log parser as it usually tries to parse RFC 3164 and RFC 5424 formats. We will ingest logs from journaled. Since system log the parser is quite liberal in what it parses, making it a good candidate for ingesting logs from journald. If you are unsure which parser to use for your case, the list of built-in parsers can be found here.

Humio new token

Once the ingestion token has been created, we click on the “View” icon and copy its value to a safe place. We will use it in a few moments.

Step 3: Download and install Humio Log Collector

Now that we have a place to store our data and a token to authenticate our log collector, we need to upload our log sender. We sail to Organization Settings, Log Collector. There we follow the steps to download and install the Falcon LogScale log collector.

Steps of Humio Log Collector

First, we select the package that corresponds to our environment, and we click on Download the installer. In our case, we are going to install the log collector on a Fedora virtual machine, so we selected the Red Hat RPMs option.

With the RPM located in our Downloads/ folder, we run the installation.

$ cd ~/Downloads
$ sudo rpm -i humio-log-collector_1.1.0_linux_amd64.rpm

Simple enough!

At this point, Humio Log Collector is not running, but it is installed. The installation does a few things for you, such as creating a Falcon LogScale user and a directory under /etc which contains an example configuration. Let’s check these parts.

Verify user creation

$ sudo cat /etc/passwd | grep humio
humio-log-collector:x:980:976:humio-log-collector service account:/var/lib/humio-log-collector:/sbin/nologin

Check the directory and creation of the sample configuration

$ sudo ls -d1 /etc/humio-log-collector/
/etc/humio-log-collector/
$ sudo ls -1 /etc/humio-log-collector/
config.yaml
EULA.txt

Excellent. All the expected parts are in place.

Step 4: Configure and run Humio Log Collector

By default, Humio Log Collector will look for your configuration in /etc/humio-log-collector/config.yaml. The example configuration included during installation is pretty close to what we need, but we will have a few changes to make. In particular, we want the log collector to read from the log. We do this with the following configuration:

dataDirectory: /var/lib/humio-log-collector
sources:
  journal:
    type: journald
    directory: /var/log/journal
    sink: mylogsink
sinks:
  mylogsink:
    type: humio
    token: 
    url: https://cloud.community.humio.com

There’s nothing too magical here, but it deserves a quick explanation.

Instead of using the var_log source of the configuration example, we use the journald source and pointing it to the default log location of /var/log/log. The location of the log may vary depending on your operating system. Before going any further, it’s worth checking your system to know for sure what value to use for phone book.

Additionally, we need to paste the ingest token from our repository that we copied before. The URLs

for Falcon LogScale will depend on the type of Falcon LogScale account you have. LCE account users will use the URL shown in the example configuration above. If you have another type of account, you can reference the terminal documentation from Falcon LogScale for the exact endpoint to use.

Configure Permissions for Falcon LogScale Log Collector

We have one more step to take, which will ensure that the Falcon LogScale log collector can see everything we must see. We need to grant the log collector process special permissions to read the log from other units that would normally require elevated permissions.

To do this on Fedora and RHEL systems, we add Ambient capacities to systemd service record. Start by editing the systemd service file for log collector; it should be /usr/lib/systemd/system/humio-log-collector.service. We add a single line (highlighted) below the [Service] section. The resulting file should look like this:

[Unit]
Description=Humio Log Collector
After=network.target

[Service]
EnvironmentFile=/etc/default/humio-log-collector
ExecStart=/usr/bin/humio-log-collector -cfg ${CONFIG_FILE}
WorkingDirectory=/var/lib/humio-log-collector
User=humio-log-collector
Group=humio-log-collector
AmbientCapabilities=CAP_DAC_READ_SEARCH

[Install]
WantedBy=multi-user.target

Start Collector

Now that our configuration is in place and the service has the right capabilities, it’s time to start Humio Log Collector!

$ sudo systemctl start humio-log-collector.service

To have the service start on boot, you can also enable it:

$ sudo systemctl enable humio-log-collector.service

As a quick health check, we use the systemctl status command to check the log collection service.

$ sudo systemctl status humio-log-collector.service                                                                                                       

● humio-log-collector.service - Humio Log Collector
Loaded: loaded (/usr/lib/systemd/system/humio-log-collector.service;
disabled; vendor preset: disabled)
Active: active (running) since Sun 2022-07-31 01:45:59 UTC; 13min ago
Main PID: 8845 (humio-log-colle)
Tasks: 7 (limit: 19155)
Memory: 18.2M
CPU: 498ms
CGroup: /system.slice/humio-log-collector.service
└─8845 /usr/bin/humio-log-collector -cfg
/etc/humio-log-collector/config.yamlJul 31 01:45:59 fedora systemd[1]: Started humio-log-collector.service - Humio Log Collector.

Check repository for data

Now that we can see the service running successfully, it’s time to check our repository for logs. We navigate to the repository we created in step 1. We can see a list of events that were ingested.

In addition to the number of events, we also see a list of events that can be queried immediately. For example, we can run a search to see commands that landed in the audit log:

Some of these commands will look familiar! Falcon LogScale ingested all of the audit events related to the commands we used throughout this tutorial, including raising permissions to install the log collector and modifying the log collector configuration.

Wrap

Now that we’ve created a repository and successfully ingested the logs with Humio Log Collector, the next steps depend on your next goal. Some good ideas are to check the options for Configuring the Humio Log Collector and details about data ingest. If you are already happy with the setup, a good next step might be to check how query data in Falcon LogScale.

Happy journaling!

Additional Resources

[ad_2]
Source link