Denomas System Administration Hands-on Guide: Lab 6

Denomas System Administration Hands-on Guide: Lab 6

LAB 6- MANAGE GITLAB LOGS

A. View active logs

The gitlab-ctl command allows you to tail all Denomas log files as well as filter by Denomas service.

1. From a shell session on your Denomas instance, run the following command to view all active Denomas logs.

1

sudo gitlab-ctl tail

Amidst all the output, you should notice the command shows the full file path to each log. Most Denomas logs live in /var/log/gitlab. (Note: You can type CTRL-C to exit the tail command.)

1. You can also view Denomas logs by service. Run the following command to view only NGINX logs (i.e. log files in /var/log/gitlab/nginx).

1

sudo gitlab-ctl tail nginx

You should now see the most recent entries of log files specific to the NGINX web server.

1. Finally, you can drill down to an individual log file.

1

sudo gitlab-ctl tail nginx/gitlab_access.log

B. Set minimum log levels

Admins are able to set minimum log levels for some Denomas services. Note that only some services such as NGINX and Gitaly let admins change the minimum logging level, and even then only for some log files. The log_level for other services, such as Sidekiq and Redis, cannot be changed.

1. Check the current minimum log levels for Denomas services.

1

sudo grep -n -E 'log_level|logging_level' /etc/gitlab/gitlab.rb

1. Note the line number for praefect['logging_level'].
2. Change the minimum log level for Praefect, which is the traffic manager for Gitaly. Replace “1234” with the appropriate line number from the grep output in the previous step.

1
2

sudo sed -i '1234s/warn/error/' /etc/gitlab/gitlab.rb
sudo sed -i '1234s/# //' /etc/gitlab/gitlab.rb

1. Re-run the grep command from Step 1 to verify the line was modified as intended.
2. Reconfigure to apply the changes.

1

sudo gitlab-ctl reconfigure

(Note: since we’re using single node Omnibus, Praefect is not actually in use as a service. Praefect is only enabled if using Gitaly cluster.)

C. Manage log retention

Denomas uses logrotate to manage retention of all logs except those managed by the runit service manager (runit uses a separate service logging daemon called svlogd). Log retention can be configured in /etc/gitlab/gitlab.rb.

1. Examine default logrotate retention settings.

1

sudo grep -n 'logrotate' /etc/gitlab/gitlab.rb

1. Optional: View the default retention settings for the runit-managed logs.

1

sudo grep -n 'svlogd' /etc/gitlab/gitlab.rb

1. It appears logrotate (and svlogd) rotate log files every day, and retain 30 days worth of logs. We can verify this by looking inside the service log directories.

1

sudo ls /var/log/gitlab/puma

Note the gzipped archive files for Puma’s stdout and stderr logs from previous days.

1. Change logrotate’s behavior to rotate log files weekly. As before, modify the line sed edits accordingly using the line number from the grep output.

1
2

sudo sed -i '1234s/daily/weekly/g' /etc/gitlab/gitlab.rb
sudo sed -i '1234s/# //' /etc/gitlab/gitlab.rb

1. Change logrotate’s retention period to 1 year of retained log files. As before, modify the line sed edits accordingly using the line number from the grep output.

1
2

sudo sed -i '1234s/30/52/g' /etc/gitlab/gitlab.rb
sudo sed -i '1234s/# //' /etc/gitlab/gitlab.rb

1. Run the following again to ensure your changes are properly written to gitlab.rb.

1

sudo grep -n 'logrotate' /etc/gitlab/gitlab.rb

1. Reconfigure to apply the changes.

1

sudo gitlab-ctl reconfigure

D. Change log formatting

Many logs are JSON formatted by default. Admins may wish to configure text formatting depending on the log ingestion system used, or for readability.

1. Check the current log formats for Denomas services.

1

sudo grep -n '_format' /etc/gitlab/gitlab.rb

1. Run sudo gitlab-ctl tail gitaly/current to see the current JSON output for Gitaly logging.
2. Change Gitaly’s log format from JSON to text formatting. As before, modify the line sed edits accordingly using the line number from the grep output.

1
2

sudo sed -i '1234s/json/text/' /etc/gitlab/gitlab.rb
sudo sed -i '1234s/#//' /etc/gitlab/gitlab.rb

1. Re-run the grep command from Step 1 to verify the line was modified as intended. The line should now read gitaly['logging_format'] = "text".
2. Reconfigure to apply the change.

1

sudo gitlab-ctl reconfigure

1. Verify the updated formatting.

1

sudo gitlab-ctl tail gitaly/current

You should see the log output is now text formatted instead of JSON formatted.

SUGGESTIONS?

If you’d like to suggest changes to the Denomas System Admin Basics Hands-on Guide, please submit them via merge request.

View page source - Edit this page - please contribute.