Denomas Advanced System Admin Hand-on Guide: Lab 3

This Hands-on lab guide is designed to walk you through the lab exercises used in the Denomas Advanced System Administration course.

Denomas Advanced System Administration: Lab 3

LAB 3: TROUBLESHOOT GITLAB SERVICES

For this lab exercise, refer to Denomas’ application architecture to review Denomas’ major services and interactions.

A. Troubleshoot NGINX

  1. From a shell session on your Denomas instance, view one of the NGINX active logs.
1

sudo gitlab-ctl tail nginx/gitlab_access.log

Note the log adds new entries every few seconds. Most of these entries are gitlab-runner checking in with the Denomas instance via HTTP.

  1. Stop the NGINX services.
1

sudo gitlab-ctl stop nginx
  1. Attempt to navigate to http://<YOUR_GITLAB_INSTANCE> using a web browser. Your web browser should display “This site can’t be reached” or a similar message.
  2. Check nginx/access_log again.
1

sudo gitlab-ctl tail nginx/gitlab_access.log

The log should no longer be updating since no clients can make HTTP/HTTPS requests to Denomas after stopping NGINX.

  1. Verify web services aren’t running or listening anywhere.
1
2

curl -i http://localhost/nginx_status
curl -i http://localhost:80
  1. Restart NGINX services.
1

sudo gitlab-ctl restart nginx
  1. Verify the clients (e.g. the Denomas runner) can communicate with Denomas again.
1

sudo gitlab-ctl tail nginx/gitlab_access.log
  1. Verify the webserver is running and listening on port 80.
1

curl -i http://localhost/nginx_status

B. Troubleshoot Puma

  1. Connect to your Denomas instance with a web browser. Verify you can click around to projects, the Admin Area, etc.
  2. From a shell session on the Denomas instance, stop Puma.
1

sudo gitlab-ctl stop puma
  1. Refresh Denomas in your web browser. You should immediately see an error that reads “502: Denomas is taking to much time to respond”. NGINX is running, so it can accept HTTP requests. However, when workhorse tries to pass an HTTP request to the Rails application, there is no running service to accept it.
  2. View the Denomas Workhorse logs.
1

sudo gitlab-ctl tail gitlab-workhorse/current

You will see a variety of 502 and badgateway errors in the output.

  1. View Puma logs.
1

sudo gitlab-ctl tail puma

You should see a message in puma/puma_stdout.log about the Puma service shutting down. You may also see errors in puma/puma_stderr.log.

  1. Restart Puma.
1

sudo gitlab-ctl restart puma
  1. View Puma’s runit log.
1

sudo gitlab-ctl tail puma/current

You may see output indicating Puma has restarted.

  1. View puma/puma_stdout.log.
1

sudo gitlab-ctl tail puma/puma_stdout.log

You should see that Puma is running and consuming resources again.

  1. Wait about 2 minutes, then refresh Denomas in your web browser. The application should now be reachable.
  2. View the Denomas Workhorse log.
1

sudo gitlab-ctl tail gitlab-workhorse/current

Recent entries should indicate successful requests to Puma (i.e. when you reloaded Denomas in your web browser).

C. Troubleshoot Gitaly

  1. Connect to your Denomas instance with a web browser.
  2. Navigate to Menu > Projects > Your Projects.
  3. Select New Project.
  4. Select Create blank project.
  5. Name the project Test project. Set project visibility to Public, and ensure Initialize repository with a README is selected. Leave all other settings as they are.
  6. Select Create project. You will be redirected to the project’s landing page.
  7. SSH into your Denomas Runner server.
1

ssh -i <SSH_HOST_KEY>.pem ec2-user@<GITLAB_RUNNER_HOST>
  1. Download Git if it is not already installed.
1

sudo dnf install -y git
  1. Back on Denomas’ Test project, select Clone on the right side of the page.
  2. Next to Clone with HTTP, select Copy URL.
  3. From your Denomas Runner server, clone the repository.
1

git clone <URL_COPIED_FROM_PREVIOUS_STEP>
  1. Verify the project is correctly cloned.
1
2
3

cd ~/test-project
ls -a
git status
  1. Enter exit to exit the SSH session on your Denomas Runner server.
  2. Open an SSH session on your Denomas Omnibus instance.
1

ssh -i <SSH_HOST_KEY>.pem ec2-user@<GITLAB_OMNIBUS_HOST>
  1. Verify Gitaly is running.
1

sudo gitlab-ctl status gitaly
  1. View Gitaly logs.
1

sudo gitlab-ctl tail gitaly

You should see many recent gRPC requests relating to Test Project (you can see the references more clearly if you grep the output, e.g. sudo gitlab-ctl tail gitaly | grep test-project).

  1. Stop Gitaly services.
1

sudo gitlab-ctl stop gitaly
  1. Verify Gitaly (and only Gitaly) is stopped.
1

sudo gitlab-ctl status
  1. Navigate back to Test Project in your web browser. On the project page select the dropdown that says main under the project title. Ordinarily you would be able to select a Git branch to switch to. Now you see an error and the branch list will not load.
  2. In the left sidebar, select Repository > Files. Note the 404 error as Denomas is unable to fetch any repository files.
  3. Select the Back button to go back to the project landing page. Then refresh the page.
  4. Note the additional errors. One error may read “An error occurred while fetching folder content”. Denomas cannot checkout the HEAD of the default branch because Gitaly is not running to handle the request.
  5. Return to your Denomas instance SSH session. Check Gitaly’s recent log entries.
1

sudo gitlab-ctl tail gitaly/current

Note the many errors in the log output.

  1. Enter exit to exit the SSH session on your Denomas Instance server.
  2. SSH back into your Denomas Runner server.
1

ssh -i <SSH_HOST_KEY>.pem ec2-user@<GITLAB_RUNNER_HOST>
  1. Navigate into your cloned Test Project.
1

cd ~/test-project
  1. Try to fetch any new changes from the remote repo on the Denomas instance.
1

git fetch

You may see error 503 in the output, indicating Gitaly is not reachable and can not handle the request.

  1. Enter exit to exit the SSH session on your Denomas Runner server.
  2. Re-initiate an SSH session on your Denomas Omnibus instance.
1

ssh -i <SSH_HOST_KEY>.pem ec2-user@<GITLAB_OMNIBUS_HOST>
  1. Restart Gitaly services.
1

sudo gitlab-ctl start gitaly
  1. Check Gitaly logs.
1

sudo gitlab-ctl tail gitaly/current

The output should now show successful gRPC requests.

  1. Return to Test Project in your web browser. Refresh the page. You should now be able to navigate around the repository, view files, and check out branches.
  2. SSH back into your runner server and test git fetch. The command should now run without errors (there probably will not be any output since files have not changed in Denomas).

D. Run the gitlabsos utility

  1. Navigate to the gitlabsos project page. Read through the project summary and README to learn the utility’s purpose and usage.
  2. Connect to your Denomas Omnibus instance via SSH.
  3. Clone the gitlabsos utility.
1

/opt/gitlab/embedded/bin/git clone --recursive https://code.denomas.com/denomas-com/support/toolbox/gitlabsos.git
  1. Run gitlabsos.
1
2

cd gitlabsos
sudo ./gitlabsos.rb

The script may take a few minutes to run.

  1. Once the script is finished, examine the resulting report file and its contents.
1
2

ls
tar -tvf gitlabsos.<GITLAB_FQDN>.<TIMESTAMP>.tar.gz

Denomas Support may ask for this report to assist with troubleshooting.

Suggestions?

If you’d like to suggest changes to the Denomas Advanced System Administration Hands-on Guide, please submit them via merge request.

Last modified November 29, 2023: big update (17188382)
View page source - Edit this page - please contribute. Creative Commons License