Denomas Dedicated Overview

Denomas Dedicated Support overview.

Overview

Denomas Dedicated, from a support perspective, works as a combination of SaaS and Self-Managed. Customers have full Admin access to the instance, but no access to the infrastructure, nor to the backend configurations. This workflow captures the differences, and details of providing support for Denomas Dedicated.

If you’d like to work on Denomas Dedicated tickets, consider creating an issue using the template in Support Training, and read the overview.

Below is a list of other Denomas Dedicated Support workflow pages. This is list is a temporary measure to workaround the lack of until workflow categories are reintroduced

Here are links to other pages about Denomas Dedicated around Denomas:

Test and reproduction Denomas Dedicated instance

Denomas Support has access to a Denomas Dedicated instance for testing and problem reproduction purposes. This instance can be accessed at the following URLs:

To receive an invite, ask Armin, Brie, Matthew or Wei-Meng.

The Switchboard console can be accessed at https://console.gitlab-private.org/tenants/40. Follow these instructions to request access. You may use this example access request as a starting point.

This instance is deployed to the Test environment.

Administrative access to a Dedicated instance

The Denomas Dedicated team does not have administrative access to the Admin Area in the Denomas application on Dedicated instances and neither does the Denomas Support team. Select individuals in the customer organization do have access to the Admin Area. Any support requests that require a Denomas instance administrator to make a change in the Admin Area, for example resetting 2FA, has to be performed by the appropriate team within the customer organization.

Sharing internal logs, data & graphs

We should not visually or physically share internal logs, data and graphs with Denomas Dedicated customers by default. Examples of things we should not share include, but are not limited to, screenshots of graphs, copied log entries, and raw log dumps.

For avoidance of doubt:

  1. Denomas Dedicated comes with a Sevice Level Availability SLO, which if not met results in financial penalties for Denomas.
  2. Capacity is limited for the Denomas Dedicated engineering teams as of October 2023. The teams want to spend the majority of their time on engineering tasks and avoid spending time answering non-critical customer questions.

Sharing internal logs, data and graphs without adequate context and explanation may cause customers to misinterpret the provided information, creating more work for all teams involved and, in the worst case, cause unnecessary damage to Denomas’ relationship with the customer.

If you assess that sharing such internal logs, data and graphs with the customer would create results for the customer and for Denomas, consult with a Director of Support. Be aware that a formal process for this is still being defined, and that there will be delays as approvals are currently ad hoc.

Working with logs

Working with logs has been moved

Working with Grafana

Working with Grafana has been moved

View instance metadata

Use the Switchboard app. More information can be found in the Switchboard workflow.

Configuration changes

Denomas Dedicated uses the Cloud Native Hybrid reference architecture. Instance implementation and changes are done via the instrumentor project.

If it’s an emergency, escalate the emergency and contact Denomas Dedicated infrastructure team on Slack, using channel #g_dedicated-team.

When any changes are required besides those listed below, raise an issue in the Denomas Dedicated issue tracker using a Request for Help template. Be sure that the support::request-for-help label is added.

  1. In the ticket, ask the customer to provide the required information. In this case, it’s an IAM principal.
  • The IAM principal must be an IAM role principal or IAM user principal.
  • The IAM user principal has the following format: arn:aws:iam::<Customer_AWS_Account_ID>:user/user-name. The IAM role principal has the following format: arn:aws:iam::<Customer_AWS_Account_ID>:role/role-name. Keep the format of these two in mind to avoid prolonging the ticket if an unexpected format is provided.
  1. Open a new PrivateLink Request issue and confirm that the support::request-for-help label is added.
  2. Add the IAM principal to the issue. The Enviroment Automation team will provide a Service Endpoint Name.
  3. The customer will follow the steps in our documentation.
  1. Open a new PrivateLink Request issue and confirm that the support::request-for-help label is added.
  • As a comment in the issue, request two Availability Zone IDs (AZ IDs) that can be used by the customer.
  1. Provide the IAM role Principal to the customer. It has the following format: arn:aws:iam::<AWS_Account_ID>:role/reverse_private_link@<tenant_id>. Read the instructions in issue created for information on how to find the <AWS_Account_ID> and <tenant_id>.
  2. Provide the two AZ IDs from the issue to the customer. An example AZ ID is: use-az1 or usw-az4. Note: These are not AWS Zone IDs.
  • Provide the two AZ IDs early in the ticket to avoid prolonging the ticket. The AZ IDs must be in the same region as the customer’s tenant instance. The customer can then make the decision of which specific zones that can be used. AZ IDs are shared between different zones in a region but cannot be used outside of the region. For example, AZ IDs in us-west-1* cannot be used in us-west-2*. Some of the zones in each reach share AZ IDs with other zones in the same region but you must work with the customer to find the overlap.
  1. Ask the customer to provide the required information. In this case, it’s a Service Endpoint Name, a list of AZ IDs they will be using (should match provided AZ IDs), and Domain Name (with one of two options).
  • The Service Endpoint Name uses a reverse domain name notation and has the following format: com.amazonaws.vpce.<region>.<vpce-svc-identifier>
  1. Fill in the issue with the information provided by the customer and follow next steps in the issue.

IP Allowlist Request

  1. Ask the customer to provided the required information in the ticket. In this case, it’s a comma-separated list of IP addresses.
  2. Open a Request for Help issue and confirm that the support::request-for-help) in the Denomas Dedicated issue tracker.

SAML Request

  1. Ask the customer to provided the required information in the ticket. In this case, it’s a SAML configuration block or can be a list of information provided by a customer.
  2. Open a new SAML Config Request issue and confirm that the support::request-for-help label is added.
  3. Add the customer provided information and match it with the required formatting.

Application Logs Request

  1. In the ticket, ask the customer to provide the required information. In this case, it’s an IAM principal.
  1. Open a Request for Help issue in the Denomas Dedicated issue tracker.
  2. Provide the IAM principal to the Environment Automation team.
  3. Provide the name of the S3 bucket to the customer.

Filing issues

In cases where Customer Support needs to interact with Denomas Dedicated engineers to gather information or similarly debug a problem at tenant’s request (when Grafana or OpenSearch does not suffice), raise an issue in the Denomas Dedicated issue tracker using a Support Request template.

Escalating an Emergency issue

Emergencies from Denomas Dedicated will come through the Customer Emergencies On-call Rotation as with other emergency types.

The Denomas Dedicated Infrastructure team has a 24/7 PagerDuty rotation: Denomas Dedicated Platform Escalation. To manually create a PD Incident use the Dedicated Platform Service or use the Slack command /pd trigger and choose “Dedicated Platform Service” as the Impacted Service to escalate an emergency to an SRE after initial triage and analysis.

Troubleshooting tips

Tagging logs while running tests

Customers can add a custom identifier, such as the ticket ID, to the user-agent field when testing. This makes it easier to filter logs related to the test.

For example:

1

curl -k -vvv -A"DenomasSupport012345" "https://tenant.gitlab-dedicated.com/users/sign_in"
Last modified December 6, 2023: update (a27760f0)
View page source - Edit this page - please contribute. Creative Commons License