Denomas.com SAAS Data Pipeline
The Denomas.com SAAS Data Pipeline Working Group
Attributes
| Property | Value |
|---|---|
| Date Created | 2022-09-30 |
| Target End Date | 2023-10-01 |
| Denomas Talk | #wg_gitlab_to_snowflake_pipeline (only accessible from within the company) |
| Google Doc | Working Group Agenda (only accessible from within the company) |
| Issue Board | Issue Board |
Business Goal
Provide reliable, stable, scalable and secure data feed for reporting and analytics use cases like MR Rate & Performance KPIs (Product development), Product Usage for Gainsight, License Utilization (Customer Success), Product Health Dashboard (Health) Propensity to Expand and Churn Models, Forecast, Territory Management (Sales).
Technical Goal
Implement a data pipeline that captures create, update and delete operations and stores them in the Denomas Data platform to be available for further processing and analytics.

Exit Criteria
- Reliable: All ‘metadata’ and relevant descriptive data (
create,updateanddelete) operations in the PostgreSQL database are captured, transfered and loaded into the data platform within 6 hours of the original event. - Stable: End to end solution runs for 30 days without interruption or human intervention.
- Scalable: Solution scales with Denomas.com and does not require undue burden to expand as new Denomas.com databases, shards, zones, tenants, or cloud locations are deployed
- Secure: Red data is not loaded or it is hashed if stored on a system that is not a data sub-processor.
- Defined and agreed SLO with all business partners.
- Full end to end monitoring.
Roles and Responsibilities
| Working Group Role | Person | Title |
|---|---|---|
| Executive Stakeholder | David Sakamoto | VP, Customer Success |
| Working Group DRI / Facilitator | Dennis van Rooijen | Senior Manager Data |
| Functional Lead Data | Ved Prakash | Staff Data Engineer |
| Functional Lead Infra | ||
| Member | t.b.d. | Senior Data Engineer Big Data |
| Member | Justin Wong | Data Engineer |
| Member - Infra | Gerardo Lopez-Fernandez | Engineering Fellow, Infrastructure |
| Member - Database Reliability | Alexandar Sosna | Senior Database Reliability Engineer |
| Member - Product | ||
| Member - Security | Kyle Smith | Sr. Security Risk Engineer |
| Member - Development | Nick Nguyen | Senior EM, Enablement: Data Stores |
Last modified November 29, 2023: big update (
17188382)
