2 Data Team Roles and the Data Lifecycle
In modern organizations, analytics projects are rarely developed by a single individual. As data volume grows and technologies specialize, the need for multidisciplinary teams arises. According to the DAMA-DMBOK data governance framework (DAMA International 2017) and the characterization of the data scientist role by Davenport and Patil (2012), there are four main profiles, each with specific responsibilities, tools, and deliverables.
2.0.1 Comparison of Data Engineering and Data Science Roles
| Role | Definition & Objectives | Primary Responsibilities | Common Tools |
|---|---|---|---|
| Data Engineer | Builds and maintains the physical and logical infrastructure required for data to flow securely and efficiently across the enterprise. | • Designs data pipelines (ETL/ELT). • Manages the Data Warehouse and Data Lake. • Optimizes databases. |
PySpark, SQL, dbt, Apache Airflow, Docker, Snowflake, BigQuery. |
| Data Analyst | Cleans, organizes, and interprets historical data to extract operational insights and support business decision-making. | • Designs reports and interactive dashboards. • Performs basic exploratory data analysis. • Responds to operational data requests (tickets). |
SQL, Tableau, Power BI, Python (Pandas/Polars), Excel, BigQuery. |
| Data Scientist | Uses mathematics, statistics, and Machine Learning algorithms to generate predictions or automate complex data-driven decisions. | • Develops regression, classification, and clustering models. • Conducts statistical validation experiments (A/B testing). • Optimizes hyperparameters. |
Python (Scikit-Learn, PyTorch, XGBoost), R, MLflow, Jupyter Notebooks. |
| Data Architect | Defines the organization’s overarching data architecture, creating the blueprint for data integration across corporate systems. | • Designs the enterprise conceptual database schema. • Establishes dimensional modeling and governance policies. • Aligns technical infrastructure with corporate strategy. |
Data modeling tools (Erwin, dbdiagram.io), Cloud Architectures. |
The dimensional modeling practices employed by the Data Architect, which we will revisit in Part I of the course, follow the classic methodology of Kimball and Ross (2013).
Below is a diagram detailing the logical workflow between different data roles:
Transactional Data Sources (OLTP)
Operational databases, server logs, third-party APIs.
Data Engineer
Builds and automates extraction, transformation, and loading (ETL) pipelines into the data warehouse.
Data Warehouse / Lakehouse
Central repository of consolidated, clean data optimized for multidimensional querying.
Data Analyst
Extracts reports, builds dashboards, and interprets historical data to answer business questions.
Data Scientist
Develops predictive data mining models and validates advanced statistical hypotheses.
2.1 The Data Lifecycle
The data lifecycle describes the sequence of phases through which information passes within an organization. This cycle is fundamental for structuring the development of this course: Part I (Data Warehouses) mainly covers the capture and storage phases, while Part II (Data Mining) focuses on strategic and predictive analysis.
Neither infrastructure management (storage phase) nor applied analytics (exploration and modeling phase) occurs in isolation. They are integrated within a sequential and iterative process for data projects. Before building any model or database, we must go through stages that help us evaluate project viability: business understanding, data collection and inspection, preprocessing, and subsequent evaluation of results to ensure successful production deployment. This structured approach is what sets professional analytical projects apart from academic proofs-of-concept.
The lifecycle consists of six key stages (Google 2022):
- Plan: Determine data requirements to solve the business problem, define necessary sources, and identify the required team.
- Capture: Collect data from internal and external sources (transactional databases, web logs, APIs). In this phase, data is extracted and prepared.
- Manage: Store data in a structured manner within relational databases or analytical data warehouses. Here, security policies, initial cleaning, and data lineage are enforced.
- Analyze: Explore data using statistics and data mining (EDA), and train analytical models to answer key questions or automate processes.
- Archive: Move historical data that is no longer queried frequently to low-cost passive storage systems (e.g., Glacier or Google Cloud Coldline Storage), safeguarding it against accidental modifications.
- Destroy: Permanently delete data once the required retention period has expired, ensuring that no residual or unsecure copies of personally identifiable information (PII) remain under secure erasure policies.
2.2 Critical Phases of Data Management
Regardless of the stage of the lifecycle, two questions cross the entire process: how sensitive is the data being handled and who is permitted to access it. Neither is resolved informally; both are governed through explicit policies that the data team must know before touching a single table.
2.2.1 Data Requiring Special Handling
Not all data presents the same risk level if exposed or misused. Certain categories—known as sensitive data or special categories—require additional collection, storage, and access controls:
- Personally Identifiable Information (PII): Name, address, official ID number, email, or any data allowing a person to be directly or indirectly identified.
- Health Data: Clinical records, diagnoses, or lab results; explicitly protected in Mexico as sensitive personal data under the Federal Law on Protection of Personal Data Held by Private Parties (Congreso de los Estados Unidos Mexicanos 2010).
- Financial and Biometric Data: Credit card numbers, credit history, fingerprints, or facial recognition.
Thus, before capturing any of these data types, the minimization principle is applied: collect only what is strictly necessary to answer the business question, rather than hoarding sensitive information “just in case”, which increases risk without contributing analytical value.
2.2.2 Governance, Usage Policy, and Access Control
The most common way to mitigate these risks is working with a data usage policy: a formal document defining what data exists, who can query it, under what conditions, and for what purpose. In modern enterprises, this policy is typically designed by the Data Architect alongside legal and security teams, and translates operationally into:
- Least Privilege Access Control: Each user or role receives specific access capabilities limited strictly to data essential for their function—a Data Analyst can query aggregated sales by region, but not necessarily individual credit card numbers—implemented via Identity and Access Management (IAM) policies and Role-Based Access Control (RBAC).
- Encryption: Of data at rest and in transit.
- Data Catalog: An updated inventory documenting sensitive fields, data owners (data owners), and applicable retention rules, preventing misinterpretations between teams.
This principle of least privilege—granting each user only essential access, and no more—is now the de facto standard across the industry: it is as relevant for the Data Engineer configuring permissions in Snowflake or BigQuery as it is for the Data Analyst requesting access to a Data Warehouse table.
2.3 Guided Activity: Designing a Lifecycle Plan for TheLook eCommerce
As an introductory practical activity for this course, you will design the initial data plan for the fictitious company TheLook eCommerce, based on requirement requests from the executive board of the Marketing and Sales departments.
2.3.1 Business Scenario
TheLook eCommerce has grown rapidly thanks to its omnichannel offering. However, its current data infrastructure is chaotic. The Marketing team has identified four main business questions they need answered to launch effective campaigns next quarter.
2.3.2 Goal Alignment and Data Planning Exercise
For each business question, we will associate its analytical objective and identify the necessary data that must be captured and managed in our future Data Warehouse.
| # | Business Question (Stakeholder) | Analytical Objective | Data Required to Capture |
|---|---|---|---|
| 1 | “Which products should we promote together on our homepage?” | Discover association rules between product categories co-purchased by users. | • Order history (orders and order_items).• Product ID and clickstream history from browsing sessions. |
| 2 | “How can we optimize inventory levels of women’s dresses across regional warehouses?” | Predict weekly future demand for specific products by geographical location. | • Daily warehouse inventory logs (inventory_items).• Shipping addresses and postal codes of historical purchases. |
| 3 | “What is the long-term cumulative value of customers acquired through paid Google ads?” | Calculate Customer Lifetime Value (CLV) grouped by acquisition channel. | • User tables (users) with registration date and referral channel.• Total historical transaction value linked to each unique user ID. |
| 4 | “How do we detect and prevent fraudulent returns of high-cost merchandise?” | Develop a suspicious transaction classifier prior to return processing. | • Return history by user. • Payment methods and elapsed time between delivery and return request. |