Data Pipeline for e-commerce analytics #1

Code Example

# dbt model: fact_e-commerce_analytics
{{ config(materialized='incremental') }}

SELECT
  user_id,
  event_date,
  COUNT(*) as event_count
FROM {{ ref('staging_events') }}
{% if is_incremental() %}
WHERE event_date > (SELECT MAX(event_date) FROM {{ this }})
{% endif %}
GROUP BY 1, 2

-- Run: INSERT INTO logs VALUES (...)

Commands & References

• SELECT * FROM users LIMIT 10
• INSERT INTO logs VALUES (...)
• pip install pandas sqlalchemy

Lab Steps

1. Prepare environment with: SELECT * FROM users LIMIT 10
2. Design or modify the data pipeline for the scenario.
3. Validate data quality and document lineage.
4. Propose one optimization for production.

Exercises

• Add one data quality check.
• Implement one incremental loading pattern.
• Write a rollback procedure for this pipeline.