## 1. Quick Start Guide **Type: checklist** - [ ] Provision data warehouse infrastructure (e.g., Snowflake, BigQuery, Redshift). - [ ] Set up a data orchestration tool (e.g., Apache Airflow, Dagster, Prefect). - [ ] Configure your ELT/ETL data integration platform (e.g., Fivetran, Airbyte) for ingestion. - [ ] Initialize a data transformation project (e.g., dbt) and connect it to your warehouse. - [ ] Write initial SQL models to structure and clean raw data into base tables. - [ ] Implement data quality tests and continuous integration (CI) pipelines. - [ ] Schedule and deploy pipelines to production for downstream analytics and ML use. ## 2. CLI Commands & Shortcuts **Type: quick-ref** - dbt run: Execute all models in your dbt project - dbt test: Run data quality tests defined in your schema.yml - dbt build: Run and test all models, seeds, and snapshots - dbt compile: Compile dbt project code to raw SQL - dbt seed: Load CSV files into your data warehouse - dbt docs generate: Generate project documentation and DAG lineage - airflow db init: Initialize the Airflow metadata database - airflow webserver: Start the Airflow web UI - airflow scheduler: Start the Airflow scheduler process - airflow dags list: List all available DAGs in your Airflow environment - airflow tasks test : Test a specific Airflow task without dependencies - spark-submit : Submit a PySpark job to a Spark cluster - pyspark: Launch the interactive PySpark shell - aws s3 ls: List S3 buckets or objects (AWS CLI) - aws s3 cp : Copy files to/from Amazon S3 - bq query --use_legacy_sql=false 'SELECT *': Run a standard SQL query in BigQuery CLI - bq load : Load data into a BigQuery table - snowsql -a -u : Connect to Snowflake via CLI - docker-compose up -d: Start your local data engineering stack in detached mode - git checkout -b : Create a new Git branch for pipeline features - ⌘ + Enter: Execute the current SQL query in most Data IDEs (e.g., DataGrip, DBeaver) - ⌘ + /: Toggle line comments in SQL/Python IDEs - ⌘ + Shift + F: Search globally across your data pipeline repository ## 3. Best Practices **Type: tips** - Always implement idempotent data pipelines so rerunning them doesn't create duplicate data. - Adopt ELT over ETL when possible to leverage the scalable compute of modern cloud data warehouses. - Version control all data models, orchestration scripts, and infrastructure configurations. - Separate your environment into Dev, Staging, and Production to prevent breaking downstream reports. - Implement strict naming conventions for tables (e.g., stg_*, int_*, fct_*, dim_*). - Write data quality tests for primary keys (unique, not null) and relational integrity. - Use incremental loading for massive datasets to reduce compute costs and execution time. - Secure sensitive data by masking PII and managing role-based access control (RBAC). - Monitor pipeline execution times and set up automated alerts for failed jobs or data SLAs. - Document your data models comprehensively using metadata and data dictionaries. ## 4. Common Use Cases **Type: tips** - Migrating legacy on-premise databases to a cloud data warehouse like Snowflake or BigQuery. - Building a real-time streaming pipeline using Apache Kafka and Spark Structured Streaming. - Consolidating third-party API data (e.g., Salesforce, Zendesk) into a central data lake for analytics. - Creating a Star Schema (fact and dimension tables) to optimize BI dashboard performance. - Preparing and flattening heavily nested JSON datasets for machine learning feature engineering. - Automating regulatory compliance reports by scheduling daily robust ETL pipelines. - Reverse ETL syncing transformed customer metrics from the warehouse back to marketing tools. ## 5. Troubleshooting **Type: troubleshooting** - Out of Memory (OOM) Errors in Spark: Increase the executor memory or repartition your data to handle data skew. - Airflow DAG not appearing in UI: Check the scheduler logs for Python syntax errors or ensure the DAG is in the correct directory. - dbt models failing on unique tests: Investigate upstream data sources for duplicates and add necessary grouping or deduplication logic. - Slow data warehouse queries: Check for full table scans, missing clustering/partition keys, or unoptimized joins. - Connection timeout to database: Verify network security groups, firewall rules, and ensure database credentials are correct. - Data discrepancies between source and target: Review timezone conversions, character encoding issues, or dropped records during extraction.


About
Pricing
Rating
Ease of Use
Key Features
Link

Enterprise Data & Solutions Architecture Toolkit

Report an issue

Report an issue

What's Included

Data Engineer Stack Cheat Sheet

Enterprise Solutions Architecture Pre-Sales Workflow

Intelligent ETL & Machine Learning Pipeline Workflow

Solutions Architect Stack Cheat Sheet