M05-06 · AI + Data & Decision Science

Data Infrastructure and Architecture

Teaches students to design, set up, and maintain the data infrastructure that makes analysis possible. Covers the modern data stack (warehouse, ingestion, transformation, orchestration, BI), read replica configuration, pipeline monitoring, event tracking instrumentation, and the architectural tradeoffs between cost, speed, and scale. Students learn to build data infrastructure from zero — the reality for startup data hires and consultants advising small companies.

25 Hours

7 Learning objectives

Create Bloom's ceiling (?)

4 Competencies

What You'll Be Able To Do

Learning Objectives

Objectives

Depth

Diagram the modern data stack components (ingestion, warehouse, transformation, orchestration, BI) and explain how data flows from source systems to analyst-facing dashboards Understand
Evaluate infrastructure options for different organizational contexts: Option A (read replica + dbt + Metabase, $50-200/month) vs. Option B (Fivetran + Snowflake + dbt + Looker, $500-2K/month) based on company size, budget, and data volume Evaluate
Set up a read replica of a production PostgreSQL database and configure it for safe analytical querying without impacting application performance Apply
Configure data pipeline orchestration (Airflow/Dagster) with monitoring, alerting on failures, and automated scheduling for transformation jobs Apply
Design an event tracking instrumentation plan: event names, properties, triggers, naming conventions, and ownership — that prevents the inconsistency problems found in unplanned tracking Create
Analyze pipeline failures and data freshness issues, diagnosing root causes (source system changes, credential expiry, schema drift, resource limits) and implementing fixes Analyze
Implement data ingestion from external APIs (Stripe, Mixpanel, Amplitude) using Fivetran/Airbyte or custom Python scripts, with error handling and incremental loading Apply

Levels: Remember · Understand · Apply · Analyze · Evaluate · Create — highest demands most original thinking.

Core Competencies

What You'll Master

Data Stack Architecture

Understanding how warehouse, ingestion, transformation, orchestration, and BI components fit together and when to add/change each.

Infrastructure Setup

Standing up read replicas, configuring warehouse access, installing and connecting dbt, deploying BI tools from zero.

Pipeline Operations

Monitoring data pipelines, diagnosing failures, managing scheduled runs, ensuring data freshness.

Event Instrumentation

Designing tracking plans, defining event schemas, coordinating with engineering on implementation, auditing existing tracking for quality.

Portfolio Artifact

What You'll Build

Data Stack Architecture Proposal — Student designs a complete data infrastructure for a realistic startup scenario: documents current state (production database, scattered SaaS tools, no warehouse), proposes architecture with component selection rationale (cost/speed/scale tradeoffs), creates an implementation plan (week-by-week), designs an event tracking plan for 10 key user actions, and includes pipeline monitoring configuration. Delivers as a technical proposal document suitable for presenting to a CTO.

Tools You'll Use

Industry Tools, Not Toy Projects

PostgreSQL

Production database and read replica setup for safe analytical querying alongside live applications.

Snowflake / BigQuery

Cloud data warehouse platforms for scalable analytical storage and compute.

dbt

Transformation layer for building modeled, tested data marts from raw warehouse tables.

Fivetran / Airbyte

Data ingestion platforms for syncing data from SaaS tools and APIs into the warehouse.

Airflow / Dagster

Pipeline orchestration tools for scheduling, monitoring, and alerting on data transformation jobs.

Claude

AI assistant for architecture review, troubleshooting pipeline issues, and infrastructure planning.