Analytics & Streaming

TL;DR

• Kinesis Data Streams is for custom real-time processing (custom consumers, per-record logic); Firehose is for managed delivery to fixed destinations.
• Glue has two roles: Data Catalog (metadata layer) and Glue ETL (managed Spark jobs). Data Catalog is foundational for discoverability.
• Athena is pay-per-query, serverless SQL on S3; appropriate for ad-hoc/infrequent queries. Redshift is for high-volume, sustained analytical queries.
• EMR provides deep cluster control (custom libraries, specific tuning); use only when Athena/Glue ETL don’t meet requirements.
• Lake Formation provides column- and row-level access control; S3 bucket policies only provide object/prefix-level granularity.

Core Concepts

Kinesis Data Streams vs. Firehose

Kinesis Data Streams: A stream that application consumers read from independently. Multiple consumers can read the same stream at different paces. Consumer logic is custom (Lambda, KCL application, EC2 code).

Use case: Real-time fraud detection. Each record needs custom scoring logic before writing to S3. Consumer: custom Lambda function.

Firehose: A managed delivery pipeline. Data flows from source to fixed destination (S3, Redshift, OpenSearch). Built-in transformation (format conversion, basic enrichment) but no custom per-record logic.

Use case: Log ingestion. Collect logs, buffer them, write to S3 nightly. Simple transformation (JSON to Parquet) but no custom business logic. Consumer: automatic delivery to S3.

Mistake: Using Firehose for fraud scoring (requires custom logic per record). Firehose’s transformations are limited; custom fraud scoring requires Kinesis Data Streams + Lambda.

Glue: Data Catalog and ETL

Glue Data Catalog: Central metadata registry. Tables, columns, schemas, S3 locations. Athena, EMR, Redshift Spectrum all query the catalog to discover what data exists and where.

Without a catalog: Each team maintains its own documentation of what tables exist. As the data lake grows, discoverability breaks down. A new analyst doesn’t know what tables exist or what they contain.

Glue ETL: Managed, serverless Spark jobs. Schedule jobs to transform/clean/enrich data. Example: ingest raw logs, transform to cleaned, deduplicated format, output to S3 in Parquet.

Distinction: Glue Data Catalog is metadata; Glue ETL is processing. Both are valuable; catalog is essential for any mature data lake.

Athena: Pay-Per-Query SQL

Athena is serverless SQL query engine on S3. No cluster provisioning. Cost: per TB of data scanned.

Good for:

• Ad-hoc exploration (few queries per week).
• Infrequent analytical queries.
• Queries that run rarely (no sustained load).

Cost math: If you run 10 queries per week, each scanning 100 GB, Athena costs $0.005 * 100 GB * 10 = $5/week. Provisioning a Redshift cluster ($1000+/month) would be wasteful.

Trade-off: Athena is slower than Redshift for complex queries. If you need sub-second response times on large datasets, Redshift’s provisioned compute is worth the cost.

Redshift: Data Warehouse

Redshift is a columnar data warehouse optimized for analytical queries on large datasets. Provisioned compute (you pay for reserved capacity).

Good for:

• High query volume (100+ queries/day).
• Complex analytical queries (joins across large tables).
• Low latency required (<1 second).

Cost: Base cost is $0.25/hour for a single DC2 node (roughly $2000/month). Only justified if query volume is high enough to amortize the cluster cost.

Trade-off: You’re paying for reserved capacity even when not querying. Athena’s pay-per-query is more efficient for low-volume, exploratory use.

EMR: Big Data Processing Engine Control

EMR runs Apache Spark, Hive, Presto, or other big data engines on EC2. Requires infrastructure management (cluster sizing, scaling, cost).

When to use:

• Custom processing logic Glue ETL can’t express (e.g., custom ML training).
• Specific Spark/Hive behavior or tuning needed.
• Complex data processing that Athena can’t handle.

When not to use:

• Simple Spark ETL jobs (use Glue ETL; it’s managed).
• SQL queries (use Athena or Redshift).
• General data transformation (Glue ETL is simpler).

Trade-off: EMR provides control but requires infrastructure expertise. Glue ETL is managed; you trade some control for simplicity.

Lake Formation: Fine-Grained Access Control

Lake Formation adds column- and row-level access control on top of data lake tables.

S3 bucket policies: Grant/deny at object/prefix level. Can’t express “user A sees columns B and C, but not column D in the same table.”

Lake Formation: Define permissions like “analyst_team can read columns name, order_total but not payment_card_details in the orders table.” Enforces at query time (Athena, Redshift, etc.).

Use case: A table has public product data and sensitive customer PII. Instead of splitting into separate S3 buckets, use Lake Formation to control column access. Analysts see product data; compliance teams see PII.

Comparison Table

Common Pitfalls

Interview Questions

• Design an ingestion pipeline for clickstream data that needs custom real-time fraud scoring on each event. — Tests whether Kinesis Data Streams + custom consumer is the right choice vs. Firehose (which has limited transformation).

• A data lake has grown to hundreds of tables with no central catalog. What’s the first thing you’d fix? — Tests understanding of Glue Data Catalog as foundational for discoverability and governance.

• A table contains both public product catalogs and sensitive customer PII in different columns. How do you govern access without splitting the table? — Tests whether Lake Formation’s column-level control is the reached-for solution.

• You’re querying S3 data with Athena for occasional exploratory analysis. When would you justify migrating to Redshift? — Tests understanding of cost trade-off: Athena (pay-per-query) vs. Redshift (provisioned capacity).