Memory hook: Separate ingestion, storage, metadata, permissions, computation and visualization so each requirement has a clear owner.
Must remember
Catalog and govern the lake before choosing the dashboard
- S3 holds data-lake objects. Glue Data Catalog holds metadata describing datasets, locations, schemas and partitions; a catalog entry does not contain or transform all of the underlying data.
- Glue crawlers discover supported schemas/partitions; Glue ETL jobs clean, transform and move data. Schema discovery and transformation are separate compute activities with separate costs.
- A known small schema can be declared directly instead of running a crawler. A newly declared column does not populate missing values in existing JSON objects.
- Lake Formation centrally governs lake access, including supported fine-grained permissions over cataloged data. IAM, storage access and key permissions remain relevant; enabling governance does not mean every existing access path is automatically secured. Glue concepts, Lake Formation
Athena and Redshift are different query choices
- Athena is serverless SQL for supported data sources, especially S3. For occasional queries, it avoids keeping a warehouse cluster running purely to wait for work.
- Partitioning allows queries with appropriate predicates to skip irrelevant data. Columnar formats such as Parquet/ORC, compression and selecting only needed columns reduce scanned work.
- Many tiny files add overhead; poor partition choices can also hurt performance. A
LIMITor a post-read filter is not a universal guarantee of low scan cost. - Workgroups isolate settings and controls such as query output location and scan limits. Keep result objects outside the table's source-data prefix so later scans do not mix source JSON with generated result formats.
- Federated queries access supported non-S3 sources through connectors. Some connectors introduce Lambda execution, spill storage and source-system load; “serverless SQL” does not make all dependencies free. Athena data optimization
- Redshift is an analytical warehouse suited to substantial joins, aggregations and recurring BI workloads. Columnar/parallel execution and table distribution/sort design matter; simply adding nodes is not every query optimization.
- Redshift Spectrum queries external S3 data without first loading it into Redshift tables. This can combine warehouse data with a larger lake rather than copying every cold dataset into the warehouse.
- Snapshots and cross-region copies provide recovery options. Storage, encryption/key access and restore time still matter; an available snapshot does not prove the required recovery time.
- Provisioned versus serverless deployment changes operations and billing, not the need to size, govern and monitor the analytical workload. Spectrum
Processing, search and streaming
- EMR runs frameworks such as Spark and Hadoop with supported cluster/serverless execution choices. Choose it when the workload needs that processing ecosystem or detailed framework control; idle cluster capacity can remain costly.
- OpenSearch indexes data for relevance search, aggregations and log analytics. An index is not automatically the authoritative transactional record; ingestion, retention and recovery need design.
- MSK supplies managed Kafka-compatible infrastructure. It fits existing Kafka clients, tooling and stream semantics; it does not by itself execute all application analytics.
- Managed Service for Apache Flink runs stateful streaming computations with windows, event-time handling and checkpoints. It can consume streams and maintain continuous results rather than rerunning isolated batch queries.
- Kinesis Data Streams provides a retained stream for independent consumers; Firehose provides managed buffered delivery. Choose the ingestion/replay and delivery functions separately. See messaging and stream choices.
- The old Kinesis Data Analytics for SQL service is retired. That is not the same product as current Managed Service for Apache Flink. Flink concepts, availability record
Current BI naming and external datasets
- The current exam service list names Amazon Quick. AWS describes Amazon Quick Sight as the continuing BI/visualization capability within Quick; older material calls it Amazon QuickSight. Preserve that name mapping rather than assuming the dashboard capability disappeared.
- For the familiar exam pattern, choose Quick Sight/QuickSight for interactive dashboards, business-user visualization and embedded analytics. SPICE stores imported analytical datasets in memory; refresh behavior determines when new source data appears. Direct query and imported data have different performance/freshness tradeoffs.
- Quick also includes broader research, automation, connected knowledge and application features. Its existence does not turn a dashboard into the database or ETL engine. No subscription is created for these labs. Current Quick terminology
- AWS Data Exchange manages access/entitlements to externally shared datasets and marketplace data products. It fits acquiring or sharing third-party data, not converting file formats or operating a query engine.
- Data grants/subscriptions, dataset revisions and allowed use must be understood before incorporating external data into analytics. Receiving a dataset does not automatically catalog, cleanse or visualize it. Data Exchange
Trace one complete pipeline
A plausible design is Kinesis/MSK → Flink or Firehose → S3 → Glue/Lake Formation → Athena/Redshift → Quick Sight. These are choices, not mandatory boxes: a simple batch dataset may need only S3, a catalog, Athena and a dashboard. Specify latency, duplicate handling, failure recovery, schema evolution and access controls at every handoff.
Choose under exam pressure
| Requirement in the question | Best direction |
|---|---|
| Occasional SQL over S3 files | Athena |
| Repeated warehouse joins and analytical reporting | Evaluate Redshift |
| Query cold S3 data from Redshift | Spectrum |
| Discover schemas or transform data | Glue crawler or ETL, respectively |
| Govern lake permissions centrally | Lake Formation |
| Preserve Kafka ecosystem | MSK |
| Continuous stateful window aggregation | Managed Service for Apache Flink |
| Search relevance over indexed documents | OpenSearch |
| Dashboards and embedded BI | Amazon Quick Sight, formerly QuickSight |
| Acquire/manage external dataset access | Data Exchange |
Traps
- Metadata, data files, query execution and visualization are separate layers.
- “Real time” requires a latency definition; buffered delivery and stateful event processing are not identical.
- A cache/imported BI dataset can be fast but stale until refreshed.
- Adding a schema field is not an ETL transformation, and an S3 bucket policy alone is not a complete lake-governance design.
- Serverless query/processing choices can still incur scans, capacity minimums, storage, request and transfer charges.
Active recall
1. Analysts query one month's records but every query scans the whole lake. What changes address the underlying work?
Use a suitable partition layout and matching predicates, columnar/compressed files and only needed columns. A dashboard refresh schedule or a small result LIMIT does not automatically prevent scanning irrelevant source data.
2. A crawler discovered a new column, but old records still have no value. Why?
The crawler changed metadata, not the underlying records. A data-writing or ETL process must populate or derive the missing values; schema discovery alone cannot invent them.
3. Existing Kafka producers need continuously updated five-minute event-time aggregates. Which two responsibilities should be separated?
MSK can provide Kafka-compatible ingestion, while Flink performs the stateful window computation. A broker or a buffered delivery service alone does not implement the aggregate logic.
4. A team needs a supplier's licensed dataset, then dashboards over it. Data Exchange or Quick Sight?
Both may have a role: Data Exchange governs acquisition/access to external data; Quick Sight visualizes prepared analytical data. Neither automatically supplies every intermediate catalog, transformation or query step.
5. A dashboard is fast but omits source changes made moments ago. What should you inspect before scaling the database?
Determine whether it uses an imported SPICE dataset and check refresh behavior. Stale imported data is a freshness issue; more source-database capacity does not refresh the BI copy.
Terraform anchor: Typed schema inputs and dynamic Glue column blocks describe metadata; keep source data, query results and governance permissions explicitly separated.