Memory hook: Trust the dataset only when freshness, correctness, access and recovery have evidence.
Must remember
- Define completeness, uniqueness, validity, consistency, freshness and reconciliation rules. Compare source/sink counts and totals, account for legitimate late data and quarantine bad records with a reason. Glue Data Quality can evaluate supported rules; passing a weak rule set is weak assurance.
- Monitor job failures, duration, backlog, event age, lag, throughput, rejected rows and query cost. Alert on missing scheduled data as well as explicit failures. Use structured logs and lineage to trace a bad dashboard value back through transformations.
- Lake Formation governs supported lake access with table/column/row controls and tag-based permissions. IAM, S3, KMS and Lake Formation controls must align; a catalog permission alone may not establish every required data path. Cross-account sharing needs recipient-side configuration and grants.
- Encrypt data in transit/at rest, isolate network access, scope roles and avoid secrets in job arguments/logs. Mask or tokenise sensitive fields where appropriate; encryption is not anonymisation. Keep raw, curated and serving layers deliberately separated.
- Classify data, record consent/licensing and provenance, define residency and retention, and propagate deletion requirements to derived datasets where required. CloudTrail and relevant service logs support audit; scope data events and retention to the evidence requirement.
- Recovery uses versioned code/configuration, source replay or snapshots, reproducible transformations and tested output reconciliation. Define RPO/RTO, reprocess a bounded range and prevent a backfill from racing live ingestion into inconsistent output.
- Optimise after measuring: appropriate file size, partitioning, compression, worker type/count, query patterns and storage lifecycle. A cheaper worker that repeatedly spills or retries can raise total cost.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| A successful job delivers no new rows | Alert on freshness/reconciliation, not just exit status. |
| Malformed records block a daily batch | Quarantine with traceability and an explicit quality threshold. |
| Backfill several months while live ingestion continues | Isolate/reconcile writes using deterministic keys and commit rules. |
Traps
- Encryption does not remove personal-data classification.
- A green job can publish logically wrong results.
- Broad catalog grants can expose more data than intended.
Active recall
1. What should a quality rule specify besides a threshold?
The business meaning, allowed exceptions, failure action and evidence recorded.
2. How do you investigate one wrong metric?
Trace its source records, transformation versions, joins and aggregation assumptions through lineage.
3. Why test a backfill on a bounded interval first?
To verify duplicate handling, schema compatibility, cost and reconciliation before expanding impact.
4. Which permissions can affect an encrypted lake query?
IAM, data-store/resource policies, Lake Formation grants and KMS permissions as applicable.
5. Does a zero-error log prove data arrived on time?
No. Measure expected arrivals and freshness explicitly.