Semi-Structured or Unstructured Data Extraction

Scenario: You need to extract data from logs, sensor data, or other semi-structured/unstructured sources where the schema may change frequently.

Semi-structured/unstructured data extraction decision path

From General ETL Data Extraction

Key factors

• Data structure: Semi-structured (logs, JSON) or unstructured (sensor data)
• Schema evolution: Schema changes frequently
• Data freshness: Real-time or periodic
• Volume: Moderate to high
• Source system impact: Must be low, as these systems generate data passively

What to discuss

• Schema-on-read: Use tools like S3 or Delta Lake for flexible schema-on-read, allowing schema to evolve without requiring changes to ingestion pipelines.
• Event-driven extraction: If data is unpredictable, discuss event-driven extraction (trigger-based ingestion) instead of scheduled extraction.
• Handling schema drift: Explain how to manage schema drift (e.g., version control, validation rules).

Trade-offs

• Schema flexibility vs. performance: Schema-on-read offers flexibility but can hurt performance. Discuss how you’ll balance this, especially as data volumes grow.
• Data quality vs. schema evolution: Schema drift can impact data quality. Implement validation rules to ensure extraction remains consistent.

What interviewers want to hear

• A thoughtful approach to handling evolving schemas without breaking the pipeline.
• Strong understanding of event-driven extraction in unpredictable, unstructured data environments.
• Plans for long-term scalability as unstructured data grows.