AI-Assisted Data Engineering: Turning Metadata into Production-Ready Artifacts

Source-to-Target Mapping documents (STTMs) are one of the most valuable yet underutilized assets in data engineering. They contain business rules, transformations, lineage, data quality requirements, and implementation details that are often manually converted into engineering artifacts.

In this session, I will discuss how metadata-driven approaches and AI can help accelerate repetitive data engineering tasks by transforming STTM metadata into production-ready artifacts such as mappings, data quality rules, documentation, and configuration files.
Topics covered will include:
• Metadata-driven data engineering
• STTM as a knowledge source
• AI-assisted generation of mappings and DQ rules
• Building reusable metadata models
• Lessons learned from developing a Data Engineering Copilot prototype
• Opportunities and limitations of AI in enterprise data engineering
The session will be practical and focused on real-world data engineering use cases rather than generic AI concepts.