Data Ingestion with Python Cookbook: A practical guide to ingesting, monitoring, and identifying errors in the data ingestion process

Key Features

• Harness best practices to create a Python and PySpark data ingestion pipeline
• Seamlessly automate and orchestrate your data pipelines using Apache Airflow
• Build a monitoring framework by integrating the concept of data observability into your pipelines

Book Description

Data Ingestion with Python Cookbook offers a practical approach to designing and implementing data ingestion pipelines. It presents real-world examples with the most widely recognized open source tools on the market to answer commonly asked questions and overcome challenges.

You'll be introduced to designing and working with or without data schemas, as well as creating monitored pipelines with Airflow and data observability principles, all while following industry best practices. The book also addresses challenges associated with reading different data sources and data formats.