Publication Date

Fall 2024

Document Type

Capstone Project

Degree Name

Master of Science

Department

Computer Science

First Advisor

Steve Shih

Abstract

CT The aviation industry generates a significant amount of data from various sources, such as flight operations, passenger information, and logistics. Effectively analyzing this data is crucial for improving operational efficiency, making accurate forecasts, and supporting data-driven decision-making. This project focuses on designing and implementing an end-to-end data engineering pipeline using Azure cloud services, enabling efficient data processing, transformation, and analysis, with insights delivered through Power BI. The solution aims to address the growing need for aviation data analysis by creating a scalable and automated ETL pipeline that can integrate data from multiple sources. The motivation behind this project stems from the challenges the aviation industry faces in managing and analyzing large-scale data that is often siloed across different systems. Traditional data management infrastructures struggle to handle the vast volume, variety, and velocity of data generated in the aviation sector. These limitations can prevent timely access to actionable insights, affecting operational efficiency and safety. By creating a robust ETL pipeline, this project seeks to break down these data silos, enabling seamless data integration, transformation, and processing to improve decision-making.The proposed solution will leverage Azure Data Factory for extracting and transforming data from various aviation-related sources, Azure Data Lake for scalable storage, and Power BI for real-time analytics and visualization. The data will be structured using a star schema to optimize query performance, allowing for the analysis of key aviation metrics such as flight delays, fuel consumption, and passenger trends. The project will also implement a CI/CD pipeline using Azure DevOps, ensuring that updates and deployments are smooth and continuous. Ultimately, this solution will provide a comprehensive approach to aviation data analysis, enabling real-time insights that can enhance operational efficiency, reduce delays, and improve overall passenger experience.

Download

Share

COinS