Skip to content

deepakray4all/gc_de_prjct2

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GCP Data Engineering and Analytics Project

This project demonstrates a streamlined ELT data pipeline, showcasing a seamless flow of data from a web application to advanced analytics. The pipeline integrates multiple Google Cloud Platform (GCP) services: the WebApp collects and ingests data, which is stored in Google Cloud Storage (GCS). A Cloud Function triggers automated transformations, loading the processed data into BigQuery for scalable, efficient analysis. Finally, Looker Studio visualizes insights, enabling interactive reporting and data-driven decision-making. This architecture highlights the end-to-end data journey, leveraging cloud-native services for a robust, scalable, and efficient solution.

Requirement:

  1. Global Retail Chain needs to consolidate and analyze its sales revenue
  2. The company needs a web application to load the sales data kept in csv from all the geogaphy to a single location
  3. From the single location the company wants to have analytics performed by it's Business Analyst

Solution

  1. A simple Web application to upload the csv files of sales data
  2. The Web Application should store these files in GCS Bucket
  3. When the file is uploaded to GCS bucket one GCP Cloud function event must be triggered
  4. The uploaded file in GCS should be loaded to Biggquery with help cloud run function
  5. Required transformation is performed in BigQuery to get the sales revenue values
  6. The transformed data must be saved in BigQuery View
  7. Using Looker Studio the Dashboardcan be created for the bigquery data

Architecture

image

Screenshots

WebApp for Uploading file

image

GCS Bucket with files

image

Cloud Function

image image

image

BigQuery Data set ,Tables and Views

image image

Looker Studio Dashboard

image

About

This project is for extracting data from web app to GCS and from their to BigQuery using Cloud Function

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors