Overview:

Responsibilities:

Data Pipeline Development:
- Design, build, and maintain scalable data pipelines to process large datasets using GCP services such as Dataflow, BigQuery, and Pub/Sub.
Data Integration:
- Collaborate with data scientists and analysts to understand their data requirements and integrate various data sources, including Cloud Spanner for relational data management.
Data Modeling:
- Develop and maintain data models and architecture that support business intelligence initiatives and analytics, leveraging BigQuery and Spanner for optimized performance.
Monitoring and Optimization:
- Implement monitoring and logging of data processes to ensure data quality and optimize performance across the data landscape.
Workflow Orchestration:
- Use Apache Airflow to orchestrate complex data workflows, creating and managing directed acyclic graphs (DAGs) for automating ETL processes.
Data Storage Management:
- Utilize Google Cloud Storage (GCS) for storing raw and processed data, implementing lifecycle management strategies to meet compliance requirements.
Documentation:
- Prepare technical documentation for data systems and processes to facilitate knowledge sharing and maintainability.
Security and Compliance:
- Ensure data security and compliance with relevant regulations and best practices in data governance.

Requirements:

Cloud Technologies: Proficiency in GCP services, including BigQuery, Dataflow, Cloud Spanner, GCS, and Pub/Sub.
Programming Languages: Strong programming skills in languages such as Python, Java, or SQL.
Data Warehousing: Knowledge of data warehousing concepts and experience with data modeling techniques using BigQuery.
ETL Processes: Experience in designing and implementing ETL (Extract, Transform, Load) processes, particularly with Airflow.
Big Data Technologies: Familiarity with big data tools and frameworks like Apache Spark, Hadoop, or similar technologies.
CI/CD: Understanding of Continuous Integration and Continuous Deployment practices for data solutions.
Data Analysis: Skills in analyzing datasets to extract meaningful insights.
Education: Bachelor’s degree in Computer Science, Information Technology, Data Science, or a related field.
Experience: 6-10 years of experience in data engineering or related roles, with a focus on cloud environments and data analysis.
Certifications: GCP certifications, such as Google Cloud Professional Data Engineer, preferred.

Prodapt

Latest Jobs