We are seeking an experienced Data Quality Engineer to join our dynamic team. The ideal candidate will be responsible for ensuring the accuracy, consistency, and reliability of our organization's data assets. As a Data Quality Engineer, you will play a critical role in developing and implementing data quality processes, and tools to uphold the integrity of our data throughout its lifecycle. This role requires a deep understanding of data quality principles, quality assurance methods, and the ability to collaborate with cross-functional teams to identify and resolve data quality issues.
Responsibilities:
Data Quality Framework Development:
-
Implement, and maintain a robust data quality framework to assess, monitor, and enhance the quality of data across various systems and platforms.
Data Profiling and Analysis:
-
Conduct thorough data profiling and analysis to identify anomalies, inconsistencies, and inaccuracies in datasets.
Quality Assurance Testing:
-
Develop and execute comprehensive data quality testing strategies and plans to verify the implementation of data pipelines and data validations.
Data Cleansing and Remediation:
-
Implement data cleansing and remediation processes to address identified data quality issues, ensuring data is accurate and compliant with organizational standards.
Collaboration:
-
Collaborate with data engineers, data scientists, and other cross-functional teams to integrate data quality checks into the data pipeline and maintain quality throughout.
Documentation:
-
Create and maintain documentation related to data quality processes, standards, and issue resolution procedures.
Monitoring and Reporting:
-
Establish monitoring mechanisms to proactively identify data quality issues, and generate regular reports on data quality metrics for management review.
Continuous Improvement:
-
Continuously assess and enhance data quality processes and methodologies to adapt to evolving business needs and industry best practices.
Training and Knowledge Sharing:
-
Provide training and guidance to team members on data quality best practices and principles. Facilitate knowledge sharing sessions to promote a culture of data quality awareness.
Requirements:
SQL Expertise:
-
Apply strong SQL skills for querying with extensive join conditions and updating data based on specified conditions.
CI/CD Lifecycle:
-
Contribute to and enhance the continuous integration and continuous delivery (CI/CD) lifecycles.
Testing:
-
Design and implement Unit, integration, and regression tests to ensure the reliability of data pipelines and data validations.
Agile SCRUM Projects:
-
Collaborate effectively within an Agile SCRUM framework, participating in sprint planning, reviews, and retrospectives.
Highly Desired:
Data Management:
-
Utilize expertise in Databricks, PySpark, Spark Structured Streaming, Delta Live Tables, and Delta Sharing for advanced data management.
Expert SQL Knowledge:
-
Demonstrate expert-level knowledge of SQL for complex data querying and manipulation.
Python Development:
-
Utilize your 4+ years of Python experience to develop and maintain data quality scripts, tools, and frameworks.
Streaming Technologies:
-
Apply experience with streaming technologies such as Apache Kafka, Azure EventHubs, and Avro to enhance data processing capabilities.
DevOps Proficiency:
-
Work within a DevOps environment, demonstrating expertise in Linux, GitHub, and Bash scripting.
Education and Experience:
-
Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
-
Proven experience as a Data Quality Engineer or in a similar role.
-
Strong understanding of data quality concepts, methodologies, and best practices.
-
Familiarity with data governance frameworks and practices.
-
Excellent problem-solving and analytical skills.
-
Strong communication and interpersonal skills, with the ability to collaborate effectively with cross-functional teams.
-
Experience with data quality tools and technologies is a plus.
-
Relevant certifications (e.g., Certified Data Management Professional) are advantageous.
If you are a data quality enthusiast with a strong technical background, we invite you to join our team and contribute to the excellence of our data-driven solutions.
