We are seeking a collaborative Data Engineer II with 6+years of experience to join our Data & GenAI team. In this role, you will design and optimize batch and streaming data pipelines, ensure data quality and governance, and support analytics and AI/ML initiatives using Azure, Synapse, Databricks, and modern CI/CD frameworks. You will work with cross-functional stakeholders, mentor junior engineers, and contribute to best practices—serving as a bridge between early-career data engineering and senior-level leadership.
YOU’VE GOT WHAT IT TAKES IF YOU HAVE:
o Proficiency in Azure Synapse, Azure Data Factory, Databricks, and related tools.
o Hands-on experience with streaming data pipelines in Databricks, including the use of Structured Streaming, Delta Live Tables, and event-driven architectures.
o Familiarity with Unity Catalog for centralized data governance, access control, and metadata management across the Databricks Lakehouse platform.
o Strong programming skills in Python, SQL, or Spark SQL.
o Experience designing and optimizing scalable data lake and medallion architectures.
o Knowledge of distributed systems and data processing frameworks (e.g., Spark).
o Experience with source control tools such as Git and CI/CD pipelines.
o Proven ability to design and implement data ingestion pipelines from API sources, including REST and GraphQL endpoints and handling data in various formats like JSON and XML.
o Familiarity designing and deploying Azure Functions for serverless data processing and event-driven workflows.
- Analytical and Problem-Solving Skills:
o Ability to diagnose and resolve complex data-related issues.
o Strong aptitude for performance tuning and optimization of data pipelines.
- Collaboration and Communication:
o Effective at working with cross-functional teams to translate business requirements into technical solutions.
o Strong interpersonal skills with the ability to communicate complex technical concepts clearly to non-technical stakeholders.
- Leadership and Mentorship:
o Experience mentoring junior team members and fostering a culture of collaboration.
o Ability to take ownership of projects and drive them to successful completion.
o Commitment to staying updated on emerging data engineering technologies and best practices.
- 6+ years of relevant work experience in Data Engineering
- Bachelor’s Degree in computer science or related field
