CEIs client is a national leader in Investment Management, and they are adding an Azure Databricks Engineer to their Enterprise Data and Reporting Group. This role will focus on building and maintaining robust data engineering pipelines within our Lakehouse architecture. The ideal candidate will have a strong background in data modeling, pipeline development, and Databricks-specific tools and features.

Hybrid Position working 3 days per week in office.
$60-65 / HR

Key Responsibilities:

Design, develop, and optimize complex data pipelines using Azure Databricks.
Acquire and integrate data from diverse sources including APIs, on-prem databases, and file transfers.
Implement and maintain medallion architecture (bronze, silver, gold zones) with increasing levels of data transformation and modeling.
Apply 3NF and dimensional modeling techniques to curate and transform data.
Utilize Databricks tools such as Delta Live Tables (DLT), Lakeflow, Autoloader, Unity Catalog, and Delta Sharing.
Integrate Databricks with REST APIs using OAuth and connect to various relational data sources.
Forecast and monitor cloud costs associated with Databricks usage.
Collaborate with cross-functional teams to gather requirements and mentor junior team members.
Stay current with new Databricks features and best practices.

Required Skills & Experience:

3–5 years of hands-on experience with Azure Databricks.
7+ years in enterprise data warehousing and data modeling.
Strong proficiency in SQL, Python, and PySpark.
Deep understanding of medallion architecture and data modeling (3NF and dimensional).
Experience with cluster tuning and troubleshooting in Databricks.
Familiarity with Unity Catalog, Delta Live Tables, and Delta Sharing.
Ability to work independently and collaboratively in a hybrid work environment.

About CEI: As a trusted technology partner, CEI delivers solutions that help our customers transform their business and achieve meaningful results. From strategy and custom application development through application management - our technology and digital experience services are tailored to meet each unique need of our customers. Our staffing solutions bring specialized skills to complement our customers' workforce and project requirements.

CEI

Latest Jobs

Data Engineer ( Azure Databricks )