Data Engineer
OmegaHires
Contract
On-site
Charlotte, North Carolina, United States
Data Engineering
Role: Data Engineer
Location: Charlotte, NC
Job Description:
We are looking for a Data Engineer to join our team and help build out our Data Mesh platform, focusing on Ingestion as a Service. The ideal candidate will work on Spark-based frameworks, leverage AWS native services, and apply functional programming principles to deliver scalable, cloud-native data solutions. This role will collaborate closely with both business and technology teams to deliver robust, consumer-focused data platforms.
Key Responsibilities:
Location: Charlotte, NC
Job Description:
We are looking for a Data Engineer to join our team and help build out our Data Mesh platform, focusing on Ingestion as a Service. The ideal candidate will work on Spark-based frameworks, leverage AWS native services, and apply functional programming principles to deliver scalable, cloud-native data solutions. This role will collaborate closely with both business and technology teams to deliver robust, consumer-focused data platforms.
Key Responsibilities:
- Design and implement AWS cloud-native solutions for Data Mesh, specifically the Ingestion as a Service module.
- Build and enhance Spark-based frameworks to support scalable and reusable data ingestion pipelines.
- Apply functional programming concepts to develop modular, maintainable microservices for the data mesh domain.
- Develop, build, and test data engineering pipelines using Python and PySpark.
- Collaborate with business and technology stakeholders to deliver modern, consumer-focused data platforms.
- Contribute to the development of technology standards, tools, and processes in support of the overall data strategy.
- Assist with conceptual data modeling and ensure alignment with data domain principles.
- Communicate and apply data management concepts such as Master Data Management, Critical Data Elements, Granularity, and Normalization.
- Operate and manage CI/CD pipelines using Jenkins, CloudFormation, etc.
- Work with AWS Data Services including Glue, Lambda, Step Functions, DynamoDB, RDS (Postgres), Event Triggers, and Lake Formation.