Job title: Data Engineer
Location: Charlotte, NC (Hybrid)
Duration: 12+ Months
Tax terms: W2
Job Overview:
- We are seeking a highly skilled Data Engineer with strong expertise in PySpark, ETL development, and Data Warehousing/Business Intelligence (DW/BI) projects. The ideal candidate will have Headspace and Fabric experience and will be responsible for end-to-end data engineering in support of Financial Attribution, SCD, Booking and Referring Agreements, Data Aggregations, and SOR Onboarding.
Required Qualifications:
- Strong hands-on experience with PySpark for data processing and transformation.
- Proficiency in ETL tools such as Informatica, Oracle PL/SQL, and Teradata.
- Experience with enterprise frameworks, UNIX scripting, and job scheduling.
- Solid background in data analysis, defect resolution, and batch processing.
- Strong understanding of Data Warehousing concepts (e.g., star/snowflake schema, slowly changing dimensions (SCD)).
- Experience working with cloud platforms (Azure or GCP) and services such as S3.
- Strong SQL skills for data extraction, transformation, and analysis.
- Experience with version control systems (Git) and CI/CD pipelines.
- Excellent problem-solving and communication skills with the ability to work in a collaborative environment.
- Knowledge of Agile/Scrum development practices is a plus.
- Headspace and Fabric experience strongly preferred.
Job Description:
- Design, develop, and optimize ETL pipelines using PySpark, S3, and Dremio.
- Work on ProfitView Modernization projects requiring PySpark, Python, Dremio, ETL, and financial domain experience.
- Handle large-scale structured and unstructured data from multiple sources.
- Implement data ingestion, transformation, and loading processes into data lakes and warehouses.
- Collaborate with BI developers, data analysts, and business stakeholders to gather and translate requirements into scalable solutions.
- Ensure data quality, integrity, and governance across all data pipelines.
- Monitor and troubleshoot performance issues, ensuring high availability and efficiency.
- Participate in code reviews, testing, and deployment processes following best practices.
- Document technical solutions, data flows, and architecture for ongoing maintenance and knowledge sharing.
