Data Engineer
Location: Austin, TX
Duration: 6 Months with possible extension
Visa: Only US Citizen and Green Card Holders
About the Role
We are looking for a highly experienced Data Engineer with a strong background in building production-grade data pipelines, working with complex automotive or time-series data formats (especially MF4/MDF4), and deploying large-scale solutions in Databricks.
The ideal candidate is fluent in Python and Data Storage formats, comfortable working at the intersection of data engineering and data analytics. This is a senior technical role requiring deep expertise, independence, and the ability to drive end-to-end delivery of data
________________________________________
Key Responsibilities
• Design and build robust data pipelines in Python to extract, transform, and load data from MF4/MDF4 files(e.g., automotive telemetry, sensor logs).
• Architect scalable ETL/ELT workflows in Databricks, leveraging Delta Lake and cloud-native storage.
• Optimize performance and ensure reliability of pipelines handling large-scale, high-frequency time-series datasets.
• Mentor junior engineers and contribute to technical design reviews, architecture discussions, and code quality.
• Stay ahead of industry trends ,data lake house architecture, and data workflows.
________________________________________
Required Qualifications
• 5+ years of experience in data engineering and software development.
• Advanced proficiency in Python, with experience in performance tuning and large-scale data processing.
• Strong experience with Databricks, Delta Lake, and Spark (PySpark or Scala).
• Demonstrated ability to design and implement high-throughput, fault-tolerant pipelines in production environments.
• Familiarity with cloud platforms (AWS, Azure, or GCP), including data storage, compute, and security best practices.
________________________________________
Preferred Qualifications
• Experience in automotive, IoT, or telemetry-heavy industries.
• Hands-on experience parsing and processing MF4/MDF4 files (ASAM standards).
• Prior exposure to Databricks Unity Catalog, Delta Live Tables.
• Understanding of GitOps and CI/CD in software engineering
Degree – Bachelor Degree (Computer Science degree is great)
Years of experience – at least 5 years
Experience with Python – advanced experience (at least a 5 years)
Build and design data pipeline in Python
Data that is captured from the test fleet vehicle need to gather data and store it in Azure Databricks, use this data for models/analytics
Hands on and has a lot of data knowledge
• 5+ years of experience in data engineering and software development.
• Advanced proficiency in Python, with experience in performance tuning and large-scale data processing.
• Strong experience with Databricks, Delta Lake, and Spark (PySpark or Scala).
• Demonstrated ability to design and implement high-throughput, fault-tolerant pipelines in production environments.
• Familiarity with cloud platforms (AWS, Azure, or GCP), including data storage, compute, and security best practices.
