Senior Data Engineer

Spartan Capital • Full-time • Hazlet, New Jersey, United States • $100.90k - $155.24k / year • 18h ago

Spartan capital provides top world leading data-driven Finance solutions for Small and Medium size businesses -SMB’s. Spartan Capital is a dynamic, fast-growing FinTech company that is revolutionizing the financial services industry with cutting-edge technology and innovative solutions.

Job Summary
We are seeking a skilled Data Engineer to join our dynamic team. The ideal candidate will be responsible for designing, developing, and maintaining robust data pipelines and architectures that facilitate the collection, storage, and analysis of large datasets. You will work closely with data scientists and analysts to ensure that data is accessible and usable for business intelligence and analytics purposes.

Duties

Design and implement data processing systems using AWS, Azure Data Lake, Hadoop, and Spark.
Develop ETL processes to extract, transform, and load data from various sources into data warehouses.
Collaborate with cross-functional teams to define data requirements and ensure alignment with business objectives.
Utilize SQL, Oracle, Microsoft SQL Server, and other database technologies for data storage and retrieval.
Create and maintain documentation related to database design, ETL processes, and data architecture.
Implement analytics solutions using tools such as Looker for reporting and visualization.
Conduct model training and support machine learning initiatives by providing clean datasets.
Optimize existing data pipelines for performance improvements.
Engage in Agile methodologies to enhance project delivery efficiency.

Skills

Proficiency in programming languages such as Java, Python, and SQL.
Experience with big data technologies including Hadoop, Apache Hive, Spark, and Informatica.
Familiarity with cloud platforms like AWS or Azure Data Lake.
Strong understanding of database design principles and experience with relational databases (Oracle, Microsoft SQL Server).
Knowledge of ETL tools such as Talend or similar frameworks.
Ability to work with linked data concepts and RESTful APIs for data integration.
Competence in shell scripting (Bash/Unix shell) for automation tasks.
Excellent analytical skills with the ability to interpret complex datasets effectively.
Experience in Agile project management methodologies is a plus.
Familiarity with VBA for automation tasks can be advantageous. Join us in leveraging your expertise in data engineering to drive impactful business decisions through effective data management.

Required Experience:

+5 years experience building complex data pipelines and working with both technical and business stakeholders
Experience in at least one primary language (e.g., Java, Scala, Python) and SQL (any variant)
Experience with technologies like BigQuery, Spark, AWS Redshift, Kafka, or Kinesis streaming
Experience creating and maintaining ETL processes
Experience designing, building, and operating a DataLake or Data Warehouse
Experience with DBMS and SQL tuning
Strong fundamentals in big data and machine learning

Responsibilities

Spartan Capital is looking for a Data Engineer for the Credit Modeling team to conceptualize, design and implement improvements to ETL processes and data through independent communication with our data-hungry stakeholders
Design, build, and maintain distributed batch and real-time data pipelines and data models
Facilitate real-life actionable use cases leveraging our data with a user- and product-oriented mindset
Be curious and eager to work across a variety of engineering specialties (i.e., Data Science, and Machine Learning to name a few)
Enforce privacy and security standards by design

PREFERRED SKILLS

Methodologies: SDLC, Agile, Waterfall

Programming Languages: Python, SQL, R

Libraries: TensorFlow, PyTorch, Scikit-learn, Keras, Pandas, NumPy, SciPy

Big Data and ETL Tools: PySpark, Apache Kafka, Hadoop (HDFS, Hive), Apache Airflow, dbt

Databases & Warehousing: MySQL, PostgreSQL, SQL Server, Snowflake, MongoDB

Data Visualization: Tableau, Power BI, Advanced Excel, Matplotlib

Cloud Platforms: AWS (S3, Glue, Redshift), Azure (Data Factory, Databricks, Blob Storage)

Version Control and CI/CD: Git, GitHub, Bitbucket, Azure DevOps, Jenkins

Infrastructure as Code and Containerization Tools: Terraform, Docker, Kubernetes

Soft Skills: Problem-Solving, Effective Communication, Cross-Functional Collaboration, Attention to Detail, Stakeholder

Job Type: Full-time