Data Engineer Integration Lead (Streaming & Warehousing) — Retail Wealth

Location: Remote

No H1b, OPT, CPT

Role

The role is a hands-on Data Engineering Lead to build streaming and batch data pipelines powering our digital self‑service roadmap. Design robust ETL/ELT on AWS, Kafka, Databricks, Oracle, and guide the transition to Snowflake. The role blends architecture and delivery—owning data quality, observability, and governance in the Retail Wealth domain.

Domain Expertise: Deep experience in retail wealth, broker dealers, warehouses, family offices, custodians, trade order management, and digital self-service platforms (e.g., robo advisors).
Technical Stack: AWS, Kafka, Databricks, Oracle, Snowflake (future), ETL processes.
Hands-On Leadership: Integration lead must design/develop APIs and coordinate with architects; data lead must be strong in ETL/data integration.

Responsibilities

Design and build streaming & batch pipelines for ingestion, curation, and consumption (real‑time + micro‑batch).
Engineer scalable ELT/ETL on Databricks (PySpark/Spark SQL), integrating sources including APEX, custodians, broker dealers, and market/reference data.
Optimize workloads on AWS (S3, Glue, EMR/Databricks, Lakehouse patterns); manage Oracle sources; drive Snowflake migration strategy and execution.
Enforce data quality (DQ), lineage, metadata, and governance with best‑practice frameworks and tooling.
Partner with analytics, product, and integration teams to support dashboards, operational reporting, advanced analytics, and ODS.
Establish DevSecOps for data (versioned transformations, CI/CD, IaC patterns, secrets mgmt).
Define and track SLAs/SLOs, cost controls, and performance baselines.

Must-Have Qualifications

5+ years (7+ preferred) in data engineering with lead-level ownership delivering production pipelines.
Retail Wealth expertise: custodians, broker dealers, warehouses/family offices; order/trade and position/transaction data.
Hands-on with Kafka (topics, partitions, schema/registry), Databricks (PySpark/Spark SQL), AWS data stack, and Oracle sources.
Strong SQL/performance tuning; ELT/ETL design patterns; batch orchestration (e.g., Airflow/Databricks Jobs).
Practical data governance: lineage, DQ, PII controls, encryption, RBAC, and regulatory awareness (FINRA/SEC).
Experience planning/executing Snowflake migrations (data modeling, performance, cost/pricing levers).

Nice to Have

Familiarity with Apex Fintech data domains and vendor ecosystems (Orion, Envestnet, Pershing/Schwab).
Knowledge of DTCC/NSCC, Morningstar data, advisory/UMA/SMA billing & commissions.
Observability for data (Great Expectations/Deequ, Delta Live Tables), cost optimization, and dbt or equivalent.
Bachelor’s in CS/Engineering/Math/IS (Master’s a plus).

Job Types: Full-time, Contract, Permanent

Pay: $30.00 - $45.00 per hour

Expected hours: 40 per week

Work Location: Remote

Lams Technology LLC

Latest Jobs

Data Engineer