Data Engineer Integration Lead (Streaming & Warehousing) — Retail Wealth
Location: Remote
No H1b, OPT, CPT
Role
The role is a hands-on Data Engineering Lead to build streaming and batch data pipelines powering our digital self‑service roadmap. Design robust ETL/ELT on AWS, Kafka, Databricks, Oracle, and guide the transition to Snowflake. The role blends architecture and delivery—owning data quality, observability, and governance in the Retail Wealth domain.
- Domain Expertise: Deep experience in retail wealth, broker dealers, warehouses, family offices, custodians, trade order management, and digital self-service platforms (e.g., robo advisors).
- Technical Stack: AWS, Kafka, Databricks, Oracle, Snowflake (future), ETL processes.
- Hands-On Leadership: Integration lead must design/develop APIs and coordinate with architects; data lead must be strong in ETL/data integration.
Responsibilities
- Design and build streaming & batch pipelines for ingestion, curation, and consumption (real‑time + micro‑batch).
- Engineer scalable ELT/ETL on Databricks (PySpark/Spark SQL), integrating sources including APEX, custodians, broker dealers, and market/reference data.
- Optimize workloads on AWS (S3, Glue, EMR/Databricks, Lakehouse patterns); manage Oracle sources; drive Snowflake migration strategy and execution.
- Enforce data quality (DQ), lineage, metadata, and governance with best‑practice frameworks and tooling.
- Partner with analytics, product, and integration teams to support dashboards, operational reporting, advanced analytics, and ODS.
- Establish DevSecOps for data (versioned transformations, CI/CD, IaC patterns, secrets mgmt).
- Define and track SLAs/SLOs, cost controls, and performance baselines.
Must-Have Qualifications
- 5+ years (7+ preferred) in data engineering with lead-level ownership delivering production pipelines.
- Retail Wealth expertise: custodians, broker dealers, warehouses/family offices; order/trade and position/transaction data.
- Hands-on with Kafka (topics, partitions, schema/registry), Databricks (PySpark/Spark SQL), AWS data stack, and Oracle sources.
- Strong SQL/performance tuning; ELT/ETL design patterns; batch orchestration (e.g., Airflow/Databricks Jobs).
- Practical data governance: lineage, DQ, PII controls, encryption, RBAC, and regulatory awareness (FINRA/SEC).
- Experience planning/executing Snowflake migrations (data modeling, performance, cost/pricing levers).
Nice to Have
- Familiarity with Apex Fintech data domains and vendor ecosystems (Orion, Envestnet, Pershing/Schwab).
- Knowledge of DTCC/NSCC, Morningstar data, advisory/UMA/SMA billing & commissions.
- Observability for data (Great Expectations/Deequ, Delta Live Tables), cost optimization, and dbt or equivalent.
- Bachelor’s in CS/Engineering/Math/IS (Master’s a plus).
Job Types: Full-time, Contract, Permanent
Pay: $30.00 - $45.00 per hour
Expected hours: 40 per week
Work Location: Remote
