About this role
Lead the strategic transformation of Smile Digital Health's Data Ingress Engine using Apache Spark, Databricks, and Medallion Lakehouse Architecture. Own product roadmap for Master Data Management and Data Remediation Engines processing millions of clinical records into Golden Patient Records.
Must have
5+ years of Technical Product Management experience leading complex backend data platforms or big-data ingestion systems
Proven track record productizing solutions powered by Apache Spark and distributed data processing frameworks
Deep familiarity with Medallion Architecture and Delta Lake/Parquet performance patterns
Demonstrated experience optimizing complex nested arrays and hierarchical data structures for analytical query engines
Direct experience productizing Master Data Management, entity resolution, or identity matching tools
Experience with event-driven architectures, Dead-Letter Queues, and data replay/remediation workflows
Ability to comfortably write and execute SQL and Python to analyze raw datasets and define logic rules
Technologies
Responsibilities
Lead product vision and roadmap to decouple core data ingestion modules into highly scalable standalone micro-services
Productize event-driven and streaming ingestion pipelines capable of processing billions of high-velocity records
Establish clear API contracts, event interfaces, and integration specs for external sources feeding raw payloads into the platform
Define requirements for transitioning mass analytics pipelines onto Databricks/Medallion Architecture using Apache Spark
Drive design of optimized Silver and Gold layer datasets with flattened FHIR data into high-performance columnar models
Own product requirements for FHIR-native MDM Engine defining rules for deterministic and probabilistic patient matching at scale
Productize Survivorship Logic to establish rules for creating and maintaining the Golden Record across fragmented data feeds
Build out end-to-end exception lifecycle for data validation failures from isolation in DLQs to structured error reporting
Define requirements for Data Steward interfaces allowing users to review exceptions and perform manual record merges/unmerges
Architect Idempotent Replay Mechanisms to re-ingest corrected payloads without generating duplicate data or corrupting state
Benefits
Amenities
Recruitment process
Company ranked #19 on Deloitte's Technology Fast 50 Ranking for 2024
FHIR-based data liberation platform used in over 20 countries
Position is a replacement role supporting continued growth and operational excellence
AI may be used in portions of recruitment and selection process such as resume screening
All hiring decisions are made by qualified human decision-makers
Company values diversity, equity, and inclusion with focus on respect and belonging
Candidates encouraged to inform company if accommodations are needed during interviews or while working
Similar offers
Staff Backend Engineer - Databases Tempo
186,368 – 223,642 CAD
/ year
Senior Solutions Architect - AI Infrastructure
184,000 – 356,500 USD
/ year
Research Scientist, Generalist Embodied Agent Research - PhD New College Grad 20
168,000 – 304,750 USD
/ year
AI and FSI Developer Technology Engineer - New College Grad 2026
124,000 – 195,500 USD
/ year
Sr. Security Engineer I - Enterprise Security
135,482 – 227,700 USD
/ year