Lead Data Engineer – Global Security
| Company | Royal Bank of Canada |
|---|---|
| Location | Toronto, ON, Canada |
| Salary | $Not Provided – $Not Provided |
| Type | Full-Time |
| Degrees | Bachelor’s, Master’s |
| Experience Level | Senior |
Requirements
- Bachelor’s or master’s degree in computer science, Data Engineering a, or a related field.
- 6+ years of proven experience in data engineering, delivering business-critical software solutions for large enterprises with a consistent track record of success.
- Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Delta Live Pipelines, Databricks Runtime, Cluster management etc.)
- Proficiency in Azure Cloud Services.
- Solid understanding of Spark and PySpark for big data processing.
- English fluency, verbal and written.
- Knowledge of SCM, Infrastructure-as-code, and CI/CD pipelines.
Responsibilities
- Design, develop, and maintain end-to-end data pipelines in Azure Databricks using Spark (SQL, PySpark) to transform large datasets efficiently.
- Develop and optimize ELT/ELT workflows using Databricks Workflows or Apache Airflow ensuring data integrity, quality, and reliability.
- Design and manage Delta Lake solutions for data versioning, incremental data loads, and efficient data storage.
- Collaborate with cross-functional teams to understand data requirements, create robust data models, and deliver actionable insights.
- Implement Site Reliability Engineering (SRE) practices for data pipelines by building automated monitoring, alerting, and incident managements solution to ensure data reliability, availability, and performance.
- Apply best practices in data governance, ensuring compliance using Unity Catalog for access management and data lineage tracking.
- Monitor, troubleshoot, and optimize Spark jobs for performance, addressing data pipelines bottlenecks and ensuring cost efficiency.
- Implement infrastructure-as-code solutions using Terraform for automated resource provisioning and management.
- Develop and maintain comprehensive documentation for data pipelines, transformations and data models.
- Provide mentorship and technical guidance to junior engineers, fostering a culture of learning and best practices in data engineering.
Preferred Qualifications
- Databricks certifications (e.g., Databricks Certified Data Engineer, Spark Engineer).
- Exposure to Kubernetes, Docker, and Terraform.
- Strong understanding of business intelligence and reporting tools.
- Familiarity with Cyber Security Concepts.