Skip to content

Senior Data Engineer
Company | AllTrails |
---|
Location | San Francisco, CA, USA |
---|
Salary | $170000 – $210000 |
---|
Type | Full-Time |
---|
Degrees | |
---|
Experience Level | Senior |
---|
Requirements
- Minimum of 6 years of experience working in data engineering
- Expertise both in using SQL and Python for data cleansing, transformation, modeling, pipelining, etc.
- Proficient in working with other stakeholders and converting requirements into detailed technical specifications; owning and leading projects from inception to completion
- Proficiency in working with high volume datasets in SQL-based warehouses such as BigQuery
- Proficiency with parallelized python-based data processing frameworks such as Google Dataflow (Apache Beam), Apache Spark, etc.
- Experience using ELT tools like Dataform or dbt
- Professional experience maintaining data systems in GCP and AWS
- Deep understanding of data modeling, access, storage, caching, replication, and optimization techniques
- Experienced with orchestrating data pipelines and Kubernetes-based jobs with Apache Airflow
- Understanding of the software development lifecycle and CI/CD
- Monitoring and metrics-gathering (e.g. Datadog, NewRelic, Cloudwatch, etc)
- Willingness to participate in a weekly on-call support rotation – currently the rotation is monthly
- Proficiency with git and working collaboratively in a shared codebase
- Excellent documentation skills
- Self motivation and a deep sense of pride in your work
- Passion for the outdoors
- Comfort with ambiguity, and an instinct for moving quickly
- Humility, empathy and open-mindedness – no egos
Responsibilities
- Work cross-functionally to ensure data scientists have access to clean, reliable, and secure data, the backbone for new algorithmic product features
- Build, deploy, and orchestrate large-scale batch and stream data pipelines to transform and move data to/from our data warehouse and other systems
- Deliver scalable, testable, maintainable, and high-quality code
- Investigate, test-for, monitor, and alert on inconsistencies in our data, data systems, or processing costs
- Create tools to improve data and model discoverability and documentation
- Ensure data collection and storage adheres to GDPR and other privacy and legal compliance requirements
- Uphold best data-quality standards and practices, promoting such knowledge throughout the organization
- Deploy and build systems that enable machine learning and artificial intelligence product solutions
- Mentoring others on best industry practices
Preferred Qualifications
- Experience working in a multi-cloud environment
- Experience with GIS, H3, or other mapping technologies
- Experience with Amplitude
- Experience with infrastructure-as-code, such as Terraform
- Experience with machine learning frameworks and platforms such as VertexAI, SageMaker, MLFlow, or related frameworks