Posted in

Senior Data Engineer

Senior Data Engineer

CompanyAllTrails
LocationSan Francisco, CA, USA
Salary$170000 – $210000
TypeFull-Time
Degrees
Experience LevelSenior

Requirements

  • Minimum of 6 years of experience working in data engineering
  • Expertise both in using SQL and Python for data cleansing, transformation, modeling, pipelining, etc.
  • Proficient in working with other stakeholders and converting requirements into detailed technical specifications; owning and leading projects from inception to completion
  • Proficiency in working with high volume datasets in SQL-based warehouses such as BigQuery
  • Proficiency with parallelized python-based data processing frameworks such as Google Dataflow (Apache Beam), Apache Spark, etc.
  • Experience using ELT tools like Dataform or dbt
  • Professional experience maintaining data systems in GCP and AWS
  • Deep understanding of data modeling, access, storage, caching, replication, and optimization techniques
  • Experienced with orchestrating data pipelines and Kubernetes-based jobs with Apache Airflow
  • Understanding of the software development lifecycle and CI/CD
  • Monitoring and metrics-gathering (e.g. Datadog, NewRelic, Cloudwatch, etc)
  • Willingness to participate in a weekly on-call support rotation – currently the rotation is monthly
  • Proficiency with git and working collaboratively in a shared codebase
  • Excellent documentation skills
  • Self motivation and a deep sense of pride in your work
  • Passion for the outdoors
  • Comfort with ambiguity, and an instinct for moving quickly
  • Humility, empathy and open-mindedness – no egos

Responsibilities

  • Work cross-functionally to ensure data scientists have access to clean, reliable, and secure data, the backbone for new algorithmic product features
  • Build, deploy, and orchestrate large-scale batch and stream data pipelines to transform and move data to/from our data warehouse and other systems
  • Deliver scalable, testable, maintainable, and high-quality code
  • Investigate, test-for, monitor, and alert on inconsistencies in our data, data systems, or processing costs
  • Create tools to improve data and model discoverability and documentation
  • Ensure data collection and storage adheres to GDPR and other privacy and legal compliance requirements
  • Uphold best data-quality standards and practices, promoting such knowledge throughout the organization
  • Deploy and build systems that enable machine learning and artificial intelligence product solutions
  • Mentoring others on best industry practices

Preferred Qualifications

  • Experience working in a multi-cloud environment
  • Experience with GIS, H3, or other mapping technologies
  • Experience with Amplitude
  • Experience with infrastructure-as-code, such as Terraform
  • Experience with machine learning frameworks and platforms such as VertexAI, SageMaker, MLFlow, or related frameworks