Principal Data Engineer-Data as a Service
| Company | Visa |
|---|---|
| Location | San Mateo, CA, USA |
| Salary | $Not Provided – $Not Provided |
| Type | Full-Time |
| Degrees | Bachelor’s, Master’s, MBA, JD, MD |
| Experience Level | Expert or higher |
Requirements
- 12 or more years of work experience with a Bachelor’s Degree or at least 10 years of work experience with an Advanced degree (e.g. Masters/MBA /JD/MD), or a minimum of 5 years of work experience with a PhD.
- Strong hands on proficiency in programming languages like Java, Scala, SQL, and Python.
- Expertise with Apache Spark, Kafka, Hadoop, Hive, Trino, Presto, Apache Airflow and NoSQL databases like HBase and Cassandra.
- Experience in on-prem (Hadoop) and cloud-based data platforms (AWS, Azure, DataBricks) and related data storage (HDFS, S3) and processing tools (Spark).
- Experience in deployment with automated and scalable CI/CD tools, including Jenkins and Maven.
Responsibilities
- Oversee the entire data lifecycle, from data acquisition and ingestion to transformation, storage, and analysis for both streaming and batch data pipelines.
- Lead architecture, design, and development of highly scalable and reliable data engineering solutions.
- Ensure data security, privacy, governance, and compliance with all relevant regulations, and develop and implement auditable policies and procedures.
- Future-proof data architecture for the payment processing pipelines that aligns with the product vision and accelerates innovation and time to market.
- Actively contribute with hands on development to critical projects by developing reusable modules, core frameworks and automation tools.
- Establish engineering best practices for application development, testing, deployment and monitoring.
- Leverage AI/ML technologies in bringing productivity across the SDLC phases.
- Champion the adoption of GenAI and Agentic AI technologies and develop strategies to integrate them into existing data pipeline workflows or develop new ones.
- Provide technology leadership and motivate a high performing team of data engineers through coaching and mentoring and elevate team talent expertise fostering a culture of innovation and continuous learning.
- Collaborate with business partners to convert product requirements into high quality solutions that comply with all non-functional requirements, including security, scalability, availability, and reliability.
- Effectively communicate technical strategy and engineering solutions to leadership and business stakeholder.
Preferred Qualifications
- 15 or more years of experience with a Bachelor’s Degree or 12 years of experience with an Advanced Degree (e.g. Masters, MBA, JD, or MD), PhD with 9+ years of experience.
- Proven track record of building and deploying complex architectures for streaming and batch ETL pipelines using the latest Big Data technologies.
- Experience developing proper metrics instrumentation in software components using Prometheus, Grafana for monitoring, logging, auditing, and security implementation to help facilitate real-time and remote problem solving /performance monitoring is highly preferred.
- Exposure with GenAI or Agentic AI technologies (e.g., Large Language Models, NLP) and RAG-based architecture and vector databases.
- Experience with containerization technologies and orchestration tools, including Docker and Kubernetes preferred.