Senior Data Engineer I

Senior Data Engineer I

Requirements

• 4+ years’ experience performing ETL (extraction, transform, loading) with overall 8+ years of Software Engineering experience.
• BE/BS in computer science or equivalent practical experience
• Expertise with the following programming language: SQL, Python, Scala
• Experience working with data technologies that power analytics – Spark, Databricks
• Experience with any cloud solutions (AWS, GCP, Azure)
• NoSql and Document Oriented models (familiar with)
• Data quality testing (like Deequ)
• Experience in Devops (Jenkins / CloudFormation templates/ Azure pipelines)
• Visualization / dashboard tools like Tableau, PowerBI, and other reporting tools
• Information Retrieval (search) esp. on Elastic or Solr
• Familiar with core text analytics tasks like classification, clustering, LDA, NER, and Relationship Extraction.
• Knowledge Bases, Taxonomies, Graph Databases, knowledge of Open Data sets like Wikidata, Yago, DBPedia, etc.
• Graph Databases and Graph Algorithms (shortest path, connected components, message passing, PageRank, triangle counting, etc.)

Responsibilities

• Perform daily data loads ensuring recurring updates are logged and tracked.
• Produce code that is efficient, repeatable, without defects, and adherent to best practices such as naming conventions, encapsulation, etc.
• Write and review portions of detailed specifications for the development of data components.
• Complete complex data engineering bug fixes and issues, researching and identifying root causes as appropriate.
• Works with more senior members of the team to identify areas where it is an advantage to work with other teams to improve overall quality, and, with peers or others, implement initiatives improving capabilities and efficiency.
• Takes on developmental assignments to train entry-level data engineers as directed by department management, ensuring they are knowledgeable in critical aspects of their roles.
• Design and work with complex data models.
• Begins to Mentor junior data engineers on methodologies and optimization techniques.
• Design and build scalable data ingestion and enrichment pipelines (machine learning inference) infrastructure that powers batch and real-time data processing of billions of records.
• Automate and handle life cycle of the systems and platforms that process our data.
• Provide visibility into the health of our data platform (comprehensive view of data flow, resources usage, data lineage, etc.).
• Mentor teammates on algorithms, data structures, design patterns.
• Evolve maturity of our data quality and monitoring systems.
• Implement development processes, coding best practices, and code reviews.

Preferred Qualifications

No preferred qualifications provided.