Senior Data Engineer

Senior Data Engineer

Requirements

• Hands on experience programming in Python, Scala or Java
• Expertise with RDBMS and Data Warehousing (Strong SQL) with Redshift, Snowflake or similar
• In-depth knowledge and experience with data and information architecture patterns and implementation approaches for Operational Data Stores, Data Warehouses, Data Marts and Data Lakes
• Proficiency in logical/physical data architecture, design and development
• Experience in Data lake / Big data analytics platform implementation either cloud based or on-premise; AWS preferred
• Experience working with high volumes of data; experience in design, implementation and support of highly distributed data applications
• Experience with Development Tools for CI/CD, Unit and Integration testing, Automation and Orchestration E.g. GitHub, Jenkins, Concourse, Airflow, Terraform
• Experience with writing Kafka producers and consumers or experience with AWS Kinesis
• Hands-on experience developing a distributed data processing platform with Big Data technologies like Hadoop, Spark etc
• Excellent analytical and problem-solving skills, often in light of ill-defined issues or conflicting information.
• Adept in the ability to elicit, gather, and manage requirements in an Agile delivery environment
• Excellent communication and presentation skills (verbal, written, presentation) across all levels of the organization. Ability to translate ambiguous concepts into tangible ideas.

Responsibilities

• Work across multiple projects and efforts to orchestrate and deliver cohesive data engineering solutions in partnership with various functional teams at Thrive Market
• Be hands-on and take ownership of the complete cycle of data services, from data ingestion, data processing, and ETL to data delivery for reporting
• Collaborate with other technical teams to deliver data solutions that meet business and technical requirements; define technical requirements and implementation details for the underlying data lake, data warehouse, and data marts
• Identify, troubleshoot, and resolve production data integrity and performance issues
• Collaborate with all areas of data management as lead to ensure patterns, decisions, and tooling are implemented by enterprise standards
• Perform data source gap analysis and create data source/target catalogs and mappings
• Develop a thorough knowledge and understanding of cross system integration, interactions, and relationships to develop an enterprise view of Thrive Market’s future data needs
• Design, coordinate, and execute pilots/prototypes/POC to provide validation on specific scenarios and provide an implementation roadmap
• Recommend/Ensure technical functionality (e.g. scalability, security, performance, data recovery, reliability, etc.) for Data Engineering
• Facilitate workshops to define requirements and develop data solution designs
• Apply enterprise and solution architecture decisions to data architecture frameworks and data models
• Maintain a repository of all data architecture artifacts and procedures
• Collaborate with IT teams, software providers, and business owners to predict and devise data architecture that addresses business needs for collection, aggregation, and interaction with multiple data streams

Preferred Qualifications

• Experience with streaming data ingestion, machine-learning, Apache Spark a plus