Senior Data Engineer – Databricks Platform
| Company | Live Nation Entertainment |
|---|---|
| Location | Pennsylvania, USA, California, USA, New York, NY, USA, Massachusetts, USA, Michigan, USA |
| Salary | $128000 – $160000 |
| Type | Full-Time |
| Degrees | |
| Experience Level | Senior, Expert or higher |
Requirements
- Good understanding of Data Lakes, Data Warehouses is MUST to have
- Software development, coding expertise in the Data engineering field using any of the Python/PySpark is MUST to have
- Hands on experience using version control systems such as Git and CI/CD workflows and practice
- Decent expertise using ANSI SQL and Spark SQL is key to have
- Workflow automation, orchestration using Airflow or equivalent tools stack
- Hands on working knowledge on at least in one of these: Databricks, Hadoop and related stacks
- Working experience in at least one of cloud services from any of the Amazon AWS, Google GCP or Microsoft Azure, preferably AWS
- Working experience with streaming (Kafka) and batch based data sources
- Working experience with diverse data sources and data formats (xml, json, yaml, parquet, avro, delta) and respective use cases
- Agile development methodologies using the Atlassian suite: Jira, Confluence
Responsibilities
- Contribute to the enhancements and continuous improvement of performance and reliability of the existing Data Platform Services to meet the requirements from Data Engineering/Product team
- Design and build self servicing onboarding and automation capabilities in Core Data Platform services in a scalable, reliable and secure way
- Design and build platform capabilities to support onboarding and operation effectiveness
- Design and develop scalable data platform and integration solutions for various data sources leveraging Databricks Unified Platform
- Ensure data quality and integrity across all data systems
- Ensure data security and compliance with relevant regulations
- Implement best practices for data management and governance
- Develop and maintain documentation for data systems and processes
- Monitor and troubleshoot data platform issues
- Participate in on-call rotations/Pagerduty for data platform support
Preferred Qualifications
- Any visualization experience is an advantage so you can bring clarity to prod incidents by using them is a nice to have
- An excellent understanding of the nuances that add complexities around time zones, geo, various data formats, data types across different storage systems is nice to have