Skip to content
Senior Data Engineer
| Company | Mastercard |
|---|
| Location | O’Fallon, MO, USA |
|---|
| Salary | $115000 – $184000 |
|---|
| Type | Full-Time |
|---|
| Degrees | |
|---|
| Experience Level | Senior |
|---|
Requirements
- Excellent problem-solving skills with good understanding of data engineering concepts
- Proficient in Apache Spark with Python and related technologies
- Strong knowledge of SQL and performance tuning
- Experience in Big Data technologies like Hadoop and Oracle Exadata
- Solid knowledge of Linux environments and proficiency with bash scripting
- Effective verbal and written communication skills
Responsibilities
- Build and maintain big data technologies, environments, and applications, seeking opportunities for improvements and efficiencies
- Performing ETL (Extract, Transform, Load) processes based on business requirements using Apache Spark and data ingestion from Apache Kafka
- Work with various data platforms including Apache Hadoop, Apache Ozone, AWS S3, Delta Lake, Apache Iceberg
- Utilizing orchestration tools like Apache NiFi for managing and scheduling data flows efficiently
- Writes performant SQL statement to analyze data with Hive/Impala/Oracle
- Full application development lifecycle (SDLC) from design to deployment
- Working with multiple stakeholders across teams to fulfill ad-hoc investigations, including large-scale data extraction, transformation, and analyses
Preferred Qualifications
- Apache Kafka, Apache Spark with Scala
- Orchestration with Apache Nifi, Apache Airflow
- Java development and microservices architecture
- Build tools like Jenkins
- Log analysis and monitoring using Splunk
- Databricks, AWS
- Working with large data sets with terabytes of data
