JOB DESCRIPTION
We are looking for a Data Operations Engineer to join our team. In this role, you will be responsible for managing, optimizing, and scaling our data processing and storage infrastructure. You will work closely with our data science team to ensure data is easily accessible and properly stored. You will also collaborate with our software development team to identify and address critical data-related issues as they arise.
RESPONSIBILITIES
- Develop and maintain data processing pipelines using tools such as Spark, and Hadoop, Hive
- Manage the data storage system, including backups and disaster recovery plans
- Monitor data metrics and KPIs to ensure optimal performance and make suggestions for improvement
- Collaborate with the software development team to incorporate new data sources and improve data integration
- Troubleshoot data issues, such as data quality or missing data, and work with the data analyst team to address them
- Stay up-to-date on new technologies and best practices in data processing and storage