Data Operation Engineer

JOB DESCRIPTION

We are looking for a Data Operations Engineer to join our team. In this role, you will be responsible for managing, optimizing, and scaling our data processing and storage infrastructure. You will work closely with our data science team to ensure data is easily accessible and properly stored. You will also collaborate with our software development team to identify and address critical data-related issues as they arise.

RESPONSIBILITIES

• Develop and maintain data processing pipelines using tools such as Spark, and Hadoop, Hive
• Manage the data storage system, including backups and disaster recovery plans
• Monitor data metrics and KPIs to ensure optimal performance and make suggestions for improvement
• Collaborate with the software development team to incorporate new data sources and improve data integration
• Troubleshoot data issues, such as data quality or missing data, and work with the data analyst team to address them
• Stay up-to-date on new technologies and best practices in data processing and storage

REQUIREMENTS:

• Bachelor's degree in computer science, engineering, or related field
• 1+ years of experience working with large-scale data processing and storage infrastructure
• Experience with data pipeline and workflow management tools: Airflow etc
• Experience with operating system: Linux, Window Server, etc.
• Strong knowledge of data processing tools such as Spark, Hadoop, hive
• Working knowledge of SQL and NoSQL databases
• Strong problem-solving and troubleshooting skills
• Excellent communication and collaboration skills