Big Data Engineer (Hadoop & Cloud Specialist)

We are looking for a Big Data Engineer (Hadoop & Cloud Specialist) to join our growing team. The ideal candidate will have deep expertise in the Hadoop ecosystem, Spark/PySpark, Python, and Cloud platforms (AWS, Cloudera, etc.). This role will focus on developing and deploying big data applications, providing advanced user support and troubleshooting, and driving platform optimization both on-premise and in the cloud. Beyond migration, the position will play a key role in onboarding, training, and long-term platform success, making it a vital contributor to our data strategy and business growth.

Key Responsibilities

• Provide advanced support and troubleshooting for users working on Hadoop, Spark, and PySpark environments.

• Develop, deploy, and maintain applications and processes on Hadoop clusters.

• Review, optimise, and enhance code and processes to improve performance, efficiency, and cost-effectiveness (especially in cloud environments).

• Support both on-premise and cloud-based big data environments, ensuring stability, scalability, and security.

• Contribute to the migration of big data workloads to the cloud, working closely with the ingestion team.

• Drive post-migration activities including onboarding of new teams, user training, and ongoing platform optimisation.

• Collaborate with cross-functional teams to identify gaps, provide technical guidance, and ensure best practices in big data development and operations.

Required Skills & Experience

• Strong knowledge of the Hadoop ecosystem (HDFS, YARN, Hive, HBase, etc.).

• Proven experience with Apache Spark and PySpark for data processing.

• Proficiency in Python for data engineering and application development.

• Hands-on experience with Cloud platforms (AWS, Cloudera, or equivalent), including:

  • Cloud Networking

  • Cloud Security

  • Cloud resource optimization for cost management

• Experience reviewing and optimizing big data code and processes.

• Ability to design, build, and support scalable big data solutions in both on-prem and cloud environments.

• Strong troubleshooting and problem-solving skills.

Preferred Qualifications

• Experience with data migration projects, particularly moving workloads from on-prem to cloud.

• Exposure to platform onboarding, user enablement, and training.

• Familiarity with performance tuning and cost optimization strategies in big data/cloud ecosystems.

• Developer-oriented background with the ability to contribute to long-term platform success and innovation.