Senior Data Engineer

Requirements

• Five-plus years of experience as a data engineer or relevant experience
• Proven experience in designing and implementing data pipelines
• Experience with end-to-end testing of analytics pipelines
• Expertise in managing and optimising Elasticsearch clusters, including performance tuning and scalability
• Strong proficiency with data extraction, transformation, and loading (ETL) processes
• Familiarity with data modelling and schema design for efficient data storage and retrieval
• Good programming and scripting skills using languages like Python, Scala, or Java
• Knowledge of DevOps and automation practices related to data engineering
• Experience with data engineering in an agile / scrum environment
• Familiarity with ksqlDB / Kafka or other stream processing frameworks
• Familiarity with data lakes and querying them
• Experience with integrating machine learning models into data pipelines

Tech stack

• Kafka / ksqlDB
• Python
• Redis
• Elasticsearch, cluster management and optimisation
• AWS S3
• PostgreSQL
• AWS

Responsibilities

• Designing, implementing, and maintaining robust data pipelines, ensuring the efficient and reliable flow of data across systems
• Developing and maintaining Elasticsearch clusters, fine-tuning them for high performance and scalability
• Collaborating with cross-functional teams to extract, transform, and load (ETL) data into Elasticsearch for advanced analytics and search capabilities
• Troubleshooting data pipeline and Elasticsearch issues, ensuring the integrity and availability of data for analytics and reporting
• Participating in the design and development of data models and schemas to support business requirements
• Continuously monitoring and optimising data pipeline and Elasticsearch performance to meet growing data demands
• Collaborating with data scientists and analysts to enable efficient data access and query performance
• Contributing to the evaluation and implementation of new technologies and tools that enhance data engineering capabilities

Posted on 18 Sep 12:42, Closing date 18 Oct