Lead Data Engineer

If you like providing efficient, effective solutions to complex challenges, join us at Acuity! Be at the heart of helping achieve extraordinary things with software and data for our customers!

We are looking for an experienced Lead Data Engineer with 7+ years' commercial experience to lead the design, delivery, and continuous evolution of our modern data platform built on Microsoft technologies.

This is a hands-on leadership role, combining technical expertise with people management, where you'll lead a team of Data Engineers while delivering scalable, secure, and high-performing data solutions across Databricks, Microsoft Fabric, Azure Synapse, and Azure.

You'll work closely with architects, stakeholders, and delivery teams to define technical direction, establish engineering best practices, and ensure successful project delivery. This is an opportunity to play a key role in shaping our Data Engineering capability while mentoring and developing a high-performing engineering team.

• Lead and mentor a team of Data Engineers, supporting their technical development and delivery.
• Act as the technical SME for Data Engineering, providing guidance and support to other engineers.
• Lead technical delivery across projects, defining solutions and implementation approaches.
• Design, build, and optimise scalable data platforms using Microsoft Fabric, Databricks, Azure Synapse, and Azure technologies.
• Develop and maintain robust data pipelines using PySpark, Spark SQL, and Azure Data Factory, Synapse, or Fabric Pipelines.
• Scope technical tasks and provide accurate estimates for delivery.
• Review code and establish engineering standards and best practices across the team.
• Build and optimise SQL solutions, including complex queries, stored procedures, and database security.
• Manage Azure DevOps Boards, Repositories, and CI/CD Pipelines throughout the development lifecycle.
• Collaborate with Solution Architects, Product Owners, and stakeholders to deliver high-quality data solutions.
• Drive Agile delivery, sprint planning, technical reviews, and continuous improvement initiatives.

• Experience leading and mentoring Data Engineering teams.
• Strong stakeholder management and communication skills.
• Experience leading technical projects and Agile delivery teams.
• Ability to scope technical work and define implementation approaches.
• Ownership of code quality, code reviews, and engineering best practices.

• Strong commercial experience with Microsoft Fabric, Databricks, or Azure Synapse.
• Expert knowledge of PySpark and Spark SQL, with the ability to act as a technical SME for the wider engineering team.
• Strong SQL skills, including query optimisation, stored procedures, and database security.
• Experience building and managing ETL/ELT pipelines.
• Pipeline orchestration using Azure Data Factory, Synapse Pipelines, or Microsoft Fabric.
• Experience developing scalable cloud-based data platforms.

• Azure DevOps (Boards, Repositories, and CI/CD Pipelines).
• Feature branching, deployment, and release management.
• CI/CD best practices for modern data platforms.

• Experience with Graph Databases (Neo4j).
• Experience with Snowflake.
• Experience using AI-assisted development tools such as Claude and GitHub Copilot.
• Experience with Azure DevOps Pipelines.
• Experience working across Microsoft data platforms including Fabric, Databricks, and Synapse.

• Lead and grow a talented team of Data Engineers.
• Play a key role in shaping our Nearshore Data Engineering capability.
• Work across modern Microsoft technologies including Fabric, Databricks, Synapse, and Azure.
• Blend hands-on technical leadership with people management.
• Collaborate on high-impact enterprise data transformation projects with leading clients.