We are looking for a Lead Data Engineer with 7-10 years of experience to join our Hyderabad team. You will build and optimize scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake, working across scrum teams to modernize core insurance back-office systems.

About the Company

NationsBenefits is a forward-thinking company at the forefront of transforming the insurance industry. We develop innovative benefits management solutions that modernize complex back-office systems, creating scalable, secure, and high-performing platforms that streamline operations for our clients. As part of our strategic growth, we are focused on platform modernization — transitioning legacy systems to modern, cloud-native architectures that support the scalability, reliability, and high performance of core back-office functions in the insurance domain. Our Hyderabad team is central to this transformation, driving data engineering excellence to enable real-time insights and operational efficiency.

Role Overview

We are seeking a self-driven Lead Data Engineer with 7-10 years of experience to join our Hyderabad office. In this pivotal role, you will be responsible for building and optimizing scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake. You will work closely with multiple scrum teams to develop data solutions, ensure robust data governance with Unity Catalog, and support both real-time and batch processing. This role demands strong problem-solving skills, deep T-SQL expertise, and hands-on experience with Azure cloud tools. A background in healthcare or insurance domain is a plus, as you will be instrumental in modernizing systems that handle sensitive benefits data.

Key Responsibilities

• Collaborate with different scrum teams to develop all quality database programming requirements for each sprint.
• Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL.
• Build and manage Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing.
• Implement Unity Catalog for centralized data governance, access control, and data lineage tracking.
• Independently analyze, solve, and correct issues in real time, providing end-to-end problem resolution.
• Develop comprehensive unit tests to enable automated testing and continuous integration.
• Apply SOLID development principles to maintain data integrity and cohesiveness across data platforms.
• Support the modernization of legacy systems to cloud-native architectures, ensuring high performance and reliability.

Tech Stack

• Programming & Frameworks: Python (Advanced), PySpark, Scala (preferred), T-SQL
• Data Platforms & Storage: Azure Databricks, Delta Lake, ADLS Gen2, Azure SQL Database, Data Lake, Data Storage
• ETL & Integration: Azure Data Factory (ADF), SSIS
• Data Governance: Unity Catalog
• DevOps & Tools: Microsoft TFS, Visual Studio, Azure DevOps
• Cloud Platform: Microsoft Azure
• Development Practices: SOLID principles, unit testing, automated testing

10+ years of experience as a Lead Data Engineer. Self-driven with minimal supervision. Proven experience with T-SQL programming, Azure Databricks, Spark (PySpark/Scala), Delta Lake, Unity Catalog, ADLS Gen2. Microsoft TFS, Visual Studio, DevOps exposure. Experience with cloud platforms such as Azure. Analytical, problem-solving mindset.

Opportunity to lead modernization of legacy systems to cloud-native architectures. Work with cutting-edge Azure data stack. Collaborative scrum environment. Professional growth in a strategic domain (insurance tech). Competitive compensation.