Design and build scalable data pipelines using Databricks, PySpark, ADF, Logic Apps, and Airflow for a hybrid, pan-India role.

About the Company

Technology Next is a forward-thinking technology solutions provider focused on delivering innovative data engineering and cloud modernization services. With a strong emphasis on leveraging the latest Azure and Databricks technologies, we empower businesses to unlock the full potential of their data assets. Our team is dedicated to building scalable, high-performance data platforms that drive intelligent decision-making and business growth.

Role Overview

We are seeking an experienced Senior Data Engineer to join our growing data engineering team. In this role, you will design, develop, and maintain robust data pipelines and lakehouse architectures on the Azure platform. You will work closely with data scientists, analysts, and business stakeholders to enable ML-ready datasets and support enterprise-wide analytics initiatives. This is a hybrid role offering flexibility to work from anywhere in India, with core hours aligned to 11:00 AM – 9:00 PM IST.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using Databricks, PySpark, Azure Data Factory, Logic Apps, and Apache Airflow.
• Build and manage Airflow DAGs, ADF pipelines, and Logic Apps workflows.
• Implement scheduled, event-driven, and micro-batch data processing workflows.
• Develop high-performance data pipelines leveraging Delta Lake and Lakehouse architecture.
• Integrate data from diverse sources including ADLS Gen2, Blob Storage, APIs, and other Azure services.
• Implement robust monitoring, alerting, retries, and failure-handling mechanisms.
• Optimize pipelines for performance, scalability, and cost-efficiency.
• Implement CI/CD pipelines using GitHub Actions or Azure DevOps.
• Ensure data quality, governance, and security standards are met across all solutions.
• Enable AI/ML and GenAI initiatives by building reliable, ML-ready datasets.
• Collaborate with Data Science, Analytics, and Business teams to understand data requirements and deliver solutions.
• Troubleshoot and resolve pipeline issues in a timely manner.

Tech Stack

• Databricks
• PySpark
• Azure Data Factory (ADF)
• Azure Logic Apps
• Apache Airflow
• Delta Lake / Lakehouse Architecture
• Python
• SQL (Advanced)
• Azure Cloud Services (ADLS Gen2, Blob Storage, etc.)
• GitHub Actions / Azure DevOps (CI/CD)
• MLflow (preferred)
• AI/ML, LLM, GenAI technologies (preferred)

Why Join Us?

• Opportunity to work with cutting-edge data technologies and contribute to AI/ML initiatives.
• Competitive compensation and a flexible hybrid work environment.
• Collaborate with a talented, cross-functional team committed to innovation.
• Immediate or short-notice joiners are preferred, so if you are ready to make an impact, we want to hear from you!

7+ years of Data Engineering experience with strong hands-on Databricks & PySpark, Azure Data Factory (ADF) and Logic Apps. Must have Apache Airflow, Delta Lake/Lakehouse architecture, advanced Python & SQL, ETL/ELT development, and Azure Cloud services. Experience with workflow orchestration, pipeline monitoring & troubleshooting, and CI/CD (GitHub Actions/Azure DevOps).

Competitive salary range of ₹90,000 - ₹110,000 per month. Hybrid work model with flexibility. Exposure to AI/ML and GenAI initiatives. Collaborate with cross-functional teams in a dynamic environment.