Join a leading IT services firm as a Databricks Developer, driving scalable data pipeline solutions in cloud environments.

About the Company

Kumaran Systems has been a pioneer in engineering future-ready transformations since 1992, helping enterprises navigate the transition from legacy systems to AI-led innovations. With over 34 years of experience, the company specializes in re-engineering core applications and facilitating cloud transformations in various sectors, including Banking, Insurance, Telecom, and Automotive. Their mission is to not only upgrade systems but to radically reimagine what's achievable in technology-driven environments.

Role Overview

We are looking for a skilled Databricks Developer to join our team. In this role, you will design, develop, and optimize scalable data pipelines and analytics solutions leveraging Apache Spark, Python (PySpark), and SQL within modern cloud environments. The ideal candidate will have hands-on experience with Databricks, Delta Lake, and cloud data services, taking on the critical responsibility of creating high-performance data-processing workflows that empower data-driven decision-making across the organization.

Key Responsibilities

• Pipeline Development
  Design, develop, and maintain ETL/ELT data pipelines using PySpark and SQL in Databricks notebooks, ensuring reliable and efficient data transformations for large-scale datasets.
• Architecture Design
  Implement data lake and warehouse architectures utilizing Databricks Delta Lake and Delta Live Tables, while managing Medallion Architecture for structured data processing.
• Performance Optimization
  Enhance Spark jobs and queries for improved performance and cost-efficiency while configuring cluster settings and optimizing partitioning strategies.
• Cloud Integration
  Seamlessly integrate Databricks solutions with cloud services, including Azure Data Factory and AWS S3, ensuring efficient data workflows.
• Data Governance & Quality
  Establish data quality checks and monitoring processes, ensuring data security, encryption, and lineage tracking are integral to your solutions.
• Workflow Automation
  Build and manage automated workflows using tools such as Databricks Jobs and integrate DevOps practices for continuous improvement.
• Collaboration
  Work closely with data engineers and business stakeholders to translate requirements into scalable solutions, actively participating in Agile processes like sprint planning and technical discussions.

Tech Stack

• Databricks
• Apache Spark
• Python (PySpark)
• SQL
• Azure Data Factory
• Delta Lake
• Git
• CI/CD Tools (Azure DevOps, Jenkins)

Required Skills & Qualifications

• Core Technologies
  3-5+ years of experience with Databricks, Apache Spark, and Python (PySpark).
• SQL Expertise
  Advanced knowledge of Spark SQL or Databricks SQL for data transformation and analysis.
• Cloud Platforms
  Hands-on experience with Azure Databricks, AWS, or Google Cloud Platform (GCP).
• Data Modeling
  Experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold layers).
• Version Control & CI/CD
  Proficiency with Git and CI/CD tools such as Azure DevOps or Jenkins.

Preferred Qualifications

• Experience with Data Build Tool (dbt).
• Knowledge of workflow orchestration tools such as Apache Airflow.
• Experience with Machine Learning workflows using MLflow.
• Familiarity with data governance and enterprise data platforms.

Benefits

• Competitive salary based on experience.
• Opportunity to work with cutting-edge technology.
• Flexible working hours and remote working opportunities.
• Professional development and training programs.
• Health insurance and wellness programs.