Join a dynamic team as an Azure Data Engineer, leveraging your expertise in data engineering and analytics to develop innovative data solutions.

About the Company

Vidpro Consultancy Services is a forward-thinking firm specializing in providing innovative solutions to its clients through data engineering and analytics. They are known for ensuring that their data solutions are state-of-the-art and tailored to meet the unique challenges of their clients within diverse industries.

Role Overview

As an Azure Data Engineer, you will play a crucial role in developing modern data warehouse solutions using the Databricks and Azure stack. This position demands a blend of technical acumen, analytical skills, and the ability to collaborate with cross-functional teams to enhance the data analytics capabilities of our client’s organization.

Key Responsibilities

• Develop modern data warehouse solutions utilizing Databricks and the AWS/Azure stack, ensuring efficient data extraction, transformation, and loading strategies.
• Collaborate effectively with Data Warehouse and Business Intelligence teams to gather ETL pipeline development requirements.
• Diagnose and triage issues within existing pipelines, applying your problem-solving skills to rectify challenges and enhance performance.
• Engage with business stakeholders to refine reporting layer requirements and develop corresponding data models.
• Mentor junior team members, offering guidance in resolving technical challenges and ensuring adherence to best practices.
• Conduct technical discussions with client architects and development teams, ensuring that project deliverables align with architectural guidelines and client expectations.
• Manage data pipelines efficiently using Airflow, ensuring timely and reliable data availability across the organization.

Tech Stack

• Languages: Python, SQL, PySpark
• Technologies: Azure Databricks, Azure Data Factory, AWS, Apache Kafka, Terraform, CircleCI, GIT
• Methodologies: Agile, DevOps
• Data Management: NoSQL (MongoDB, Cassandra), Hadoop (Hive, Pig, Impala)

Salary

Estimated salary: ₹80,000 - ₹1,50,000 per month

Requirements

• Bachelor's and/or master’s degree in computer science or equivalent experience.
• Total 6+ years of IT experience, with 3+ years in Data Warehouse/ETL projects.
• Proficiency in Azure Databricks, PySpark, and SQL.
• Experience with data management principles and dimensional modelling (Star and Snowflake).
• Hands-on experience with SQL, Python, and Spark (PySpark).
• Familiarity with AWS/Azure stack and ETL with batch and streaming (Kinesis).
• Experience in building ETL/data warehouse processes and working with Apache Kafka.
• Understanding of big data tools such as Hadoop (including Hive, Pig, and Impala) and NoSQL databases (MongoDB, Cassandra, Neo4J).
• Experience in a DevOps environment; familiarity with Terraform, CircleCI, and GIT.
• Strong analytical and problem-solving skills.
• Clear verbal and written communication skills.
• Agile methodology experience preferred.
• Databricks Certified Data Engineer Associate/Professional Certification is desirable.

Benefits

• Competitive salary with performance bonuses.
• Flexible working hours with opportunities for remote work.
• Health and wellness programs.
• Career advancement opportunities and professional development.
• Engaging and collaborative work culture.