Digit88 is looking for a Lead Big Data Engineer to build and manage Big Data Pipelines for handling huge structured datasets. The role involves leading the India engineering team, collaborating with US teams, and using tools like Spark, Akka, Java, Scala, NiFi, and Airflow.

About the Company

Digit88 empowers digital transformation for innovative and high growth B2B and B2C SaaS companies as their trusted offshore software product engineering partner. We are a lean mid-stage software company with a team of 75+ fantastic technologists, backed by executives with deep understanding of and extensive experience in consumer and enterprise product development across large corporations and startups. We build highly efficient and effective engineering teams that solve real and complex problems for our partners. With more than 50+ years of collective experience in areas ranging from B2B and B2C SaaS, web and mobile apps, e-commerce platforms and solutions, custom enterprise SaaS platforms and domains spread across Conversational AI, Chatbots, IoT, Health-tech, ESG/Energy Analytics, Data Engineering, the founding team thrives in a fast paced and challenging environment that allows us to showcase our best.

Role Overview

Digit88 is looking for a Lead Big Data Engineer to join our expanding data engineering team. You will work on building and managing Big Data Pipelines to handle huge structured data sets that are used as input to accurately generate analytics at scale for our valued customers. The primary focus will be on choosing optimal solutions for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company. Applicants must have a passion for engineering with accuracy and efficiency, be highly motivated and organized, able to work as part of a team, and also possess the ability to work independently with minimal supervision.

Key Responsibilities

• Collaborate closely with Product Management and Engineering leadership to devise and build the right solution.
• Participate in Design discussions and brainstorming sessions to select, integrate, and maintain Big Data tools and frameworks required to solve Big Data problems at scale.
• Design and implement systems to cleanse, process, and analyze large data sets using distributed processing tools like Akka and Spark.
• Understanding and critically reviewing existing data pipelines, and coming up with ideas in collaboration with Technical Leaders and Architects to improve upon current bottlenecks.
• Take initiatives, and show the drive to pick up new stuff proactively, and work as a Senior Individual contributor on multiple products and features.
• Work closely with US and India engineering teams to help build Java/Scala based data pipelines.
• Lead the India engineering team in technical excellence and ownership of critical modules; own the development of new modules and features.
• Troubleshoot live production server issues.
• Handle client coordination and be able to work as a part of a team, drive the team to exceptional contributions with minimal supervision.
• Follow Agile methodology and use JIRA for work planning, issue management/tracking.

Tech Stack

• Programming Languages: Java, Scala
• Big Data Frameworks: Spark, Akka, Storm, Hadoop
• Data Pipeline & ETL: Apache NiFi, Airflow
• Build Tools: Gradle, Maven, SBT
• Version Control: Git
• Other: Distributed processing, object-oriented design, data structures, algorithms, profiling, optimization
• Methodology: Agile, JIRA

Education: Bachelor's degree in Engineering, Computer Science, or a related field. 8+ years of experience in developing highly scalable Big Data pipelines. Hands-on experience in team leading and leading product or module development. In-depth understanding of the Big Data ecosystem including processing frameworks like Spark, Akka, Storm, and Hadoop, and the file types they deal with. Experience with ETL and Data pipeline tools like Apache NiFi, Airflow etc. Excellent coding skills in Java or Scala, including the understanding to apply appropriate Design Patterns when required. Experience with Git and build tools like Gradle/Maven/SBT. Strong understanding of object-oriented design, data structures, algorithms, profiling, and optimization. Have elegant, readable, maintainable and extensible code style. Should be able to work independently with India & US based team members. Strong verbal and written communication with ability to articulate problems and solutions over phone and emails. Strong sense of urgency, with a passion for accuracy and timeliness. Ability to work calmly in high pressure situations and manage multiple projects/tasks. Ability to work independently and possess superior skills in issue resolution. Should have the passion to learn and implement, analyse and troubleshoot issues.

Comprehensive Insurance (Life, Health, Accident); Flexible Work Model; Accelerated learning & non-linear growth; Flat organization structure driven by ownership and accountability; Global Peers – Working with some of the best engineers/professionals globally from the likes of Apple, Amazon, IBM Research, Adobe and other innovative product companies; Ability to make a global impact with your work, leading innovations in Conversational AI, Tele-Medicine, Healthcare and more.