Digit88 is seeking a Lead Big Data Engineer to join a new offshore product development team for a partner building next-generation Big Data, Cloud-Based Business Operation Support technology. This role involves building, managing, and optimizing Big Data pipelines using tools like Spark, Akka, and Hadoop, while leading a team of engineers in India and collaborating with US-based teams. The ideal candidate has 8+ years of experience, strong Java/Scala skills, and expertise in the Big Data ecosystem.

About the Company

Digit88 is a lean, mid-stage software company that empowers digital transformation for innovative, high-growth B2B and B2C SaaS companies as a trusted offshore software product engineering partner. With a team of 75+ fantastic technologists, Digit88 is backed by executives who bring deep understanding and extensive experience in consumer and enterprise product development across large corporations and startups. The company builds highly efficient and effective engineering teams that solve real and complex problems for its partners. With over 50 years of collective experience in areas such as B2B and B2C SaaS, web and mobile apps, e-commerce platforms, custom enterprise SaaS platforms, and domains spanning Conversational AI, Chatbots, IoT, Health-tech, ESG/Energy Analytics, and Data Engineering, the founding team thrives in a fast-paced and challenging environment. The vision of Digit88 is to be the most trusted technology partner to innovative software product companies worldwide.

Role Overview

Digit88 is establishing a new offshore product development team for a partner that is building next-generation Big Data, Cloud-Based Business Operation Support technology for utilities, retail energy suppliers, and Community Choice Aggregators (CCA). As a Lead Big Data Engineer, you will join an existing team of outstanding data engineers in the US and help expand the data engineering team. You will work on different products and layers of the infrastructure, responsible for building, managing, and optimizing Big Data pipelines that handle huge structured datasets to accurately generate analytics at scale. The primary focus is on choosing optimal solutions, maintaining, implementing, and monitoring them, and integrating them with the company’s architecture. This role requires a passion for engineering with accuracy and efficiency, being highly motivated and organized, and working both as part of a team and independently with minimal supervision.

Key Responsibilities

• Collaborate closely with Product Management and Engineering leadership to devise and build the right solutions.
• Participate in design discussions and brainstorming sessions to select, integrate, and maintain Big Data tools and frameworks required to solve Big Data problems at scale.
• Design and implement systems to cleanse, process, and analyze large data sets using distributed processing tools like Akka and Spark.
• Critically review existing data pipelines and collaborate with Technical Leaders and Architects to improve current bottlenecks.
• Take initiatives, drive to pick up new stuff proactively, and work as a Senior Individual contributor on multiple products and features.
• Work closely with US and India engineering teams to build Java/Scala-based data pipelines.
• Lead the India engineering team in technical excellence and ownership of critical modules; own the development of new modules and features.
• Troubleshoot live production server issues.
• Handle client coordination, contribute independently, and drive the team to exceptional contributions with minimal supervision.
• Follow Agile methodology and use JIRA for work planning and issue management/tracking.

Tech Stack

• Languages: Java, Scala
• Big Data Frameworks: Apache Spark, Apache Hadoop, Apache Storm, Akka
• ETL & Pipeline Tools: Apache NiFi, Apache Airflow
• File Types: Parquet, Avro, ORC (implied from Big Data ecosystem)
• Build Tools: Gradle, Maven, SBT
• Version Control: Git
• Other: Distributed computing, object-oriented design, data structures, algorithms, profiling, optimization

Education: Bachelor's degree in Engineering, Computer Science, or a related field.

• 8+ years of experience in developing highly scalable Big Data pipelines. (Note: Job description originally stated 6-9 years but later mentions 8+ years. We have used 8+ years as the more specific requirement.)
• Hands-on experience in team leading and leading product or module development.
• In-depth understanding of the Big Data ecosystem including processing frameworks like Spark, Akka, Storm, and Hadoop, and the file types they deal with.
• Experience with ETL and Data pipeline tools like Apache NiFi, Airflow, etc.
• Excellent coding skills in Java or Scala, including understanding of Design Patterns.
• Experience with Git and build tools like Gradle/Maven/SBT.
• Strong understanding of object-oriented design, data structures, algorithms, profiling, and optimization.
• Elegant, readable, maintainable, and extensible code style.
• Strong verbal and written communication skills.
• Strong sense of urgency, with a passion for accuracy and timeliness.
• Ability to work calmly in high pressure situations and manage multiple projects/tasks.
• Ability to work independently and possess superior skills in issue resolution.
• Passion to learn, implement, analyze, and troubleshoot issues.

Comprehensive Insurance (Life, Health, Accident); Flexible Work Model; Accelerated learning & non-linear growth; Flat organization structure driven by ownership and accountability; Global Peers – Working with some of the best engineers/professionals globally from the likes of Apple, Amazon, IBM Research, Adobe and other innovative product companies; Ability to make a global impact with your work, leading innovations in Conversational AI, Tele-Medicine, Healthcare and more.