Join Techsophy as a Data Engineer in Hyderabad! Design scalable big data pipelines using Scala & Spark in a hybrid work environment. 4+ years experience required.

About the Company

TECHSOPHY is a technology solutions company that specializes in productizing solutions based on new technology. Founded in 2009, the company focuses on emerging platforms including BPM & ECM, Low Code, AI (ML/RPA/NLP), and big data. Headquartered in California, USA, TECHSOPHY has regional offices in Dubai, UAE, and an offshore innovation center in Hyderabad, India. The company serves as engineering partners for client development teams in India and the US, building highly scalable and high-performance applications on top of client platforms, as well as platform components that run on big data environments with over 10,000 nodes and petabytes of data.

Role Overview

We are currently looking for a Data Engineer with strong Scala and Spark skills to join our engineering team. In this role, you will work closely with the client engineering team in India and the US to build highly scalable, high-performance applications running on a massive big data environment (10K+ nodes, petabytes of data). You will contribute to both client platform development and core platform components. The ideal candidate has solid fundamentals in algorithms and data structures, exceptional problem-solving abilities, and 4 to 7 years of experience. We value intelligent people who can quickly pick up new technologies, so even if you don't meet every requirement, we encourage you to apply if you can demonstrate strong learning ability.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL processes using Scala and Spark.
• Work with the engineering team to build and optimize applications on big data platforms.
• Implement data processing solutions that handle petabytes of data across a 10K+ node environment.
• Collaborate with US-based and India-based engineering teams to deliver high-performance platform components.
• Apply strong software design and debugging skills to solve complex data engineering challenges.
• Mine and analyze data to obtain actionable insights, leveraging modern big data technologies.
• Ensure code quality, performance, and reliability in a hybrid work environment (3 days in office: Tue, Wed, Thu; 2 days remote).

Tech Stack

• Core: Scala, Apache Spark, Hadoop ecosystem (HDFS, MapReduce)
• Streaming & Messaging: Kafka
• Databases & NoSQL: Cassandra, NoSQL databases
• Data Processing: Spark SQL, DataFrames, RDDs
• Big Data Ecosystem: Hadoop, MapReduce, HDFS
• Other: Data analytics, data mining, problem-solving, debugging
• Work Environment: Hybrid (3 days in office, 2 days remote)

Note: Experience with modern big data technologies (Spark, NoSQL, Cassandra, Kafka, MapReduce, Hadoop ecosystem) is good to have, and data analytics/mining skills are highly valued.

Solid fundamentals and exceptional problem-solving skills. Solid and fluent understanding of algorithms and data structures. Proficiency in Scala + Spark. Experience range: 4 to 7 years. Excellent programming skills and knowledge of Scala Spark. Excellent software design, problem-solving, and debugging skills. Good to have experience with modern big data technologies such as Spark, NoSQL, Cassandra, Kafka, MapReduce, Hadoop ecosystem. Experience with data analytics and ability to mine data to obtain insights is much appreciated.

Work from home, Flexible schedule, Health insurance, Paid sick time, Paid time off, Provident Fund