Seeking a Testing Lead to ensure AI quality for Deep Learning and LLM products, with a strong focus on documentation and testing strategies.

About the Company

We are a forward-thinking company in the Health Care sector, specializing in Deep Learning, Large Language Models (LLMs), and Vision-Language Models (VLMs). Our mission is to leverage advanced AI technologies to enhance healthcare solutions, which requires a rigorous focus on quality assurance across all products. We operate in a fast-paced startup environment, which offers exciting challenges and opportunities for growth.

Role Overview

We are seeking a hands-on Testing Lead to take ownership of the quality and documentation for our cutting-edge AI products, specifically in the realms of Deep Learning, LLMs, and VLMs. The ideal candidate will have a deep understanding of how to test and document AI systems effectively, recognizing the critical importance that clear documentation plays, especially in non-deterministic environments. You will work closely with our Machine Learning, Engineering, and Product teams to establish robust testing methodologies that ensure the highest quality products.

Key Responsibilities

• Own Quality & Documentation End-to-End: Define testing strategies for LLMs, VLMs, and DL pipelines while creating and maintaining clear documentation that includes model testing strategies, evaluation metrics, known limitations, risks, and quality sign-offs.
• LLM / GenAI Testing: Design thorough tests for prompt templates, retrieval quality, and multi-turn conversations, while documenting prompt behavior and maintaining regression test suites.
• Vision & Multimodal Testing: Test VLM performance for image-text alignment, OCR, and reasoning across various domains and quality levels.
• Automation, MLOps & Reporting: Develop Python-based automation for regression testing and integrate these tests into CI/CD and MLOps pipelines. Produce quality reports to monitor production issues related to model drift and quality degradation.
• Build a Quality-First Culture: Establish QA standards, mentor engineers on writing testable code, and serve as the main authority on AI quality and testing.

• Strong background in software testing with lead or ownership experience.
• Hands-on experience testing LLMs, DL models, or GenAI systems.
• Strong Python skills for test automation and data validation.
• Proven ability to write clear, structured technical documentation.
• Understanding of transformer-based models and DL workflows.
• Familiarity with model evaluation metrics and non-deterministic system testing.
• Comfortable working in ambiguity and moving fast in a startup environment.

Nice-to-Have

• Experience with VLMs, multimodal models, or computer vision.
• Exposure to RAG architectures, vector databases, and embeddings.
• Familiarity with tools like LangChain, LlamaIndex, MLflow or similar.
• Experience documenting AI risks, limitations, or compliance requirements.

• Competitive salary with performance bonuses.
• Flexible working model (remote, hybrid, on-site).
• Opportunities for professional development and career advancement.
• A collaborative and supportive work environment.