[Job-30854] AI SDET (Software Development Engineer in Test)

To move beyond traditional QA by owning the "Evaluation Harness," ensuring the AI agents are safe, compliant, and accurate through rigorous automated testing of non-deterministic outputs.

Key Responsibilities

• Own and manage the Langfuse eval harness, including the creation of "Golden Paths" and adversarial test scenarios.
• Validate regulatory quality gates (SR 26-2, NYDFS Reg 187) and run shadow mode validation to compare AI performance against human benchmarks.
• Test for data quality degradation, population segmentation errors, and human-in-the-loop (HITL) checkpoint reliability.
• Verify that the Designer’s output standards (explanation blocks, signal types) are maintained in all evaluation scenarios.

Required Skills and Qualifications

• Advanced proficiency in Python 3.11+, Pytest, and automated testing frameworks.
• Direct experience with AI Evaluation tools (Langfuse, RAGAS, or similar) and LLM-as-a-judge patterns.
• Knowledge of model risk management and regulatory compliance testing.
• Experience with "Shadow Mode" deployment strategies and statistical validation.

Soft Skills

• A "skeptical" and adversarial mindset to find edge cases.
• Uncompromising attention to detail regarding regulatory evidence.
• Collaborative approach to working with engineers to fix logic failures.