Agent Evaluation Engineer — Build-Time Framework & Deployment Gates

• Competitive compensation depending on experience and skills
• Variety of projects within one company
• Being a part of a project following engineering excellence standards
• Individual career path and professional growth opportunities
• Internal events and communities
• Flexible work hours

We're looking for an Agent Evaluation Engineer - Build-Time Framework & Deployment Gates to join our team in Portugal in a fully remote working mode. In this role, you will design and maintain an evaluation framework for AI agents, ensuring quality and compliance through automated tests and CI/CD deployment gates. You will develop multi-layer evaluation suites that blend deterministic checks with LLM-powered graders, simulate multi-turn conversations, and define reliability metrics. The position also involves implementing staging validations, shadow-mode traffic analysis, and A/B rollout strategies, with feedback loops from production environments to enhance overall system robustness.

• Design and implement build-time evaluation frameworks for agentic workflows using LangGraph or comparable orchestration frameworks
• Create deterministic and LLM-as-judge grading pipelines covering reasoning, trajectory accuracy, and output quality
• Develop test harnesses for multi-turn conversational simulations and context-retention scoring
• Define reliability assessment methods including multi-trial metrics (pass@k, pass^k)
• Implement CI/CD deployment gates that enforce quality thresholds and block releases not meeting standards
• Integrate staging validation, shadow-mode traffic comparison, and A/B rollout control in deployment pipelines
• Leverage AWS AgentCore Evaluations for on-demand and online scoring components connected to production feedback
• Convert production incidents into reusable regression cases for continuous quality improvement
• Collaborate with engineering and DevOps teams to embed evaluation gates into automated workflows