Software Engineer – AI Code Evaluation & Benchmarking (US candidates only)
Explore this software engineering opportunity. The information below summarizes the supplied role details; check the application page for the latest requirements and availability.
Role highlights & qualifications
- PythonGitDocker
- Seeking experienced Software Engineers to help evaluate, benchmark, and improve the coding capabilities of frontier AI models. In this role, you will assess AI-generated code, validate solutions against real-world software engineering tasks, identify correctness and quality issues, and contribute to the development of high-quality evaluation datasets and benchmarks.
- This position is ideal for engineers who enjoy code review, debugging, problem-solving, and applying strong software engineering judgment to complex technical scenarios. Your work will directly contribute to measuring and improving the performance of advanced AI coding systems.
- Review and evaluate AI-generated code for correctness, efficiency, maintainability, and adherence to requirements.
- Analyze software engineering tasks and validate whether proposed solutions meet expected outcomes.
- Assess model-generated explanations, reasoning, and implementation approaches for technical accuracy.
- Create, refine, and maintain evaluation datasets, benchmarks, and grading rubrics for coding tasks.
- Document findings clearly and provide structured feedback to improve evaluation quality and consistency.
- Collaborate with project teams to establish quality standards and evaluation methodologies.
- Bachelor's or Master's degree in Computer Science, Software Engineering, or a related technical field.
Information is summarized from the supplied job posting. Please verify location eligibility, compensation, contractual terms and current availability on the application page.