SWE-Bench AI Task Auditor - Freelance AI Trainer Project

Accountabilities

• Evaluate software engineering tasks for technical accuracy, realism, solvability, reproducibility, and alignment with established SWE-Bench standards.


• Review complex codebases and real-world application scenarios to determine whether tasks accurately reflect practical software engineering challenges.


• Assess the reliability and completeness of tests, evaluation criteria, and expected outcomes associated with assigned tasks.


• Identify codebase integration issues, logic errors, test failures, and other technical inconsistencies that could affect task quality or evaluation accuracy.


• Troubleshoot complex technical scenarios and provide clear, actionable feedback to improve task implementation and evaluation workflows.


• Apply specialized software engineering knowledge to ensure AI training and evaluation tasks meet high standards of technical rigor and practical relevance.


• Document findings clearly and consistently so that identified issues can be addressed efficiently.

• Demonstrable professional experience and deep technical expertise in software engineering, including complex codebase navigation and real-world application development.


• Strong familiarity with software engineering concepts and the ability to evaluate tasks against practical development standards, including SWE-Bench-related requirements.


• Strong analytical and problem-solving skills, with the ability to rigorously test, troubleshoot, and evaluate complex technical scenarios.


• Strong attention to detail and the ability to identify subtle logic, integration, testing, and reproducibility issues.


• Excellent written communication skills for providing precise, constructive, and actionable technical feedback.


• Ability to work independently and apply specialized expertise to project-based freelance assignments.


• Candidates will be evaluated for a specific area of specialization based on their professional background; expertise across multiple domains is not required.


• Ability to provide a secure computer and reliable high-speed internet connection for freelance project work.

• Compensation of $70–$100 per hour, with the exact rate determined based on experience, expertise, and geographic location.


• Final compensation may vary from the published range following evaluation of your qualifications and location.


• Freelance contract structure offering flexibility for project-based work.


• Remote work environment.


• Opportunity to contribute directly to the training and evaluation of advanced AI systems.


• Opportunity to apply specialized software engineering expertise to technically challenging AI evaluation projects.


• Health insurance, paid time off, and other company-sponsored employee benefits are not provided under the freelance contract.


• Contractors are responsible for supplying their own secure computer and high-speed internet connection.

Dette opslag er fra Lever. Se originalopslag ↗