Software Engineer

Responsibilities

• Evaluate and compare multiple AI-generated responses to technical prompts for clarity, correctness, tone, and usefulness.
• Assess AI-generated technical documentation, error messages, code comments, architectural explanations, instructions, and related engineering content.
• Assign analytical quality scores and write detailed rationales supporting each evaluation.
• Identify responses that are technically accurate but poorly communicated or inappropriate for their intended audience.
• Detect fluent and convincing responses that contain factual or technical inaccuracies.
• Document evaluation observations and findings with precision, consistency, and strong attention to detail.
• Identify and clearly explain subtle errors, inconsistencies, ambiguities, and communication problems in technical content.
• Evaluate technical responses for audience appropriateness and instructional value.
• Collaborate asynchronously with project managers and reviewers to maintain alignment on evaluation standards.
• Work independently to meet project requirements and deliverables within defined timeframes.

Skills

Requirements

• Native-level English fluency and exceptional written communication skills.
• Professional experience as a software engineer working with a modern technology stack.
• Proven experience authoring or reviewing technical documentation, design documents, API references, specifications, or comparable engineering materials.
• Experience conducting code reviews, design reviews, or providing detailed feedback on technical writing.
• Demonstrated ability to assess technical communication for correctness, clarity, audience appropriateness, and instructional value.
• Strong analytical and critical-thinking skills.
• Meticulous attention to detail and ability to identify subtle factual errors, inconsistencies, ambiguities, and misleading technical explanations.
• Ability to produce clear and detailed written rationales supporting evaluation decisions.
• Ability to work independently and meet defined project requirements and timelines.
• Familiarity with AI model evaluation, annotation, or RLHF projects is a plus but not required.
• No prior professional AI experience is required.

Benefits

• Fully remote contractor opportunity.
• Compensation of $90–$140 per hour.
• Opportunity to apply software engineering expertise to next-generation AI training and evaluation.
• Work focused on technical analysis and evaluation rather than software feature development.
• Opportunity to evaluate advanced AI-generated technical documentation, explanations, and engineering content.
• Asynchronous collaboration with project managers and reviewers.
• Opportunity to gain experience with AI model evaluation and technical content quality assessment.
• No prior AI evaluation or RLHF experience required.