LLM - Sr. SW Engineer – LLM Evaluation & Repository Validation

Remote

Role

LLM - Sr. SW Engineer – LLM Evaluation & Repository Validation

Type of Role

Short-term roles

YoE

Special reqs

Should have worked for at least 2 years at top-tier product or research companies as a Full Time Employee

Geo

US, UK, Canada, France, Germany, Switzerland, Singapore, Denmark, Finland, Netherlands, Sweden, Iceland, Italy, Austria, Ireland, Norway

Engagement Type

Short Term Contract ( 1 month)

Total number of positions

200

Start Date

Immediate

Vetting

ICF compulsory + Technical Interview

Skill

Fullstack (Backend - Java, Go, Node, Python, C++ &* Frontend *- Typescript, JavaScript, JQuery, React, Vue, Angular)

Availability

Minimum of 10 hrs/week - 40hrs/week

Project Overview

We're building high-quality evaluation and training datasets to improve how Large Language Models (LLMs) interact with realistic software engineering tasks. You will have the opportunity to work on a diverse range of projects from helping models traverse complex code bases to building agents that improve model performance.

Role Overview — What Does a Typical Day Look Like?

Work across multiple different projects to improve LLM performance on code: sample projects

Leading and delivering end-to-end agent use cases such as home automation agents, coding copilots, or creative design assistants.
Collaborate with the team to identify edge cases and ambiguities in model behavior.
Review and compare 3–4 model-generated code responses per task using a structured ranking system.
Evaluate code diffs for correctness, code quality, style, and efficiency. Provide clear, detailed rationales explaining the reasoning behind each ranking decision.

Required Skills & Experience

Several years of software engineering experience, including 2+ continuous years at a top-tier product company (e.g., Google, Stripe, Amazon, Apple, Meta, Netflix, Microsoft, Datadog, Dropbox, Shopify, PayPal, IBM Research).
Strong expertise in building full-stack applications and deploying scalable, production-grade software using modern languages and tools.
Deep understanding of software architecture, design, development, debugging, and code quality/review assessment.
Proven ability to review code diffs and evaluate correctness, maintainability, and efficiency.
Excellent oral and written communication skills for clear, structured evaluation rationales.

Engagement Details

Commitment: flexible engagement, minimum 10 hrs/week, up to 40 hrs/week (partial PST overlap required).
Type: Contractor (no medical/paid leave).
Duration: 1 month potential extensions based on performance and fit.

Top companies:

Google (Alphabet), Apple, Amazon, Meta (Facebook), Netflix, Microsoft, Tesla, NVIDIA, Adobe, Salesforce, Github, Atlassian, hashiCorp, Databricks, Snowflake, Cloudflare, DigitalOcean, MongoDB, Elastic, Confluent, Airbnb, Dropbox, Stripe, Palantir, Uber, Lyft, Square (Block), Twilio, Snap Inc., Pinterest, Figma, Oracle, Cisco, Paypal, Doordash, Rivian, Reddit, Coinbase, Splunk, Spotify, Goldman Sachs, Morgan Stanley, JP Morgan Chase, Capital One, Plaid, Shopify, Intuit, Workday, ServiceNow

Job Type: Contract

Pay: $60.00 - $70.00 per hour

Work Location: Remote

Acies IT Solutions

Latest Jobs

Software Engineer