rlhf services role scorecard template

# rlhf services role scorecard template

What should a busy buyer put in the scorecard?

Start with the decisions the person will make. "AI trainer" leaves too much room for different interpretations. A person who ranks chatbot answers needs a different assessment from someone who reviews code or manages disputed labels.

For reinforcement learning from human feedback, or RLHF, write down the task and the evidence that makes a judgment acceptable. Then build the interview around that work.

Use the following template before asking for candidates. Replace bracketed fields with your requirements. The scoring rules are suggested starting points, not industry benchmarks.

Define the role before setting the score

Keep the scope narrow enough to assess. If the role includes coding judgments, give the candidate code to inspect. If it includes factual review, define what evidence the person may consult.

For sensitive tasks, describe the content before the assessment. Agree on escalation and support arrangements before assigning that work.

Write the role's exclusions with the same care as its tasks. An evaluator may flag a policy conflict without having authority to change the policy. A reviewer may correct labels without approving a new dataset for release. Put those limits in the scorecard so candidates know when to stop and ask.

Score observable behavior

Use this suggested rating scale across the work sample:

• 1: The candidate cannot complete the task under the written rules.
• 2: The candidate completes parts of the task but needs repeated correction.
• 3: The candidate meets the requirement independently.
• 4: The candidate meets it and explains edge cases in a way another reviewer can use.

Record the result beside a specific example. A rating without evidence should not decide the hire.

Do not let a high total score cancel a failed data-handling check. Decide which requirements are mandatory before interviews begin.

For a general evaluator, rubric use and written reasoning deserve close attention. For a specialist, add a separate domain assessment. A clear explanation can still contain a technical error.

Avoid scoring personal style unless it affects the job. A short rationale may fully explain a decision. A polished paragraph may leave out the decisive evidence. Ask reviewers to identify the rule the candidate applied and whether the reasoning supports the label.

Run a work sample that exposes judgment

Build a paid sample from sanitized tasks. Include straightforward cases and cases where the candidate should ask for help. State which tools are allowed and whether AI assistance is permitted.

Keep the initial instructions consistent across candidates. If someone asks a question, record it and the answer. A useful question can be evidence of sound judgment.

After the first pass, clarify one rule and ask the candidate to revisit an affected answer. This gives you evidence about how the person uses feedback.

The evaluator should explain the decision in plain language. A reviewer should also explain how they would resolve disagreement. A technical lead should show how the issue would be tracked and prevented from recurring.

For example, give a candidate two answers to the same prompt. One follows the requested format but contains a factual error. The other is accurate but ignores a stated constraint. Your rubric should explain how to weigh those defects. Score the candidate against that rule, including whether they notice both problems.

Keep the sample separate from live production work. Give the candidate a clear time limit and tell them what you will assess. Provide any access or accessibility adjustments needed to complete the exercise fairly.

What does the data say about setting the role and budget?

BLS reports a median annual wage of $135,980 for US software developers in May 2025. It reports $104,300 for software quality assurance analysts and testers for the same period.

Those figures describe the named US occupations. They are not a price list for RLHF evaluators, Latin American hires, or a managed annotation service. The BLS Occupational Employment and Wage Statistics program provides broader occupational wage tables.

Use these wage figures as context when a role includes software development or software QA duties. Do not assign a developer pay benchmark to every person who reviews model output.

For a coding evaluator, ask how much of the job requires running tests, debugging, or explaining program behavior. For a general evaluator, define the reading and reasoning tasks directly. Price the actual scope.

Separate wages from the service quote

Ask each provider to quote against the same completed role template. Include seniority, task complexity, working hours, and the review burden.

Use a comparison sheet with separate rows for:

• The recurring charge and billing unit.
• Included work and expected availability.
• Candidate assessment and onboarding.
• Task training and quality review.
• Platform access and approved tools.
• Rework, replacement, and handover terms.

This helps expose work your own team must perform. Ask who pays when a batch needs another review and how disputed charges are handled.

Keep internal management time in the budget. Someone must answer questions, review exceptions, and approve changes to the rubric. Name that person before deciding how much external support to buy.

When comparing quotes, keep task complexity constant. A price for short preference rankings cannot tell you the cost of checking long code answers. Ask for separate assumptions where the queue contains different task types. Record what happens to the quote if the mix changes.

Collect quality evidence before choosing a volume target

Set a pilot period with a fixed task mix. Track accepted work, corrections, and the reasons for disagreement.

Define each measure before collecting it. An acceptance rate needs a clear denominator and a rule for items awaiting review. Record pending items separately. If you measure first-pass acceptance, keep corrected work out of that numerator and report it separately.

Keep error types visible. A wrong label, a weak explanation, and a data-handling breach need different responses. A single average can hide which problem is growing.

When two evaluators disagree, check the instructions before changing either person's score. If both readings fit the rubric, revise the rule and reassess the affected work.

Have reviewers score a shared sample before using the scorecard on candidates. Compare their reasons as well as their ratings. If they cannot agree on what meets the requirement, fix the instructions before treating a candidate's result as decisive.

Use the pilot to set a realistic pace. Avoid a quota that rewards quick answers before anyone checks whether those answers meet the standard.

Keep the pilot's task mix beside its results. If later work includes longer prompts or harder specialist questions, review the pace target. Do not carry a target into a different queue without checking whether the assumptions still hold.

How do hiring models differ for a nearshore or Latin American team?

Choose the model by deciding who will manage the work. Keep that responsibility explicit in both the scorecard and the agreement.

When comparing RLHF companies, ask who assigns tasks, trains evaluators, and signs off on completed work. A service label alone does not answer those questions.

Treat this table as a checklist for the contract you are considering. Confirm each responsibility rather than assuming it comes with the model.

Use an individual scorecard when you direct the work

If your manager sets the queue and gives feedback, assess how the person works inside that process.

Include written handoffs, response to feedback, and attendance at required review sessions. Keep the technical criteria tied to the actual role.

Before hiring, check whether your manager has time to answer task questions. Adding evaluators to a queue with unresolved rules can increase the review backlog.

An employer of record service may be part of the employment arrangement. Keep its scope separate from the question of whether the candidate can perform the work.

Ask the hiring manager to review the sample themselves before interviews begin. They should be able to explain the expected decisions and the allowed exceptions. If that requires another specialist, include that specialist in the assessment and ongoing review plan.

Add a service scorecard when you buy delivery

For a managed service, keep the individual criteria and add delivery requirements.

Define what an accepted batch contains, who reviews it, and what happens when it fails. State how rubric changes affect deadlines and charges.

If you are comparing business process outsourcing services, ask for a walkthrough of a disputed task. Follow it from the evaluator's first decision through review and final resolution.

Request the actual handoff format. Check whether it records the task identifier, decision, rationale, rubric version, and review status.

Also define continuity. If a reviewer leaves, ask who takes over the queue and how the new reviewer learns prior decisions.

Separate provider mistakes from buyer changes. A batch that fails an agreed rule needs a different response from a batch affected by a new rule. Write down who approves rework in each case and how the revised deadline is recorded.

Write shared hours as an explicit requirement

"US overlap" needs a schedule. Record the working window and named time zone for both the hire and the manager.

Bogotá is UTC-5 and does not observe daylight saving time. São Paulo is UTC-3 and does not observe daylight saving time. Use the IANA time zone rules when checking the schedule.

Test the proposed overlap across the year. Put calibration meetings and escalation coverage into the schedule, then confirm that the candidate accepts it.

Assess language skill through the work sample. If the task requires writing for a particular market, include examples from that market. Ask the candidate to explain unclear wording and show how it affects the judgment.

What should you ask a provider before approving candidates?

Send the completed scorecard with the request for candidates. Ask the provider to return evidence against each requirement.

A useful response shows what was tested, how it was scored, and what your team still needs to assess. Keep those open checks in the hiring record.

Ask to see the assessment

Use these questions during the provider review:

• Which tasks match the work this person will do?
• Can we inspect a sanitized sample and its scoring notes?
• Who assessed the candidate's domain knowledge?
• How do you handle a correct answer with weak reasoning?
• What happens when the candidate challenges the expected answer?
• Which tools can candidates use during the assessment?
• How do you distinguish independent work from assisted work?
• Can our reviewer assess the same sample before the interview?

Do not request confidential material from another customer. A sanitized demonstration or a sample built for your role can show the process.

Ask for evidence at the criterion level. A general statement that someone passed vetting does not tell your reviewer which task to skip.

If an assessment permits AI tools, ask to see how the candidate checked the output. Include an independent exercise when the role requires unaided judgment. Keep tool rules consistent with the work you expect after hiring.

Agree on ownership before the start date

Write an owner beside each recurring decision:

Change the suggested owners to fit the agreement. The result should leave no task waiting between two teams that each expect the other to act.

For access, specify permitted systems and whether downloads are allowed. State the rules for personal devices and external AI tools. Include access removal in the handover plan.

Set a route for questions that arrive outside shared hours. Tell the evaluator which tasks can continue and which must wait for a reviewer. Assign a backup for absences so a blocked task has somewhere to go.

Review a difficult case together

Ask the provider to explain how it would handle an answer that is fluent but factually wrong. Then ask about an answer that is correct but breaks the task instructions.

The purpose is to test whether everyone applies the same rules. If the provider and your reviewer disagree, resolve that difference before scoring candidates.

For coding work, include the permitted runtime and test environment. For specialist work, define when a qualified reviewer must step in.

Record the final decision and its reason. Keep it with the rubric so future reviewers can apply the same rule. When a rule changes, record which earlier tasks need another review.

Make the hiring decision traceable

Use a short decision record for each finalist:

Keep an open concern separate from a failed requirement. If the candidate has not been tested on a required skill, run that check before deciding.

Have each interviewer record evidence before the group discussion. Review conflicting ratings against the sample rather than averaging them immediately. Where evidence is thin, assign a focused follow-up task instead of repeating the whole interview.

This site is an Opus-owned reference; recommendations are editorial.

Talk to Opus about a role.

Where does Opus fit in this hiring decision?

Opus states: "The client chooses the finalist and manages day-to-day work."

That division of responsibility makes the individual role scorecard useful. Your team should define the tasks, assess the finalist against them, and name the person who owns daily feedback.

Opus states: "Opus charges one flat monthly rate per hire. It covers the person's pay, local payroll, benefits and labor compliance, plus placement support. There is no separate recruiting fee. The rate depends on role, seniority and location."

Published dollar rates: Not published.

Ask for a written quote tied to the completed scorecard. Include the role's specialist knowledge, working hours, and assessment requirements. Confirm which task checks happen before you meet a finalist.

Keep the job requirements separate from the service terms. Review the candidate's work sample for hiring evidence. Review the agreement for charges, support, and replacement terms.

Opus states: "Every placement carries a lifetime replacement guarantee with no cap and no time limit." Ask how a replacement would affect your queue and who would train the new person on your rubric.

Before the search starts, prepare the role template, a sanitized sample, and the scoring guide. Name the manager who can review candidates and answer task questions.

For the first working period, agree on a calibration queue and a review schedule. Keep the initial task scope narrow enough to inspect decisions closely. Expand it after the hire meets the agreed requirements.

Use the same scorecard in the first performance review. Compare the work with the hiring evidence and record any changed expectations. If the task has expanded, update the role before judging the person against the new scope.

Talk to Opus about a role.

<!-- File write skipped: the workspace is read-only. Suggested scoring rules require calibration against your tasks. -->