Human data research program manager

The human data research program manager is the professional who scopes and leads human data collection for research and the evaluation of artificial intelligence models. They turn researchers’ needs into workable campaigns, coordinate those involved and monitor the quality of the data collected. The role’s value lies in maintaining this connection between scientific requirements and operational delivery.

Working with researchers, they clarify collection needs, success indicators and quality measures. They organise partnerships, monitor campaigns and adjust their delivery as priorities change. Providers and evaluators produce the human contributions. Engineers work on platforms, address platform issues and deal with access. The manager needs to understand each party’s constraints to make operational decisions explicit.

The scope described here draws on the common ground between a campaign management role at OpenAI and an adjacent human data operations role at Anthropic, two US laboratories. The exact job title therefore does not provide a general framework for French organisations. When hiring, specify the reporting line, the campaigns assigned and the decisions delegated. Authority over budgets, contracts, final scientific choices or a team of employees must be defined within your organisation.

Campaign manager, AI researcher and platform engineer: who does what?

• The campaign manager translates research needs into data collection, organises delivery and monitors results. Depending on the role, they analyse data or automate certain operations.
• The researcher brings the scientific requirements that they and the manager use to define success and quality criteria. Campaign management alone does not constitute a research role.
• The platform engineer works on tools and addresses technical obstacles. Collaboration may include temporary solutions proposed by the manager, without making the manager the platform’s main developer.

Annotation coordination focused on a single campaign may cover some of these activities. To distinguish the roles, look particularly at their connection to scientific requirements and responsibility for multiple campaigns.

A campaign can produce many contributions without meeting the research need. The hiring challenge is to assign campaign monitoring to someone who can connect scientific requirements with collection decisions. Delivery volume, progress and quality need to be discussed together, using indicators whose meaning both researchers and providers understand.

Quality needs particular attention. A scientific review of textual datasets identifies annotation errors, bias and artefacts, among other issues. These problems matter for model training and evaluation. This finding provides no universal quality threshold. It points to the need to define, for each campaign, what checks should measure and which limitations should remain visible when the data is used.

Poorly defined responsibilities can also leave obstacles without anyone to address them. Whether the issue involves ambiguous instructions, campaign analysis or platform access, the manager must be able to identify the difficulty and coordinate its resolution. They should not implicitly be assigned final scientific decisions or all technical changes. Clarify who they should involve and which decisions they can make independently.

Hypothetical example: a team wants to extend data collection to a new language while the initial campaign still has annotation disagreements. The manager prepares the information needed to decide: instructions to test, contributors’ skills and results from existing checks. Researchers clarify the scientific requirements for the extension. The team can then decide how to proceed without treating the translation of instructions as sufficient preparation.

Finally, distinguish analytical ability from coordination ability when hiring. A candidate may provide clear campaign updates without explaining the decisions drawn from the results. Another may be proficient with analytical tools without knowing how to organise communication with providers. Assessment should relate both dimensions to the actual complexity of your campaigns.

• Scope human data collection needs with researchers.
• Define success indicators and quality measures for each campaign with researchers.
• Turn research requirements into workable campaigns using the tools available.
• Organise collection partnerships and coordinate providers and evaluators.
• Build dashboards to analyse campaign results and adjust delivery.
• Prioritise campaigns according to research needs and operational dependencies.
• Coordinate with engineers to resolve platform obstacles and address access issues.

Prioritise the skills and experience that enable a candidate to move from a research request to a campaign that can be monitored and understood. Relevant experience should show which responsibilities they have held: scoping requirements, coordinating those involved, analysing results and resolving difficulties. A candidate’s previous job title is not enough to establish these abilities.

For its adjacent human data operations role, Anthropic accepts backgrounds including operations, consulting, product and programme management. The employer specifies a bachelor’s degree or an equivalent combination of education and experience. This openness describes a particular vacancy. It does not establish a mandatory qualification for the profession or an automatic connection between a background and proficiency in human data campaigns.

Technical skills should match the work assigned. For a role involving independent analysis, look for the ability to use campaign data and explain the indicators chosen. If the manager needs to integrate tools through APIs, Python experience directly relevant to that task becomes useful. Understanding researchers’ needs and LLM training methods helps them communicate with the scientific team, without requiring the candidate to have worked as a researcher.

Additional training can provide grounding in annotation and crowdsourcing. The Johns Hopkins course Training AI with Humans covers, among other topics, calculating inter-annotator agreement. This knowledge can help in understanding measures of data consistency. It replaces neither experience in delivering campaigns nor comprehensive training in programme management.

To assess this profile, adapt the following steps to the campaigns assigned and the decisions your organisation delegates to the manager.

1. Define a role-specific assessment framework

Use distinct criteria: scoping with researchers, measuring quality, analysing campaigns, coordinating providers and resolving obstacles. For each criterion, specify the level of autonomy expected.

Involve researchers in assessing scientific requirements and engineers in assessing platform constraints. If the team lacks expertise, bring in an expert for a specific assessment of the aspects it cannot judge on its own.

A positive sign is the candidate’s ability to distinguish operational decisions from those requiring scientific or technical agreement. A warning sign is a claim to autonomy without explicit limits.

2. Examine a past achievement

Ask the candidate to describe a campaign, from the initial request to the results delivered. Have them clarify their personal role, the indicators chosen and the adjustments made during delivery.

Invite them to present an anonymised dashboard or set of instructions where sharing is possible. Ask what these documents helped people understand and which decision followed.

Look for an explanation connecting results to a specific action. Pay attention to an account that attributes every achievement to the candidate without distinguishing the contributions of researchers, providers and engineers.

3. Present a case similar to your campaigns

Hypothetical example: a campaign is running late, some evaluators interpret an instruction differently and a platform dependency is preventing part of the collection.

Ask the candidate to explain which information to gather, which priorities to propose and whom to involve. Have them distinguish an instruction problem, a technical difficulty and a question requiring scientific discussion.

Examine how they make uncertainty visible. A positive sign is a response that plans to test instructions and understand disagreements before applying a correction more widely.

A warning sign is removing divergent contributions without examining their cause. Do not expect a universal procedure: assess the quality of the explanation and how well it fits the context provided.

4. Check autonomy in analysis and integration

If the role requires analysis, provide a small dataset drawn from the preceding hypothetical case. Ask the candidate to choose indicators, explain their limitations and propose a decision.

Where API integrations are part of the role, assess separately their ability to send a request, process a response and integrate the result into the workflow. Adapt the exercise to the tools actually used.

A clear explanation of assumptions and missing data is a positive sign. A numerical result without an explanation of how it was produced should prompt further discussion.

5. Assess coordination and take up references

Have the candidate explain how they communicate an obstacle to a researcher, a provider and an engineer. Ask how they support those involved without making decisions on their behalf.

If the role includes direct management, assess this responsibility separately from provider coordination. Do not infer line management experience from the title of program manager alone.

With the candidate’s agreement, take up references from contacts who participated in the campaigns described. Ask these contacts about the candidate’s responsibilities, how they explained difficulties and which decisions they actually made.

Draw your conclusion from all the observations. A strong candidate makes their choices, the limits of their role and the points requiring another specialist’s involvement understandable.