AI Cybersecurity Capabilities Researcher

The AI Cybersecurity Capabilities Researcher studies what artificial intelligence models or agents can accomplish on defined computer security tasks. They build datasets, benchmarks and prototypes to measure performance, understand failures and establish the limits of the results. Their work gives research and security leaders evidence to decide which capabilities to investigate further and which uses still need to be examined.

The subject of the research must be explicit: the ability to find a vulnerability, propose a fix or complete a cyber task in a test environment. The researcher connects this question to data and an evaluation protocol. They provide more than a score: they explain the task, the conditions for success and the technical choices that influenced the result. Public research by the AI Security Institute and Meta's benchmarks illustrate this experimental measurement of cyber capabilities.

The scope can combine software development, knowledge of vulnerabilities and experimentation with AI agents. Google DeepMind's job advertisement on secure code brings these activities together with work on industrialisation. This profile covers the research work, without including the deployment of a solution for a client. When hiring, specify the scientific lead, the cyber specialists the researcher will work with and the person who will decide how to act on the results. These contexts do not establish a single reporting line.

AI capabilities researcher, penetration tester and AI engineer: who does what?

• The researcher studies the capabilities of models. Their deliverables are evaluations, data, prototypes and an analysis of the results.
• Operational penetration testing and vulnerability research are not the purpose of the role described here. Cyber techniques can, however, be used to build its experiments.
• Deploying an AI solution is a responsibility that needs to be defined separately. A research prototype does not validate a production use.

The purpose of this hire is to obtain conclusions whose scope management can understand. An evaluation should answer a specific question about a model, a task and execution conditions. It can guide the choice of a new experiment or the examination of a possible use. On its own, it does not establish real-world risk or effectiveness in production.

The work of Lukošiūtė and Swanda highlights that risk assessment must also consider adoption by malicious actors and potential impact. The AI Security Institute's programme, for its part, seeks to connect evaluations with real-world risks. For a business leader, this distinction avoids asking the researcher for a conclusion that their protocol cannot produce. Ask which decision each study should inform, then what additional information that decision would require.

The quality of the protocol matters as much as the reported result. At the AI Security Institute, evaluations can vary prompts and the tools available to agents. Its agenda also notes that quick tests can fail to reveal the upper limit of capabilities. Comparing scores without knowing these conditions therefore risks giving a misleading view of performance. The candidate must be able to explain what was tested and what remains outside the scope of the experiment.

Fictional example: a team compares two agents on code correction tasks. The first has additional tools. Before concluding that one model is more capable, ask the researcher to distinguish the effect of the model from that of the experimental environment. A useful deliverable sets out this limitation and proposes a better-controlled comparison.

A poorly defined role can also create simultaneous expectations of research, cyber auditing and product delivery. Define the responsibilities before selecting the candidate. A negative result remains useful if the experiment is reproducible and the analysis explains what it reveals about the capability being studied. The success of the hire can then be judged by the quality of the knowledge produced, including its limitations.

• Define the cyber capabilities to be measured and the questions the experiments must answer.
• Build datasets suited to the cyber tasks being studied.
• Design benchmarks and metrics to measure the performance of models or agents.
• Develop research prototypes to experiment with agentic techniques.
• Compare variations in prompts and tools, documenting the conditions of each evaluation.
• Analyse results and failures to establish the limits of the capabilities measured.
• Document the environments and data needed to reproduce the experiments.
• Present conclusions and their limitations to technical and non-technical audiences.

The candidate's background should combine learning and experience in research, programming and computer security suited to the assigned tasks. Examine what they can build and explain: a protocol, an evaluation tool, a prototype or a vulnerability analysis. A qualification indicates training, but does not in itself describe autonomy in conducting an experiment.

The documented routes vary by role. Google DeepMind mentions a doctorate in a technical field or equivalent practical experience for its research role on secure code. The AI Security Institute particularly highlights Python programming and cyber experience, with support from the manager and technical leads. These examples do not make a doctorate compulsory across the occupation.

Past work can take several forms. AI publications shed light on scientific work. Tools, machine learning applications or work on vulnerabilities demonstrate other learning and experience. Participation in CTFs, their design or contributions to bug bounty programmes are also among the experiences cited by AISI. Candidates do not need to meet all these criteria: select those that match your research programme.

For a role with support, look for the ability to carry out and explain a defined experiment. For an autonomous role, examine the responsibilities already held in choosing tasks, building evaluations and interpreting results. Also identify the skills available in the team to complement those of the candidate. The need for knowledge of fine-tuning or reinforcement learning depends on the planned research and should not become an automatic requirement.

To assess this profile, you can examine the following criteria using public work, an authorised presentation or a fictional case suited to your research.

1. Define the criteria for the role

Separate the expected capabilities: experimental design, programming, understanding cyber tasks and analysing results. State which must be mastered on arrival and which can be developed with the team.

Ask the candidate to restate a research question related to the role. A positive response names the capability to be measured, the task and the decision the study could inform. Be alert to promises of operational effectiveness that go beyond what the experiment could establish.

Involve a model evaluation specialist and a cyber specialist in this assessment. If your company does not have these skills, bring in an external expert to assess the protocol and the relevance of the tasks.

2. Examine a past piece of work

Choose work the candidate is authorised to present. Ask what they personally designed, developed and analysed. Reconstruct the initial question, the data used, how the prototype worked and the success criteria.

Examine how the candidate documents the inputs, the tools available to the model and the outputs of the experiment. Ask what another person would need to know to reproduce the result. Reproducibility and checking that a task is solvable are principles documented by Inspect Cyber.

A positive signal is an explanation that connects each conclusion to an observation. A concern is an inability to distinguish the candidate's results from those of the team, or to specify the test conditions.

3. Discuss a focused practical case

Fictional example: your organisation wants to compare two models on tasks involving fixing vulnerabilities. Provide an authorised working environment and ask the candidate to propose an evaluation, without requiring a solution usable in production.

Ask them to explain the choice of tasks, how the data would be assembled and the metric selected. Ask how they would check that the tasks are solvable. Have them specify the conditions that must remain comparable between trials.

Then introduce an unexpected result. Ask what checks would help explain the failure and what new experiment would be useful. Assess the approach rather than the ability to name a cause immediately.

A positive proposal makes clear what will be measured and what will remain unknown. A protocol whose conclusion is fixed before the experiment is a concern.

4. Assess communication and coordination

Ask for a short presentation for a business leader, followed by a technical discussion of the same result. Observe whether the candidate retains the study's limitations in both versions.

Present a situation where cyber and AI specialists do not share the same interpretation. Ask how the candidate would clarify the disagreement and what observation might help resolve it. If the role includes management, also examine the responsibilities they have held in supporting other researchers.

A clear explanation of uncertainty is a positive signal. A response that consistently turns a score into proof of general effectiveness calls for further examination.

5. Cross-check past responsibilities

With the candidate's consent, speak with people who have observed their work. Focus the questions on their autonomy, their contribution to experiments and how they discuss conclusions.

Cross-check the responsibilities described during the interview, without requesting protected data or results. Use these conversations to clarify a specific point, such as the choice of metrics or the conduct of a study. Keep observed facts separate from your contacts' assessments.