Applied AI Engineer, Coding Agents

An applied AI engineer specialising in coding agents designs and improves systems that can complete tasks in a software repository. Such an agent receives a task, uses tools in an execution environment, acts in several steps and may change code. Its output can be checked through tests, among other methods. The role concerns how the agent works and the quality of its output, beyond the day-to-day use of a coding assistant.

Depending on the organisation, the engineer may work on the loop that coordinates the agent’s actions, the tools it can access, its instructions and safeguards, or the evaluation of its results. These responsibilities may sit with separate teams. At Cursor, for example, job listings distinguish agent-system design from evaluation and quality. The expected scope should therefore be defined before looking for a candidate.

The system handles programming tasks such as writing tests, fixing bugs and adding features. The engineer in this role does not necessarily make every code change requested of the agents themselves. They build the conditions in which those changes can be carried out, reviewed and improved.

Coding-agent engineer, software engineer and evaluation specialist: who does what?

• Coding-agent engineer: works on the execution loop, tools and measurement of the agent’s quality, according to the agreed remit.
• Software engineer: may integrate or use an agent in a product without ongoing responsibility for its behaviour.
• Evaluation specialist: focuses on test tasks, criteria, regressions and failure analysis. This responsibility may form a separate team.

This distinction helps define the hire. It does not fix job titles or how companies divide responsibilities.

A coding agent can act on a repository and alter its working environment. Its reliability therefore depends on how it acts as well as on the visible result. Convincing behaviour in a demonstration does not establish reliability on other tasks. Tests, execution traces and failure analysis help distinguish a real improvement from an effect confined to a few cases.

The first decision concerns scope. If the problems involve the sequence of actions, tools or execution environment, the work resembles agent-system design. If results are hard to measure or regressions are the main concern, evaluation needs strengthening. Cursor’s job listings illustrate these two specialisms, without requiring every company to divide the work this way. A job profile that combines all these missions without priorities makes the expected autonomy hard to assess.

The second decision concerns what an evaluation measures. The chosen tasks, agent configuration and criteria affect the result. Microsoft states that, for its Scope platform, a score is not a universal ranking and an automated judgement may need human review. Recruitment should therefore favour someone who can explain what their tests demonstrate and what remains unresolved.

Fictional example: an agent completes a series of short fixes but fails when a task requires reading several files and running tests. Before changing the model, the engineer examines the available tools, the steps taken and the failure traces. They then change one part of the system and rerun the tasks to see whether the improvement holds.

Finally, perceived quality, reliability and cost can pull in different directions. Choices about models, tools and behaviour depend on the product and its users’ constraints. The candidate should be able to justify those choices with the available data, without presenting a technical preference as a general truth.

• Design the execution loop that coordinates the agent’s steps and tool calls.
• Choose the tools available to the agent and define their functions and limits.
• Adapt the instructions and execution environment to the intended programming tasks.
• Define representative tasks and criteria for evaluating changes in behaviour.
• Run tests and evaluations to identify regressions after a change.
• Analyse traces and failures to distinguish problems with a tool, context or behaviour.
• Interpret usage signals to prioritise improvements relevant to the product.
• Explain trade-offs between the quality of results, user experience and cost in context.

No single degree defines this specialism. A background in software development can provide command of code, tests and execution environments. Experience in tool design can prepare someone to define the functions available to an agent and the context those tools return. Evaluation engineering develops the ability to build representative tasks, interpret traces and spot regressions. These skills can be combined in different ways; no single qualification or length of experience is required for this role. Work on code, tests or tools can show progression towards this specialism, even if the previous job had a different title. The scope of each piece of work matters: contributing to a tool, building an evaluation system and taking responsibility for an agent’s behaviour call for different levels of autonomy. This distinction helps define the level sought without requiring a uniform career path.

The expected level depends on the remit. Improving a specific component requires less autonomy than defining a product’s evaluation programme. Development experience may be relevant if it includes work on how the agent operates or on its quality, beyond using it. Previous job titles tell you less than the responsibilities actually held.

To assess this profile, follow these steps and link the role’s skills to observable evidence.

1. Set the criteria before interviews

Specify whether the role mainly concerns the execution loop, tools or agent quality. Choose a few criteria tied to that remit: code quality, tool choices, test design, reading traces and judgement about results. State the expected level of autonomy for each decision. During interviews, a positive sign is a candidate’s ability to connect each criterion to a real product task. Be cautious if the role combines responsibilities without priorities or anyone designated to validate the results.

2. Examine previous work

Ask the candidate to describe a change they made to an agent or its evaluation system. Ask them to clarify the task, available tools, tests and failures encountered. Establish what they decided themselves and what their team handled. A strong account also explains a limitation: for example, why a result on some tasks does not predict all others. If they attribute an improvement to a single change without examining other parameters, explore their approach further.

3. Offer a case close to the role

Use a small fictional repository or an evaluation pack made for the exercise. You could ask the candidate to choose which tools an agent should have, define a success test or analyse a failure trace. Allow a reasonable amount of time and give every candidate the same materials. Fictional example: an agent changes the right file but fails to run a test that would have revealed a regression. Ask what information is missing, what change to try and how to check its effect. Assess the approach and the limits the candidate recognises, without making the number of lines produced the sole criterion.

4. Observe communication and trade-offs

Invite the candidate to explain their choices to someone responsible for product and someone responsible for engineering. Ask them to distinguish measured quality, user experience and cost when these matter in your context. A positive sign is a decision supported by observed tasks and explicit constraints. Be wary of a general conclusion drawn from a single score. Evaluation results depend on the tasks and configurations tested; an automated judgement may need human review.

5. Corroborate the evidence

With the candidate’s consent, ask a professional referee what scope they actually held: design, testing, diagnosis or coordination. Seek examples of decisions and collaboration rather than a vague appraisal. If your company lacks the expertise to judge the technical case, involve an experienced software engineer or agent-evaluation engineer. Give them the criteria grid and ask for written reasons for their observations. Then compare the evidence gathered with the initial remit, without treating a limited exercise as a universal judgement.