LLMOps Engineer

An LLMOps Engineer, or engineer responsible for large language model operations, deploys and maintains applications that use LLMs in production. They organise evaluations, monitor service operation and prepare version changes. Their work helps product and technical teams understand whether the application still serves its intended use cases, with acceptable response times and costs.

LLMOps refers to a set of practices. When defining the role, specify the applications the engineer will be responsible for and the decisions they can make. The model may be provided through an API or run on the company's infrastructure. In both cases, application integration, response evaluation and production monitoring still need to be organised. The role described here does not involve creating a foundation model.

Operations cover several components: the model, prompts, evaluation data and, where the application uses one, a document retrieval system. This system provides the model with context to generate its response: this is the principle of RAG, or retrieval-augmented generation. The diagnostic process must distinguish a document-related problem from a generation problem.

LLMOps Engineer, MLOps Engineer and Prompt Engineer: who does what?

• The LLMOps Engineer described here focuses on operating LLM applications, managing changes and monitoring them over time.
• The MLOps Engineer provides a point of comparison for shared deployment, testing and version management practices applied more broadly to machine learning.
• The Prompt Engineer works in greater depth on the instructions given to the model. These instructions are one of the components that need testing in an LLM application.

To establish where the role sits, identify the team that actually handles operations: the platform, infrastructure or application team, depending on your organisation. Clarify how it works with product teams, developers, data specialists and security teams. The job title alone does not allocate their responsibilities.

An LLM application can continue to respond even as its results become less useful. Technical availability alone is therefore insufficient to assess service quality. Monitoring must combine performance, resource consumption and examination of responses. Without this combined view, the company risks treating a symptom without understanding which component has deteriorated.

Decisions about changes also require explicit criteria. A model may deliver better results for one use case while responding more slowly or costing more. The challenge is to compare these dimensions against product priorities. Before deployment, suitable evaluations allow teams to examine differences between versions. These evaluations must then evolve with user feedback, without being treated as a guarantee of accuracy.

Hypothetical example: a document assistant provides less relevant responses after a change to its retrieval system. The team immediately considers changing the model. The LLMOps Engineer first proposes comparing the documents passed to the model and the responses generated using the same evaluation dataset. This approach helps locate the deterioration before choosing a correction.

Controlling expenditure requires an understanding of the calls made by the application. Tracking tokens, the units of text processed by the model, and estimating the cost per call inform this analysis. These estimates do not necessarily account for the whole bill. An optimisation decision must therefore remain linked to the result achieved and the expected response time.

Finally, day-to-day operations must account for the risks of prompt injection, data leakage and inappropriate responses. Addressing them requires cooperation with security teams, access controls and testing. Retaining versions and preparing backups contribute to service continuity. Recruiting solely on proficiency with a tool, without examining diagnosis and change management, would leave these responsibilities insufficiently covered.

• Work with those responsible for the product to define evaluations that reflect actual use cases.
• Compare application versions and expand validation datasets using user feedback.
• Automate testing and deployments while retaining the components needed for rollback.
• Monitor performance and responses to identify anomalies in production.
• Analyse token consumption and estimated costs per call.
• Diagnose deterioration by distinguishing document retrieval, the context passed to the model and generation in RAG applications.
• Test risky behaviours and address data access issues with those responsible for security.
• Document versions and prepare the backups needed for service continuity.

Useful foundations for operations include programming, component integration, data pipelines and interpreting metrics. These foundations must be connected to how an LLM application works. Familiarity with a model or a conversational interface is not enough on its own to maintain a service in production.

Experience in development, MLOps or operations can provide a starting point. What someone has learnt varies with the responsibilities they have held: carrying out tests, taking part in a supervised deployment and deciding how to respond to deterioration represent different experiences. Length of experience alone therefore does not determine their level of autonomy in these responsibilities.

Further learning can combine structured training and practical work. The LLMOps course from DeepLearning.AI, developed with Google Cloud, offers practice in areas including versioning, model adaptation pipelines and checking model behaviour. It illustrates one learning route, without constituting a mandatory qualification or demonstrating professional autonomy on its own. For a role that only uses an API, adjust the emphasis placed on model training to the work actually planned.

The foundations of GetPro's assessment method

GetPro structures assessment around a framework of prioritised criteria. It distinguishes what can be verified from a candidate's career history from what needs further exploration in an interview, and specifies an assessment method for each criterion. Open questions are accompanied by requests for concrete examples. Reference checks help clarify the responsibilities held and explore strengths or areas of concern identified during the interview.

Suggested ways to apply the framework when assessing an LLMOps Engineer

The criteria and exercises below are suggestions to adapt to the applications concerned and the decisions assigned to the role. They do not describe a GetPro protocol specific to this profession.

1. Define the criteria before the interview

Specify the components to be operated, the use cases and the support available. Choose observable criteria: building an evaluation, explaining deterioration, preparing a rollback and justifying a cost or performance decision.

Distinguish what the candidate must carry out with support from what they will need to decide independently. For each criterion, ask for a deliverable or an explanation. Avoid turning knowledge of one provider into a general requirement for success.

2. Examine a past piece of work

Ask the candidate to describe an application they helped operate, while respecting their former employer's confidentiality. Have them clarify their personal role, the changes made and the information available when decisions were taken.

Explore an incident or a regression. Ask which signals alerted the team, which hypotheses were examined and how the correction was evaluated. Look for the link between observations and the decision taken.

Treat a clear distinction between facts, hypotheses and limitations as a positive signal. Probe an account that attributes every problem to the model without examining integration or the available data.

3. Set a diagnostic exercise relevant to the role

Hypothetical example: after a change to document retrieval, an assistant responds more slowly and its users report less relevant answers.

Provide a version history, a few call traces and example responses. Ask the candidate what information they are missing before proposing a correction. Invite them to distinguish document ingestion, retrieved context and generation.

Observe whether they choose useful comparisons using the same evaluation dataset. Ask how they would verify a rollback to a known version. Do not make the speed of their answer the only assessment criterion.

4. Ask the candidate to explain trade-offs

Ask the candidate to compare evaluation results, response times and token consumption. Have them specify the business priorities they need to know before recommending an option.

Ask what the cost estimates cover and what limitations the test dataset has. Value a recommendation that is conditional on the available information. Probe any promise of quality or savings that is not based on a comparison the candidate can explain.

5. Assess communication and coordination

Ask for an explanation of the diagnosis aimed at someone responsible for the product. Observe whether the candidate names the problem, its possible effects and the next decision to be made.

Have them clarify which matters need to be addressed with infrastructure, data and security teams. If the role includes management, ask how they would support a colleague in the analysis and allocate responsibilities. Assess this aspect separately from technical proficiency.

6. Cross-check observations and references

Summarise the evidence gathered for each initial criterion. During reference checks, clarify the responsibilities actually held and how the candidate worked with others during changes or incidents.

If your company lacks the technical expertise, involve a professional who can examine evaluations, traces and deployment. Ask business stakeholders to assess the use cases and explanations. Document the points that still need further exploration before reaching a conclusion on autonomy.