An MLOps Audit Checklist for Flight Training Teams

An MLOps Audit Checklist for Flight Training Teams

Flight students meet checklists on day one and soon learn that a checklist is only useful if it is run honestly. The same is true of the quiet routines that keep a machine learning model trustworthy after it goes live. This audit sheet turns the Ops Atlas description of MLOps into questions an instructor, an operator or a technically minded student can put to any team that relies on a prediction model, for example one that estimates training demand, schedules aircraft or flags students who might need extra support. Work through it in order, answering yes, no or “not sure”, and treat every “not sure” as a no.

How to use the audit

Each section lists items to verify and the evidence to ask for. The reference material is the Ops Atlas entry covering MLOps, which frames the discipline as the work of putting trained models into service and then keeping their answers right. It does not describe a product; it describes habits. A team that can show proof for the items below is practising the habits, whatever tools it uses.

Score generously where evidence exists and strictly where it does not. Half-credit for work done manually and full credit for work that is automated is a convention borrowed from a published readiness rubric the entry mentions, which also lets the weakest section set the overall result, so a single strong area cannot excuse a weak one.

Section A: Lineage and records

  • Can the team name, for the model in service today, the code revision that built it?
  • Can they identify the data used, through a fingerprint or snapshot identifier and not a vague description?
  • Are the settings and the software environment recorded alongside?
  • Has anyone rebuilt the model from those records and compared the results within an agreed tolerance?

The last item deserves emphasis. The entry defines reproducibility as metrics within tolerance, not identical output, because training can involve randomness. Ask for the tolerance in writing. If nobody can quote one, record a no.

Section B: Consistency between learning and live use

  • Is each input computed from one shared definition, or are there two implementations?
  • Are the inputs used at decision time logged for later comparison?
  • Is there a replay test that runs recorded requests through both paths?
  • What is the tolerated mismatch, and is it zero?

Mismatches here are called skew in the Atlas, and the correct response is to fix the pipelines, not to train again. An auditor who sees a team retraining as a first response to odd results should note it.

Section C: The release gate

  • Is the comparison made on a fixed test set split by time, so the candidate cannot have seen the answers?
  • Does the gate compare more than one overall number: slices of the population, speed and the format of outputs?
  • Are the pass marks written down before the test, such as a minimum improvement, a maximum allowed fall for any slice and a response-time ceiling?
  • Has the gate ever rejected a candidate? A gate that has never said no deserves suspicion.

The entry’s worked example is a handy yardstick. In that lab a newcomer improved overall AUC by 0.009 while a single slice slipped 0.029, beyond the 0.02 that policy allowed, so the newcomer waited until the slice recovered. If your team can narrate a similar story, the gate is real. The same entry cites a review of leakage in published research, with errors reaching 294 papers across 17 fields, to explain why careless test design is a persistent hazard.

Section D: Introducing a new model

  • Does a new model first run in shadow mode while users see only the old answers?
  • Is there a canary stage with a small share of traffic?
  • Is there an online experiment that measures the real outcome and not only model scores?

Ask why the last step exists. According to the Atlas, one travel-booking company found that better model metrics did not guarantee a business gain. If the team has never seen that happen, they have probably not looked.

Section E: Monitoring

  • Are service health, incoming data and prediction quality monitored as three separate layers?
  • For slow labels, is there a proxy, such as the distribution of predictions or their bias against a matured base rate?
  • Are action limits agreed in advance, with a named person for each?
  • Are alert thresholds calibrated on the team’s own history instead of copied from a default?
  • Do critical inputs have their own checks, so a dataset-wide rule cannot hide one bad column?

On thresholds, the entry is blunt: popular drift cut-offs are conventions and the underlying value depends on how data is binned and how much there is. Its reference implementation suggests alerting on a feature drift score above 0.2 only after calibration, paging for schema violations after fifteen minutes and flagging a statistics job stale after 26 hours. Treat those as examples of specificity, not as numbers to copy.

Diagnosis questions

  • When an alert fires, does the team ask first whether the pipelines disagree, whether upstream data is defective, or whether the world genuinely changed?
  • Do they know which of the three allows retraining as the remedy?

Section F: Retraining policy

  • Is there a written reason for the retraining cadence?
  • Has a staleness test been run: train to a date, then measure week-by-week decay?
  • Does the team know which triggers apply? Google Cloud lists on demand, schedule, new data, degraded performance and distribution change.
  • Is automatic promotion avoided, or at least gated and watched?

The Atlas lists self-promoting nightly retrains as an anti-pattern because feedback loops can cause outages. Cost and training time matter here, much as they do in flight training, where the number of hours drives the bill, a topic our piece on why logged hours move the final training bill unpacks. Retraining that is too frequent wastes money in just the same way.

Section G: Rollback and fallback

  • Is the previous model version still registered and addressable?
  • Does rollback mean switching an alias or traffic split, rather than starting a fresh training job?
  • Is there a plain non-ML fallback?
  • When was the rollback last rehearsed? The entry suggests quarterly.

Pilots will appreciate this one. A checklist that has never been practised does not exist; it is a document.

Section H: Ownership and retirement

  • Is there a named owner for each model?
  • Is there a list of every system that consumes its output, with access control on those outputs?
  • Is there a defined point at which the model is retired?

The Atlas draws on two widely cited papers on hidden technical debt to explain why undeclared consumers make retirement impossible. If the answer to the consumer question is “we are not sure”, you have found your first priority. For a sense of how reliability practices are taught more broadly, our guide to a roadmap for learning site reliability offers context.

Evidence to request during the audit

An audit without paper is a conversation. Ask for the registry page for the live model, a screenshot or export of the last gate result, the most recent rollback rehearsal note, and the consumer list. Ask to see one alert that fired in the past quarter and what the team did about it. If the answer is a shrug, that tells you as much as any score. If a team has a lot of material, sample it: pick one model at random and trace it from training record to live alias, noting every point where the trail goes cold.

Be fair about context. A school with one small scheduling model does not need the apparatus of a large operator, and the entry itself says the level of automation is a choice. What every team needs, whatever its size, is a believable answer to three questions: how was this built, how do we know it is still right, and how do we undo it?

Scoring your results

  1. Count the yes answers in each section separately.
  2. Identify your weakest section; that is your overall rating, in keeping with the minimum-of-sections idea.
  3. Fix that section first, then repeat the audit in a quarter.

For teams starting out, the Atlas counsels modest goals: a pipeline that reruns cleanly, a registry showing ancestry and an escape hatch will repay you better than a polished platform that nobody touches. It also reminds readers that the level of automation is a decision and that not every model needs full automation; Google’s own first rule is to avoid launching with machine learning when it is not needed. If the audit leaves you with many gaps, resist the urge to purchase a platform. Pick the lowest-scoring section, assign an owner and a date, and close one gap at a time.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *