Choosing AI tools for IT operations: a vendor-neutral checklist

Somewhere in your inbox right now there's a vendor promising that their AI will "transform your service desk." There were three last month. There will be four next month.

Most IT teams pick AI tooling the same way: someone sees a demo, gets excited, starts a trial, and six weeks later the team has a half-adopted tool, a new line item, and a vague sense that it isn't doing what the demo did. The problem usually isn't the tool. It's that nobody wrote down what the tool had to be true about before the demo started.

Here's the evaluation approach we recommend, in the order that actually matters.

Start with the workload, not the tool

Before you look at a single product, name the work you want to move. Not "we want to use AI" but something you can point at in your ticket queue. First responses on password resets. Escalation summaries between tier 1 and tier 2. Turning resolved tickets into knowledge base articles.

This sounds obvious. Almost nobody does it. And it changes everything downstream, because a tool that's excellent at drafting customer replies can be useless at log analysis, and the demo will never tell you which one you're buying.

Write down your top three workloads. If a tool doesn't clearly serve at least one of them, the evaluation is over, no matter how good the demo looks.

The questions that disqualify tools fast

Data handling comes first, and it's binary. Where does your data go when your engineer pastes a ticket into this tool? Is it used for model training? Can you turn that off, and is it off by default on the plan you're actually buying, not the enterprise plan they show in the security PDF? If the vendor can't answer in plain language, that's your answer. A surprising number of evaluations should end right here, before pricing ever comes up.

Then workflow fit. The honest test is whether the tool works inside the place the work already happens. Every extra tab costs adoption. A mediocre assistant inside your PSA or ITSM tool will beat a brilliant one that lives in a separate browser tab, because your team will actually use the mediocre one on a busy Tuesday.

Then the exit cost. What happens when you leave? If your prompts, workflows, and generated documentation are locked in a proprietary format, the real price of the tool is the subscription plus the cost of leaving it. Prefer tools where your assets survive the relationship.

Score it, briefly

You don't need a procurement framework. A simple sheet with your three workloads and five criteria (data handling, workflow fit, output quality on your real tickets, exit cost, price per seat at the size you'll actually run) beats gut feel and beats a 40-row RFP equally.

One rule makes the scoring honest: test with your own tickets, not the vendor's examples. Take five real tickets, anonymise them first, and run them through the trial. Vendor demo data is chosen to look good. Your data is chosen by reality.

We keep a one-page version of this evaluation as a cheat sheet and a fuller pre-purchase checklist in the Opstimio library, mostly because teams kept asking for something they could hand to a colleague before the next vendor call. But the short version above will already filter out most of the bad decisions.

The traps we see most

The per-seat creep trap. The pilot is ten seats. The rollout is sixty. The price that looked harmless at pilot size quietly becomes your third biggest software line. Model the full-team cost before the trial starts, not after.

The "it does everything" trap. Tools that claim ticket handling, documentation, monitoring, and reporting in one product are usually adequate at all of them and excellent at none. You're allowed to buy nothing. You're also allowed to solve one workload well and stop there.

The pilot-that-never-ends trap. Set a decision date when the trial starts. Four weeks is enough for a service desk workload. If nobody would fight to keep the tool at week four, you have your answer, and the kindest thing you can do is cancel before it becomes furniture.

What good looks like

A good AI tool decision in IT operations is boring. The team knows which workload it serves. The data policy is written down and someone has actually read it. The output has been tested on your own tickets. Finance knows the cost at full rollout. And there's a date in the calendar to review whether it's still earning its seat.

None of that requires a bigger budget. It requires half a day of thinking before the demo instead of after the invoice, and that's a trade most teams only learn to make once.