Resources · Framework
Integrator Evaluation Scorecard
Ten criteria that determine how an integration actually goes, weighted, with three marked pass/fail because no amount of strength elsewhere compensates for them. It is published knowing it will be used on us, which is the only condition under which publishing it means anything.
Score one vendor
UNIT 01Run it once per vendor and keep the printed copies side by side. Score the answer you were actually given, not the answer you expect they would give if pressed — the difference between those two is most of the value here.
Only used to label your printed copy. Nothing is sent anywhere.
They have worked on equipment like yours
Pass / failAsk: Which controllers, protocols and equipment vintages have you actually read data from, and where?
Good answerSpecific makes, models and protocols, with the awkward parts named — a licence they had to buy, a bus they could not tap, an OEM who refused.
Bad answerSector experience offered instead of equipment experience. "We've done a lot of work in manufacturing" is not an answer to this question.
A reference doing the same work, reachable
Pass / failAsk: Can I speak to a client where you did something structurally similar, and may I ask them what went wrong?
Good answerA name, a phone number, and no anxiety about the second half of the question.
Bad answerA logo wall, an NDA cited for everything, or a reference who turns out to have bought something quite different.
You can leave with your own system
Pass / failAsk: If I engaged somebody else next year, what would I hold — source, configuration, credentials, data — and what would I be missing?
Good answerA clear list, offered without hesitation, and a willingness to put it in the contract as a deliverable.
Bad answer"That's never come up." It has. The answer is being avoided rather than assembled.
They have a threshold for saying no
Weight 3Ask: What would make you tell me not to do this?
Good answerA real example of an engagement they declined or scoped down, and why. Specific, recent, and costly to them.
Bad answerAn assurance that they always find a way. An integrator with no threshold will build whatever is asked for, including the wrong thing.
They ask for documentation before quoting
Weight 2Ask: What do you need from me before you can give a price you would stand behind?
Good answerAn asset schedule, a tag list, a site visit, and a stated position on what the price covers if those turn out to be wrong.
Bad answerA price on the first call. That is either a contingency you will pay for, or a variation you will pay for later.
Handover is specified, not promised
Weight 2Ask: List the documents, training and artefacts delivered at handover, and who signs them off.
Good answerA named list with an acceptance step. As-builts, tag schedules, credentials transfer, recorded training.
Bad answer"Full documentation" as a line item, undefined. It is defined at handover, by whoever is more tired.
They raise segmentation before you do
Weight 3Ask: How will this change what is reachable on my network, and what do you propose about it?
Good answerThey bring it up unprompted, ask about your current separation, and price a boundary into the scope.
Bad answerSilence, or an assurance that it is all behind the firewall. Integration is what makes an informal boundary load-bearing.
They will quote the running cost
Weight 2Ask: What does this cost to own in year three — licences, support, hosting, and my own people's hours?
Good answerA number, with the assumptions behind it and what happens to it as the system grows.
Bad answerA build price only. Running cost typically lands between 15% and 22% of the build each year, and a vendor unwilling to estimate it has not thought past the invoice.
You know who is actually doing the work
Weight 2Ask: Who will be on site, what have they done before, and are they employed by you?
Good answerNamed engineers, their background, and a straight answer about subcontracting.
Bad answerSales attends, delivery is unnamed. The gap between the pitch team and the delivery team is where most disappointment originates.
Change is priced before it is needed
Weight 1Ask: How are changes handled during the build, and what is the mechanism and the rate?
Good answerA stated process and a rate agreed up front, plus a clear line between clarification and variation.
Bad answer"We'll be reasonable." Reasonableness is not a rate, and it is negotiated from your weakest position rather than your strongest.
0 of 10 answered — the result appears when the last one is in
Verdict
UNIT 02Weak
0%Weighted score
This vendor scores poorly on what determines the outcome
Score every criterion to get a verdict. An unscored criterion counts as zero here, which is deliberate — an unanswered question is not a neutral one.
Take it with you
↓The result as plain text. Paste it into an email, a board paper, or a request for quotation — no address needed, and it stays readable when it is forwarded to somebody who was not in the room.
Why publish this
UNIT 03There is a commercial argument and it is not modesty. A buyer scoring integrators on price and headcount picks badly, the project disappoints, and the next three buyers in that sector are harder to sell to. The criteria above are the ones that actually determine how an integration goes, and we would rather compete on them.
Why three criteria are pass/fail rather than heavily weighted
Because weighting lets them be averaged away, and they cannot be. A vendor unable to evidence work on equipment like yours has not shown they can do the job. One with no reachable reference is asking you to take the last five years on trust. One who will not say what you hold if you leave has told you what the relationship is going to be. None of those is offset by a competitive price.
The criterion we would most like to be asked
“What would make you tell me not to do this?” It is number four, it is not pass/fail, and it is the one that separates an engineering firm from a sales operation. Two of the free calculators on this shelf — the payback calculator and the predictive maintenance screener — return “don't” by construction, which is the most direct answer to that question we can give without you having to ask it.
Technical references
UNIT 04What the reasoning on this page is drawn from. Where a standard costs money to read it is marked, and where a free document covers the same ground better it is listed first.
- ISA/IEC 62443Paid standardISAThe industrial automation security series. What a customer or insurer will name when they ask how your OT is secured.
- NIST SP 800-82 Rev. 3NISTGuide to Operational Technology Security. Free, several hundred pages, and the most useful single document in this field. The segmentation logic in our planner follows its zone and conduit reasoning.
- ISO 42001 explainedISOISO's own free summary. The right first read if you are deciding whether certification is relevant before buying the standard.
Links open in a new tab so anything you have entered above survives. Every one was checked at build time; if one has rotted since, tell us and it comes out rather than getting patched from memory.
Every one of these tools is a compressed version of a conversation. If yours turned up something you would rather talk through than read about, that is what the scoping call is for — bring your result with you.
Score us against this, then talk to usAlso on the shelf
- Framework
The Readiness Model
The five dimensions behind the readiness assessment, written out in full: what each one measures, how it fails, and what it costs to fix before a build rather than during one.
- Framework
Spec template
The sections an integration brief needs before it goes out to vendors, so that three quotes come back describing the same job.
- Framework
Pilot to production
The gates a working pilot has to clear before it is allowed to run the business, and the specific ones most SME automation dies at.