Resources · Framework
Integrator Evaluation Scorecard
This scorecard weights the ten criteria that decide how an integration project goes. Three criteria are pass/fail, because strength elsewhere cannot make up for failing them. We expect buyers to use it to assess us, and we publish it for that reason.
Score one vendor
UNIT 01Complete one scorecard for each vendor and compare the printed copies side by side. Score the answer the vendor actually gave you. Do not score the answer you think they would give if you pressed them. Most of the scorecard's value comes from that difference.
Used only to label your printed copy. Nothing is sent to us or anyone else.
They have worked on equipment like yours
Pass / failAsk: Which controllers, protocols and equipment vintages have you actually read data from, and where?
Good answerSpecific makes, models and protocols, including the difficulties: a licence they had to buy, a bus they could not connect to, an OEM that refused access.
Bad answerThey describe sector experience instead of equipment experience. "We've done a lot of work in manufacturing" does not answer this question.
A contactable reference for similar work
Pass / failAsk: Can I speak to a client where you did similar work, and may I ask them what went wrong?
Good answerA name and a phone number, and no hesitation about the second part of the question.
Bad answerA page of client logos, a confidentiality agreement cited for every project, or a reference who bought something quite different.
You can leave with your own system
Pass / failAsk: If I engaged somebody else next year, what would I hold (source, configuration, credentials, data) and what would I be missing?
Good answerA clear list, given without hesitation, and willingness to include it in the contract as a deliverable.
Bad answer"That's never come up." It certainly has, and the vendor is avoiding the question.
They have clear reasons for saying no
Weight 3Ask: What would make you tell me not to do this?
Good answerA real, recent example of a project they declined or reduced in scope, with the reason, where the decision cost them revenue.
Bad answerA promise that they always find a way. An integrator that never says no will build whatever you ask for, including the wrong thing.
They ask for documentation before quoting
Weight 2Ask: What do you need from me before you can give a price you would stand behind?
Good answerAn asset schedule, a tag list and a site visit, plus a clear statement of what the price covers if any of these prove wrong.
Bad answerA price on the first call. You will pay for that price later, either as a contingency or as a contract variation.
Handover is specified in writing
Weight 2Ask: List the documents, training and other deliverables at handover, and who signs them off.
Good answerA named list with an acceptance step: as-built drawings, tag schedules, credentials transfer and recorded training.
Bad answer"Full documentation" as an undefined line item. Its content is then decided at handover, when both sides want to finish quickly.
They raise network segmentation before you do
Weight 3Ask: How will this change what can be reached on my network, and what do you propose to do about it?
Good answerThey raise it without being asked, ask how your networks are currently separated, and include a network boundary in the price.
Bad answerSilence, or a promise that everything is behind the firewall. Integration connects networks, so an informal boundary starts to carry real risk.
They will quote the running cost
Weight 2Ask: What will this cost to own in year three, including licences, support, hosting and my own staff's time?
Good answerA figure, with the assumptions behind it and how it will change as the system grows.
Bad answerA build price only. Running cost is typically 15% to 22% of the build cost each year. A vendor that will not estimate it has not planned beyond the first invoice.
You know who will do the work
Weight 2Ask: Who will be on site, what have they done before, and do you employ them directly?
Good answerNamed engineers, their background, and a direct answer about subcontracting.
Bad answerThe sales team attends, and the delivery team is not named. Most disappointment starts with the difference between the sales team and the delivery team.
Changes are priced before they are needed
Weight 1Ask: How do you handle changes during the build, and what are the process and the rate?
Good answerA written process and a rate agreed in advance, with a clear distinction between a clarification and a variation.
Bad answer"We'll be reasonable." Being reasonable is not a rate. You will negotiate it later, when you are in your weakest position.
0 of 10 answered. Your result appears when you answer the last one.
Result
UNIT 02Weak
0%Weighted score
This vendor scores poorly on the criteria that decide the outcome
Score every criterion to get a result. The scorecard counts an unscored criterion as zero, because an unanswered question is not neutral.
Save your result
↓Your result as plain text. You can paste it into an email, a board paper or a request for quotation without giving us your email address. It stays clear when you forward it to a colleague who has not seen the tool.
Why we publish this scorecard
UNIT 03We publish it for a commercial reason. Buyers who choose integrators on price and headcount often choose badly. The project disappoints, and the next buyers in that sector become more cautious. The criteria in this scorecard decide how an integration goes, and we prefer to compete on them.
Three criteria are pass/fail because no strength elsewhere offsets them
Weighting would let a high score elsewhere hide a failure on these three. A vendor who cannot show evidence of work on equipment like yours has not shown they can do the job. A vendor with no reference you can contact is asking you to trust the last five years without proof. A vendor who will not say what you keep if you leave has shown you how the relationship will work. A competitive price does not make up for any of these.
The criterion we most want buyers to ask us about
Criterion four asks: “What would make you tell me not to do this?” It is not pass/fail, but it shows most clearly whether a firm is led by engineering or by sales. Two of our free calculators, the payback calculator and the predictive maintenance screener, are designed to return “don't” when the numbers do not support a project. That is our most direct answer to the question.
Technical references
UNIT 04These are the sources for the reasoning on this page. We mark standards that you must pay to read. Where a free document covers the same subject better, we list it first.
- ISA/IEC 62443Paid standardISAThe series of security standards for industrial automation. Customers and insurers usually cite it when they ask how your operational technology (OT) is secured.
- NIST SP 800-82 Rev. 3NISTGuide to Operational Technology Security. It is free, several hundred pages long and the most useful single document in this field. Our segmentation planner follows its model of zones and conduits.
- ISO 42001 explainedISOISO's free summary. Read this first to decide whether certification is relevant to you before you buy the standard.
Links open in a new tab, so you do not lose anything you have entered on this page. We checked every link when we built the page. If a link no longer works, please tell us and we will remove it.
Each of these tools is a short version of a conversation with one of our engineers. If your result raised a question you want to discuss, book a scoping call and bring your result with you.
More resources
- Framework
The Readiness Model
The full method behind the readiness assessment. For each of the five dimensions, it explains what it measures, how it fails and what it costs to fix before a build.
- Framework
Spec template
The sections an integration brief needs before you send it to vendors, so that every quote you receive describes the same job.
- Framework
Pilot to production
The checks a working pilot must pass before it runs live operations, and the ones where most automation projects at small and mid-sized firms fail.