An IoT platform demonstration usually begins with a device that already works. Your evaluation should begin one step earlier, with an unclaimed unit, an ordinary account, and the instructions a customer will receive. That small change reveals work a polished presentation can skip. It also gives competing products the same starting point, which makes the comparison more useful than a collection of feature checkmarks.
This guide is for an engineering or operations team evaluating a device platform. It proposes a test plan, not a vendor ranking. Use a small workload and hardware you are authorized to test. Record the product version, account tier, configuration, and date. Results belong to that environment; a successful trial is evidence for the next decision, not proof of every future deployment.
Define the job and the exit conditions
Write down the first workflow the platform must support. It might be onboarding a new device, assigning it to a customer, receiving a reading, and granting a support user limited access. Name the people involved and the system boundaries. If your application already handles billing or identity, state that clearly so the evaluation does not reward features you do not need.
Choose a small set of pass, fail, and investigate outcomes before the demo. “A support user cannot change ownership” is testable. “The interface feels enterprise-ready” is not a stable acceptance condition. Leave room for observations that matter but are not blockers. The purpose is to distinguish a missing requirement from a preference that could reasonably be accommodated.
Onboard a device from a clean start
Use a device that has not already been prepared by the vendor. Follow the documented route to create or claim its identity. Note where credentials come from, how they are installed, and who can replace them. Test an incorrect claim code and an already claimed device. A useful evaluation records the recovery path, not only the successful final state.
Then ask a colleague who did not attend the demonstration to repeat the procedure. Time can be recorded for your own comparison, but the more useful observations are where that person needed help and what information was missing. Check whether the customer-facing workflow requires privileges that only an internal engineer should hold. A convenient trial account can conceal that problem.
Prove the permission boundary
Create separate administrator, installer, and read-only support users if those roles match your needs. Try the same operations with each account. Include device reassignment, configuration changes, exports, and credential access. Record both the interface behavior and the API result where an API is part of the product. A hidden button alone does not establish an authorization boundary.
For a multi-customer service, create two test customers and verify that a user from one cannot access the other's devices or reports. Keep the test within an authorized environment and agreed scope. The NIST device cybersecurity capability baseline offers a useful set of topics for organizing access and configuration questions, while your deployment determines the actual requirements.
Interrupt the connection deliberately
Disconnect the test device using a safe method suitable for the hardware. Observe when the platform marks the last contact and how it distinguishes an absent reading from a measured zero. Reconnect and inspect any delayed messages. The timestamp for the observation and the time the platform received it may answer different questions, so both need clear meaning in your application.
If MQTT is involved, consult the OASIS MQTT specification when discussing delivery and session behavior. Ask which protocol version and settings the implementation supports. Protocol delivery semantics do not automatically settle application deduplication, event ordering across devices, or the business meaning of a repeated message. Test the cases your workflow actually depends on.
Exercise a configuration change and an update
Choose a harmless configuration change and follow it from request to device acknowledgment. What does the operator see while the device is offline? Can a newer change replace a pending one? Is there a record of who made the request? Write down what each status means. A status called “complete” should have an explanation the team can rely on.
For software updates, ask the vendor to demonstrate a supported test path on suitable hardware. Discuss failure recovery, version visibility, and how the team limits the initial rollout. Do not interrupt an update on production equipment as an improvised experiment. The evaluation should establish a safe, agreed method for learning how failures are handled before the product is selected.
Build a cost worksheet from a real sample
Count the devices, messages per device, average payload, retained history, exports, and user seats in your sample. Ask which of those dimensions affect the bill. Include development environments and inactive devices. Some costs may sit outside the platform, such as connectivity or a separate data store, so keep a column for costs the vendor quote does not cover.
Run several hypothetical workloads using stated assumptions. One might use the expected reporting interval; another might increase message frequency during troubleshooting. These are planning scenarios, not forecasts. Ask the vendor to confirm how each scenario would be metered and which limits would apply. Keep the written answer with the evaluation so the commercial decision has the same context as the technical one.
Test the support and departure paths
Open a normal support question during the trial and inspect the information needed to investigate it. Can your team retrieve relevant logs without sending credentials or unnecessary customer data? Is there a documented way to report a security issue? Make sure the support arrangement applies to the tier under consideration, rather than the special attention available during a sales demonstration.
Export the device inventory and a sample of readings, then read them with a separate tool. Check identifiers, units, timestamps, and missing-value conventions. Ask what happens to access and retained data when the subscription ends. A practical departure plan helps assess how much of the business would depend on platform-specific behavior, even if the intention is to stay for years.
Close with evidence, owners, and open questions
Summarize each required workflow with its outcome and a link to your internal test record. Assign an owner to unresolved questions and distinguish a promised feature from one you exercised. If an integration needs custom work, estimate that work separately instead of letting it disappear into a general impression of product fit. The final comparison should explain the effort your team would inherit.
A useful decision meeting can be short when the evaluation is concrete. Review blockers first, then operational obligations and commercial assumptions. Choose the next bounded step: reject, investigate, or run a broader pilot with explicit criteria. Keep the checklist for that next stage and revise it when evidence changes. A repeatable test plan is valuable because it survives the presentation and follows the product into real work.