What to Evaluate During a Verification & Test Software Trial
Verification and test software can look remarkably impressive in a demonstration.
A test sequence runs automatically. Instruments begin collecting data. Requirements turn green.

Dashboards populate. Reports appear. A hardware-in-the-loop system injects a failure and the controller reacts exactly as expected.
Then everyone goes home thinking:
That worked beautifully.
But that is not the question a software trial needs to answer.
The question is:
Will it work beautifully with your products, your hardware, your requirements, your engineers, your existing test infrastructure, and the ugly exceptions that happen in a real lab?
Verification and test software can sit at the intersection of requirements management, simulation, physical testing, test automation, data acquisition, HIL systems, laboratory hardware, reporting and regulatory evidence.
Changing it later can be expensive.
So don’t evaluate a verification and test platform as though you were trying a new productivity app.
Test the entire workflow.
Start With the Requirement, Not the Test
A test that runs successfully is not necessarily a useful test.
The first thing to evaluate is whether the software can maintain a clear connection between:
Requirement โ verification method โ test case โ test configuration โ test result โ evidence
That chain is fundamental to verification.
Siemens, for example, describes verification management around linking requirements to test cases and maintaining traceability throughout development.
Keysight’s Requirements Verification Suite similarly connects requirements with verification plans spanning simulation and physical testing, records pass/fail outcomes and identifies requirements that remain unverified.
dSPACE SYNECT supports traceability from requirements through test cases, parameters and results across MIL, SIL and HIL testing.
That gives you one of the first practical trial exercises.
Pick a real requirement.
Follow it all the way through your verification process.
Then change the requirement.
What happens?
Can you immediately determine:
- which test cases are affected,
- which test configurations may need revision,
- which existing results are no longer valid,
- which product variants are affected,
- and what needs to be rerun?
If the answer requires three spreadsheets, four meetings and somebody named Bob who remembers how the original test was configured, the software has not solved the problem.
1. Can You Maintain Requirements Traceability?
Requirements traceability sounds administrative until something fails.
Then it becomes one of the most important capabilities in the system.
During the trial, determine whether engineers can easily navigate both directions:
Requirement โ tests
and
Failed test โ requirement
That second direction is particularly important.
Suppose a thermal test fails.
Can the engineer quickly determine:
- which requirement was violated,
- what requirement revision was tested,
- which product configuration was tested,
- which test procedure was used,
- what measurement limits were applied,
- and whether the failure affects related requirements?
NI’s guidance on verification and validation emphasizes detailed specifications and controlling changes to those specifications because verification results are meaningful only when tied to the correct requirements, settings and limits.
During the trial, deliberately revise a requirement.
Don’t tell the vendor representative exactly what downstream effects you expect.
See whether the system finds them.
2. Evaluate Verification Planning
Complex products may have thousandsโor tens of thousandsโof requirements.
Some are verified through:
- analysis,
- simulation,
- inspection,
- demonstration,
- physical testing,
- software testing,
- MIL,
- SIL,
- HIL,
- or combinations of several methods.
Can the platform tell you how each requirement will be verified?
More importantly:
Can it tell you what hasn’t been verified?
That sounds trivial until you are approaching a major design review or certification milestone.
During the trial, create a small but realistic verification plan containing:
- multiple requirements,
- multiple verification methods,
- several test environments,
- more than one product configuration,
- and a few intentionally missing verification activities.
Then ask the software:
What is still unverified?
The answer should not require manually comparing spreadsheets.
3. Test the Full MIL-to-SIL-to-HIL Workflow
If your organization uses model-based development, one of the most valuable trial exercises is moving the same verification intent through multiple environments.
For example:
Test the behavior of the mathematical or control model.
Test compiled production or near-production software in a simulated environment.
Run the real controller against a real-time simulated plant.
The goal should not be to recreate every test three times.
Ask:
How much of the test can move with us?
dSPACE, for example, explicitly positions SYNECT around managing and automating MIL, SIL and HIL testing while maintaining traceability between requirements, tests, parameters and results.
During an evaluation, try to take one meaningful test from MIL through SIL and into HIL.
Record:
- what had to be rewritten,
- what could be reused,
- what had to be manually reconfigured,
- whether expected results remained consistent,
- and whether reporting remained traceable across environments.
If each stage becomes a separate verification universe, you may be buying several workflows rather than one.
4. Evaluate the Hardware-in-the-Loop Experience
HIL deserves special attention because it combines software, models, real-time hardware, I/O, networks and physical controllers.
That creates many opportunities for friction.
During a HIL trial, evaluate:
- real-time model deployment,
- I/O configuration,
- signal mapping,
- fault insertion,
- bus simulation,
- restbus simulation,
- timing accuracy,
- synchronization,
- automated scenario execution,
- data logging,
- debugging,
- test reuse,
- and repeatability.
Do not test only normal operation.
Inject failures.
Disconnect something.
Create incorrect sensor values.
Introduce network faults.
Push the controller into states that rarely occur during ordinary testing.
That is one of the reasons HIL exists.
5. Test the Instruments You Actually Own
This one can save a company an enormous amount of grief.
Do not accept:
โYes, we support third-party instruments.โ
Bring the instruments.
Or at least bring a representative list.
Verification labs may contain equipment accumulated over many years from:
- Keysight,
- NI,
- Tektronix,
- Rohde & Schwarz,
- Yokogawa,
- Fluke,
- Vector,
- dSPACE,
- and numerous specialized manufacturers.
Some equipment may be decades old.
Some may have proprietary drivers.
Some may be controlled through:
- VISA,
- SCPI,
- Ethernet,
- USB,
- PXI,
- GPIB,
- serial interfaces,
- CAN,
- LIN,
- FlexRay,
- Automotive Ethernet,
- or custom interfaces.
NI TestStand, for example, supports test code written in multiple environments including LabVIEW, Python, C/C++ and .NET.
Keysight’s current PathWave Test Automation offering is built around OpenTAP and integration with test instruments and simulation tools.
Those are useful capabilities.
But vendor compatibility lists are not your laboratory.
During the trial, connect your equipment.
6. Test Old Equipment, Not Just New Equipment
This deserves its own section.
Engineering test systems tend to live for a very long time.
A company might replace laptops every three years.
It might continue operating a specialized test bench for fifteen.
That makes backward compatibility unusually important in verification and test software.
Bring at least one awkward old instrument into the evaluation.
Something everyone still needs but nobody particularly likes supporting.
See what happens.
If the vendor’s response begins with:
โWell, normally customers replace thatโฆโ
keep listeningโbut calculate the replacement cost.
7. Evaluate Test Authoring
Creating the first automated test is rarely the difficult part.
Maintaining 2,000 of them is.
During the trial, have your engineers build real test sequences.
Evaluate:
- test authoring speed,
- conditional logic,
- loops,
- parameterization,
- reusable functions,
- templates,
- libraries,
- error handling,
- parallel execution,
- debugging,
- documentation,
- and code review.
Also determine who can author tests.
Does everything require a software developer?
Can test engineers create sequences?
Can domain specialists modify test parameters without accidentally changing test logic?
Can operators run approved sequences without being able to alter them?
Those distinctions become important as the system spreads through an organization.
8. Test Automation Before Production
Automation should not begin only when a test system is considered finished.
NI’s current guidance specifically argues for bringing automation into design and validation earlier because manual-only workflows can hide timing problems, create inconsistent results and make later automation harder.
During a software trial, therefore, don’t evaluate automation only as:
Press button โ execute test.
Look at the entire process:
- Configure the test article.
- Confirm instrumentation.
- Load the correct software.
- Execute the procedure.
- Collect results.
- Detect failures.
- Repeat or branch when necessary.
- Generate the report.
- Store the evidence.
- Associate the result with the correct requirement and configuration.
The more manual steps remaining between those stages, the less automated your test process really is.
9. Evaluate Repeatability
Run the same test repeatedly.
Then run it:
- on another workstation,
- on another test bench,
- with another operator,
- on another shift,
- and, if applicable, at another facility.
Do you obtain comparable results?
Modern distributed validation environments can suffer from configuration driftโdifferent software versions, settings, drivers or system configurations gradually appearing across test stations. NI highlights configuration consistency as a significant factor in maintaining repeatable distributed testing.
During the evaluation, deliberately introduce configuration differences.
See whether the platform notices.
A verification system that cannot tell you why yesterday’s test differs from today’s may create more uncertainty than it removes.
10. Test Configuration Management
Ask one very simple question:
Exactly what did we test?
The answer may include:
- hardware revision,
- firmware version,
- software build,
- calibration state,
- simulation model version,
- test sequence revision,
- requirement revision,
- parameter set,
- test fixture,
- environmental conditions,
- instrument configuration,
- and operator.
Now ask whether the software captured all of it automatically.
If six months later an engineer opens a failed result, can that person reconstruct the test?
That is especially important for products with multiple configurations and variants.
Siemens, for example, emphasizes alignment between engineering BOMs, test BOMs and physical BOMs as part of verification management.
A test result against the wrong configuration may be perfectly accurate and completely useless.
11. Change Something
One of the best things you can do during any engineering software evaluation is break the happy path.
Change:
- a requirement,
- a test procedure,
- a component,
- an instrument,
- a calibration,
- a software build,
- a simulation model,
- or a measurement limit.
Then evaluate impact analysis.
Which tests must be repeated?
Which previous results remain valid?
Which reports become obsolete?
Which requirements are no longer verified?
NI’s V&V guidance identifies impact analysis as a fundamental part of maintaining test systems as requirements and system components change.
If your current process requires engineers to figure this out manually, this is an opportunity for the software to provide measurable value.
12. Evaluate Data Collection
Verification produces data.
Sometimes astonishing quantities of it.
Before evaluating fancy dashboards, determine whether the software can reliably collect the information you actually need.
Test:
- sampling rates,
- timestamps,
- synchronization,
- metadata,
- waveform storage,
- images,
- logs,
- video where relevant,
- pass/fail results,
- derived values,
- environmental conditions,
- and raw measurement preservation.
Ask whether raw data and processed results are stored separately.
You may discover years later that a processing algorithm was wrong.
If the raw data remains available, you may be able to reprocess it.
If only the calculated result was stored, the original evidence may be gone.
13. Evaluate Time Synchronization
This can become surprisingly important.
Suppose several systems record:
- CAN traffic,
- sensor measurements,
- video,
- ECU states,
- environmental chamber conditions,
- simulation values,
- and power consumption.
Can you accurately determine what happened at the same moment?
During the trial, create an event visible to several data sources.
Then see how easily an engineer can correlate them.
For complex system failures, a few milliseconds can matter.
14. Test the Data Analysis Workflow
Collecting enormous quantities of test data is not useful if nobody can efficiently interpret it.
Give engineers a large dataset.
Ask them to:
- locate a failure,
- compare two tests,
- compare two configurations,
- identify an outlier,
- plot a specific signal,
- calculate a derived measurement,
- and determine what changed.
Measure the number of steps.
Pay attention to whether engineers repeatedly export data to Excel, MATLAB, Python or another tool.
Exporting is not inherently bad.
But if every useful analysis begins with:
โFirst, export everythingโฆโ
you should understand what role the V&T software is actually playing.
15. Evaluate Search
Imagine you have accumulated five years of test results.
Now someone asks:
Have we ever seen this failure before?
Can you find out?
Search should be tested using:
- part number,
- serial number,
- software version,
- date,
- test procedure,
- requirement,
- failure code,
- engineer,
- test station,
- product configuration,
- and measurement values.
A verification database that becomes impossible to search eventually becomes an expensive archive.
16. Evaluate Reporting
Every demonstration includes a beautiful report.
Ignore it.
Build your report.
Try recreating:
- a regulatory report,
- an internal engineering report,
- a customer report,
- a design-review summary,
- and a simple management dashboard.
Determine how easily you can control:
- branding,
- tables,
- plots,
- units,
- tolerances,
- pass/fail indicators,
- revision information,
- approvals,
- requirement links,
- and supporting evidence.
Then change the report format.
Can an engineer make the change?
Or do you need professional services?
17. Evaluate Failure Handling
Most demos show passing tests.
Ask the vendor to show you a terrible day in the lab.
What happens when:
- an instrument stops responding,
- the network drops,
- a sensor produces impossible data,
- the test article loses power,
- the test aborts halfway through,
- the operator presses the wrong button,
- storage runs out,
- a test exceeds its allowed duration,
- or the computer crashes?
Does the system:
- preserve the data already collected,
- explain what happened,
- resume intelligently,
- distinguish test-system failure from product failure,
- and prevent invalid results from being treated as legitimate?
This may reveal more about the platform than another hour of feature demonstrations.
18. Can the Test System Verify Itself?
There is a wonderfully circular problem in verification:
How do you know the system doing the testing is correct?
NI’s V&V guidance makes this explicit: the test system itself must be verified and validated to ensure that its hardware and software perform their intended functions.
During the evaluation, determine how the platform supports:
- test-system validation,
- calibration,
- self-test,
- reference measurements,
- controlled software revisions,
- known-good devices,
- audit trails,
- and validation after system changes.
For regulated industries, this can be critical.
For everyone else, it is still good engineering.
19. Evaluate Calibration Management
Your software can execute the perfect test using an instrument whose calibration expired yesterday.
What happens?
Can the system:
- track calibration dates,
- prevent use of expired equipment,
- notify engineers before calibration expires,
- associate calibration status with test results,
- and determine which historical tests may be affected if an instrument later proves inaccurate?
NI SystemLink, for example, includes capabilities for monitoring test-system assets and upcoming calibration dates.
If calibration management is handled elsewhere, evaluate the integration.
20. Evaluate Usability for the Operator
The engineer who creates the test and the person who runs it may be entirely different people.
Put an actual operator in front of the trial system.
Don’t train that person for three hours first.
See whether the workflow makes sense.
Can the operator:
- identify the correct test,
- select the correct product,
- connect the equipment correctly,
- understand instructions,
- recognize a failure,
- recover from common mistakes,
- and know when to stop and call an engineer?
Human error is part of the test system.
Your evaluation should include it.
21. Evaluate Human Factors in the Product Being Tested
This is a slightly different questionโand an important one.
Verification cannot stop at:
Did the hardware perform to specification?
Sometimes the person operating the product is part of the system being verified.
That can include:
- vehicle drivers,
- pilots,
- machine operators,
- technicians,
- maintenance personnel,
- medical professionals,
- sailors,
- or ordinary consumers.
A mechanism can meet its structural load requirement and still be difficult or dangerous for the intended user to operate.
A control system can respond perfectly but present information badly.
A maintenance procedure can technically work but require an impossible body position.
When evaluating V&T software, ask whether it allows qualitative observations, video evidence, operator feedback and human-factors results to remain connected to the same verification framework as quantitative test data.
Not every valid verification result is a number.
22. Evaluate Environmental Testing
Products do not live at 72ยฐF on a laboratory bench forever.
Depending on the industry, verification may involve:
- temperature,
- humidity,
- vibration,
- shock,
- dust,
- water,
- altitude,
- corrosion,
- electromagnetic interference,
- radiation,
- or combinations of several conditions.
In other words:
You may need to shake it, freeze it, cook it, drop it and drown it.
The software needs to coordinate with those environments.
During a trial, determine whether environmental conditions can be:
- automatically controlled,
- synchronized with measurements,
- recorded with the test result,
- incorporated into test sequences,
- and associated with requirements.
This becomes particularly important when failures are intermittent.
23. Evaluate Cross-Domain Testing
Modern products increasingly combine:
- mechanical systems,
- electronics,
- embedded software,
- networks,
- controls,
- sensors,
- batteries,
- cloud services,
- and user interfaces.
The verification software should not assume that every failure belongs conveniently to one engineering discipline.
Try a systems-level test.
Can a mechanical engineer, electrical engineer, software engineer and systems engineer all understand the same result?
Can they trace the failure back into their own engineering environments?
This is where integration with PLM, ALM, MBSE and requirements platforms may become important.
24. Evaluate Open Interfaces
No verification and test system exists alone.
Ask about:
- REST APIs,
- Python interfaces,
- MATLAB integration,
- C/C++,
- .NET,
- databases,
- OPC UA,
- ASAM standards,
- FMI,
- OSLC,
- instrument APIs,
- PLM integration,
- ALM integration,
- CI/CD integration,
- and cloud services.
You may not need all of them.
But you should understand how easily the platform can exchange information with systems you already use.
Avoid creating another engineering island.
25. Try Your Existing Test Code
Years of test development represent intellectual property.
Bring it into the evaluation.
Determine whether you can reuse:
- LabVIEW code,
- Python,
- C/C++,
- MATLAB scripts,
- .NET assemblies,
- existing TestStand sequences,
- CANoe tests,
- custom executables,
- or proprietary internal test libraries.
dSPACE SYNECT, for example, explicitly supports test authoring using tools including AutomationDesk, NI TestStand, Vector CANoe/vTESTstudio, Python and MATLAB scripts.
NI TestStand similarly supports sequences combining multiple programming environments.
The question is not whether your code can technically be called.
The question is how much effort is required to make it part of a maintainable workflow.
26. Evaluate Version Compatibility
This is an easy one to forget during an exciting evaluation.
Ask:
What happens five years from now?
Test infrastructure often lasts longer than the software releases used to build it.
Evaluate compatibility among:
- test management software,
- development environments,
- instrument drivers,
- operating systems,
- real-time systems,
- databases,
- hardware interfaces,
- and test sequences.
Ask what happens when one component must be upgraded and another cannot be.
Long-lived test environments make lifecycle planning unusually important.
27. Evaluate Deployment
Building the test is only half the job.
Deploy it.
Preferably to several systems.
Measure:
- installation time,
- dependency management,
- configuration,
- licensing,
- test-package distribution,
- updates,
- rollback,
- permissions,
- and remote management.
Then update the test.
How difficult is it to ensure that every station is running the correct version?
Distributed test environments can become difficult to control precisely because software and configuration gradually diverge among systems.
28. Evaluate Scalability
A trial often involves:
one engineer + one test bench + one device
Your deployment may eventually involve:
200 engineers + 80 benches + 12 facilities + 40 product configurations.
Ask the vendor to demonstrate what changes.
Evaluate:
- central management,
- concurrent users,
- server architecture,
- data volume,
- geographic distribution,
- role management,
- test scheduling,
- equipment scheduling,
- and reporting across programs.
Keysight, for example, positions PathWave Test Automation as capable of scaling test automation across teams and geographic locations.
Verify that claim against your likely deployment model.
29. Evaluate Collaboration
Verification is rarely performed by one person.
Try:
- handing a test from one engineer to another,
- reviewing a test procedure,
- approving changes,
- adding comments,
- assigning failures,
- tracking corrective action,
- and comparing results between teams.
Then determine what happens when teams are geographically separated.
If collaboration mostly consists of emailing exported reports, the platform may not be delivering much collaboration.
30. Evaluate Security and Access Control
Not everyone should be allowed to alter an approved test.
Check whether permissions can distinguish among:
- test developers,
- verification engineers,
- operators,
- reviewers,
- managers,
- administrators,
- suppliers,
- and auditors.
Evaluate:
- authentication,
- role-based access,
- change history,
- electronic approvals where required,
- audit trails,
- data integrity,
- and supplier access.
For regulated or safety-critical industries, include your quality and cybersecurity teams in the trial.
31. Evaluate Compliance Evidence
If your organization works under standards such as:
- ISO 26262,
- DO-178C,
- DO-254,
- IEC 61508,
- FDA quality requirements,
- IEC 62304,
- ISO 13485,
- or other industry-specific processes,
do not accept:
โYes, we support compliance.โ
Ask the vendor to show you.
Take one requirement through:
requirement โ verification plan โ execution โ result โ approval โ report โ audit trail
Then ask someone from quality or compliance whether the evidence would actually be useful.
dSPACE, for example, explicitly positions SYNECT traceability as supporting ISO 26262 workflows.
The software does not make you compliant.
It can make producing and preserving the evidence considerably easierโor considerably harder.
32. Evaluate Test Asset Management
What equipment do you own?
Where is it?
Who is using it?
Is it calibrated?
Is it available next Tuesday?
Those questions become increasingly important in large validation organizations.
Evaluate whether the software helps manage:
- instruments,
- benches,
- fixtures,
- HIL systems,
- environmental chambers,
- licenses,
- calibration,
- maintenance,
- utilization,
- and reservations.
NI describes SystemLink use cases around viewing test systems, deployed software, equipment utilization, system status and calibration information.
If the product you are evaluating does not handle asset management itself, determine how it connects to the system that does.
33. Evaluate Test Throughput
Test time costs money.
Particularly in production.
Measure:
- sequence execution time,
- parallel test capability,
- instrument utilization,
- initialization time,
- data processing time,
- report generation,
- and downtime between test articles.
NI TestStand includes parallel execution and automatic scheduling capabilities intended for increasing test throughput and hardware utilization.
But benchmark throughput using your test, not the vendor’s example.
A 10% reduction in test time may matter enormously when millions of products are tested.
It may matter almost not at all when an aerospace qualification test takes three weeks.
Context matters.
34. Evaluate Test Maintenance
Now imagine the engineer who built the system has retired.
Can someone else maintain it?
Look at:
- architecture,
- naming conventions,
- documentation,
- reusable libraries,
- dependencies,
- custom code,
- configuration files,
- version control,
- and diagnostic tools.
NI’s TestStand guidance emphasizes modular test architectures partly because components can be changed and validated independently rather than forcing the entire system to be reconsidered after every modification.
This is one of the areas where a slightly slower system may be substantially cheaper over its lifetime.
35. Evaluate AI Carefully
Verification and test vendors are increasingly introducing AI-assisted capabilities.
Some may help with:
- generating tests,
- analyzing data,
- identifying anomalies,
- creating scripts,
- diagnosing failures,
- summarizing results,
- or optimizing test coverage.
Try them.
But don’t evaluate them by how impressive the generated answer looks.
Give the AI something for which you already know the correct answer.
Then check:
- accuracy,
- repeatability,
- traceability,
- explainability,
- hallucination risk,
- proprietary-data handling,
- and whether engineers can verify the generated output.
The central rule for AI in verification should be particularly strict:
Never allow the tool that is supposed to provide confidence in the product to quietly introduce new uncertainty into the evidence.
36. Let the People Who Will Actually Use It Evaluate It
Do not allow the software-selection committee to consist entirely of:
- executives,
- procurement,
- IT,
- and the vendor.
Include the engineers.
Better yet, include several types of users:
- systems engineers,
- verification engineers,
- test engineers,
- software developers,
- lab technicians,
- operators,
- quality personnel,
- and whoever maintains your existing test infrastructure.
Each one will notice a different problem.
The engineer may love the automation.
The operator may hate the interface.
IT may discover the deployment model creates a security problem.
Quality may discover the reports cannot satisfy an audit.
Procurement may discover that deploying the system requires six additional licenses nobody mentioned.
That is exactly why you conduct the trial.
37. Understand the Licensing Before the Trial Ends
Verification and test platforms can involve several layers of software.
You may need licenses for:
- development,
- execution,
- deployment,
- real-time targets,
- data management,
- test management,
- specialized protocols,
- analysis modules,
- report generation,
- requirements management,
- HIL systems,
- servers,
- floating users,
- and individual engineers.
Before approving the platform, recreate your likely deployment.
Then ask:
What would this exact architecture cost?
Also ask what changes if you double:
- engineers,
- test stations,
- facilities,
- or test volume.
The cheapest trial configuration may not resemble the production configuration at all.
38. Calculate the Cost of the Whole Test System
Software price is only one part of the investment.
Include:
- licenses,
- hardware,
- instruments,
- real-time systems,
- computers,
- servers,
- storage,
- integration,
- custom development,
- training,
- migration,
- maintenance,
- calibration,
- deployment,
- and support.
Then calculate something frequently forgotten:
engineering time.
If one platform saves each of 30 engineers four hours every week, that may overwhelm a seemingly large licensing difference.
Conversely, inexpensive software that requires extensive custom development may become the expensive option.
39. Ask What Happens If You Leave
This is one of my favorite software-evaluation questions.
Suppose five years from now you decide to change vendors.
What happens to:
- your test sequences,
- test data,
- results,
- requirements links,
- reports,
- metadata,
- scripts,
- models,
- instrument drivers,
- and historical verification evidence?
Can you export them?
In what format?
Can you still interpret the data without maintaining an old license server indefinitely?
Vendor lock-in is not automatically bad.
Sometimes deep integration is worth it.
But it should be understood before the purchase.
40. Test the Ugly Day, Not the Perfect Day
Perhaps this should be the central rule of the entire verification and test software trial.
Don’t spend the evaluation reproducing the vendor’s polished demonstration.
Create the kind of day engineers actually experience.
The requirement changes.
The wrong firmware gets loaded.
One instrument is old.
Another instrument’s calibration expires.
The HIL rack loses communication.
A test aborts.
Two facilities get different results.
Someone changes a limit without documenting it.
A software update breaks an old driver.
A test produces 400 GB of data.
An operator clicks the wrong thing.
Now ask:
Can the software help us understand what happened?
That is the test.
Create Your Own Verification & Test Trial
A good V&T software trial should include at least:
- one real product or representative subsystem,
- real requirements,
- real test cases,
- existing test code,
- representative instrumentation,
- multiple users,
- at least one change to a requirement,
- at least one deliberately induced failure,
- representative test data,
- real reporting requirements,
- and a realistic deployment scenario.
Do not allow the vendor to convert the evaluation into an extended product demonstration.
Their engineers can help.
They should help.
But your team should operate the system.
What Success Should Look Like
Before the trial begins, establish measurable criteria.
For example:
Traceability
Can we trace requirements through test cases and results without separate spreadsheets?
Test development
How long does it take to create and debug a representative automated test?
Reuse
How much existing test code can we retain?
Automation
How many manual steps can be eliminated?
Repeatability
Can different test stations reproduce the same test environment?
Failure analysis
How quickly can an engineer move from failed test to likely cause?
Data
Can engineers find and analyze historical test results efficiently?
Deployment
How difficult is it to distribute and update approved tests?
Maintenance
Could another engineer maintain this system five years from now?
Cost
What does the complete production deployment costโnot merely the trial license?
Those measurements give you something much more useful than:
โThe engineers seemed to like Product A.โ
The Software Is Part of the Verification System
There is an interesting paradox in buying verification software.
You are purchasing software whose purpose is to help determine whether engineering systems behave as intended.
So use the same philosophy when evaluating the software itself.
Define your requirements.
Build realistic verification scenarios.
Test normal operation.
Test abnormal operation.
Measure the results.
Document the limitations.
Include the humans.
And don’t assume that because something looks polished, automated or technically impressive that it actually satisfies your operational requirements.
That lesson applies to aircraft, vehicles, electronics, medical devices, industrial machinery, shipsโand the software used to verify them.
Verification software should have to prove itself, too.
