Research method
B2B software evaluation framework
A reusable way to evaluate business software on workflows, controls, data and contract terms instead of feature lists. Written so a buyer, a procurement team or an analyst can run it without us.
Published 2026-11-20 · Updated 2026-11-20 · Khyati Sharma
Why feature lists are not evaluation
A demo is built to show the successful end state. It rarely shows what happens when a compensation change carries the wrong number forward, when an import fails at finalisation, or when an approval policy nobody configured lets a change through. Those moments decide whether software is safe to run.
This framework tests six things a feature list cannot: workflows, controls, data behaviour, pricing, implementation risk and buyer fit. Each observation carries an evidence level, an environment and a date, so a reader can tell a documented run from a vendor claim.
Evidence levels exist to prevent the most common dishonesty in software reviewing: calling documentation reading "testing". No vendor can buy a higher level, and a level describes what was done, not how good the product is.
Evidence levels
Every claim on TrulyCritic carries one of these. Vendors cannot purchase a higher level, and a higher level does not imply a positive finding.
| Level | What it supports | What it does not |
|---|---|---|
| Public-Source Researched | What the vendor publishes and what independent public sources say. | That the workflow behaves that way when you run it. |
| Vendor Demo Reviewed | That the vendor demonstrated a capability, and how they described it. | That it survives your data, your permissions or your approval chain. |
| Hands-On Tested | That TrulyCritic operated a specific workflow in a named environment on a dated run. | That the result generalises to another configuration or to production. |
| Full Buyer Benchmark | Multiple workflows, evidence, limitations, implementation risk and pricing questions in one record. | That every module was covered, or that the vendor will not change the product. |
| Not Independently Verified | Nothing on its own. It marks a claim that could not be tested directly. | Carry the weight of a finding. Treat it as a question to ask the vendor. |
Eight evaluation modules
Run these in order. Skipping module 1 makes every later finding unfalsifiable, because you will not know what the environment could or could not show.
Module 1
Product access and environment
- Trial, demo, sandbox or production, named explicitly
- Seeded or synthetic data, and whether it is realistic
- Access duration and whether it renews
- Permissions granted to the tester
- Reset behaviour and whether state persists
- Safety of external systems the product touches
- What the environment structurally could not show
Module 2
Core buyer workflows
- Realistic tasks written before testing starts
- Expected behaviour stated in advance
- Observed behaviour, recorded at the time
- Setup and completion prerequisites
- Workflow dependencies across modules
- Failure and recovery behaviour, not just the happy path
Module 3
Data, permissions and controls
- Role-based access and what each role can actually do
- Approval chains and who can bypass them
- Validation rules and what they block
- Audit history and whether it survives changes
- Data ownership, export and deletion
- Separation of duties where money or access is involved
Module 4
Integrations and APIs
- Authentication model and token lifetime
- Scopes and least-privilege behaviour
- Pagination and export limits
- Field mapping and silent data loss
- Retries, backoff and idempotency
- Webhooks and missed-event handling
- Rate limits and throttling behaviour
- Revocation and deprovisioning
Module 5
Reporting and AI features
- Source data behind each report
- Repeatability across runs
- Calculation accuracy on reconciled examples
- Explainability of a number or a recommendation
- Exportability and whether formulas survive
- Human verification steps the product expects
- Latency measured as a sample, not a benchmark
Module 6
Implementation and migration risk
- Data preparation and cleanup effort
- Who owns configuration after go-live
- Training load and who carries it
- Change management across teams
- Rollback and what a partial cutover leaves behind
- Cleanup after a failed import
- Support and escalation paths
- Timeline assumptions and who states them
Module 7
Pricing and contract diligence
- Seat minimums and platform fees
- Modules sold separately
- Implementation and onboarding fees
- Integration and API charges
- Support tiers and what is excluded
- Usage limits and overage rates
- Renewal increases and uplift clauses
- Data-export and termination terms
- Contract length and notice period
Module 8
Buyer fit and proof of concept
- Who benefits, stated as a profile rather than a market
- Who should avoid it, stated plainly
- Must-test workflows before signing
- Acceptance criteria written down in advance
- Stakeholders who must see the test
- Scoring or decision rules agreed beforehand
- Go/no-go conditions that are not negotiable after the demo
The standard finding record
One finding, one record, one stable ID. The ID is what lets a correction be traced, and it is what stops a rewritten page from erasing a result.
- Stable ID
- Title
- Workflow
- Evidence level
- Environment
- Date
- Expected behaviour
- Observed behaviour
- Result
- Severity
- Limitation
- Vendor context
- Buyer implication
- Recommended control
- Proof-of-concept test
- Sources
- Change history
A record without an environment, a date and a limitation is an opinion. Delete it or downgrade it to Not Independently Verified.
Evidence collection checklist
- Screenshots, with the environment visible
- Recordings only where the vendor permits them
- Calculations shown as inputs and outputs
- Official sources linked, not summarised
- API requests and responses with secrets removed
- Timestamps on every observation
- Environment label on every record
- Synthetic data, never real customer data
- Reviewer notes kept separate from findings
- Product version or release under test
- Public or private classification for each record
Keep a private evidence register while testing. Publish only what the vendor has agreed can be shown, and never a credential, a token or a customer record.
Use it yourself
The fastest useful version needs no download. Before your next proof of concept, write these five lines for each workflow you care about:
- 1. The workflow, in one sentence a colleague could run.
- 2. What should happen, written before you touch the product.
- 3. What actually happened, with a screenshot and a timestamp.
- 4. What a failure would cost after go-live.
- 5. The control that would catch it, or the reason none exists.
That is a finding record. Everything else in this framework is discipline around it. We do not currently publish a spreadsheet template, and we will not pretend to.
What limits every evaluation
- Sandbox is not production. No real bank settlement, no carrier transmission, no year-end filing, no scale.
- Configuration varies. The same product behaves differently with different templates, approvals and permissions.
- Sample size is one. A single run proves the behaviour is possible, not that it is typical.
- Access is time-limited. Products change after the test and before you buy.
- Modules go untested. Say which, or the absence reads as approval.
- Pricing changes. Every figure needs a date and a source.
- Judgement is editorial. Severity is a human call, and it should be argued, not asserted.
Vendor factual verification, and its limits
A vendor may check its own facts before publication: product names, pricing, feature availability, technical detail and links. It may not change conclusions, severity, competitor coverage, ratings or verdicts, and it does not get to approve the evaluation. Where a correction comes in, the finding is updated in public with its ID, the date and what changed.
Facts and conclusions are separate things. A fact can be wrong and get fixed. A conclusion is an argument built on facts, and it changes only when the facts do.
See the framework applied
The Rippling evaluation is the first full use of this framework: a named sandbox, a written workflow plan, nine documented findings including one API control test, a published list of what was not tested, and a method page that explains every limit.