Skip to main content

Research method

B2B software evaluation framework

A reusable way to evaluate business software on workflows, controls, data and contract terms instead of feature lists. Written so a buyer, a procurement team or an analyst can run it without us.

Published 2026-11-20 · Updated 2026-11-20 · Khyati Sharma

Why feature lists are not evaluation

A demo is built to show the successful end state. It rarely shows what happens when a compensation change carries the wrong number forward, when an import fails at finalisation, or when an approval policy nobody configured lets a change through. Those moments decide whether software is safe to run.

This framework tests six things a feature list cannot: workflows, controls, data behaviour, pricing, implementation risk and buyer fit. Each observation carries an evidence level, an environment and a date, so a reader can tell a documented run from a vendor claim.

Evidence levels exist to prevent the most common dishonesty in software reviewing: calling documentation reading "testing". No vendor can buy a higher level, and a level describes what was done, not how good the product is.

Evidence levels

Every claim on TrulyCritic carries one of these. Vendors cannot purchase a higher level, and a higher level does not imply a positive finding.

LevelWhat it supportsWhat it does not
Public-Source ResearchedWhat the vendor publishes and what independent public sources say.That the workflow behaves that way when you run it.
Vendor Demo ReviewedThat the vendor demonstrated a capability, and how they described it.That it survives your data, your permissions or your approval chain.
Hands-On TestedThat TrulyCritic operated a specific workflow in a named environment on a dated run.That the result generalises to another configuration or to production.
Full Buyer BenchmarkMultiple workflows, evidence, limitations, implementation risk and pricing questions in one record.That every module was covered, or that the vendor will not change the product.
Not Independently VerifiedNothing on its own. It marks a claim that could not be tested directly.Carry the weight of a finding. Treat it as a question to ask the vendor.

Eight evaluation modules

Run these in order. Skipping module 1 makes every later finding unfalsifiable, because you will not know what the environment could or could not show.

Module 1

Product access and environment

  • Trial, demo, sandbox or production, named explicitly
  • Seeded or synthetic data, and whether it is realistic
  • Access duration and whether it renews
  • Permissions granted to the tester
  • Reset behaviour and whether state persists
  • Safety of external systems the product touches
  • What the environment structurally could not show

Module 2

Core buyer workflows

  • Realistic tasks written before testing starts
  • Expected behaviour stated in advance
  • Observed behaviour, recorded at the time
  • Setup and completion prerequisites
  • Workflow dependencies across modules
  • Failure and recovery behaviour, not just the happy path

Module 3

Data, permissions and controls

  • Role-based access and what each role can actually do
  • Approval chains and who can bypass them
  • Validation rules and what they block
  • Audit history and whether it survives changes
  • Data ownership, export and deletion
  • Separation of duties where money or access is involved

Module 4

Integrations and APIs

  • Authentication model and token lifetime
  • Scopes and least-privilege behaviour
  • Pagination and export limits
  • Field mapping and silent data loss
  • Retries, backoff and idempotency
  • Webhooks and missed-event handling
  • Rate limits and throttling behaviour
  • Revocation and deprovisioning

Module 5

Reporting and AI features

  • Source data behind each report
  • Repeatability across runs
  • Calculation accuracy on reconciled examples
  • Explainability of a number or a recommendation
  • Exportability and whether formulas survive
  • Human verification steps the product expects
  • Latency measured as a sample, not a benchmark

Module 6

Implementation and migration risk

  • Data preparation and cleanup effort
  • Who owns configuration after go-live
  • Training load and who carries it
  • Change management across teams
  • Rollback and what a partial cutover leaves behind
  • Cleanup after a failed import
  • Support and escalation paths
  • Timeline assumptions and who states them

Module 7

Pricing and contract diligence

  • Seat minimums and platform fees
  • Modules sold separately
  • Implementation and onboarding fees
  • Integration and API charges
  • Support tiers and what is excluded
  • Usage limits and overage rates
  • Renewal increases and uplift clauses
  • Data-export and termination terms
  • Contract length and notice period

Module 8

Buyer fit and proof of concept

  • Who benefits, stated as a profile rather than a market
  • Who should avoid it, stated plainly
  • Must-test workflows before signing
  • Acceptance criteria written down in advance
  • Stakeholders who must see the test
  • Scoring or decision rules agreed beforehand
  • Go/no-go conditions that are not negotiable after the demo

The standard finding record

One finding, one record, one stable ID. The ID is what lets a correction be traced, and it is what stops a rewritten page from erasing a result.

  • Stable ID
  • Title
  • Workflow
  • Evidence level
  • Environment
  • Date
  • Expected behaviour
  • Observed behaviour
  • Result
  • Severity
  • Limitation
  • Vendor context
  • Buyer implication
  • Recommended control
  • Proof-of-concept test
  • Sources
  • Change history

A record without an environment, a date and a limitation is an opinion. Delete it or downgrade it to Not Independently Verified.

Evidence collection checklist

  • Screenshots, with the environment visible
  • Recordings only where the vendor permits them
  • Calculations shown as inputs and outputs
  • Official sources linked, not summarised
  • API requests and responses with secrets removed
  • Timestamps on every observation
  • Environment label on every record
  • Synthetic data, never real customer data
  • Reviewer notes kept separate from findings
  • Product version or release under test
  • Public or private classification for each record

Keep a private evidence register while testing. Publish only what the vendor has agreed can be shown, and never a credential, a token or a customer record.

Use it yourself

The fastest useful version needs no download. Before your next proof of concept, write these five lines for each workflow you care about:

  1. 1. The workflow, in one sentence a colleague could run.
  2. 2. What should happen, written before you touch the product.
  3. 3. What actually happened, with a screenshot and a timestamp.
  4. 4. What a failure would cost after go-live.
  5. 5. The control that would catch it, or the reason none exists.

That is a finding record. Everything else in this framework is discipline around it. We do not currently publish a spreadsheet template, and we will not pretend to.

What limits every evaluation

  • Sandbox is not production. No real bank settlement, no carrier transmission, no year-end filing, no scale.
  • Configuration varies. The same product behaves differently with different templates, approvals and permissions.
  • Sample size is one. A single run proves the behaviour is possible, not that it is typical.
  • Access is time-limited. Products change after the test and before you buy.
  • Modules go untested. Say which, or the absence reads as approval.
  • Pricing changes. Every figure needs a date and a source.
  • Judgement is editorial. Severity is a human call, and it should be argued, not asserted.

Vendor factual verification, and its limits

A vendor may check its own facts before publication: product names, pricing, feature availability, technical detail and links. It may not change conclusions, severity, competitor coverage, ratings or verdicts, and it does not get to approve the evaluation. Where a correction comes in, the finding is updated in public with its ID, the date and what changed.

Facts and conclusions are separate things. A fact can be wrong and get fixed. A conclusion is an argument built on facts, and it changes only when the facts do.

See the framework applied

The Rippling evaluation is the first full use of this framework: a named sandbox, a written workflow plan, nine documented findings including one API control test, a published list of what was not tested, and a method page that explains every limit.