Budgeting flash storage qualification: acceptance before sample quantity

Define the unanswered acceptance questions before choosing sample quantity or a test budget. Can the device enumerate and boot, sustain the actual write workload, recover within agreed limits after interruption, and remain traceable to the production configuration? These questions determine samples, hosts, observation time and cost.

A few samples and one speed run cannot cover every risk. An extensive generic checklist can also consume money without answering the customer's acceptance question. The following framework combines purchasing decisions with the technical plan; it is not a report of a completed customer project.

Image: AI-generated workshop illustration, not an actual factory photograph or evidence that a test was performed or passed.

Turn the requirement into a decision

"Stable", "fast" and "industrial" are not complete acceptance criteria. State host and firmware, workload, environment, observed metrics, limits and the action required after a failure. For recording, define data generation, overwrite behaviour and clip inspection. A system drive may require boot and update checks performed by the customer's engineering team.

If a metric cannot yet be measured, record the missing tool or evidence instead of marking a pass. Resolve critical gaps before release. Kalstor supplies storage hardware; software installation, image preparation or recovery, licensing and application validation remain with the customer or its appointed provider.

Split the budget into five categories

A practical budget separates samples and consumables, dedicated equipment or external services, staff hours multiplied by the agreed project rate, and an approved retest reserve. Keep fixture occupancy hours separate from staff time: unattended runtime is not automatically the same number of labour hours. Apply the company's consistent allocation policy for existing equipment and avoid counting the same cost twice.

Supplier shipment checks are not a substitute for customer system validation. Confirm who supplies samples and documents, who performs each check, what is excluded and what incurs a charge before ordering. Do not infer a laboratory service or a software service from a hardware quotation.

Sample count must follow the intended conclusion

Choose quantity using expected device and host variation, destructive-test needs, project risk and the statistical conclusion sought. Repeated runs on one card increase exposure of that card; they do not create independent samples. A single lot also limits what can be inferred about other lots.

Zero failures in a small sample can support further evaluation, not a promise of zero failures in future production. A numerical confidence claim requires an appropriate sampling and statistical design. More samples of the same type will not correct a plan that omits the critical host or failure condition.

Agree failure handling before testing: preserve configuration, time, logs, media state and reproduction steps. Stop when further testing may destroy evidence or unique data. Replacing samples repeatedly without comparable records does not explain a fault. When firmware, host or storage configuration changes, reassess which earlier conclusions remain applicable.

Start with the product requirement, not a benchmark

Write application requirements before choosing tools:

• interface, capacity and mechanical envelope;
• operating and storage temperature;
• startup and enumeration time;
• sustained read/write throughput after cache effects;
• average and tail latency limits;
• endurance and expected host-write profile;
• retention and service interval;
• power-loss and unsafe-shutdown behavior;
• health reporting and field diagnostics;
• regulatory, material and traceability needs.

Separate must pass requirements from characterization data. A camera that must sustain a defined video stream needs a minimum write floor and maximum stall duration, not only an average MB/s result. An embedded controller may care more about boot latency and power-cycle recovery than peak throughput.

Bind approval to an exact configuration

Record every device under test:

If a later shipment cannot be tied to that identity, the qualification cannot prove equivalence. Pair the release with a PCN/EOL and change-control process and a fixed-BOM requirement where variation is unacceptable.

Build a layered test matrix

1. Interface and compatibility

Test cold boot, warm reset, sleep/resume, removal where applicable, all required host ports and expected protocol modes. Record negotiated link width or speed, capacity and identifiers. Exercise error paths such as timeout, retry and device reset without injecting destructive faults into uncontrolled equipment.

2. Functional and data-integrity behavior

Cover full-address-range write/read verification where feasible, filesystem creation, large and small files, boundary sizes, trim/deallocate or erase behavior and application data structures. Verify that the system detects a failed comparison rather than only recording that the command completed.

3. Workload performance

Use the application’s block sizes, read/write mix, queue depth, access pattern, sync behavior and duty cycle. Report throughput, IOPS and latency percentiles over time. Avoid approving from a single best run.

SNIA's performance specifications provide methods for comparable device-level measurements [1]. Its v2.0.2 specification distinguishes preparation and preconditioning from measurements used to characterize steady-state behaviour [2]. Select the applicable procedure rather than treating every flash product as the same SSD workload.

4. Environment and mechanics

Run the required workload at defined ambient and device temperatures in the final enclosure. Include airflow, heatsink, vibration or connector retention where the product requirement calls for them. For NVMe designs, correlate performance with the thermal-throttling counters rather than accepting one temperature screenshot.

5. Power and recovery

Test normal shutdown, repeated power cycle, brownout or unexpected removal using an approved fixture and safe procedure. Verify bootability, filesystem consistency, application recovery and health logs after each sequence. Power-loss testing can damage data by design; isolate samples and preserve results.

6. Endurance and health

Map the estimated field workload to host writes, write amplification and service life. Record a baseline of supported SMART, NVMe or eMMC health fields, then trend them during stress. The WAF guide explains why the same host-write volume can create different media wear.

Preconditioning prevents a false pass

Managed flash often starts fast. Empty blocks and a temporary SLC cache can make a fresh device look better than it behaves after sustained use.

SNIA defines workload-independent and workload-dependent preconditioning for SSD measurement [2]. Use the procedure and stopping conditions appropriate to the selected test; a fixed amount of writing alone does not establish every application's steady-state behaviour.

Do not copy that SSD procedure unchanged to every microSD or eMMC. Preserve the principle:

1. Define the starting state.
2. Fill or exercise the address range in a documented way.
3. Apply the representative workload long enough for behavior to converge.
4. Measure the window that represents service, not only the first minutes.

If the product uses a burst cache, include a transfer larger than the cache and report both burst and sustained write speed.

Measure distributions and transitions

Average latency can hide stalls that break an application. Retain:

• median, p95, p99, p99.9 and maximum latency where the workload supports them;
• performance versus time, temperature and bytes written;
• transition from fresh to steady state;
• recovery after idle, trim/deallocate and workload changes;
• device errors, retries, resets and thermal events.

NVM Express distinguishes protocol compliance from customer qualification: meeting a particular latency may be an end-user requirement even when it is not a compliance item [3]. Compliance certificates are useful inputs, not substitutes for the final host test.

Control samples and failures

Use production-representative samples. Engineering samples, hand-selected units or devices with undocumented firmware do not establish production capability.

For a failure:

1. Stop destructive retesting.
2. Preserve the exact device, host logs, workload and time of failure.
3. Record device identity and health/error logs using read-only methods where possible.
4. Reproduce on a controlled sample only when the plan allows it.
5. Request supplier failure analysis with a defined question and evidence package.

NVMe-CLI can read identity, SMART/Health, error and self-test information where supported [4]. Store raw outputs, not only screenshots or a pass/fail summary.

Define the release package

Keep three decisions separately traceable. Technical approval means the candidate meets the agreed conditions. Commercial approval also requires confirmed price, quantities, lead time, responsibilities and payment terms. Production release requires evidence linking delivered configuration to approved samples. None silently replaces the others.

A production approval should contain:

• approved part number and configuration;
• qualification plan and revision;
• sample and lot table;
• calibrated equipment and software versions;
• raw data, plots and test logs;
• pass/fail limits and results;
• deviations, risk acceptance and owners;
• approved golden sample if one is used;
• effective production lot/date;
• requalification triggers.

Triggers commonly include controller, NAND or firmware changes, manufacturing transfer, capacity change, major host revision, repeated field failure or an unapproved substitution.

Request hardware candidates with a defined validation scope

Send Kalstor the device, storage interface, capacity, workload, environment, project quantity and time window through the project RFQ. Separate sample supply, available technical documentation, agreed supplier hardware checks and customer system validation. Confirm scope, responsibility and cost for each. A supplier report supports a decision only to the extent that its samples and conditions match the project; it cannot automatically approve the customer's equipment or business software.

Bottom line

Flash qualification is a controlled release decision, not a speed screenshot. Bind approval to the exact device and host, test the real workload after preconditioning, measure tail behavior across environment and power events, and preserve auditable evidence. Then connect production shipments to the qualified baseline through traceability and change control.