Why Copying Many Small Files Is Slower Than a Single Large File
- File transfer performance depends heavily on per-file transaction overhead and metadata updates, not just the aggregate byte volume.
- Network transfers over SMB incur round-trip latency for every file operation, while local transfers are constrained by file system structures and I/O serialization.
- Consolidating files into an uncompressed archive reduces transaction counts during transit but shifts the resource overhead to packaging and extraction.
Transferring many small files can take longer than one file with the same total payload, despite both operations moving approximately the same amount of payload data. This difference alone does not prove a media hardware failure. It may reflect the architectural overhead inherent in managing file system metadata, security filters, and transport protocols.
The Cost of Per-File Operations
A file copy is not a continuous raw stream of data. For every individual file, the software stack may perform several operations:
- Creation and Directory Allocation: The file system must locate free directory entries, update index structures (such as the Master File Table or inode tables), and allocate block pointers.
- Payload Writing: The actual data payload is written to storage blocks. For small files, per-file operations can become a larger share of elapsed time.
- Attribute and Journal Updates: Timestamps, access control lists (ACLs), and file sizes are updated. The details depend on the filesystem and copy tool.
- Filter Driver Interception: Security or backup filters may inspect many operations, adding CPU and latency overhead to each file in the batch.
Local Disks vs. SMB Network Transfers
The performance bottleneck manifests differently depending on the transport medium:
- Local Transfers (e.g., USB, internal SATA/NVMe): Opening, creating and closing each file, plus metadata and optional scanning, can dominate. A particular copy tool need not issue a synchronous flush per file.
- Network Transfers (e.g., SMB): Remote copy can add protocol round trips. Microsoft documents this specifically for SMB and single-threaded tools; its network figures cannot be applied to a local USB copy.
Controlled Comparison Framework
To accurately isolate why a specific workflow is slow, isolate the transfer variables using a standardized test design:
| Variable | Baseline Test (Single File) | Target Test (Multiple Files) | Control Requirement |
|---|---|---|---|
| Total Payload | One file | Many files | Same aggregate bytes; record file count and size distribution |
| Storage Targets | Source A to Destination B | Source A to Destination B | Identical hardware and connection path |
| Cache State | Defined starting cache state | Defined starting cache state | Document how cache state was established |
| Filter Drivers | Scanner enabled | Scanner enabled | Consistent antivirus and inspection rules |
| Copy Mechanism | Command-line copy tool | Command-line copy tool | Same thread count and buffer sizing |
Mitigation: Archiving and Trade-offs
Packaging multiple small files into a single archive (such as a .tar or .zip container) reduces file count during the transport stage, if the workflow permits archives.
However, archiving introduces trade-offs:
- Compute Overhead: Packaging and extracting files consumes local CPU cycles and memory.
- Storage Headroom: Creating an archive requires temporary scratch space on the source or destination volume.
- Access Latency: The recipient may need extra steps to inspect or use individual files.
If selecting media for this workflow, send the file-count distribution and measured copy conditions with a quote request.
We publish measured usable capacity and define trial-batch verification against the quoted product and agreed test scope.