cp and rsync crawl on hard drives
They read files in directory order, forcing the disk head to jump all over the platter. fast-copy sorts every file by its physical block offset first, so the drive reads in one smooth sweep.
Open Source · Apache 2.0 · v3.1.2
File transfersat full speed.
fast-copy is a Python CLI that squeezes every MB/s out of your hardware — block-order I/O, SSH tar pipe streaming, content-aware deduplication, sparse-file awareness, and a tamper-resistant audit log under sudo. 3–5× faster than scp/rsync/SFTP for remote transfers. No bloat. Just fast.
# Copy a folder — block-order I/O + dedup on by default
python fast_copy.py /home/kai/my-app /mnt/usb/my-app
🔍 Scanning block order… 2,341 files sorted by disk offset
⚡ Copying 48.3 GB… done in 38s · 41.2 MB/s
🔑 Deduped: 4.1 GB saved ✅ Verified
# Multi-source + sparse VM disks under sudo (v3.1.0+)
fast-copy --use-sudo /var/lib/longhorn/replicas/pvc-* /mnt/backup/
💾 Sparse: 2.3 TB logical → 12 GB on disk
# Incident snapshot — preserve full parent paths (-R, v3.1.2+)
fast-copy --use-sudo -R /etc /var/log /home/operator /mnt/snap/
🗂 Layout: /mnt/snap/etc/, /mnt/snap/var/log/, /mnt/snap/home/operator/
# SSH tar pipe — local → remote, no temp files
python fast_copy.py /data user@server:/backup/data --ssh-dst-password
🚀 SSH tar pipe · 4.0 MB/s · 6 batches of ~100 MB · ETA 41s
What it does
Copying files
shouldn't be the slow part.
fast-copy is a command-line tool that copies files and folders as fast as your disks and network physically allow — locally, to USB drives and NAS boxes, or across SSH. It exists because the everyday tools leave most of your hardware's speed on the table. Here's what it fixes:
Thousands of tiny files take forever
Per-file overhead murders throughput. fast-copy bundles small files into streamed tar batches, so 10,000 little files move as fast as one big one.
Duplicate files waste space and time
fast-copy fingerprints content with xxHash-128, copies each unique file once, and reflinks or hard-links the rest. Identical data is never written twice.
scp / SFTP top out at 1–2 MB/s
SFTP's chunked protocol is the bottleneck, not your network. fast-copy streams a raw SSH tar pipe with zero temp files — typically 3–5× faster on the same link.
Copies fail halfway or corrupt silently
A pre-flight space check refuses to start a copy that can't finish, and every file is hash-verified after landing — so a cheap USB drive can't corrupt your data unnoticed.
VM disks and volumes are huge but mostly empty
Sparse-file awareness (SEEK_DATA / SEEK_HOLE) skips the unallocated holes entirely. A 2.3 TB VM image with 12 GB of real data copies 12 GB, not 2.3 TB.
Get started
Install in seconds.
Run directly with Python 3.8+, or grab a standalone binary from GitHub Releases — no runtime required on the target machine.
Python (all platforms)
# requires Python 3.8+ python fast_copy.py <source> <destination> # SSH support pip install paramiko # Optional: ~10x faster hashing pip install xxhash
Standalone Binary (GitHub Releases)
# Linux / macOS — no Python needed curl -L https://github.com/gekap/fast-copy/releases/latest/download/fast_copy-linux -o fast-copy chmod +x fast-copy && ./fast-copy --help # Windows — download fast_copy-windows.exe # from the GitHub Releases page fast-copy.exe "C:\Projects\my-app" "E:\Backup\my-app"
Core features
Built for throughput,
not checkbox demos.
Every feature in fast-copy exists because it moves bits faster or wastes fewer of them. No GUI. No phone-home. No surprises.
Block-Order I/O
Uses FIEMAP on Linux, fcntl on macOS, and FSCTL on Windows to sort files by physical disk offset before reading — eliminating random-seek overhead and maximising sequential throughput.
Content-Aware Deduplication
Files are fingerprinted with xxHash-128 — the fastest non-cryptographic hash available — before writing. Identical content is skipped, saving bandwidth and write cycles.
SSH Tar Pipe Streaming
Remote transfers bypass SFTP entirely. fast-copy pipes a tar stream over SSH for a true single-hop server-to-server transfer — significantly faster and writing zero bytes to temp storage on either end.
Reflink-Based Copy
On btrfs / XFS / APFS / bcachefs, files are cloned via FICLONE / clonefile(2) — metadata-only CoW. A 10 GB copy on the same volume completes in milliseconds.
Sparse-File Awareness
VM disk images and Longhorn replicas are auto-detected via SEEK_DATA / SEEK_HOLE. Holes never hit the wire or the destination disk — 2.3 TB logical → 12 GB on disk on real backups.
--use-sudo + Audit Log
Re-execs under sudo for root-only paths, writes a hidden chattr +i JSONL audit log of every elevated run. Even root can't quietly delete a record.
Small-File Bundling
Instead of thousands of individual write calls, small files are bundled into a tar stream and block-written in one pass, then extracted locally. Dramatically improves throughput on directories full of tiny files.
Post-Copy Verification
After every transfer, fast-copy re-hashes the destination and compares against the source manifest. Corrupted or incomplete copies are reported immediately, before you delete the source.
Cross-Platform
First-class support for Linux, macOS, and Windows. PyInstaller standalone binaries on every GitHub release — no Python runtime required on the target machine.
Pre-Flight Space Check
Before writing a single byte, fast-copy calculates exactly how much unique data will land on the destination — accounting for deduplication and sparse-file holes — and verifies the target has enough free space. Works for local paths, USB drives, and remote SSH destinations (walking parent directories as fallback). Copy jobs never fail halfway through a 500 GB transfer because the drive was full.
Under the hood
What happens when
you hit Enter.
fast-copy runs a five-stage pipeline optimised for maximum I/O efficiency on every transfer.
01
Scan & Sort
Query the OS for the physical disk offset of every file. Sort the read queue to minimise head movement and maximise sequential I/O.
02
Fingerprint
Hash each file with xxHash-128. Cross-reference against the destination manifest. Flag duplicates before a single byte is written.
03
Space Check
Calculate exactly how much unique data will land on the destination — after dedup and sparse-hole elision — and verify the target has enough free space.
04
Stream
Write files in block-sorted order — locally or via SSH tar pipe. Small files are bundled first to coalesce write calls into large sequential blocks.
05
Verify
Re-hash the destination and diff against the source manifest. Report any mismatch instantly.
How it stacks up
fast-copy vs
the alternatives.
A feature-level comparison with the tools you're probably already using.
| Feature | fast-copy | rsync | cp (Unix) | robocopy (Win) | scp / SFTP |
|---|---|---|---|---|---|
| Block-order I/O (physical offset sort) | ✓ | ✗ | ✗ | ✗ | ✗ |
| Content deduplication (xxHash-128) | ✓ | checksum only | ✗ | ✗ | ✗ |
| SSH tar pipe (single hop) | ✓ | ✗ | ✗ | ✗ | ✗ |
| Zero temp files on remote transfer | ✓ | partial | ✗ | ✗ | ✗ |
| Reflink / CoW clones (btrfs/XFS/APFS) | ✓ | ✗ | ✗ | ✗ | ✗ |
| Sparse-file awareness (SEEK_DATA) | ✓ | partial | ✗ | ✗ | ✗ |
| Tamper-resistant audit log under sudo | ✓ | ✗ | ✗ | ✗ | ✗ |
| Post-copy hash verification | ✓ | --checksum | ✗ | ✗ | ✗ |
| Pre-flight space check (before writing) | ✓ | ✗ | ✗ | ✗ | ✗ |
| Small-file bundling | ✓ | ✗ | ✗ | ✗ | ✗ |
| Standalone binary (no runtime needed) | ✓ | system pkg | system pkg | ✓ | system pkg |
| Platform support | Linux/macOS/Win | Linux/macOS | Linux/macOS | Windows only | Linux/macOS/Win |
v3.1.2What's new in this releaseReleased 2026-05-24
Forensic-clean
multi-source copies.
New flag v3.1.2
-R, --keep-parentspreserves each source's full parent path under the destination (rsync-Rsemantics).fast-copy -R /etc /var/log dst/now producesdst/etc/anddst/var/log/instead ofdst/etc/anddst/log/.- Why it matters — the basename layout is fine for Longhorn
pvc-*replicas with unique names, but wrong for incident snapshots: a future analyst staring atdst/log/can't tell whether the original was/var/logor/usr/local/log, and two sources sharing a basename would silently merge.-Rpreserves provenance. - Scope — affects multi-source mode (≥2 paths) and glob mode (quoted patterns fast-copy expands internally). Single-source copies emit a one-line note and are unchanged. No change to dedup, sparse-file detection,
--use-sudohardening, or SSH transfers. - Forensic use case
fast-copy --use-sudo -R /etc /var/log /home/operator /mnt/snap/— snapshot tree mirrors the source filesystem exactly, so every analyst tool (Volatility, plaso, sleuthkit) recognizes the layout.
Previously v3.1.0 + v3.1.1
- Multi-source CLI (v3.1.0) —
fast-copy SRC1 SRC2 … DST/. Shell glob expands to N source trees, each preserved by basename by default.-Ropts into full-path layout. - Sparse-file awareness (v3.1.0) on Linux/macOS via
SEEK_DATA/SEEK_HOLE. 2.3 TB logical trees with 12 GB of real data no longer reject 900 GB destinations. --use-sudoself-elevation + tamper-resistant audit log (v3.1.0) — re-execs under sudo and writes a hiddenchattr +iJSONL audit trail. Even root can't quietly delete a record.O_NOFOLLOWon every destination + source (v3.1.1) — planted symlinks can no longer redirect root-privileged I/O to/root/.bashrcor exfiltrate/etc/shadow.--updaterefused under sudo (v3.1.1) plus--update-sha256 <hex>for out-of-band integrity pinning. Compromised release can't auto-trojan root.- Script-permission preflight (v3.1.1) —
--use-sudorefuses to elevate iffast_copy.pyor its directory is group/world-writable. Closes the "edit the script and wait" trojan path. - Reflink-based copy on btrfs / XFS / APFS / bcachefs — a 10 GB clone on the same volume completes in milliseconds via
FICLONE/clonefile.
Documentation
Complete Reference.
CLI Syntax
python fast_copy.py [options] source [source ...] destination
When two or more positionals are given, the last is the destination and earlier ones are sources (cp -r style). Each source can be a local path, a glob pattern, or a remote SSH path in the form user@host:/path.
All Options
Positional arguments
| Argument | Description |
|---|---|
| source | One or more source folders, files, globs, or remote user@host:/path |
| destination | Destination path or remote user@host:/path |
General options
| Flag | Default | Description |
|---|---|---|
| --buffer BUFFER | 64 | Buffer size in MB. 8 for low-memory systems, 128 for fast SSDs. |
| --threads THREADS | 4 | Threads used for hashing and physical layout resolution. |
| --dry-run | — | Show the copy plan without writing any data. |
| --no-verify | — | Skip post-copy hash verification. |
| --no-dedup | — | Disable deduplication. Faster for known-unique files. |
| --hash ALGO | auto | Hash algorithm: auto (xxh128 if installed, else sha256), xxh128, sha256. |
| --no-cache | — | Disable the persistent SQLite hash cache. |
| --force | — | Skip pre-flight space check. |
| --overwrite | — | Overwrite all files, skipping identical-file detection. |
| --exclude EXCLUDE | — | Exclude files/dirs by name or glob (repeatable). |
| -R, --keep-parents | — | Preserve each source's full parent path under the destination (rsync -R semantics, v3.1.2+). Multi-source and glob copies only — single-source copies emit a one-line note and are unchanged. fast-copy -R /etc /var/log dst/ produces dst/etc/ and dst/var/log/ instead of dst/etc/ and dst/log/. Use for incident snapshots and forensic captures where source provenance must be preserved. |
| --use-sudo | — | Re-exec self under sudo if not already root (v3.1.0+, hardened in v3.1.1+). Refuses to elevate if the script or its directory is group/world-writable. Linux/macOS only. |
| --update [VERSION] | — | Download and install latest (or a specific version). Refused under sudo (v3.1.1+). Supports --update-sha256 <hex> for out-of-band integrity pinning (v3.1.1+). |
SSH source options
| Flag | Description |
|---|---|
| --ssh-src-port PORT | SSH port for the remote source (default: 22) |
| --ssh-src-key PATH | Path to SSH private key for the remote source |
| --ssh-src-password | Prompt for SSH password for the remote source |
SSH destination options
| Flag | Description |
|---|---|
| --ssh-dst-port PORT | SSH port for the remote destination (default: 22) |
| --ssh-dst-key PATH | Path to SSH private key for the remote destination |
| --ssh-dst-password | Prompt for SSH password for the remote destination |
General SSH options
| Flag | Description |
|---|---|
| -z, --compress | Enable SSH compression. Recommended for slow or high-latency links. |
Local Copy — Examples
Copy a folder to a USB drive
python fast_copy.py /home/kai/my-app /mnt/usb/my-appMultiple sources at once (v3.1.0+)
fast-copy /var/lib/longhorn/replicas/pvc-* /mnt/backup_pvc/Copy a single file
python fast_copy.py ~/Downloads/Rocky-10.0-x86_64-dvd1.iso /mnt/usb/Glob pattern
python fast_copy.py "~/Downloads/*.zip" /mnt/usb/zips/
Windows
python fast_copy.py "C:\Projects\my-app" "E:\Backup\my-app"
Bulk Backups with Sudo + Audit Log
When source or destination requires root (Longhorn replicas, container volumes, system paths),
--use-sudo re-execs the tool under sudo. The audit log is written to ~$SUDO_USER/.fast_copy_audit.jsonl with chattr +i after every write — even root can't quietly delete a record.Snapshot /etc, /var/log, and an operator home dir
fast-copy --use-sudo /etc /var/log /home/operator /mnt/incident_snapshot/Multi-source CLI (v3.1.0+) accepts any number of source paths followed by the destination; each source is copied as its own subtree under the destination, preserving its basename.
SSH Transfers — Examples
SSH transfers require
When the remote server has
paramiko: pip install paramiko.When the remote server has
tar available, all transfers use raw SSH channels instead of SFTP — bypassing protocol overhead and working even on servers with SFTP disabled (e.g. Synology NAS).Local → Remote
python fast_copy.py /data user@server:/backup/data --ssh-dst-passwordRemote → Local
python fast_copy.py user@server:/data /local/backup --ssh-src-passwordRemote → Remote (relay through your machine)
python fast_copy.py user@src-host:/data admin@dst-host:/backup/data \
--ssh-src-password --ssh-dst-passwordCustom port and SSH key
python fast_copy.py user@host:/data /local \
--ssh-src-port 2222 --ssh-src-key ~/.ssh/id_ed25519Synology NAS (non-standard port, path with spaces)
python fast_copy.py /local/data "user@nas:/volume1/Shared Folder/backup" \ --ssh-dst-port 2205 --ssh-dst-password
Other Options — Examples
Dry run — preview without copying
python fast_copy.py /data /mnt/usb/data --dry-runDisable deduplication (faster for unique files)
python fast_copy.py /data /mnt/usb/data --no-dedupExclude directories
python fast_copy.py /project /mnt/usb/project \
--exclude node_modules --exclude .gitLow-memory system
python fast_copy.py /data /mnt/usb/data --buffer 8Verify a self-update against a known hash
fast-copy --update --update-sha256 <64-hex-from-release-page>
Real-World Benchmarks
Local → Local · 59,925 files (593 MB) to HDD
12.1sTotal time
41.2 MB/sSpeed
15,471Files hard-linked
92.5 MBSaved by dedup
25.8% of files were duplicates. Files read in physical disk order; small files bundled into block streams.
Sparse VM backup · 346 sparse files (Longhorn replicas)
2.3 TBLogical size
12.2 GBOn disk
2.2 TBHoles skipped
∞×vs naive cp
SEEK_DATA / SEEK_HOLE walked data-extents only. Pre-flight space check used allocated count, so a 900 GB destination accepted the job.
Remote → Local · 91,669 files (888 MB) over 100 Mbps LAN
14m 2sTotal time
619.5 KB/sSpeed
46,951Files hard-linked
378.5 MBSaved by dedup
51.2% duplicates. Streamed in 6 tar batches of ~100 MB with streaming extraction (no temp files). All 91,669 files verified after copy.
Local → Remote · 91,663 files (888 MB) over 100 Mbps LAN
2m 7sTotal time
4.0 MB/sSpeed
46,951Files hard-linked
6 batchesTar streams
3× faster than SFTP-based transfer. Remote hard links created via batched Python script over SSH (5,000 links per batch).
Questions
Frequently asked
questions.
The things people usually want to know before trying fast-copy.
How much faster is it than scp, rsync or SFTP?
For remote transfers, typically 3–5× faster than scp/rsync/SFTP. SFTP often tops out at 1–2 MB/s because of its chunked protocol; fast-copy streams a raw SSH tar pipe instead. Locally, the win comes from reading files in physical disk order and skipping duplicate and sparse data — on the same volume with a reflink-capable filesystem, copies can finish in milliseconds.
Do I have to install it on both machines for SSH transfers?
No. fast-copy runs on your side and drives the remote over a normal SSH connection using the tar and ssh that are already there — no daemon, agent, or matching install required on the remote. It even works against busybox-based systems like Synology NAS.
Will it ever modify or delete my source files?
No. The source is only ever read. fast-copy never writes to, renames, or deletes anything in the source tree — it builds the copy at the destination and verifies it.
Do I need Python installed?
Not necessarily. You can grab the standalone binary from the releases page (no runtime needed), or run it as python fast_copy.py if you already have Python 3. Both behave identically.
What does deduplication actually do to my files?
Identical files are stored once and the duplicates become reflinks (copy-on-write clones) or hard links on filesystems that support them — the files still appear normally, they just share underlying data. On link-incapable drives like FAT32 or exFAT, nothing is linked; fast-copy simply reports the bandwidth it saved by not re-reading the duplicate.
Is it safe to run under sudo?
Yes. The --use-sudo flow is security-hardened: it uses O_NOFOLLOW to resist symlink attacks, runs a script-permission preflight, refuses self-update while elevated, and writes a tamper-resistant (chattr +i) JSONL audit log recording who copied what.
Which platforms are supported?
Linux, macOS and Windows — each with native I/O optimizations (FIEMAP on Linux, fcntl on macOS, FSCTL on Windows). Windows long paths over 260 characters are handled too.
Does it phone home or collect any data?
No. There is no telemetry, no analytics, and no network calls beyond the SSH connection you explicitly ask for. It's open source under Apache 2.0 — you can read every line.
Do I have to pay to use fast-copy?
No. fast-copy is completely free and open source under the Apache 2.0 license. There's no paid tier, no license fee, and nothing locked behind a paywall — every feature you see is the whole tool.
Get in touch
Questions, bug reports,
or just to say hi.
Plain text only — no uploads, no tracking pixels. Spam protected by Cloudflare Turnstile. Goes straight to my inbox.
Open source
Star it. Fork it.
Make it faster.
fast-copy is Apache 2.0 licensed. Contributions, bug reports, and feature ideas are all welcome on GitHub.