File and SFTP Transfers in Camel: Read Locks, Partner Drops, and Not Losing Files

Every integration estate we review has a Kafka story, a REST story, and — quietly doing a third of the real business volume — a pile of file transfers. Partner drops a CSV on SFTP at 02:00. A mainframe writes a fixed-width extract to a mount. Finance expects a settlement file by 06:00 and will phone someone if it is late.

File integration looks like the easy part. It is not. The failure modes are quiet: a half-written file read as complete, the same file processed twice by two pods, a poller that stopped three days ago and nobody noticed because nothing threw an exception. Camel's file and sftp components handle all of this well, but only if you set the options deliberately. Defaults are a starting point, not a design.

The half-written file problem

The first rule of consuming a directory you do not control: a file appearing is not a file finished. Your partner's upload takes ninety seconds. Your poller runs every ten. Without a read lock you will eventually parse a truncated CSV and insert half a day of orders.

Camel's readLock option picks the strategy:

• changed — poll the file's size and modification time and wait until they stop moving. Works across protocols and across writers you do not control. Tune readLockCheckInterval and readLockMinAge; this is the default choice for SFTP and for partner drops.
• markerFile (local file default) — Camel writes a .camelLock sidecar. Fine when Camel is the only consumer, useless against an external writer, because the writer does not know about the marker.
• rename — attempt an atomic rename into a work area. Fast and decisive, but only atomic within a filesystem or SFTP server that treats it that way, and a rename can succeed on a file still being appended to.
• idempotent / idempotent-changed — use an idempotent repository as the lock so multiple nodes agree. More on clustering below.
• none — only for files you produced yourself, atomically.

The cleanest answer, when you can negotiate it, is not a read lock at all: ask the sender for a done file. The partner writes orders-20260612.csv, then writes an empty orders-20260612.done. You consume with doneFileName=${file:name.noext}.done and the ambiguity disappears. Half the partner integrations we migrate could have had this from day one; it just never got asked for. Ask for it on new interfaces.

Move, don't delete

delete=true is tidy right up to the morning someone asks what exactly was in yesterday's 11:40 file. Default Camel behaviour — move to a .camel subdirectory — is better, and an explicit dated archive is better still:

from("sftp://partner@host/outbound"
    + "?readLock=changed&readLockMinAge=30s"
    + "&move=.done/${date:now:yyyyMMdd}/${file:name}"
    + "&moveFailed=.error/${date:now:yyyyMMdd}/${file:name}"
    + "&idempotent=true&idempotentRepository=#fileRepo")
  .routeId("partner-orders-in")
  .to("direct:processOrders");

Two things matter here as much as the read lock. moveFailed gives you a quarantine directory instead of a poison file that the poller retries forever. And an archive with the original bytes is the artifact that settles disputes with a partner — you can show them precisely what arrived. Pair the archive with a retention job, or you will meet the disk-full incident instead.

Clustering: the duplicate nobody tests for

Run two replicas of the same file route and both pollers will see the same file. On a local filesystem the marker lock usually saves you. On SFTP, where locking semantics belong to someone else's server, it often does not.

Two workable answers:

1. Shared idempotent repository. Use readLock=idempotent (or idempotent-changed) backed by a JDBC or Infinispan repository so all nodes check the same table. Key on file name plus something that changes with content — size, modification time — because partners reuse file names more often than you would like.
2. Single-active consumer. Camel's cluster service support lets only one node own the route (master: endpoint prefix with a Kubernetes or JGroups cluster service). Simpler to reason about, and the right call for strictly ordered feeds. The cost is throughput and a failover window.

Pick one on purpose. The failure we see most often is neither: three replicas, no shared lock, and duplicate-order incidents blamed on the upstream partner for a year.

Absence is the hardest error

File integrations fail loudest when nothing happens. No exception, no error handler, no alert — just a file that never arrived and a reconciliation break discovered at month end.

Camel's error handler cannot catch a non-event, so instrument around it:

• Emit a metric per successful consume (file.received{route="partner-orders-in"}) and alert on staleness, not on errors. Any decent monitoring stack can fire when a counter has not moved in six hours.
• For hard deadlines, add a scheduled watchdog route — a timer at 06:15 that checks whether today's file was recorded, and raises an alert if not.
• Watch the poller itself. A consumer that throws on connect will back off and, depending on configuration, go quiet. backoffMultiplier, backoffIdleThreshold, and an explicit pollStrategy keep connection failures visible rather than merely logged at DEBUG.

Migration notes

ESB file adapters tend to carry years of accumulated behaviour: archive naming conventions downstream teams secretly depend on, trailing-newline quirks, character encodings set once in 2011. When we move these onto Camel, the file flows are the ones where byte-level reconciliation during the parallel run earns its keep — tee the input, write the Camel output to a shadow directory, and diff. Encoding and line-ending differences surface immediately there, and nowhere else until a partner complains.

The short version: use a read lock that matches who writes the file, ask for done files on anything new, move and archive rather than delete, agree on one clustering story, and alert on silence. File transfer is not glamorous, but it is load-bearing, and it is entirely possible to make it boring.