A 2 GB positional-format file lands on an SFTP drop at 03:00. A partner posts a JSON array with 400,000 order lines. A nightly extract produces a CSV that used to be 50 MB and is now 900 MB because somebody widened a date range. These are the jobs that turn a healthy Camel estate into a 3 a.m. page, and they almost always fail the same way: the route reads the whole payload into a String or a List, the JVM spends its last minutes in full GC, and the broker redelivers the same message so the next node can die too.
Camel has the patterns for this. They are not new and they are not clever — Splitter, Aggregator, Claim Check, and the streaming variants of each. What is hard is using them together without quietly reintroducing the memory problem you were trying to avoid.
Rule one: never hold the whole body
Most large-payload incidents trace to a single line. .convertBodyTo(String.class) on a file endpoint, a getBody(List.class) in a processor, a log statement that formats the body, or an unmarshal into a fully materialized object graph. Any one of these pins the entire payload in heap, and worse, does it per concurrent message. A route sized for one 900 MB file at a time will not survive two.
The alternative is to keep the body as a stream and let the Splitter consume it lazily:
from("file:/data/in?noop=false")
.split(body().tokenize("\n")).streaming()
.to("direct:handleLine")
.end();
.streaming() is the important word. Without it, the Splitter materializes every sub-message before sending the first one — which for a line-tokenized 900 MB file means several gigabytes of String objects. With it, Camel pulls sub-messages one at a time from an iterator and the resident set stays flat.
The same applies to structured formats. Use the streaming-capable readers rather than whole-document parsers: Jackson's streaming mode for large JSON arrays, StAX (camel-stax) for large XML with a repeating element, and Bindy or the CSV data formats configured to iterate rather than collect. The general test: if the component hands you an Iterator or a Stream, it can stay streaming; if it hands you a List, it has already read everything.
One caveat that catches people: a streamed body can only be read once. If your route logs the body, then splits it, the split sees an empty stream. Camel's stream caching (context.setStreamCaching(true)) fixes re-readability by spilling to a temp file past a threshold, and that is the right default for routes that genuinely need multiple reads. But it is not free — you have traded heap for disk I/O and a temp directory that needs monitoring. Prefer restructuring the route so the body is read once.
Rule two: aggregate with a bounded, durable store
Splitting is the easy half. The trouble starts when you split a file into 400,000 records, enrich each one, and then need to recombine — a per-file result, a batch of 500 for a bulk API, a per-customer roll-up.
from("direct:handleLine")
.aggregate(header("customerId"), new GroupedBodyAggregationStrategy())
.completionSize(500)
.completionTimeout(30000)
.aggregationRepository(jdbcRepo)
.to("direct:submitBatch");
Three things decide whether this route is production-grade.
Completion conditions, plural. Size alone stalls the tail: the last 37 records of a batch of 500 sit in memory forever. Timeout alone makes throughput lumpy. Use both, and for file-derived flows add completionFromBatchConsumer() so the aggregator knows the file is done and flushes the remainder. A correlation group that never completes is not an idle group — it is a slow memory leak with a business consequence at the other end.
A repository that survives a restart. The default aggregation repository is in-memory. If the pod is rescheduled mid-batch, every partially aggregated group is gone, and because the source messages were already acknowledged, the data is gone with them. A JDBC or Infinispan-backed AggregationRepository persists in-flight groups and recovers them on startup. For anything with financial or regulatory weight, this is not optional.
A bounded correlation key space. Aggregating by customerId across a file with 200,000 distinct customers means 200,000 open groups. Aggregating by a value that includes a timestamp means unbounded groups forever. Ask what the maximum cardinality of the key can be on the worst day, not the typical one, and set closeCorrelationKeyOnCompletion and eviction accordingly.
The aggregation strategy itself deserves the same scrutiny. GroupedBodyAggregationStrategy accumulates a list in memory — fine for 500 rows, fatal for 500,000. When the aggregate is large, aggregate into a file or a database row rather than into the exchange body: append to a temp file, return a lightweight handle, and let the completion step pick it up.
Rule three: use Claim Check when the payload is not the point
Sometimes the large body is simply in the way. A route needs to route, enrich, and audit on ten header fields, and the 40 MB attachment is dead weight being copied through every hop, every queue, and every JMS broker page-out.
The Claim Check EIP separates the two. Store the payload once, carry a reference, retrieve it only where it is needed:
from("jms:queue:documents")
.claimCheck(ClaimCheckOperation.Push)
.to("direct:enrichMetadata")
.to("direct:routeDecision")
.claimCheck(ClaimCheckOperation.Pop)
.to("direct:deliverDocument");
Camel's built-in claimCheck uses an in-memory repository by default, which is fine within a single JVM's route but is not a distributed claim check. When the hops cross processes or brokers, do it explicitly: write the payload to object storage or a blob table keyed by a business id, put the key in a header, and fetch it at the far end. That version also fixes a problem the in-memory one does not — broker message size limits. Most JMS and Kafka deployments have a maximum message size measured in single-digit megabytes, and "we'll raise the broker limit" is a decision with cluster-wide consequences that outlives the integration that prompted it.
Claim check has a cost of its own: the payload now has a lifecycle. Something must delete it. Decide the retention rule when you build the route, not when storage bills arrive.
Rule four: size the pipe, not just the pattern
Streaming keeps each message small; it does not stop you from running too many at once. A .split().streaming().parallelProcessing() with a thread pool of 50 against a database that comfortably handles 10 concurrent writes converts a memory problem into a connection-pool problem. Set the pool explicitly with executorService, size it from the slowest downstream dependency, and put a throttle in front of anything with a published rate limit.
Watch these while the big job runs, not after:
- Heap after full GC, not peak heap. A flat post-GC line means the streaming is real; a rising one means something is still accumulating.
- Inflight exchange count (
camel.exchanges.inflight) per route. If it climbs steadily during a batch, your aggregator groups or thread queues are filling faster than they drain. - Aggregation repository size, if you persisted it. Open groups that never close show up here first.
- Temp directory usage, if stream caching is on.
Rule five: decide what "restart" means before you need it
The unglamorous question that decides whether the 03:00 page is a five-minute fix or a reconciliation project: if this job dies 70% of the way through a 400,000-record file, what happens when it runs again?
There are only a few honest answers. Reprocess everything, safe only if the downstream write is idempotent on a business key — which is the argument for pairing large-batch routes with an idempotent consumer keyed on record identity, not file name. Resume from a checkpoint, which requires recording progress durably as you go and costs a write per unit of progress. Or quarantine and repair, where failed records go to a dead letter destination with enough context to be replayed individually while the rest of the batch proceeds — usually the best fit for record-level data errors, because one malformed row should never fail 399,999 good ones.
Pick one deliberately and write it down in the route's runbook. The failure mode of large-batch integration is rarely a subtle bug; it is a job that half-ran and nobody can say which half.
The short version
Stream the split, bound the aggregate, persist the repository, claim-check anything the route does not need to look at, size concurrency from the slowest dependency, and know your restart story. None of this is exotic Camel. It is the difference between a batch route that scales with the data and one that works until the day the file gets bigger.