Package io.github.jchejarla.springbatch.clustering.mgmt

Class ClusterJobRecoveryManager

java.lang.Object
io.github.jchejarla.springbatch.clustering.mgmt.ClusterJobRecoveryManager
public class ClusterJobRecoveryManager extends Object
Recovers jobs whose master node has been lost from the cluster.

In this framework the node that launches a job is that job's master: it splits the work, monitors partition completion, reassigns orphaned partitions, and finalizes the job. Because mastership is per-job-execution, losing a master only affects that one job — the rest of the cluster is unaffected. However, a job whose master JVM dies mid-run would otherwise hang forever: its Spring Batch JobExecution stays STARTED with no living process to complete or fail it.

This manager closes that gap. Every node periodically scans for coordination rows that are still STARTED but whose master node has left the cluster (see DatabaseBackedClusterService.findOrphanedMasterJobs()). For each such job it atomically claims the row — so exactly one surviving node acts — then marks the stranded JobExecution (and any still-running step executions) FAILED. A FAILED execution is restartable, so the job can simply be re-submitted and resumed via standard Spring Batch restart semantics, rather than remaining a permanent zombie.

This is the detection-and-reaping phase of master failover. It does not (yet) automatically resume the job on another node; it makes recovery a clean, operator- or client-driven restart. Because recovery is fail-and-restart rather than resume, a job is restarted even in the edge case where its master died after all partitions completed but before finalizing — so restarts must be idempotent. The claim takes ownership of the coordination row, so a recovery that is itself interrupted is re-detected and retried by another node (recovery is idempotent). The scan cadence is controlled by the spring.batch.cluster.orphaned-master-scan-interval property (see BatchClusterProperties).

Author:
Janardhan Chejarla

  • Constructor Details

    • ClusterJobRecoveryManager

      public ClusterJobRecoveryManager()

  • Method Details

    • start

      @EventListener(org.springframework.boot.context.event.ApplicationReadyEvent.class) public void start()

    • reapOrphanedMasterJobs

      protected void reapOrphanedMasterJobs()
      Scans for jobs whose master node has left the cluster and abandons each one (once), so the stranded execution stops hanging and becomes restartable.