Original design intent behind JobIsolatedProcessor and the dyn JobProcessor trait boundary
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 203
Description
**Problem**
Hi! I've been studying CubeStore's job execution architecture and had a question about a pattern I noticed in the ingestion subsystem.
## Context
PR https://github.com/cube-js/cube/pull/7094 extracted JobRunner, JobProcessorImpl, and JobIsolatedProcessor from cluster/mod.rs into cluster/ingestion/. The resulting architecture has a clear trait boundary:
JobRunner.route_job()
└── self.job_processor.process_job(job) // dyn JobProcessor
└── self.processor.process_separate_job() // JobIsolatedProcessor
This mirrors the WorkerPool / WorkerConfigurator pattern used for SELECT queries, where:
- A trait abstraction (WorkerProcessing) enables subprocess isolation
- respawn() spawns OS subprocesses with IPC channels
- WorkerConfigurator::configure() bootstraps services from Config inside the subprocess
The job-side counterpart has analogous scaffolding:
- dyn JobProcessor trait with wait_processing_loops() / stop_processing_loops() lifecycle methods (currently no-ops in JobProcessorImpl)
- JobIsolatedProcessor::new_from_config(config: &Config) — a constructor that resolves services from the DI injector, matching the pattern a subprocess worker_main() would use
- The name process_separate_job and error message "Cannot process in separate process" reference a "separate process" concept
But no JobConfigurator or JobWorkerProcessing implementation exists — jobs run in-process as async Tokio tasks, same as before the refactoring.
## Questions
1. Was subprocess isolation for background jobs (compaction, imports, etc.) a planned feature that was extracted as groundwork in #7094 but never completed? Or is the current in-process design intentional and the naming inherited from an earlier
prototype?
2. Is JobIsolatedProcessor::new_from_config() dead code, or is it a deliberate hook for future subprocess isolation?
3. Was there a specific motivation for the dyn JobProcessor trait boundary beyond code organization? The lifecycle methods (wait_processing_loops, stop_processing_loops) being no-ops suggests they were intended for a future implementation that would
need them.
Thanks for any insight!
Contributor guide
Research direction
Start with PR #7094 and the JobRunner, JobProcessorImpl, and JobIsolatedProcessor definitions moved from cluster/mod.rs into cluster/ingestion/. Trace the dyn JobProcessor lifecycle methods, process_separate_job, and new_from_config() alongside the WorkerPool/WorkerConfigurator pattern. Done means documenting whether subprocess isolation was intended, whether the constructor is dead code or a future hook, and why the trait boundary exists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100