cube-js / cube-js/cube

Original design intent behind JobIsolatedProcessor and the dyn JobProcessor trait boundary

Open
#10,381 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 10h
Merged PRs (30d)
203

Description

**Problem**

Hi! I've been studying CubeStore's job execution architecture and had a question about a pattern I noticed in the ingestion subsystem.

## Context

PR https://github.com/cube-js/cube/pull/7094 extracted JobRunner, JobProcessorImpl, and JobIsolatedProcessor from cluster/mod.rs into cluster/ingestion/. The resulting architecture has a clear trait boundary:

JobRunner.route_job()
└── self.job_processor.process_job(job) // dyn JobProcessor
└── self.processor.process_separate_job() // JobIsolatedProcessor

This mirrors the WorkerPool / WorkerConfigurator pattern used for SELECT queries, where:
- A trait abstraction (WorkerProcessing) enables subprocess isolation
- respawn() spawns OS subprocesses with IPC channels
- WorkerConfigurator::configure() bootstraps services from Config inside the subprocess

The job-side counterpart has analogous scaffolding:
- dyn JobProcessor trait with wait_processing_loops() / stop_processing_loops() lifecycle methods (currently no-ops in JobProcessorImpl)
- JobIsolatedProcessor::new_from_config(config: &Config) — a constructor that resolves services from the DI injector, matching the pattern a subprocess worker_main() would use
- The name process_separate_job and error message "Cannot process in separate process" reference a "separate process" concept

But no JobConfigurator or JobWorkerProcessing implementation exists — jobs run in-process as async Tokio tasks, same as before the refactoring.

## Questions

1. Was subprocess isolation for background jobs (compaction, imports, etc.) a planned feature that was extracted as groundwork in #7094 but never completed? Or is the current in-process design intentional and the naming inherited from an earlier
prototype?
2. Is JobIsolatedProcessor::new_from_config() dead code, or is it a deliberate hook for future subprocess isolation?
3. Was there a specific motivation for the dyn JobProcessor trait boundary beyond code organization? The lifecycle methods (wait_processing_loops, stop_processing_loops) being no-ops suggests they were intended for a future implementation that would
need them.

Thanks for any insight!

Contributor guide

Open the contributing guide

Research direction

Start with PR #7094 and the JobRunner, JobProcessorImpl, and JobIsolatedProcessor definitions moved from cluster/mod.rs into cluster/ingestion/. Trace the dyn JobProcessor lifecycle methods, process_separate_job, and new_from_config() alongside the WorkerPool/WorkerConfigurator pattern. Done means documenting whether subprocess isolation was intended, whether the constructor is dead code or a future hook, and why the trait boundary exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.