Experiment: low-cost mesh-to-VoxelPlan pipeline using ordinary LLMs

Context

CraftDAG currently has a strong semantic architecture path:

Prompt
→ BuildIntent / ComponentPlan
→ CraftDAG
→ VoxelPlan
→ preview / materials / layers / schematic export

This works well for architecture and other structures that can be expressed cleanly with bounded semantic components.

However, the current demand coverage matrix already identifies a weaker area:

• sculptures
• organic creatures
• pixel art
• freeform voxel forms
• mesh/image/heightmap-style inputs

See #81. The adapter boundary in #29 also explicitly allows inputs that produce VoxelPlan directly when semantic recovery is not useful.

Recent general-purpose coding/agent models are increasingly capable of driving Blender or producing 3D geometry. That suggests a second generation frontend worth testing:

Prompt / reference
→ ordinary LLM + Blender or mesh tooling
→ 3D scene / GLB / OBJ
→ deterministic mesh voxelization
→ Minecraft-native block resolution
→ VoxelPlan
→ existing CraftDAG downstream pipeline

The important architectural point is that this should not require reverse-compiling arbitrary geometry into ComponentPlan.

VoxelPlan is already the natural target representation for geometry that has no useful semantic component decomposition.

Core hypothesis

A low-cost, ordinary model such as GPT-5.6 Luna can generate or manipulate sufficiently useful 3D geometry, while deterministic tooling handles the expensive precision work needed to convert that geometry into a Minecraft-quality VoxelPlan.

The experiment should explicitly test whether this route works without depending on the most expensive frontier model.

If the pipeline only works with a top-tier model on every generation, it is probably not an attractive production direction for MinePilot because inference cost would dominate a feature whose deterministic parts should be handled by code.

Design principle

The desired split is:

LLM responsibility
- understand the request
- create/modify coarse 3D geometry
- optionally attach semantic/material hints

Deterministic compiler responsibility
- normalize scale
- voxelize geometry
- enforce block budgets
- classify shell/interior occupancy
- map materials to Minecraft palettes
- choose Minecraft-native block states/shapes
- validate and emit VoxelPlan

Do not ask the LLM to emit hundreds of thousands of blocks directly.

Do not require the LLM to infer exact Minecraft coordinates if deterministic geometry tooling can do that more reliably and cheaply.

Proposed spike

This is an experiment, not a commitment to a new production subsystem.

Stage 0 — prove mesh → VoxelPlan without any LLM

Build the smallest possible adapter:

GLB / OBJ
→ normalize bounds + target resolution
→ voxel occupancy grid
→ simple Minecraft palette
→ VoxelPlan
→ existing preview / schematic path

This separates the fundamental compiler question from model capability.

Minimum parameters:

targetHeight / targetBounds
maxBlocks
voxelMode: surface | solid | shell
wallThickness (if shell)
defaultPalette

A successful Stage 0 proves that arbitrary continuous geometry can enter CraftDAG through the existing target IR.

Stage 1 — low-cost model generation

Use a normal GPT-5.6 Luna-class model through Codex/agent tooling to generate or edit Blender geometry for a small fixed benchmark set.

Do not optimize the benchmark around the model.

Suggested cases:

1. simple house
2. castle tower
3. ship or vehicle silhouette
4. statue
5. dragon / organic creature

For each object, try at least a few target resolutions, for example:

32 blocks high
64 blocks high
128 blocks high

The goal is not photorealistic Blender quality. The goal is whether the resulting geometry retains enough structure after voxelization to become a recognizable and useful Minecraft build.

Stage 2 — Minecraft-native post-processing

Only if Stage 0/1 are promising, evaluate deterministic post-processing such as:

sloped voxel run      → stairs
half-height surface   → slabs
thin transparent wall → glass panes
thin vertical edge    → walls / fences
pillar-like structure → logs / walls

This stage is important because a raw voxelized mesh can look like cheap voxel art rather than an intentional Minecraft build.

Stage 3 — optional semantic hints

If needed, allow the upstream model/Blender scene to emit bounded hints such as:

{
  "object": "main_tower",
  "voxelMode": "shell",
  "wallThickness": 2,
  "materialRole": "stone_wall"
}

Hints should improve deterministic compilation, not replace it.

Benchmark questions

Record answers to these questions for every fixture:

Geometry

• Is the result recognizable at 32 / 64 / 128 block scale?
• Which details disappear at each resolution?
• Does shell voxelization produce usable interiors?
• Does block count remain bounded and predictable?

Minecraft quality

• Does the result look like an intentional Minecraft build rather than arbitrary voxelized geometry?
• Which post-processing operations give the largest quality improvement?
• Can material mapping produce coherent palettes without model-authored per-block output?

Model dependence

• Can GPT-5.6 Luna create/edit the required Blender geometry reliably enough?
• How many retries are needed per fixture?
• Can deterministic scripts repair most failures?
• Is a more expensive model materially better enough to justify its cost?

Cost

Track approximate per-generation model usage and retries.

The desired production shape is:

cheap model reasoning + deterministic geometry compiler

not:

expensive frontier model repeatedly reasoning over every block

Success criteria

The route is worth continuing if a small spike can demonstrate all of the following:

• At least one GLB/OBJ fixture converts deterministically into a valid VoxelPlan.
• Existing CraftDAG/MinePilot preview and schematic/export tooling can consume that result without a parallel target format.
• At least 4/5 benchmark objects are recognizable at a practical Minecraft scale.
• Freeform/organic cases show a clear advantage over forcing the same geometry through ComponentPlan.
• A GPT-5.6 Luna-class model can drive the upstream geometry workflow with acceptable retry rates.
• No top-tier model is required for the normal path.
• Block count and scale can be bounded deterministically.
• The experiment identifies a plausible path from raw voxels to Minecraft-native stairs/slabs/walls/panes or explicitly concludes that this is the main blocker.

Failure / stop criteria

Stop or defer this direction if one of these remains true after the spike:

• usable geometry generation requires an expensive frontier model for ordinary cases;
• voxelized outputs are only useful at impractically large resolutions;
• Minecraft-native post-processing requires as much model reasoning as direct block generation;
• outputs are consistently inferior to ComponentPlan/CraftDAG for both architectural and freeform builds;
• the resulting pipeline requires a separate execution/export stack instead of reusing VoxelPlan.

A negative result is still useful: it tells us not to expand this into a production feature.

Architecture boundary

If this works, the intended shape is a new input adapter/compiler frontend, not a replacement for CraftDAG semantic construction:

                     ┌→ ComponentPlan → CraftDAG ─┐
Prompt / reference ──┤                            ├→ VoxelPlan
                     └→ Mesh / Blender → Voxelizer┘

Use semantic construction where semantics are valuable.

Use mesh voxelization where geometry is inherently freeform.

A future MinePilot planner could choose between the two, or combine both in one build (for example, semantic castle + voxelized dragon statue).

Non-goals

• Do not integrate a specific frontier model as a hard dependency.
• Do not make Blender a required runtime dependency for CraftDAG core.
• Do not add sculpture/dragon-specific ComponentPlan primitives.
• Do not attempt mesh → ComponentPlan semantic reconstruction in the first experiment.
• Do not let the model emit huge per-block JSON payloads.
• Do not redesign VoxelPlan unless the spike exposes a concrete missing target capability.
• Do not productionize this before the benchmark gives evidence that the route is cost-effective.

Suggested implementation shape

If Stage 0 succeeds, consider a narrow adapter/package boundary such as:

adapter-mesh
  parse GLB/OBJ
  normalize transform
  voxelize
  material-role mapping
  emit VoxelPlan

Blender can be one upstream producer of mesh files, but the durable capability should be mesh-to-Minecraft compilation, not a Blender-specific integration.

Related

• #81 — Minecraft build demand coverage; freeform/sculpture gap
• #29 — input adapter architecture
• #28 — schematic-to-VoxelPlan precedent
• #110 — intentionally keeps true freeform mesh work outside semantic curve primitives

Start with the Stage 0 mesh-to-VoxelPlan spike, using #28 as the schematic-to-VoxelPlan precedent and #29 for the adapter boundary. First verify the existing VoxelPlan preview and export path, then measure whether a GLB/OBJ fixture can be normalized, voxelized, bounded, and consumed without a parallel format; the benchmark and success criteria define done.