docling-project / docling-project/docling-core

`delete_items` remaps references to cascade-deleted descendants onto unrelated items and does not update `GraphCell.item_ref`

Open
#768 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 15h
Merged PRs (30d)
21

Description

`DoclingDocument.delete_items` removes the items it is handed and their descendants. It then attempts to renumber references throughout the document. The renumbering treats only the handed refs as deleted. `_update_refitems_with_lookup` skips a ref when `ref_item in refs_to_be_deleted`, and that list is the argument to `delete_items`, not the set of items removed. A caption, footnote, reference, comment or `RichTableCell.ref` pointing at a descendant of a deleted item is therefore not dropped. It goes through the index lookup, which maps the deleted index onto the slot below it. With nothing below it the result is `#/texts/-1`. The caption, footnote, reference and `RichTableCell.ref` paths construct a new `RefItem` there and raise. The comments path mutates the existing ref and leaves `#/texts/-1` in place without an error. With anything below it the ref resolves to an unrelated item, and nothing reports it.

Separately, `_update_breadth_first_with_lookup` updates `comments`, `captions`, `references`, `footnotes`, `RichTableCell.ref`, `parent`, `self_ref` and `children`, and nothing else. `GraphCell.item_ref` on `KeyValueItem.graph` and `FormItem.graph` is never rewritten. Any deletion that shifts an index leaves graph refs pointing at the wrong item or past the end of the list, whether or not the graph's target was involved.

Reproduced on docling-core 2.96.0. The same code is on `main`, in `document.py`, `_update_refitems_with_lookup` and `_update_breadth_first_with_lookup`. Both date from #220, which added the delete and insert methods. I found no earlier report.

## Reproduction

```python
from docling_core.types.doc.document import (
DoclingDocument, GraphCell, GraphCellLabel, GraphData, GraphLink, GraphLinkLabel, RefItem,
)
from docling_core.types.doc.common.reference import FineRef
from docling_core.types.doc.labels import DocItemLabel

def base(preceding):
doc = DoclingDocument(name="t")
for i in range(preceding):
doc.add_text(label=DocItemLabel.TEXT, text=f"lead {i}")
group = doc.add_inline_group()
run = doc.add_text(label=DocItemLabel.TEXT, text="Figure 1:", parent=group)
doc.add_text(label=DocItemLabel.TEXT, text="a caption", parent=group)
trailing = doc.add_text(label=DocItemLabel.TEXT, text="trailing")
return doc, group, run, trailing

# 1. caption -> a run inside the group, run is texts/0
doc, group, run, _ = base(0)
pic = doc.add_picture(caption=run)
doc.delete_items(node_items=[group]) # ValidationError: RefItem '#/texts/-1'

# 1b. comment -> the same run: no error, the ref is left invalid
doc, group, run, _ = base(0)
holder = doc.add_text(label=DocItemLabel.TEXT, text="holder")
holder.comments.append(FineRef(cref=run.self_ref))
doc.delete_items(node_items=[group])
holder.comments[0].cref # '#/texts/-1'

# 2. same, one text before the group
doc, group, run, _ = base(1)
pic = doc.add_picture(caption=run)
doc.delete_items(node_items=[group])
pic.captions[0].cref, pic.caption_text(doc) # '#/texts/0', 'lead 0' (was '#/texts/1', 'Figure 1:')

# 3. control: caption -> a survivor after the group
doc, group, run, trailing = base(1)
pic = doc.add_picture(caption=trailing)
doc.delete_items(node_items=[group])
pic.captions[0].cref, pic.caption_text(doc) # '#/texts/1', 'trailing' (correct)

# 4. control: caption -> the group itself, a handed ref
doc, group, run, trailing = base(1)
pic = doc.add_picture(caption=trailing); pic.captions = [RefItem(cref=group.self_ref)]
doc.delete_items(node_items=[group])
pic.captions # [] (dropped, correct)

def kv(doc, target):
cells = [GraphCell(cell_id=0, label=GraphCellLabel.KEY, text="k", orig="k",
item_ref=RefItem(cref=target.self_ref)),
GraphCell(cell_id=1, label=GraphCellLabel.VALUE, text="v", orig="v")]
links = [GraphLink(source_cell_id=0, target_cell_id=1, label=GraphLinkLabel.TO_VALUE)]
return doc.add_key_values(graph=GraphData(cells=cells, links=links))

# 5. GraphCell.item_ref -> a run inside the deleted group
doc, group, run, trailing = base(1)
k = kv(doc, run)
doc.delete_items(node_items=[group])
ref = k.graph.cells[0].item_ref
ref.cref, ref.resolve(doc).text # '#/texts/1', 'trailing' (was 'Figure 1:')

# 6. GraphCell.item_ref -> a survivor; only an unrelated group is deleted
doc, group, run, trailing = base(1)
k = kv(doc, trailing)
doc.delete_items(node_items=[group])
k.graph.cells[0].item_ref.resolve(doc) # IndexError: cref still '#/texts/3', texts has 2 items
```

Actual output, docling-core 2.96.0:

```
[1] RAISED ValidationError - 1 validation error for RefItem
[1b] no error; comment cref = '#/texts/-1'
[2] before: #/texts/1 -> 'Figure 1:' after: #/texts/0 -> 'lead 0'
[3] before: #/texts/3 -> 'trailing' after: #/texts/1 -> 'trailing'
[4] after: captions = []
[5] before: #/texts/1 -> 'Figure 1:' after: #/texts/1 -> 'trailing'
[6] before: #/texts/3 -> 'trailing' after: #/texts/3 -> IndexError: list index out of range
```

## Expected

Cases 3 and 4 show the intended behaviour. A ref to a survivor is renumbered. A ref to a deleted item is dropped. A ref to a cascade-deleted descendant should be treated like the handed ref in case 4, since the same call removes it. What dropping means depends on the field:

- `captions`, `footnotes`, `references` and `comments` are lists. The entry goes, as in case 4.
- `GraphCell.item_ref` is `RefItem | None`. It can become `None`. It also needs the renumbering every other ref gets, so that case 6 resolves to `'trailing'`.
- `RichTableCell.ref` is required, so there is no value to drop to. Whether the deletion is refused, the cell replaced, or something else is a policy question for the library.

For anyone editing a document programmatically the consequence can be silent corruption. After `delete_items` a picture can carry another paragraph's text as its caption, and a key-value graph can point at the wrong item or raise on resolve.

Contributor guide

Open the contributing guide

Research direction

Start in document.py at _update_refitems_with_lookup and _update_breadth_first_with_lookup, then run the reproduction against delete_items. Trace cascade-deleted descendants and GraphCell.item_ref alongside the existing caption and reference paths. Done means survivor references resolve correctly, references to removed descendants are handled consistently, and the GraphCell cases no longer point at unrelated or missing items.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.