Bulk loader: add --preserve-schema flag to error instead of auto-converting uid predicates to [uid]
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 21.8k
- Forks
- 1.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 9
Description
Summary
The bulk loader silently rewrites a predicate's schema from uid to [uid] when it encounters more than one UID for a subject/predicate pair. This proposal adds a --preserve-schema flag so users can opt into strict behavior: fail the load with a clear error instead of widening the schema.
Current behavior
Given this schema:
<somepred>: uid .
and RDF like:
_:abc <somepred> _:123 .
_:abc <somepred> _:456 .
the bulk loader detects the conflict in the reduce phase, prints a warning, and forces the schema to [uid]:
https://github.com/dgraph-io/dgraph/blob/main/dgraph/cmd/bulk/reduce.go#L876-L891
This behavior is deliberate. It came from #3659, which fixed a data-loss bug: before that change, extra UIDs on a scalar uid predicate were silently dropped. Forcing the schema to a list was the conservative choice (keep all the data, let the user sort it out once the cluster is up). The problem is there's no way to opt out. Users who treat their schema file as the source of truth only discover the rewrite after the load completes, and the warning names the predicate but is easy to miss in bulk loader output.
Note this is a bulk-loader-only behavior. The live loader never modifies an existing schema; with <somepred>: uid already defined, live-loaded mutations follow scalar semantics (last write wins).
Proposal
Add a --preserve-schema flag to dgraph bulk (default false, preserving current behavior):
-
--preserve-schema=false(default): current behavior. Warn and widen the predicate to a list. -
--preserve-schema=true: the schema file is authoritative. When the reducer finds more than one UID for a scalaruidpredicate, fail the load with an error naming the predicate and the subject UID, e.g.:predicate <somepred> is defined as uid (not a list), but subject 0x2 has 2 values; rerun without --preserve-schema to widen the schema to [uid], or fix the input data
Implementation should be small: the detection already exists in reduce.go, and setSchemaAsList in dgraph/cmd/bulk/schema.go is only called from that one site. The strict path just returns an error instead.
Scope notes
- The check runs in the reduce phase over merged posting lists, so the original RDF line is no longer available. We can report the predicate and subject UID, but per-line attribution would require map-phase tracking across shards. I'd keep that out of scope for the first pass.
- The original request also floated a "keep the scalar schema and last-write-wins" mode. I'd skip that too: which value wins is nondeterministic in a map/reduce load, so it's a footgun. Error-or-widen covers the real use cases.
testBulkSingleUidin systest/bulk_live/common/bulk_live_cases.go locks in the current force-to-list behavior and should gain a strict-mode counterpart.
References
- Original fix that introduced the auto-widening: #3659
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in dgraph/cmd/bulk/reduce.go at the existing UID-conflict detection and read dgraph/cmd/bulk/schema.go, especially setSchemaAsList. Review testBulkSingleUid in systest/bulk_live/common/bulk_live_cases.go and add a strict-mode counterpart. Done means dgraph bulk accepts --preserve-schema, errors with the predicate and subject UID for scalar conflicts, and retains the current warning-and-widening behavior by default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100