dgraph-io / dgraph-io/dgraph

Bulk loader: add --preserve-schema flag to error instead of auto-converting uid predicates to [uid]

Open
#9,780 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Stale
Dominant language
Go
Stars
21.8k
Forks
1.6k
Avg merge
2d 5h
Merged PRs (30d)
9

Description

Summary

The bulk loader silently rewrites a predicate's schema from uid to [uid] when it encounters more than one UID for a subject/predicate pair. This proposal adds a --preserve-schema flag so users can opt into strict behavior: fail the load with a clear error instead of widening the schema.

Current behavior

Given this schema:

<somepred>: uid .

and RDF like:

_:abc <somepred> _:123 .
_:abc <somepred> _:456 .

the bulk loader detects the conflict in the reduce phase, prints a warning, and forces the schema to [uid]:

https://github.com/dgraph-io/dgraph/blob/main/dgraph/cmd/bulk/reduce.go#L876-L891

This behavior is deliberate. It came from #3659, which fixed a data-loss bug: before that change, extra UIDs on a scalar uid predicate were silently dropped. Forcing the schema to a list was the conservative choice (keep all the data, let the user sort it out once the cluster is up). The problem is there's no way to opt out. Users who treat their schema file as the source of truth only discover the rewrite after the load completes, and the warning names the predicate but is easy to miss in bulk loader output.

Note this is a bulk-loader-only behavior. The live loader never modifies an existing schema; with <somepred>: uid already defined, live-loaded mutations follow scalar semantics (last write wins).

Proposal

Add a --preserve-schema flag to dgraph bulk (default false, preserving current behavior):

  • --preserve-schema=false (default): current behavior. Warn and widen the predicate to a list.

  • --preserve-schema=true: the schema file is authoritative. When the reducer finds more than one UID for a scalar uid predicate, fail the load with an error naming the predicate and the subject UID, e.g.:

    predicate <somepred> is defined as uid (not a list), but subject 0x2 has 2 values;
    rerun without --preserve-schema to widen the schema to [uid], or fix the input data
    

Implementation should be small: the detection already exists in reduce.go, and setSchemaAsList in dgraph/cmd/bulk/schema.go is only called from that one site. The strict path just returns an error instead.

Scope notes
  • The check runs in the reduce phase over merged posting lists, so the original RDF line is no longer available. We can report the predicate and subject UID, but per-line attribution would require map-phase tracking across shards. I'd keep that out of scope for the first pass.
  • The original request also floated a "keep the scalar schema and last-write-wins" mode. I'd skip that too: which value wins is nondeterministic in a map/reduce load, so it's a footgun. Error-or-widen covers the real use cases.
  • testBulkSingleUid in systest/bulk_live/common/bulk_live_cases.go locks in the current force-to-list behavior and should gain a strict-mode counterpart.

References

  • Original fix that introduced the auto-widening: #3659

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dgraph/cmd/bulk/reduce.go at the existing UID-conflict detection and read dgraph/cmd/bulk/schema.go, especially setSchemaAsList. Review testBulkSingleUid in systest/bulk_live/common/bulk_live_cases.go and add a strict-mode counterpart. Done means dgraph bulk accepts --preserve-schema, errors with the predicate and subject UID for scalar conflicts, and retains the current warning-and-widening behavior by default.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.