"supportsAggregatePushdown": does not work correctly on sharded Mongo Cluster
- Dominant language
- Java
- Stars
- 2k
- Forks
- 990
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 5
Description
Drill 1.2.1
Mongo 4.4
Mongodb Config: 2 shards/3 replicas
Mongos on port 27017
DrillBit:
{
"type": "DB",
"connection": "mongodb://mongo01:27017,mongo02:27017/?readPreference=secondaryPreferred",
"pluginOptimizations": {
"supportsProjectPushdown": true,
"supportsFilterPushdown": true,
"supportsAggregatePushdown": true,
"supportsSortPushdown": true,
"supportsUnionPushdown": true,
"supportsLimitPushdown": true
},
"batchSize": 100,
"enabled": true,
"authMode": "SHARED_USER"
Running a count(*) on the above drill only returns the # of records on one shard. Changing "supportsAggregatePushdown": to false reports the correct count from both shards.
Same query from mongo shell reports total across both shards.
Since we are querying mongos from drill, the count should include records from both shards, but for some reason it only reports a count on the shard that it is talking to.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing count(*) with the supplied DrillBit Mongo configuration against the two-shard Mongo 4.4 cluster, first with supportsAggregatePushdown enabled and then disabled. Compare the results with the Mongo shell total; done means aggregate pushdown returns records from both shards. The report does not name a source file or test entry point.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100