appbaseio / appbaseio/abc

src_filter not working

Open
#178 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
471
Forks
47
PR merge metrics
No merged PRs in 30d

Description

```
abc --version
Version: 0.6.7
Variant: !oss
Go version: go1.11.1
OS: linux
Arch: amd64
```

I'm trying to import only one collection from a mongodb database which has three collections, but I think because two of my collections are similarly named the regex picks up both, this then fails in elastic because it tries to create two mapping types, e.g. I see this on the import:

```
INFO[0000] Connection to mongo_server:27017 established.
INFO[0000] collection count db="data" num_collections=3
INFO[0000] adding for iteration... collection="market_raw_data" db="data"
INFO[0000] adding for iteration... collection="raw_data" db="data"
INFO[0000] skipping iteration... collection=updateQueue db="data"
INFO[0000] done iterating collections db="data"
```

My transform file looks like:

```
t.Source("source", source, "/raw_data/")
.Mapping({
"raw_data": {
"properties":{
"mongo_id" : {
"properties" : {
"date": {
"type":"date",
"format": "yyyyMMdd"
}
}
}
}
}
})
// more transforms
.Save("sink", sink, "/.*/")
```
How can I exclude it from picking up this collection:

`INFO[0000] adding for iteration... collection="market_raw_data" db="data"
`

Contributor guide

Open the contributing guide

Research direction

Start by locating the src_filter handling and the MongoDB collection-iteration path, then reproduce the issue with the shown transform and three collection names. Confirm that filtering selects only raw_data and that the import no longer attempts to create a second mapping type for market_raw_data.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go, mongodb
Domain
cli, databases, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.