opensearch-project / opensearch-project/data-prepper
Add GeoIP commercial databases
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Hello
Is your feature request related to a problem? Please describe.
The documentation is not clear if we can you GeoIP commercial database (other than GeoIP enterprise database). Today I have subscription for GeoIP2-Country, GeoIP2-ISP and GeoIP2-Anonymous, I wanted to use at least GeoIP2-Country but didn't work.
It seems that geoip processor only works with Geo2Lite.
I have the following configuration:
extensions:
geoip_service:
maxmind:
databases:
country: "/usr/share/GeoIP/GeoIP2-Country.mmdb"
database_refresh_interval: PT1H
In case you can access database samples here: https://github.com/maxmind/MaxMind-DB/tree/main/test-data
Describe the solution you'd like
Have the possible to use GeoIP2 database with configuration like:
extensions:
geoip_service:
maxmind:
databases:
geoip2-country: "/usr/share/GeoIP/GeoIP2-Country.mmdb"
geoip2-isp: "/usr/share/GeoIP/GeoIP2-ISP.mmdb"
database_refresh_interval: PT1H
Describe alternatives you've considered (Optional)
Or similar to what it's done in logstash, determine the database in the processor like:
processor:
- geoip:
entries:
- source: "/clientIp"
database: "/usr/share/GeoIP/GeoIP2-Country.mmdb"
Additional context
It's seems to don't work ever when providing GeoLite2-Country.mmdb by localpath directly.
data-prepper-config.yaml:
extensions:
geoip_service:
maxmind:
databases:
country: "/usr/share/GeoIP/GeoLite2-Country.mmdb"
database_refresh_interval: PT1H
pipeline.yaml:
version: "2"
test-pipeline:
source:
http:
processor:
- parse_json:
source: "message"
- geoip:
entries:
- source: "/clientIp"
sink:
- stdout:
input data sample:
{"clientIP":"185.126.231.50"}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the geoip_service MaxMind configuration and geoip processor shown in the issue; compare the requested commercial database names with the existing country configuration. Reproduce the pipeline using the supplied GeoLite2-Country.mmdb or MaxMind test data, then consider the issue done when the configured GeoIP2 databases load and the processor enriches the sample event.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100