opensearch-project / opensearch-project/data-prepper

Add GeoIP commercial databases

Open
#4,407 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Hello

Is your feature request related to a problem? Please describe.
The documentation is not clear if we can you GeoIP commercial database (other than GeoIP enterprise database). Today I have subscription for GeoIP2-Country, GeoIP2-ISP and GeoIP2-Anonymous, I wanted to use at least GeoIP2-Country but didn't work.
It seems that geoip processor only works with Geo2Lite.

I have the following configuration:

extensions:
  geoip_service:
    maxmind:
      databases:
        country: "/usr/share/GeoIP/GeoIP2-Country.mmdb"
      database_refresh_interval: PT1H

In case you can access database samples here: https://github.com/maxmind/MaxMind-DB/tree/main/test-data

Describe the solution you'd like
Have the possible to use GeoIP2 database with configuration like:

extensions:
  geoip_service:
    maxmind:
      databases:
        geoip2-country: "/usr/share/GeoIP/GeoIP2-Country.mmdb"
        geoip2-isp: "/usr/share/GeoIP/GeoIP2-ISP.mmdb"
      database_refresh_interval: PT1H

Describe alternatives you've considered (Optional)
Or similar to what it's done in logstash, determine the database in the processor like:

  processor:
    - geoip:
        entries:
          - source: "/clientIp"
        database: "/usr/share/GeoIP/GeoIP2-Country.mmdb"

Additional context
It's seems to don't work ever when providing GeoLite2-Country.mmdb by localpath directly.

data-prepper-config.yaml:

extensions:
  geoip_service:
    maxmind:
      databases:
        country: "/usr/share/GeoIP/GeoLite2-Country.mmdb"
      database_refresh_interval: PT1H

pipeline.yaml:

version: "2"
test-pipeline:
  source:
    http:
  processor:
    - parse_json:
        source: "message"
    - geoip:
        entries:
          - source: "/clientIp"
  sink:
    - stdout:

input data sample:

{"clientIP":"185.126.231.50"}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the geoip_service MaxMind configuration and geoip processor shown in the issue; compare the requested commercial database names with the existing country configuration. Reproduce the pipeline using the supplied GeoLite2-Country.mmdb or MaxMind test data, then consider the issue done when the configured GeoIP2 databases load and the processor enriches the sample event.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.