cloudflare / cloudflare/py-mmdb-encoder
inserting subnets in non-strict mode fail if more-specific subnet is inserted first
- Dominant language
- Python
- Stars
- 31
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
Hi Cloudflare,
first please let me thank you for your code. It is really amazing.
We at Showmax are using it and have found an issue (a corner case, if you wish) which results in some prefixes not being stored into the generated file. The issue can be demonstrated on the following piece of code:
```
#!/usr/bin/env python3
"""Geoexceptions file generator."""
import argparse
import logging
import logging.handlers
import sys
import geoip2.database
import mmdbencoder
from geoip2.errors import AddressNotFoundError
def exception_data(country_code):
"""
Format the data as needed by the MaxMind Country database format.
Do not modify unless really sure what are you doing!
:param country_code: ISO 2-character country code, e.g. ZA
:return: MaxMind Country DB structure needed for an IP network record.
"""
return {
'continent': {},
'registered_country': {},
'represented_country': {},
'traits': {},
'country': {
'iso_code': country_code,
},
}
def tryip(reader, ip):
"""Try to retrieve country code from the MaxMind database."""
try:
return reader.country(ip).country.iso_code
except AddressNotFoundError:
# Address is not present in the database
return None
def main():
"""Main."""
logging.basicConfig(stream=sys.stdout)
logging.getLogger()
logging.getLogger().setLevel(logging.DEBUG)
logging.debug('Test case #1')
test_pfxs = ['2001:db8:807:1::1', '2001:db8:807:102::1', '2001:db8:807:111::1']
fname = 'test.mmdb'
cc_za = exception_data('ZA')
cc_ke = exception_data('KE')
enc = mmdbencoder.Encoder(
6, # IP version
32, # Size of the pointers
'Country', # Name of the table
['en'], # Languages
{'en': 'Geoexceptions Country Database'}, # Description
compat=True)
data_za = enc.insert_data(cc_za)
data_ke = enc.insert_data(cc_ke)
enc.insert_network('2001:db8:807::/48', data_za, False)
enc.insert_network('2001:db8:807:102::/64', data_ke, False)
enc.write_file(fname)
reader = geoip2.database.Reader(fname)
for pfx in test_pfxs:
logging.debug('prefix %s, present? %s', pfx, tryip(reader, pfx))
logging.debug('Test case #2')
fname = 'test.mmdb'
cc_za = exception_data('ZA')
cc_ke = exception_data('KE')
enc = mmdbencoder.Encoder(
6, # IP version
32, # Size of the pointers
'Country', # Name of the table
['en'], # Languages
{'en': 'Geoexceptions Country Database'}, # Description
compat=True)
data_za = enc.insert_data(cc_za)
data_ke = enc.insert_data(cc_ke)
enc.insert_network('2001:db8:807:102::/64', data_ke, False)
enc.insert_network('2001:db8:807::/48', data_za, False)
enc.write_file(fname)
reader = geoip2.database.Reader(fname)
for pfx in test_pfxs:
logging.debug('prefix %s, present? %s', pfx, tryip(reader, pfx))
logging.debug('Tests completed.')
if __name__ == '__main__':
main()
```
The resulting output is:
```
$ python test_subnet.py
DEBUG:root:Test case #1
DEBUG:root:prefix 2001:db8:807:1::1, present? ZA
DEBUG:root:prefix 2001:db8:807:102::1, present? KE
DEBUG:root:prefix 2001:db8:807:111::1, present? ZA
DEBUG:root:Test case #2
DEBUG:root:prefix 2001:db8:807:1::1, present? ZA
DEBUG:root:prefix 2001:db8:807:102::1, present? KE
DEBUG:root:prefix 2001:db8:807:111::1, present? None
DEBUG:root:Tests completed.
```
If the more-specific prefix is inserted first, the less-specific data for `2001:db8:807:100::/63`, `2001:db8:807:103::`..`2001:db8:807:1ff:ffff:ffff:ffff:ffff` are missing from the resulting file.
I believe this should not be happening.
(If strict mode is disabled, both cases end up with error `Exception: Encoder: add_to_trie: try setting data on a non-final: (...128bit-int-number) already has child. Not updating in strict mode.`)
Contributor guide
Assessment
This issue has not been assessed yet.