yammdb - just another .mmdb

NeoonNeoon OG
edited May 2023 in Technical

Hey,

Today I wanted to present you, yammdb.
Which is another, different, geodatabase, based on real measurements.

You can find it here: https://github.com/Ne00n/yammdb
The Database is build weekly, on fridays.

As long the buildserver doesn't blow up.
The primary use case for me is to compare existing geo data, maybe someone of you will find it useful.

If you have any ideas and feedback, please lemme know.

Enjoy.

Free NAT KVM | Free NAT LXC

Tagged:
«1

Comments

  • NeoonNeoon OG

    Latency is now included from the closest location.

    Free NAT KVM | Free NAT LXC

  • schismschism

    Umm I'm stupid and new to all this. What is this???

  • somiksomik OG Hostbusters

    @schism said:
    Umm I'm stupid and new to all this. What is this???

    Put in a IP, it'll tell you where the physical location of the server/host of the IP is.

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • schismschism

    Oh cool!

  • FrankZFrankZ ModeratorOG

    I'm trying this out with my gDNSd servers. The db seems much smaller than the ip-to-city-lite db I have been using, 16MB compared to 99MB. If this works as well it is going to save me a bunch of RAM. :)
    Thank you for this !!

  • NeoonNeoon OG

    @FrankZ said:
    I'm trying this out with my gDNSd servers. The db seems much smaller than the ip-to-city-lite db I have been using, 16MB compared to 99MB. If this works as well it is going to save me a bunch of RAM. :)
    Thank you for this !!

    It has way less "useless" data on it, hence its so smol.
    However, It should work with auto_dc maps using geo cords but I have no idea how accurate it is.

    Free NAT KVM | Free NAT LXC

  • dwightdwight

    Thank you! I hope there is an acl version for bind9

  • NeoonNeoon OG

    @Neoon said:

    @FrankZ said:
    I'm trying this out with my gDNSd servers. The db seems much smaller than the ip-to-city-lite db I have been using, 16MB compared to 99MB. If this works as well it is going to save me a bunch of RAM. :)
    Thank you for this !!

    It has way less "useless" data on it, hence its so smol.
    However, It should work with auto_dc maps using geo cords but I have no idea how accurate it is.

    I wrote a smol, 30 lines benchmark script, using a dump of the global routing table.
    65%, 638k from 975k in the routing table, which is good, I expected less.

    Github page said 600k.
    Meanwhile the other .mmdb's have a 99% hit rate.

    TLDR: Yes use it, but not as primary database, build your own, as my primary purpose was.

    Free NAT KVM | Free NAT LXC

  • FrankZFrankZ ModeratorOG

    @Neoon said: TLDR: Yes use it, but not as primary database, build your own, as my primary purpose was.

    Fair enough.

  • NeoonNeoon OG
    edited May 2023

    @FrankZ said:

    @Neoon said: TLDR: Yes use it, but not as primary database, build your own, as my primary purpose was.

    Fair enough.

    if you build your own, you prob, can drop the memory usage even further.
    Throw everything out and turn it into a flying gas can.

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG

    Speaking of hit rate, the benchmark I ran yesterday, gave me a 65% hit rate, against a routing table dump.
    Which is more than I expected, roughly 648k from 975k, however, the second benchmark I ran, hit only 35% with 8.5 Million IP addresses.

    This was due, that bigger subnets are splitted into smaller ones for more accurate data, however the ones which didn't respond where not filled, which has been fixed.

    After fixing these bugs, the hit rate is at 74%.
    The next step would be to see, where I can further improve the data I use for all of this, so I end up with higher hit rates.

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG

    Thanks to https://virtury.com/ we got a new Probe in Pakistan!

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG
    edited May 2023

    I took a bit longer than expected, however the software is now mostly optimized for more probes.
    Expect more probes in the next weeks.

    Daily test builds, not guaranteed, will be available under https://yammdb.serv.app/test.mmdb
    Weekly build will happen as usual.

    Also, thanks to https://ginernet.com for a new probe in Madrid.

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG

    @Neoon said:
    I took a bit longer than expected, however the software is now mostly optimized for more probes.
    Expect more probes in the next weeks.

    Daily test builds, not guaranteed, will be available under https://yammdb.serv.app/test.mmdb
    Weekly build will happen as usual.

    Also, thanks to https://ginernet.com for a new probe in Madrid.

    It was supposedly to be already done, however I did a fuck up.
    One function had the build hang for hours over hours.

    This is fixed, thanks to GPT4 once again.
    It should finish in a bit, once this is done, I will make a second test build, including a bunch of new locations.

    Free NAT KVM | Free NAT LXC

  • someTomsomeTom

    i noticed its now smaller than the previous version i tested. did you reduce the coverage or change the format?
    however, cool and useful idea to crosscheck other geolocators.

  • NeoonNeoon OG

    @someTom said:
    i noticed its now smaller than the previous version i tested. did you reduce the coverage or change the format?
    however, cool and useful idea to crosscheck other geolocators.

    No, but I did noticed it too.
    The only way I can explain it, is how the writer builds the database.

    Basically I tried to aggregate the prefixes, to make the database even smaller, however it seems like the writer already does this.
    So the size did not change after all, a while ago, the database had a lot of gaps, because the way it does ping bigger subnets.

    These gaps have been closed, hence I do assume, that the writer now can optimize / aggregate the database even further, hence its smaller. The code definitely does not or has remove data.

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG

    I added a few more Locations for this test run.

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG

    Gonna be the biggest Friday run, yet.

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG
    edited May 2023

    Thanks to some people that followed my github repo, they actually gave me the idea, to make an mtr only geo database.
    I did code it in less than 24 hours, however, the hit rates where to low and my brain did not manage to figure out yet where the fuck up was.

    However, today I found the mapping error.
    db/mtr.mmdb {'fail': 126502, 'success': 849072, 'percentage': 87.03306976200678}

    From 64% to 74% now 87% hitrate, not bad.
    I put the .mmdb as usual on https://yammdb.serv.app/mtr.mmdb

    This database is only 4.2MB in size, only contains geo coordinates, right now.
    I will add the usual info in a later build, such as country, continent etc.

    Plus I will add a combined build later, with geo.mmdb and mtr.mmdb which first uses latency, then mtr for better accuracy.

    Free NAT KVM | Free NAT LXC

  • somiksomik OG Hostbusters

    Any plans to release a CSV version of the mmdb?

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • NeoonNeoon OG

    I updated the mtr.mmdb, it does now include continent, country and latency same as the geo.mmdb.
    @somik Sure, I added the csv file: https://yammdb.serv.app/mtr.csv

    Currently they are smaller than the geo.mmdb due to less measurements per subnet, this will change once I run them again.

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG

    I also added geo.mmdb as csv: https://yammdb.serv.app/geo.csv
    There is no compression or anything, hence the file is so big.

    Usually the .mmdb writer does the compression.

    Free NAT KVM | Free NAT LXC

  • somiksomik OG Hostbusters

    @Neoon said:
    I also added geo.mmdb as csv: https://yammdb.serv.app/geo.csv
    There is no compression or anything, hence the file is so big.

    Usually the .mmdb writer does the compression.

    Best to have it without compression for maximum compatibility. I'm visiting our neighbouring country for some good foods now, so I'll test it out once I go back to Singapore.

    On that note, seems like a lot of shops closed down over the last pandemic... Sad days.

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • NeoonNeoon OG

    Well, I guess a .mmtr only database with more tests per subnet, won't be happening.
    It takes to long, roughly 1-2 days to finish a build with roughly 8+ million targets.

    Even with 20 probes, running, at the same time.
    Instead I am going to run another test build next week, which does .mtr on subnet's that doesn't ping and combines them with the latency results as mentioned before.

    Free NAT KVM | Free NAT LXC

  • somiksomik OG Hostbusters

    @Neoon I think your CSV headers (table titles) are missing for both geo.csv and mtr.csv

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • NeoonNeoon OG

    @somik said:
    @Neoon I think your CSV headers (table titles) are missing for both geo.csv and mtr.csv

    I will change that before the next build tomorraw.

    Free NAT KVM | Free NAT LXC

  • NeoonNeoon OG

    It seems that a recent masscan is mandatory, I still used a 2 months old one.
    The build just finished 3 hours earlier and with +7% higher hitrate, so 80% without mtr.

    Lesson learned, masscan will be updated at least once per week, gg.
    As soon I get the mtr integration working It should easily get a 90%+ hitrate.

    Free NAT KVM | Free NAT LXC

  • somiksomik OG Hostbusters

    @Neoon said:
    It seems that a recent masscan is mandatory, I still used a 2 months old one.
    The build just finished 3 hours earlier and with +7% higher hitrate, so 80% without mtr.

    Lesson learned, masscan will be updated at least once per week, gg.
    As soon I get the mtr integration working It should easily get a 90%+ hitrate.

    I thought port scanning was frowned upon by most data-centers/hosts?

    Btw, what's MRT?

    90% hit rate as in for IPs or returning correct geo-loc/country?

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • NeoonNeoon OG

    @somik said:

    @Neoon said:
    It seems that a recent masscan is mandatory, I still used a 2 months old one.
    The build just finished 3 hours earlier and with +7% higher hitrate, so 80% without mtr.

    Lesson learned, masscan will be updated at least once per week, gg.
    As soon I get the mtr integration working It should easily get a 90%+ hitrate.

    I thought port scanning was frowned upon by most data-centers/hosts?

    I never said I was port scanning.

    90% hit rate as in for IPs or returning correct geo-loc/country?

    hit rate means, you get a result.
    Accuracy depends on the amount of locations.

    Free NAT KVM | Free NAT LXC

  • somiksomik OG Hostbusters

    @Neoon said:

    @somik said:

    @Neoon said:
    It seems that a recent masscan is mandatory, I still used a 2 months old one.
    The build just finished 3 hours earlier and with +7% higher hitrate, so 80% without mtr.

    Lesson learned, masscan will be updated at least once per week, gg.
    As soon I get the mtr integration working It should easily get a 90%+ hitrate.

    I thought port scanning was frowned upon by most data-centers/hosts?

    I never said I was port scanning.

    90% hit rate as in for IPs or returning correct geo-loc/country?

    hit rate means, you get a result.
    Accuracy depends on the amount of locations.

    Isn't masscan used for port scanning? Are you using it to scan for something else?

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • NeoonNeoon OG
    edited June 2023

    @somik said:

    @Neoon said:

    @somik said:

    @Neoon said:
    It seems that a recent masscan is mandatory, I still used a 2 months old one.
    The build just finished 3 hours earlier and with +7% higher hitrate, so 80% without mtr.

    Lesson learned, masscan will be updated at least once per week, gg.
    As soon I get the mtr integration working It should easily get a 90%+ hitrate.

    I thought port scanning was frowned upon by most data-centers/hosts?

    I never said I was port scanning.

    90% hit rate as in for IPs or returning correct geo-loc/country?

    hit rate means, you get a result.
    Accuracy depends on the amount of locations.

    Isn't masscan used for port scanning?

    You should read up what masscan can do.

    Are you using it to scan for something else?

    Yes, as I said a few times.

    Free NAT KVM | Free NAT LXC

  • GanonkGanonk

    Hello Sir, can you explain to me how to use that mmdb?

    i never using mmdb before. thanks

  • tuctuc

    Any database guru please help with an indexing and sql statement for best performance on searching the dataset given an IP address.
    Thanks in advance :3

    Virmach Deals

  • NeoonNeoon OG
    edited June 2023

    @tuc said:
    Any database guru please help with an indexing and sql statement for best performance on searching the dataset given an IP address.
    Thanks in advance :3

    Probably if you store it in binary, the .mmdb uses a binary search tree and does lookups in a fraction of a second, also just 15MB in size compared to the csv.

    @Ganonk said:
    Hello Sir, can you explain to me how to use that mmdb?

    i never using mmdb before. thanks

    You provide a IP address, it gives you the approximate location.
    Python example is on github.

    Free NAT KVM | Free NAT LXC

  • GanonkGanonk
    edited June 2023

    sorry, i mean step by step running/read file .mmdb , like as apt install blabla , and than next step.
    on linux , or android , or windows

  • somiksomik OG Hostbusters
    edited June 2023

    @tuc said:
    Any database guru please help with an indexing and sql statement for best performance on searching the dataset given an IP address.
    Thanks in advance :3

    Convert the IP/subnet (CIDR) to a IP range containing the starting and ending IP.
    Convert the starting and ending IP to a "long" number.

    For PHP, you can use:

    function ipRange($cidr) {
    $range = array();
    $cidr = explode('/', $cidr);
    $range[0] = long2ip((ip2long($cidr[0])) & ((-1 << (32 - (int)$cidr[1]))));
    $range[1] = long2ip((ip2long($range[0])) + pow(2, (32 - (int)$cidr[1])) - 1);
    return $range;
}

    Save it in the DB like: index, startingIPLong, endingIPLong, Location

    When you want to check a IP, convert it to "long" number, and find the row WHERE startingIPLong>= searchIP AND endingIPLong<= searchIP

    I think you'll be able to figure it out from there?

    For PHP to convert the IP to long, use function: https://www.php.net/manual/en/function.ip2long.php

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • tuctuc

    Thanks @somik for the function. That's really help.

    Virmach Deals

  • somiksomik OG Hostbusters

    @somik said:
    When you want to check a IP, convert it to "long" number, and find the row WHERE startingIPLong>= searchIP AND endingIPLong<= searchIP

    Btw, i did it other way around. Guess should not be browsing LES first thing after waking up:

    WHERE startingIPLong <= searchIP AND endingIPLong >= searchIP

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • tuctuc

    @somik said:

    @somik said:
    When you want to check a IP, convert it to "long" number, and find the row WHERE startingIPLong>= searchIP AND endingIPLong<= searchIP

    Btw, i did it other way around. Guess should not be browsing LES first thing after waking up:

    WHERE startingIPLong <= searchIP AND endingIPLong >= searchIP

    Just a small typo mistake :)

    Virmach Deals

  • tuctuc
    edited June 2023

    I just made a tiny shell script to add two columns: ipfrom and ipto to the datasets made available by @Neoon. Not sure if my style of making data ready before importing into databases will help :)

    #!/bin/bash

if [ -z "$1" ]; then exit; fi
while IFS=',' read -r ipr cont cnt lat lng rad; do
  if [ -z "$ipr" ]; then break; fi
  IFS='/' read -r IP CIDR <<<"$ipr"
  MASK=$((-1 << (32 - ${CIDR})))
  IFS='.' read a b c d <<<${IP}
  base_ip=$((($b << 16) + ($c << 8) + $d))
  ipfrom=$((${base_ip} & ${MASK}))
  ipto=$((($a << 24) + ((${ipfrom} | ~${MASK}) & 0x7FFFFFFF)))
  ipfrom=$((($a << 24) + $ipfrom))
  echo "$ipr,$ipfrom,$ipto,$cont,$cnt,$lat,$lng,$rad"
done <$1

    To use the script:

    script-name data-file | cat > new-data-file

    output sample

    1.0.0.0/24,16777216,16777471,EU,CZ,50.08,14.41,0
1.1.1.0/24,16843008,16843263,EU,CZ,50.08,14.41,0
2.16.2.0/24,34603520,34603775,EU,CZ,50.08,14.41,0
2.56.253.0/24,37289216,37289471,EU,CZ,50.08,14.41,0
3.33.144.0/20,52531200,52535295,EU,CZ,50.08,14.41,0
5.23.39.0/24,85403392,85403647,EU,CZ,50.08,14.41,0
5.39.200.0/22,86493184,86494207,EU,CZ,50.08,14.41,0
5.59.34.0/24,87761408,87761663,EU,CZ,50.08,14.41,0
5.145.104.0/22,93415424,93416447,EU,CZ,50.08,14.41,0
5.145.104.0/22,93415424,93416447,EU,CZ,50.08,14.41,0

    Virmach Deals

  • somiksomik OG Hostbusters

    @tuc said:
    I just made a tiny shell script to add two columns: ipfrom and ipto to the datasets made available by @Neoon. Not sure if my style of making data ready before importing into databases will help :)

    #!/bin/bash

if [ -z "$1" ]; then exit; fi
while IFS=',' read -r ipr cont cnt lat lng rad; do
  if [ -z "$ipr" ]; then break; fi
  IFS='/' read -r IP CIDR <<<"$ipr"
  MASK=$((-1 << (32 - ${CIDR})))
  IFS='.' read a b c d <<<${IP}
  base_ip=$((($b << 16) + ($c << 8) + $d))
  ipfrom=$((${base_ip} & ${MASK}))
  ipto=$((($a << 24) + ((${ipfrom} | ~${MASK}) & 0x7FFFFFFF)))
  ipfrom=$((($a << 24) + $ipfrom))
  echo "$ipr,$ipfrom,$ipto,$cont,$cnt,$lat,$lng,$rad"
done <$1

    To use the script:

    script-name data-file | cat > new-data-file

    output sample

    1.0.0.0/24,16777216,16777471,EU,CZ,50.08,14.41,0
1.1.1.0/24,16843008,16843263,EU,CZ,50.08,14.41,0
2.16.2.0/24,34603520,34603775,EU,CZ,50.08,14.41,0
2.56.253.0/24,37289216,37289471,EU,CZ,50.08,14.41,0
3.33.144.0/20,52531200,52535295,EU,CZ,50.08,14.41,0
5.23.39.0/24,85403392,85403647,EU,CZ,50.08,14.41,0
5.39.200.0/22,86493184,86494207,EU,CZ,50.08,14.41,0
5.59.34.0/24,87761408,87761663,EU,CZ,50.08,14.41,0
5.145.104.0/22,93415424,93416447,EU,CZ,50.08,14.41,0
5.145.104.0/22,93415424,93416447,EU,CZ,50.08,14.41,0

    That's what I did as well, just with PHP. Process the data before importing it into DB to make subsequent queries faster.

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • Not_OlesNot_Oles Provider

    @Neoon said: @Ganonk said:

    Hello Sir, can you explain to me how to use that mmdb?

    i never using mmdb before. thanks

    You provide a IP address, it gives you the approximate location.
    Python example is on github.

    Hi @Ganonk! Hi @Neoon!

    I never tried something like this before. Here is a transcript of the steps I tried. Maybe the transcript will be helpful to anybody who can show me my mistakes or who wants a beginner style recipe.

    • Step 1: Make an yammdb neighborhood
    notoles@fsn:~$ mkdir neoon
notoles@fsn:~$ cd neoon/
notoles@fsn:~/neoon$
    • Step 2: Grab the data
    notoles@fsn:~/neoon$ time wget https://yammdb.serv.app/geo.mmdb
--2023-06-04 05:17:49--  https://yammdb.serv.app/geo.mmdb
Resolving yammdb.serv.app (yammdb.serv.app)... 45.130.21.225
Connecting to yammdb.serv.app (yammdb.serv.app)|45.130.21.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16172538 (15M) [application/octet-stream]
Saving to: ‘geo.mmdb’

geo.mmdb                100%[==============================>]  15.42M  71.8MB/s    in 0.2s    

2023-06-04 05:17:49 (71.8 MB/s) - ‘geo.mmdb’ saved [16172538/16172538]


real    0m0.382s
user    0m0.013s
sys     0m0.017s
notoles@fsn:~/neoon$ ls
geo.mmdb
notoles@fsn:~/neoon$ ls -al
total 15804
drwxr-xr-x 2 notoles notoles     4096 Jun  4 05:17 .
drwx--x--x 9 notoles notoles     4096 Jun  4 05:17 ..
-rw-r--r-- 1 notoles notoles 16172538 Jun  2 17:03 geo.mmdb
notoles@fsn:~/neoon$
    • Step 3: Grab the program with which to read the data.

    I cut and pasted the program from https://github.com/Ne00n/yammdb

    notoles@fsn:~/neoon$ ed read.py
read.py: No such file or directory
a
import geoip2.database

reader = geoip2.database.Reader("geo.mmdb")
response = reader.city("1.1.1.1")
print("Continent",response.continent.code)
print("Country",response.country.iso_code)
print("Latitude",response.location.latitude,"Longitude",response.location.longitude)
print(f"Latency {response.location.accuracy_radius}ms")
.
w
329
q
notoles@fsn:~/neoon$

    notoles@fsn:~/neoon$ sudo apt-get install python3-geoip2
[sudo] password for notoles: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  python3-maxminddb
Suggested packages:
  python-maxmindb-doc
The following NEW packages will be installed:
  python3-geoip2 python3-maxminddb
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 50.4 kB of archives.
After this operation, 203 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://deb.debian.org/debian unstable/main amd64 python3-maxminddb amd64 2.2.0-1+b1 [27.6 kB]
Get:2 http://deb.debian.org/debian unstable/main amd64 python3-geoip2 all 2.9.0+dfsg1-5 [22.8 kB]
Fetched 50.4 kB in 0s (1,435 kB/s)        
Selecting previously unselected package python3-maxminddb.
(Reading database ... 56808 files and directories currently installed.)
Preparing to unpack .../python3-maxminddb_2.2.0-1+b1_amd64.deb ...
Unpacking python3-maxminddb (2.2.0-1+b1) ...
Selecting previously unselected package python3-geoip2.
Preparing to unpack .../python3-geoip2_2.9.0+dfsg1-5_all.deb ...
Unpacking python3-geoip2 (2.9.0+dfsg1-5) ...
Setting up python3-maxminddb (2.2.0-1+b1) ...
Setting up python3-geoip2 (2.9.0+dfsg1-5) ...
notoles@fsn:~/neoon$ ls
geo.mmdb  read.py
notoles@fsn:~/neoon$
    • Step 5: Get an IP address to check
    notoles@fsn:~/neoon$ w
 20:22:21 up 3 days, 21:00,  1 user,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
notoles  pts/0    187.189.238.1    20:17    0.00s  0.03s   ?    w
notoles@fsn:~/neoon$
    • Step 6: Read the data for the IP address
    notoles@fsn:~/neoon$ time python3 ./read.py 187.189.238.1
Continent EU
Country CZ
Latitude 50.08 Longitude 14.41
Latency 0.0ms

real    0m0.026s
user    0m0.022s
sys     0m0.004s
notoles@fsn:~/neoon$
    • Step 7: Ponder

    The IP address that I am using seems to be shown as located in CZ. However, the address best might be located here where I am using it, visiting, in Sonora, Mexico. :) Probably I made some mistakes! Is everything reported as EU/CZ because I need to add additional, basic steps? Can someone please help me understand why my geolocation result seems possibly incorrect?

  • NeoonNeoon OG
    edited June 2023

    @Not_Oles said:

    @Neoon said: @Ganonk said:

    Hello Sir, can you explain to me how to use that mmdb?

    i never using mmdb before. thanks

    You provide a IP address, it gives you the approximate location.
    Python example is on github.

    Hi @Ganonk! Hi @Neoon!

    I never tried something like this before. Here is a transcript of the steps I tried. Maybe the transcript will be helpful to anybody who can show me my mistakes or who wants a beginner style recipe.

    • Step 1: Make an yammdb neighborhood
    notoles@fsn:~$ mkdir neoon
notoles@fsn:~$ cd neoon/
notoles@fsn:~/neoon$
    • Step 2: Grab the data
    notoles@fsn:~/neoon$ time wget https://yammdb.serv.app/geo.mmdb
--2023-06-04 05:17:49--  https://yammdb.serv.app/geo.mmdb
Resolving yammdb.serv.app (yammdb.serv.app)... 45.130.21.225
Connecting to yammdb.serv.app (yammdb.serv.app)|45.130.21.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16172538 (15M) [application/octet-stream]
Saving to: ‘geo.mmdb’

geo.mmdb                100%[==============================>]  15.42M  71.8MB/s    in 0.2s    

2023-06-04 05:17:49 (71.8 MB/s) - ‘geo.mmdb’ saved [16172538/16172538]


real    0m0.382s
user    0m0.013s
sys     0m0.017s
notoles@fsn:~/neoon$ ls
geo.mmdb
notoles@fsn:~/neoon$ ls -al
total 15804
drwxr-xr-x 2 notoles notoles     4096 Jun  4 05:17 .
drwx--x--x 9 notoles notoles     4096 Jun  4 05:17 ..
-rw-r--r-- 1 notoles notoles 16172538 Jun  2 17:03 geo.mmdb
notoles@fsn:~/neoon$
    • Step 3: Grab the program with which to read the data.

    I cut and pasted the program from https://github.com/Ne00n/yammdb

    notoles@fsn:~/neoon$ ed read.py
read.py: No such file or directory
a
import geoip2.database

reader = geoip2.database.Reader("geo.mmdb")
response = reader.city("1.1.1.1")
print("Continent",response.continent.code)
print("Country",response.country.iso_code)
print("Latitude",response.location.latitude,"Longitude",response.location.longitude)
print(f"Latency {response.location.accuracy_radius}ms")
.
w
329
q
notoles@fsn:~/neoon$

    notoles@fsn:~/neoon$ sudo apt-get install python3-geoip2
[sudo] password for notoles: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  python3-maxminddb
Suggested packages:
  python-maxmindb-doc
The following NEW packages will be installed:
  python3-geoip2 python3-maxminddb
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 50.4 kB of archives.
After this operation, 203 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://deb.debian.org/debian unstable/main amd64 python3-maxminddb amd64 2.2.0-1+b1 [27.6 kB]
Get:2 http://deb.debian.org/debian unstable/main amd64 python3-geoip2 all 2.9.0+dfsg1-5 [22.8 kB]
Fetched 50.4 kB in 0s (1,435 kB/s)        
Selecting previously unselected package python3-maxminddb.
(Reading database ... 56808 files and directories currently installed.)
Preparing to unpack .../python3-maxminddb_2.2.0-1+b1_amd64.deb ...
Unpacking python3-maxminddb (2.2.0-1+b1) ...
Selecting previously unselected package python3-geoip2.
Preparing to unpack .../python3-geoip2_2.9.0+dfsg1-5_all.deb ...
Unpacking python3-geoip2 (2.9.0+dfsg1-5) ...
Setting up python3-maxminddb (2.2.0-1+b1) ...
Setting up python3-geoip2 (2.9.0+dfsg1-5) ...
notoles@fsn:~/neoon$ ls
geo.mmdb  read.py
notoles@fsn:~/neoon$
    • Step 5: Get an IP address to check
    notoles@fsn:~/neoon$ w
 20:22:21 up 3 days, 21:00,  1 user,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
notoles  pts/0    187.189.238.1    20:17    0.00s  0.03s   ?    w
notoles@fsn:~/neoon$
    • Step 6: Read the data for the IP address
    notoles@fsn:~/neoon$ time python3 ./read.py 187.189.238.1
Continent EU
Country CZ
Latitude 50.08 Longitude 14.41
Latency 0.0ms

real    0m0.026s
user    0m0.022s
sys     0m0.004s
notoles@fsn:~/neoon$
    • Step 7: Ponder

    The IP address that I am using seems to be shown as located in CZ. However, the address best might be located here where I am using it, visiting, in Sonora, Mexico. :) Probably I made some mistakes! Is everything reported as EU/CZ because I need to add additional, basic steps? Can someone please help me understand why my geolocation result seems possibly incorrect?

    Because you didn't changed the IP in the script.
    You probably want to replace the line 1.1.1.1 with
    response = reader.city(sys.argv[1])

    which takes your argument.
    Need to import sys though.

    Free NAT KVM | Free NAT LXC

  • Not_OlesNot_Oles Provider
    edited June 2023

    @Neoon said: Because you didn't changed the IP in the script.
    You probably want to replace the line 1.1.1.1 with
    response = reader.city(sys.argv[1])

    which takes your argument.

    Thanks so much! I will make that change and post the result.

    If you wish, here is another question:

    notoles@fsn:~/neoon$ wget https://yammdb.serv.app/geo.mmdb
--2023-06-04 21:56:05--  https://yammdb.serv.app/geo.mmdb
Resolving yammdb.serv.app (yammdb.serv.app)... 45.130.21.225
Connecting to yammdb.serv.app (yammdb.serv.app)|45.130.21.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16172538 (15M) [application/octet-stream]
Saving to: ‘geo.mmdb.1’

geo.mmdb.1              100%[==============================>]  15.42M  73.2MB/s    in 0.2s    

2023-06-04 21:56:05 (73.2 MB/s) - ‘geo.mmdb.1’ saved [16172538/16172538]

notoles@fsn:~/neoon$ # Cut and paste checksum from https://github.com/Ne00n/yammdb/blob/master/sha256sum + add filename
notoles@fsn:~/neoon$ ed SHA256SUM
SHA256SUM: No such file or directory
a
e5ba4b5e3fd4fd19fb3e056adbb956ad3640c14e72eb86d73aafdd39b2cf39b8 geo.mmdb
.
w
74
q
notoles@fsn:~/neoon$ sha256sum -c SHA256SUM 
geo.mmdb: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
notoles@fsn:~/neoon$ sha256sum geo.mmdb
daac3daeca83835400584b5ab81b2ec84657428bcd6290b7b392a6ad7e4e9922  geo.mmdb
notoles@fsn:~/neoon$

    What mistake am I making here? Have I wrongly identified the file whose checksum should match? Thanks!

  • NeoonNeoon OG

    @Not_Oles said:

    @Neoon said: Because you didn't changed the IP in the script.
    You probably want to replace the line 1.1.1.1 with
    response = reader.city(sys.argv[1])

    which takes your argument.

    Thanks so much! I will make that change and post the result.

    If you wish, here is another question:

    notoles@fsn:~/neoon$ wget https://yammdb.serv.app/geo.mmdb
--2023-06-04 21:56:05--  https://yammdb.serv.app/geo.mmdb
Resolving yammdb.serv.app (yammdb.serv.app)... 45.130.21.225
Connecting to yammdb.serv.app (yammdb.serv.app)|45.130.21.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16172538 (15M) [application/octet-stream]
Saving to: ‘geo.mmdb.1’

geo.mmdb.1              100%[==============================>]  15.42M  73.2MB/s    in 0.2s    

2023-06-04 21:56:05 (73.2 MB/s) - ‘geo.mmdb.1’ saved [16172538/16172538]

notoles@fsn:~/neoon$ # Cut and paste checksum from https://github.com/Ne00n/yammdb/blob/master/sha256sum + add filename
notoles@fsn:~/neoon$ ed SHA256SUM
SHA256SUM: No such file or directory
a
e5ba4b5e3fd4fd19fb3e056adbb956ad3640c14e72eb86d73aafdd39b2cf39b8 geo.mmdb
.
w
74
q
notoles@fsn:~/neoon$ sha256sum -c SHA256SUM 
geo.mmdb: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
notoles@fsn:~/neoon$ sha256sum geo.mmdb
daac3daeca83835400584b5ab81b2ec84657428bcd6290b7b392a6ad7e4e9922  geo.mmdb
notoles@fsn:~/neoon$

    What mistake am I making here? Have I wrongly identified the file whose checksum should match? Thanks!

    The checksum wasn't updated since the last build, probably a issue in my build script.

    Free NAT KVM | Free NAT LXC

  • Not_OlesNot_Oles Provider

    @Neoon said: The checksum wasn't updated since the last build, probably a issue in my build script.

    Thanks!

  • Not_OlesNot_Oles Provider

    Making the change that @Neoon suggested:

    notoles@fsn:~/neoon$ cat read.py
import geoip2.database

reader = geoip2.database.Reader("geo.mmdb")
response = reader.city("1.1.1.1")
print("Continent",response.continent.code)
print("Country",response.country.iso_code)
print("Latitude",response.location.latitude,"Longitude",response.location.longitude)
print(f"Latency {response.location.accuracy_radius}ms")
notoles@fsn:~/neoon$ ed read.py
329
/res
response = reader.city("1.1.1.1")
d
i
response = reader.city(sys.argv[1])
.
w
331
q
notoles@fsn:~/neoon$ cat read.py
import geoip2.database

reader = geoip2.database.Reader("geo.mmdb")
response = reader.city(sys.argv[1])
print("Continent",response.continent.code)
print("Country",response.country.iso_code)
print("Latitude",response.location.latitude,"Longitude",response.location.longitude)
print(f"Latency {response.location.accuracy_radius}ms")
notoles@fsn:~/neoon$ 

    notoles@fsn:~/neoon$ time python3 ./read.py 187.189.238.1
Traceback (most recent call last):
  File "/home/notoles/neoon/./read.py", line 4, in <module>
    response = reader.city(sys.argv[1])
                           ^^^
NameError: name 'sys' is not defined

real    0m0.026s
user    0m0.019s
sys     0m0.007s
notoles@fsn:~/neoon$

    Looks like we need to import the sys module. :)

    notoles@fsn:~/neoon$ ed read.py
331
2

i
import sys
.
w
342
q
notoles@fsn:~/neoon$ time python3 ./read.py 187.189.238.1
Continent NA
Country US
Latitude 32.77 Longitude -96.8
Latency 43.0ms

real    0m0.024s
user    0m0.024s
sys     0m0.000s
notoles@fsn:~/neoon$

    According to findlatitudeandlongitude.com the reported latitude and longitude is in Dallas, TX USA.

    The ISP here, as of the last time I checked awhile ago, seemed to use multiple hops with private addresses between my location and 187.189.238.1. Based on ping times and other factors, I previously imagined that 187.189.238.1 might be located in Missouri. Maybe it was in Dallas, or it is in Dallas now.

    Vendors seem universally to identify 187.189.238.1 as a Mexican IP. I can pretty much count on being offered default pages in Spanish and default prices in pesos.

    Thanks @Neoon! :)

  • somiksomik OG Hostbusters

    @Not_Oles said:
    Vendors seem universally to identify 187.189.238.1 as a Mexican IP. I can pretty much count on being offered default pages in Spanish and default prices in pesos.

    Yes, seems like the DB i am using also identifies your IP as mexico: https://ip2c.ziox.us/?ip=187.189.238.1

    The source code is located here: https://github.com/somik123/IP-to-Country

    I am currently using the "lite" database from ip2location.com

    I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.

  • NeoonNeoon OG

    I don't have a probe in Mexico, the closest is Texas, so its as close as it can get.

    Free NAT KVM | Free NAT LXC

  • someTomsomeTom

    @Neoon said:
    I don't have a probe in Mexico, the closest is Texas, so its as close as it can get.

    how do you calculate the positions? i assumed you are triangulating different "nearby" probes? eg in this case the results of Texas and Sao Paulo.

This discussion has been closed.