2025 Tap And Map

 Tap And Map is back with new and improved installation instructions!  

Here is an old video showing what TapAndMap is, for those who may not know:

https://www.youtube.com/watch?v=gRfQzBjrhEE

you'll notice that it's been a while since I've touched Tap And Map.   Because it's been so long since I built this, the old installation instructions would NEVER work to build it today (Feb 2025).   Plus, there was a Google Maps API bug that has troubled me all this time, as Google Maps would decide not to reload the page and I'd have to clear my history (lame).  Because my old code was built as an all-in-one method, so packets were sniffed, looked up, displayed, and logged, all in one large monolithic module, there was no easy means to troubleshoot.   

So I've now decided to Sniff packets, lookup their IP information, and store the data in one script, and read that file, parse and publish html files in a separate module.  That allowed me to better separate and find errors.  For examples, when MaxMind could not find a lat/long or city, and then the whole process of plotting the packet on a map was out the window.  Anyway, the new TapAndMap separates separate functions to help troubleshoot and just make it cleaner.  

These instructions should work on any Raspberry Pi, and I've got it running on a couple.  Before you start the installation process, be aware that TapAndMap requires two external dependencies (free).  Those are highlighted below.  

- The MaxMind GeoIP database requires registration to download their City-Level database and get a license key.
- GoogleMaps requires that you register with them to get your own GoogleMaps key.

I'll walk you through how to get those below.  If you like, you can skip ahead and get your keys before you proceed, if you're worried about this step.  

Here are the new Installation instructions.  I've put things you need to do in red:

The files that I made that you'll need are at:

https://github.com/johnbebo/TapAndMap

Put those in a safe place for later.

- Download "Raspberry Pi OS with Desktop" from https://www.raspberrypi.com/software/operating-systems/.  This was the 2024-11-19-raspios-bookworm-armhf.img.xz image as of Feb 2025.  

- I extracted it with 'xz -d 2024-11-19-raspios-bookworm-armhf.img.xz ' and then copied it to a micro SD card with dd (dd if=2024-11-19-raspios-bookworm-armhf.img of=/dev/sdx) where x is your SD card

- I booted it up and at the Welcome screen I selected Next, Country, Language, and Timezone. 
- I gave it a userid and password
- I set the default browser as chromium (I'll never use it anyway)

The desktop wallpaper was cool

- Since I have one internal Ethernet, and one USB-Dongle Ethernet, I set up the internal with an IP address (Eth0/Wired Connection1) in the GUI, and disabled the other (eth1/Wired Connection 2) to configure via the terminal later.  Disabling it avoids any NIC manager confusion of having them both plugged in.  I shut off the wireless on my Pi that has it.  Another way to set this up vs sniffing on one USB dongle, is to have TWO USB dongles, and bridge them, and sniff the bridge.  That allows you to sniff in the middle of a specific computer or something instead of needing a spanning port, tap, or hub.  I'll explain more in the Ethernet section later on how to set this up.  But for these instructions, as this is how I use it, I have a managed network switch with a spanning port that I can listen to.  This could be an expensive Cisco switch, or a small, TP-Link TL-SG108E switch for like $30.  Both work as well as far as I can tell. 

I had to reboot for the Ethernet settings to take effect.  

- apt install tcpdump screen vim

- To run ssh at bootup, I Selected the Raspberry from the Upper Left Raspberry Icon -> Preferences -> Raspberry Pi Configuration -> Interfaces Tab -> turn on SSH

to test this, reboot and do a 'netstat -pant' and you should see port 22 open. 

- While I had the GUI up, I installed updates via the GUI (it was a down arrow in the upper right).

You can now follow the rest of these instructions on the pi, or through an ssh tunnel.  From here on out you won't need a keyboard, mouse, or monitor.  The Pi is designed to run headless.  

- apt install python3-pcapy scapy  # Needed to sniff packets

--------------- Configure GeoIP ---------------------------
-  #apt-get install geoip-bin geoip-database
   -- #Unfortunately, this will only give you COUNTRY level geolocation.  
   -- BUT, if you 

go to https://dev.maxmind.com/geoip/geolite2-free-geolocation-data/ 

you can create a free account and download a free CITY level GeoLite database. 

Here's that site:

  To get your database:

          --- Click on "Sign up for GeoLite."  This asked you for your name, email and to set up an account, which they verify by emailing you, and then they verify again when you change your password by emailing you a code.  But after that you can download the database.  
          --- I downloaded the 'GeoLite2 City' in both binary and csv.
          --- When I untar'd GeoLite2-City_20250207.tar, there as a
 GeoLite2-City.mmdb.
          ---I put this in a directory called:

/home/pi/TapAndMap/GeoDB/GeoLite2-City_20250207/ GeoLite2-City.mmdb

We will use this file later to test out IP lookup.    

I then did:
---- sudo apt install python3-geoip2

        ---- that worked, but I seemed to be missing geoipupdate.  When I tried: sudo apt install geoipupdate, it was not found.  
        ---- To get around this, I tried to change my /etc/apt/sources.list, but could not get that to work, as the sources are untrusted. 
So I downloaded it by going to: https://packages.debian.org/buster/armhf/geoipupdate/download
and clicking on a link (I had to do this in Firefox, as Google blocked ftp).  

This downloaded the package geoipupdate_3.1.1-1_armhf.deb

This file is in my github folder if you want to get it from there.  

I then did a 'dpkg -i geoipupdate_3.1.1-1_armhf.deb' to install it.

      ----'geoipupdate -v' worked, but gave me a bunch of license errors.  Specifically, it said:

Your account ID or license key is invalid

   ----- I went back to the maxmind site, maxmind.com, and looked down the left to "manage License Keys" 

and there I saw a "Generate New License".  I clicked on it, and it made me a key.  I copied it via their clipboard and downloaded the GeoIP.conf file.  This has the key you'll need.  

then put this file in the /etc/path (replacing the old /etc/GeoIP.conf file).  

Now, when I did a 'geoupdate -v' it then configured the geoip for me.   

to test it, I ran the following python script:

#!/usr/bin/python3
import geoip2.database
# Replace with your database path
db = geoip2.database.Reader('path/to/GeoLite2-City.mmdb')
def get_location_by_ip(ip_address): 
    try:
        response = db.city(ip_address)
        latitude = response.location.latitude
        longitude = response.location.longitude
        return latitude, longitude
    except Exception as e:
        print(f"Error looking up IP: {e}")
        return None, None

# Example usage
ip = "8.8.8.8"
lat, lon = get_location_by_ip(ip)
print(f"Latitude: {lat}, Longitude: {lon}") 

This spit out a lat and long for 8.8.8.8!   Yay.  

I also needed to spit out the city and country, so with the help of ChatGPT, I came up with this script:

#!/usr/bin/python3
import geoip2.database
DATABASE_PATH = '/your/path/to/GeoLite2-City.mmdb'
def get_city(ip_address):
   # open with database
   with geoip2.database.Reader(DATABASE_PATH) as reader:
       try:
           response = reader.city(ip_address)
           city = response.city.name
           country = response.country.name
           lat, lon = response.location.latitude, response.location.longitude
           print(f"IP Address:  {ip_address}")
           print(f"City: {city}")
           print(f"Country:  {country}")
           print(f"Latitude:  {lat}  Longitude:  {lon}")
       except geoip2.errors.AddressNotFoundError:
           print(f"No Location Found for IP:  {ip_address}")

# Example Usage
get_city("4.4.4.4")
root@raspberrypi:/home/pi/TapAndMap# ./test_city.py 
IP Address:  4.4.4.4
City: Omaha
Country:  United States
Latitude:  41.263  Longitude:  -95.9623

--------  Device GeoIP Capable. Check ------------

That is a good start!   Now that you can look up IP addresses, time to install a webserver to host the data.

------- Install Apache Web Server -----------------

- apt install apache2

This will install a webserver.  'service apache2 stop', and 'service apache2 start' will stop and start it, but it should always be running now that you've installed it.  

/var/www/html/ is where apache2 will look for files.  

When you point a browser to your IP you should now see the default Apache2 web page

We will make a, index.html webpage later.  But for now, make these two directories you'll need later:

mkdir /var/www/html/logs

mkdir /var/www/html/maps

-------------  Configure NIC to sniff ------------------

Next was to make the eth1 (USB Dongle) a promiscuous interface so I can sniff it. If you're using two USB dongles as a bridge, sniff the bridge instead.  I tried 

adding 'sudo ifconfig eth1 promisc up' to a /etc/rc.local file that I made, as on bootup the device is supposed to load /etc/rc.local.  I then made that file executable with 'chmod +x /etc/rc.local'

This worked when I ran it manually, but upon reboot it seems that this file gets run before the NIC comes up, so it does nothing.  To get around this, I edited the crontab to start the /etc/rc.local file 5 seconds after bootup.   To edit the crontab, I ran 

'crontab -e' 

and selected nano as my editor, and then put this at the last line:

@reboot sleep 5; /etc/rc.local

I rebooted to test that ran /etc/rc.local 5 seconds after booting up, viola, my NIC is now in promiscuous mode when I ssh'd in.  This is a semi-ghetto way of configuring the NIC, but since it takes me more than 5 seconds to ssh in and do stuff I called it good.

To make sure this works, after reboot, I ran 

ifconfig eth1

and I saw this:

eth1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500

        ether 00:50:b6:1b:81:72  txqueuelen 1000  (Ethernet)

        RX packets 150595  bytes 30212103 (28.8 MiB)

        RX errors 0  dropped 15357  overruns 0  frame 0

        TX packets 22  bytes 1240 (1.2 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Bridging instructions:

NOTE:   If you want to sniff THROUGH the device, as in you have two USB dongles and you want them bridged so that traffic goes in one side and out the other because you have no Spanning Port or Network Tap or such to sniff on one interface, you'll need to make a bridge.  That should be done something like this:

 ifconfig eth1 up
 ifconfig eth2 up
 brctl addbr br0
 brctl addif br0 eth1 eth2
 ifconfig br0 up

I HAVE NOT TRIED THIS in a long time, as I have a spanning port.  But the last time I tried it, it worked, so I'll post it as a starting, perhaps working, method.  

Anyway, to test that you are able to sniff, after reboot, run this:

tcpdump -i eth1

and you should see packets flowing

-----  Device Packet Sniffing Capable.  Check.  ---------

Next was to build a sniffer to sniff the NIC, ignore all private only IP addresses, and use the MaxMind database we downloaded to geolocate the public IPs, and store those as a file.  This wasn't too hard, as shown below, with some randomly generated UDP packets.  This is done with a script I called 

Send_A_Random_UDP_Packet.py, which is in the github folder.  

First, copy the TapAndMap.conf file from github to your /var/www/ directory.  Change the variables (like your home location) as you need to.  

Then copy the below file from github and run it, as shown.  You should see IPs and locations.

Sniff_Dev_Save_CSV.py

This file stores packet count, time, src IP/port, dst IP/port, protocol, lat/long, city, country, and zip.  For reference if you need it, the sniff'd data is stored in /tmp/tempfile.  This is intentional, so that in case you leave your device for long periods of time and it fills up the hard drive, a reboot will clean you up, since /tmp/ gets wiped.  

----- Device can Sniff, Lookup, and Store.  Check. ------

Next was to build some python that would read in the stored file above and  plot the points on a Google Maps map.  This turned out to be the hardest part, as even with a clean file, Google Maps seemed to continually want to hang, even on a static web page that reads in points and plots them, requiring me to clear the browser history or go to a different browser or browser profile to have the cache not hang me.  

To troubleshoot, and because I had thousands of packets, and because plotting the same packets over and over was silly, and because I thought that perhaps Google didn't want me to plot so many markers, I elected to write a loop to only plot UNIQUE packets.  Meaning, if the same lat/long was seen more than once, it would only plot the first one.  This helped troubleshoot, and seemed like the way to go for the final product.  I ended up with my main page working.  Then I built the rightmost pane to keep track of connections.   Then I stored maps and logs every hour in separate folders for analysis, if required.   The resulting script I called:

./Read_CSV_Write_Html.py

If you run this script (in my github folder), and don't have a proper GoogleMapsAPI Key, as described below, you'll see this:

YOUR Google Key.

If you download my files and try them and get a dark, unhelpful GoogleMap, that is because you need to get your own GoogleAPI key.   To do that, visit:

https://cloud.google.com/maps-platform/#get-started

and you'll be welcomed wtih:

Once you click get started, it will walk you through generating a key.  

It will probably ask you to create a project, Enable a GoogleMaps Platform, and create a billing account (I've never been billed, as it takes massive queries to be billed).

After you put in your credit card (I've never been charged), you'll get an EnableAppsAPI screen -> Next

Here's how my key was found:

Put that key in

<script src="https://maps.googleapis.com/maps/api/js?libraries=geometry&key=YOUR_KEY"></script>

Add that key in the TWO places in the Read_CSV_Write_Html.py file.  One is for the main page, the other is for the hourly stored maps.  

Once you've copied that key, TapAndMap should work.   To run TapAndMap, I recommend this after it boots up:

1) ssh in as pi, and sudo su up to root.
2) cd /var/www
3) screen -S Sniffer    #This will start a sniffer that you can detach from and leave running
4) ./Sniff_Dev_Save_CSV.py
5) CTRL-A, D           #This will detach from the Sniffer screen.  To resume that screen, type 'screen -r'
6) ./Read_CSV_Write_Html.py 

And it the final result is this:

 

Whenever you want to refresh the screen, just run the Read_CSV_Write_Html.py script.  I could put this on a "while true, sleep 20' loop, but I will leave that to you.  I like manually running it and watching it complete!  There is something satisfying about that.   

If you like what you see, and you're feeling generous, feel free to donate a few fractions of a bitcoin to: 1Pq1pwxauSj1zQQLLQA4Suc9azVfFYU2dX

(That QR code is below in case you want to take a pic with your phone)