n0mad1k/CoM-geo-scout
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial build: geo-scout geo-targeted network scanner

Browse Source 
masscan port discovery → nmap fingerprinting → probe engine → pattern/version matching
Dynamic YAML inputs: countries.yaml (country codes + CIDR resolution) and targets.yaml (fingerprint probes)
RIR delegated stats for authoritative country CIDR data (no external API)
Supports tcp_banner, http, https, rtsp, udp probe types
Version extraction and comparison operators (<=, >=, ==, etc.)
SQLite results per engagement, CSV/JSON export
format-spec/ docs for generating input files externally
This commit is contained in:
n0mad1k
2026-04-30 10:38:20 -04:00
commit c20f189d49
18 changed files with 1221 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
format-spec/countries-format.md
+77
View File
@@ -0,0 +1,77 @@
# countries.yaml — Format Specification
## Purpose
Defines which countries to scan and scan parameters per country.
This file is read by geo-scout.py at runtime. The scanner resolves each
country code to its CIDR ranges using RIR delegated stats files.
## Schema
```yaml
scan_profile:
name: string # Label for this profile (used in logs/reports)
rate: integer # masscan packets/sec (default: 1000, max: 100000)
max_hosts_per_country: integer|null # Cap IPs per country. null = no limit
countries:
- code: string # ISO 3166-1 alpha-2 (e.g. US, VE, NL, GB, NG)
priority: high|medium|low # Scan order. high = first
exclude_cidrs: # Optional CIDR blocks to skip within this country
- "x.x.x.x/xx"
notes: string # Optional context (not used by scanner)
```
## Rules
- `code` must be a valid ISO 3166-1 alpha-2 code
- GB is the correct code for United Kingdom (not UK)
- `rate` applies globally to the masscan run, not per-country
- `exclude_cidrs` entries must be valid CIDR notation
- Countries are scanned in priority order: high → medium → low
- Within same priority, order in the list is preserved
## Valid priority values
`high` | `medium` | `low`
## Common country codes
| Country | Code |
|---------|------|
| United States | US |
| United Kingdom | GB |
| Venezuela | VE |
| Netherlands | NL |
| Canada | CA |
| Nigeria | NG |
| Russia | RU |
| Germany | DE |
| China | CN |
| Brazil | BR |
| Iran | IR |
| India | IN |
| France | FR |
| Australia | AU |
## Example
```yaml
scan_profile:
name: "latam-europe-sweep"
rate: 2000
max_hosts_per_country: 100000
countries:
- code: VE
priority: high
exclude_cidrs: []
notes: "Primary target"
- code: US
priority: medium
exclude_cidrs:
- "10.0.0.0/8"
- "172.16.0.0/12"
- "192.168.0.0/16"
- code: NL
priority: low
```
format-spec/targets-format.md
+174
View File
@@ -0,0 +1,174 @@
# targets.yaml — Format Specification
## Purpose
Defines what to look for during a scan. Each target is a fingerprint with
one or more probes. The scanner runs masscan to find open ports, then fires
each probe against matching hosts, and applies pattern/version matching.
## Schema
```yaml
targets:
- name: string # Human-readable label (appears in results)
tags: [string, ...] # Free-form tags for grouping/filtering results
ports: [integer, ...] # Ports masscan will scan for this target
probes:
- type: tcp_banner|http|https|rtsp|udp
# --- For http/https ---
path: string # URL path (e.g. "/", "/login.asp", "/api/version")
method: GET|POST # Default: GET
headers: # Optional extra request headers
Header-Name: value
body: string # POST body (optional)
match_in: body|headers|[body, headers] # Where to search for patterns
# --- For tcp_banner ---
# (no extra fields — reads the raw TCP banner on connect)
# --- For rtsp ---
# (sends OPTIONS * RTSP/1.0 and matches banner)
# --- Pattern matching (all probe types) ---
patterns: # At least one must match (OR logic)
- "regex or plain string"
all_patterns: # All must match (AND logic, optional)
- "regex or plain string"
# --- Version extraction (optional) ---
version_extract: "regex with one capture group"
version_compare: # Optional — filter by version
operator: "<="|">="|"=="|"!="|"<"|">"
value: "string or number"
# --- Confidence ---
confidence: high|medium|low # Default: medium
```
## Rules
- A target matches a host if ANY probe matches
- Within a probe, `patterns` uses OR logic (any one pattern is enough)
- `all_patterns` uses AND logic — use when you need multiple strings to co-occur
- `version_extract` must contain exactly one regex capture group `()`
- Version comparison is string-aware for dotted versions (e.g. "20.3" < "20.17")
- `match_in` defaults to `body` for http/https probes
- For `tcp_banner` type, the banner is the raw bytes received on connect
- Patterns are case-insensitive by default; prefix with `(?-i)` to force case-sensitive
- Tags are free-form strings — use them for filtering with `--tag` at runtime
## Probe types
| Type | Description |
|------|-------------|
| `tcp_banner` | Connect and read raw banner |
| `http` | HTTP GET/POST, match response |
| `https` | HTTPS GET/POST, match response (cert errors ignored) |
| `rtsp` | RTSP OPTIONS probe, match response |
| `udp` | Send empty UDP, match response |
## Common port reference
| Service | Ports |
|---------|-------|
| HTTP | 80, 8080, 8000, 8888 |
| HTTPS | 443, 8443 |
| SSH | 22 |
| Telnet | 23 |
| FTP | 21 |
| RTSP (cameras) | 554, 8554 |
| ONVIF (cameras) | 80, 8080 |
| DVR/NVR | 37777, 34567 |
| RDP | 3389 |
| SMB | 445 |
| Redis | 6379 |
| Elasticsearch | 9200 |
| MongoDB | 27017 |
| MySQL | 3306 |
| PostgreSQL | 5432 |
## Examples
### D-Link DIR-823X firmware 240126 or 240802
```yaml
- name: "D-Link DIR-823X fw 240126/240802"
tags: [router, d-link, cpe, iot]
ports: [80, 443, 8080]
probes:
- type: http
path: "/"
match_in: body
patterns: ["DIR-823X"]
all_patterns: []
- type: http
path: "/"
match_in: body
patterns: ["240126", "240802"]
confidence: high
```
### Exposed IP cameras (any brand)
```yaml
- name: "Exposed IP Camera"
tags: [camera, iot, surveillance]
ports: [80, 554, 8080, 8443, 37777, 34567]
probes:
- type: http
path: "/"
match_in: [body, headers]
patterns:
- "(?i)hikvision"
- "(?i)dahua"
- "(?i)ip camera"
- "(?i)ipcam"
- "(?i)webcam"
- "(?i)nvr"
- "(?i)dvr"
- "(?i)axis"
- "(?i)reolink"
- "(?i)amcrest"
- type: rtsp
patterns: ["RTSP/1.0 200"]
confidence: medium
```
### Cisco Catalyst SD-WAN Manager <= 20.17
```yaml
- name: "Cisco SD-WAN vManage <= 20.17"
tags: [cisco, sdwan, network, cve]
ports: [443, 8443]
probes:
- type: https
path: "/dataservice/client/server"
method: GET
match_in: body
patterns: ["vmanage", "platformVersion"]
version_extract: '"platformVersion":"([0-9.]+)"'
version_compare:
operator: "<="
value: "20.17"
confidence: high
- type: https
path: "/"
match_in: [body, headers]
patterns: ["vManage", "Cisco SD-WAN"]
confidence: low
```
### Ubuntu 24.04 SSH
```yaml
- name: "Ubuntu 24.04 SSH"
tags: [linux, ubuntu, ssh]
ports: [22]
probes:
- type: tcp_banner
patterns:
- "Ubuntu-24"
- "OpenSSH.*Ubuntu"
version_extract: "SSH-2.0-OpenSSH_([0-9p.]+)"
confidence: high
```
### Open Redis (unauthenticated)
```yaml
- name: "Open Redis"
tags: [database, redis, exposed]
ports: [6379]
probes:
- type: tcp_banner
patterns: ["redis_version"]
version_extract: "redis_version:([0-9.]+)"
confidence: high
```
geo-scout.py
+323
View File
@@ -0,0 +1,323 @@
#!/usr/bin/env python3
"""
geo-scout — geo-targeted network scanner
masscan for port discovery, nmap for fingerprinting, probe engine for matching
"""
import json
import os
import sys
import tempfile
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import yaml
BASE_DIR = Path(__file__).parent
sys.path.insert(0, str(BASE_DIR))
from lib.cidr import fetch_rir_data, load_index, get_cidrs, merge_cidrs
from lib.scanner import check_deps, masscan_sweep, nmap_fingerprint
from lib.prober import run_probe
from lib.matcher import match_target
from lib import db
INPUTS_DIR = BASE_DIR / "inputs"
DEFAULT_COUNTRIES = INPUTS_DIR / "countries.yaml"
DEFAULT_TARGETS = INPUTS_DIR / "targets.yaml"
# ── ANSI colors ───────────────────────────────────────────────────────────────
R = "\033[0;31m"
G = "\033[0;32m"
Y = "\033[0;33m"
C = "\033[0;36m"
B = "\033[1;34m"
W = "\033[0;37m"
BOLD = "\033[1m"
RST = "\033[0m"
def err(msg): print(f"{R}[!]{RST} {msg}", file=sys.stderr)
def ok(msg): print(f"{G}[+]{RST} {msg}")
def info(msg): print(f"{C}[*]{RST} {msg}")
def warn(msg): print(f"{Y}[-]{RST} {msg}")
def banner(): print(f"""{B}
__ _ ___ ___ ___ ___ ___ _ _ _
/ _` / _ \\/ _ \\ ___ / __|/ __/ _ \\| | | | |_
| (_| | __/ (_) |___\\__ \\ (_| (_) | |_| | _|
\\__, |\\___|\\___/ |___/\\___\\___/ \\__,_|\\__|
|___/
{RST}{W} geo-targeted network scanner v1.0{RST}
""")
# ── Loaders ───────────────────────────────────────────────────────────────────
def load_countries(path: Path) -> dict:
if not path.exists():
err(f"Countries file not found: {path}")
sys.exit(1)
with open(path) as f:
return yaml.safe_load(f)
def load_targets(path: Path) -> list[dict]:
if not path.exists():
err(f"Targets file not found: {path}")
sys.exit(1)
with open(path) as f:
data = yaml.safe_load(f)
return data.get("targets", [])
def collect_all_ports(targets: list[dict]) -> list[int]:
ports = set()
for t in targets:
ports.update(t.get("ports", []))
return sorted(ports)
# ── Probe worker ──────────────────────────────────────────────────────────────
def probe_host(ip: str, port: int, targets: list[dict]) -> list[dict]:
matches = []
for target in targets:
if port not in target.get("ports", []):
continue
probe_results = []
for probe in target.get("probes", []):
result = run_probe(ip, port, probe)
probe_results.append((probe, result))
hit_probes = match_target(ip, port, probe_results)
for hp in hit_probes:
matches.append({
"ip": ip,
"port": port,
"target_name": target["name"],
"tags": target.get("tags", []),
**hp,
})
return matches
# ── Commands ──────────────────────────────────────────────────────────────────
def cmd_run(args):
banner()
missing = check_deps()
if missing:
err(f"Missing required tools: {', '.join(missing)}")
err("Install with: sudo apt install " + " ".join(missing))
sys.exit(1)
countries_path = Path(args.countries) if args.countries else DEFAULT_COUNTRIES
targets_path = Path(args.targets) if args.targets else DEFAULT_TARGETS
country_data = load_countries(countries_path)
targets = load_targets(targets_path)
profile = country_data.get("scan_profile", {})
rate = profile.get("rate", 1000)
max_hosts = profile.get("max_hosts_per_country", None)
countries = sorted(
country_data.get("countries", []),
key=lambda c: {"high": 0, "medium": 1, "low": 2}.get(c.get("priority", "medium"), 1)
)
info(f"Engagement : {BOLD}{args.engagement}{RST}")
info(f"Countries : {', '.join(c['code'] for c in countries)}")
info(f"Targets : {len(targets)}")
info(f"Rate : {rate} pps")
print()
info("Loading country CIDR data...")
fetch_rir_data()
cidr_index = load_index()
conn = db.get_db(args.engagement)
run_id = db.start_run(
conn,
[c["code"] for c in countries],
[t["name"] for t in targets],
)
all_ports = collect_all_ports(targets)
info(f"Scanning ports: {', '.join(str(p) for p in all_ports)}")
print()
total_hosts = 0
total_matches = 0
workers = args.workers
with tempfile.TemporaryDirectory() as tmpdir:
tmp = Path(tmpdir)
for country in countries:
cc = country["code"].upper()
exclude = country.get("exclude_cidrs", [])
cidrs = get_cidrs(cc, cidr_index)
cidrs = merge_cidrs(cidrs, exclude)
if not cidrs:
warn(f"{cc}: no CIDR ranges found, skipping")
continue
if max_hosts:
cidrs = cidrs[:max_hosts // 256 + 1]
info(f"{BOLD}{cc}{RST}: {len(cidrs)} CIDR blocks → masscan...")
open_ports = masscan_sweep(cidrs, all_ports, rate, tmp)
hosts_found = len(set(e["ip"] for e in open_ports))
total_hosts += hosts_found
ok(f"{cc}: {hosts_found} live hosts ({len(open_ports)} open ports)")
if not open_ports:
continue
# Group by IP for nmap
ip_ports: dict[str, list[int]] = {}
for entry in open_ports:
ip_ports.setdefault(entry["ip"], []).append(entry["port"])
info(f"{cc}: fingerprinting {hosts_found} hosts with nmap...")
nmap_results: dict[str, dict] = {}
with ThreadPoolExecutor(max_workers=min(workers, 10)) as ex:
futs = {ex.submit(nmap_fingerprint, ip, ports, tmp): ip
for ip, ports in ip_ports.items()}
for fut in as_completed(futs):
ip = futs[fut]
try:
nmap_results[ip] = fut.result()
except Exception:
nmap_results[ip] = {}
info(f"{cc}: probing and matching targets...")
with ThreadPoolExecutor(max_workers=workers) as ex:
futs = []
for entry in open_ports:
futs.append(ex.submit(probe_host, entry["ip"], entry["port"], targets))
for fut in as_completed(futs):
try:
matches = fut.result()
except Exception:
continue
for match in matches:
ip = match["ip"]
port = match["port"]
nmap_info = nmap_results.get(ip, {}).get(port, {})
hit = {
**match,
"country_code": cc,
"nmap_service": nmap_info.get("service", ""),
"nmap_banner": nmap_info.get("banner", ""),
}
db.insert_hit(conn, run_id, hit)
total_matches += 1
ver = f" [{match['version']}]" if match.get("version") else ""
print(f" {G}HIT{RST} {BOLD}{ip}:{port}{RST} {Y}{match['target_name']}{RST}{ver} [{match['confidence']}] {cc}")
print()
db.finish_run(conn, run_id, total_hosts, total_matches)
conn.close()
print()
ok(f"Run complete. {total_hosts} hosts scanned, {G}{BOLD}{total_matches}{RST} matches.")
info(f"Results: results/{args.engagement}.db (run_id={run_id})")
def cmd_export(args):
conn = db.get_db(args.engagement)
fmt = args.format.lower()
if fmt == "json":
data = db.export_json(conn)
out = json.dumps(data, indent=2)
elif fmt == "csv":
lines = db.export_csv_lines(conn)
out = "\n".join(lines)
else:
err(f"Unknown format: {fmt}")
sys.exit(1)
if args.output:
Path(args.output).write_text(out)
ok(f"Exported {fmt.upper()} → {args.output}")
else:
print(out)
conn.close()
def cmd_update_cidr(args):
banner()
info("Force-refreshing RIR delegated data...")
fetch_rir_data(force=True)
from lib.cidr import load_index
info("Rebuilding CIDR index...")
index = load_index(force_rebuild=True)
country_count = len(index)
total_cidrs = sum(len(v) for v in index.values())
ok(f"Index built: {country_count} countries, {total_cidrs} CIDR blocks")
def cmd_list(args):
conn = db.get_db(args.engagement)
rows = conn.execute(
"SELECT target_name, country_code, COUNT(*) as c FROM hits GROUP BY target_name, country_code ORDER BY c DESC"
).fetchall()
if not rows:
warn(f"No hits in engagement: {args.engagement}")
return
print(f"\n{'Target':<40} {'Country':<8} {'Hits':>6}")
print("-" * 58)
for r in rows:
print(f" {r['target_name']:<38} {r['country_code']:<8} {r['c']:>6}")
conn.close()
# ── Entry point ───────────────────────────────────────────────────────────────
def main():
import argparse
parser = argparse.ArgumentParser(
prog="geo-scout",
description="Geo-targeted network scanner"
)
sub = parser.add_subparsers(dest="command", required=True)
# run
p_run = sub.add_parser("run", help="Run a scan")
p_run.add_argument("--engagement", "-e", required=True, help="Engagement name (used for DB filename)")
p_run.add_argument("--countries", "-c", default=None, help="Path to countries.yaml")
p_run.add_argument("--targets", "-t", default=None, help="Path to targets.yaml")
p_run.add_argument("--workers", "-w", type=int, default=20, help="Probe thread workers (default: 20)")
# export
p_exp = sub.add_parser("export", help="Export results")
p_exp.add_argument("--engagement", "-e", required=True)
p_exp.add_argument("--format", "-f", default="csv", choices=["csv", "json"])
p_exp.add_argument("--output", "-o", default=None, help="Output file (stdout if omitted)")
# update-cidr
sub.add_parser("update-cidr", help="Force refresh RIR CIDR data")
# list
p_list = sub.add_parser("list", help="Summarize hits for an engagement")
p_list.add_argument("--engagement", "-e", required=True)
args = parser.parse_args()
if args.command == "run":
cmd_run(args)
elif args.command == "export":
cmd_export(args)
elif args.command == "update-cidr":
cmd_update_cidr(args)
elif args.command == "list":
cmd_list(args)
if __name__ == "__main__":
main()
inputs/countries.yaml
+20
View File
@@ -0,0 +1,20 @@
scan_profile:
name: "example-sweep"
rate: 1000
max_hosts_per_country: null
countries:
- code: VE
priority: high
exclude_cidrs: []
- code: US
priority: medium
exclude_cidrs:
- "10.0.0.0/8"
- "172.16.0.0/12"
- "192.168.0.0/16"
- code: NL
priority: low
exclude_cidrs: []
inputs/targets.yaml
+72
View File
@@ -0,0 +1,72 @@
targets:
- name: "Ubuntu 24.04 SSH"
tags: [linux, ubuntu, ssh]
ports: [22]
probes:
- type: tcp_banner
patterns:
- "Ubuntu-24"
- "OpenSSH.*Ubuntu"
version_extract: "SSH-2\\.0-OpenSSH_([0-9p.]+)"
confidence: high
- name: "D-Link DIR-823X fw 240126/240802"
tags: [router, d-link, cpe, iot]
ports: [80, 443, 8080]
probes:
- type: http
path: "/"
match_in: body
patterns: ["DIR-823X"]
confidence: medium
- type: http
path: "/"
match_in: body
all_patterns: ["DIR-823X"]
patterns: ["240126", "240802"]
confidence: high
- name: "Exposed IP Camera"
tags: [camera, iot, surveillance]
ports: [80, 554, 8080, 8443, 37777, 34567]
probes:
- type: http
path: "/"
match_in: [body, headers]
patterns:
- "(?i)hikvision"
- "(?i)dahua"
- "(?i)ip.?camera"
- "(?i)ipcam"
- "(?i)webcam"
- "(?i)reolink"
- "(?i)amcrest"
- "(?i)axis"
confidence: medium
- type: rtsp
patterns: ["RTSP/1.0 200"]
confidence: medium
- name: "Cisco SD-WAN vManage <= 20.17"
tags: [cisco, sdwan, network]
ports: [443, 8443]
probes:
- type: https
path: "/dataservice/client/server"
method: GET
match_in: body
patterns: ["platformVersion"]
version_extract: '"platformVersion":"([0-9.]+)"'
version_compare:
operator: "<="
value: "20.17"
confidence: high
- name: "Open Redis"
tags: [database, redis, exposed]
ports: [6379]
probes:
- type: tcp_banner
patterns: ["redis_version"]
version_extract: "redis_version:([0-9.]+)"
confidence: high
lib/__init__.py
View File
lib/__pycache__/__init__.cpython-313.pyc
BIN
View File
Binary file not shown.
lib/__pycache__/cidr.cpython-313.pyc
BIN
View File
Binary file not shown.
lib/__pycache__/db.cpython-313.pyc
BIN
View File
Binary file not shown.
lib/__pycache__/matcher.cpython-313.pyc
BIN
View File
Binary file not shown.
lib/__pycache__/prober.cpython-313.pyc
BIN
View File
Binary file not shown.
lib/__pycache__/scanner.cpython-313.pyc
BIN
View File
Binary file not shown.
lib/cidr.py
+104
View File
@@ -0,0 +1,104 @@
import ipaddress
import json
import os
import time
import urllib.request
from pathlib import Path
CACHE_DIR = Path(__file__).parent.parent / "data" / "cidr-cache"
CACHE_TTL = 86400 * 7 # 7 days
RIR_URLS = [
"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-latest",
"https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest",
"https://ftp.apnic.net/pub/stats/apnic/delegated-apnic-latest",
"https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-latest",
"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-latest",
]
RIR_NAMES = ["ripe", "arin", "apnic", "lacnic", "afrinic"]
def _cache_path(rir: str) -> Path:
return CACHE_DIR / f"{rir}.txt"
def _index_path() -> Path:
return CACHE_DIR / "index.json"
def _is_stale(path: Path) -> bool:
if not path.exists():
return True
return time.time() - path.stat().st_mtime > CACHE_TTL
def fetch_rir_data(force: bool = False) -> None:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
for name, url in zip(RIR_NAMES, RIR_URLS):
path = _cache_path(name)
if not force and not _is_stale(path):
continue
print(f" Fetching {name}...", end=" ", flush=True)
try:
req = urllib.request.Request(url, headers={"User-Agent": "geo-scout/1.0"})
with urllib.request.urlopen(req, timeout=30) as resp:
path.write_bytes(resp.read())
print("done")
except Exception as e:
print(f"failed ({e})")
def _build_index() -> dict[str, list[str]]:
index: dict[str, list[str]] = {}
for name in RIR_NAMES:
path = _cache_path(name)
if not path.exists():
continue
for line in path.read_text(errors="replace").splitlines():
if line.startswith("#") or not line.strip():
continue
parts = line.split("|")
if len(parts) < 7:
continue
_, cc, record_type, start, value, _, status = parts[:7]
if record_type != "ipv4" or status not in ("allocated", "assigned"):
continue
if not cc or cc == "*":
continue
cc = cc.upper()
try:
count = int(value)
prefix_len = 32 - (count - 1).bit_length()
cidr = f"{start}/{prefix_len}"
ipaddress.ip_network(cidr, strict=False)
except (ValueError, TypeError):
continue
index.setdefault(cc, []).append(cidr)
return index
def load_index(force_rebuild: bool = False) -> dict[str, list[str]]:
idx_path = _index_path()
if not force_rebuild and idx_path.exists() and not _is_stale(idx_path):
return json.loads(idx_path.read_text())
index = _build_index()
idx_path.write_text(json.dumps(index))
return index
def get_cidrs(country_code: str, index: dict[str, list[str]]) -> list[str]:
return index.get(country_code.upper(), [])
def merge_cidrs(cidrs: list[str], exclude: list[str]) -> list[str]:
if not exclude:
return cidrs
exclude_nets = [ipaddress.ip_network(c, strict=False) for c in exclude]
result = []
for cidr in cidrs:
net = ipaddress.ip_network(cidr, strict=False)
excluded = any(net.overlaps(ex) for ex in exclude_nets)
if not excluded:
result.append(cidr)
return result
lib/db.py
+117
View File
@@ -0,0 +1,117 @@
import json
import sqlite3
from datetime import datetime
from pathlib import Path
RESULTS_DIR = Path(__file__).parent.parent / "results"
def get_db(engagement: str) -> sqlite3.Connection:
RESULTS_DIR.mkdir(parents=True, exist_ok=True)
db_path = RESULTS_DIR / f"{engagement}.db"
conn = sqlite3.connect(str(db_path))
conn.row_factory = sqlite3.Row
_init(conn)
return conn
def _init(conn: sqlite3.Connection) -> None:
conn.executescript("""
CREATE TABLE IF NOT EXISTS runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
started_at TEXT NOT NULL,
finished_at TEXT,
countries TEXT,
targets TEXT,
total_hosts INTEGER DEFAULT 0,
total_matches INTEGER DEFAULT 0
);
CREATE TABLE IF NOT EXISTS hits (
id INTEGER PRIMARY KEY AUTOINCREMENT,
run_id INTEGER NOT NULL,
ip TEXT NOT NULL,
port INTEGER NOT NULL,
country_code TEXT,
target_name TEXT NOT NULL,
tags TEXT,
probe_type TEXT,
confidence TEXT,
version TEXT,
nmap_service TEXT,
nmap_banner TEXT,
found_at TEXT NOT NULL,
FOREIGN KEY (run_id) REFERENCES runs(id)
);
CREATE INDEX IF NOT EXISTS idx_hits_ip ON hits(ip);
CREATE INDEX IF NOT EXISTS idx_hits_target ON hits(target_name);
CREATE INDEX IF NOT EXISTS idx_hits_country ON hits(country_code);
CREATE INDEX IF NOT EXISTS idx_hits_run ON hits(run_id);
""")
conn.commit()
def start_run(conn: sqlite3.Connection, countries: list, targets: list) -> int:
cur = conn.execute(
"INSERT INTO runs (started_at, countries, targets) VALUES (?, ?, ?)",
(datetime.utcnow().isoformat(), json.dumps(countries), json.dumps(targets)),
)
conn.commit()
return cur.lastrowid
def finish_run(conn: sqlite3.Connection, run_id: int, total_hosts: int, total_matches: int) -> None:
conn.execute(
"UPDATE runs SET finished_at=?, total_hosts=?, total_matches=? WHERE id=?",
(datetime.utcnow().isoformat(), total_hosts, total_matches, run_id),
)
conn.commit()
def insert_hit(conn: sqlite3.Connection, run_id: int, hit: dict) -> None:
conn.execute(
"""INSERT INTO hits
(run_id, ip, port, country_code, target_name, tags, probe_type,
confidence, version, nmap_service, nmap_banner, found_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?)""",
(
run_id,
hit["ip"],
hit["port"],
hit.get("country_code", ""),
hit["target_name"],
json.dumps(hit.get("tags", [])),
hit.get("probe_type", ""),
hit.get("confidence", ""),
hit.get("version", ""),
hit.get("nmap_service", ""),
hit.get("nmap_banner", ""),
datetime.utcnow().isoformat(),
),
)
conn.commit()
def export_json(conn: sqlite3.Connection, run_id: int | None = None) -> list[dict]:
if run_id:
rows = conn.execute("SELECT * FROM hits WHERE run_id=?", (run_id,)).fetchall()
else:
rows = conn.execute("SELECT * FROM hits").fetchall()
return [dict(r) for r in rows]
def export_csv_lines(conn: sqlite3.Connection, run_id: int | None = None) -> list[str]:
header = "ip,port,country_code,target_name,confidence,version,nmap_service,nmap_banner,found_at"
if run_id:
rows = conn.execute("SELECT * FROM hits WHERE run_id=?", (run_id,)).fetchall()
else:
rows = conn.execute("SELECT * FROM hits ORDER BY found_at").fetchall()
lines = [header]
for r in rows:
lines.append(",".join([
str(r["ip"]), str(r["port"]), r["country_code"] or "",
r["target_name"], r["confidence"] or "", r["version"] or "",
r["nmap_service"] or "", r["nmap_banner"] or "", r["found_at"],
]))
return lines
lib/matcher.py
+90
View File
@@ -0,0 +1,90 @@
import re
from typing import Optional
from .prober import ProbeResult
def _version_compare(extracted: str, operator: str, threshold: str) -> bool:
def to_tuple(v: str):
try:
return tuple(int(x) for x in v.split("."))
except ValueError:
return (v,)
ev = to_tuple(extracted)
tv = to_tuple(threshold)
if operator == "<=":
return ev <= tv
elif operator == ">=":
return ev >= tv
elif operator == "<":
return ev < tv
elif operator == ">":
return ev > tv
elif operator == "==":
return ev == tv
elif operator == "!=":
return ev != tv
return False
def _get_search_text(result: ProbeResult, match_in) -> str:
if isinstance(match_in, list):
parts = []
if "body" in match_in:
parts.append(result.body or result.banner)
if "headers" in match_in:
parts.append(" ".join(f"{k}: {v}" for k, v in result.headers.items()))
return "\n".join(parts)
if match_in == "headers":
return " ".join(f"{k}: {v}" for k, v in result.headers.items())
return result.body or result.banner
def match_probe(result: ProbeResult, probe: dict) -> tuple[bool, Optional[str]]:
if not result.success:
return False, None
match_in = probe.get("match_in", "body")
text = _get_search_text(result, match_in)
patterns = probe.get("patterns", [])
all_patterns = probe.get("all_patterns", [])
if patterns:
if not any(re.search(p, text) for p in patterns):
return False, None
if all_patterns:
if not all(re.search(p, text) for p in all_patterns):
return False, None
extracted_version = None
version_extract = probe.get("version_extract")
if version_extract:
m = re.search(version_extract, text)
if m:
extracted_version = m.group(1)
version_compare = probe.get("version_compare")
if version_compare and extracted_version:
op = version_compare.get("operator", "==")
val = str(version_compare.get("value", ""))
if not _version_compare(extracted_version, op, val):
return False, None
elif version_compare and not extracted_version:
return False, None
return True, extracted_version
def match_target(host: str, port: int, probe_results: list[tuple[dict, ProbeResult]]) -> list[dict]:
matches = []
for probe, result in probe_results:
matched, version = match_probe(result, probe)
if matched:
matches.append({
"probe_type": result.probe_type,
"confidence": probe.get("confidence", "medium"),
"version": version,
})
return matches
lib/prober.py
+125
View File
@@ -0,0 +1,125 @@
import socket
import ssl
import urllib.request
import urllib.error
from dataclasses import dataclass, field
CONNECT_TIMEOUT = 5
READ_TIMEOUT = 8
@dataclass
class ProbeResult:
probe_type: str
success: bool
banner: str = ""
body: str = ""
headers: dict = field(default_factory=dict)
error: str = ""
def _tcp_banner(host: str, port: int) -> ProbeResult:
try:
sock = socket.create_connection((host, port), timeout=CONNECT_TIMEOUT)
sock.settimeout(READ_TIMEOUT)
try:
data = sock.recv(4096)
return ProbeResult("tcp_banner", True, banner=data.decode("utf-8", errors="replace"))
except socket.timeout:
return ProbeResult("tcp_banner", True, banner="")
finally:
sock.close()
except Exception as e:
return ProbeResult("tcp_banner", False, error=str(e))
def _http_probe(host: str, port: int, probe: dict, use_ssl: bool) -> ProbeResult:
scheme = "https" if use_ssl else "http"
path = probe.get("path", "/")
method = probe.get("method", "GET").upper()
extra_headers = probe.get("headers", {})
body_data = probe.get("body", "")
url = f"{scheme}://{host}:{port}{path}"
data = body_data.encode() if body_data else None
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
req = urllib.request.Request(url, data=data, method=method)
req.add_header("User-Agent", "Mozilla/5.0")
req.add_header("Connection", "close")
for k, v in extra_headers.items():
req.add_header(k, v)
try:
handler = urllib.request.HTTPSHandler(context=ctx) if use_ssl else urllib.request.HTTPHandler()
opener = urllib.request.build_opener(handler)
opener.addheaders = []
with opener.open(req, timeout=READ_TIMEOUT) as resp:
raw = resp.read(65536)
body = raw.decode("utf-8", errors="replace")
headers = dict(resp.headers)
return ProbeResult(
"https" if use_ssl else "http",
True,
body=body,
headers=headers,
)
except urllib.error.HTTPError as e:
try:
body = e.read(16384).decode("utf-8", errors="replace")
except Exception:
body = ""
return ProbeResult("https" if use_ssl else "http", True, body=body, headers=dict(e.headers))
except Exception as e:
return ProbeResult("https" if use_ssl else "http", False, error=str(e))
def _rtsp_probe(host: str, port: int) -> ProbeResult:
try:
sock = socket.create_connection((host, port), timeout=CONNECT_TIMEOUT)
sock.settimeout(READ_TIMEOUT)
request = f"OPTIONS * RTSP/1.0\r\nCSeq: 1\r\nUser-Agent: geo-scout\r\n\r\n"
sock.sendall(request.encode())
try:
data = sock.recv(4096)
return ProbeResult("rtsp", True, banner=data.decode("utf-8", errors="replace"))
except socket.timeout:
return ProbeResult("rtsp", False, error="timeout")
finally:
sock.close()
except Exception as e:
return ProbeResult("rtsp", False, error=str(e))
def _udp_probe(host: str, port: int) -> ProbeResult:
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.settimeout(CONNECT_TIMEOUT)
sock.sendto(b"", (host, port))
try:
data, _ = sock.recvfrom(4096)
return ProbeResult("udp", True, banner=data.decode("utf-8", errors="replace"))
except socket.timeout:
return ProbeResult("udp", False, error="timeout")
finally:
sock.close()
except Exception as e:
return ProbeResult("udp", False, error=str(e))
def run_probe(host: str, port: int, probe: dict) -> ProbeResult:
ptype = probe.get("type", "tcp_banner")
if ptype == "tcp_banner":
return _tcp_banner(host, port)
elif ptype == "http":
return _http_probe(host, port, probe, use_ssl=False)
elif ptype == "https":
return _http_probe(host, port, probe, use_ssl=True)
elif ptype == "rtsp":
return _rtsp_probe(host, port)
elif ptype == "udp":
return _udp_probe(host, port)
return ProbeResult(ptype, False, error=f"unknown probe type: {ptype}")
lib/scanner.py
+118
View File
@@ -0,0 +1,118 @@
import json
import shutil
import subprocess
import tempfile
import xml.etree.ElementTree as ET
from pathlib import Path
def check_deps() -> list[str]:
missing = []
for tool in ("masscan", "nmap"):
if not shutil.which(tool):
missing.append(tool)
return missing
def masscan_sweep(cidrs: list[str], ports: list[int], rate: int, output_dir: Path) -> list[dict]:
if not cidrs or not ports:
return []
port_str = ",".join(str(p) for p in sorted(set(ports)))
cidr_file = output_dir / "targets.txt"
cidr_file.write_text("\n".join(cidrs))
out_file = output_dir / "masscan.json"
cmd = [
"masscan",
f"--rate={rate}",
f"--ports={port_str}",
"-iL", str(cidr_file),
"-oJ", str(out_file),
"--output-format", "json",
"--wait", "3",
]
try:
subprocess.run(cmd, capture_output=True, timeout=3600)
except subprocess.TimeoutExpired:
pass
except FileNotFoundError:
raise RuntimeError("masscan not found — install it first")
if not out_file.exists():
return []
raw = out_file.read_text(errors="replace").strip()
if not raw or raw == "[]":
return []
raw = raw.rstrip(",\n")
if not raw.endswith("]"):
raw += "]"
if not raw.startswith("["):
raw = "[" + raw
try:
data = json.loads(raw)
except json.JSONDecodeError:
return []
results = []
for entry in data:
ip = entry.get("ip")
for port_entry in entry.get("ports", []):
port = port_entry.get("port")
if ip and port:
results.append({"ip": ip, "port": port})
return results
def nmap_fingerprint(host: str, ports: list[int], output_dir: Path) -> dict:
if not ports:
return {}
port_str = ",".join(str(p) for p in sorted(set(ports)))
out_file = output_dir / f"nmap_{host.replace('.', '_')}.xml"
cmd = [
"nmap",
"-sV", "--version-intensity", "5",
"-p", port_str,
"-T4",
"--open",
"-oX", str(out_file),
host,
]
try:
subprocess.run(cmd, capture_output=True, timeout=60)
except (subprocess.TimeoutExpired, FileNotFoundError):
return {}
if not out_file.exists():
return {}
try:
tree = ET.parse(out_file)
except ET.ParseError:
return {}
result = {}
for port_el in tree.findall(".//port"):
portid = int(port_el.get("portid", 0))
state = port_el.find("state")
if state is None or state.get("state") != "open":
continue
service = port_el.find("service")
info = {}
if service is not None:
info["service"] = service.get("name", "")
info["product"] = service.get("product", "")
info["version"] = service.get("version", "")
info["extrainfo"] = service.get("extrainfo", "")
info["banner"] = " ".join(filter(None, [
info["product"], info["version"], info["extrainfo"]
]))
result[portid] = info
return result
requirements.txt
+1
View File
@@ -0,0 +1 @@
pyyaml>=6.0