Initial commit: Phase 1 & Phase 2 infrastructure complete
This commit is contained in:
@@ -0,0 +1,801 @@
|
||||
# GigLez Deployment Considerations: Development vs Production
|
||||
|
||||
**Created**: 2026-01-12
|
||||
**Purpose**: Identify critical differences between local development and production deployment
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
GigLez needs to support two distinct deployment scenarios:
|
||||
|
||||
1. **Local Development** (Termux on Android)
|
||||
- Resource-constrained (mobile device)
|
||||
- Single-user access
|
||||
- Direct USB serial communication
|
||||
- Local file storage
|
||||
- Development/testing workflow
|
||||
|
||||
2. **Production Server** (Cloud/VPS)
|
||||
- Multi-user platform
|
||||
- Public API access
|
||||
- No hardware connection
|
||||
- Object storage for files
|
||||
- High availability requirements
|
||||
|
||||
**Key Insight**: We need a **modular architecture** that supports both scenarios without code duplication.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Strategy
|
||||
|
||||
### Hybrid Architecture: "Capture" vs "Platform" Mode
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ GigLez Application │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌────────────────┐ ┌────────────────┐ │
|
||||
│ │ Capture Mode │ │ Platform Mode │ │
|
||||
│ │ (Termux) │ │ (Server) │ │
|
||||
│ ├────────────────┤ ├────────────────┤ │
|
||||
│ │ - T-Embed USB │ │ - No hardware │ │
|
||||
│ │ - Local GPS │ │ - File uploads │ │
|
||||
│ │ - Auto-capture │ │ - Multi-user │ │
|
||||
│ │ - Local API │ │ - Public API │ │
|
||||
│ └────────────────┘ └────────────────┘ │
|
||||
│ │ │ │
|
||||
│ └──────────┬───────────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────▼────────┐ │
|
||||
│ │ Shared Core │ │
|
||||
│ ├────────────────┤ │
|
||||
│ │ - Database │ │
|
||||
│ │ - Parser │ │
|
||||
│ │ - Matcher │ │
|
||||
│ │ - GPS Validator│ │
|
||||
│ │ - API Endpoints│ │
|
||||
│ └────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Detailed Comparison
|
||||
|
||||
### 1. Hardware Access
|
||||
|
||||
#### Development (Termux)
|
||||
- **T-Embed Connection**: USB serial communication (pyserial)
|
||||
- **GPS Source**: Android native GPS via SL4A or Termux:API
|
||||
- **Real-time Capture**: Direct signal scanning and capture
|
||||
- **Storage**: Local filesystem on Android device
|
||||
|
||||
**Implications**:
|
||||
- Need hardware abstraction layer
|
||||
- Support offline operation
|
||||
- Handle USB disconnections
|
||||
- Batch upload when network available
|
||||
|
||||
#### Production (Server)
|
||||
- **No Hardware**: Accept pre-captured .sub files only
|
||||
- **GPS Source**: Embedded in upload manifest
|
||||
- **No Real-time**: Historical data only
|
||||
- **Storage**: Object storage (S3, MinIO, or filesystem)
|
||||
|
||||
**Implications**:
|
||||
- No serial communication code needed
|
||||
- Network-dependent operation
|
||||
- Focus on file processing pipeline
|
||||
- Horizontal scaling possible
|
||||
|
||||
**Solution**:
|
||||
```python
|
||||
# Abstract hardware interface
|
||||
class CaptureSource(ABC):
|
||||
@abstractmethod
|
||||
def scan_signals(self): pass
|
||||
|
||||
@abstractmethod
|
||||
def get_gps(self): pass
|
||||
|
||||
class TEmbedSource(CaptureSource):
|
||||
"""Development: Direct hardware"""
|
||||
pass
|
||||
|
||||
class UploadSource(CaptureSource):
|
||||
"""Production: File uploads"""
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Database Configuration
|
||||
|
||||
#### Development (Termux)
|
||||
- **PostgreSQL**: Local installation via `pkg install postgresql`
|
||||
- **Host**: localhost or 127.0.0.1
|
||||
- **Port**: 5432
|
||||
- **Connections**: Small pool (5 + 5 overflow)
|
||||
- **Storage**: Device internal storage or SD card
|
||||
|
||||
**Considerations**:
|
||||
- Limited RAM on mobile devices
|
||||
- Smaller connection pool
|
||||
- May need to tune PostgreSQL for mobile
|
||||
- Backup to SD card or cloud
|
||||
|
||||
#### Production (Server)
|
||||
- **PostgreSQL**: Dedicated server or managed database (RDS, Cloud SQL)
|
||||
- **Host**: Remote database server
|
||||
- **Port**: 5432 (or custom)
|
||||
- **Connections**: Larger pool (20 + 40 overflow)
|
||||
- **Storage**: SSD/NVMe for performance
|
||||
|
||||
**Considerations**:
|
||||
- Connection pooling via PgBouncer
|
||||
- Read replicas for scaling
|
||||
- Automated backups
|
||||
- Point-in-time recovery
|
||||
|
||||
**Solution**: Environment-based configuration
|
||||
```bash
|
||||
# Development (.env)
|
||||
GIGLEZ_DB_HOST=localhost
|
||||
GIGLEZ_DB_POOL_SIZE=5
|
||||
GIGLEZ_DB_MAX_OVERFLOW=5
|
||||
|
||||
# Production (.env.production)
|
||||
GIGLEZ_DB_HOST=db.giglez.com
|
||||
GIGLEZ_DB_POOL_SIZE=20
|
||||
GIGLEZ_DB_MAX_OVERFLOW=40
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. File Storage
|
||||
|
||||
#### Development (Termux)
|
||||
- **Location**: `/data/data/com.termux/files/home/giglez/storage/`
|
||||
- **Structure**: Organized by session/date
|
||||
- **Size**: Limited by device storage (typically < 64GB)
|
||||
- **Access**: Direct filesystem access
|
||||
|
||||
**Structure**:
|
||||
```
|
||||
storage/
|
||||
├── captures/
|
||||
│ ├── 2026-01-12/
|
||||
│ │ ├── session_abc123/
|
||||
│ │ │ ├── capture_001.sub
|
||||
│ │ │ ├── capture_002.sub
|
||||
│ │ │ └── manifest.json
|
||||
```
|
||||
|
||||
#### Production (Server)
|
||||
- **Location**: Object storage (S3, MinIO, GCS) or networked filesystem
|
||||
- **Structure**: Content-addressed by file hash
|
||||
- **Size**: Unlimited (cloud storage)
|
||||
- **Access**: Via storage API or CDN
|
||||
|
||||
**Structure**:
|
||||
```
|
||||
s3://giglez-captures/
|
||||
├── ab/
|
||||
│ ├── c1/
|
||||
│ │ └── abc123...def456.sub # First 2+2 chars of hash for sharding
|
||||
```
|
||||
|
||||
**Solution**: Storage abstraction
|
||||
```python
|
||||
class StorageBackend(ABC):
|
||||
@abstractmethod
|
||||
def save_file(self, file_hash: str, content: bytes): pass
|
||||
|
||||
@abstractmethod
|
||||
def get_file(self, file_hash: str) -> bytes: pass
|
||||
|
||||
class LocalStorage(StorageBackend):
|
||||
"""Development: Filesystem"""
|
||||
pass
|
||||
|
||||
class S3Storage(StorageBackend):
|
||||
"""Production: S3-compatible storage"""
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. API Access & Security
|
||||
|
||||
#### Development (Termux)
|
||||
- **Access**: Local only (127.0.0.1) or local network
|
||||
- **Port**: 8000 (or any available)
|
||||
- **HTTPS**: Not required (localhost)
|
||||
- **Authentication**: Optional (single user)
|
||||
- **CORS**: Permissive (development)
|
||||
- **Rate Limiting**: None needed
|
||||
|
||||
**Use Cases**:
|
||||
- Testing API endpoints
|
||||
- Web UI development
|
||||
- Local data management
|
||||
- Debugging
|
||||
|
||||
#### Production (Server)
|
||||
- **Access**: Public internet
|
||||
- **Port**: 443 (HTTPS) behind reverse proxy
|
||||
- **HTTPS**: Required (Let's Encrypt)
|
||||
- **Authentication**: JWT + API keys mandatory
|
||||
- **CORS**: Strict origin whitelisting
|
||||
- **Rate Limiting**: Aggressive (per-user, per-IP)
|
||||
|
||||
**Security Requirements**:
|
||||
- Nginx/Caddy reverse proxy
|
||||
- SSL/TLS certificates
|
||||
- API key rotation
|
||||
- Request validation
|
||||
- SQL injection prevention
|
||||
- File upload limits
|
||||
- DDoS protection
|
||||
|
||||
**Solution**: Environment-based security
|
||||
```python
|
||||
# config/security.py
|
||||
if os.getenv("GIGLEZ_MODE") == "production":
|
||||
REQUIRE_AUTH = True
|
||||
RATE_LIMIT = "10/minute"
|
||||
ALLOWED_ORIGINS = ["https://giglez.com"]
|
||||
else:
|
||||
REQUIRE_AUTH = False
|
||||
RATE_LIMIT = None
|
||||
ALLOWED_ORIGINS = ["*"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Performance & Scaling
|
||||
|
||||
#### Development (Termux)
|
||||
- **Workers**: 1 Uvicorn worker
|
||||
- **Concurrency**: Minimal (single user)
|
||||
- **Caching**: In-memory only (LRU)
|
||||
- **Background Tasks**: Simple queue or synchronous
|
||||
- **Resource Limits**: RAM < 2GB, CPU 2-4 cores
|
||||
|
||||
**Optimization Focus**:
|
||||
- Memory efficiency
|
||||
- Minimal background processes
|
||||
- Battery life considerations
|
||||
- Offline operation
|
||||
|
||||
#### Production (Server)
|
||||
- **Workers**: Multiple Uvicorn workers (4-8)
|
||||
- **Concurrency**: High (100+ concurrent users)
|
||||
- **Caching**: Redis + LRU
|
||||
- **Background Tasks**: Celery + RabbitMQ/Redis
|
||||
- **Resource Limits**: RAM 8-32GB, CPU 4-16 cores
|
||||
|
||||
**Optimization Focus**:
|
||||
- Query performance
|
||||
- Connection pooling
|
||||
- Horizontal scaling
|
||||
- CDN for static assets
|
||||
- Load balancing
|
||||
|
||||
**Solution**: Conditional imports
|
||||
```python
|
||||
# src/api/main.py
|
||||
if PRODUCTION_MODE:
|
||||
from celery import Celery
|
||||
celery_app = Celery('giglez', broker='redis://localhost')
|
||||
else:
|
||||
# Use FastAPI BackgroundTasks for development
|
||||
celery_app = None
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Logging & Monitoring
|
||||
|
||||
#### Development (Termux)
|
||||
- **Logging**: Console output (stdout)
|
||||
- **Level**: DEBUG
|
||||
- **Format**: Colorized, verbose
|
||||
- **Storage**: Optional file logging
|
||||
- **Monitoring**: Manual observation
|
||||
|
||||
**Example**:
|
||||
```python
|
||||
logger.add(sys.stdout, level="DEBUG", colorize=True)
|
||||
```
|
||||
|
||||
#### Production (Server)
|
||||
- **Logging**: Structured JSON to file + syslog
|
||||
- **Level**: INFO/WARNING
|
||||
- **Format**: JSON for parsing
|
||||
- **Storage**: Log rotation + cloud storage
|
||||
- **Monitoring**: Prometheus + Grafana, error tracking (Sentry)
|
||||
|
||||
**Example**:
|
||||
```python
|
||||
logger.add(
|
||||
"/var/log/giglez/app.log",
|
||||
level="INFO",
|
||||
format="{time:YYYY-MM-DD HH:mm:ss} | {level} | {message}",
|
||||
rotation="100 MB",
|
||||
compression="gz"
|
||||
)
|
||||
```
|
||||
|
||||
**Solution**: Environment-based logging
|
||||
```python
|
||||
# config/logging.py
|
||||
if PRODUCTION_MODE:
|
||||
setup_production_logging()
|
||||
else:
|
||||
setup_development_logging()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Backup & Disaster Recovery
|
||||
|
||||
#### Development (Termux)
|
||||
- **Database**: Manual pg_dump to SD card
|
||||
- **Files**: Synced to cloud storage (Google Drive, Dropbox)
|
||||
- **Frequency**: On-demand
|
||||
- **Recovery**: Restore from local backups
|
||||
|
||||
#### Production (Server)
|
||||
- **Database**: Automated daily backups with PITR
|
||||
- **Files**: Object storage replication (S3 versioning)
|
||||
- **Frequency**: Hourly incremental, daily full
|
||||
- **Recovery**: Automated restore procedures
|
||||
|
||||
**Backup Strategy**:
|
||||
```bash
|
||||
# Development
|
||||
pg_dump giglez > /sdcard/giglez_backup_$(date +%Y%m%d).sql
|
||||
|
||||
# Production
|
||||
pg_dump giglez | gzip | aws s3 cp - s3://giglez-backups/$(date +%Y%m%d).sql.gz
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 8. Network Considerations
|
||||
|
||||
#### Development (Termux)
|
||||
- **Connectivity**: WiFi or cellular (intermittent)
|
||||
- **Bandwidth**: Potentially limited (mobile data)
|
||||
- **Latency**: Variable (mobile networks)
|
||||
- **Offline Support**: Critical (wardriving in field)
|
||||
|
||||
**Implications**:
|
||||
- Queue uploads for when network available
|
||||
- Support offline capture and analysis
|
||||
- Minimize API calls
|
||||
- Compress uploads
|
||||
|
||||
#### Production (Server)
|
||||
- **Connectivity**: Always online (datacenter)
|
||||
- **Bandwidth**: High (1Gbps+)
|
||||
- **Latency**: Low (< 50ms typical)
|
||||
- **Offline Support**: Not applicable
|
||||
|
||||
**Implications**:
|
||||
- Assume network availability
|
||||
- No offline mode needed
|
||||
- Can make external API calls freely
|
||||
- Stream large responses
|
||||
|
||||
**Solution**: Offline queue
|
||||
```python
|
||||
# src/capture/uploader.py
|
||||
class UploadQueue:
|
||||
def queue_capture(self, capture_data):
|
||||
if network_available():
|
||||
upload_immediately(capture_data)
|
||||
else:
|
||||
save_to_local_queue(capture_data)
|
||||
|
||||
def process_queue(self):
|
||||
"""Called when network becomes available"""
|
||||
for item in get_queued_items():
|
||||
try:
|
||||
upload_immediately(item)
|
||||
mark_as_uploaded(item)
|
||||
except NetworkError:
|
||||
break # Stop and try again later
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Strategy
|
||||
|
||||
### Environment Variables Approach
|
||||
|
||||
Use different `.env` files for each environment:
|
||||
|
||||
```bash
|
||||
# .env.development (Termux)
|
||||
GIGLEZ_MODE=development
|
||||
GIGLEZ_DB_HOST=localhost
|
||||
GIGLEZ_DB_POOL_SIZE=5
|
||||
GIGLEZ_API_HOST=127.0.0.1
|
||||
GIGLEZ_API_PORT=8000
|
||||
GIGLEZ_STORAGE_TYPE=filesystem
|
||||
GIGLEZ_STORAGE_PATH=/data/data/com.termux/files/home/giglez/storage
|
||||
GIGLEZ_REQUIRE_AUTH=false
|
||||
GIGLEZ_LOG_LEVEL=DEBUG
|
||||
GIGLEZ_ENABLE_HARDWARE=true
|
||||
GIGLEZ_TEMBED_PORT=/dev/ttyUSB0
|
||||
|
||||
# .env.production (Server)
|
||||
GIGLEZ_MODE=production
|
||||
GIGLEZ_DB_HOST=db.giglez.com
|
||||
GIGLEZ_DB_POOL_SIZE=20
|
||||
GIGLEZ_API_HOST=0.0.0.0
|
||||
GIGLEZ_API_PORT=8000
|
||||
GIGLEZ_STORAGE_TYPE=s3
|
||||
GIGLEZ_STORAGE_BUCKET=giglez-captures
|
||||
GIGLEZ_REQUIRE_AUTH=true
|
||||
GIGLEZ_LOG_LEVEL=INFO
|
||||
GIGLEZ_ENABLE_HARDWARE=false
|
||||
```
|
||||
|
||||
### Configuration Loading
|
||||
|
||||
```python
|
||||
# config/settings.py
|
||||
import os
|
||||
from enum import Enum
|
||||
|
||||
class DeploymentMode(Enum):
|
||||
DEVELOPMENT = "development"
|
||||
PRODUCTION = "production"
|
||||
|
||||
class Settings:
|
||||
def __init__(self):
|
||||
self.mode = DeploymentMode(os.getenv("GIGLEZ_MODE", "development"))
|
||||
|
||||
@property
|
||||
def is_production(self) -> bool:
|
||||
return self.mode == DeploymentMode.PRODUCTION
|
||||
|
||||
@property
|
||||
def is_development(self) -> bool:
|
||||
return self.mode == DeploymentMode.DEVELOPMENT
|
||||
|
||||
@property
|
||||
def enable_hardware(self) -> bool:
|
||||
"""Only enable hardware access in development"""
|
||||
return os.getenv("GIGLEZ_ENABLE_HARDWARE", "false").lower() == "true"
|
||||
|
||||
@property
|
||||
def storage_backend(self) -> str:
|
||||
return os.getenv("GIGLEZ_STORAGE_TYPE", "filesystem")
|
||||
|
||||
settings = Settings()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
### Modular Component Design
|
||||
|
||||
```
|
||||
giglez/
|
||||
├── src/
|
||||
│ ├── core/ # Shared core (both modes)
|
||||
│ │ ├── database/ # Database models & queries
|
||||
│ │ ├── parser/ # .sub file parser
|
||||
│ │ ├── matcher/ # Signature matching
|
||||
│ │ ├── gps/ # GPS validation
|
||||
│ │ └── storage/ # Storage abstraction
|
||||
│ │
|
||||
│ ├── capture/ # Capture mode only (Termux)
|
||||
│ │ ├── tembed.py # T-Embed communication
|
||||
│ │ ├── scanner.py # Signal scanning
|
||||
│ │ ├── gps_manager.py # Android GPS integration
|
||||
│ │ └── uploader.py # Offline queue & upload
|
||||
│ │
|
||||
│ ├── api/ # API endpoints (both modes)
|
||||
│ │ ├── main.py # FastAPI app
|
||||
│ │ ├── routes/
|
||||
│ │ │ ├── captures.py # Upload endpoints
|
||||
│ │ │ ├── devices.py # Device catalog
|
||||
│ │ │ ├── query.py # Search endpoints
|
||||
│ │ │ └── stats.py # Statistics
|
||||
│ │ ├── auth.py # Authentication
|
||||
│ │ ├── middleware.py # Rate limiting, CORS
|
||||
│ │ └── dependencies.py # Dependency injection
|
||||
│ │
|
||||
│ └── web/ # Web interface (both modes)
|
||||
│ ├── static/ # CSS, JS, images
|
||||
│ └── templates/ # HTML templates
|
||||
│
|
||||
└── config/
|
||||
├── settings.py # Environment-based config
|
||||
├── database.py # Database connection
|
||||
├── storage.py # Storage backend selection
|
||||
└── logging.py # Logging configuration
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Recommendations
|
||||
|
||||
### 1. Use Feature Flags
|
||||
|
||||
```python
|
||||
# src/api/main.py
|
||||
from config.settings import settings
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
# Conditional hardware endpoints
|
||||
if settings.enable_hardware:
|
||||
@app.post("/api/capture/start")
|
||||
async def start_capture():
|
||||
# Only available in Termux mode
|
||||
pass
|
||||
|
||||
# Always available endpoints
|
||||
@app.post("/api/captures/upload")
|
||||
async def upload_captures():
|
||||
# Available in both modes
|
||||
pass
|
||||
```
|
||||
|
||||
### 2. Storage Abstraction
|
||||
|
||||
```python
|
||||
# config/storage.py
|
||||
from src.core.storage import LocalStorage, S3Storage
|
||||
|
||||
def get_storage_backend():
|
||||
if settings.storage_backend == "s3":
|
||||
return S3Storage(
|
||||
bucket=os.getenv("GIGLEZ_STORAGE_BUCKET"),
|
||||
access_key=os.getenv("AWS_ACCESS_KEY_ID"),
|
||||
secret_key=os.getenv("AWS_SECRET_ACCESS_KEY")
|
||||
)
|
||||
else:
|
||||
return LocalStorage(
|
||||
base_path=os.getenv("GIGLEZ_STORAGE_PATH")
|
||||
)
|
||||
|
||||
storage = get_storage_backend()
|
||||
```
|
||||
|
||||
### 3. Optional Dependencies
|
||||
|
||||
```python
|
||||
# requirements.txt (core)
|
||||
fastapi==0.109.0
|
||||
sqlalchemy==2.0.25
|
||||
psycopg2-binary==2.9.9
|
||||
...
|
||||
|
||||
# requirements-dev.txt (Termux additions)
|
||||
-r requirements.txt
|
||||
pyserial==3.5
|
||||
aioserial==1.3.2
|
||||
|
||||
# requirements-prod.txt (Server additions)
|
||||
-r requirements.txt
|
||||
boto3==1.34.0 # S3 storage
|
||||
redis==5.0.1 # Caching
|
||||
celery==5.3.4 # Background tasks
|
||||
gunicorn==21.2.0 # WSGI server
|
||||
```
|
||||
|
||||
### 4. Graceful Degradation
|
||||
|
||||
```python
|
||||
# src/api/routes/captures.py
|
||||
@app.post("/api/captures/upload")
|
||||
async def upload_captures(files: List[UploadFile]):
|
||||
# Core functionality works in both modes
|
||||
captures = []
|
||||
|
||||
for file in files:
|
||||
# Parse .sub file (works everywhere)
|
||||
metadata = parse_sub_file(file)
|
||||
|
||||
# Store file (abstracted)
|
||||
storage.save_file(metadata.file_hash, file.content)
|
||||
|
||||
# Save to database (works everywhere)
|
||||
capture = Capture(**metadata)
|
||||
db.add(capture)
|
||||
|
||||
captures.append(capture)
|
||||
|
||||
db.commit()
|
||||
|
||||
# Background matching (production) or synchronous (development)
|
||||
if settings.is_production:
|
||||
celery_app.send_task('match_signatures', args=[capture.file_hash])
|
||||
else:
|
||||
match_signatures_sync(capture.file_hash)
|
||||
|
||||
return {"uploaded": len(captures)}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Development Testing (Termux)
|
||||
- Direct hardware testing
|
||||
- USB serial communication
|
||||
- GPS acquisition
|
||||
- Offline queue processing
|
||||
- Local file storage
|
||||
|
||||
### Production Testing (Server)
|
||||
- API endpoint testing
|
||||
- File upload validation
|
||||
- Multi-user concurrency
|
||||
- Rate limiting
|
||||
- Authentication
|
||||
- S3 storage integration
|
||||
|
||||
### Shared Testing
|
||||
- Database operations
|
||||
- .sub file parsing
|
||||
- Signature matching
|
||||
- GPS validation
|
||||
- Query performance
|
||||
|
||||
**Test Structure**:
|
||||
```
|
||||
tests/
|
||||
├── unit/ # Unit tests (both modes)
|
||||
│ ├── test_parser.py
|
||||
│ ├── test_matcher.py
|
||||
│ └── test_gps.py
|
||||
│
|
||||
├── integration/ # Integration tests
|
||||
│ ├── test_api.py # Both modes
|
||||
│ ├── test_hardware.py # Development only
|
||||
│ └── test_storage.py # Both modes
|
||||
│
|
||||
└── e2e/ # End-to-end tests
|
||||
├── test_capture.py # Development only
|
||||
└── test_upload.py # Both modes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
### Development (Termux) Setup
|
||||
- [ ] Install Termux from F-Droid
|
||||
- [ ] Install PostgreSQL: `pkg install postgresql`
|
||||
- [ ] Install Python: `pkg install python`
|
||||
- [ ] Install dependencies: `pip install -r requirements-dev.txt`
|
||||
- [ ] Configure USB permissions for T-Embed
|
||||
- [ ] Set up Termux:API for GPS
|
||||
- [ ] Create `.env.development`
|
||||
- [ ] Run database setup: `./scripts/setup_database.sh`
|
||||
- [ ] Test hardware connection
|
||||
- [ ] Start API: `python src/api/main.py`
|
||||
|
||||
### Production (Server) Setup
|
||||
- [ ] Provision VPS (2GB+ RAM, 2+ cores)
|
||||
- [ ] Install PostgreSQL 16 + PostGIS
|
||||
- [ ] Install Python 3.10+
|
||||
- [ ] Install Redis (for caching)
|
||||
- [ ] Install Nginx (reverse proxy)
|
||||
- [ ] Configure SSL with Let's Encrypt
|
||||
- [ ] Set up S3/MinIO for file storage
|
||||
- [ ] Configure firewall (allow 80, 443)
|
||||
- [ ] Create `.env.production`
|
||||
- [ ] Run database setup
|
||||
- [ ] Set up systemd service
|
||||
- [ ] Configure automated backups
|
||||
- [ ] Set up monitoring (optional)
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Development (Termux) Optimizations
|
||||
- Smaller connection pool (5 + 5)
|
||||
- In-memory caching only
|
||||
- Synchronous processing (no Celery)
|
||||
- Minimal logging
|
||||
- Local file storage
|
||||
|
||||
### Production (Server) Optimizations
|
||||
- Larger connection pool (20 + 40)
|
||||
- Redis caching
|
||||
- Async background tasks (Celery)
|
||||
- Structured logging with rotation
|
||||
- CDN for file delivery
|
||||
- Read replicas for scaling
|
||||
- Materialized view refresh (cron)
|
||||
|
||||
---
|
||||
|
||||
## Cost Considerations
|
||||
|
||||
### Development (Termux)
|
||||
- **Hardware**: LilyGo T-Embed (~$30-50)
|
||||
- **Android Device**: Existing phone/tablet
|
||||
- **Infrastructure**: $0 (local)
|
||||
- **Storage**: Device storage only
|
||||
- **Total**: ~$30-50 one-time
|
||||
|
||||
### Production (Server)
|
||||
- **VPS**: $10-50/month (2-8GB RAM)
|
||||
- **Database**: Included or $10-30/month (managed)
|
||||
- **Storage**: $0.02/GB/month (S3) or included (filesystem)
|
||||
- **Bandwidth**: Typically included (1-5TB)
|
||||
- **Domain**: $10-15/year
|
||||
- **SSL**: $0 (Let's Encrypt)
|
||||
- **Total**: $10-80/month
|
||||
|
||||
---
|
||||
|
||||
## Recommended Phase 2 Approach
|
||||
|
||||
### Build for Both from the Start
|
||||
|
||||
**Strategy**: Implement the API with abstraction layers that work in both environments.
|
||||
|
||||
**Implementation Order**:
|
||||
1. ✅ Core API structure (FastAPI app)
|
||||
2. ✅ Storage abstraction (filesystem + S3)
|
||||
3. ✅ Upload endpoint (works in both modes)
|
||||
4. ✅ Authentication (optional in dev, required in prod)
|
||||
5. ✅ Background tasks (sync in dev, Celery in prod)
|
||||
6. ✅ Environment-based configuration
|
||||
|
||||
**Key Principle**:
|
||||
> Write code that works in both environments by default, with conditional imports/features for mode-specific functionality.
|
||||
|
||||
---
|
||||
|
||||
## Summary: Key Differences
|
||||
|
||||
| Aspect | Development (Termux) | Production (Server) |
|
||||
|--------|---------------------|---------------------|
|
||||
| **Hardware** | T-Embed USB + GPS | None (file uploads) |
|
||||
| **Users** | Single user | Multi-user |
|
||||
| **Database** | Local PostgreSQL (small pool) | Remote PostgreSQL (large pool) |
|
||||
| **Storage** | Local filesystem | S3/object storage |
|
||||
| **API Access** | localhost only | Public internet |
|
||||
| **Security** | Optional auth | Required auth + HTTPS |
|
||||
| **Background Tasks** | Synchronous | Celery async |
|
||||
| **Caching** | In-memory LRU | Redis + LRU |
|
||||
| **Logging** | Console (DEBUG) | File + JSON (INFO) |
|
||||
| **Monitoring** | Manual | Automated (Prometheus) |
|
||||
| **Scaling** | N/A | Horizontal |
|
||||
| **Cost** | $0/month | $10-80/month |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
Before implementing Phase 2, we need to:
|
||||
|
||||
1. ✅ **Create environment configuration** (`.env.development`, `.env.production`)
|
||||
2. ✅ **Implement storage abstraction** (`src/core/storage/`)
|
||||
3. ✅ **Create settings module** (`config/settings.py`)
|
||||
4. ✅ **Document deployment procedures** (development guide, production guide)
|
||||
5. ✅ **Begin FastAPI implementation** with environment awareness
|
||||
|
||||
**Recommendation**: Start with a **shared API implementation** that works in both modes, then add mode-specific features as needed.
|
||||
|
||||
---
|
||||
|
||||
**Document Version**: 1.0
|
||||
**Status**: Ready for Phase 2 Implementation
|
||||
**Next Action**: Create environment-aware FastAPI application structure
|
||||
Reference in New Issue
Block a user