giglez/DEPLOYMENT_CONSIDERATIONS.md

GigLez Deployment Considerations: Development vs Production

Created: 2026-01-12 Purpose: Identify critical differences between local development and production deployment

---

Executive Summary

GigLez needs to support two distinct deployment scenarios:

1. Local Development (Termux on Android)

   • Resource-constrained (mobile device)
   • Single-user access
   • Direct USB serial communication
   • Local file storage
   • Development/testing workflow

2. Production Server (Cloud/VPS)

   • Multi-user platform
   • Public API access
   • No hardware connection
   • Object storage for files
   • High availability requirements

Key Insight: We need a modular architecture that supports both scenarios without code duplication.

---

Architecture Strategy

Hybrid Architecture: "Capture" vs "Platform" Mode

┌─────────────────────────────────────────────────────────┐
│                    GigLez Application                    │
├─────────────────────────────────────────────────────────┤
│                                                           │
│  ┌────────────────┐         ┌────────────────┐          │
│  │  Capture Mode  │         │ Platform Mode  │          │
│  │   (Termux)     │         │   (Server)     │          │
│  ├────────────────┤         ├────────────────┤          │
│  │ - T-Embed USB  │         │ - No hardware  │          │
│  │ - Local GPS    │         │ - File uploads │          │
│  │ - Auto-capture │         │ - Multi-user   │          │
│  │ - Local API    │         │ - Public API   │          │
│  └────────────────┘         └────────────────┘          │
│           │                          │                    │
│           └──────────┬───────────────┘                    │
│                      │                                    │
│              ┌───────▼────────┐                           │
│              │  Shared Core   │                           │
│              ├────────────────┤                           │
│              │ - Database     │                           │
│              │ - Parser       │                           │
│              │ - Matcher      │                           │
│              │ - GPS Validator│                           │
│              │ - API Endpoints│                           │
│              └────────────────┘                           │
└─────────────────────────────────────────────────────────┘

---

Detailed Comparison

1. Hardware Access

Development (Termux)

• T-Embed Connection: USB serial communication (pyserial)
• GPS Source: Android native GPS via SL4A or Termux:API
• Real-time Capture: Direct signal scanning and capture
• Storage: Local filesystem on Android device

Implications:

• Need hardware abstraction layer
• Support offline operation
• Handle USB disconnections
• Batch upload when network available

Production (Server)

• No Hardware: Accept pre-captured .sub files only
• GPS Source: Embedded in upload manifest
• No Real-time: Historical data only
• Storage: Object storage (S3, MinIO, or filesystem)

Implications:

• No serial communication code needed
• Network-dependent operation
• Focus on file processing pipeline
• Horizontal scaling possible

Solution:

# Abstract hardware interface
class CaptureSource(ABC):
    @abstractmethod
    def scan_signals(self): pass

    @abstractmethod
    def get_gps(self): pass

class TEmbedSource(CaptureSource):
    """Development: Direct hardware"""
    pass

class UploadSource(CaptureSource):
    """Production: File uploads"""
    pass

---

2. Database Configuration

Development (Termux)

• PostgreSQL: Local installation via pkg install postgresql
• Host: localhost or 127.0.0.1
• Port: 5432
• Connections: Small pool (5 + 5 overflow)
• Storage: Device internal storage or SD card

Considerations:

• Limited RAM on mobile devices
• Smaller connection pool
• May need to tune PostgreSQL for mobile
• Backup to SD card or cloud

Production (Server)

• PostgreSQL: Dedicated server or managed database (RDS, Cloud SQL)
• Host: Remote database server
• Port: 5432 (or custom)
• Connections: Larger pool (20 + 40 overflow)
• Storage: SSD/NVMe for performance

Considerations:

• Connection pooling via PgBouncer
• Read replicas for scaling
• Automated backups
• Point-in-time recovery

Solution: Environment-based configuration

# Development (.env)
GIGLEZ_DB_HOST=localhost
GIGLEZ_DB_POOL_SIZE=5
GIGLEZ_DB_MAX_OVERFLOW=5

# Production (.env.production)
GIGLEZ_DB_HOST=db.giglez.com
GIGLEZ_DB_POOL_SIZE=20
GIGLEZ_DB_MAX_OVERFLOW=40

---

3. File Storage

Development (Termux)

• Location: /data/data/com.termux/files/home/giglez/storage/
• Structure: Organized by session/date
• Size: Limited by device storage (typically < 64GB)
• Access: Direct filesystem access

Structure:

storage/
├── captures/
│   ├── 2026-01-12/
│   │   ├── session_abc123/
│   │   │   ├── capture_001.sub
│   │   │   ├── capture_002.sub
│   │   │   └── manifest.json

Production (Server)

• Location: Object storage (S3, MinIO, GCS) or networked filesystem
• Structure: Content-addressed by file hash
• Size: Unlimited (cloud storage)
• Access: Via storage API or CDN

Structure:

s3://giglez-captures/
├── ab/
│   ├── c1/
│   │   └── abc123...def456.sub  # First 2+2 chars of hash for sharding

Solution: Storage abstraction

class StorageBackend(ABC):
    @abstractmethod
    def save_file(self, file_hash: str, content: bytes): pass

    @abstractmethod
    def get_file(self, file_hash: str) -> bytes: pass

class LocalStorage(StorageBackend):
    """Development: Filesystem"""
    pass

class S3Storage(StorageBackend):
    """Production: S3-compatible storage"""
    pass

---

4. API Access & Security

Development (Termux)

• Access: Local only (127.0.0.1) or local network
• Port: 8000 (or any available)
• HTTPS: Not required (localhost)
• Authentication: Optional (single user)
• CORS: Permissive (development)
• Rate Limiting: None needed

Use Cases:

• Testing API endpoints
• Web UI development
• Local data management
• Debugging

Production (Server)

• Access: Public internet
• Port: 443 (HTTPS) behind reverse proxy
• HTTPS: Required (Let's Encrypt)
• Authentication: JWT + API keys mandatory
• CORS: Strict origin whitelisting
• Rate Limiting: Aggressive (per-user, per-IP)

Security Requirements:

• Nginx/Caddy reverse proxy
• SSL/TLS certificates
• API key rotation
• Request validation
• SQL injection prevention
• File upload limits
• DDoS protection

Solution: Environment-based security

# config/security.py
if os.getenv("GIGLEZ_MODE") == "production":
    REQUIRE_AUTH = True
    RATE_LIMIT = "10/minute"
    ALLOWED_ORIGINS = ["https://giglez.com"]
else:
    REQUIRE_AUTH = False
    RATE_LIMIT = None
    ALLOWED_ORIGINS = ["*"]

---

5. Performance & Scaling

Development (Termux)

• Workers: 1 Uvicorn worker
• Concurrency: Minimal (single user)
• Caching: In-memory only (LRU)
• Background Tasks: Simple queue or synchronous
• Resource Limits: RAM < 2GB, CPU 2-4 cores

Optimization Focus:

• Memory efficiency
• Minimal background processes
• Battery life considerations
• Offline operation

Production (Server)

• Workers: Multiple Uvicorn workers (4-8)
• Concurrency: High (100+ concurrent users)
• Caching: Redis + LRU
• Background Tasks: Celery + RabbitMQ/Redis
• Resource Limits: RAM 8-32GB, CPU 4-16 cores

Optimization Focus:

• Query performance
• Connection pooling
• Horizontal scaling
• CDN for static assets
• Load balancing

Solution: Conditional imports

# src/api/main.py
if PRODUCTION_MODE:
    from celery import Celery
    celery_app = Celery('giglez', broker='redis://localhost')
else:
    # Use FastAPI BackgroundTasks for development
    celery_app = None

---

6. Logging & Monitoring

Development (Termux)

• Logging: Console output (stdout)
• Level: DEBUG
• Format: Colorized, verbose
• Storage: Optional file logging
• Monitoring: Manual observation

Example:

logger.add(sys.stdout, level="DEBUG", colorize=True)

Production (Server)

• Logging: Structured JSON to file + syslog
• Level: INFO/WARNING
• Format: JSON for parsing
• Storage: Log rotation + cloud storage
• Monitoring: Prometheus + Grafana, error tracking (Sentry)

Example:

logger.add(
    "/var/log/giglez/app.log",
    level="INFO",
    format="{time:YYYY-MM-DD HH:mm:ss} | {level} | {message}",
    rotation="100 MB",
    compression="gz"
)

Solution: Environment-based logging

# config/logging.py
if PRODUCTION_MODE:
    setup_production_logging()
else:
    setup_development_logging()

---

7. Backup & Disaster Recovery

Development (Termux)

• Database: Manual pg_dump to SD card
• Files: Synced to cloud storage (Google Drive, Dropbox)
• Frequency: On-demand
• Recovery: Restore from local backups

Production (Server)

• Database: Automated daily backups with PITR
• Files: Object storage replication (S3 versioning)
• Frequency: Hourly incremental, daily full
• Recovery: Automated restore procedures

Backup Strategy:

# Development
pg_dump giglez > /sdcard/giglez_backup_$(date +%Y%m%d).sql

# Production
pg_dump giglez | gzip | aws s3 cp - s3://giglez-backups/$(date +%Y%m%d).sql.gz

---

8. Network Considerations

Development (Termux)

• Connectivity: WiFi or cellular (intermittent)
• Bandwidth: Potentially limited (mobile data)
• Latency: Variable (mobile networks)
• Offline Support: Critical (wardriving in field)

Implications:

• Queue uploads for when network available
• Support offline capture and analysis
• Minimize API calls
• Compress uploads

Production (Server)

• Connectivity: Always online (datacenter)
• Bandwidth: High (1Gbps+)
• Latency: Low (< 50ms typical)
• Offline Support: Not applicable

Implications:

• Assume network availability
• No offline mode needed
• Can make external API calls freely
• Stream large responses

Solution: Offline queue

# src/capture/uploader.py
class UploadQueue:
    def queue_capture(self, capture_data):
        if network_available():
            upload_immediately(capture_data)
        else:
            save_to_local_queue(capture_data)

    def process_queue(self):
        """Called when network becomes available"""
        for item in get_queued_items():
            try:
                upload_immediately(item)
                mark_as_uploaded(item)
            except NetworkError:
                break  # Stop and try again later

---

Configuration Strategy

Environment Variables Approach

Use different .env files for each environment:

# .env.development (Termux)
GIGLEZ_MODE=development
GIGLEZ_DB_HOST=localhost
GIGLEZ_DB_POOL_SIZE=5
GIGLEZ_API_HOST=127.0.0.1
GIGLEZ_API_PORT=8000
GIGLEZ_STORAGE_TYPE=filesystem
GIGLEZ_STORAGE_PATH=/data/data/com.termux/files/home/giglez/storage
GIGLEZ_REQUIRE_AUTH=false
GIGLEZ_LOG_LEVEL=DEBUG
GIGLEZ_ENABLE_HARDWARE=true
GIGLEZ_TEMBED_PORT=/dev/ttyUSB0

# .env.production (Server)
GIGLEZ_MODE=production
GIGLEZ_DB_HOST=db.giglez.com
GIGLEZ_DB_POOL_SIZE=20
GIGLEZ_API_HOST=0.0.0.0
GIGLEZ_API_PORT=8000
GIGLEZ_STORAGE_TYPE=s3
GIGLEZ_STORAGE_BUCKET=giglez-captures
GIGLEZ_REQUIRE_AUTH=true
GIGLEZ_LOG_LEVEL=INFO
GIGLEZ_ENABLE_HARDWARE=false

Configuration Loading

# config/settings.py
import os
from enum import Enum

class DeploymentMode(Enum):
    DEVELOPMENT = "development"
    PRODUCTION = "production"

class Settings:
    def __init__(self):
        self.mode = DeploymentMode(os.getenv("GIGLEZ_MODE", "development"))

    @property
    def is_production(self) -> bool:
        return self.mode == DeploymentMode.PRODUCTION

    @property
    def is_development(self) -> bool:
        return self.mode == DeploymentMode.DEVELOPMENT

    @property
    def enable_hardware(self) -> bool:
        """Only enable hardware access in development"""
        return os.getenv("GIGLEZ_ENABLE_HARDWARE", "false").lower() == "true"

    @property
    def storage_backend(self) -> str:
        return os.getenv("GIGLEZ_STORAGE_TYPE", "filesystem")

settings = Settings()

---

Recommended Architecture

Modular Component Design

giglez/
├── src/
│   ├── core/                    # Shared core (both modes)
│   │   ├── database/            # Database models & queries
│   │   ├── parser/              # .sub file parser
│   │   ├── matcher/             # Signature matching
│   │   ├── gps/                 # GPS validation
│   │   └── storage/             # Storage abstraction
│   │
│   ├── capture/                 # Capture mode only (Termux)
│   │   ├── tembed.py            # T-Embed communication
│   │   ├── scanner.py           # Signal scanning
│   │   ├── gps_manager.py       # Android GPS integration
│   │   └── uploader.py          # Offline queue & upload
│   │
│   ├── api/                     # API endpoints (both modes)
│   │   ├── main.py              # FastAPI app
│   │   ├── routes/
│   │   │   ├── captures.py      # Upload endpoints
│   │   │   ├── devices.py       # Device catalog
│   │   │   ├── query.py         # Search endpoints
│   │   │   └── stats.py         # Statistics
│   │   ├── auth.py              # Authentication
│   │   ├── middleware.py        # Rate limiting, CORS
│   │   └── dependencies.py      # Dependency injection
│   │
│   └── web/                     # Web interface (both modes)
│       ├── static/              # CSS, JS, images
│       └── templates/           # HTML templates
│
└── config/
    ├── settings.py              # Environment-based config
    ├── database.py              # Database connection
    ├── storage.py               # Storage backend selection
    └── logging.py               # Logging configuration

---

Implementation Recommendations

1. Use Feature Flags

# src/api/main.py
from config.settings import settings

app = FastAPI()

# Conditional hardware endpoints
if settings.enable_hardware:
    @app.post("/api/capture/start")
    async def start_capture():
        # Only available in Termux mode
        pass

# Always available endpoints
@app.post("/api/captures/upload")
async def upload_captures():
    # Available in both modes
    pass

2. Storage Abstraction

# config/storage.py
from src.core.storage import LocalStorage, S3Storage

def get_storage_backend():
    if settings.storage_backend == "s3":
        return S3Storage(
            bucket=os.getenv("GIGLEZ_STORAGE_BUCKET"),
            access_key=os.getenv("AWS_ACCESS_KEY_ID"),
            secret_key=os.getenv("AWS_SECRET_ACCESS_KEY")
        )
    else:
        return LocalStorage(
            base_path=os.getenv("GIGLEZ_STORAGE_PATH")
        )

storage = get_storage_backend()

3. Optional Dependencies

# requirements.txt (core)
fastapi==0.109.0
sqlalchemy==2.0.25
psycopg2-binary==2.9.9
...

# requirements-dev.txt (Termux additions)
-r requirements.txt
pyserial==3.5
aioserial==1.3.2

# requirements-prod.txt (Server additions)
-r requirements.txt
boto3==1.34.0  # S3 storage
redis==5.0.1   # Caching
celery==5.3.4  # Background tasks
gunicorn==21.2.0  # WSGI server

4. Graceful Degradation

# src/api/routes/captures.py
@app.post("/api/captures/upload")
async def upload_captures(files: List[UploadFile]):
    # Core functionality works in both modes
    captures = []

    for file in files:
        # Parse .sub file (works everywhere)
        metadata = parse_sub_file(file)

        # Store file (abstracted)
        storage.save_file(metadata.file_hash, file.content)

        # Save to database (works everywhere)
        capture = Capture(**metadata)
        db.add(capture)

        captures.append(capture)

    db.commit()

    # Background matching (production) or synchronous (development)
    if settings.is_production:
        celery_app.send_task('match_signatures', args=[capture.file_hash])
    else:
        match_signatures_sync(capture.file_hash)

    return {"uploaded": len(captures)}

---

Testing Strategy

Development Testing (Termux)

• Direct hardware testing
• USB serial communication
• GPS acquisition
• Offline queue processing
• Local file storage

Production Testing (Server)

• API endpoint testing
• File upload validation
• Multi-user concurrency
• Rate limiting
• Authentication
• S3 storage integration

Shared Testing

• Database operations
• .sub file parsing
• Signature matching
• GPS validation
• Query performance

Test Structure:

tests/
├── unit/                    # Unit tests (both modes)
│   ├── test_parser.py
│   ├── test_matcher.py
│   └── test_gps.py
│
├── integration/             # Integration tests
│   ├── test_api.py          # Both modes
│   ├── test_hardware.py     # Development only
│   └── test_storage.py      # Both modes
│
└── e2e/                     # End-to-end tests
    ├── test_capture.py      # Development only
    └── test_upload.py       # Both modes

---

Deployment Checklist

Development (Termux) Setup

• Install Termux from F-Droid
• Install PostgreSQL: pkg install postgresql
• Install Python: pkg install python
• Install dependencies: pip install -r requirements-dev.txt
• Configure USB permissions for T-Embed
• Set up Termux:API for GPS
• Create .env.development
• Run database setup: ./scripts/setup_database.sh
• Test hardware connection
• Start API: python src/api/main.py

Production (Server) Setup

• Provision VPS (2GB+ RAM, 2+ cores)
• Install PostgreSQL 16 + PostGIS
• Install Python 3.10+
• Install Redis (for caching)
• Install Nginx (reverse proxy)
• Configure SSL with Let's Encrypt
• Set up S3/MinIO for file storage
• Configure firewall (allow 80, 443)
• Create .env.production
• Run database setup
• Set up systemd service
• Configure automated backups
• Set up monitoring (optional)

---

Performance Considerations

Development (Termux) Optimizations

• Smaller connection pool (5 + 5)
• In-memory caching only
• Synchronous processing (no Celery)
• Minimal logging
• Local file storage

Production (Server) Optimizations

• Larger connection pool (20 + 40)
• Redis caching
• Async background tasks (Celery)
• Structured logging with rotation
• CDN for file delivery
• Read replicas for scaling
• Materialized view refresh (cron)

---

Cost Considerations

Development (Termux)

• Hardware: LilyGo T-Embed (~$30-50)
• Android Device: Existing phone/tablet
• Infrastructure: $0 (local)
• Storage: Device storage only
• Total: ~$30-50 one-time

Production (Server)

• VPS: $10-50/month (2-8GB RAM)
• Database: Included or $10-30/month (managed)
• Storage: $0.02/GB/month (S3) or included (filesystem)
• Bandwidth: Typically included (1-5TB)
• Domain: $10-15/year
• SSL: $0 (Let's Encrypt)
• Total: $10-80/month

---

Recommended Phase 2 Approach

Build for Both from the Start

Strategy: Implement the API with abstraction layers that work in both environments.

Implementation Order:

1. ✅ Core API structure (FastAPI app)
2. ✅ Storage abstraction (filesystem + S3)
3. ✅ Upload endpoint (works in both modes)
4. ✅ Authentication (optional in dev, required in prod)
5. ✅ Background tasks (sync in dev, Celery in prod)
6. ✅ Environment-based configuration

Key Principle:

Write code that works in both environments by default, with conditional imports/features for mode-specific functionality.

---

Summary: Key Differences

Aspect	Development (Termux)	Production (Server)
Hardware	T-Embed USB + GPS	None (file uploads)
Users	Single user	Multi-user
Database	Local PostgreSQL (small pool)	Remote PostgreSQL (large pool)
Storage	Local filesystem	S3/object storage
API Access	localhost only	Public internet
Security	Optional auth	Required auth + HTTPS
Background Tasks	Synchronous	Celery async
Caching	In-memory LRU	Redis + LRU
Logging	Console (DEBUG)	File + JSON (INFO)
Monitoring	Manual	Automated (Prometheus)
Scaling	N/A	Horizontal
Cost	$0/month	$10-80/month

---

Next Steps

Before implementing Phase 2, we need to:

1. ✅ Create environment configuration (.env.development, .env.production)
2. ✅ Implement storage abstraction (src/core/storage/)
3. ✅ Create settings module (config/settings.py)
4. ✅ Document deployment procedures (development guide, production guide)
5. ✅ Begin FastAPI implementation with environment awareness

Recommendation: Start with a shared API implementation that works in both modes, then add mode-specific features as needed.

---

Document Version: 1.0 Status: Ready for Phase 2 Implementation Next Action: Create environment-aware FastAPI application structure