System Design

Production Architecture

A containerized VPS deployment with Docker, managed PostgreSQL, cloud storage, and CDN delivery — built for reliability and scale.

Infrastructure Stack

A modern cloud-native stack built on Digital Ocean services with Docker containerization

Digital Ocean VPS

Single droplet hosting all Docker containers with isolated environments

  • Ubuntu 22.04 LTS
  • Docker Engine 24.x
  • Docker Compose v2
  • Automatic restarts

Container Orchestration

Four isolated Docker containers working in harmony

  • Frontend (Next.js 16)
  • Backend (Python Scrapers)
  • Nginx Reverse Proxy
  • Certbot SSL Manager

Managed PostgreSQL

Digital Ocean database cluster with connection pooling

  • Production + Dev + Staging
  • Automated daily backups
  • Connection pooling
  • SSL/TLS encryption

Cloud Storage & CDN

Digital Ocean Spaces with edge CDN for media delivery

  • DO Spaces object storage
  • CDN edge caching
  • Custom domain masking
  • media.rsadigital.co.za

Container Architecture

Four Docker containers orchestrated via Docker Compose on a single VPS

Digital Ocean VPS (Ubuntu 22.04)
rsa-digital-frontend

Frontend Application

Next.js 16 application serving the public website and admin dashboard with App Router and Server Components.

Next.js 16 App Router
React Server Components
Subdomain-based routing
Drizzle ORM integration
Server Actions for mutations
Framer Motion animations
rsadigital.co.za, app.rsadigital.co.za
gumtree-monitor

Scraper Service

Python-based data ingestion engine with Selenium automation for two-phase scraping across multiple categories.

Two-phase scraping (Lite → Heavy)
VehiclesScraper with multi-category support
Phone extraction with auth
Image upload to DO Spaces
Data validation measures
Randomized intervals
Internal container
notifier-service

Notifier Service

WhatsApp notification delivery with database-driven recipient management, lead quality filtering, and quiet hours.

Twilio WhatsApp API
Content Templates
Quiet hours enforcement
Lead quality routing
Database-driven recipients
Delivery tracking
Twilio API integration
nginx-proxy

Nginx Reverse Proxy

High-performance reverse proxy handling SSL termination, subdomain routing, rate limiting, and load balancing.

SSL/TLS termination
Subdomain routing
Rate limiting
Gzip compression
Security headers
WebSocket support
Ports 80/443

Data Flow Pipeline

Gumtree SASearch pages
Lite ScrapeCollect URLs
Heavy ScrapeExtract details
DO SpacesStore images
PostgreSQLUPSERT data
NotifierFilter & match
WhatsAppDeliver alerts

Security & Best Practices

SSL/TLS Encryption

Let's Encrypt certificates with automated renewal via Certbot cron jobs

Network Isolation

Docker bridge networks isolating internal services from external access

Rate Limiting

Nginx-based rate limiting protecting against abuse and DDoS attempts

Environment Isolation

Separate production, development, and staging database environments

Technology Stack

Frontend
  • Next.js 16
  • React 19
  • TypeScript
  • Tailwind CSS v4
  • Framer Motion
Backend
  • Python 3.11
  • Selenium
  • BeautifulSoup
  • asyncio
  • Twilio SDK
Database
  • PostgreSQL 16
  • Drizzle ORM
  • Connection Pooling
  • Drizzle Kit
Infrastructure
  • Docker
  • Nginx
  • Certbot
  • Digital Ocean
  • DO Spaces