Implementation Blueprint

Prerequisites

sudo apt update && sudo apt upgrade -y
sudo apt install docker.io docker-compose -y
sudo systemctl enable docker
sudo systemctl start docker

Docker Compose Production Setup

version: '3'

services:
  ckan:
    image: ckan/ckan:latest
    ports:
      - "5000:5000"
    environment:
      - CKAN_SQLALCHEMY_URL=postgresql://ckan:password@db/ckan
      - CKAN_SOLR_URL=http://solr:8983/solr/ckan
      - CKAN_REDIS_URL=redis://redis:6379/1
    depends_on:
      - db
      - solr
      - redis

  db:
    image: ckan/postgresql:latest
    environment:
      - POSTGRES_USER=ckan
      - POSTGRES_PASSWORD=password
    volumes:
      - pg_data:/var/lib/postgresql/data

  solr:
    image: ckan/solr:latest
    volumes:
      - solr_data:/opt/solr/server/solr/ckan/data

  redis:
    image: redis:6-alpine

volumes:
  pg_data:
  solr_data:

Kubernetes Production Deployment (Recommended)

helm repo add ckan https://ckan.github.io/ckan-helm/
helm install my-portal ckan/ckan --namespace data-portal --create-namespace

• Horizontal Scaling: Scale web pods to handle thousands of simultaneous users.
• Resilient Data Store: Use managed PostgreSQL and Solr clusters for maximum uptime.
• Storage Flexibility: Easily attach S3 or Azure Blob Storage for dataset file storage.

Scaling & Performance

• Caching: Implement a heavy caching layer (Varnish or NGINX) in front of the CKAN API.
• Dedicated Workers: Run DataPusher and harvester tasks on separate pods to avoid impacting web performance.
• Solr Optimization: Tune Solr's memory and shard the index if you have hundreds of thousands of datasets.

Backup & Maintenance

• Database Dumps: Regularly backup the primary PostgreSQL and the DataStore DB separately.
• Metadata Integrity: Use CKAN's hashing tools to ensure data consistency across the harvest and store lifecycle.
• Volume Backups: Ensure persistent volumes for Solr and file storage (if local) are snapshotted daily.