Backup & Restore

Protect your Prisme.ai platform data with comprehensive backup and restore procedures

A robust backup and restore strategy is essential for protecting your Prisme.ai platform data and ensuring business continuity. This guide provides detailed instructions for backing up and restoring all components of your self-hosted Prisme.ai environment.

Backup Strategy

What to Back Up

Your Prisme.ai platform requires backing up several components:

Client-Managed Databases

• MongoDB/compatible database
• Elasticsearch/OpenSearch
• Redis

Object Storage

• S3 or compatible object storage
• Document files and attachments

Configuration

• Kubernetes manifests
• Helm values
• Terraform state files

Secrets

• Kubernetes secrets
• Certificate files
• API keys and credentials

Backup Frequency

Recommended backup frequencies based on data criticality:

Component	Frequency	Retention
MongoDB	Daily	30 days
Elasticsearch/OpenSearch	Daily	14 days
Redis	Daily	7 days
S3 Storage	Weekly (incremental)	90 days
Configuration	After changes	Last 10 versions

Adjust backup frequency based on your organization’s Recovery Point Objective (RPO) requirements.

Database Backup Procedures

MongoDB

1. Create MongoDB Backup

   Use mongodump to create a full backup of your MongoDB database:

   # For standalone MongoDB
   mongodump --uri="mongodb://username:password@hostname:port/database" \
     --out=/path/to/backup/mongo/$(date +%Y-%m-%d)
   
   # For MongoDB Atlas
   mongodump --uri="mongodb+srv://username:[e-mail]/database" \
     --out=/path/to/backup/mongo/$(date +%Y-%m-%d)
   

   MongoDB Backup Options

   Additional options to consider:

   # Compressed backup (reduces storage requirements)
   mongodump --uri="mongodb://username:password@hostname:port/database" \
     --out=/path/to/backup/mongo/$(date +%Y-%m-%d) \
     --gzip
   
   # Backup specific collections
   mongodump --uri="mongodb://username:password@hostname:port/database" \
     --collection=users --collection=agents \
     --out=/path/to/backup/mongo/$(date +%Y-%m-%d)
   
2. Verify MongoDB Backup

   Ensure the backup contains all expected data:

   # List databases in the backup
   find /path/to/backup/mongo/$(date +%Y-%m-%d) -type d -depth 1
   
   # Count documents in a specific collection
   mongorestore --uri="mongodb://username:password@hostname:port/database" \
     --nsInclude="database.collection" \
     --dryRun /path/to/backup/mongo/$(date +%Y-%m-%d) | grep "documents to restore"
   
3. Schedule Regular Backups

   Create a cron job to automate daily backups:

   # Add to crontab
   0 1 * * * /path/to/mongodb-backup-script.sh > /path/to/logs/mongodb-backup-$(date +\%Y-\%m-\%d).log 2>&1
   

   Example backup script (mongodb-backup-script.sh):

   #!/bin/bash
   
   BACKUP_DIR="/path/to/backup/mongo"
   DATE=$(date +%Y-%m-%d)
   RETENTION_DAYS=30
   
   # Create backup
   mongodump --uri="mongodb://username:password@hostname:port/database" \
     --out=${BACKUP_DIR}/${DATE} --gzip
   
   # Clean up old backups
   find ${BACKUP_DIR} -type d -mtime +${RETENTION_DAYS} -exec rm -rf {} \;
   

Elasticsearch/OpenSearch

1. Create Elasticsearch/OpenSearch Backup

   Use Elasticsearch snapshot API or elasticdump for backups:

   Using Snapshot API (recommended)

   # 1. Register a snapshot repository (S3)
   curl -X PUT "localhost:9200/_snapshot/backup_repository" -H "Content-Type: application/json" -d'
   {
     "type": "s3",
     "settings": {
       "bucket": "your-backup-bucket",
       "region": "your-region",
       "role_arn": "arn:aws:iam::account-id:role/role-name"
     }
   }'
   
   # 2. Create a snapshot
   curl -X PUT "localhost:9200/_snapshot/backup_repository/snapshot_$(date +%Y%m%d)" -H "Content-Type: application/json" -d'
   {
     "indices": "*",
     "ignore_unavailable": true,
     "include_global_state": true
   }'
   
   # 3. Check snapshot status
   curl -X GET "localhost:9200/_snapshot/backup_repository/snapshot_$(date +%Y%m%d)/_status"
   

   Using elasticdump

   # Install elasticdump
   npm install -g elasticdump
   
   # Backup all indices data
   elasticdump \
     --input=http://elasticsearch:9200/ \
     --output=/path/to/backup/elasticsearch/data_$(date +%Y%m%d).json \
     --type=data
   
   # Backup index mappings
   elasticdump \
     --input=http://elasticsearch:9200/ \
     --output=/path/to/backup/elasticsearch/mapping_$(date +%Y%m%d).json \
     --type=mapping
   
2. Verify Elasticsearch/OpenSearch Backup

   Ensure the backup contains all expected indices and data:

   # For snapshot API
   curl -X GET "localhost:9200/_snapshot/backup_repository/snapshot_$(date +%Y%m%d)"
   
   # For elasticdump
   jq 'keys' /path/to/backup/elasticsearch/mapping_$(date +%Y%m%d).json
   
3. Schedule Regular Backups

   Create a cron job to automate daily backups:

   # Add to crontab (for elasticdump method)
   0 2 * * * /path/to/elasticsearch-backup-script.sh > /path/to/logs/elasticsearch-backup-$(date +\%Y-\%m-\%d).log 2>&1
   

Redis

1. Configure Redis Persistence

   Ensure Redis is configured for persistence:

   # In redis.conf
   
   # RDB persistence
   save 900 1
   save 300 10
   save 60 10000
   
   # AOF persistence
   appendonly yes
   appendfsync everysec
   
2. Create Redis Backup

   Use the SAVE command or copy the RDB file:

   # Trigger a SAVE operation
   redis-cli -h redis-host -a password SAVE
   
   # Copy the dump.rdb file
   cp /path/to/redis/data/dump.rdb /path/to/backup/redis/dump_$(date +%Y%m%d).rdb
   
3. Schedule Regular Backups

   Create a cron job to automate daily backups:

   # Add to crontab
   0 3 * * * /path/to/redis-backup-script.sh > /path/to/logs/redis-backup-$(date +\%Y-\%m-\%d).log 2>&1
   

S3 Storage

1. Back Up S3 Data

   Use AWS CLI or compatible tools to back up your S3 storage:

   # Sync to another bucket (cross-region for disaster recovery)
   aws s3 sync s3://your-prisme-bucket s3://your-backup-bucket
   
   # Or sync to local storage
   aws s3 sync s3://your-prisme-bucket /path/to/backup/s3-data
   
2. Enable Versioning

   Enable versioning on your S3 bucket for built-in backup protection:

   aws s3api put-bucket-versioning \
     --bucket your-prisme-bucket \
     --versioning-configuration Status=Enabled
   
3. Set Up Lifecycle Policies

   Configure lifecycle policies to manage backup retention:

   aws s3api put-bucket-lifecycle-configuration \
     --bucket your-backup-bucket \
     --lifecycle-configuration file://lifecycle-config.json
   

   Example lifecycle-config.json:

   {
     "Rules": [
       {
         "ID": "Delete old backups",
         "Status": "Enabled",
         "Prefix": "",
         "Expiration": {
           "Days": 90
         }
       }
     ]
   }
   

Configuration Backup

1. Back Up Kubernetes Resources

   Save your Kubernetes configuration resources:

   # Back up all resources in the Prisme namespace
   mkdir -p /path/to/backup/kubernetes/$(date +%Y-%m-%d)
   
   # Export deployments, services, configmaps, secrets, etc.
   kubectl get all,pvc,cm,secret -n prisme-system -o yaml > \
     /path/to/backup/kubernetes/$(date +%Y-%m-%d)/prisme-resources.yaml
   
2. Back Up Helm Values

   Save your Helm chart values for each release:

   # Get values for each Helm release
   helm get values prisme-core -n prisme-system -o yaml > \
     /path/to/backup/helm/$(date +%Y-%m-%d)/prisme-core-values.yaml
   
   helm get values prisme-securechat -n prisme-system -o yaml > \
     /path/to/backup/helm/$(date +%Y-%m-%d)/prisme-securechat-values.yaml
   
   # Repeat for each product module
   
3. Back Up Terraform State

   If using Terraform, back up your state files:

   # If using local state
   cp terraform.tfstate terraform.tfstate.backup
   cp -r terraform.tfstate.d /path/to/backup/terraform/$(date +%Y-%m-%d)
   
   # If using remote state (recommended), ensure your remote backend has proper backup
   

   Using remote state in Terraform (like S3 with versioning or Terraform Cloud) provides built-in backup capabilities.

Restore Procedures

MongoDB Restore

1. Prepare for Restore

   Before restoring, stop services that interact with the database:

   # Scale down Prisme.ai deployments
   kubectl scale deployment -n prisme-system --replicas=0 \
     prisme-api prisme-worker prisme-securechat prisme-knowledge
   
2. Restore MongoDB Data

   Use mongorestore to restore from your backup:

   # Full restore
   mongorestore --uri="mongodb://username:password@hostname:port/database" \
     --nsFrom="database.*" --nsTo="database.*" \
     /path/to/backup/mongo/YYYY-MM-DD
   
   # Or restore specific collections
   mongorestore --uri="mongodb://username:password@hostname:port/database" \
     --nsInclude="database.users" --nsInclude="database.agents" \
     /path/to/backup/mongo/YYYY-MM-DD
   

   Restoring will overwrite existing data. Be sure to validate your backup before proceeding.

3. Restart Services

   After restore is complete, scale the services back up:

   # Scale up Prisme.ai deployments
   kubectl scale deployment -n prisme-system --replicas=1 \
     prisme-api prisme-worker prisme-securechat prisme-knowledge
   
   # Verify pods are running
   kubectl get pods -n prisme-system
   

Elasticsearch/OpenSearch Restore

1. Prepare for Restore

   Stop services that interact with Elasticsearch:

   kubectl scale deployment -n prisme-system --replicas=0 \
     prisme-api prisme-worker
   
2. Restore Data

   Using Snapshot API

   # List available snapshots
   curl -X GET "localhost:9200/_snapshot/backup_repository/_all"
   
   # Restore from snapshot
   curl -X POST "localhost:9200/_snapshot/backup_repository/snapshot_YYYYMMDD/_restore" -H "Content-Type: application/json" -d'
   {
     "indices": "*",
     "ignore_unavailable": true,
     "include_global_state": true
   }'
   
   # Check restore status
   curl -X GET "localhost:9200/_recovery?human"
   

   Using elasticdump

   # Restore mappings first
   elasticdump \
     --input=/path/to/backup/elasticsearch/mapping_YYYYMMDD.json \
     --output=http://elasticsearch:9200/ \
     --type=mapping
   
   # Then restore data
   elasticdump \
     --input=/path/to/backup/elasticsearch/data_YYYYMMDD.json \
     --output=http://elasticsearch:9200/ \
     --type=data
   
3. Restart Services

   After restore is complete, restart the services:

   kubectl scale deployment -n prisme-system --replicas=1 \
     prisme-api prisme-worker
   
   # Verify pods are running
   kubectl get pods -n prisme-system
   

Redis Restore

1. Stop Redis Server

   If possible, gracefully shut down Redis:

   redis-cli -h redis-host -a password SHUTDOWN SAVE
   

   If using Redis in Kubernetes:

   # Scale down Redis deployment
   kubectl scale statefulset -n prisme-system redis --replicas=0
   
2. Replace RDB File

   Copy the backup RDB file to Redis data directory:

   # If using persistent volume in Kubernetes
   kubectl cp /path/to/backup/redis/dump_YYYYMMDD.rdb \
     prisme-system/redis-0:/data/dump.rdb
   
   # For standalone Redis
   cp /path/to/backup/redis/dump_YYYYMMDD.rdb /path/to/redis/data/dump.rdb
   
3. Restart Redis

   Start Redis service with the restored data:

   # For Kubernetes
   kubectl scale statefulset -n prisme-system redis --replicas=1
   
   # For standalone Redis
   systemctl start redis
   
   # Verify Redis is running with restored data
   redis-cli -h redis-host -a password info keyspace
   

S3 Storage Restore

1. Prepare for Restore

   Consider the impact of restoration on your application:

   # Scale down services that interact with object storage
   kubectl scale deployment -n prisme-system --replicas=0 \
     prisme-document-processor prisme-file-handler
   
2. Restore S3 Data

   Sync data from your backup location:

   # From backup bucket
   aws s3 sync s3://your-backup-bucket s3://your-prisme-bucket
   
   # Or from local backup
   aws s3 sync /path/to/backup/s3-data s3://your-prisme-bucket
   

   If using versioning, you can restore specific versions of objects:

   # List versions of a specific object
   aws s3api list-object-versions --bucket your-prisme-bucket --prefix path/to/object
   
   # Restore specific version
   aws s3api copy-object --copy-source your-prisme-bucket/path/to/object?versionId=VERSION_ID \
     --bucket your-prisme-bucket --key path/to/object
   
3. Verify Restoration

   Check that files have been properly restored:

   # List objects in bucket
   aws s3 ls s3://your-prisme-bucket --recursive | head
   
   # Count objects
   aws s3 ls s3://your-prisme-bucket --recursive | wc -l
   
4. Restart Services

   Resume normal operations:

   kubectl scale deployment -n prisme-system --replicas=1 \
     prisme-document-processor prisme-file-handler
   

Configuration Restore

1. Restore Kubernetes Resources

   Apply your backed-up Kubernetes configurations:

   # First, clean up the namespace if necessary
   # WARNING: This will delete all resources! Use with caution.
   # kubectl delete namespace prisme-system
   # kubectl create namespace prisme-system
   
   # Apply backed-up resources
   kubectl apply -f /path/to/backup/kubernetes/YYYY-MM-DD/prisme-resources.yaml
   

   Be cautious when restoring resources. Consider restoring specific resource types instead of everything at once:

   # Restore only ConfigMaps and Secrets first
   kubectl apply -f /path/to/backup/kubernetes/YYYY-MM-DD/configmaps.yaml
   kubectl apply -f /path/to/backup/kubernetes/YYYY-MM-DD/secrets.yaml
   
   # Then restore other resources
   kubectl apply -f /path/to/backup/kubernetes/YYYY-MM-DD/deployments.yaml
   
2. Restore Helm Releases

   Use your backed-up values to reinstall or upgrade Helm releases:

   # Reinstall core
   helm upgrade --install prisme-core prisme/prisme-core \
     -f /path/to/backup/helm/YYYY-MM-DD/prisme-core-values.yaml \
     --namespace prisme-system
   
   # Reinstall product modules
   helm upgrade --install prisme-securechat prisme/prisme-securechat \
     -f /path/to/backup/helm/YYYY-MM-DD/prisme-securechat-values.yaml \
     --namespace prisme-system
   
   # Repeat for other product modules
   
3. Restore Terraform State

   If you need to restore Terraform state:

   # For local state
   cp /path/to/backup/terraform/YYYY-MM-DD/terraform.tfstate .
   cp -r /path/to/backup/terraform/YYYY-MM-DD/terraform.tfstate.d .
   
   # Verify state
   terraform state list
   

   For remote state, follow your backend provider’s restoration process.

Disaster Recovery Planning

Recovery Objectives

Define your recovery objectives to guide your backup strategy:

RPO (Recovery Point Objective)

Maximum acceptable data loss in time:

• Critical data: RPO < 1 hour
• Important data: RPO < 24 hours
• Regular data: RPO < 1 week

RTO (Recovery Time Objective)

Maximum acceptable time to restore service:

• Critical services: RTO < 4 hours
• Important services: RTO < 24 hours
• Regular services: RTO < 3 days

DR Scenarios

Plan for different disaster recovery scenarios:

Single Component Failure

Example: MongoDB database corruption

Recovery Plan:

1. Scale down dependent services
2. Restore MongoDB from most recent backup
3. Verify data integrity
4. Scale up services
5. Monitor application functionality

Entire Platform Failure

Example: Kubernetes cluster compromise

Recovery Plan:

1. Deploy new Kubernetes cluster
2. Restore configuration (secrets, configmaps)
3. Reconnect to database services (if intact)
4. Restore databases if necessary
5. Deploy Helm charts with backed-up values
6. Verify connectivity and functionality

Datacenter/Region Failure

Example: Cloud provider region outage

Recovery Plan:

1. Activate secondary region (if configured)
2. Restore from cross-region backups
3. Update DNS/load balancing to point to new region
4. Verify application functionality
5. Communicate status to users

DR Testing

Regularly test your disaster recovery procedures:

1. Scheduled Testing

   Conduct regular DR tests:

   • Quarterly: Component-level recovery
   • Bi-annually: Full platform recovery
   • Annually: Regional failover (if applicable)
2. Test Documentation

   Document each test with:

   • Date and scope
   • Procedures followed
   • Time to recover
   • Issues encountered
   • Improvements identified
3. Continuous Improvement

   Update procedures based on test results:

   • Optimize recovery scripts
   • Improve automation
   • Address gaps or failures
   • Update documentation

Testing and Validation

1. Verify Backup Integrity

   Regularly test your backups to ensure they can be restored:

   # For MongoDB
   mongorestore --uri="mongodb://username:password@hostname:port/test_db" \
     --nsFrom="database.*" --nsTo="test_db.*" \
     --dryRun /path/to/backup/mongo/YYYY-MM-DD
   
   # For S3
   aws s3 cp s3://your-backup-bucket/sample-file.pdf /tmp/test-restore.pdf
   
2. Validation Checkpoints

   Establish validation points for successful restoration:

   Data Validation

   • Record counts match pre-backup state
   • Sample record content is intact
   • Relationships between data are preserved
   • Application-specific data tests pass

   Functionality Validation

   • Core services start successfully
   • API endpoints respond correctly
   • Authentication and authorization work
   • Data processing functions operate properly
   • UI elements display and function as expected
3. Document Restoration Procedures

   Maintain detailed, tested restoration runbooks:

   • Step-by-step instructions
   • Required credentials and access
   • Validation checkpoints
   • Troubleshooting guidance
   • Contact information for support

Next Steps

Updates

Keep your platform current with the latest updates

Scaling

Scale your platform to meet growing demands