Emergency Hotfix Deployment

Deploy critical hotfix: $ARGUMENTS

Current Production State

• Current version: !git describe --tags --abbrev=0 2>/dev/null || echo "No tags found"
• Production branch: !git branch --show-current
• Recent commits: !git log --oneline -5
• Deployment status: !curl -s https://api.example.com/health 2>/dev/null | jq -r '.version // "Unknown"' || echo "Health check failed"
• Staging environment: Check for staging deployment capabilities

Emergency Response Protocol

Execute emergency hotfix deployment: $ARGUMENTS

• Emergency Assessment and Triage

- Assess the severity and impact of the issue

- Determine if a hotfix is necessary or if it can wait

- Identify affected systems and user impact

- Estimate time sensitivity and business impact

- Document the incident and decision rationale

• Incident Response Setup

- Create incident tracking in your incident management system

- Set up war room or communication channel

- Notify stakeholders and on-call team members

- Establish clear communication protocols

- Document initial incident details and timeline

• Branch and Environment Setup

# Create hotfix branch from production tag
   git fetch --tags
   git checkout tags/v1.2.3  # Latest production version
   git checkout -b hotfix/critical-auth-fix
   
   # Alternative: Branch from main if using trunk-based development
   git checkout main
   git pull origin main
   git checkout -b hotfix/critical-auth-fix

• Rapid Development Process

- Keep changes minimal and focused on the critical issue only

- Avoid refactoring, optimization, or unrelated improvements

- Use well-tested patterns and established approaches

- Add minimal logging for troubleshooting purposes

- Follow existing code conventions and patterns

• Accelerated Testing

# Run focused tests related to the fix
   npm test -- --testPathPattern=auth
   npm run test:security
   
   # Manual testing checklist
   # [ ] Core functionality works correctly
   # [ ] Hotfix resolves the critical issue
   # [ ] No new issues introduced
   # [ ] Critical user flows remain functional

• Fast-Track Code Review

- Get expedited review from senior team member

- Focus review on security and correctness

- Use pair programming if available and time permits

- Document review decisions and rationale quickly

- Ensure proper approval process even under time pressure

• Version and Tagging

# Update version for hotfix
   # 1.2.3 -> 1.2.4 (patch version)
   # or 1.2.3 -> 1.2.3-hotfix.1 (hotfix identifier)
   
   # Commit with detailed message
   git add .
   git commit -m "hotfix: fix critical authentication vulnerability
   
   - Fix password validation logic
   - Resolve security issue allowing bypass
   - Minimal change to reduce deployment risk
   
   Fixes: #1234"
   
   # Tag the hotfix version
   git tag -a v1.2.4 -m "Hotfix v1.2.4: Critical auth security fix"
   git push origin hotfix/critical-auth-fix
   git push origin v1.2.4

• Staging Deployment and Validation

# Deploy to staging environment for final validation
   ./deploy-staging.sh v1.2.4
   
   # Critical path testing
   curl -X POST staging.example.com/api/auth/login \
        -H "Content-Type: application/json" \
        -d '{"email":"test@example.com","password":"testpass"}'
   
   # Run smoke tests
   npm run test:smoke:staging

• Production Deployment Strategy

Blue-Green Deployment:

# Deploy to blue environment
   ./deploy-blue.sh v1.2.4
   
   # Validate blue environment health
   ./health-check-blue.sh
   
   # Switch traffic to blue environment
   ./switch-to-blue.sh
   
   # Monitor deployment metrics
   ./monitor-deployment.sh

Rolling Deployment:

# Deploy to subset of servers first
   ./deploy-rolling.sh v1.2.4 --batch-size 1
   
   # Monitor each batch deployment
   ./monitor-batch.sh
   
   # Continue with next batch if healthy
   ./deploy-next-batch.sh

• Pre-Deployment Checklist

# Verify all prerequisites are met
    # [ ] Database backup completed successfully
    # [ ] Rollback plan documented and ready
    # [ ] Monitoring alerts configured and active
    # [ ] Team members standing by for support
    # [ ] Communication channels established
    
    # Execute production deployment
    ./deploy-production.sh v1.2.4
    
    # Run immediate post-deployment validation
    ./validate-hotfix.sh

• Real-Time Monitoring

# Monitor key application metrics
    watch -n 10 'curl -s https://api.example.com/health | jq .'
    
    # Monitor error rates and logs
    tail -f /var/log/app/error.log | grep -i "auth"
    
    # Track critical metrics:
    # - Response times and latency
    # - Error rates and exception counts
    # - User authentication success rates
    # - System resource usage (CPU, memory)

• Post-Deployment Validation

# Run comprehensive validation tests
    ./test-critical-paths.sh
    
    # Test user authentication functionality
    curl -X POST https://api.example.com/auth/login \
         -H "Content-Type: application/json" \
         -d '{"email":"test@example.com","password":"testpass"}'
    
    # Validate security fix effectiveness
    ./security-validation.sh
    
    # Check overall system performance
    ./performance-check.sh

• Communication and Status Updates

- Provide regular status updates to stakeholders

- Use consistent communication channels

- Document deployment progress and results

- Update incident tracking systems

- Notify relevant teams of deployment completion

• Rollback Procedures

# Automated rollback script
    #!/bin/bash
    PREVIOUS_VERSION="v1.2.3"
    
    if [ "$1" = "rollback" ]; then
        echo "Rolling back to $PREVIOUS_VERSION"
        ./deploy-production.sh $PREVIOUS_VERSION
        ./validate-rollback.sh
        echo "Rollback completed successfully"
    fi
    
    # Manual rollback steps if automation fails:
    # 1. Switch load balancer back to previous version
    # 2. Validate previous version health and functionality
    # 3. Monitor system stability after rollback
    # 4. Communicate rollback status to team

• Post-Deployment Monitoring Period

- Monitor system for 2-4 hours after deployment

- Watch error rates and performance metrics closely

- Check user feedback and support ticket volume

- Validate that the hotfix resolves the original issue

- Document any issues or unexpected behaviors

• Documentation and Incident Reporting

- Document the complete hotfix process and timeline

- Record lessons learned and process improvements

- Update incident management systems with resolution

- Create post-incident review materials

- Share knowledge with team for future reference

• Merge Back to Main Branch

# After successful hotfix deployment and validation
    git checkout main
    git pull origin main
    git merge hotfix/critical-auth-fix
    git push origin main
    
    # Clean up hotfix branch
    git branch -d hotfix/critical-auth-fix
    git push origin --delete hotfix/critical-auth-fix

• Post-Incident Activities

- Schedule and conduct post-incident review meeting

- Update runbooks and emergency procedures

- Identify and implement process improvements

- Update monitoring and alerting configurations

- Plan preventive measures to avoid similar issues

Hotfix Best Practices:

• Keep It Simple: Make minimal changes focused only on the critical issue
• Test Thoroughly: Maintain testing standards even under time pressure
• Communicate Clearly: Keep all stakeholders informed throughout the process
• Monitor Closely: Watch the fix carefully in production environment
• Document Everything: Record all decisions and actions for post-incident review
• Plan for Rollback: Always have a tested way to revert changes quickly
• Learn and Improve: Use each incident to strengthen processes and procedures

Emergency Escalation Guidelines:

# Emergency contact information
SENIOR_ENGINEER="+1-555-0124"
ENGINEERING_MANAGER="+1-555-0125"
INCIDENT_COMMANDER="+1-555-0126"

# Escalation timeline thresholds:
# 15 minutes: Escalate to senior engineer
# 30 minutes: Escalate to engineering manager
# 60 minutes: Escalate to incident commander

Important Reminders:

• Hotfixes should only be used for genuine production emergencies
• When in doubt about severity, follow the normal release process
• Always prioritize system stability over speed of deployment
• Maintain clear audit trails for all emergency changes
• Regular drills help ensure team readiness for real emergencies