Skip to content

Refreshing ECS Container Instance

This guide explains how to refresh an ECS container instance when it becomes disconnected from the ECS service. This is different from restarting containers - this addresses issues with the underlying EC2 instance that hosts the containers.

When to Refresh Container Instance

You may need to refresh the ECS container instance in the following situations: - Container instance shows as "DISCONNECTED" in the ECS console - Tasks fail to start with capacity or connectivity issues - Container instance appears unhealthy or unresponsive - ECS agent has stopped communicating with the ECS service - After network connectivity issues or EC2 instance problems

Understanding the Difference

Container Restart vs Container Instance Refresh: - Container Restart: Stops and starts application containers (see Restarting ECS Containers) - Container Instance Refresh: Restarts the underlying EC2 instance that hosts the containers

Prerequisites

  • Access to AWS Console with ECS and EC2 permissions
  • Understanding that this will cause downtime for all running containers on the instance
  • Coordination with team members if this affects production services

Step-by-Step Process

1. Identify the Disconnected Container Instance

  1. Access ECS Console: Navigate to AWS Console → ECS
  2. Select Cluster: Click on Odoo_ECS_Cluster
  3. Container Instances Tab: Click on the "Container instances" tab
  4. Check Status: Look for instances with status:
  5. DISCONNECTED - Instance is not communicating with ECS
  6. INACTIVE - Instance is not accepting new tasks
  7. UNHEALTHY - Instance is experiencing issues

2. Note Running Tasks (Important)

Before refreshing the instance: 1. Check Running Tasks: Note which tasks are currently running on the instance 2. Service Impact: Understand which services will be affected 3. Backup Plan: Ensure other instances can handle the workload if needed

3. Access the EC2 Instance

  1. Get Instance ID: In the container instance details, note the EC2 instance ID
  2. Navigate to EC2: Go to AWS Console → EC2
  3. Find Instance: In the EC2 dashboard, locate the instance by its ID
  4. Verify Instance: Confirm this is the correct instance by checking:
  5. Instance ID matches
  6. Instance type and configuration

4. Stop the EC2 Instance

  1. Select Instance: Check the box next to the EC2 instance
  2. Instance State: Click "Instance state" dropdown
  3. Stop Instance: Select "Stop instance"
  4. Confirm: Confirm the stop action when prompted
  5. Wait for Stop: Wait for the instance state to change to "Stopped"

5. Start the EC2 Instance

  1. Select Stopped Instance: Ensure the instance is selected
  2. Instance State: Click "Instance state" dropdown
  3. Start Instance: Select "Start instance"
  4. Monitor Startup: Watch the instance state change to "Running"
  5. Check Status Checks: Wait for both system and instance status checks to pass

6. Verify ECS Agent Reconnection

  1. Return to ECS Console: Navigate back to ECS → Odoo_ECS_Cluster
  2. Container Instances: Check the container instances tab
  3. Verify Status: Confirm the instance status is now:
  4. ACTIVE - Instance is healthy and accepting tasks
  5. Connected and communicating with ECS service
  6. Check Agent Version: Verify ECS agent is running the correct version

7. Monitor Task Restart

  1. Automatic Task Restart: ECS will automatically restart tasks that were running
  2. Check Services: Go to the Services tab and verify odoo-http service
  3. Monitor Tasks: Watch for new tasks to start and reach RUNNING status
  4. Verify Health: Ensure application health checks pass

Troubleshooting

If Instance Won't Start

  1. Check EC2 Limits: Verify you haven't hit EC2 instance limits
  2. Review Security Groups: Ensure security groups allow proper communication
  3. Check Subnets: Verify subnet has available IP addresses
  4. Instance Health: Check if the instance has underlying health issues

If ECS Agent Won't Connect

  1. Check IAM Roles: Verify the instance has proper ECS permissions
  2. Network Connectivity: Ensure the instance can reach ECS endpoints
  3. ECS Agent Logs: Check EC2 instance logs for ECS agent errors
  4. Agent Version: Verify ECS agent is up to date

If Tasks Don't Restart

  1. Service Configuration: Check if service has desired task count set
  2. Resource Limits: Verify instance has enough CPU/memory
  3. Task Definition: Ensure task definition is valid
  4. Placement Constraints: Check if any constraints prevent task placement

Alternative Methods

Using AWS CLI

# Stop EC2 instance
aws ec2 stop-instances --instance-ids i-1234567890abcdef0

# Wait for instance to stop
aws ec2 wait instance-stopped --instance-ids i-1234567890abcdef0

# Start EC2 instance
aws ec2 start-instances --instance-ids i-1234567890abcdef0

# Wait for instance to be running
aws ec2 wait instance-running --instance-ids i-1234567890abcdef0

Quick Reference