Rolling / gradual / sequential change example for AWS EC2, aws-py-ec2-rolling
Open
@pgavlin is already working on this.
Since Apr 30, 2020.
- Dominant language
- TypeScript
- Stars
- 2.6k
- Forks
- 894
- Avg merge
- 12h 19m
- Merged PRs (30d)
- 39
Description
Hello and thank you for pulumi,
I am new to pulumi (and terraform) and am trying pulumi out with a workflow I could not solve with terraform.
The workflow is for deploying an AWS EC2 instance type change over multiple EC2 instances in a rolling manner (without ASG), in my case this is for some clustered stacks where only one node at a time should be down. Using pulumi I am able to daisy chain the nodes in a cluster with WaitForResource between each EC2 node, the WaitForProvider used always returns changes=True for diff and the create and update methods always call a wait method, in the example below the wait method is waiting for TCP port 22 to accept a connection but this would be a check for the cluster service port and response , the links in the chain are depends_on ResourceOptions.
I am wondering if there is a better (simpler/easier) way to accomplish this?
__main__.py
import sys
import time
import socket
from datetime import datetime
from typing import Any, Optional
import pulumi
from pulumi import Output, ResourceOptions
from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult, DiffResult
from pulumi.log import info
import pulumi_aws as aws
size = 't2.micro'
ami_id = '' # fill in
#vpc_id = ''
#subnet_id = ''
num_nodes = 3
class WaitForProvider(ResourceProvider):
def wait(self, props):
hostname = props['hostname']
ip = props['ip']
port = props['port']
info("%s WaitForProvider::wait for hostname=%s ip=%s port=%d" % (datetime.now(), hostname, ip, port) )
timeout = 300
start_time = time.perf_counter()
while True:
try:
with socket.create_connection((ip, int(port)), timeout=timeout):
break
except OSError as ex:
time.sleep(0.5)
if ex:
s_ex = "(%s)" % ex
if time.perf_counter() - start_time >= timeout:
raise TimeoutError('Waited too long for the port %d on host %s (%s) to start accepting '
'connections %s' % (port, hostname, ip, s_ex)) from ex
info("%s WaitForProvider::wait done, waited %.2f seconds" % (datetime.now(),(time.perf_counter() - start_time)) )
def diff(self, _id: str, _olds: Any, _news: Any) -> DiffResult:
return DiffResult( changes = True )
def create(self, props):
info("%s WaitForProvider::create %s" % (datetime.now(), name) )
self.wait(props)
return CreateResult(id_=name, outs={})
def update(self, _id: str, _olds: Any, _news: Any) -> UpdateResult:
info("%s WaitForProvider::update %s" % (datetime.now(),_id) )
self.wait({
'hostname': _news['hostname'],
'ip': _news['ip'],
'port': _news['port']
})
return UpdateResult()
class WaitForResource(Resource):
def __init__(self, name: str, props: Any, opts: Optional[ResourceOptions] = None):
super().__init__(WaitForProvider(), name, props, opts)
group = aws.ec2.SecurityGroup('rolling-secgrp',
# vpc_id = vpc_id,
description = 'Enable SSH access',
name = 'rolling-secgrp',
ingress=[
{ 'protocol': 'tcp', 'from_port': 22, 'to_port': 22, 'cidr_blocks': ['172.16.0.0/12'], 'self': True }
]
)
previous_wait_for_list = []
for i in range(num_nodes):
name = 'node%02d' % (i+1)
server = aws.ec2.Instance(
name,
opts = pulumi.resource.ResourceOptions(
depends_on = previous_wait_for_list
),
tags = {
'Name': name
},
instance_type = size,
vpc_security_group_ids = [group.id],
# subnet_id = subnet_id,
ami = ami_id
)
wait_for_name = 'WaitFor-%s' % (name)
wfr = WaitForResource(
wait_for_name,
{
'hostname': name,
'ip': server.private_ip,
'port': 22
},
opts = pulumi.resource.ResourceOptions(
depends_on = [ server ]
)
)
previous_wait_for_list = [ wfr ]
> pulumi up
Previewing update (rolling-dev):
Type Name Plan
+ pulumi:pulumi:Stack rolling-rolling-dev create
+ ├─ aws:ec2:SecurityGroup rolling-secgrp create
+ ├─ aws:ec2:Instance node01 create
+ ├─ pulumi-python:dynamic:Resource WaitFor-node01 create
+ ├─ aws:ec2:Instance node02 create
+ ├─ pulumi-python:dynamic:Resource WaitFor-node02 create
+ ├─ aws:ec2:Instance node03 create
+ └─ pulumi-python:dynamic:Resource WaitFor-node03 create
Updating (rolling-dev):
Type Name Status Info
+ pulumi:pulumi:Stack rolling-rolling-dev created 9 messages
+ ├─ aws:ec2:SecurityGroup rolling-secgrp created
+ ├─ aws:ec2:Instance node01 created
+ ├─ pulumi-python:dynamic:Resource WaitFor-node01 created
+ ├─ aws:ec2:Instance node02 created
+ ├─ pulumi-python:dynamic:Resource WaitFor-node02 created
+ ├─ aws:ec2:Instance node03 created
+ └─ pulumi-python:dynamic:Resource WaitFor-node03 created
Diagnostics:
pulumi:pulumi:Stack (rolling-rolling-dev):
info: 2020-04-30 05:56:04.267053 WaitForProvider::create node01
info: 2020-04-30 05:56:04.267116 WaitForProvider::wait for hostname=node01 ip=172.16.0.1 port=22
info: 2020-04-30 05:56:35.328949 WaitForProvider::wait done, waited 31.06 seconds
info: 2020-04-30 05:57:07.370874 WaitForProvider::create node02
info: 2020-04-30 05:57:07.370911 WaitForProvider::wait for hostname=node02 ip=172.16.0.2 port=22
info: 2020-04-30 05:57:25.973276 WaitForProvider::wait done, waited 18.60 seconds
info: 2020-04-30 05:57:47.657190 WaitForProvider::create node03
info: 2020-04-30 05:57:47.657227 WaitForProvider::wait for hostname=node03 ip=172.16.0.3 port=22
info: 2020-04-30 05:58:02.689045 WaitForProvider::wait done, waited 15.03 seconds
Duration: 2m38s
pulumi up after changing t2.micro to t3.micro:
Previewing update (rolling-dev):
Type Name Plan Info
pulumi:pulumi:Stack rolling-rolling-dev
~ ├─ aws:ec2:Instance node01 update [diff: ~instanceType]
~ ├─ pulumi-python:dynamic:Resource WaitFor-node01 update [diff: ~ip]
~ ├─ aws:ec2:Instance node02 update [diff: ~instanceType]
~ ├─ pulumi-python:dynamic:Resource WaitFor-node02 update [diff: ~ip]
~ ├─ aws:ec2:Instance node03 update [diff: ~instanceType]
~ └─ pulumi-python:dynamic:Resource WaitFor-node03 update [diff: ~ip]
Updating (rolling-dev):
Type Name Status Info
pulumi:pulumi:Stack rolling-rolling-dev 9 messages
~ ├─ aws:ec2:Instance node01 updated [diff: ~instanceType]
~ ├─ pulumi-python:dynamic:Resource WaitFor-node01 updated
~ ├─ aws:ec2:Instance node02 updated [diff: ~instanceType]
~ ├─ pulumi-python:dynamic:Resource WaitFor-node02 updated
~ ├─ aws:ec2:Instance node03 updated [diff: ~instanceType]
~ └─ pulumi-python:dynamic:Resource WaitFor-node03 updated
Diagnostics:
pulumi:pulumi:Stack (rolling-rolling-dev):
info: 2020-04-30 07:17:33.857615 WaitForProvider::update node01
info: 2020-04-30 07:17:33.857696 WaitForProvider::wait for hostname=node01 ip=172.16.0.1 port=22
info: 2020-04-30 07:17:48.896784 WaitForProvider::wait done, waited 15.04 seconds
info: 2020-04-30 07:19:21.139589 WaitForProvider::update node02
info: 2020-04-30 07:19:21.139654 WaitForProvider::wait for hostname=node02 ip=172.16.0.2 port=22
info: 2020-04-30 07:19:23.702064 WaitForProvider::wait done, waited 2.56 seconds
info: 2020-04-30 07:20:46.527078 WaitForProvider::update node03
info: 2020-04-30 07:20:46.527116 WaitForProvider::wait for hostname=node03 ip=172.16.0.3 port=22
info: 2020-04-30 07:20:48.069626 WaitForProvider::wait done, waited 1.54 seconds
Duration: 5m10s
stack graph
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.