Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supports managing EBS Volumes by ECS services/tasks. #659

Merged
merged 5 commits into from
Jan 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -573,6 +573,61 @@ You can define `serviceConnectConfiguration` in service definition files and `po

For more details, see also [Service Connect parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect.html#service-connect-parameters)

### EBS Volume support

ecspresso supports managing [Amazon EBS Volumes](https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/ebs-volumes.html).

To use EBS volumes, define `volumeConfigurations` in service definitions, and `mountPoints` and `volumes` attributes in task definition files.

```json
// ecs-service-def.json
"volumeConfigurations": [
{
"managedEBSVolume": {
"filesystemType": "ext4",
"roleArn": "arn:aws:iam::123456789012:role/ecsInfrastructureRole",
"sizeInGiB": 10,
"tagSpecifications": [
{
"propagateTags": "SERVICE",
"resourceType": "volume"
}
],
"volumeType": "gp3"
},
"name": "ebs"
}
]
```

```json
// ecs-task-def.json
// containerDefinitions[].mountPoints
"mountPoints": [
{
"containerPath": "/mnt/ebs",
"sourceVolume": "ebs"
}
]
// volumes
"volumes": [
{
"name": "ebs",
"configuredAtLaunch": true
}
]
```

`ecspresso run` command supports EBS volumes too.

The EBS volumes attached to the standalone tasks will be deleted when the task is stopped by default. But you can keep the volumes by `--no-ebs-delete-on-termination` option.

```console
$ ecspresso run --no-ebs-delete-on-termination
```

The EBS volumes attached to the tasks run by ECS services will always be deleted when the task is stopped. This behavior is by the ECS specification, so ecspresso can't change it.

### How to check diff and verify service/task definitions before deploy.

ecspresso supports `diff` and `verify` subcommands.
Expand Down
108 changes: 66 additions & 42 deletions cli_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -416,40 +416,42 @@ var cliTests = []struct {
args: []string{"run"},
sub: "run",
subOption: &ecspresso.RunOption{
DryRun: false,
TaskDefinition: "",
Wait: true,
Count: int32(1),
WatchContainer: "",
PropagateTags: "",
TaskOverrideStr: "",
TaskOverrideFile: "",
SkipTaskDefinition: false,
LatestTaskDefinition: false,
Tags: "",
WaitUntil: "stopped",
Revision: ptr(int64(0)),
ClientToken: nil,
DryRun: false,
TaskDefinition: "",
Wait: true,
Count: int32(1),
WatchContainer: "",
PropagateTags: "",
TaskOverrideStr: "",
TaskOverrideFile: "",
SkipTaskDefinition: false,
LatestTaskDefinition: false,
Tags: "",
WaitUntil: "stopped",
Revision: ptr(int64(0)),
ClientToken: nil,
EBSDeleteOnTermination: ptr(true),
},
},
{
args: []string{"run", "--no-wait", "--dry-run"},
sub: "run",
subOption: &ecspresso.RunOption{
DryRun: true,
TaskDefinition: "",
Wait: false,
Count: int32(1),
WatchContainer: "",
PropagateTags: "",
TaskOverrideStr: "",
TaskOverrideFile: "",
SkipTaskDefinition: false,
LatestTaskDefinition: false,
Tags: "",
WaitUntil: "stopped",
Revision: ptr(int64(0)),
ClientToken: nil,
DryRun: true,
TaskDefinition: "",
Wait: false,
Count: int32(1),
WatchContainer: "",
PropagateTags: "",
TaskOverrideStr: "",
TaskOverrideFile: "",
SkipTaskDefinition: false,
LatestTaskDefinition: false,
Tags: "",
WaitUntil: "stopped",
Revision: ptr(int64(0)),
ClientToken: nil,
EBSDeleteOnTermination: ptr(true),
},
},
{
Expand All @@ -463,20 +465,42 @@ var cliTests = []struct {
},
sub: "run",
subOption: &ecspresso.RunOption{
DryRun: false,
TaskDefinition: "foo.json",
Wait: true,
Count: int32(2),
WatchContainer: "app",
PropagateTags: "SERVICE",
TaskOverrideStr: `{"foo":"bar"}`,
TaskOverrideFile: "overrides.json",
SkipTaskDefinition: false,
LatestTaskDefinition: true,
Tags: "KeyFoo=ValueFoo,KeyBar=ValueBar",
WaitUntil: "running",
Revision: ptr(int64(1)),
ClientToken: ptr("3abb3a41-c4dc-4c16-a3be-aaab729008a0"),
DryRun: false,
TaskDefinition: "foo.json",
Wait: true,
Count: int32(2),
WatchContainer: "app",
PropagateTags: "SERVICE",
TaskOverrideStr: `{"foo":"bar"}`,
TaskOverrideFile: "overrides.json",
SkipTaskDefinition: false,
LatestTaskDefinition: true,
Tags: "KeyFoo=ValueFoo,KeyBar=ValueBar",
WaitUntil: "running",
Revision: ptr(int64(1)),
ClientToken: ptr("3abb3a41-c4dc-4c16-a3be-aaab729008a0"),
EBSDeleteOnTermination: ptr(true),
},
},
{
args: []string{"run", "--no-ebs-delete-on-termination"},
sub: "run",
subOption: &ecspresso.RunOption{
DryRun: false,
TaskDefinition: "",
Wait: true,
Count: int32(1),
WatchContainer: "",
PropagateTags: "",
TaskOverrideStr: "",
TaskOverrideFile: "",
SkipTaskDefinition: false,
LatestTaskDefinition: false,
Tags: "",
WaitUntil: "stopped",
Revision: ptr(int64(0)),
ClientToken: nil,
EBSDeleteOnTermination: ptr(false),
},
},
{
Expand Down
1 change: 1 addition & 0 deletions create.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ func (d *App) createService(ctx context.Context, opt DeployOption) error {
ServiceRegistries: svd.ServiceRegistries,
Tags: svd.Tags,
TaskDefinition: aws.String(tdArn),
VolumeConfigurations: svd.VolumeConfigurations,
}
if _, err := d.ecs.CreateService(ctx, createServiceInput); err != nil {
return fmt.Errorf("failed to create service: %w", err)
Expand Down
1 change: 1 addition & 0 deletions deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,7 @@ func svToUpdateServiceInput(sv *Service) *ecs.UpdateServiceInput {
PropagateTags: sv.PropagateTags,
ServiceConnectConfiguration: sv.ServiceConnectConfiguration,
ServiceRegistries: sv.ServiceRegistries,
VolumeConfigurations: sv.VolumeConfigurations,
}
if sv.SchedulingStrategy == types.SchedulingStrategyDaemon {
in.PlacementStrategy = nil
Expand Down
34 changes: 22 additions & 12 deletions ecspresso.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ func taskDefinitionName(t *TaskDefinition) string {
type Service struct {
types.Service
ServiceConnectConfiguration *types.ServiceConnectConfiguration
VolumeConfigurations []types.ServiceVolumeConfiguration
DesiredCount *int32
}

Expand All @@ -55,16 +56,20 @@ func (d *App) newServiceFromTypes(ctx context.Context, in types.Service) (*Servi
Service: in,
DesiredCount: aws.Int32(in.DesiredCount),
}
for _, dp := range in.Deployments {
d.Log("[DEBUG] deployment: %s %s", *dp.Id, *dp.Status)
if aws.ToString(dp.Status) != "PRIMARY" {
continue
}
scc := dp.ServiceConnectConfiguration
sv.ServiceConnectConfiguration = scc
if scc == nil {
break
}
dps := lo.Filter(in.Deployments, func(dp types.Deployment, i int) bool {
return aws.ToString(dp.Status) == "PRIMARY"
})
if len(dps) == 0 {
d.Log("[WARNING] no primary deployment")
return &sv, nil
}
dp := dps[0]
d.Log("[DEBUG] deployment: %s %s", *dp.Id, *dp.Status)

// ServiceConnect
scc := dp.ServiceConnectConfiguration
sv.ServiceConnectConfiguration = scc
if scc != nil {
// resolve sd namespace arn to name
id := arnToName(aws.ToString(scc.Namespace))
res, err := d.sd.GetNamespace(ctx, &servicediscovery.GetNamespaceInput{
Expand All @@ -74,16 +79,21 @@ func (d *App) newServiceFromTypes(ctx context.Context, in types.Service) (*Servi
var oe *smithy.OperationError
if errors.As(err, &oe) {
d.Log("[WARNING] failed to get namespace: %s", oe)
break
} else {
return nil, fmt.Errorf("failed to get namespace: %w", err)
}
}
if res.Namespace != nil {
scc.Namespace = res.Namespace.Name
}
break
}

// VolumeConfigurations
if dp.VolumeConfigurations != nil {
d.Log("[DEBUG] VolumeConfigurations: %#v", dp.VolumeConfigurations)
sv.VolumeConfigurations = dp.VolumeConfigurations
}

return &sv, nil
}

Expand Down
24 changes: 12 additions & 12 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ require (
github.com/Songmu/prompter v0.5.1
github.com/alecthomas/kong v0.8.1
github.com/aws/aws-sdk-go-v2 v1.24.1
github.com/aws/aws-sdk-go-v2/config v1.26.0
github.com/aws/aws-sdk-go-v2/credentials v1.16.11
github.com/aws/aws-sdk-go-v2/config v1.26.3
github.com/aws/aws-sdk-go-v2/credentials v1.16.14
github.com/aws/aws-sdk-go-v2/service/applicationautoscaling v1.25.4
github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.29.4
github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.31.0
github.com/aws/aws-sdk-go-v2/service/codedeploy v1.22.0
github.com/aws/aws-sdk-go-v2/service/ecr v1.24.4
github.com/aws/aws-sdk-go-v2/service/ecs v1.37.0
Expand All @@ -18,12 +18,12 @@ require (
github.com/aws/aws-sdk-go-v2/service/s3 v1.47.4
github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.25.4
github.com/aws/aws-sdk-go-v2/service/servicediscovery v1.27.4
github.com/aws/aws-sdk-go-v2/service/ssm v1.44.4
github.com/aws/aws-sdk-go-v2/service/sts v1.26.4
github.com/aws/aws-sdk-go-v2/service/ssm v1.44.7
github.com/aws/aws-sdk-go-v2/service/sts v1.26.7
github.com/aws/smithy-go v1.19.0
github.com/fatih/color v1.16.0
github.com/fujiwara/cfn-lookup v1.0.0
github.com/fujiwara/ecsta v0.4.1
github.com/fujiwara/ecsta v0.4.3
github.com/fujiwara/logutils v1.1.2
github.com/fujiwara/tfstate-lookup v1.1.6
github.com/goccy/go-yaml v1.9.5
Expand Down Expand Up @@ -66,20 +66,20 @@ require (
github.com/BurntSushi/toml v1.2.0 // indirect
github.com/Songmu/flextime v0.1.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.4 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.10 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.11 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.15.6 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.7.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.7.2 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.9 // indirect
github.com/aws/aws-sdk-go-v2/service/cloudformation v1.42.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.9 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.9 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.10 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.9 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.26.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.18.4 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.26.7 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.18.6 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.6 // indirect
github.com/creack/pty v1.1.20 // indirect
github.com/dimchansky/utfbom v1.1.1 // indirect
github.com/fujiwara/tracer v1.0.2 // indirect
Expand Down
Loading