Skip to content

Commit aaf8977

Browse files
JosepSampeJosepSampe
authored
[AWS EC2] Add support for configuring EBS volumes in EC2 lithops workers (#1418)
1 parent c616131 commit aaf8977

File tree

7 files changed

+103
-41
lines changed

7 files changed

+103
-41
lines changed

‎CHANGELOG.md

+2
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44

55
### Added
66
- [Core] Added support for python 3.13
7+
- [AWS EC2] Add support for configuring EBS volumes in EC2 lithops workers
8+
- [AWS EC2] Add support for specifying CIDR block in EC2 public subnet
79

810
### Changed
911
-

‎config/README.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -133,12 +133,12 @@ config = {
133133
}
134134
}
135135

136-
def hello_world(name):
137-
return f'Hello {name}!'
136+
def hello_world(number):
137+
return f'Hello {number}!'
138138

139139
if __name__ == '__main__':
140140
fexec = lithops.FunctionExecutor(config=config)
141-
fexec.call_async(hello_world, 'World')
141+
fexec.map(hello_world, [1, 2, 3, 4])
142142
print(fexec.get_result())
143143
```
144144

‎docs/source/compute_config/aws_ec2.md

+46-4
Original file line numberDiff line numberDiff line change
@@ -109,13 +109,14 @@ In summary, you can use one of the following settings:
109109
|---|---|---|---|---|
110110
|aws_ec2 | region | |no | Region name, for example: `eu-west-1`. Lithops will use the `region` set under the `aws` section if it is not set here |
111111
|aws_ec2 | instance_role | | yes | EC2 Instance role name created in the configuration section above. Do not use the full ARN here; only the role name is required. For example: `ec2LithopsInstanceRole`|
112-
|aws_ec2 | vpc_id | | no | VPC id. You can find all the available VPCs in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#vpcs:) |
113-
|aws_ec2 | subnet_id | | no | Subnet id. You can find all the available Subnets in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#subnets:) |
114-
|aws_ec2 | security_group_id | | no | Security group ID. You can find the available security groups in the [VPC console page](https://console.aws.amazon.com/vpc/v2/home#SecurityGroups:). The security group must have ports 22 and 8080 open |
112+
|aws_ec2 | vpc_id | | no | VPC id. You can find all the available VPCs in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#vpcs:). If not provided, Lithops will create a new VPC |
113+
|aws_ec2 | public_subnet_id | | no | Public subnet id. You can find all the available Subnets in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#subnets:). If not provided, Lithops will create a new public subnet |
114+
|aws_ec2 | public_subnet_cidr_block | 10.0.1.0/24 | no | In case a `public_subnet_id` is not provided, Lithops will create a new subnet with this CIDR block |
115+
|aws_ec2 | security_group_id | | no | Security group ID. You can find the available security groups in the [VPC console page](https://console.aws.amazon.com/vpc/v2/home#SecurityGroups:). The security group must have ports 22, 6379, 8080 and 8081 open. If not provided, Lithops will create a new security group |
115116
|aws_ec2 | ssh_key_name | | no | SSH Key name. You can find the available keys in the [EC2 console page](https://console.aws.amazon.com/ec2/v2/home#KeyPairs:). Create a new one or upload your own key if it does not exist|
116117
|aws_ec2 | ssh_username | ubuntu |no | Username to access the VM |
117118
|aws_ec2 | ssh_password | |no | Password for accessing the worker VMs. If not provided, it is created randomly|
118-
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. It will use the default path if not provided |
119+
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. If not provided, Lithops will use the default path and create a new ssh key for the VPC |
119120
|aws_ec2 | request_spot_instances | True | no | Request spot instance for worker VMs|
120121
|aws_ec2 | target_ami | | no | Virtual machine image id. Default is Ubuntu Server 22.04 |
121122
|aws_ec2 | master_instance_type | t2.micro | no | Profile name for the master VM |
@@ -130,6 +131,47 @@ In summary, you can use one of the following settings:
130131
|aws_ec2 | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to **create**, Lithops will automatically create new VMs for each map() call based on the number of elements in iterdata. If set to **reuse** will try to reuse running workers if exist |
131132

132133

134+
## Additional configuration
135+
136+
# Elastic Block Store (EBS)
137+
138+
To attach EBS volumes to an EC2 instance in Lithops, you can configure the `aws_ec2` section as follows.
139+
140+
```yaml
141+
aws_ec2:
142+
execution_role: <EXECUTION_ROLE_ARN>
143+
region: <REGION_NAME>
144+
...
145+
ebs_volumes:
146+
- device_name: /dev/xvda
147+
ebs:
148+
volume_size: 100
149+
volume_type: gp2
150+
delete_on_termination: true
151+
encrypted: false
152+
kms_key_id: <KMS_KEY_ARN>
153+
- device_name: /dev/xvdf
154+
ebs:
155+
volume_size: 50
156+
volume_type: gp3
157+
delete_on_termination: true
158+
encrypted: false
159+
iops: 3000
160+
throughput: 125
161+
...
162+
```
163+
164+
|Group|Key|Default|Mandatory|Additional info|
165+
|---|---|---|---|---|
166+
| ebs | volume_size | 8 | No | Size of the volume in GiB |
167+
| ebs | volume_type | gp2 | No | Type of volume. Options: `gp2`, `gp3`, `io1`, `io2`, `sc1`, `st1`, `standard`|
168+
| ebs | delete_on_termination| True | No | Whether the volume is deleted automatically when the instance is terminated |
169+
| ebs | encrypted | False | No | Whether the volume is encrypted |
170+
| ebs | kms_key_i | | No | ARN of the KMS key used for encryption. If not provided, the default AWS-managed key is used |
171+
| ebs | iops | | No | Provisioned IOPS for `io1`, `io2`, or `gp3` volumes |
172+
| ebs | throughput | | No | Throughput in MiB/s for `gp3` volumes |
173+
174+
133175
## Consume mode
134176

135177
In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the `create/reuse` modes, since the entire job is executed in the same VM.

‎docs/source/configuration.rst

+24-16
Original file line numberDiff line numberDiff line change
@@ -71,22 +71,30 @@ Here is an example of providing configuration keys for IBM Cloud Functions and I
7171

7272
.. code:: python
7373
74-
import lithops
75-
76-
77-
config = {'lithops': {'backend': 'ibm_cf', 'storage': 'ibm_cos'},
78-
'ibm': {'region': 'REGION',
79-
'iam_api_key': 'IAM_API_KEY',
80-
'resource_group_id': 'RESOURCE_GROUP_ID'}
81-
'ibm_cos': {'storage_bucket': 'STORAGE_BUCKET'}}
82-
83-
def hello_world(name):
84-
return 'Hello {}!'.format(name)
85-
86-
if __name__ == '__main__':
87-
fexec = lithops.FunctionExecutor(config=config)
88-
fexec.call_async(hello_world, 'World')
89-
print(fexec.get_result())
74+
import lithops
75+
76+
config = {
77+
'lithops': {
78+
'backend': 'code_engine',
79+
'storage': 'ibm_cos'
80+
},
81+
'ibm': {
82+
'region': 'REGION',
83+
'iam_api_key': 'IAM_API_KEY',
84+
'resource_group_id': 'RESOURCE_GROUP_ID'
85+
},
86+
'ibm_cos': {
87+
'storage_bucket': 'STORAGE_BUCKET'
88+
}
89+
}
90+
91+
def hello_world(number):
92+
return f'Hello {number}!'
93+
94+
if __name__ == '__main__':
95+
fexec = lithops.FunctionExecutor(config=config)
96+
fexec.map(hello_world, [1, 2, 3, 4])
97+
print(fexec.get_result())
9098
9199
92100
.. _config-reference-label:

‎docs/source/lithops_config_keys.csv

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Group;Key;Default;Mandatory;Additional info
2-
lithops;backend;``ibm_cf``;no;Compute backend implementation. IBM Cloud Functions is the default. If not set, Lithops will check the `mode` and use the `backend` set under the `serverless` or `standalone` sections described below.
3-
lithops;storage;``ibm_cos``;no;Storage backend implementation. IBM Cloud Object Storage is the default.
2+
lithops;backend;``aws_lambda``;no;Compute backend implementation. AWS Lambda is the default.
3+
lithops;storage;``aws_s3``;no;Storage backend implementation. AWS S3 is the default.
44
lithops;data_cleaner;``True``;no;If set to True, then the cleaner will automatically delete all the temporary data that was written into `storage_bucket/lithops.jobs`.
55
lithops;monitoring;``storage``;no;Monitoring system implementation. One of: **storage** or **rabbitmq**.
66
lithops;monitoring_interval;``2``;no;Monitoring check interval in seconds in case of **storage** monitoring.

‎lithops/standalone/backends/aws_ec2/aws_ec2.py

+24-15
Original file line numberDiff line numberDiff line change
@@ -182,7 +182,8 @@ def _create_subnets(self):
182182
if 'public_subnet_id' not in self.config:
183183
logger.debug(f'Creating new public subnet in VPC {self.vpc_name}')
184184
response = self.ec2_client.create_subnet(
185-
CidrBlock='10.0.1.0/24', VpcId=self.config['vpc_id'],
185+
CidrBlock=self.config['public_subnet_cidr_block'],
186+
VpcId=self.config['vpc_id'],
186187
)
187188
public_subnet_id = response['Subnet']['SubnetId']
188189
self.config['public_subnet_id'] = public_subnet_id
@@ -1157,20 +1158,28 @@ def _create_instance(self, user_data=None):
11571158
"""
11581159
Creates a new VM instance
11591160
"""
1160-
if self.fast_io:
1161-
BlockDeviceMappings = [
1162-
{
1163-
'DeviceName': '/dev/xvda',
1164-
'Ebs': {
1165-
'VolumeSize': 100,
1166-
'DeleteOnTermination': True,
1167-
'VolumeType': 'gp2',
1168-
# 'Iops' : 10000,
1169-
},
1170-
},
1171-
]
1172-
else:
1173-
BlockDeviceMappings = None
1161+
ebs_volumes = self.config.get('ebs_volumes', [])
1162+
BlockDeviceMappings = []
1163+
1164+
for volume in ebs_volumes:
1165+
ebs_config = volume.get('ebs', {})
1166+
block_device = {
1167+
'DeviceName': volume['device_name'],
1168+
'Ebs': {
1169+
'VolumeSize': ebs_config.get('volume_size', 8), # Default 8 GiB
1170+
'VolumeType': ebs_config.get('volume_type', 'gp2'),
1171+
'DeleteOnTermination': ebs_config.get('delete_on_termination', True),
1172+
'Encrypted': ebs_config.get('encrypted', False),
1173+
}
1174+
}
1175+
if 'iops' in ebs_config:
1176+
block_device['Ebs']['Iops'] = ebs_config['iops']
1177+
if 'throughput' in ebs_config:
1178+
block_device['Ebs']['Throughput'] = ebs_config['throughput']
1179+
if 'kms_key_id' in ebs_config:
1180+
block_device['Ebs']['KmsKeyId'] = ebs_config['kms_key_id']
1181+
1182+
BlockDeviceMappings.append(block_device)
11741183

11751184
LaunchSpecification = {
11761185
"ImageId": self.config['target_ami'],

‎lithops/standalone/backends/aws_ec2/config.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@
2828
'request_spot_instances': True,
2929
'delete_on_dismantle': True,
3030
'max_workers': 100,
31-
'worker_processes': 'AUTO'
31+
'worker_processes': 'AUTO',
32+
'public_subnet_cidr_block': '10.0.1.0/24'
3233
}
3334

3435
REQ_PARAMS_1 = ('instance_id',)

0 commit comments

Comments
 (0)