Skip to content

Commit 9a27cee

Browse files
JosepSampeJosepSampe
committed
update to 3.4.1
1 parent f2f5a62 commit 9a27cee

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

68 files changed

+463
-459
lines changed

Diff for: ‎docs/_sources/source/compute_config/aws_ec2.md

+66-66
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# AWS Elastic Compute Cloud (EC2)
22

3-
The AWS EC2 client of Lithops can provide a truely serverless user experience on top of EC2 where Lithops creates new Virtual Machines (VMs) dynamically in runtime and scale Lithops jobs against them. Alternatively Lithops can start and stop an existing VM instances.
3+
The AWS EC2 client of Lithops can provide a truely serverless user experience on top of EC2 where Lithops creates new Virtual Machines (VMs) dynamically in runtime and scale Lithops jobs against them (Create & Reuse modes). Alternatively Lithops can start and stop an existing VM instance (Consume mode).
44

55
## AWS
66
The assumption that you already familiar with AWS, and you have AUTH credentials to your account (HMAC Credentials).
@@ -20,71 +20,7 @@ Any Virtual Machine (VM) need to define the instance’s operating system and ve
2020
python3 -m pip install lithops[aws]
2121
```
2222

23-
## Lithops Consume mode
24-
25-
In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the `create/reuse` modes, since the entire job is executed in the same VM.
26-
27-
### AWS Credential setup
28-
29-
Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).
30-
31-
In summary, you can use one of the following settings:
32-
33-
1. Provide the credentials via the `~/.aws/config` file, or set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
34-
35-
You can run `aws configure` command if the AWS CLI is installed to setup the credentials. Then set in the Lithops config file:
36-
```yaml
37-
lithops:
38-
backend: aws_ec2
39-
40-
aws_ec2:
41-
region : <REGION_NAME>
42-
exec_mode: consume
43-
instance_id : <INSTANCE ID OF THE VM>
44-
```
45-
46-
2. Provide the credentials in the `aws` section of the Lithops config file:
47-
```yaml
48-
lithops:
49-
backend: aws_ec2
50-
51-
aws:
52-
access_key_id: <AWS_ACCESS_KEY_ID>
53-
secret_access_key: <AWS_SECRET_ACCESS_KEY>
54-
region: <REGION_NAME>
55-
56-
aws_ec2:
57-
exec_mode: consume
58-
instance_id : <INSTANCE ID OF THE VM>
59-
```
60-
61-
62-
### Summary of configuration keys for AWS
63-
64-
|Group|Key|Default|Mandatory|Additional info|
65-
|---|---|---|---|---|
66-
|aws | region | |no | AWS Region. For example `us-east-1` |
67-
|aws | access_key_id | |no | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
68-
|aws | secret_access_key | |no | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
69-
|aws | session_token | |no | Session token for temporary AWS credentials |
70-
|aws | account_id | |no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. |
71-
72-
### Summary of configuration keys for the consume Mode
73-
74-
|Group|Key|Default|Mandatory|Additional info|
75-
|---|---|---|---|---|
76-
|aws_ec2 | instance_id | | yes | virtual server instance ID |
77-
|aws_ec2 | region | |yes | Region name of the VPC. For example `us-east-1`. Lithops will use the region set under the `aws` section if it is not set here |
78-
|aws_ec2 | ssh_username | ubuntu |no | Username to access the VM |
79-
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to create the VM. It will use the default path if not provided |
80-
|aws_ec2 | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the VM|
81-
|aws_ec2 | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the defeuv python3 interpreter of the VM |
82-
|aws_ec2 | auto_dismantle | True |no | If False then the VM is not stopped automatically.|
83-
|aws_ec2 | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
84-
|aws_ec2 | hard_dismantle_timeout | 3600 | no | Time in seconds to stop the VM instance after a job **started** its execution |
85-
86-
87-
## Lithops create and reuse modes
23+
## Create and reuse modes
8824
In the `create` mode, Lithops will automatically create new worker VM instances in runtime, scale Lithops job against generated VMs, and automatically delete the VMs when the job is completed.
8925
Alternatively, you can set the `reuse` mode to keep running the started worker VMs, and reuse them for further executions. In the `reuse` mode, Lithops checks all the available worker VMs and start new workers only if necessary.
9026

@@ -194,6 +130,70 @@ In summary, you can use one of the following settings:
194130
|aws_ec2 | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to **create**, Lithops will automatically create new VMs for each map() call based on the number of elements in iterdata. If set to **reuse** will try to reuse running workers if exist |
195131

196132

133+
## Consume mode
134+
135+
In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the `create/reuse` modes, since the entire job is executed in the same VM.
136+
137+
### AWS Credential setup
138+
139+
Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).
140+
141+
In summary, you can use one of the following settings:
142+
143+
1. Provide the credentials via the `~/.aws/config` file, or set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
144+
145+
You can run `aws configure` command if the AWS CLI is installed to setup the credentials. Then set in the Lithops config file:
146+
```yaml
147+
lithops:
148+
backend: aws_ec2
149+
150+
aws_ec2:
151+
region : <REGION_NAME>
152+
exec_mode: consume
153+
instance_id : <INSTANCE ID OF THE VM>
154+
```
155+
156+
2. Provide the credentials in the `aws` section of the Lithops config file:
157+
```yaml
158+
lithops:
159+
backend: aws_ec2
160+
161+
aws:
162+
access_key_id: <AWS_ACCESS_KEY_ID>
163+
secret_access_key: <AWS_SECRET_ACCESS_KEY>
164+
region: <REGION_NAME>
165+
166+
aws_ec2:
167+
exec_mode: consume
168+
instance_id : <INSTANCE ID OF THE VM>
169+
```
170+
171+
172+
### Summary of configuration keys for AWS
173+
174+
|Group|Key|Default|Mandatory|Additional info|
175+
|---|---|---|---|---|
176+
|aws | region | |no | AWS Region. For example `us-east-1` |
177+
|aws | access_key_id | |no | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
178+
|aws | secret_access_key | |no | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
179+
|aws | session_token | |no | Session token for temporary AWS credentials |
180+
|aws | account_id | |no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. |
181+
182+
### Summary of configuration keys for the consume Mode
183+
184+
|Group|Key|Default|Mandatory|Additional info|
185+
|---|---|---|---|---|
186+
|aws_ec2 | instance_id | | yes | virtual server instance ID |
187+
|aws_ec2 | region | |yes | Region name of the VPC. For example `us-east-1`. Lithops will use the region set under the `aws` section if it is not set here |
188+
|aws_ec2 | ssh_username | ubuntu |no | Username to access the VM |
189+
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to create the VM. It will use the default path if not provided |
190+
|aws_ec2 | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the VM|
191+
|aws_ec2 | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the defeuv python3 interpreter of the VM |
192+
|aws_ec2 | auto_dismantle | True |no | If False then the VM is not stopped automatically.|
193+
|aws_ec2 | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
194+
|aws_ec2 | hard_dismantle_timeout | 3600 | no | Time in seconds to stop the VM instance after a job **started** its execution |
195+
196+
197197
## Test Lithops
198198
Once you have your compute and storage backends configured, you can run a hello world function with:
199199

Diff for: ‎docs/_sources/source/compute_config/azure_vms.md

+34-34
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Azure Virtual Machines
22

3-
The Azure Virtual Machines client of Lithops can provide a truely serverless user experience on top of Azure VMs where Lithops creates new Virtual Machines (VMs) dynamically in runtime and scale Lithops jobs against them. Alternatively Lithops can start and stop an existing VM instances.
3+
The Azure Virtual Machines client of Lithops can provide a truely serverless user experience on top of Azure VMs where Lithops creates new Virtual Machines (VMs) dynamically in runtime and scale Lithops jobs against them (Create & Reuse modes). Alternatively Lithops can start and stop an existing VM instance (Consume Mode).
44

55
## Installation
66

@@ -48,11 +48,11 @@ az login
4848

4949
- Option 2: Alternatively, you can use a pre-built custom image that will greatly improve VM creation time for Lithops jobs. To benefit from this approach, navigate to [runtime/azure_vms](https://github.com/lithops-cloud/lithops/tree/master/runtime/azure_vms), and follow the instructions.
5050

51-
## Lithops and the VM consume mode
52-
53-
In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the `create/reuse` modes, since the entire job is executed in the same VM.
51+
## Create and reuse modes
52+
In the `create` mode, Lithops will automatically create new worker VM instances in runtime, scale Lithops job against generated VMs, and automatically delete the VMs when the job is completed.
53+
Alternatively, you can set the `reuse` mode to keep running the started worker VMs, and reuse them for further executions. In the `reuse` mode, Lithops checks all the available worker VMs and start new workers only if necessary.
5454

55-
### Lithops configuration for the consume mode
55+
### Lithops configuration for the create or reuse modes
5656

5757
Edit your lithops config and add the relevant keys:
5858

@@ -66,10 +66,7 @@ Edit your lithops config and add the relevant keys:
6666
subscription_id: <SUBSCRIPTION_ID>
6767
6868
azure_vms:
69-
exec_mode: consume
70-
instance_name: <VM_NAME>
71-
ssh_username: <SSH_USERNAME>
72-
ssh_key_filename: <SSH_KEY_PATH>
69+
exec_mode: reuse
7370
```
7471

7572

@@ -81,26 +78,32 @@ Edit your lithops config and add the relevant keys:
8178
|azure| region | |yes | Location of the resource group, for example: `westeurope`, `westus2`, etc|
8279
|azure| subscription_id | |yes | Subscription ID from your account. Find it [here](https://portal.azure.com/#view/Microsoft_Azure_Billing/SubscriptionsBlade)|
8380

84-
### Azure VMs - Consume Mode
81+
### Azure VMs - Create and Reuse Modes
8582

8683
|Group|Key|Default|Mandatory|Additional info|
8784
|---|---|---|---|---|
88-
|azure_vms | instance_name | | yes | virtual server instance Name. The instance must exists in your resource group |
89-
|azure_vms | ssh_username | ubuntu | yes | Username to access the VM. It will use `ubuntu` if not provided |
90-
|azure_vms | ssh_key_filename | ~/.ssh/id_rsa | yes | Path to the ssh key file provided to create the VM. It will use the default path if not provided |
91-
|azure_vms | region | |no | Location of the resource group, for example: `westeurope`, `westus2`, etc. Lithops will use the region set under the `azure` section if it is not set here |
92-
|azure_vms | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the VM|
93-
|azure_vms | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the defeuv python3 interpreter of the VM |
85+
|azure_vms| region | |no | Azure location for deploying the VMS. For example: `westeurope`, `westus2`, etc. Lithops will use the `region` set under the `azure` section if it is not set here|
86+
|azure_vms | image_id | Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest |no | Image ID. ARM resource identifier |
87+
|azure_vms | ssh_username | ubuntu |no | Username to access the VM |
88+
|azure_vms | ssh_password | |no | Password for accessing the worker VMs. If not provided, it is created randomly|
89+
|azure_vms | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. It will use the default path if not provided |
90+
|azure_vms | master_instance_type | Standard_B1s | no | Profile name for the master VM |
91+
|azure_vms | worker_instance_type | Standard_B2s | no | Profile name for the worker VMs |
92+
|azure_vms | delete_on_dismantle | False | no | Delete the worker VMs when they are stopped. Master VM is never deleted when stopped. `True` is NOT YET SUPPORTED |
93+
|azure_vms | max_workers | 100 | no | Max number of workers per `FunctionExecutor()`|
94+
|azure_vms | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the `worker_instance_type` VM|
95+
|azure_vms | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the default python3 interpreter of the VM |
9496
|azure_vms | auto_dismantle | True |no | If False then the VM is not stopped automatically.|
9597
|azure_vms | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
9698
|azure_vms | hard_dismantle_timeout | 3600 | no | Time in seconds to stop the VM instance after a job **started** its execution |
99+
|azure_vms | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to **create**, Lithops will automatically create new VMs for each map() call based on the number of elements in `iterdata`. If set to **reuse** will try to reuse running workers if exist |
97100
98101
99-
## Lithops and the VM auto create and reuse modes
100-
In the `create` mode, Lithops will automatically create new worker VM instances in runtime, scale Lithops job against generated VMs, and automatically delete the VMs when the job is completed.
101-
Alternatively, you can set the `reuse` mode to keep running the started worker VMs, and reuse them for further executions. In the `reuse` mode, Lithops checks all the available worker VMs and start new workers only if necessary.
102+
## Consume mode
102103
103-
### Lithops configuration for the create or reuse modes
104+
In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the `create/reuse` modes, since the entire job is executed in the same VM.
105+
106+
### Lithops configuration for the consume mode
104107
105108
Edit your lithops config and add the relevant keys:
106109
@@ -114,7 +117,10 @@ Edit your lithops config and add the relevant keys:
114117
subscription_id: <SUBSCRIPTION_ID>
115118
116119
azure_vms:
117-
exec_mode: reuse
120+
exec_mode: consume
121+
instance_name: <VM_NAME>
122+
ssh_username: <SSH_USERNAME>
123+
ssh_key_filename: <SSH_KEY_PATH>
118124
```
119125
120126
@@ -126,25 +132,19 @@ Edit your lithops config and add the relevant keys:
126132
|azure| region | |yes | Location of the resource group, for example: `westeurope`, `westus2`, etc|
127133
|azure| subscription_id | |yes | Subscription ID from your account. Find it [here](https://portal.azure.com/#view/Microsoft_Azure_Billing/SubscriptionsBlade)|
128134
129-
### Azure VMs - Create and Reuse Modes
135+
### Azure VMs - Consume Mode
130136
131137
|Group|Key|Default|Mandatory|Additional info|
132138
|---|---|---|---|---|
133-
|azure_vms| region | |no | Azure location for deploying the VMS. For example: `westeurope`, `westus2`, etc. Lithops will use the `region` set under the `azure` section if it is not set here|
134-
|azure_vms | image_id | Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest |no | Image ID. ARM resource identifier |
135-
|azure_vms | ssh_username | ubuntu |no | Username to access the VM |
136-
|azure_vms | ssh_password | |no | Password for accessing the worker VMs. If not provided, it is created randomly|
137-
|azure_vms | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. It will use the default path if not provided |
138-
|azure_vms | master_instance_type | Standard_B1s | no | Profile name for the master VM |
139-
|azure_vms | worker_instance_type | Standard_B2s | no | Profile name for the worker VMs |
140-
|azure_vms | delete_on_dismantle | False | no | Delete the worker VMs when they are stopped. Master VM is never deleted when stopped. `True` is NOT YET SUPPORTED |
141-
|azure_vms | max_workers | 100 | no | Max number of workers per `FunctionExecutor()`|
142-
|azure_vms | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the `worker_instance_type` VM|
143-
|azure_vms | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the default python3 interpreter of the VM |
139+
|azure_vms | instance_name | | yes | virtual server instance Name. The instance must exists in your resource group |
140+
|azure_vms | ssh_username | ubuntu | yes | Username to access the VM. It will use `ubuntu` if not provided |
141+
|azure_vms | ssh_key_filename | ~/.ssh/id_rsa | yes | Path to the ssh key file provided to create the VM. It will use the default path if not provided |
142+
|azure_vms | region | |no | Location of the resource group, for example: `westeurope`, `westus2`, etc. Lithops will use the region set under the `azure` section if it is not set here |
143+
|azure_vms | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the VM|
144+
|azure_vms | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the defeuv python3 interpreter of the VM |
144145
|azure_vms | auto_dismantle | True |no | If False then the VM is not stopped automatically.|
145146
|azure_vms | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
146147
|azure_vms | hard_dismantle_timeout | 3600 | no | Time in seconds to stop the VM instance after a job **started** its execution |
147-
|azure_vms | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to **create**, Lithops will automatically create new VMs for each map() call based on the number of elements in `iterdata`. If set to **reuse** will try to reuse running workers if exist |
148148
149149
150150
## Test Lithops

0 commit comments

Comments
 (0)