Our Understanding about AWS Lambda Function
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
## Background:
As stated in https://github.com/apache/texera/issues/3685#issue-3344145904, there are three phases to go through in terms of the Texera development. The project is now advancing from Phase 1 (Local Deployment) to Phase 2 (AWS Deployment). There are two major issues in phase 2, cost and latency.
We propose a solution to the cost challenge by adopting a serverless-like model where services are started on-demand rather than running continuously. This ensures that we only incur costs when the system is actively processing requests. The solution should be able to manage the lifecycle of Texera's eight microservices, waking them up as needed.
To build upon this strategy, this new issue will document our experimental results with an authentic serverless technology: AWS Lambda. We are exploring AWS Lambda as a method to implement this on-demand, cost-effective architecture. The goal is to test its ability to effectively create a system that is shut down by default and only "wakes up" to handle incoming user requests.
This issue shows the basic logic of lambda function by showing how to config a "tag trigger" using lambda function which will be triggered when a EC2 instance with a tag is created.
## The Automated Workflow
1. **Event:** A user creates or modifies an EC2 instance.
2. **Detection:** AWS Config detects this change and takes a snapshot of the instance's configuration.
3. **Trigger:** AWS Config sees that the change matches our custom rule's scope (`AWS::EC2::Instance`).
4. **Invocation:** AWS Config automatically calls our Lambda function, sending the instance's configuration data as the "event payload."
5. **Execution:** The Lambda function's Python code runs, inspects the tags in the event data, and determines if the `owner` tag is present.
6. **Reporting:** The function sends a verdict (`COMPLIANT` or `NON_COMPLIANT`) back to AWS Config.
7. **Update:** The AWS Config dashboard is updated to show the compliance status of the EC2 instance.
## Prerequisites
* An AWS Account.
* Permissions to manage IAM, Lambda, EC2, S3, and AWS Config.
* All steps must be performed in the **same AWS Region**.
## Steps to set up AWS Lambda
### Step 1: Set Up AWS Config
If you have never used AWS Config in your region, you must first enable it.
1. Navigate to the **AWS Config** service in the AWS Console.
2. If prompted, click **Get started**.
3. For **Resource types to record**, choose **Record all resources supported in this region**.
4. For the **Delivery method**, choose **Create S3 bucket**.
5. For the **AWS Config role**, choose **Create AWS Config service-linked role**.
6. Click **Next**, then **Skip** the rules page, then **Confirm**.
### Step 2: Create the IAM Role for Lambda
The Lambda function needs permission to send evaluation results to AWS Config.
1. Navigate to the **IAM** service.
2. Go to **Roles** and click **Create role**.
3. For **Trusted entity type**, select **AWS service**.
4. For **Use case**, choose **Lambda**. Click **Next**.
5. On the **Add permissions** page, search for and attach the following two policies:
* `AWSConfigRulesExecutionRole` (allows sending results to Config)
* `AWSLambdaBasicExecutionRole` (allows writing logs to CloudWatch for debugging)
6. Click **Next**.
7. For **Role name**, enter `LambdaOwnerTagCheckerRole`.
8. Click **Create role**.
### Step 3: Create the Lambda Function
This function contains the logic for our compliance check.
1. Navigate to the **Lambda** service.
2. Click **Create function**.
3. Select **Author from scratch**.
4. Configure the **Basic information**:
* **Function name:** `EC2-Owner-Tag-Checker`
* **Runtime:** **Python 3.9** (or a newer Python version)
* **Architecture:** **x86_64**
5. Expand **Change default execution role**:
* Select **Use an existing role**.
* From the **Existing role** dropdown, choose the `LambdaOwnerTagCheckerRole` you created.
6. Click **Create function**.
7. On the function page, scroll to the **Code source** editor and replace the existing code with your code.
### Step 4: Create the Custom AWS Config Rule
This rule connects the trigger (EC2 changes) to our Lambda function.
1. Navigate back to the **AWS Config** service.
2. Go to **Rules** and click **Add rule**.
3. Choose **Add custom rule**.
4. Configure the rule:
- **Name:** `Check-for-Owner-Tag`
- **AWS Lambda function ARN:** Paste the ARN of the `EC2-Owner-Tag-Checker` function. You can find this on the Lambda function's page.
- **Trigger type:** Select **Configuration changes**.
- **Scope of changes > Resources:** Select **EC2: Instance**.
5. Click **Save**.
### Step 5: Test the Solution
1. Navigate to the **EC2** service and **Launch instances**.
2. **Non-Compliant Test:** Launch an instance *without* adding any tags.
3. **Compliant Test:** Launch a second instance and, in the tags section, add a tag with **Key:** `owner` and **Value:** `your-name`.
4. Wait a few minutes, then go back to **AWS Config > Rules**.
5. Click on your `Check-for-Owner-Tag` rule. You should see the untagged instance listed as **Noncompliant** and the tagged instance listed as **Compliant**.
## Limitation
**AWS Lambda is an event-driven service**, meaning the function code runs only when it is triggered by an event.
You cannot send a direct HTTP request to the Lambda service endpoint itself to run your code. Instead, you must configure a specific trigger that will receive the request or event and then invoke your function. The way that you can trigger an AWS service is mainly by AWS Services, including Amazon S3, Amazon DynamoDB, Amazon SQS, Amazon SNS, AWS Config.
While in our service, we have to request for service in http format, meaning that we cannot wake up any lambda function directly by our current code.
Contributor guide
Research direction
Start with the Phase 2 context in issue 3685, then review the AWS Config, IAM, Lambda, EC2, and testing steps in this issue. Verify the workflow and its HTTP-trigger limitation in one AWS Region; done means the experimental findings are documented with a clear, reproducible setup and result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100