DEVELOPyo

[ Serverless Website : Todo List ]

1. This website should scale globally

2. Blogs are rarely written, but often read

3. Some of the website is purely static files, the rest is a dynamic REST API

4. Caching must be implement where possible

5. Any new users that subscribes should receive a welcome email

6. Any photo uploaded to the blog should have a thumbnail generated

[ 1. Serving static content, globally ]

※ CloudFront 를 사용하여 Amazon S3(region)를 global 로 사용

[ 2. Serving static content, globally, securely ]

S3 Bucket policy 설정을 통해 OAI (Origin Access Identity) 만 권한(authorize) 부여

client 는 Amazon S3 에 직접 접근 불가하며 CloudFont 를 통해서만 S3 에 접근 가능

[ 2-2. Serving static content, globally, securely ]

[ Sending Email ]

DynamoDB Stream 을 통해 데이터 변화시 Lambda 를 호출,

(SES(Simple Email Service) 를 사용 할 수 있는 IAM Role을 가진) Lambda 는 Amazon SES를 사용하여 email 전송

[ Making Thumbnail on photo added ]

Client 가 S3 에 직접 혹은 CloudFront 를 통해 이미지 업로드시 Lambda 를 호출(trigger),

Lambda 가 thumbnail 을 생성하여 S3 에 업로드. (이때 S3 는 SQS/SNS 를 사용하여 부가적인 동작 가능)

[ AWS Hosted Website Summary ] 

- We've seen static content being distributed using CloudFront with S3

- The REST API was serverless, didn't need Cognito because public

- We leveraged a Global DynamoDB table to serve the data globally (could use Aurora Global Tables)

- We enabled DynamoDB streams to trigger a Lambda function 

- The lambda function had an IAM role which use SES(Simple Email Service)

- SES(Simple Email Service) was used to send emails in a serverless way

- S3 can trigger SQS/SNS/Lambda to notify of events

DynamoDB Stream 사용법 관련 포스팅

[ Mobile Application : My Todo List ]

1. We want to create a mobile application with the following requirements

2. Expose as REST API with HTTPS

3. Serverless architecture

4. Users should be able to directly interact with their own folder in S3

5. Users should authenticate through a managed serverless service

6. The users can write and read to-dos, but they mostly read them

7. The database should scale, and have some high read throughput

[ 1. Mobile App : REST API layer ]

[ 2. Mobile App : giving users access to S3 ]

※ Note : save credentials on S3 (not on your mobile client)

[ 3. Mobile app : high read throughput, static data ]

- Serverless REST API : HTTPS, API Gateway, Lambda, DynamoDB

- Using Cognito to generate temporary credentials with STS to access S3 bucket with restricted policy. App users can directly access AWS resources this way. Pattern can be applied to DynamoDB, Lambda

- Caching the reads on DynamoDB using DAX

- Caching the REST requests at the API Gateway level

- Security for authentication and authrization with Cognito, STS

- SAM = Serverless Application Model

- Framework for developing and deploying serverless applications

- All the configuration is YAML code

  1) Lambda Functions

  2) DynamoDB tables

  3) API Gateway

  4) Cognito User Pools

- SAM can help you to run Lambda, API Gateway, DynamoDB locally

- SAM can use CodeDeploy to deploy Lambda functions

[ AWS Cognito ]

We want to give our users an identity so that they can interact with our application

1. Cognito User Pools :

- Sign in functionality for app users

- Integrate with API Gateway

2. Cognito Identity Pools (Federated Identity) :

- Provide AWS credentials to users so they can access AWS resources directly

- Integrate with Cognito User Pools as an identity provider

3. Cognito Sync :

- Synchronize data from device to Cognito.

- May be deprecated and replaced by AppSync

[ 1. AWS Cognito User Pools (CUP) ]

- Create a serverless database of user for your mobile apps

- Simple login : Username(or email) / password combination

- Possibility to verify emails / phone numbers and add MFA

- Can enable Federated Identities (Facebook, Gogle, SAML...)

- Sends back a JSON Web Tokens (JWT)

- Can be integrated with API Gateway for authentication

[ 2. Federated Identity Pools ]

Goal :

- Provide direct access to AWS Resources from the Client Side

How :

- Log in to federated identity provider - or remain anonymous

- Get temporary AWS credentials back from the Federated Identity Pool

- These credentials come with a pre-defined IAM policy stating their permissions

Example :

- provide (temporary) access to write to S3 bucket using Facebook Login

[ 3. AWS Cognito Sync ]

Deprecated - use AWS AppSync now

- Store preferences, configuration, state of app

- Cross device synchronization (any platform - iOS, Android, etc )

- Offline capability (synchronization when back online)

- Requires Federated Identity Pool in Cognito (not User Pool)

- Store data in datasets (up to 1MB)

- Up to 20 datasets to synchronize

SSL 인증서

ROOT 인증서

java key store

서버 인증서 설치의 내부 메카니즘/원리, 폐쇄망 엔지니어를 위한 글 (tistory.com)

Bittersweet Story: [TLS] SSL에 대해 알아보아요~ (btsweet.blogspot.com)

[ API Gateway ]

- AWS Lambda + API Gateway : No infrastructure to manage

- Support for the WebSocket Protocol

- Handle API versioning

- Handle different environments (dev/test/oper)

- Handle security (Authentication and Authorization)

- Create API keys, handle request throttling

- Swagger / Open API import to quickly define APIs 

- Transform and validate requests and responses

- Generate SDK and API specifications

- Cache API responses

[ API Gateway - Integrations High Level ]

1. Lambda Function

  - Invoke Lambda function

  - Easy way to expose REST API backed by AWS Lambda

2. HTTP

  - Expose HTTP endpoints in the backend

  eg. internal HTTP API on premise, ALB(Application Load Balancer)...

  Why? Add rate limiting, caching, user authentications, API Keys, etc...

3. AWS Service

  - Expose any AWS API through the API Gateway?

  eg. start an AWS Step Function workflow, post a message to SQS

  - Why? Add authentication, deploy publicly, rate control..

[ API Gateway - Endpoint Types ]

1. Edge-Optimized(default) : For global clients

  - Requests are routed through the CloudFront Edge locations (improves latency)

  - The API Gateway still lives in only one region

2. Regional :

  - For clients within the same region

  - Cloud manually combine with CloudFront (more control over the caching strategies and the distribution)

3. Private :

  - Can only be accessed from your VPC using an interface VPC endpoint (ENI)

  - Use a resource policy to define access

[ API Gateway - Hands on ]

1) API Lambda function 작성(언어별 코딩 지원)

2) Gate way 에서 API 종류 선택

3) context path 입력

4) deploy

[ API Gateway - Security ]

[ IAM Permissions ]

- Create an IAM policy authorization and attach to User/Role

- API Gateway verifies IAM permissions passed by the calling application

- Good to provide access within your own infrastructure

- Leverages "Sig v4" capability where IAM credential are in headers

- Great for users / roles already within your AWS account

- Handle authentication + authorization

- Leverage Sig v4

[ Lambda Authorizer (formerly Custom Authorizers) ] 

- Uses AWS Lambda to validate the token in header being passed

- Option to cache result of authentication

- Helps to use OAuth/SAML/3rd party type of authentication

- Lambda must return an IAM policy for the user

- Great for 3rd party tokens

- very flexible in terms of what IAM policy is returned

- Handle Authentication + Authorization

- Pay per Lambda invocation

[ Cognito User Pools ]

- Cognito fully manages user lifecycle

- API gateway verifies identity automatically from AWS Cognito

- No custom implementation required

- Cognito only helps with authentication, not authorization

- You manage your own user pool (can be backed by Facebook, Google login etc..)

- No need to write any custom code

- Must implement authorization in the backend

※ Authentication(인증) vs Authorization(인가)

Authentication(인증) : 사용자를 구분 및 확인 check who he is

Authorization(인가) : 사용자가 허가된 작업을 요청하는지 확인 check if he can access

cognito 관련 글

[ DynamoDB ]

- Fully Managed, Highly available with replication across 3 AZ

- NoSQL database - not a relational database

- Scales to massive workloads, distributed database

- Milions of requests per seconds, trillions of row, 100s of TB of storage

- Fast and consistent in performance (low latency on retrieval)

- Integrated with IAM for security, authorization and administration

- Enables event driven programming with DynamoDB Streams

- Low cost and auto scaling capabilities

[ DynamoDB Basic ]

DynamoDB is made of tables

- Each table has a PK (must be decided at creation time)

- Each table can have an infinite number of items

- Each item has attributes (can be added over time , can be null)

- Maximum size of a item is 400KB

- Supported data types 

  1) Scalar Types : String, Number, Binary, Boolean, Null

  2) Document Types : List, Map

  3) Set Types : String Set, Number Set, Binary Set

[ DynamoDB - Provisioned Throughput *** ]

- Table must have provisioned read and write capacity units

- RCU (Read Capacity Units) : throughput for reads ($0.00013 per RCU)

   1 RCU = 1 strongly consistent read of 4KB per second

   1 RCU = 2 eventually consistent read of 4 KB per second

- WCU (Write Capacity Unit) : throughput for writes ($0.00065 per WCU)

   1 WCU = 1 write of 1 KB per second

- Option to setup auto-scaling of throughput to meet demand

- Throughput can be exceeded temporarily using "burst credit"

- If burst credit are empty, you'll get a "ProvisionedThroughputException"

※ To prevent "ProvisionedThroughputException", use DAX

- It's then advised to do an exponential(기하급수적인) back-off retry

[ DynamoDB Advanced Features ]

[ DynamoDB - DAX ]

- DAX = DynamoDB Accelerator

- Seamless cache for DynamoDB, no application re-write

- Writes go through DAX to DynamoDB

- Micro second latency for cached reads & queries

- Solves the Hot Key problem (too many reads)

- 5 minutes TTL for caches by default

- Up to 10 nodes in the cluster

- Multi AZ (3 nodes minium recommended for production)

- Secure (encryption at rest with KMS, VPC, IAM, CloudTrail)

[ DynamoDB Streams ]

- Changes in DynamoDB (CUD) can end up in a DynamoDB Stream

- This stream can be read by AWS Lambda, and we can then do :

  1) React to changes in real time (eg. welcome email to new users)

  2) Analytics

  3) Create derivative(파생되는) tables / views

  4) Insert into Elastic Search

- Could implement cross region replication using Streams

- Stream has 24 hours of data retention

※ Before create a DynamoDB Global Table, DynamoDB Streams must be enabled first

: DynamoDB 를 글로벌로 사용하기 전에 DynamoDB Stream 을 활성화 시켜야 한다

[ DynamoDB - New Features ** ]

Transactions (new from Nov 2018)

- All or nothing type of operations

- Coordinated Insert, Update & Delete across multiple tables

- Include up to 10 unique items or up to 4 MB of data

On Demand (new from Nov 2018)

- No capacity planning needed (WCU/RCU) - scales automatically

- x2.5 more expensive than provisioned capacity (use with care)

- Helpful when spikes are un-predictable or the application is very low throughput

[ DynamoDB - Security & Other Features ]

- Security :

  1) VPC Endpoints available to access DynamoDB without internet

  2) Access fully controlled by IAM

  3) Encryption at rest using KMS

  4) Encryption in transit using SSL/TLS

- Backup and Restore feature available

  1) Point in time restore like RDS

  2) No performance impact

- Global Tables

  Multi region, fully replicated, high performance

- Amazon DMS can be used to migrate to DynamoDB (from Mongo, Oracle, MySQL, S3, etc..)

- You can launch a local DynamoDB on your computer for development purposes

- Global Tables (cross region replication)

  Active Active replication, many regions

  Must enable DynamoDB Streams

  Useful for low latency, DR purposes

- Capacity planning

  1) Planned capacity : provision WCU&RCU, can enable auto scaling

  2) On-demand capacity : get unlimited WCU & RCU, no throttle, more expensive

You have deployed a CDN using CloudFront

What if you wanted to run a global AWS Lambda alongside?

Or how to implement request filtering before reaching your application?

: you can use Lambda@Edge 

 deploy Lambda functions alongside your CloudFront CDN

   - build more responsive applications

   - don't manage servers, Lambda is deployed globally

   - Customize the CDN content

   - Pay only for what you use

You can use Lambda to change CloudFront requests and reponses :

After CloudFront receives a request from a viewer (viewer request)

Before CloudFront forwards the request to the origin (origin request)

After CloudFront receives the response from the origin (origin reponse)

Before CloudFront forwards the response to the viewer (viewer response)

Example of Lambda@Edge : Global application

Lambda@Egde : Use Cases

- Website Security and Privacy

- Dynamic Web Application at the Edge

- Search Engine Optimization (SEO)

- Intelligently Route Across Origins and Data Centers

- Bot Mitigation at the Edge

- Real-time Image Transformation

- A/B Testing

- User Authentication and Authorization

- User Prioritization

- User Tracking and Analytics

Serverless

- Serverless is a new paradigm in which the developers don't have to manage servers anymore

- just deploy code, just deploy functions

- Serverless == FaaS(Function as a Service)

- Serverless was pioneered by AWS Lambda but now also includes anything that's managed such as databases, messaging, storage, etc.

- Serverless doesn't mean there are no servers

Serverless in AWS 

: AWS Lambda, DynamoDB, AWS Cognito, AWS API Gateway, S3, Fargate...

[ Why use AWS Lambda (compare Amazon EC2 with AWS Lambda) ] 

Amazon EC2

- Virtual Servers in the Cloud

- Limited by RAM and CPU

- Continuously running

- Scaling means intervention to add /remove servers

AWS Lambda

- Virtual functions - no servers to manage

- Limited by time - short executions

- Run on-demand

- Scaling is automated

[ Benefits of AWS Lambda ]

-Easy Pricing

  Pay per request and compute time

  Free tier of 1,000,000 AWS Lambda requests and 400,000 GBs of compute time

- Integrated with the whole AWS suite of services

- Integrated with many programming languages 

- Easy monitoring through AWS CloudWatch

- Easy to get more resources per functions (up to 10GB of RAM)

- Increasing RAM will also improve CPU and network

[ AWS Lambda language support ]

- js

- Python

- Java

- C#

- Go

- Ruby

- Custom Runtime API 

- Lambda Container Image

  The container image must implement the Lambda Runtime API

  ECS / Fargate is preferred for running arbitrary(임의적인 멋대로인) Docker images

[ AWS Lambda Integrations Main ones ]

- API Gateway

- Kinesis

- Dynamo DB

- S3

- CloudFront

- CloudWatch Events Event Bridges

- CloudWatch Logs

- SNS

- SQS

- Cognito

[ Example : Serverless Thumbnail creation ]

[ Example : Serverless CRON job ]

[ AWS Lambda Pricing ]

- https://aws.amazon.com/lambda/pricing/ 

- Pay per calls 

  first 1,000,000 requests are free

  $0.20 per 1 million requests 

- Pay per duration : 

  400,000 GB seconds of compute time per month if FREE

  == 400,000 sec if function is 1GB RAM

  == 3,200,000 seconds if function is 128MB RAM

- After that $1.00 for 600,000 GB-sec

- It is usually very cheap to run AWS Lambda so it's very popular

[ AWS Lambda Limits to Know - per region ]

Execution :

1) Memory allocation 128mb-10GB (64mb increments)

2) Maximum execution time : 900 sec (15min)

3) Environment variables (4 KB)

4) Disk capacity in the function container (in /tmp) : 512MB

5) Concurrency executions : 1000 (can be increased)

Deployment :

1) Lambda function deployment size (compressed .zip) : 50MB

2) Size of uncompressed deployment (code + dependencies) : 250MB

3) Can use the /tmp directory to load other files at startup

4) Size of environment variables : 4 KB

[ ORACLE에서의 자료형 VARCHAR2, CLOB 사용 및 가공시 주의점 ]

보통 4000bytes 이하의 문자열에 사용되는 VARCHAR2

대용량 데이터 LOB 의 문자자료형인 CLOB 데이터

[ Charset 에 따른 한글 문자 bytes 크기 ]

ORACLE 의 권장 character set 은 UTF-8 이며, UTF-8 은 문자 하나에 최대 4bytes.

* 한글은 UTF-8 에서 문자 하나당 3bytes, EUC-KR 에서 문자 하나당 2bytes

* SQL에서 VARCHAR2 는 최대 4000bytes 길이 제한.

※ DB charset 확인하는 방법

1) 설정확인(권장)

SELECT name, value$
FROM sys.props$
WHERE name = 'NLS_CHARACTERSET';

2) 문자를 조회하여 확인

SELECT LENGTHB('일') FROM DUAL;

결과로 리턴되는 BYTES 크기 확인

* LENGTHB 는 BYTES 크기 반환

아래와 같이 TABLE 생성시 

CREATE TABLE sample.persons(
    name VARCHAR2(9) NOT NULL,
    PRIMARY KEY(name)
);

VARCHAR2(9) 의 9은 bytes 를 의미하며 UTF-8 기준으로 한글은 3글자밖에 들어갈 수 없다.

[ VARCHAR2 칼럼에 데이터 넣기 ]

1. Query 에서 잘라 넣기

1) SUBSTR 사용하여 CHAR 문자 잘라 넣기

INSERT INTO persons VALUES (SUBSTR('네자이름'), 1, 3));

* SUBSTR 은 문자 length 를 기준으로 자름

* TABLE DESC 에 명시된 자료형의 크기는 default가 bytes 기준이므로 CHAR length를 기준으로 위와 같이 문자열을 잘라넣는건 부적절.

2) SUBSTRB 사용하여 BYTES 단위로 문자 잘라 넣기

INSERT INTO persons VALUES (SUBSTRB('네자이름'), 1, 9));

* SUBSTRB 는 문자 bytes 를 기준으로 자름

* 위와같이 데이터를 넣을 경우 bytes단위로 문자열이 잘려나가기 때문에 한글이 깨질 수 있으므로 부적절.

※ 또 다른 문제.

SQL에서 VARCHAR2 는 최대 4000bytes 길이 제한이 있어

1), 2) 에서 사용된 SUBSTR, SUBSTRB 함수 모두 input 값이 4000bytes 가 넘을시 "ORA-01704: 문자열이 너무 깁니다" 에러가 발생한다.

2. 소스레벨에서 BYTES 단위로 잘라 넣기

정해진 bytes 크기의 한도내에서 한글이 깨지지 않게 문자열 자르기

SQL 에서 문자열을 가공하기보단 소스레벨에서 가공

[ CLOB 데이터 읽기 및 가공하기 ]

CLOB 데이터는 LENGTH, LENGTHB, SUBSTRB 함수 등을 지원하지 않음.

VARCHAR2 데이터에서 사용가능한 함수 사용을 위해 VARCHAR2 로 변환하고자 할 경우,

DBMS_LOB.SUBSTR(clob칼럼, 가져올길이, offset)을 사용하여 VARCHAR2 로 변환이 가능

* CLOB 데이터 가공시 SUBSTR 보다 DBMS_LOB.SUBSTR 의 퍼포먼스가 뛰어남 (참고)

* DBMS_LOB.SUBSTR() 사용시에도 4000bytes 를 넘는 데이터를 가져오지 않도록 주의 해야한다

※ 또 다른 문제.

temp segment 가 unrelease 되는 현상.

CLOB 칼럼에 함수를 직접 사용할 경우, 메모리와 같은 역할을 하는 temp segment를 사용하게 되며 release 하지 않고 쌓일 수 있다. 이는 서버 장애를 유발하여 서비스 전체에 영향을 줄 수 있다.

실제로 이와 관련하여, 운영중인 DB서버 전체가 마비되어 서비스 장애를 경험해 보았다..

=> CLOB 칼럼에 함수 사용은 지양하자. 소스레벨에 그대로 가져와서 소스레벨에서 자르는 걸 권장.

[ 관련에러 ]

ORA-01704: 문자열이 너무 깁니다

ORA-64203: 문자 집합 변환 후 대상 버퍼가 너무 작아 CLOB 데이터를 보유할 수 없습니다.

ORA-06502: 문자열 버퍼가 너무 작습니다.

참고 :

https://tomining.tistory.com/95