🎓

Exam Preparation - Developer - Associate (DVA-C01)


AWS Fundamentals - Part 1 ( Region, IAM & EC2)

IAM ( Identity and Access Management)

  • Categories

    Users

    Groups

    Roles

Policies are written in JSON

  • IAM Federation

    Uses SAML standard ( Active Directory)

    This is used at enterprises, so you can use their AD credentials to log into AWS console.

EC2 Userdata

Userdata are used to install bootstrap

ie) Execute commands to install software while booting.

Bydefault the commands are run as root user

Ec2 instance launch types

  • On Demand Instance types

    For Short workload

    Pay for what you use.

    Highes cost and no long term commitment

  • Reserved

    Minimum reserved for 1 Year

    → Reserved Instances: for Long workloads

    → Convertible Reserved Instances: Long workloads with flexible instances

    → Scheduled Reserved Instances: Every Thu b/w 3 and 6 pm

    upto 75% discount compared to On demand instance

    Reservation period can be 1 or 3 years

  • Spot Instances

    Less reliable, for short workloads

  • Dedicated Instances

    No other customer will share your instance

  • Dedicated Hosts

    Book an entire physical server, you can control where to place the instances

Security Groups

  • They control how traffic is into or out of our EC2 machine.

They used to control port, what IP ranges can access the ports.

Elastic IP

If the Ec2 instance is stopped and started again then the IP will change, to avoid changing of IP we use Elastic IP

You can attach one Elastic IP to instance

You can have only 5 Elastic IP address in your account.

Maximum try to avoid using Elastic IP

Security on AWS

Shared Responsibility

https://aws.amazon.com/compliance/shared-responsibility-model/
https://aws.amazon.com/security/penetration-testing/

  • AWS vs Customer responsibility Overview

  • Customer Responsibility
    • Customer data
    • Platform , application, and IAM
    • OS patching on EC2
    • Antivirus
    • Network, and firewall configuration
    • Multi-factor authentication
    • Password and key rotation
    • Security groups
    • Resource-based policies
    • Access control lists
    • VPC
    • Operating system
    • level patches
    • Data in transit and at res
  • AWS Responsibility
    • Regions, availability zones, and edge locations
    • Physical server level and below Fire/ power/ clim ate management
    • Storage device decommissioning according to industry standards
    • Personnel security
    • Network device security and ACLs
    • API access endpoints use SSL for secure communication
    • DDoS protection
    • EC2 instances and spoofing protection (ingress/ egress filtering)
    • Port scanning against rules even if it's your own environment
    • EC2 instance hypervisor isolation
    • Instances on the same physical device are separated at the hypervisor level; they are independent of each other Underlying OS patching on Lambda, RDS, DynamoDB, and other managed services; customer focuses on security

  • AWS S3 Bucket Policy
  • S3 Data Integrity
    • Versioning
    • Replication
    • MFA Delete
  • VPC, SecurityGroup , NACL
  • Security Group

    By Default All inbound traffic are denied and outbound traffic are allowed

  • Ec2
    • User Data is used to give bootup instruction.
    • Instance is billed only when in running state(not implies for reserved instance as it based on fixed duration)
    • Ec2 instance private IP remains un changed once stopped and restart
    • Public IP will change once instance stopped and relaunched.
    • Bootstrapping: Process to run any specific commands → install → Os update any action to be performed before instance launched.
    • User data will automatically run script in sudo

    → AMI is region locked can't be shared to other region with same API ID

  • Ec2 Spot instance

    Spot instance are reserved at spot, you can get instances at lower price!

    đź’ˇ
    At any time AWS can withdraw your spot instance, so not suitable for critical application.
  • IP
    đź’ˇ
    You can only have 5 Elastic IP in your account (you can ask AWS to increase that).

IAM

đź’ˇ
→ Admin policy contains almost all access except billing. → Explicit deny will always overwrite , explicit allow → IAM Policies can't be directly attached to AWS resources
  • Root User
    • The user created when an AWS account is created.
    • The credentials are the email and password used when signing up for an AWS account.
    • By default, the root user has full administrative rights and access to every part of the account.
  • Best Practices for Root User
    • The root user should not be used for daily work and administration
    • Another user should be created for daily work that has admin rights.
    • The root user account should not have access keys; delete them if they exist.
    • The root user should always use multi-factor authentication(MFA), like Google Authenticator.I AM Users
  • IAM User
    • A new user has an implicit deny for all AWS services, requiring a policy be added to grant them access.
    • Users receive unique credentials (usernam e, password, and possibly access keys).
    • Users can have IAM policies applied directly to them , or they can be a m em ber of a group that has policies attached.
    • With policies, an explicit deny always overrides an explicit allow from attached policies:
    • For instance, all policies attached to a user will be ignored if a single deny-all policy is added.
  • Best Practices for Users
    • Never store or "pass" your access credentials to an EC2 instance , use SSH forwarding.
    • MFA can and shouldbe used for user accounts.
    • Access credentials are unique
  • IAM Group
    • Allow for policy assignments to multiple users at the same time
    • Permissions may be assigned to group
    • More organized and efficient way to manage users and policies
    • Cannot be nested
    • Users may be a member of multiple groups
  • Best Practices for IAM Groups
    • Organize users by function (e.g., DB admins, developers, architects, etc.)
    • Assign IAM policies to groups, not individual users
  • IAM Policies

  • IAM Roles
  • MFA
  • Access keys

    Access keys are long-term credentials for an IAM user or the AWS account root user. You can use access keys to sign programmatic requests to the AWS CLI or AWS API (directly or using the AWS SDK). For more information, see Signing AWS API Requests in the Amazon Web Services General Reference.

    Access keys consist of two parts: an access key ID (for example, AKIAIOSFODNN7EXAMPLE) and a secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY). Like a user name and password, you must use both the access key ID and secret access key together to authenticate your requests. Manage your access keys as securely as you do your user name and password.

    As a best practice, use temporary security credentials (IAM roles) instead of access keys, and disable any AWS account root user access keys. For more information, see Best Practices for Managing AWS Access Keys in the Amazon Web Services General Reference.

    If you still need to use long-term access keys, you can create, modify, view, or rotate your access keys (access key IDs and secret access keys). You can have a maximum of two access keys. This allows you to rotate the active keys according to best practices.

    When you create an access key pair, save the access key ID and secret access key in a secure location. The secret access key is available only at the time you create it. If you lose your secret access key, you must delete the access key and create a new one.

  • External ID

    To provide access to users from other aws account


ELB & Autoscaling Group

  • Application Load balancer
  • Network Load balancer
  • Clasic loadbalancer
  • ELB- stickiness

    → To make user to connect to same server as previously connected server.

    → The stickiness to be configured in the target group

  • ELB crosszone Load balancing

    → Distribute load all across availability zone → You would need to pay extra for using this feature in NLB

    → In Clasic load balancer and Network Loadbalancer this is not enabled by default → In Application load balancer this feature is enabled by default.

  • ELB - SSL certificate

    → Flow is Browser to ALB (HTTPS) and ALB to Ec2 (HTTP)

    → SNI (Server Name indication) To host multiple domains in a same server and to get SSL handshake succeed Note: Only works for ALB and NLB (newer version) and cloudfront , but not on Classic Load balancer. The Ciphers can be controlled on Listeners

  • ELB - Connection draining

    Time to complete ongoing instances before de-registering or stopping the unhealthy resources.

    → Called as connection draining in Load balancers (for Classic Load balancers) → Called as Deregistration delay in Target group (for ALB and NLB)

  • Access Logs
    đź’ˇ
    Elastic Load Balancing provides access logs that capture detailed information about requests sent to your load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses. You can use these access logs to analyze traffic patterns and troubleshoot issues.
đź’ˇ
NLB expose the static public IP, where as ALB and classic Load balancer expose DNS URL
đź’ˇ
Default timeout for lambda func is 3seconds
đź’ˇ
Lambda can be used to transfer data where as SQS can't be used

ALB target group can't be pointed to NLB, it can be pointed to Ec2 instances , IP address, Lambda balancer

Route 53

  • CNAME vs Alias

    If domain is Top level domain then use Alias record. If domain is non TLD, you can use either Alias or CNAME

  • Healthcheck

    Amazon Route 53 health checks monitor the health and performance of your web applications, web servers, and other resources. Each health check that you create can monitor one of the following:

    • The health of a specified resource, such as a web server
    • The status of other health checks
    • The status of an Amazon CloudWatch alarm

EC2 storage - EBS and EFS

  • EBS (Elastic Block store)
    • EBS are AZ zone locked
    • It's a network drive, not a physical drive It can be detached from one Ec2 instance and can be attached to another Ec2 instance.
    • It's provisioned based on capacity ( GBs or IOPS) You pay for what you provisioned and not for what you use.
    • You can increase the capacity later
    • Only Gp2 and IO1 are to be used as boot volumes
  • You are running a high-performance database that requires an IOPS of 210,000 for its underlying filesystem. What do you recommend?

    Is running a DB on EC2 instance store possible? It is possible to run a database on EC2. It is also possible to use instance store, but there are some considerations to have. The data will be lost if the instance is stopped, but it can be restarted without problems. One can also set up a replication mechanism on another EC2 instance with instance store to have a standby copy. One can also have back-up mechanisms. It's all up to how you want to set up your architecture to validate your requirements. In this case, it's around IOPS, and we build an architecture of replication and back up around i


Developer

CLI

đź’ˇ
Policy Simulator can be used to test the generated/written policy

SQS

  • SQS Autoscales on it's own
  • You would like messages to be processed by SQS consumers only after 5 minutes. What should you do?

    Increase the delay seconds parameter.

  • You can't convert existing queue to FIFO you need to create new one
  • Default visibility timeout is 30 seconds
  • SQS cost optimization best practice

    To reduce costs, batch your message actions:

    • To send, receive, and delete messages, and to change the message visibility timeout for multiple messages with a single action, use the Amazon SQS batch API actions.
    • To combine client-side buffering with request batching, use long polling together with the buffered asynchronous client included with the AWS SDK for Java.
  • Lambda with SQS

    You can use an AWS Lambda function to process messages in an Amazon Simple Queue Service (Amazon SQS) queue. Lambda event source mappings support standard queues and first-in, first-out (FIFO) queues. With Amazon SQS, you can offload tasks from one component of your application by sending them to a queue and processing them asynchronously.

    Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages. Lambda reads messages in batches and invokes your function once for each batch. When your function successfully processes a batch, Lambda deletes its messages from the queue.

  • Delay queue

    Delay queues let you postpone the delivery of new messages to a queue for a number of seconds, for example, when your consumer application needs additional time to process messages. If you create a delay queue, any messages that you send to the queue remain invisible to consumers for the duration of the delay period. The default (minimum) delay for a queue is 0 seconds. The maximum is 15 minutes.

    To set delay seconds on individual messages, rather than on an entire queue, use message timers to allow Amazon SQS to use the message timer's DelaySeconds value instead of the delay queue's DelaySeconds value.

  • Deduplication message id

    Amazon SQS FIFO First-In-First-Out queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated.

    Amazon SQS FIFO queues follow exactly-once processing. It introduces a parameter called Message Deduplication ID, which is the token used for deduplication of sent messages. If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully but aren't delivered during the 5-minute deduplication interval.

    Amazon SQS continues to keep track of the message deduplication ID even after the message is received and deleted.

  • Dead letter queue

    Any Lambda function invoked asynchronously is retried twice before the event is discarded. If the retries fail and you're unsure why, use Dead Letter Queues (DLQ) to direct unprocessed events to an Amazon SQS queue or an Amazon SNS topic to analyze the failure.

    AWS Lambda directs events that cannot be processed to the specified Amazon SNS topic or Amazon SQS queue. Functions that don't specify a DLQ will discard events after they have exhausted their retries.

SNS

  • SNS message attributes

    Amazon SNS supports delivery of message attributes, which let you provide structured metadata items (such as timestamps, geospatial data, signatures, and identifiers) about the message. For attribute mapping between Amazon SNS and Amazon SQS, each message can have up to 10 attributes. When using raw mode or an endpoint other than Amazon SQS, a message can have more than 10 attributes.

    Message attributes are optional and separate from—but are sent together with—the message body. The receiver can use this information to decide how to handle the message without having to process the message body first.

  • Filter Policy

    A filter policy is a simple JSON object containing attributes that define which messages the subscriber receives. When you publish a message to a topic, Amazon SNS compares the message attributes to the attributes in the filter policy for each of the topic's subscriptions. If any of the attributes match, Amazon SNS sends the message to the subscriber. Otherwise, Amazon SNS skips the subscriber without sending the message. If a subscription doesn't have a filter policy, the subscription receives every message published to its topic.


DynamoDB

  • Provisioned Throughputc
    • Table must be provisioned with read & write capacity units
    • Read Capacity Units (RCU) throughput for read
    • Write Capacity Units (WCU) throughput for write
    • Burst Credit → Throughput can be exceeded temporarily using burst credit.
    • If your burst credit is empty you will receive Provisioned Throughput exception
  • Write Capacity Unit(WCU)

    1 write capacity unit represents one write per second for an item upto 1 KB in size.

    If larger then 1 KB then it requires more WCU

    • Examples:

      1→ To write 10 objects per second of 2 KB each 2 * 10 = 20 WCU

      2→ To write 6 objects per second of 4.5 KB each 6* 5 = 30 WCU (4.5 is rounded to nearest round)

      3 → To write 120 objects per minute of 2KB each

      (120/60)*2 = 4 WCU (Convert minute to second hence we do 120/60)

  • Strong consistent vs Eventually consistent

  • Read Capacity Unit(RCU)

    1 RCU represents 1 strongly consistent read per second or 2 eventually consistent read per second for an item upto 4KB

    • Example:

      1) To 10 strongly consistent reads per seconds of 4 KB each 10*1 = 10 (multiplied by 1 as at once 4KB can be done)

      2) To 16 eventually consistent reads per seconds of 12 KB each (16/2)*4 = 32 (divided by 2 since it's eventually consistent)

      3) 10 strongly consistent reads per second of 6 KB each

      10 * 2= 20 (Here 6 is rounded to nearest which is 8 so 8/4 ie- 2)

  • DynamoDB-DAX (DynamoDB Accelerator)
    • What is DAX?

      DAX is a DynamoDB-compatible caching service that enables you to benefit from fast in-memory performance for demanding applications. DAX addresses three core scenarios:

      As an in-memory cache, DAX reduces the response times of eventually consistent read workloads by an order of magnitude from single-digit milliseconds to microseconds. DAX reduces operational and application complexity by providing a managed service that is API-compatible with DynamoDB. Therefore, it requires only minimal functional changes to use with an existing application. For read-heavy or bursty workloads, DAX provides increased throughput and potential operational cost savings by reducing the need to overprovision read capacity units. This is especially beneficial for applications that require repeated reads for individual keys.

    • How it works?

    • Writes to DB goes through DAX
    • Micro second latency for cached reads and queries
    • 5 Min TTL as default cache
    • Supports upto 10 clusters
    • Multi AZ - min 3 node recommended for production
    • Secure(encryption at rest with KMS, VPC, IAM , CloudTrail)
    • For strongly consistent read request from an application, DAX Cluster pass all request to DynamoDB & does not cache for these requests.
  • Diff b/w Elasticache and DAX

    In Elasticache stores arggregation result In DAX individual object cache query/scan cache

    No need of individual servers like Redis/Memcached in DAX

  • Dynamo DB CLI
    • — projection-expression

      Attribute to retrieve data You use projection-expression while performing query operation when you want to retrieve data with filter You use projection-expression while performing scan operation when you just want to retrieve data data from table

    • — filter-expression

      To Filter results

  • Dynamo DB-TTL (Time to Live)

    TTL automatically deletes an item after an expiry date / time

    Helps to reduce storage and manage the table size over time

    TTL is enabled per row.

    DynamoDB deletes the expired item within 48 hours of the expiration

    To retrieve expired items DynamoDB Streams can be used

    TTL can be given any name


    Amazon DynamoDB Time to Live (TTL) allows you to define a per-item timestamp to determine when an item is no longer needed. Shortly after the date and time of the specified timestamp, DynamoDB deletes the item from your table without consuming any write throughput. TTL is provided at no extra cost as a means to reduce stored data volumes by retaining only the items that remain current for your workload’s needs.

    TTL is useful if you store items that lose relevance after a specific time. The following are example TTL use cases:

    • Remove user or sensor data after one year of inactivity in an application.
    • Archive expired items to an Amazon S3 data lake via DynamoDB Streams and AWS Lambda.
    • Retain sensitive data for a certain amount of time according to contractual or regulatory obligations.

  • AWS Exponential backoff

    In addition to simple retries, each AWS SDK implements exponential backoff algorithm for better flow control. The idea behind exponential backoff is to use progressively longer waits between retries for consecutive error responses. You should implement a maximum delay interval, as well as a maximum number of retries.

    In SDK exponential backoff is enabled by default / You need to enable manually

  • Database streams

    Changes in DynamoDB can end in Streams

    streams can be red by Lambda / EC2 instances

    Stream has 24 hours of data retention

    The stream datas can be

    Key only New_image → updated item Old_image → replaced item New_and_old_images → to replace both

  • Dynamo DB write types
  • Secondary Index
    • Local Secondary Index(LSI)

      LSI to be defined at the time of creation of table

      Upto 5 secondary key ( Local hash value ) to be defined

      The attribute that you choose must be a scalar string , Number or Binary

    • Global Secondary Index (GSI)

      GSI = Partition key + optional sort key

      You can modify the GSI later once even after table is created.

  • CMK

    DynamoDB uses the CMK to generate and encrypt a unique data key for the table, known as the table key.

    With DynamoDB, AWS Owned, or AWS Managed CMK can be used to generate & encrypt keys.

    AWS Owned CMK is free of charge while AWS Managed CMK is chargeable. Customer managed CMK’s are not supported with encryption at rest. "With encryption at rest, DynamoDB transparently encrypts all customer data in a DynamoDB table, including its primary key and local and global secondary indexes, whenever the table is persisted to disk

  • Read/write units

    Read Request Units and Write Request Units

    For on-demand mode tables, you don't need to specify how much read and write throughput you expect your application to perform. DynamoDB charges you for the reads and writes that your application performs on your tables in terms of read request units and write request units.

    • One read request unit represents one strongly consistent read request, or two eventually consistent read requests, for an item up to 4 KB in size. Two read request units represent one transactional read for items up to 4 KB. If you need to read an item that is larger than 4 KB, DynamoDB needs additional read request units. The total number of read request units required depends on the item size, and whether you want an eventually consistent or strongly consistent read. For example, if your item size is 8 KB, you require 2 read request units to sustain one strongly consistent read, 1 read request unit if you choose eventually consistent reads, or 4 read request units for a transactional read request.
    • One write request unit represents one write for an item up to 1 KB in size. If you need to write an item that is larger than 1 KB, DynamoDB needs to consume additional write request units. Transactional write requests require 2 write request units to perform one write for items up to 1 KB. The total number of write request units required depends on the item size. For example, if your item size is 2 KB, you require 2 write request units to sustain one write request or 4 write request units for a transactional write request.

    https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html

  • Global table

    Used to have multiple copies of your table in additional regions

  • Parallelscans

    Many applications can benefit from using parallel Scan operations rather than sequential scans. For example, an application that processes a large table of historical data can perform a parallel scan much faster than a sequential one. Multiple worker threads in a background "sweeper" process could scan a table at a low priority without affecting production traffic. In each of these examples, a parallel Scan is used in such a way that it does not starve other applications of provisioned throughput resources.

    Although parallel scans can be beneficial, they can place a heavy demand on provisioned throughput. With a parallel scan, your application has multiple workers that are all running Scan operations concurrently. This can quickly consume all of your table's provisioned read capacity. In that case, other applications that need to access the table might be throttled.

    A parallel scan can be the right choice if the following conditions are met:

    • The table size is 20 GB or larger.
    • The table's provisioned read throughput is not being fully used.
    • Sequential Scan operations are too slow.
  • Capturing Table activity with DynamoDB stream

    Many applications can benefit from the ability to capture changes to items stored in a DynamoDB table, at the point in time when such changes occur. The following are some example use cases:

    • An application in one AWS Region modifies the data in a DynamoDB table. A second application in another Region reads these data modifications and writes the data to another table, creating a replica that stays in sync with the original table.
    • A popular mobile app modifies data in a DynamoDB table, at the rate of thousands of updates per second. Another application captures and stores data about these updates, providing near-real-time usage metrics for the mobile app.
    • A global multi-player game has a multi-active topology, storing data in multiple AWS Regions. Each locale stays in sync by consuming and replaying the changes that occur in the remote Regions.
    • An application automatically sends notifications to the mobile devices of all friends in a group as soon as one friend uploads a new picture.A new customer adds data to a DynamoDB table. This event invokes another application that sends a welcome email to the new customer.

    DynamoDB Streams enables solutions such as these, and many others. DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours. Applications can access this log and view the data items as they appeared before and after they were modified, in near-real time.

    Encryption at rest encrypts the data in DynamoDB streams. For more information, see DynamoDB Encryption at Rest.

    A DynamoDB stream is an ordered flow of information about changes to items in a DynamoDB table. When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table.

    Whenever an application creates, updates, or deletes items in the table, DynamoDB Streams writes a stream record with the primary key attributes of the items that were modified. A stream record contains information about a data modification to a single item in a DynamoDB table. You can configure the stream so that the stream records capture additional information, such as the "before" and "after" images of modified items.

    DynamoDB Streams helps ensure the following:

    • Each stream record appears exactly once in the stream.
    • For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

    DynamoDB Streams writes stream records in near-real time so that you can build applications that consume these streams and take action based on the contents.

  • Handling throttling errors

    1→ Exponential backoff

    2 → Increase throughput capacity

  • Projection expression

    To read data from a table, you use operations such as GetItem, Query, or Scan. Amazon DynamoDB returns all the item attributes by default. To get only some, rather than all of the attributes, use a projection expression.

    A projection expression is a string that identifies the attributes that you want. To retrieve a single attribute, specify its name. For multiple attributes, the names must be comma-separated.

  • Global Secondary Index

    A global secondary index is an index with a partition key and a sort key that can be different from those on the base table. A global secondary index is considered "global" because queries on the index can span all of the data in the base table, across all partitions.

    To create a table with one or more global secondary indexes, use the CreateTable operation with the GlobalSecondaryIndexes parameter. For maximum query flexibility, you can create up to 20 global secondary indexes (default limit) per table. You must specify one attribute to act as the index partition key; you can optionally specify another attribute for the index sort key. It is not necessary for either of these key attributes to be the same as a key attribute in the table. Global secondary indexes inherit the read/write capacity mode from the base table.

    As shown in the above table, the following are the things that the developer should consider when using a global secondary index:

    đź’ˇ
    Global Secondary Index doesn't support Consistent Read need to use Eventual Read
  • Global secondary index -2

    A global secondary index (GSI) is an index with a partition key and a sort key that can be different from those on the base table. It is considered "global" because queries on the index can span all of the data in the base table, across all partitions.

    Every global secondary index has its own provisioned throughput settings for read and write activity. Queries or scans on a global secondary index consume capacity units from the index, not from the base table. The same holds true for global secondary index updates due to table writes.

    When you create a global secondary index on a provisioned mode table, you must specify read and write capacity units for the expected workload on that index. The provisioned throughput settings of a global secondary index are separate from those of its base table. A Query operation on a global secondary index consumes read capacity units from the index, not the base table. When you put, update or delete items in a table, the global secondary indexes on that table are also updated; these index updates consume write capacity units from the index, not from the base table.

    For example, if you Query a global secondary index and exceed its provisioned read capacity, your request will be throttled. If you perform heavy write activity on the table but a global secondary index on that table has insufficient write capacity, then the write activity on the table will be throttled.

    To avoid potential throttling, the provisioned write capacity for a global secondary index should be equal or greater than the write capacity of the base table since new updates will write to both the base table and global secondary index.

  • Queries in DynamoDB

    To specify the search criteria, you use a key condition expression—a string that determines the items to be read from the table or index.

    You must specify the partition key name and value as an equality condition.

    You can optionally provide a second condition for the sort key (if present)

  • Capacity unit consumed by query

    Query operation does not return any data on how much read capacity it consumes. However, you can specify the ReturnConsumedCapacity parameter in a Query request to obtain this information. The following are the valid settings for ReturnConsumedCapacity:

    • NONE — No consumed capacity data is returned. (This is the default.)
    • TOTAL — The response includes the aggregate number of read capacity units consumed.
    • INDEXES — The response shows the aggregate number of read capacity units consumed, together with the consumed capacity for each table and index that was accessed.
  • Store larger file size

    Amazon S3 can be used to saved items which are exceeding 400 KB. In this option , Items are saved in S3 buckets while an object identifier is saved in DynamoDB table which points to an item in S3

  • Encryption

    When creating a new table, you can choose one of the following customer master keys (CMK) to encrypt your table:

    • AWS owned CMK – Default encryption type. The key is owned by DynamoDB (no additional charge).
    • AWS managed CMK – The key is stored in your account and is managed by AWS KMS (AWS KMS charges apply).
    • Customer managed CMK – The key is stored in your account and is created, owned, and managed by you. You have full control over the CMK (AWS KMS charges apply).
  • Local secondary index

    A local secondary index maintains an alternate sort key for a given partition key value. A local secondary index also contains a copy of some or all of the attributes from its base table; you specify which attributes are projected into the local secondary index when you create the table. The data in a local secondary index is organized by the same partition key as the base table, but with a different sort key. This lets you access data items efficiently across this different dimension. For greater query or scan flexibility, you can create up to five local secondary indexes per table.

    Suppose that an application needs to find all of the threads that have been posted within the last three months. Without a local secondary index, the application would have to Scan the entire Thread table and discard any posts that were not within the specified time frame. With a local secondary index, a Query operation could use LastPostDateTime as a sort key and find the data quickly.

    To create a Local Secondary Index, make sure that the primary key of the index is the same as the primary key/partition key of the table, just as shown below. Then you must select an alternative sort key which is different from the sort key of the table.

    When you request a strongly consistent read, DynamoDB returns a response with the most up-to-date data, reflecting the updates from all prior write operations that were successful. A strongly consistent read might not be available if there is a network delay or outage. Strongly consistent reads are not supported on global secondary indexes.

    The primary key of a local secondary index must be composite (partition key and sort key). A local secondary index lets you query over a single partition, as specified by the partition key value in the query.

    Local secondary indexes are created at the same time that you create a table. You cannot add a local secondary index to an existing table, nor can you delete any local secondary indexes that currently exist.

  • Optimistic locking

    Optimistic locking is a strategy to ensure that the client-side item that you are updating (or deleting) is the same as the item in DynamoDB. If you use this strategy, then your database writes are protected from being overwritten by the writes of others — and vice-versa. Take note that:

    - DynamoDB global tables use a “last writer wins” reconciliation between concurrent updates. If you use Global Tables, last writer policy wins. So in this case, the locking strategy does not work as expected.

    - DynamoDBMapper transactional operations do not support optimistic locking.

    With optimistic locking, each item has an attribute that acts as a version number. If you retrieve an item from a table, the application records the version number of that item. You can update the item, but only if the version number on the server side has not changed. If there is a version mismatch, it means that someone else has modified the item before you did; the update attempt fails, because you have a stale version of the item. If this happens, you simply try again by retrieving the item and then attempting to update it. Optimistic locking prevents you from accidentally overwriting changes that were made by others; it also prevents others from accidentally overwriting your changes.

    Since the application is already using the AWS SDK for Java, it can support optimistic locking by simply adding the @DynamoDBVersionAttribute annotation to the objects. In the mapping class for your table, you designate one property to store the version number, and mark it using this annotation. When you save an object, the corresponding item in the DynamoDB table will have an attribute that stores the version number. The DynamoDBMapper assigns a version number when you first save the object, and it automatically increments the version number each time you update the item. Your update or delete requests will succeed only if the client-side object version matches the corresponding version number of the item in the DynamoDB table.

  • CRUD functionality

    Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. Since fast performance is one of the requirements asked in the scenario, DynamoDB should be option to consider.

    In DynamoDB, an item is a collection of attributes. Each attribute has a name and a value. An attribute value can be a scalar, a set, or a document type. DynamoDB provides four operations for basic create/read/update/delete (CRUD) functionality:

    PutItem – create an item.

    GetItem – read an item.

    UpdateItem – update an item.

    DeleteItem – delete an item.

    You can use the UpdateItem operation to implement an atomic counter—a numeric attribute that is incremented, unconditionally, without interfering with other write requests. With an atomic counter, the numeric value will increment each time you call UpdateItem.

    For example, you might use an atomic counter to keep track of the number of visitors to a website. In this case, your application would increment a numeric value, regardless of its current value. If an UpdateItem operation should fail, the application could simply retry the operation. This would risk updating the counter twice, but you could probably tolerate a slight overcounting or undercounting of website visitors.

  • Atomic counter

    You might use an atomic counter to keep track of the number of visitors to a website. In this case, your application would increment a numeric value, regardless of its current value.

  • Use case

    Today's applications have more demanding requirements than ever before. For example, an online game might start out with just a few users and a very small amount of data. However, if the game becomes successful, it can easily outstrip the resources of the underlying database management system. It is not uncommon for web-based applications to have hundreds, thousands, or millions of concurrent users, with terabytes or more of new data generated per day. Databases for such applications must handle tens (or hundreds) of thousands of reads and writes per second.

  • DynamoDB stream data retrieval time

    A DynamoDB stream is an ordered flow of information about changes to items in an Amazon DynamoDB table. When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table.

    Whenever an application creates, updates, or deletes items in the table, DynamoDB Streams writes a stream record with the primary key attribute(s) of the items that were modified. A stream record contains information about a data modification to a single item in a DynamoDB table. You can configure the stream so that the stream records capture additional information, such as the "before" and "after" images of modified items.

    All data in DynamoDB Streams is subject to a 24 hour lifetime. You can retrieve and analyze the last 24 hours of activity for any given table; however, data older than 24 hours is susceptible to trimming (removal) at any moment.

    If you disable a stream on a table, the data in the stream will continue to be readable for 24 hours. After this time, the data expires and the stream records are automatically deleted. Note that there is no mechanism for manually deleting an existing stream; you just need to wait until the retention limit expires (24 hours), and all the stream records will be deleted.

  • Return consumed capacity

    To create, update, or delete an item in a DynamoDB table, use one of the following operations:

    • PutItem
    • UpdateItem
    • DeleteItem

    For each of these operations, you need to specify the entire primary key, not just part of it. For example, if a table has a composite primary key (partition key and sort key), you must supply a value for the partition key and a value for the sort key.

    To return the number of write capacity units consumed by any of these operations, set the ReturnConsumedCapacity parameter to one of the following:

    TOTAL — returns the total number of write capacity units consumed.

    INDEXES — returns the total number of write capacity units consumed, with subtotals for the table and any secondary indexes that were affected by the operation.

    NONE — no write capacity details are returned. (This is the default.)

  • Conditional write

    By default, the DynamoDB write operations (PutItem, UpdateItem, DeleteItem) are unconditional: each of these operations will overwrite an existing item that has the specified primary key.

    DynamoDB optionally supports conditional writes for these operations. A conditional write will succeed only if the item attributes meet one or more expected conditions. Otherwise, it returns an error.

    Conditional writes are helpful in many situations. For example, you might want a PutItem operation to succeed only if there is not already an item with the same primary key. Or you could prevent an UpdateItem operation from modifying an item if one of its attributes has a certain value. Conditional writes are helpful in cases where multiple users attempt to modify the same item.

  • Table Key

    DynamoDB uses the CMK to generate and encrypt a unique data key for the table, known as the table key.

    With DynamoDB, AWS Owned, or AWS Managed CMK can be used to generate & encrypt keys.

    AWS Owned CMK is free of charge while AWS Managed CMK is chargeable.

    Customer managed CMK’s are not supported with encryption at rest. "With encryption at rest, DynamoDB transparently encrypts all customer data in a DynamoDB table, including its primary key and local and global secondary indexes, whenever the table is persisted to disk

    đź’ˇ
    Mainly used to encrypt Global secondary index
đź’ˇ
Serverside encryption is enabled at dynamodb by default
đź’ˇ
For guaranteed peformance gain from milliseconds to microseconds choose DAX

RDS

  • Intro

    Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the AWS Cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks.

  • Transparent Data Encryption (TDE)

    Amazon RDS supports using Transparent Data Encryption (TDE) to encrypt stored data on your DB instances running Microsoft SQL Server. TDE automatically encrypts data before it is written to storage, and automatically decrypts data when the data is read from storage.

    Amazon RDS supports TDE for the following SQL Server versions and editions:

    • SQL Server 2017 Enterprise Edition
    • SQL Server 2016 Enterprise Edition
    • SQL Server 2014 Enterprise Edition
    • SQL Server 2012 Enterprise Edition
    • SQL Server 2008 R2 Enterprise Edition

    To enable transparent data encryption for an RDS SQL Server DB instance, specify the TDE option in an RDS option group that is associated with that DB instance.

    Transparent data encryption for SQL Server provides encryption key management by using a two-tier key architecture. A certificate, which is generated from the database master key, is used to protect the data encryption keys. The database encryption key performs the actual encryption and decryption of data on the user database. Amazon RDS backs up and manages the database master key and the TDE certificate. To comply with several security standards, Amazon RDS is working to implement automatic periodic master key rotation.

  • Read Replicas

    Amazon RDS Read Replicas provide enhanced performance and durability for RDS database (DB) instances.

    They make it easy to elastically scale out beyond the capacity constraints of a single DB instance for read-heavy database workloads.

    You can create one or more replicas of a given source DB Instance and serve high-volume application read traffic from multiple copies of your data, thereby increasing aggregate read throughput.

    Read replicas can also be promoted when needed to become standalone DB instances. Read replicas are available in Amazon RDS for MySQL, MariaDB, PostgreSQL, Oracle, and SQL Server as well as Amazon Aurora.

    For the MySQL, MariaDB, PostgreSQL, Oracle, and SQL Server database engines, Amazon RDS creates a second DB instance using a snapshot of the source DB instance.

    It then uses the engines' nativeasynchronous replication to update the read replica whenever there is a change to the source DB instance.

    The read replica operates as a DB instance that allows only read-only connections; applications can connect to a read replica just as they would to any DB instance. Amazon RDS replicates all databases in the source DB instance.

    Amazon Aurora further extends the benefits of read replicas by employing an SSD-backed virtualized storage layer purpose-built for database workloads. Amazon Aurora replicas share the same underlying storage as the source instance, lowering costs and avoiding the need to copy data to the replica nodes. For more information about replication with Amazon Aurora

  • Encryption @ rest

    Ensure to encrypt the cluster

  • Log mechanism
    • Error log – contains diagnostic messages generated by the database engine, along with startup and shutdown times.
    • General query log – contains a record of all SQL statements received from clients, and also client connect and disconnect times.
    • Slow query log – contains a record of SQL statements that took longer to execute than a set amount of time and that examined more than a defined number of rows. Both thresholds are configurable.
đź’ˇ
If you need to handle complex query then go with RDS

Redshift

Amazon Redshift is an Internet hosting service and data warehouse product which forms part of the larger cloud-computing platform Amazon Web Services. It is built on top of technology from the massive parallel processing data warehouse company ParAccel, to handle large scale data sets and database migrations.

Elasticache

  • Session Management
    • Sticky Sessions with Local Session Caching

      Sticky sessions, also known as session affinity, allow you to route a site user to the particular web server that is managing that individual user’s session.

      The session’s validity can be determined by a number of methods, including a client-side cookies or via configurable duration parameters that can be set at the load balancer which routes requests to the web servers.

      Advantages: Cost effective due to the fact you are storing sessions on the same web servers running your applications and that retrieval of those sessions is generally fast because it eliminates network latency.

      Drawback In the event of a failure, you are likely to lose the sessions that were resident on the failed node.

      In addition, in the event the number of your web servers change, for example a scale-up scenario, it’s possible that the traffic may be unequally spread across the web servers as active sessions may exist on particular servers. If not mitigated properly, this can hinder the scalability of your applications.

    • Distributed Session Management
    • ElastiCache is the perfect solution for managing session state.
    • In order to address scalability and to provide a shared data storage for sessions that can be accessible from any individual web server, you can abstract the HTTP sessions from the web servers themselves. A common solution to for this is to leverage an In-Memory Key/Value store such as Redis and Memcached.
    ElastiCache is the perfect solution for managing session state. This is also given in the AWS Documentation</p>
    
    <p>In order to address scalability and to provide a shared data storage for sessions that can be accessible from any individual web server, you can abstract the HTTP sessions from the web servers themselves. A common solution to for this is to leverage an&nbsp;In-Memory Key/Value store&nbsp;such as&nbsp;Redis&nbsp;and&nbsp;Memcached.</p>
    
    
  • Redis vs Memcache
  • Redis

  • Parameter Groups

    Cache parameter groups are an easy way to manage runtime settings for supported engine software. Parameters are used to control memory usage, eviction policies, item sizes, and more.

    An ElastiCache parameter group is a named collection of engine-specific parameters that you can apply to a cluster. By doing this, you make sure that all of the nodes in that cluster are configured in exactly the same way.

KMS

Always use Symmetric CMK to encrypt data in other services eg: Data in S3 bucket

  • Envelope encryption

    When you encrypt your data, your data is protected, but you have to protect your encryption key. One strategy is to encrypt it. Envelope encryption is the practice of encrypting plaintext data with a data key, and then encrypting the data key under another key.

    You can even encrypt the data encryption key under another encryption key, and encrypt that encryption key under another encryption key. But, eventually, one key must remain in plaintext so you can decrypt the keys and your data. This top-level plaintext key encryption key is known as the master key.

  • CMK (Customer Master Key)

    AWS KMS helps you to protect your master keys by storing and managing them securely. Master keys stored in AWS KMS, known as customer master keys (CMKs), never leave the AWS KMS FIPS validated hardware security modules unencrypted. To use an AWS KMS CMK, you must call AWS KMS.

  • Cipherkey blob is the only required mandatory data to encrypt

    cipherkeyblob is the txt that to be encrypted. Blob is the metadata about the text

  • Grants

    A grant is another mechanism for providing permissions. It's an alternative to key policies. Because grants can be very specific, and are easy to create and revoke, they are often used to provide temporary permissions or more granular permissions.

  • Grant token

    When you create a grant, the permissions specified in the grant might not take effect immediately due to eventual consistency. If you need to mitigate the potential delay, use the grant token that you receive in the response to your CreateGrant request. You can pass the grant token with some AWS KMS API requests to make the permissions in the grant take effect immediately.

  • Generate Data key without plaintext

    GenerateDataKeyWithoutPlaintext is identical to the GenerateDataKey operation except that returns only the encrypted copy of the data key. This operation is useful for systems that need to encrypt data at some point, but not immediately. When you need to encrypt the data, you call the Decrypt operation on the encrypted copy of the key.

    It's also useful in distributed systems with different levels of trust. For example, you might store encrypted data in containers. One component of your system creates new containers and stores an encrypted data key with each container. Then, a different component puts the data into the containers. That component first decrypts the data key, uses the plaintext data key to encrypt data, puts the encrypted data into the container, and then destroys the plaintext data key. In this system, the component that creates the containers never sees the plaintext data key.

    GenerateDataKeyWithoutPlaintext returns a unique data key for each request. The bytes in the keys are not related to the caller or CMK that is used to encrypt the private key.

    To generate a data key, you must specify the symmetric customer master key (CMK) that is used to encrypt the data key. You cannot use an asymmetric CMK to generate a data key.

Assymetric encryption wont work with default enctryption type, you need to mention the type.

Lambda

  • Why Lambda?

    Serverless, easy to manage and autoscale

    Free tier of 1,000,000 AWS Lambda requests and 400,000 GBs of compute time

    per function can support upto 3 GB of RAM

  • Pricing

    Pay per calls:

    1st 1 million calls are free 0.20$ per 1 million request thereafter( $0.0000002 per request)

    Pay per duration

    If free tier 400,000GB seconds of compute time is free per month

    1$ for 600,000 GBseconds

  • Lambda synchronous invocations

    CLI, SDK , API gateway and ALB

    Response will be served immediately after requested.

    Error to be handled at client end

  • Lambda function with ALB

    Lambda to be mapped to Target group to access it via ALB

  • ALB multiheader flows
  • Lambda @ Edge

    Cloudfront

    You can also generate response to users at cloudfront without passing to backend

  • Lambda @ Edge Usecases

    Web security & privacy

    SEO

    A/B Testing

    Bot mitigation at edge

    User prioritisation

    User Authentication & Authorization

  • Lambda Asynchronous invocations

    S3, SNS , Cloudwatch events

    Events are planned in Event Queue

    Lambda attempts to retry on errors

    > 3 tries total

    > 1 minute wait after 1st then 2 minute waits

  • Lambda by default

    By default it's launched outside VPC

  • Lambda in VPC

    You need to provide your subnet and SG Lambda will create its ENI

    AWSLambdaVPCAccessExecutionRole

  • Versioning

    You can use versions to manage the deployment of your functions. For example, you can publish a new version of a function for beta testing without affecting users of the stable production version. Lambda creates a new version of your function each time that you publish the function. The new version is a copy of the unpublished version of the function.

    A function version includes the following information:

    • The function code and all associated dependencies.
    • The Lambda runtime that invokes the function.
    • All of the function settings, including the environment variables.
    • A unique Amazon Resource Name (ARN) to identify the specific version of the function.

    You can change the function code and settings only on the unpublished version of a function. When you publish a version, the code and most of the settings are locked to maintain a consistent experience for users of that version.

  • Environment variables
    • You can use environment variables to adjust your function's behavior without updating code. An environment variable is a pair of strings that are stored in a function's version-specific configuration. The Lambda runtime makes environment variables available to your code and sets additional environment variables that contain information about the function and invocation request.
    • You set environment variables on the unpublished version of your function by specifying a key and value. When you publish a version, the environment variables are locked for that version along with other version-specific configuration.
  • Handler

    The handler is the name of the method within a code that Lambda calls to execute the function

  • Best practices for function

    Separate the Lambda handler from your core logic. This allows you to make a more unit-testable function. In Node.js this may look like:

    exports.myHandler = function(event, context, callback) {	var foo = event.foo;	var bar = event.bar;	var result = MyLambdaFunction (foo, bar);	callback(null, result);
    }
    function MyLambdaFunction (foo, bar) {	// MyLambdaFunction logic here
    }
    

    Take advantage of execution environment reuse to improve the performance of your function. Initialize SDK clients and database connections outside of the function handler, and cache static assets locally in the /tmp directory. Subsequent invocations processed by the same instance of your function can reuse these resources. This saves execution time and cost.

    To avoid potential data leaks across invocations, don’t use the execution environment to store user data, events, or other information with security implications. If your function relies on a mutable state that can’t be stored in memory within the handler, consider creating a separate function or separate versions of a function for each user.

    Use a keep-alive directive to maintain persistent connections. Lambda purges idle connections over time. Attempting to reuse an idle connection when invoking a function will result in a connection error. To maintain your persistent connection, use the keep-alive directive associated with your runtime. For an example, see Reusing Connections with Keep-Alive in Node.js

    Use environment variables to pass operational parameters to your function. For example, if you are writing to an Amazon S3 bucket, instead of hard-coding the bucket name you are writing to, configure the bucket name as an environment variable.

    Control the dependencies in your function's deployment package. The AWS Lambda execution environment contains a number of libraries such as the AWS SDK for the Node.js and Python runtimes (a full list can be found here: AWS Lambda runtimes). To enable the latest set of features and security updates, Lambda will periodically update these libraries. These updates may introduce subtle changes to the behavior of your Lambda function. To have full control of the dependencies your function uses, package all of your dependencies with your deployment package.

    Minimize your deployment package size to its runtime necessities. This will reduce the amount of time that it takes for your deployment package to be downloaded and unpacked ahead of invocation. For functions authored in Java or .NET Core, avoid uploading the entire AWS SDK library as part of your deployment package. Instead, selectively depend on the modules which pick up components of the SDK you need (e.g. DynamoDB, Amazon S3 SDK modules and Lambda core libraries

    Reduce the time it takes Lambda to unpack deployment packages authored in Java by putting your dependency .jar files in a separate /lib directory. This is faster than putting all your function’s code in a single jar with a large number of .class files. See AWS Lambda deployment package in Java for instructions.

    Minimize the complexity of your dependencies. Prefer simpler frameworks that load quickly on execution environment startup. For example, prefer simpler Java dependency injection (IoC) frameworks like Dagger

    Avoid using recursive code in your Lambda function, wherein the function automatically calls itself until some arbitrary criteria is met. This could lead to unintended volume of function invocations and escalated costs. If you do accidentally do so, set the function reserved concurrency to 0 immediately to throttle all invocations to the function, while you update the code.

  • Lambda with resources in VPC

    AWS Lambda runs your function code securely within a VPC by default. However, to enable your Lambda function to access resources inside your private VPC, you must provide additional VPC-specific configuration information that includes VPC subnet IDs and security group IDs. AWS Lambda uses this information to set up elastic network interfaces (ENIs) that enable your function to connect securely to other resources within your private VPC.

  • application logging

    You can insert logging statements into your code to help you validate that your code is working as expected. Lambda automatically integrates with Amazon CloudWatch Logs and pushes all logs from your code to a CloudWatch Logs group associated with a Lambda function (/aws/lambda/). 

  • Handling code with external libraries

    Minimize your deployment package size to its runtime necessities.This will reduce the amount of time that it takes for your deployment package to be downloaded and unpacked ahead of invocation.

    For functions authored in Java or .NET Core, avoid uploading the entire AWS SDK library as part of your deployment package. Instead, selectively depend on the modules which pick up components of the SDK you need

  • Best Practices of Lambda
    • Separate the Lambda handler from your core logic.

      This allows you to make a more unit-testable function. In Node.js this may look like:

    • Take advantage of execution environment reuse to improve the performance of your function.

      Initialize SDK clients and database connections outside of the function handler, and cache static assets locally in the /tmp directory. Subsequent invocations processed by the same instance of your function can reuse these resources. This saves cost by reducing function run time.

      To avoid potential data leaks across invocations, don’t use the execution environment to store user data, events, or other information with security implications. If your function relies on a mutable state that can’t be stored in memory within the handler, consider creating a separate function or separate versions of a function for each user.

    • Use a keep-alive directive to maintain persistent connections.
      • Lambda purges idle connections over time. Attempting to reuse an idle connection when invoking a function will result in a connection error. To maintain your persistent connection, use the keep-alive directive associated with your runtime. For an example, see Reusing Connections with Keep-Alive in Node.js
    • . Use environment variables to pass operational parameters to your function.
      • For example, if you are writing to an Amazon S3 bucket, instead of hard-coding the bucket name you are writing to, configure the bucket name as an environment variable.
    • Control the dependencies in your function's deployment package.
      • The AWS Lambda execution environment contains a number of libraries such as the AWS SDK for the Node.js and Python runtimes (a full list can be found here: AWS Lambda runtimes). To enable the latest set of features and security updates, Lambda will periodically update these libraries. These updates may introduce subtle changes to the behavior of your Lambda function. To have full control of the dependencies your function uses, package all of your dependencies with your deployment package.
    • Minimize your deployment package size to its runtime necessities.
      • This will reduce the amount of time that it takes for your deployment package to be downloaded and unpacked ahead of invocation. For functions authored in Java or .NET Core, avoid uploading the entire AWS SDK library as part of your deployment package. Instead, selectively depend on the modules which pick up components of the SDK you need (e.g. DynamoDB, Amazon S3 SDK modules and Lambda core libraries
    • Reduce the time it takes Lambda to unpack deployment packages
      • authored in Java by putting your dependency .jar files in a separate /lib directory. This is faster than putting all your function’s code in a single jar with a large number of .class files. See AWS Lambda deployment package in Java for instructions.
    • Minimize the complexity of your dependencies.
    • Avoid using recursive code in your Lambda function
      • wherein the function automatically calls itself until some arbitrary criteria is met. This could lead to unintended volume of function invocations and escalated costs. If you do accidentally do so, set the function reserved concurrency to 0 immediately to throttle all invocations to the function, while you update the code.
  • Trigger Layer & Destination
    • Triggers –
      • Triggers are services and resources that you have configured to invoke your function. Choose Add trigger to create a Lambda event source mapping or to configure a trigger in another service that the Lambda console integrates with. For details about these services and others, see Using AWS Lambda with other services.
    • Layers –
      • Choose the Layers node to add layers to your application. A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies.
    • Destinations –
      • Add a destination to your function to send details about invocation results to another service. You can send invocation records when your function is invoked asynchronously, or by an event source mapping that reads from a stream.
  • Function settings
    • Code – The code and dependencies of your function. For scripting languages, you can edit your function code in the embedded editor. To add libraries, or for languages that the editor doesn't support, upload a deployment package. If your deployment package is larger than 50 MB, choose Upload a file from Amazon S3.
    • Handler – The method that the runtime runs when your function is invoked, such as index.handler. The first value is the name of the file or module. The second value is the name of the method.
    • Environment variables – Key-value pairs that Lambda sets in the execution environment. Use environment variables to extend your function's configuration outside of code.
    • Tags – Key-value pairs that Lambda attaches to your function resource. Use tags to organize Lambda functions into groups for cost reporting and filtering in the Lambda console.

      Tags apply to the entire function, including all versions and aliases.

    • Execution role – The IAM role that AWS Lambda assumes when it runs your function.
    • Description – A description of the function.
    • Memory– The amount of memory available to the function while it is running. Choose an amount between 128 MB and 3,008 MB in 64-MB increments.

      Lambda allocates CPU power linearly in proportion to the amount of memory configured. At 1,792 MB, a function has the equivalent of one full vCPU (one vCPU-second of credits per second).

    • Timeout – The amount of time that Lambda allows a function to run before stopping it. The default is 3 seconds. The maximum allowed value is 900 seconds.
    • Virtual private cloud (VPC) – If your function needs network access to resources that are not available over the internet, configure it to connect to a VPC.
    • Concurrency – Reserve concurrency for a function to set the maximum number of simultaneous executions for a function. Provision concurrency to ensure that a function can scale without fluctuations in latency.

      Reserved concurrency applies to the entire function, including all versions and aliases.

    • Asynchronous invocation – Configure error handling behavior to reduce the number of retries that Lambda attempts, or the amount of time that unprocessed events stay queued before Lambda discards them. Configure a dead-letter queue to retain discarded events.

      You can configure error handling settings on a function, version, or alias.

  • Lambda with X-Ray

    Ensure Lambda API is assigned an IAM role that has permission to access X-Ray service

  • Cloudwatch for Lambda monitoring

    AWS Lambda monitors functions on your behalf and sends metrics to Amazon CloudWatch. The metrics include total requests, duration, and error rates. The Lambda console creates graphs for these metrics and shows them on the Monitoring page for each function.

  • Lambda monitoring graphs in Lambda
    • Invocations – The number of times that the function was invoked in each 5-minute period.
    • Duration – The average, minimum, and maximum execution times.
    • Error count and success rate (%) – The number of errors and the percentage of executions that completed without error.
    • Throttles – The number of times that execution failed due to concurrency limits.
    • IteratorAge – For stream event sources, the age of the last item in the batch when Lambda received it and invoked the function.
    • Async delivery failures – The number of errors that occurred when Lambda attempted to write to a destination or dead-letter queue.
    • Concurrent executions – The number of function instances that are processing events.
  • Default lambda setting

    Timeout 3 seconds Memory 128 MB

  • Lambda Alias

    You can create one or more aliases for your Lambda function. A Lambda alias is like a pointer to a specific function version. Users can access the function version using the alias Amazon Resource Name (ARN).

    The following example creates a Lambda function alias named routing-alias that points to version 1 of the function. Version 2 of the function receives 3 percent of the traffic. The remaining 97 percent of traffic is routed to version 1.

    $ aws lambda create-alias --name routing-alias --function-name my-function --function-version 1  \
    --routing-config AdditionalVersionWeights={"2"=0.03}

    Use the update-alias command to increase the percentage of incoming traffic to version 2. In the following example, you increase the traffic to 5 percent.

    $ aws lambda update-alias --name routing-alias --function-name my-function  \
    --routing-config AdditionalVersionWeights={"2"=0.05}
  • Troubleshooting Lambda error

    AWS Lambda can occasionally experience transient service errors. In this case, invoking Lambda results in a 500 error, such as ServiceException, AWSLambdaException, or SdkClientException. As a best practice, proactively handle these exceptions in your state machine to Retry invoking your Lambda function, or to Catch the error.

    đź’ˇ
    First do retry if even failed after certain retry catch the error
  • Lambda with VPC internet

    If your Lambda function needs Internet access, just as described in this scenario, do not attach it to a public subnet or to a private subnet without Internet access. Instead, attach it only to private subnets with Internet access through a NAT instance or add a NAT gateway to your VPC. You should also ensure that the associated security group of the Lambda function allows outbound connections.

  • Lambda function with CLI

    To create a Lambda function, you need a deployment package and an execution role. The deployment package contains your function code. The execution role grants the function permission to use AWS services, such as Amazon CloudWatch Logs for log streaming and AWS X-Ray for request tracing. You can use the CreateFunction API via the AWS CLI or the AWS SDK of your choice.

    A function has an unpublished version, and can have published versions and aliases. The unpublished version changes when you update your function's code and configuration. A published version is a snapshot of your function code and configuration that can't be changed. An alias is a named resource that maps to a version, and can be changed to map to a different version.

    The InvalidParameterValueException will be returned if one of the parameters in the request is invalid. For example, if you provided an IAM role in the CreateFunction API which AWS Lambda is unable to assume.

    If you have exceeded your maximum total code size per account, the CodeStorageExceededException will be returned.

    If the resource already exists, the ResourceConflictException will be returned and not InvalidParameterValueException.

    If the AWS Lambda service encountered an internal error, the ServiceException will be returned .

  • Lambda limitation

    Can be only 1000 concurrent executions

    Throttling error - 429

    Server error - 500

    AWS SDK comes by default on each lambda function

    You also need to zip the dependant modules while uploading your code

    Lambda aliase can't refer to another aliase

  • Lambda best practice
  • Concurrency limit on individual lambda function

    You can now set a concurrency limit on individual AWS Lambda functions. The concurrency limit you set will reserve a portion of your account level concurrency limit for a given function. This feature allows you to throttle a given function if it reaches a maximum number of concurrent executions allowed, which you can choose to set. This is useful when you want to limit traffic rates to downstream resources called by Lambda (e.g. databases) or if you want to control the consumption of elastic network interfaces (ENI) and IP addresses for functions accessing a private VPC

    Over all concurrency on account is 1000

  • Lambda performance metrics
    • Duration – The amount of time that your function code spends processing an event. For the first event processed by an instance of your function, this includes initialization time. The billed duration for an invocation is the value of Duration rounded up to the nearest 100 milliseconds.
    • IteratorAge – For event source mappings that read from streams, the age of the last record in the event. The age is the amount of time between when the stream receives the record and when the event source mapping sends the event to the function.
  • Lambda environment variables
    • _X_AMZN_TRACE_ID:

      Contains the tracing header, which includes the sampling decision, trace ID, and parent segment ID. If Lambda receives a tracing header when your function is invoked, that header will be used to populate the _X_AMZN_TRACE_ID environment variable. If a tracing header was not received, Lambda will generate one for you.

    • _X_AMZN_TRACE_ID

      Contains the tracing header, which includes the sampling decision, trace ID, and parent segment ID. If Lambda receives a tracing header when your function is invoked, that header will be used to populate the _X_AMZN_TRACE_ID environment variable. If a tracing header was not received, Lambda will generate one for you.

    • AWS_XRAY_CONTEXT_MISSING

      The X-Ray SDK uses this variable to determine its behavior in the event that your function tries to record X-Ray data, but a tracing header is not available. Lambda sets this value to LOG_ERROR&nbsp;by default.

    • AWS_XRAY_DAEMON_ADDRESS

      This environment variable exposes the X-Ray daemon&#39;s address in the following format:&nbsp;IP_ADDRESS:PORT. You can use the X-Ray daemon&#39;s address to send trace data to the X-Ray daemon directly, without using the X-Ray SDK.

  • Execution context

    When AWS Lambda executes your Lambda function, it provisions and manages the resources needed to run your Lambda function. When you create a Lambda function, you specify configuration information, such as the amount of memory and maximum execution time that you want to allow for your Lambda function. When a Lambda function is invoked, AWS Lambda launches an execution context based on the configuration settings you provide. The execution context is a temporary runtime environment that initializes any external dependencies of your Lambda function code, such as database connections or HTTP endpoints. This affords subsequent invocations better performance because there is no need to "cold-start" or initialize those external dependencies, as explained below.

    It takes time to set up an execution context and do the necessary "bootstrapping", which adds some latency each time the Lambda function is invoked. You typically see this latency when a Lambda function is invoked for the first time or after it has been updated because AWS Lambda tries to reuse the execution context for subsequent invocations of the Lambda function.

    After a Lambda function is executed, AWS Lambda maintains the execution context for some time in anticipation of another Lambda function invocation. In effect, the service freezes the execution context after a Lambda function completes, and thaws the context for reuse, if AWS Lambda chooses to reuse the context when the Lambda function is invoked again. This execution context reuse approach has the following implications:

    • Any declarations in your Lambda function code (outside the handler code, see Programming Model) remains initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations. It is recommended to add logic in your code to check if a connection exists before creating one.
    • Each execution context provides 512 MB of additional disk space in the /tmp directory. The directory content remains when the execution context is frozen, providing transient cache that can be used for multiple invocations. You can add extra code to check if the cache has the data that you stored.
    • Background processes or callbacks initiated by your Lambda function that did not complete when the function ended resume if AWS Lambda chooses to reuse the execution context. You should make sure any background processes or callbacks (in case of Node.js) in your code are complete before the code exits.
  • Lambda Invocation with API

    AWS Lambda supports synchronous and asynchronous invocation of a Lambda function. You can control the invocation type only when you invoke a Lambda function (referred to as on-demand invocation). The following examples illustrate on-demand invocations:

    • Your custom application invokes a Lambda function.
    • You manually invoke a Lambda function (for example, using the AWS CLI) for testing purposes.

    In both cases, you invoke your Lambda function using the Invoke operation, and you can specify the invocation type as synchronous or asynchronous.

    When you use AWS services as a trigger, the invocation type is predetermined for each service. You have no control over the invocation type that these event sources use when they invoke your Lambda function.

    In the Invoke API, you have 3 options to choose from for the InvocationType:

    RequestResponse (default) - Invoke the function synchronously. Keep the connection open until the function returns a response or times out. The API response includes the function response and additional data.

    Event - Invoke the function asynchronously. Send events that fail multiple times to the function's dead-letter queue (if it's configured). The API response only includes a status code.

    DryRun - Validate parameter values and verify that the user or role has permission to invoke the function.

  • Types on Lambda Autherizer

    A Lambda authorizer is an API Gateway feature that uses a Lambda function to control access to your API. When a client makes a request to one of your API's methods, API Gateway calls your Lambda authorizer, which takes the caller's identity as input and returns an IAM policy as output.

    There are two types of Lambda authorizers:

    - A token-based Lambda authorizer (also called a TOKEN authorizer) receives the caller's identity in a bearer token, such as a JSON Web Token (JWT) or an OAuth token.

    - A request parameter-based Lambda authorizer (also called a REQUEST authorizer) receives the caller's identity in a combination of headers, query string parameters, stageVariables, and $context variables.

    It is possible to use an AWS Lambda function from an AWS account that is different from the one in which you created your Lambda authorizer function by using a Cross-Account Lambda Authorizer.

  • Lambda concurrency

    The unit of scale for AWS Lambda is a concurrent execution. However, scaling indefinitely is not desirable in all scenarios. For example, you may want to control your concurrency for cost reasons or to regulate how long it takes you to process a batch of events, or to simply match it with a downstream resource. To assist with this, Lambda provides a concurrent execution limit control at both the account level and the function level.

    The concurrent executions refers to the number of executions of your function code that are happening at any given time. You can estimate the concurrent execution count, but the concurrent execution count will differ depending on whether or not your Lambda function is processing events from a poll-based event source.

    If you create a Lambda function to process events from event sources that aren't poll-based (for example, Lambda can process every event from other sources, like Amazon S3 or API Gateway), each published event is a unit of work, in parallel, up to your account limits. Therefore, the number of invocations these event sources make influences the concurrency.

    If you set the concurrent execution limit for a function, the value is deducted from the unreserved concurrency pool. For example, if your account's concurrent execution limit is 1000 and you have 10 functions, you can specify a limit on one function at 200 and another function at 100. The remaining 700 will be shared among the other 8 functions.

    AWS Lambda will keep the unreserved concurrency pool at a minimum of 100 concurrent executions, so that functions that do not have specific limits set can still process requests. So, in practice, if your total account limit is 1000, you are limited to allocating 900 to individual functions.

If your application uses the AWS Lambda compute platform, the AppSpec file can be formatted with either YAML or JSON. It can also be typed directly into an editor in the console. The AppSpec file is used to specify

<<FAQ and developer guide>>

VPC

  • VPC Flow logs

    VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. Flow log data can be published to Amazon CloudWatch Logs and Amazon S3. After you've created a flow log, you can retrieve and view its data in the chosen destination.

    Flow logs can help you with a number of tasks; for example, to troubleshoot why specific traffic is not reaching an instance, which in turn helps you diagnose overly restrictive security group rules. You can also use flow logs as a security tool to monitor the traffic that is reaching your instance. CloudWatch Logs charges apply when using flow logs, whether you send them to CloudWatch Logs or to Amazon S3.

  • Route table

    A route table contains a set of rules, called routes, that are used to determine where network traffic is directed.

    Each subnet in your VPC must be associated with a route table; the table controls the routing for the subnet. A subnet can only be associated with one route table at a time, but you can associate multiple subnets with the same route table.

đź’ˇ
default route limit per VPC is 200.

X-Ray

  • Provides visual analysis for application
  • X-Ray advantages

    See application architecture

    Know which application is impacted and exact users who are impacted.

    Enables to ensure are we meeting SLA

  • X-Ray compatibility

    Compatible with AWS Lambda Elastic Beanstalk ECS ELB API Gateway EC2 instances or any application server

  • X-Ray Tracing

    Tracing is end-end following the request

    Tracing is made up of segments (+ subsegments)

    You can trace Entire Request or Sample of request % for example

    X-Ray compiles and processes segment documents to generate queryable trace summaries and full traces that you can access by using the GetTraceSummaries and BatchGetTraces APIs, respectively. In addition to the segments and subsegments that you send to X-Ray, the service uses information in subsegments to generate inferred segments and adds them to the full trace. Inferred segments represent downstream services and resources in the service map.

  • Service map

    Service map collects data from all segments and shown as visualised

  • Trouble shooting
    • If not working on EC2

      Check IAM role on ec2

      Check is Ec2 running on X-Ray daemon

    • If not working on Lambda

      Check IAM on lambda

      Ensure X-Ray is imported in code

  • X-Ray Instrumentation

    To measure performance of application You use X-Ray SDK

  • X-Ray sampling rules

    By default X-Ray sample 1 request per second and 5% of any additional request that are beyond reservoir size

  • X-Ray write API
  • X-Ray read API
  • SQS and X-Ray

    AWS X-Ray integrates with Amazon Simple Queue Service (Amazon SQS) to trace messages that are passed through an Amazon SQS queue. If a service traces requests by using the X-Ray SDK, Amazon SQS can send the tracing header and continue to propagate the original trace from the sender to the consumer with a consistent trace ID. Trace continuity enables users to track, analyze, and debug throughout downstream services.

  • Requests sampled per second

    Reservoir Rate" + Fixed Rate % [(Total Requests - Reservoir Rate)]

    60 + (200-60) * 20
    60 + (140) * 20
    60 + 28
    88

For server related error X-Ray will generate 500 status code

  • X-Ray with Lambda

    You can use AWS X-Ray to trace your AWS Lambda functions. Lambda runs the X-Ray daemon and records a segment with details about the function invocation and execution. For further instrumentation, you can bundle the X-Ray SDK with your function to record outgoing calls and add annotations and metadata.

GetTraceSummaries includes Trace id and Event Time

  • X-Ray daemon on Elastic Beanstalk

    Elastic Beanstalk platforms provide a configuration option that you can set to run the daemon automatically. You can enable the daemon in a configuration file in your source code or by choosing an option in the Elastic Beanstalk console. When you enable the configuration option, the daemon is installed on the instance and runs as a service.

  • X-Ray daemon on ECS

    In Amazon ECS, create a Docker image that runs the X-Ray daemon, upload it to a Docker image repository, and then deploy it to your Amazon ECS cluster. You can use port mappings and network mode settings in your task definition file to allow your application to communicate with the daemon container.

  • To track all requests from a specific user

    Use Annotations Annotations are key-value pairs indexed to use with filter expressions. In the above case, traces for a user A needs to be tracked, for which Annotations can be used along with a Filter expression to find all traces related to that user

  • Segment fields
    • Mandatory fields

      Name Id Trace_id Start_time End_time In_progress

    • Optional fields

      Service User Origin Parent_id http aws error, throttle, fault, and cause annotations metadata subsegments

  • X-Ray SDK provides
    • Interceptors to add to your code to trace incoming HTTP requests
    • Client handlers to instrument AWS SDK clients that your application uses to call other AWS services
    • An HTTP client to use to instrument calls to other internal and external HTTP web services
  • X-Ray SDK

    The AWS X-Ray SDK does not send trace data directly to AWS X-Ray. To avoid calling the service every time your application serves a request, the SDK sends the trace data to a daemon, which collects segments for multiple requests and uploads them in batches. Use a script to run the daemon alongside your application.

    To properly instrument your applications in Amazon ECS, you have to create a Docker image that runs the X-Ray daemon, upload it to a Docker image repository, and then deploy it to your Amazon ECS cluster. You can use port mappings and network mode settings in your task definition file to allow your application to communicate with the daemon container.

    The AWS X-Ray daemon is a software application that listens for traffic on UDP port 2000, gathers raw segment data, and relays it to the AWS X-Ray API. The daemon works in conjunction with the AWS X-Ray SDKs and must be running so that data sent by the SDKs can reach the X-Ray service.

  • Filter expression

    Even with sampling, a complex application generates a lot of data. The AWS X-Ray console provides an easy-to-navigate view of the service graph. It shows health and performance information that helps you identify issues and opportunities for optimization in your application. For advanced tracing, you can drill down to traces for individual requests, or use filter expressions to find traces related to specific paths or users.

    When you instrument your application, the X-Ray SDK records information about incoming and outgoing requests, the AWS resources used, and the application itself. You can add other information to the segment document as annotations and metadata.

    Annotations are simple key-value pairs that are indexed for use with filter expressions. Use annotations to record data that you want to use to group traces in the console, or when calling the GetTraceSummaries API. X-Ray indexes up to 50 annotations per trace.

    Metadata are key-value pairs with values of any type, including objects and lists, but that are not indexed. Use metadata to record data you want to store in the trace but don't need to use for searching traces. You can view annotations and metadata in the segment or subsegment details in the X-Ray console.

    A trace segment is a JSON representation of a request that your application serves. A trace segment records information about the original request, information about the work that your application does locally, and subsegments with information about downstream calls that your application makes to AWS resources, HTTP APIs, and SQL databases.

By default, AWS X-Ray Daemon listens for traffic on UDP port 2000. This port can be changed while configuring daemon with CLI option as – bind Different port number

API Gateway

  • Flow
  • API Integration high level

    Lambda function

    HTTP

    AWS service

  • API gateway endpoint types

    Edge-Optimised - for Global customers

    Regional - for users in same region

    Private - For resources within VPC

  • Stage variables

    You can use stage variables to configure values based on environment

    A stage variable can be used as part of an HTTP integration URL, as shown in the following examples:

    • A full URI without protocol – http://${stageVariables.<variable_name>}
    • A full domain – http://${stageVariables.<variable_name>}/resource/operation
    • A subdomain – http://${stageVariables.<variable_name>}.example.com/resource/operation
    • A path – http://example.com/${stageVariables.<variable_name>}/bar
    • A query string – http://example.com/foo?q=${stageVariables.<variable_name>}
  • Lambda Alias

    Lambda Alias are used to invoke right version of Lambda function Stage variables can be used to refer Lambda Alias and that will trigger right lambda function

  • API Gateway Canary deployment

    This is nothing but splitting traffic to test any new feature you set % of traffic to be passed to

  • Integration type: Mock

    This is will give the response without passing the request to backend

  • Integration AWS_Proxy (Lambda Proxy)

    Request from the client is inout for Lambda Lambda function works based on request of client no value is passed as header / cookie / queryparameter by the enduser.

  • Integration Http_proxy

    Just requests are passed to backend and serve response what ever received from backend

  • Mapping template

    This is used to modify request/response Add header/ query parameter Modify body content Filter output data

  • Json to XML with SOAP
  • Caching at API gateway

    Default cache duration is 300 seconds Max is 3600 seconds Min is 0 seconds

    Cache capacity from 0.5 GB to 237GB

    It's expensive so not to enable cache for dev/test environment

  • Cache Invalidation

    To invalidate Cache-Control:max-age=0

    If cache invalidate policy is not defined then any one can invalidate the case so be aware

  • API keys

    It's alphanumeric

    Throttling is applied per API key

    Quota limit is the overall number of max requests

    User pass the api key as x-api-key header

  • API Cloudwatch metrics
  • API limit

    Throttling is 10000 rps Burst is 5000

  • Gateway errors
  • Gateway- CORS

    To serve request from the other domains

    When a browser receives a non-simple HTTP request, the

    CORS protocol

    requires the browser to send a preflight request to the server and wait for approval (or a request for credentials) from the server before sending the actual request. The preflight request appears to your API as an HTTP request that:

    • Includes an Origin header.
    • Uses the OPTIONS method.
    • Includes the following headers:
      • Access-Control-Request-Method
      • Access-Control-Request-Headers

    To support CORS, therefore, a REST API resource needs to implement an OPTIONS method that can respond to the OPTIONS preflight request with at least the following response headers mandated by the Fetch standard:

    • Access-Control-Allow-Methods
    • Access-Control-Allow-Headers
    • Access-Control-Allow-Origin

  • API security - role access
  • Require Authorization box

    Only authenticated users can invalidate cache

  • Restrict access to IP

    Can be done in WAF or in Resource policy

  • Frontend vs Backend

    As an API developer, you control the behaviors of your API's frontend interactions by configuring the method request and a method response. You control the behaviors of your API's backend interactions by setting up the integration request and integration response. These involve data mappings between a method and its corresponding integration

  • Deploy rest API in API gateway

    In API Gateway, a REST API deployment is represented by a Deployment resource. It's similar to an executable of an API that is represented by a RestApi resource.

    For the client to call your API, you must create a deployment and associate a stage with it. A stage is represented by a Stage resource. It represents a snapshot of the API, including methods, integrations, models, mapping templates, and Lambda authorizers (formerly known as custom authorizers). When you update the API, you can redeploy the API by associating a new stage with the existing deployment. We discuss creating a stage in Setting up a stage for a REST API.

  • Authorizing API calls
    • Resource policies

      let you create resource-based policies to allow or deny access to your APIs and methods from specified source IP addresses or VPC endpoints. For more information, see Controlling access to an API with API Gateway resource policies.

    • Standard AWS IAM roles and policies

      offer flexible and robust access controls that can be applied to an entire API or individual methods. IAM roles and policies can be used for controlling who can create and manage your APIs, as well as who can invoke them. For more information, see Control access to an API with IAM permissions.

    • IAM tags

      can be used together with IAM policies to control access. For more information, see Using tags to control access to API Gateway resources. Endpoint policies

    • Endpoint policies

      for interface VPC endpoints allow you to attach IAM resource policies to interface VPC endpoints to improve the security of your private APIs. For more information, see Use VPC endpoint policies for private APIs in API Gateway.

    • Lambda authorizers

      are Lambda functions that control access to REST API methods using bearer token authentication—as well as information described by headers, paths, query strings, stage variables, or context variables request parameters. Lambda authorizers are used to control who can invoke REST API methods. For more information, see Use API Gateway Lambda authorizers.

    • Amazon Cognito user pools

      let you create customizable authentication and authorization solutions for your REST APIs. Amazon Cognito user pools are used to control who can invoke REST API methods. For more information, see Control access to a REST API using Amazon Cognito User Pools as authorizer.

  • Integration

    Programmatically, you choose an integration type by setting the type property on the Integration resource.

    • For the Lambda proxy integration, the value is AWS_PROXY.
    • For the Lambda custom integration and all other AWS integrations, it is AWS.
    • For the HTTP proxy integration and HTTP integration, the value is HTTP_PROXY and HTTP, respectively.
    • For the mock integration, the type value is MOCK.

    You can integrate an API method in your API Gateway with a custom HTTP endpoint of your application in two ways:

    - HTTP proxy integration

    - HTTP custom integration

  • Metrics

    Monitor the IntegrationLatency metrics to measure the responsiveness of the backend.

    - Monitor the Latency metrics to measure the overall responsiveness of your API calls.

    - Monitor the CacheHitCount and CacheMissCount metrics to optimize cache capacities to achieve a desired performance.

  • 504 error

    A gateway response is identified by a response type defined by API Gateway. The response consists of an HTTP status code, a set of additional headers that are specified by parameter mappings, and a payload that is generated by a non-VTL (Apache Velocity Template Language) mapping template.

    You can set up a gateway response for a supported response type at the API level. Whenever API Gateway returns a response of the type, the header mappings and payload mapping templates defined in the gateway response are applied to return the mapped results to the API caller.

    The following are the Gateway response types which are associated with the HTTP 504 error in API Gateway:

    INTEGRATION_FAILURE - The gateway response for an integration failed error. If the response type is unspecified, this response defaults to the DEFAULT_5XX type.

    INTEGRATION_TIMEOUT - The gateway response for an integration timed out error. If the response type is unspecified, this response defaults to the DEFAULT_5XX type.

    For the integration timeout, the range is from 50 milliseconds to 29 seconds for all integration types, including Lambda, Lambda proxy, HTTP, HTTP proxy, and AWS integrations.

  • Stage variables

SAM(Serverless Application Model)

  • Intro

    The AWS Serverless Application Model (AWS SAM) is an open source framework for building serverless applications. It provides shorthand syntax to express functions, APIs, databases, and event source mappings. You define the application you want with just a few lines per resource and model it using YAML.

    AWS SAM is natively supported by AWS CloudFormation and provides a simplified way of defining the Amazon API Gateway APIs, AWS Lambda functions, and Amazon DynamoDB tables needed by your serverless application. During deployment, SAM transforms and expands the SAM syntax into AWS CloudFormation syntax. Then, CloudFormation provisions your resources with reliable deployment capabilities.

  • SAM works based on cloudformation
  • Simple SAM YAML file can be used to create complex cloudformation template
  • SAM can help you to run Lambda, API gateway , DunamoDB locally
  • SAM and codedeploy
  • Commands to know

    sam build → fetch resources and build local deployment artifacts

    sam package → package and upload to amazon s3, generates cloudformation template

    sam policy templates → for easy IAM policy definition

    SAM is integrated to codedeploy to do deploy to lambda aliases

  • SAM does following for you
    1. Deploys new versions of your Lambda function, and automatically creates aliases that point to the new version.
    1. Gradually shifts customer traffic to the new version until you're satisfied that it's working as expected, or you roll back the update.
    1. Defines pre-traffic and post-traffic test functions to verify that the newly deployed code is configured correctly and your application operates as expected.
    1. Rolls back the deployment if CloudWatch alarms are triggered
  • AutoPublishAlias:

    By adding this property and specifying an alias name, AWS SAM:

    • Detects when new code is being deployed, based on changes to the Lambda function's Amazon S3 URI.
    • Creates and publishes an updated version of that function with the latest code.
    • Creates an alias with a name that you provide (unless an alias already exists), and points to the updated version of the Lambda function. Function invocations should use the alias qualifier to take advantage of this.
  • Deployment Preference Type:

    If you use AWS SAM to create your serverless application, it comes built-in with CodeDeploy to help ensure safe Lambda deployments. There are various deployment preference types that you can choose from.

    For example:

    If you choose Canary10Percent10Minutes then 10 percent of your customer traffic is immediately shifted to your new version. After 10 minutes, all traffic is shifted to the new version.

    However, if your pre-hook/post-hook tests fail, or if a CloudWatch alarm is triggered, CodeDeploy rolls back your deployment. The following table outlines other traffic-shifting options that are available:

    • Canary: Traffic is shifted in two increments. You can choose from predefined canary options. The options specify the percentage of traffic that's shifted to your updated Lambda function version in the first increment, and the interval, in minutes, before the remaining traffic is shifted in the second increment.
    • Linear: Traffic is shifted in equal increments with an equal number of minutes between each increment. You can choose from predefined linear options that specify the percentage of traffic that's shifted in each increment and the number of minutes between each increment.
    • All-at-once: All traffic is shifted from the original Lambda function to the updated Lambda function version at once.
    đź’ˇ
    CodeDeployDefault.HalfAtATime is only applicable for EC2/On-premises compute platform and not for Lambda.
  • Alarms

    These are CloudWatch alarms that are triggered by any errors raised by the deployment. They automatically roll back your deployment. An example is if the updated code you're deploying is creating errors within the application.

  • Hooks
    • PreTraffic: Before traffic shifting starts, CodeDeploy invokes the pre-traffic hook Lambda function. This Lambda function must call back to CodeDeploy and indicate success or failure. If the function fails, it aborts and reports a failure back to AWS CloudFormation. If the function succeeds, CodeDeploy proceeds to traffic shifting.
    • PostTraffic: After traffic shifting completes, CodeDeploy invokes the post-traffic hook Lambda function. This is similar to the pre-traffic hook, where the function must call back to CodeDeploy to report a success or failure. Use post-traffic hooks to run integration tests or other validation actions.
  • What is SAM and hot it works?

    Serverless application architectures enable you to break large projects into smaller, more manageable services that are highly reusable and independently scalable, secured, and evolved over time.

    As serverless architectures grow, we have seen common patterns that get reimplemented across companies, teams, and projects, hurting development velocity and leading to wasted effort.

    We have made it easier to develop new serverless architectures that meet organizational best practices by supporting nested applications in AWS SAM and the AWS Serverless Application Repository.

  • Transform

    For serverless applications (also referred to as Lambda-based applications), the optional Transform section specifies the version of the AWS Serverless Application Model (AWS SAM) to use. When you specify a transform, you can use AWS SAM syntax to declare resources in your template. The model defines the syntax that you can use and how it is processed.

    This section specifies one or more macros that AWS CloudFormation uses to process your template. The Transform section builds on the simple, declarative language of AWS CloudFormation with a powerful macro system.

    You can declare one or more macros within a template. AWS CloudFormation executes macros in the order that they are specified. When you create a change set, AWS CloudFormation generates a change set that includes the processed template content. You can then review the changes and execute the change set.

    AWS CloudFormation also supports the AWS::Serverless and AWS::Include transforms, which are macros hosted by AWS CloudFormation. AWS CloudFormation treats these transforms the same as any macros you create in terms of execution order and scope.

  • Mappings

    this is just a literal mapping of keys and associated values that you can use to specify conditional parameter values, similar to a lookup table.

  • Parameters

    this only contains the values that will be passed to your template at runtime (when you create or update a stack). You can refer to parameters from the Resources and Outputs sections of the template but this is not used to specify the AWS SAM version.

  • Format version

    This just refers to the AWS CloudFormation template version that the template conforms to, and not the version of the AWS Serverless Application Model (AWS SAM)

ECS

  • ECS intro

    Amazon Elastic Container Service (Amazon ECS) is a highly scalable, high-performance https://aws.amazon.com/containers/ container orchestration service that supports  https://aws.amazon.com/docker/ Docker containers and allows you to easily run and scale containerized applications on AWS. Amazon ECS eliminates the need for you to install and operate your own container orchestration software, manage and scale a cluster of virtual machines, or schedule containers on those virtual machines

  • ECS container instance

    An Amazon ECS container instance is an Amazon EC2 instance that is running the Amazon ECS container agent and has been registered into an Amazon ECS cluster. When you run tasks with Amazon ECS using the EC2 launch type or an Auto Scaling group capacity provider, your tasks are placed on your active container instances.

    If ECS instance is stopped then container instance will be still active just ECS container Agent status changes to FALSE
  • Task placement strategies

    A task placement strategy is an algorithm for selecting instances for task placement or tasks for termination. Task placement strategies can be specified when either running a task or creating a new service.

    Amazon ECS supports the following task placement strategies:

    binpack - Place tasks based on the least available amount of CPU or memory. This minimizes the number of instances in use.

    random - Place tasks randomly.

    spread - Place tasks evenly based on the specified value. Accepted values are attribute key-value pairs, instanceId, or host.

  • Cluster queries

    Cluster queries are expressions that enable you to group objects. For example, you can group container instances by attributes such as Availability Zone, instance type, or custom metadata. You can add custom metadata to your container instances, known as attributes. Each attribute has a name and an optional string value. You can use the built-in attributes provided by Amazon ECS or define custom attributes.

    After you have defined a group of container instances, you can customize Amazon ECS to place tasks on container instances based on group. Running tasks manually is ideal in certain situations. For example, suppose that you are developing a task but you are not ready to deploy this task with the service scheduler. Perhaps your task is a one-time or periodic batch job that does not make sense to keep running or restart when it finishes.

Mechanism

In addition to simple retries, each AWS SDK implements exponential backoff algorithm for better flow control. The idea behind exponential backoff is to use progressively longer waits between retries for consecutive error responses. You should implement a maximum delay interval, as well as a maximum number of retries. The maximum delay interval and maximum number of retries are not necessarily fixed values, and should be set based on the operation being performed, as well as other local factors, such as network latency.

Step Function

AWS Step Functions is a web service that enables you to coordinate the components of distributed applications and microservices using visual workflows. You build applications from individual components that each perform a discrete function, or task, allowing you to scale and change applications quickly. Step Functions provides a reliable way to coordinate components and step through the functions of your application. 

Web Identity Federation

With web identity federation, you don't need to create custom sign-in code or manage your own user identities. Instead, users of your app can sign in using a well-known identity provider (IdP) —such as Login with Amazon, Facebook, Google, or any other  OpenID Connect (OIDC)compatible IdP, receive an authentication token, and then exchange that token for temporary security credentials in AWS that map to an IAM role with permissions to use the resources in your AWS account. Using an IdP helps you keep your AWS account secure, because you don't have to embed and distribute long-term security credentials with your application.

EC2

  • Instance profile

    Use an instance profile to pass an IAM role to an EC2 instance.

    Managing instance profiles (console)

    If you use the AWS Management Console to create a role for Amazon EC2, the console automatically creates an instance profile and gives it the same name as the role. When you then use the Amazon EC2 console to launch an instance with an IAM role, you can select a role to associate with the instance. In the console, the list that's displayed is actually a list of instance profile names. The console does not create an instance profile for a role that is not associated with Amazon EC2.

    Managing instance profiles (AWS CLI or AWS API)

    If you manage your roles from the AWS CLI or the AWS API, you create roles and instance profiles as separate actions. Because roles and instance profiles can have different names, you must know the names of your instance profiles as well as the names of roles they contain. That way you can choose the correct instance profile when you launch an EC2 instance.


IAM

  • Introduction
  • IAM Federation

    uses the SAML standard(AD)

    uses in larger organisation so users can login to aws using Active Directory

  • STS ( Security Token Service)

    Allow to grant temp access to AWS resources ( max upto 1 hour)

    Assume Role : To assume role within your account or cross account

    AssumRolewithSAML : return credentials with logged with SAML

    AssumeRolewithIdentity : To login via 3rd party app like google , fb AWS used to recommends Cognito Identity Pools instead

    Getsessiontoken : for MFA

    GetFederationToken : obtain temp creds for a federated user

    Getcalleridentity : return details about IAM user or role used to call API

    DecodeAuthorizationMessage : To decode error message when AWS API is denied

  • STS to assume a role

    STS permission is not cumulative ie) permission of user won't be implied when another role is opted

    Temp credentials can vary from 15min to 1 hour

  • Cross account with STS
  • Evaluation fow of policy
  • IAM policy & S3 bucket policy

    IAM policies are attached to users, groups roles

    S3 policies are attached to s3 bucket

  • Dynamic policy with IAM

    ${aws:username} - variable to use in policy

  • Inline vs Managed Policies

    AWS managed policies :- Managed by AWS, auto update if any change in API

    Customer Managed policies :- Custom made, version controlled and central change management

    Inline: policy mapped one to one.

  • iam:passrole

    To pass role to service eg: pass role to EC2 server

  • Trust policy

    Roles can't be passed to any service, it should be in what trust allows

  • AWS Direcotry Services

    AWS Managed Microsoft AD Create you AD in AWS, you can enable trust with onpremis AD

    AD connector Director Gateway (proxy) redirect to onpremis AD Users are managed in on premis AD

    Simple AD AD compatible managed direcotory on AWS

    Can't link with onpremis AD

  • IAM access Analyzer

    AWS IAM Access Analyzer helps you identify the resources in your organization and accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity. This lets you identify unintended access to your resources and data, which is a security risk.

  • NFS file system policy

    NFS clients can identify themselves using an IAM role when connecting to an EFS file system. When a client connects to a file system, Amazon EFS evaluates the file system’s IAM resource policy, which is called a file system policy, along with any identity-based IAM policies to determine the appropriate file system access permissions to grant.

đź’ˇ
Remember if to restrict for certain any OU then SCP to be applied for all and denied for all except allowed OU

CI-CD

  • Code star

    AWS CodeStar is a cloud-based service for creating, managing, and working with software development projects on AWS. You can quickly develop, build, and deploy applications on AWS with an AWS CodeStar project. An AWS CodeStar project creates and integrates AWS services for your project development toolchain. Depending on your choice of AWS CodeStar project template, that toolchain might include source control, build, deployment, virtual servers or serverless resources, and more. AWS CodeStar also manages the permissions required for project users (called team members). By adding users as team members to an AWS CodeStar project, project owners can quickly and simply grant each team member role-appropriate access to a project and its resources.

Code Build

  • To override build config

    To override the default buildspec file name, location, or both, do one of the following:

    • Run the AWS CLI create-project or update-project command, setting the buildspec value to the path to the alternate buildspec file relative to the value of the built-in environment variable CODEBUILD_SRC_DIR. You can also do the equivalent with the create project operation in the AWS SDKs. For more information, see Create a build project or Change a build project's settings.
    • Run the AWS CLI start-build command, setting the buildspecOverride value to the path to the alternate buildspec file relative to the value of the built-in environment variable CODEBUILD_SRC_DIR. You can also do the equivalent with the start build operation in the AWS SDKs. For more information, see Run a build.
    • In an AWS CloudFormation template, set the BuildSpec property of Source in a resource of type AWS::CodeBuild::Project to the path to the alternate buildspec file relative to the value of the built-in environment variable CODEBUILD_SRC_DIR. For more information, see the BuildSpec property in AWS CodeBuild project source in the AWS CloudFormation User Guide.
  • Automatic rollback on codebuild

    Rollback on failure will just rollback to previous version but won't restore any deleted file

  • Resources in codebuild

    The content in the 'resources' section of the AppSpec file varies, depending on the compute platform of your deployment. The 'resources' section for an Amazon ECS deployment contains your Amazon ECS task definition, container and port for routing traffic to your updated Amazon ECS task set, and other optional information. The 'resources' section for an AWS Lambda deployment contains the name, alias, current version, and target version of a Lambda function.

    https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-resources.html

  • Code build password

    You need to specify the --with-decryption option, this allows the CodeDeploy service to decrypt the password so that it can be used in the application.

  • Codebuild with vpc

    Typically, AWS CodeBuild cannot access resources in a VPC. To enable access, you must provide additional VPC-specific configuration information in your CodeBuild project configuration. This includes the VPC ID, the VPC subnet IDs, and the VPC security group IDs. VPC-enabled builds can then access resources inside your VPC.

Code Deploy

  • Code deploy rollback

    During AWS CodeDeploy automatic roll-back, it will try to retrieve files which were part of previous versions. If these files are deleted or missing, you need to manually add those files to Instance or create a new application revision.

  • Code deploy flow
  • Lambda serverless flow
  • CodeDeploy log file if it was accidentally deleted

    Then you would need to restart codedeploy agent service

  • In-place deployment

    In-place deployment: The application on each instance in the deployment group is stopped, the latest application revision is installed, and the new version of the application is started and validated. You can use a load balancer so that each instance is deregistered during its deployment and then restored to service after the deployment is complete. Only deployments that use the EC2/On-Premises compute platform can use in-place deployments. AWS Lambda compute platform deployments cannot use an in-place deployment type.

đź’ˇ
Appsec.Yaml file is used to specify info on application that to be deployed for underlying instance
đź’ˇ
CodeDeploy agent is not required for deployments that use the Amazon ECS or AWS Lambda compute platform.

Code commit

  • Code commit crossaccount

    You can configure access to AWS CodeCommit repositories for IAM users and groups in another AWS account. This is often referred to as cross-account access. This section provides examples and step-by-step instructions for configuring cross-account access for a repository named MySharedDemoRepo in the US East (Ohio) Region in an AWS account (referred to as AccountA) to IAM users who belong to an IAM group named DevelopersWithCrossAccountRepositoryAccess in another AWS account (referred to as Account B).

  • Migrating to code commit

    You can migrate a Git repository to a CodeCommit repository in a number of ways: by cloning it, mirroring it, migrating all or just some of the branches, and so on. You can also migrate local, unversioned content on your computer to CodeCommit.

    The following topics show you some of the ways you can migrate a repository. Your steps might vary, depending on the type, style, or complexity of your repository and the decisions you make about what and how you want to migrate. For very large repositories, you might want to consider migrating incrementally.

  • HTTPS using git credentials

    The simplest way to set up connections to AWS CodeCommit repositories is to configure Git credentials for CodeCommit in the IAM console, and then use those credentials for HTTPS connections.

    You can also use these same credentials with any third-party tool or individual development environment (IDE) that supports HTTPS authentication using a static user name and password. For

  • AWS Codecommit

    Access to AWS CodeCommit requires credentials. Those credentials must have permissions to access AWS resources, such as CodeCommit repositories, and your IAM user, which you use to manage your Git credentials or the SSH public key that you use for making Git connections.

    With HTTPS connections and Git credentials, you generate a static user name and password in IAM. You then use these credentials with Git and any third-party tool that supports Git user name and password authentication. This method is supported by most IDEs and development tools. It is the simplest and easiest connection method to use with CodeCommit.

Code Pipeline

  • Code pipeline

    In AWS CodePipeline, an action is a task performed on an artifact in a stage. If an action or a set of parallel actions is not completed successfully, the pipeline stops running

  • Jenkins as pipeline

    As a best practice, when you use a Jenkins build provider for your pipeline’s build or test action, install Jenkins on an Amazon EC2 instance and configure a separate EC2 instance profile. Make sure the instance profile grants Jenkins only the AWS permissions required to perform tasks for your project, such as retrieving files from Amazon S3.

    The instance profile provides applications running on an Amazon EC2 instance with the credentials to access other AWS services. As a result, you do not need to configure AWS credentials (AWS access key and secret key).

  • Monitoring and Logging bestpractice code pipeline

    You can use logging features in AWS to determine the actions users have taken in your account and the resources that were used. The log files show:

    • The time and date of actions.
    • The source IP address for an action.
    • Which actions failed due to inadequate permissions.

    Logging features are available in the following AWS services:

Error codes of Code deploy

https://docs.aws.amazon.com/codedeploy/latest/userguide/error-codes.html

Data pipeline

  • Datanode

    AWS Data Pipeline, a data node defines the location and type of data that a pipeline activity uses as input or output. AWS Data Pipeline supports the following types of data nodes:

    DynamoDBDataNode A DynamoDB table that contains data for HiveActivity or EmrActivity to use.

    SqlDataNode An SQL table and database query that represent data for a pipeline activity to use.

    đź’ˇ
    NotePreviously, MySqlDataNode was used. Use SqlDataNode instead.

    RedshiftDataNode

    An Amazon Redshift table that contains data for RedshiftCopyActivity to use.

    S3DataNode

    An Amazon S3 location that contains one or more files for a pipeline activity to use.

Beanstalk

  • Deployment policy

    Deployment policy

    Choose from the following deployment options

    All at once

    Deploy the new version to all instances simultaneously. All instances in your environment are out of service for a short time while the deployment occurs.

    Rolling –

    Deploy the new version in batches. Each batch is taken out of service during the deployment phase, reducing your environment's capacity by the number of instances in a batch. Rolling with additional batch –

    Deploy the new version in batches, but first launch a new batch of instances to ensure full capacity during the deployment process. Immutable –

    Deploy the new version to a fresh group of instances by performing an immutable update. Traffic splitting –

    Deploy the new version to a fresh group of instances and temporarily split incoming client traffic between the existing application version and the new one.

  • Deployment methods

    In Elastic Beanstalk, you can choose from a variety of deployment methods:

    All at once – Deploy the new version to all instances simultaneously. All instances in your environment are out of service for a short time while the deployment occurs.

    Rolling – Deploy the new version in batches. Each batch is taken out of service during the deployment phase, reducing your environment's capacity by the number of instances in a batch.

    Rolling with additional batch – Deploy the new version in batches, but first launch a new batch of instances to ensure full capacity during the deployment process.

    Immutable – Deploy the new version to a fresh group of instances by performing an immutable update.

    Blue/Green - Deploy the new version to a separate environment, and then swap CNAMEs of the two environments to redirect traffic to the new version instantly.

  • Customize Deployment policy

    For the Rolling and Rolling with additional batch deployment policies you can configure:

    • Batch size – The size of the set of instances to deploy in each batch.

      Choose Percentage to configure a percentage of the total number of EC2 instances in the Auto Scaling group (up to 100 percent), or choose Fixed to configure a fixed number of instances (up to the maximum instance count in your environment's Auto Scaling configuration).

    For the Traffic splitting deployment policy you can configure the following:

    • Traffic split – The initial percentage of incoming client traffic that Elastic Beanstalk shifts to environment instances running the new application version you're deploying.
    • Traffic splitting evaluation time – The time period, in minutes, that Elastic Beanstalk waits after an initial healthy deployment before proceeding to shift all incoming client traffic to the new application version that you're deploying.
  • Configuration

    Elastic Beanstalk supports two methods of saving configuration option settings. Configuration files in YAML or JSON format can be included in your application's source code in a directory named .ebextensions and deployed as part of your application source bundle. You create and manage configuration files locally.

  • Custom application in EC2

    Mention the config in YAML or JSON file

    Place the file in the .ebextension folder

  • Blue/Green deployment

    Because AWS Elastic Beanstalk performs an in-place update when you update your application versions, your application can become unavailable to users for a short period of time. You can avoid this downtime by performing a blue/green deployment, where you deploy the new version to a separate environment, and then swap CNAMEs of the two environments to redirect traffic to the new version instantly.

    A blue/green deployment is also required when you want to update an environment to an incompatible platform version. For more information, see Updating your Elastic Beanstalk environment's platform version.

    Blue/green deployments require that your environment runs independently of your production database, if your application uses one. If your environment has an Amazon RDS DB instance attached to it, the data will not transfer over to your second environment, and will be lost if you terminate the original environment.

  • EB CLI

    The EB CLI is a command line interface for AWS Elastic Beanstalk that provides interactive commands that simplify creating, updating and monitoring environments from a local repository.

    Use the EB CLI as part of your everyday development and testing cycle as an alternative to the Elastic Beanstalk console.

  • Custom AMI

    When you create an AWS Elastic Beanstalk environment, you can specify an Amazon Machine Image (AMI) to use instead of the standard Elastic Beanstalk AMI included in your platform version. A custom AMI can improve provisioning times when instances are launched in your environment if you need to install a lot of software that isn't included in the standard AMIs.

    Using configuration files is great for configuring and customizing your environment quickly and consistently. Applying configurations, however, can start to take a long time during environment creation and updates. If you do a lot of server configuration in configuration files, you can reduce this time by making a custom AMI that already has the software and configuration that you need.

    A custom AMI also allows you to make changes to low-level components, such as the Linux kernel, that are difficult to implement or take a long time to apply in configuration files. To create a custom AMI, launch an Elastic Beanstalk platform AMI in Amazon EC2, customize the software and configuration to your needs, and then stop the instance and save an AMI from it.

  • ASG

    Your AWS Elastic Beanstalk environment includes an Auto Scaling group that manages the Amazon EC2 instances in your environment. In a single-instance environment, the Auto Scaling group ensures that there is always one instance running. In a load-balanced environment, you configure the group with a range of instances to run, and Auto Scaling adds or removes instances as needed, based on load.

    The Auto Scaling group also applies the launch configuration for the instances in your environment. You can modify the launch configuration to change the instance type, key pair, Amazon Elastic Block Store (Amazon EBS) storage, and other settings that can only be configured when you launch an instance.

  • Managing app version

    Elastic Beanstalk creates an application version whenever you upload source code. This usually occurs when you create an environment or upload and deploy code using the environment management console or EB CLI. Elastic Beanstalk deletes these application versions according to the application's lifecycle policy and when you delete the application. For details about application lifecycle policy, see Configuring application version lifecycle settings.

    You can also upload a source bundle without deploying it from the application management console. Elastic Beanstalk stores source bundles in Amazon Simple Storage Service (Amazon S3) and doesn't automatically delete them.

  • DynamoDB

    You can create DynamoDB with Elastic beanstalk

    Ensure to select create snapshot to create snapshot before termination

  • Application source bundle

    When you use the AWS Elastic Beanstalk console to deploy a new application or an application version, you'll need to upload a source bundle. Your source bundle must meet the following requirements:

    • Consist of a single ZIP file or WAR file (you can include multiple WAR files inside your ZIP file)
    • Not exceed 512 MB
    • Not include a parent folder or top-level directory (subdirectories are fine)

    If you want to deploy a worker application that processes periodic background tasks, your application source bundle must also include a cron.yaml file. For more information

  • Beanstalk worker environment

    If your application performs operations or workflows that take a long time to complete, you can offload those tasks to a dedicated worker environment. Decoupling your web application front end from a process that performs blocking operations is a common way to ensure that your application stays responsive under load.

    A long-running task is anything that substantially increases the time it takes to complete a request, such as processing images or videos, sending email, or generating a ZIP archive. These operations can take only a second or two to complete, but a delay of a few seconds is a lot for a web request that would otherwise complete in less than 500 ms.

    One option is to spawn a worker process locally, return success, and process the task asynchronously. This works if your instance can keep up with all of the tasks sent to it. Under high load, however, an instance can become overwhelmed with background tasks and become unresponsive to higher priority requests. If individual users can generate multiple tasks, the increase in load might not correspond to an increase in users, making it hard to scale out your web server tier effectively.

    To avoid running long-running tasks locally, you can use the AWS SDK for your programming language to send them to an Amazon Simple Queue Service (Amazon SQS) queue, and run the process that performs them on a separate set of instances. You then design these worker instances to take items from the queue only when they have the capacity to run them, preventing them from becoming overwhelmed.

    Elastic Beanstalk worker environments simplify this process by managing the Amazon SQS queue and running a daemon process on each instance that reads from the queue for you. When the daemon pulls an item from the queue, it sends an HTTP POST request locally to http://localhost/ on port 80 with the contents of the queue message in the body. All that your application needs to do is perform the long-running task in response to the POST. You can configure the daemon to post to a different path, use a MIME type other than application/JSON, connect to an existing queue, or customize connections (maximum concurrent requests), timeouts, and retries.

  • Update application

    The EB CLI is a command line interface for Elastic Beanstalk that provides interactive commands that simplify creating, updating and monitoring environments from a local repository. It is recommended that you use the EB CLI as part of your everyday development and testing cycle as an alternative to the AWS Management Console.

    You can tell the EB CLI to deploy a ZIP file or WAR file that you generate as part of a separate build process by adding the following lines to .elasticbeanstalk/config.yml in your project folder:

    deploy:   artifact: path/to/buildartifact.zip 

Cloudfront

  • Viewer policy

    You can configure one or more cache behaviors in your CloudFront distribution to require HTTPS for communication between viewers and CloudFront. You also can configure one or more cache behaviors to allow both HTTP and HTTPS, so that CloudFront requires HTTPS for some objects but not for others.

    If you're using the domain name that CloudFront assigned to your distribution, such as dtut0ria1sd0jo.cloudfront.net, you can change the Viewer Protocol Policy setting for one or more cache behaviors to require HTTPS communication by setting it as either Redirect HTTP to HTTPS, or HTTPS Only.

  • If your origin is an Elastic Load Balancing load balancer, you can use a certificate provided by AWS Certificate Manager (ACM). You can also use a certificate that is signed by a trusted third-party certificate authority and imported into ACM. Note that you can't use a self-signed certificate for HTTPS communication between CloudFront and your origin.
  • To have minimum latency on lambda authentication

    You can use [email protected] to help authenticate and authorize users for the premium pay-wall content on your website, filtering out unauthorized requests before they reach your origin infrastructure. For example, you can trigger a Lambda function to authorize each viewer request by calling authentication and user management service such as Amazon Cognito.

Secret Manager

  • Intro

    AWS Secrets Manager is an AWS service that makes it easier for you to manage secrets. Secrets can be database credentials, passwords, third-party API keys, and even arbitrary text. You can store and control access to these secrets centrally by using the Secrets Manager console, the Secrets Manager command line interface (CLI), or the Secrets Manager API and SDKs.

    In the past, when you created a custom application that retrieves information from a database, you typically had to embed the credentials (the secret) for accessing the database directly in the application. When it came time to rotate the credentials, you had to do much more than just create new credentials. You had to invest time to update the application to use the new credentials. Then you had to distribute the updated application. If you had multiple applications that shared credentials and you missed updating one of them, the application would break. Because of this risk, many customers have chosen not to regularly rotate their credentials, which effectively substitutes one risk for another.

    Secrets Manager enables you to replace hardcoded credentials in your code (including passwords), with an API call to Secrets Manager to retrieve the secret programmatically. This helps ensure that the secret can't be compromised by someone examining your code, because the secret simply isn't there. Also, you can configure Secrets Manager to automatically rotate the secret for you according to a schedule that you specify. This enables you to replace long-term secrets with short-term ones, which helps to significantly reduce the risk of compromise.

  • @

Account alias

  • Account alias

    An account alias substitutes for an account ID in the web address for your account. You can create and manage an account alias from the AWS Management Console, AWS CLI, or AWS API. Your sign-in page URL has the following format, by default:

    https://Your_AWS_Account_ID.signin.aws.amazon.com/console/

    If you create an AWS account alias for your AWS account ID, your sign-in page URL looks like the following example.

    https://Your_Alias.signin.aws.amazon.com/console/

    The original URL containing your AWS account ID remains active and can be used after you create your AWS account alias. For example, the following create-account-alias command creates the alias tutorialsdojo for your AWS account:

    aws iam create-account-alias --account-alias tutorialsdojo

ASG

  • Intro

    At any time, you can change the size of an existing Auto Scaling group manually. You can either update the desired capacity of the Auto Scaling group, or update the instances that are attached to the Auto Scaling group. Manually scaling your group can be useful when automatic scaling is not needed or when you need to hold capacity at a fixed number of instances.

    With each Auto Scaling group, you can control when it adds instances (referred to as scaling out) or removes instances (referred to as scaling in) from your network architecture. You can scale the size of your group manually by adjusting your desired capacity, or you can automate the process through the use of scheduled scaling or a scaling policy.

    Consider an Auto Scaling group that uses a launch configuration. It has one instance type, two Availability Zones, a desired capacity of two instances, and scaling policies that increase and decrease the number of instances by one when certain thresholds are met. When the threshold for the scale-out policy is met, the policy takes effect and the Auto Scaling group launches a new instance. The Auto Scaling group now has three instances.

    When the threshold for the scale-in policy is met, the policy takes effect and the Auto Scaling group terminates one of the instances. If you did not assign a specific termination policy to the group, it uses the default termination policy. It selects the Availability Zone with two instances, and terminates the instance launched from the oldest launch configuration. If the instances were launched from the same launch configuration, the Auto Scaling group selects the instance that is closest to the next billing hour and terminates it.

    If all instances in an Auto Scaling group are protected from termination during scale in, and a scale-in event occurs, its desired capacity is decremented. However, the Auto Scaling group can't terminate the required number of instances until their instance scale-in protection settings are disabled.

Consolidated billing

The consolidated billing feature in AWS Organizations allows you to consolidate payment for multiple AWS accounts or multiple AISPL accounts. Each organization in AWS Organizations has a master account that pays the charges for all the member accounts. If you have access to the master account, you can see a combined view of the AWS charges that are incurred by the member accounts. You also can get a cost report for each member account. Hence the correct answer is to create separate AWS accounts for each project and use consolidated billing.`

SAML integration

  • For unspported on-premise

    If your identity store is not compatible with SAML 2.0, then you can build a custom identity broker application to perform a similar function. The broker application authenticates users, requests temporary credentials for users from AWS, and then provides them to the user to access AWS resources.

    The application verifies that employees are signed into the existing corporate network's identity and authentication system, which might use LDAP, Active Directory, or another system. The identity broker application then obtains temporary security credentials for the employees.

    To get temporary security credentials, the identity broker application calls either AssumeRole or GetFederationToken to obtain temporary security credentials, depending on how you want to manage the policies for users and when the temporary credentials should expire. The call returns temporary security credentials consisting of an AWS access key ID, a secret access key, and a session token. The identity broker application makes these temporary security credentials available to the internal company application. The app can then use the temporary credentials to make calls to AWS directly. The app caches the credentials until they expire, and then requests a new set of temporary credentials.

S3

  • Restoring Archived Objects
    • S3 Batch Operations

      To restore more than one Amazon S3 object with a single request, you can use S3 Batch Operations. You provide S3 Batch Operations with a list of objects to operate on. S3 Batch Operations call the respective API to perform the specified operation. A single S3 Batch Operations job can perform the specified operation on billions of objects containing exabytes of data.

      S3 Batch Operations track progress, send notifications, and store a detailed completion report of all actions, providing a fully managed, auditable, serverless experience. You can use S3 Batch Operations through the AWS Management Console, AWS CLI, AWS SDKs, or REST API.

    • Restoration types

      The time it takes a restore job to finish depends on which archive storage class you use and which retrieval option you specify: Expedited (only available for S3 Glacier), Standard, or Bulk. You can be notified when your restore is complete using Amazon S3 event notifications.

    đź’ˇ
    Amazon S3 restores a temporary copy of the object only for the specified duration. After that, it deletes the restored object copy. You can modify the expiration period of a restored copy by reissuing a restore.
  • Amazon S3 event notifications

    You can configure to send notification when a certain event occur

    • Notification can be triggered on below scenarios

      New object created events

      Object removal events

      Restore object events

      Reduced Redundancy Storage (RRS) object lost events

      Replication events

    • S3 supports following destinations to notify service

      SNS

      SQS

      Lambda

  • S3 access logs records below parameters

    Bucket Owner Bucket → Bucket name Time → Time of request received eg: [06/Feb/2019:00:00:38 +0000] Remote IP Requestor →The canonical user ID of the requester Request ID Operation Key Request-URI HTTP Status Error Code Bytes Sent Object Size Total Time Turn-Around Time Referer User-Agent Version Id Host Id Signature Version Cipher Suite Authentication Type Host Header TLS version

    đź’ˇ

  • S3 inventory

    You can use it to audit and report on the replication and encryption status of your objects for business, compliance, and regulatory needs. You can also simplify and speed up business workflows and big data jobs using Amazon S3 inventory, which provides a scheduled alternative to the Amazon S3 synchronous List API operation.

    You can configure multiple inventory lists for a bucket. You can configure what object metadata to include in the inventory, whether to list all object versions or only current versions, where to store the inventory list file output, and whether to generate the inventory on a daily or weekly basis. You can also specify that the inventory list file be encrypted.

    You can query Amazon S3 inventory using standard SQL by using Amazon Athena, Amazon Redshift Spectrum, and other tools

  • Replication configure overview
    • You add one rule in replication configuration in the following scenarios:
      • You want to replicate all objects.
      • You want to replicate a subset of objects. You identify the object subset by adding a filter in the rule. In the filter, you specify an object key prefix, tags, or a combination of both, to identify the subset of objects that the rule applies to.
    • You add multiple rules in a replication configuration if you want to select a different subset of objects. In each rule, you specify a filter that selects a different subset of objects. For example, you might choose to replicate objects that have either tax/ or document/ key prefixes. You would add two rules and specify the tax/ key prefix filter in one rule and the document/ key prefix in the other.
  • CORS for S3

    Scenario 1

    Suppose that you are hosting a website in an Amazon S3 bucket named website as described in Hosting a static website on Amazon S3. Your users load the website endpoint:

    http://website.s3-website.us-east-1.amazonaws.com

    Now you want to use JavaScript on the webpages that are stored in this bucket to be able to make authenticated GET and PUT requests against the same bucket by using the Amazon S3 API endpoint for the bucket, website.s3.us-east-1.amazonaws.com. A browser would normally block JavaScript from allowing those requests, but with CORS you can configure your bucket to explicitly enable cross-origin requests from website.s3-website.us-east-1.amazonaws.com.

    Scenario 2

    Suppose that you want to host a web font from your S3 bucket. Again, browsers require a CORS check (also called a preflight check) for loading web fonts. You would configure the bucket that is hosting the web font to allow any origin to make these requests.

  • To have server side encryption for all objects

    Block uploads that doesn't contain x-amz-server-side-encryption header

  • S3 log files

    An S3 bucket with server access logging enabled can accumulate many server log objects over time. Your application might need these access logs for a specific period after creation, and after that, you might want to delete them. You can use Amazon S3 lifecycle configuration to set rules so that Amazon S3 automatically queues these objects for deletion at the end of their life.

    You can define a lifecycle configuration for a subset of objects in your S3 bucket by using a shared prefix (that is, objects that have names that begin with a common string). If you specified a prefix in your server access logging configuration, you can set a lifecycle configuration rule to delete log objects that have that prefix. For example, if your log objects have the prefix logs/, you can set a lifecycle configuration rule to delete all objects in the bucket that have the prefix /logs after a specified period of time. For more information about lifecycle configuration, see Object lifecycle management.

  • Performance

    Your applications can easily achieve thousands of transactions per second in request performance when uploading and retrieving storage from Amazon S3.

    Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket.

    There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by parallelizing reads. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second.

  • S3 optimization
  • To ensure Encryption for all object

    Deny permission to upload an object if the header does not include x-amz-server-side-encryption

  • 503 errors

    If you notice a significant increase in the number of HTTP 503-slow down responses received for Amazon S3 PUT or DELETE object requests to a bucket that has versioning enabled, you might have one or more objects in the bucket for which there are millions of versions. When you have objects with millions of versions, Amazon S3 automatically throttles requests to the bucket to protect the customer from an excessive amount of request traffic, which could potentially impede other requests made to the same bucket.

    To determine which S3 objects have millions of versions, use the Amazon S3 inventory tool. The inventory tool generates a report that provides a flat file list of the objects in a bucket.

  • KMS encryption

    s3:PutObject include the x-amz-server-side-encryption header requesting server-side encryption with SSE-KMS.

    To upload an object to the S3 bucket which uses SSE-KMS, you have to send a request with an x-amz-server-side-encryption header with the value of aws:kms. There's also an optional x-amz-server-side-encryption-aws-kms-key-id header which specifies the ID of the AWS KMS master encryption key that was used for the object. The Amazon S3 API also supports encryption context, with the x-amz-server-side-encryption-context header.

    When you upload an object, you can specify the KMS key using the x-amz-server-side-encryption-aws-kms-key-id header. If the header is not present in the request, Amazon S3 assumes the default KMS key. Regardless, the KMS key ID that Amazon S3 uses for object encryption must match the KMS key ID in the policy, otherwise Amazon S3 denies the request.

  • SSE-C encryption

    Server-side encryption is about protecting data at rest. Using server-side encryption with customer-provided encryption keys (SSE-C) allows you to set your own encryption keys. With the encryption key you provide as part of your request, Amazon S3 manages both the encryption, as it writes to disks, and decryption, when you access your objects. Therefore, you don't need to maintain any code to perform data encryption and decryption. The only thing you do is manage the encryption keys you provide.

    When you upload an object, Amazon S3 uses the encryption key you provide to apply AES-256 encryption to your data and removes the encryption key from memory. It is important to note that Amazon S3 does not store the encryption key you provide. Instead, it is stored in a randomly salted HMAC value of the encryption key in order to validate future requests. The salted HMAC value cannot be used to derive the value of the encryption key or to decrypt the contents of the encrypted object. That means, if you lose the encryption key, you lose the object.

    When you retrieve an object, you must provide the same encryption key as part of your request. Amazon S3 first verifies that the encryption key you provided matches, and then decrypts the object before returning the object data to you.

    When using server-side encryption with customer-provided encryption keys (SSE-C), you must provide encryption key information using the following request headers:

    x-amz-server-side-encryption-customer-algorithm - This header specifies the encryption algorithm. The header value must be "AES256".

    x-amz-server-side-encryption-customer-key This header provides the 256-bit, base64-encoded encryption key for Amazon S3 to use to encrypt or decrypt your data.

    x-amz-server-side-encryption-customer-key-MD5 This header provides the base64-encoded 128-bit MD5 digest of the encryption key according to RFC 1321. Amazon S3 uses this header for a message integrity check to ensure the encryption key was transmitted without error.

  • Encrypt multipart upload

    If you are getting an Access Denied error when trying to upload a large file to your S3 bucket with an upload request that includes an AWS KMS key then you have to confirm that you have the permission to perform kms:Decrypt actions on the AWS KMS key that you're using to encrypt the object.

    To perform a multipart upload with encryption using an AWS Key Management Service (AWS KMS) customer master key (CMK), the requester must have permission to the kms:Decrypt and kms:GenerateDataKey* actions on the key. These permissions are required because Amazon S3 must decrypt and read data from the encrypted file parts before it completes the multipart upload.

  • 503 error

    If you notice a significant increase in the number of HTTP 503-slow down responses received for Amazon S3 PUT or DELETE object requests to a bucket that has versioning enabled, you might have one or more objects in the bucket for which there are millions of versions. When you have objects with millions of versions, Amazon S3 automatically throttles requests to the bucket to protect the customer from an excessive amount of request traffic, which could potentially impede other requests made to the same bucket.

    To determine which S3 objects have millions of versions, use the Amazon S3 inventory tool. The inventory tool generates a report that provides a flat file list of the objects in a bucket. For more information, see Amazon S3 inventory.

    The Amazon S3 team encourages customers to investigate applications that repeatedly overwrite the same S3 object, potentially creating millions of versions for that object, to determine whether the application is working as intended. If you have a use case that requires millions of versions for one or more S3 objects, contact the AWS Support team at AWS Support

    to discuss your use case and to help us assist you in determining the optimal solution for your use case scenario.

    To help prevent this issue, consider the following best practices:

    • Enable a lifecycle management "NonCurrentVersion" expiration policy and an "ExpiredObjectDeleteMarker" policy to expire the previous versions of objects and delete markers without associated data objects in the bucket.
    • Keep your directory structure as flat as possible and make each directory name unique.
đź’ˇ
IAM roles can't be assigned to S3

đź’ˇ
If you run Javascript on website which is on S3 then enable CORS

Cognito

  • What is Cognito?

    Amazon Cognito handles user authentication and authorization for your web and mobile apps.

    With user pools, you can easily and securely add sign-up and sign-in functionality to your apps. With identity pools (federated identities), your apps can get temporary credentials that grant users access to specific AWS resources, whether the users are anonymous or are signed in.

  • Amazon Cognito User Pools

    A user pool is a user directory in Amazon Cognito. With a user pool, your users can sign in to your web or mobile app through Amazon Cognito.

    Your users can also sign in through social identity providers like Google, Facebook, Amazon, or Apple, and through SAML identity providers.

    Whether your users sign in directly or through a third party, all members of the user pool have a directory profile that you can access through a Software Development Kit (SDK).

    • User pools provide
      • Sign-up and sign-in services.
      • A built-in, customizable web UI to sign in users.
      • Social sign-in with Facebook, Google, Login with Amazon, and Sign in with Apple, as well as sign-in with SAML identity providers from your user pool.
      • User directory management and user profiles.
      • Security features such as multi-factor authentication (MFA), checks for compromised credentials, account takeover protection, and phone and email verification.
      • Customized workflows and user migration through AWS Lambda triggers
    • After successfully authenticating a user, Amazon Cognito issues JSON web tokens (JWT) that you can use to secure and authorize access to your own APIs, or exchange for AWS credentials.
  • Amazon Cognito Sync

    Amazon Cognito Sync is an AWS service and client library that enables cross-device syncing of application-related user data.

    You can use it to synchronize user profile data across mobile devices and the web without requiring your own backend.

    The client libraries cache data locally so your app can read and write data regardless of device connectivity status.

    When the device is online, you can synchronize data, and if you set up push sync, notify other devices immediately that an update is available.

  • User pools vs Identity pools

    User pools are for authentication (identify verification). With a user pool, your app users can sign in through the user pool or federate through a third-party identity provider (IdP).

    Identity pools are for authorization (access control). You can use identity pools to create unique identities for users and give them access to other AWS services.

  • Userpools with MFA

    Multi-factor authentication (MFA) increases security for your app by adding another authentication method, and not relying solely on user name and password. You can choose to use SMS text messages, or time-based one-time (TOTP) passwords as second factors in signing in your users.

    With adaptive authentication, you can configure your user pool to require second factor authentication in response to an increased risk level

  • Cognito streams

    Amazon Cognito Streams gives developers control and insight into their data stored in Amazon Cognito. Developers can now configure a Kinesis stream to receive events as data is updated and synchronized. Amazon Cognito can push each dataset change to a Kinesis stream you own in real time.

    Using Amazon Cognito Streams, you can move all of your Sync data to Kinesis, which can then be streamed to a data warehouse tool such as Amazon Redshift for further analysis. To learn more about Kinesis, see Getting Started Using Amazon Kinesis.

  • Multiple Roles evaluation

    Rules are evaluated in order, and the IAM role for the first matching rule is used, unless CustomRoleArn is specified to override the order.

  • Userpool with Lamda

    You can create an AWS Lambda function and then trigger that function during user pool operations such as user sign-up, confirmation, and sign-in (authentication) with a Lambda trigger. You can add authentication challenges, migrate users, and customize verification messages.

  • Cognito sync

    Amazon Cognito Sync is an AWS service and client library that enable cross-device syncing of application-related user data. You can use it to synchronize user profile data across mobile devices and web applications. The client libraries cache data locally so your app can read and write data regardless of device connectivity status. When the device is online, you can synchronize data, and if you set up push sync, notify other devices immediately that an update is available.

  • Userpools vs Identity pools

    The two main components of Amazon Cognito are user pools and identity pools.

    User pools are user directories that provide sign-up and sign-in options for your app users.

    Identity pools enable you to grant your users access to other AWS services. You can use identity pools and user pools separately or together.

AWS App sync

  • Intro

    AWS AppSync simplifies application development by letting you create a flexible API to securely access, manipulate, and combine data from one or more data sources. AppSync is a managed service that uses GraphQL to make it easy for applications to get exactly the data they need.

    With AppSync, you can build scalable applications, including those requiring real-time updates, on a range of data sources such as NoSQL data stores, relational databases, HTTP APIs, and your custom data sources with AWS Lambda. For mobile and web apps, AppSync additionally provides local data access when devices go offline, and data synchronization with customizable conflict resolution, when they are back online.

    AWS AppSync is quite similar with Amazon Cognito Sync which is also a service for synchronizing application data across devices. It enables user data like app preferences or game state to be synchronized as well however, the key difference is that, it also extends these capabilities by allowing multiple users to synchronize and collaborate in real time on shared data.

CloudWatch

  • Metrics Resolution

    Each metric is one of the following:

    • Standard resolution, with data having a one-minute granularity
    • High resolution, with data at a granularity of one second

    Metrics produced by AWS services are standard resolution by default. When you publish a custom metric, you can define it as either standard resolution or high resolution. When you publish a high-resolution metric, CloudWatch stores it with a resolution of 1 second, and you can read and retrieve it with a period of 1 second, 5 seconds, 10 seconds, 30 seconds, or any multiple of 60 seconds.

    High-resolution metrics can give you more immediate insight into your application's sub-minute activity. Keep in mind that every PutMetricData call for a custom metric is charged, so calling PutMetricData more often on a high-resolution metric can lead to higher charges. For more information about CloudWatch pricing, see Amazon CloudWatch Pricing

    If you set an alarm on a high-resolution metric, you can specify a high-resolution alarm with a period of 10 seconds or 30 seconds, or you can set a regular alarm with a period of any multiple of 60 seconds. There is a higher charge for high-resolution alarms with a period of 10 or 30 seconds.

  • Detailed log

    send metric data of the EC2 instance to CloudWatch in 1-minute periods instead of 5-minute intervals.

  • CloudwatchEvents

    Amazon CloudWatch Events is a web service that monitors your AWS resources and the applications you run on AWS. You can use Amazon CloudWatch Events to detect and react to changes in the state of a pipeline, stage, or action. Then, based on rules you create, CloudWatch Events invokes one or more target actions when a pipeline, stage, or action enters the state you specify in a rule. Depending on the type of state change, you might want to send notifications, capture state information, take corrective action, initiate events, or take other actions.

    Amazon CloudWatch Events are composed of:

    • Rules. An event in Amazon CloudWatch Events is configured by first creating a rule with a selected service as the event source.
    • Targets. The new rule receives a selected service as the event target. For a list of services available as Amazon CloudWatch Events targets, see What Is Amazon CloudWatch Events.

    Examples of Amazon CloudWatch Events rules and targets:

    • A rule that sends a notification when the instance state changes, where an EC2 instance is the event source and Amazon SNS is the event target.
    • A rule that sends a notification when the build phase changes, where a CodeBuild configuration is the event source and Amazon SNS is the event target.
    • A rule that detects pipeline changes and invokes an AWS Lambda function.
  • High resolution metrics

    Using the existing PutMetricData API, you can now publish Custom Metrics down to 1-second resolution. This gives you more immediate visibility and greater granularity into the state and performance of your custom applications, such as observing short-lived spikes and functions. In addition, you can also alert sooner with High-Resolution Alarms, as frequently as 10-second periods. High-Resolution Alarms allow you to react and take actions faster, and support the same actions available today with standard 1-minute alarms. You can add these high-resolution metrics and alarms widgets to your Dashboards giving you easy observability of critical components. Additionally, if you use collectd to gather your metrics, you can publish these metrics to CloudWatch using our updated collectd plugin supporting high-resolution periods down to 1-second.

  • Alarm

    When you create an alarm, you specify three settings to enable CloudWatch to evaluate when to change the alarm state:

    • Period is the length of time to evaluate the metric or expression to create each individual data point for an alarm. It is expressed in seconds. If you choose one minute as the period, there is one datapoint every minute.
    • Evaluation Period is the number of the most recent periods, or data points, to evaluate when determining alarm state.
    • Datapoints to Alarm is the number of data points within the evaluation period that must be breaching to cause the alarm to go to the ALARM state. The breaching data points do not have to be consecutive, they just must all be within the last number of data points equal to Evaluation Period.

    In the following figure, the alarm threshold is set to three units. The alarm is configured to go to the ALARM state and both Evaluation Period and Datapoints to Alarm are 3. That is, when all three datapoints in the most recent three consecutive periods are above the threshold, the alarm goes to the ALARM state. In the figure, this happens in the third through fifth time periods. At period six, the value dips below the threshold, so one of the periods being evaluated is not breaching, and the alarm state changes to OK. During the ninth time period, the threshold is breached again, but for only one period. Consequently, the alarm state remains OK.

  • Schedule Lambda trigger

    Amazon CloudWatch Events help you to respond to state changes in your AWS resources. When your resources change state, they automatically send events into an event stream. You can create rules that match selected events in the stream and route them to your AWS Lambda function to take action. For example, you can automatically invoke an AWS Lambda function to log the state of an EC2 instance or AutoScaling Group. You maintain event source mapping in Amazon CloudWatch Events by using a rule target definition.

    You can also create a Lambda function and direct AWS Lambda to execute it on a regular schedule. You can specify a fixed rate (for example, execute a Lambda function every hour or 15 minutes), or you can specify a Cron expression.

CloudTrail

  • IAM and STS logs

    CloudTrail captures all API calls for IAM and AWS STS as events, including calls from the console and from API calls. If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket. If you don't configure a trail, you can still view the most recent events in the CloudTrail console in Event history. You can use CloudTrail to get information about the request that was made to IAM or AWS STS. For example, you can view the IP address from which the request was made, who made the request, when it was made, and additional details.

AWS Systems Manager Parameter Store

  • Explanation

    AWS Systems Manager Parameter Store provides secure, hierarchical storage for configuration data management and secrets management.

    You can store data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values.

    You can store values as plain text or encrypted data.

    You can reference Systems Manager parameters in your scripts, commands, SSM documents, and configuration and automation workflows by using the unique name that you specified when you created the parameter.

Kinesis

  • Kinesis Firehose

    Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), Splunk, and any custom HTTP endpoint or HTTP endpoints owned by supported third-party service providers, including Datadog, MongoDB, and New Relic.

    Kinesis Data Firehose is part of the Kinesis streaming data platform, along with Kinesis Data Streams, Kinesis Video Streams, and Amazon Kinesis Data Analytics. With Kinesis Data Firehose, you don't need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it.

  • Server side encryption at Kinesis dataware house

    When you configure a Kinesis data stream as the data source of a Kinesis Data Firehose delivery stream, Kinesis Data Firehose no longer stores the data at rest. Instead, the data is stored in the data stream.

    When you send data from your data producers to your data stream, Kinesis Data Streams encrypts your data using an AWS Key Management Service (AWS KMS) key before storing the data at rest. When your Kinesis Data Firehose delivery stream reads the data from your data stream, Kinesis Data Streams first decrypts the data and then sends it to Kinesis Data Firehose. Kinesis Data Firehose buffers the data in memory based on the buffering hints that you specify. It then delivers it to your destinations without storing the unencrypted data at rest.

  • Resharding

    The purpose of resharding in Amazon Kinesis Data Streams is to enable your stream to adapt to changes in the rate of data flow. You split shards to increase the capacity (and cost) of your stream. You merge shards to reduce the cost (and capacity) of your stream.

  • Kinesis Data stream

    Amazon Kinesis Data Streams is a massively scalable, highly durable data ingestion and processing service optimized for streaming data. You can configure hundreds of thousands of data producers to continuously put data into a Kinesis data stream. Data will be available within milliseconds to your Amazon Kinesis applications, and those applications will receive data records in the order they were generated.

    The purpose of resharding in Amazon Kinesis Data Streams is to enable your stream to adapt to changes in the rate of data flow. You split shards to increase the capacity (and cost) of your stream. You merge shards to reduce the cost (and capacity) of your stream.

    One approach to resharding could be to split every shard in the stream—which would double the stream's capacity. However, this might provide more additional capacity than you actually need and therefore create unnecessary cost.

    You can also use metrics to determine which are your "hot" or "cold" shards, that is, shards that are receiving much more data, or much less data, than expected. You could then selectively split the hot shards to increase capacity for the hash keys that target those shards. Similarly, you could merge cold shards to make better use of their unused capacity.

    You can obtain some performance data for your stream from the Amazon CloudWatch metrics that Kinesis Data Streams publishes. However, you can also collect some of your own metrics for your streams. One approach would be to log the hash key values generated by the partition keys for your data records. Recall that you specify the partition key at the time that you add the record to the stream.

  • Lambda function process kinesis data stream

    For Lambda functions that process Kinesis or DynamoDB streams, the number of shards is the unit of concurrency. If your stream has 100 active shards, there will be at most 100 Lambda function invocations running concurrently. This is because Lambda processes each shard’s events in sequence.

Cloudformation

  • Nested stack

    As your infrastructure grows, common patterns can emerge in which you declare the same components in each of your templates. You can separate out these common components and create dedicated templates for them. That way, you can mix and match different templates but use nested stacks to create a single, unified stack. Nested stacks are stacks that create other stacks. To create nested stacks, use the AWS::CloudFormation::Stack resource in your template to reference other templates.

    For example, assume that you have a load balancer configuration that you use for most of your stacks. Instead of copying and pasting the same configurations into your templates, you can create a dedicated template for the load balancer. Then, you just use the AWS::CloudFormation::Stack resource to reference that template from within other templates. If the load balancer template is updated, any stack that is referencing it will use the updated load balancer (only after you update the stack). In addition to simplifying updates, this approach lets you use exports to create and maintain components that you might not be necessarily familiar with. All you need to do is reference their templates.

  • Install software while launching EC2 instance

    When you launch stacks, you can install and configure software applications on Amazon EC2 instances by using the cfn-init helper script and the AWS::CloudFormation::Init resource. By using AWS::CloudFormation::Init, you can describe the configurations that you want rather than scripting procedural steps. You can also update configurations without recreating instances. And if anything goes wrong with your configuration, AWS CloudFormation generates logs that you can use to investigate issues.

    In your template, specify installation and configuration states in the AWS::CloudFormation::Init resource.

  • Parameter types

    String

    Number

    List

    Comma delimitedlist

    AWS Specific parameter types

    SSM parameter types

  • CloudFormation StackSets

    AWS CloudFormation StackSets extends the functionality of stacks by enabling you to create, update, or delete stacks across multiple accounts and regions with a single operation. Using an administrator account, you define and manage an AWS CloudFormation template, and use the template as the basis for provisioning stacks into selected target accounts across specified regions.

    A stack set lets you create stacks in AWS accounts across regions by using a single AWS CloudFormation template. All the resources included in each stack are defined by the stack set's AWS CloudFormation template. As you create the stack set, you specify the template to use, as well as any parameters and capabilities that the template requires.

  • Flexible template for multiple environments

    The optional Conditions section contains statements that define the circumstances under which entities are created or configured. For example, you can create a condition and then associate it with a resource or output so that AWS CloudFormation only creates the resource or output if the condition is true

    Similarly, you can associate the condition with a property so that AWS CloudFormation only sets the property to a specific value if the condition is true. If the condition is false, AWS CloudFormation sets the property to a different value that you specify.

    You might use conditions when you want to reuse a template that can create resources in different contexts, such as a test environment versus a production environment.

STS

  • The STS API calls will return

    The Amazon Resource Name (ARN) of the role that the app should assume. The duration, which specifies the duration of the temporary security credentials. A role session name, which is a string value that you can use to identify the session. This value can be captured and logged by CloudTrail to help you distinguish between your role users during an audit.

  • Decode error

    Decodes additional information about the authorization status of a request from an encoded message returned in response to an AWS request. For example, if a user is not authorized to perform an action that he or she has requested, the request returns.

    Client.UnauthorizedOperation response (an HTTP 403 response). Some AWS actions additionally return an encoded message that can provide details about this authorization failure Because of the right command used in the documentation, all other options are incorrect

    decode-authorization-message command to decode STS

Assume role with web identity

Returns a set of temporary security credentials for users who have been authenticated in a mobile or web application with a web identity provider. Example providers include Amazon Cognito, Login with Amazon, Facebook, Google, or any OpenID Connect-compatible identity provider.

Stepfunction

  • Intro

    AWS Step Functions provides serverless orchestration for modern applications. Orchestration centrally manages a workflow by breaking it into multiple steps, adding flow logic, and tracking the inputs and outputs between the steps. As your applications execute, Step Functions maintains application state, tracking exactly which workflow step your application is in, and stores an event log of data that is passed between application components. That means that if networks fail or components hang, your application can pick up right where it left off.

    Application development is faster and more intuitive with Step Functions, because you can define and manage the workflow of your application independently from its business logic. Making changes to one does not affect the other. You can easily update and modify workflows in one place, without having to struggle with managing, monitoring and maintaining multiple point-to-point integrations. Step Functions frees your functions and containers from excess code, so your applications are faster to write, more resilient, and easier to maintain.

AWS OpsWork

  • Intro

    AWS OpsWorks is a configuration management service that helps you configure and operate applications in a cloud enterprise by using Puppet or Chef. AWS OpsWorks Stacks and AWS OpsWorks for Chef Automate let you use Chef cookbooks and solutions for configuration management, while OpsWorks for Puppet Enterprise lets you configure a Puppet Enterprise master server in AWS. Puppet offers a set of tools for enforcing the desired state of your infrastructure, and automating on-demand tasks.

Cheat sheets

Lambda: https://tutorialsdojo.com/aws-cheat-sheet-aws-lambda/

Service comparission: https://tutorialsdojo.com/comparison-of-aws-services-for-udemy-students/

Cloudfront: https://tutorialsdojo.com/aws-cheat-sheet-amazon-cloudfront/

ECS : https://tutorialsdojo.com/aws-cheat-sheet-amazon-elastic-container-service-amazon-ecs/

S3: https://tutorialsdojo.com/aws-cheat-sheet-amazon-s3/

Elasticache: https://tutorialsdojo.com/amazon-elasticache/?src=udemy

Api gateway : https://tutorialsdojo.com/aws-cheat-sheet-amazon-api-gateway/

DynamoDB: https://tutorialsdojo.com/aws-cheat-sheet-amazon-dynamodb/

RDS vs Dynamodb: https://tutorialsdojo.com/aws-cheat-sheet-amazon-rds-vs-dynamodb/

DynamoDB scan vs query https://tutorialsdojo.com/dynamodb-scan-vs-query/?src=udemy

Cloudformation : https://tutorialsdojo.com/aws-cheat-sheet-aws-cloudformation/

SAM: https://tutorialsdojo.com/aws-cheat-sheet-aws-serverless-application-model-sam/

Cloudwatch : https://tutorialsdojo.com/aws-cheat-sheet-amazon-cloudwatch/

SNS : https://tutorialsdojo.com/aws-cheat-sheet-amazon-sns/

VPC https://tutorialsdojo.com/aws-cheat-sheet-amazon-vpc/

X-Ray https://tutorialsdojo.com/aws-cheat-sheet-aws-x-ray/

IAM : https://tutorialsdojo.com/aws-identity-and-access-management-iam/?src=udemy

Secret Manager : https://tutorialsdojo.com/aws-secrets-manager/

Code commit repository : https://tutorialsdojo.com/aws-codecommit-repository/

Instrumenting with x-ray https://tutorialsdojo.com/aws-cheat-sheet-instrumenting-your-application-with-aws-x-ray/

KMS: https://tutorialsdojo.com/aws-key-management-service-aws-kms/

ASG : https://tutorialsdojo.com/aws-auto-scaling/?src=udemy

Check out this AWS CloudFormation Cheat Sheet:

https://tutorialsdojo.com/aws-cheat-sheet-aws-cloudformation/

Check out this AWS Serverless Application Model Cheat Sheet:

https://tutorialsdojo.com/aws-cheat-sheet-aws-serverless-application-model-sam/

To Read: https://docs.aws.amazon.com/codedeploy/latest/userguide/error-codes.html

Code deploy flow https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html#reference-appspec-file-structure-hooks-run-order

Lambda traffic shifting with alias

https://docs.aws.amazon.com/lambda/latest/dg/configuration-aliases.html

Cloudformation - Template anotamy https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-anatomy.html

Buildspec- codebuild

https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html

Elasticache: caching strategies

https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/Strategies.html

ECS instances:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_instances.html

Code Deploy:

Appspec.yml file is used to define how your application will be deployed