AWS Kinesis

Send streaming data to Kinesis in AWS for analysis

• Streams

Producers send data to Streams

Stored from 24 hrs to 7 days

Stored in shards -> EC2 data consumers process the data -> store the processed data

Shard – 5 TPS  for reads

• Firehose

Data producers send data to Firehose . No shards . No streams etc. Analytics are run directly using lambda and send it to S3 and then to Redshift. No retention of data

• Analytics

Allows to run sql queries on Streams or Firehose data

AWS Elastic Beanstalk

Web service for deploying and scaling web applications supported  programming languages like .Net, JAVA, PHP etc

GUI based service that handles deplyoment , scaling , load balancing and application health

Deployment options:

• All at once – Updates and deploys to all instances at the same time . There is downtime. Deployment fails then rollback to previous version

• Rolling – Deployment in batches. Capacity is reduced during deployment. Not good for performance sensitive applications

• Rolling with additional batch – Maintains full capacity. Additional instances are provisioned, to rollback perform further rolling update

• Immutable – Deployments happen in new auto scaling group. Old group is deleted after new group is live

Elastic Beanstalk is customizable using configuration files written in YAML or JSON. These configuration files have  “.config” extension and saved in “.ebextensions” which should in top level of source code folder

EBS and RDS

Launching RDS from EBS – good for dev and test not for prod. Do not couple for Prod as RDS instance will get deleted

Additional security group added to Autoscaling group and Connection information added to EBS application.

 

 

 

 

 

AWS SNS

Web service that helps setup , maintain and send notifications from AWS

Push based system – Push messages to any device

Delivers messages  – trigger lambda, http endpoints, SMS, email to AWS SQS

Topics – Access points for recepients to recieve messages. Multiple subscribers to a topic

Instantaneous, simple apis, flexible delivery (different protocols)

50 cents / 1 millions notifications

SES is Email service only. Can be used for automated emails etc

AWS SQS

Pull based system

Visibility timeout window – message is inivisible during this time. Default is 30 secs and Maximum is 12 hours

upto 256kb of text in any format

allows decoupling of components

acts as buffer between producer and consumer of messages

Messages can be kept in queue from 1 min to 14 days

SQS Long Polling – Response is not returned until a message arrives in the queue or Long Polling timeout occurs. This reduces cost

Types of queues :

Standard(default) and FIFO

Standard –

• unlimited number of transactions per second
• messages are delivered atleast once
• order cannot be guaranteed

FIFO–

• Ordered is preserved
• No duplicates
• messages are preserved until consumed
• 300 messages/sec limit