AWS Certified Developer Exam Notes – SQS/SNS

Posted on by antonogeo

SQS

  • Fully managed queue service, which decouples applications
  • Multiple producers send messages in the queue
  • Multiple consumers poll messages from the queue
  • Unlimited throughput, unlimited number of messages
  • Default retention 4 days, maximum 14 days
  • Low latency <10ms
  • Message size limit: 256KB
  • We can have duplicated messages (at least one delivery)
  • Messages are out of order.
  • Producers use SDK to send messages to SQS (SendMessage API)
  • Receive up to 10 messages at a time
  • The consumer deletes the message that he reads with the DeleteMessage API
  • We can set up a Cloudwatch metric with ApproximateNumberOfMessages on the SQS Queue, and if the threshold is reached, we can auto-scale the EC2 consumers.
  • Queue Access Policy
    • We need an access policy for the consumer (for example ec2 instance) to poll (sqs:ReceiveMessage)
    • We need an access policy for a producer to send a message (sqs:SendMessage). An example of a producer could be S3 bucket with an Event Notification.
  • Delay Queue
    • Default: 0 seconds. Up to 15 minutes
    • Can override it using the DelaySeconds parameter
  • API
    • CreateQueue
    • PurgeQueue – Deletes the messages in a queue 
    • DeleteQueue – Deletes the queue
    • SendMessage (DelaySeconds), ReceiveMessage, DeleteMessage
    • MaxNumberOfMessages (default:1, max: 10) How many messages the consumer receives at a time
    • ReceiveMessageWaitTimeSeconds: Long Polling
    • ChangeMessageVisibility

 

Message Visibility Timeout

  • 30 seconds, during which the message is being read from one consumer. This message is invisible to other consumers during this period. After the period, and if the message is not deleted, it will become available again.
  • Use ChangeMessageVisibility API to change that
  • Possible values: 0 seconds – 12 hours

 

SQS Extended Client

  • Java Library
  • If we want to send a large message, first send it to S3, and send the metadata to the SQS queue. Consumers use this metadata to read from S3.

FIFO

  • Messages ordered in the queue
  • Limited throughput : 300 messages/s 3000 message/s with batching
  • Exactly once processing
  • Deduplication in 5 minutes. Disabled by default. Enable it on queue creation.
    • If you send the same message twice in 5 minutes, the second will be refused
    • It is implemented with a hash SHA-256
  • Messages with the same MessageGroupID will be in order, and they are going to be processed by the same consumer

 

Dead Letter Queue

  • We set how many times the message goes back to the queue. After some tries(MaximumReceives), the error may not be in the consumer but in the message. So we should send this to the Dead Letter Queue.
  • Configurable retention period. 1-14 days. After that, the messages are deleted.

 

Long polling

  • The consumer does a poll and waits. Set the WaitTimeSeconds to 1 sec to 20 sec
  • Decreases the API calls, and reads the message quicker, as soon as it is available
  • It is preferable to Short polling.

SNS

  • One message to many receivers. Pub/Sub pattern
  • We create SNS topic, with many subscribers
  • Up to 10m subscriptions per topic
  • 100k topics limit
  • Common usages: Cloudwatch(alarms), ASG, S3 etc
  • SQS + SQS: Fan Out
    • Push once in SNS, receive in all SQS queues that are subscribers
    • By this pattern, we can have many queues, and we only need to send the message once
    • We use this when we want to send a message that must be read by many different services. We create many queues, one for each consumer.
    • This is also used on S3 Events. We can have only one event per type and prefix(create images/). If we want to send this event to multiple destinations, we should use this pattern.
    • This is a common pattern where only one message is sent to the SNS topic and then “fan-out” to multiple SQS queues. This approach has the following features: it’s fully decoupled, no data loss, and you have the ability to add more SQS queues (more applications) over time.
  • Fifo Topic
    • Ordered messages with exactly one processing.
    • Subscription: SQS
  • Standard Topic
    • Subscription: SQS, Lambda, HTTP, SMS, email
    • Unordered and at least one processing
    • Highest throughput