AWS SQS: Overview and Case-study

AWS Logo

Amazon Web Services(AWS) is a subsidiary of Amazon providing on-demand cloud computing platforms and APIs to individuals, companies, and governments, on a metered pay-as-you-go basis. These cloud computing web services provide a variety of basic abstract technical infrastructure and distributed computing building blocks and tools.

Amazon Web services hold many services in its package. These services are widely used around the globe to lessen the amount of burden caused due to managing them. One such managed service is SQS, lets get a know-how of the benefits it has to render us.

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.

In Amazon SQS there are two types of queues which are Standard Queue and AWS SQS FIFO. Standard queue offers at least one delivery, best-effort ordering and maximum throughput. The SQS FIFO queues guarantee that the processed message takes place only once in the first in first out basis i.e. messages are processed exactly once, in the exact order that they are sent.

AWS Standard Queue Vs AWS SQS FIFO

Benefits of using Amazon SQS

Benefits of Amazon SQS
  • Security : You control who can send messages to and receive messages from an Amazon SQS queue. Server-side encryption (SSE) helps in transmitting sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).
  • Durability : For the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support at-least-once message delivery, and FIFO queues support exactly-once message processing.
  • Availability :Amazon SQS uses redundant infrastructure to provide highly-concurrent access to messages and high availability for producing and consuming messages.
  • Scalability :Amazon SQS can process each buffered request independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.
  • Reliability :Amazon SQS locks your messages during processing, so that multiple producers can send and multiple consumers can receive messages at the same time.
  • Customization :We can customize the queues as we like for example, we can set a default delay on a queue, store the contents of messages larger than 256 KB using Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB, with Amazon SQS holding a pointer to the Amazon S3 object, or we can split a large message into smaller messages.

Functionality of Amazon SQS:

  • Unlimited queues and messages: Create unlimited Amazon SQS queues with an unlimited number of message in any region
  • Payload Size: Message payloads can contain up to 256KB of text in any format. Each 64KB ‘chunk’ of payload is billed as 1 request.
  • Batches: Send, receive, or delete messages in batches of up to 10 messages or 256KB. Batches cost the same amount as single messages, meaning SQS can be even more cost effective for customers that use batching.
  • Long polling: Reduce extraneous polling to minimize cost while receiving new messages as quickly as possible. When your queue is empty, long-poll requests wait up to 20 seconds for the next message to arrive. Long poll requests cost the same amount as regular requests.
  • Retain messages : n queues for up to 14 days.
  • Send and read messages simultaneously.
  • Message locking: When a message is received, it becomes “locked” while being processed. This keeps other computers from processing the message simultaneously. If the message processing fails, the lock will expire and the message will be available again.
  • Queue sharing: Securely share Amazon SQS queues anonymously or with specific AWS accounts. Queue sharing can also be restricted by IP address and time-of-day.
  • Server-side encryption (SSE): Protect the contents of messages in Amazon SQS queues using keys managed in the AWS Key Management Service (AWS KMS). SSE encrypts messages as soon as Amazon SQS receives them. The messages are stored in encrypted form and Amazon SQS decrypts messages only when they are sent to an authorized consumer.
  • Dead Letter Queues (DLQ): Handle messages that have not been successfully processed by a consumer with Dead Letter Queues.

Case-Study: Oyster using AWS

Lots of companies reportedly use Amazon SQS in their tech stacks, including Pinterest, Amazon, BMW, NASA, EMS Driving Fuel IQ, Capital One, Redbus, Lyft, Oyster and many more.

New York-based Oyster.com shares unvarnished reviews of hotels in nearly 200 destinations worldwide.

The company’s own investigators visit each location to assess cleanliness, amenities, service and overall quality. Oyster takes hundreds of photos for each property, and every review includes dozens of untouched images (submitted by guests as well as investigators) that allow potential visitors to compare a hotel’s marketing material with reality.

Challenge

Since its 2009 launch, Oyster has published more than one million high-quality digital images. When this massive volume of images became too cumbersome to handle in-house, the company decided to offload the content to a central repository on Amazon Simple Storage Service (Amazon S3). In 2010, the company migrated to Amazon S3. Oyster chose moving to the cloud and Amazon S3 because storing images in our data center would have been too costly. Amazon S3 was a more economical solution.

Oyster reprocesses its entire collection of photographic images a few times each year to update the copyright year and, if necessary, to change the watermarks. Using their previous solution, reprocessing the entire collection of photographs required about 800 hours to complete. In addition, Oyster often recreated existing images in new formats and sizes for mobile and tablet devices. Resizing existing images and adding new ones was slowing down the rate at which the company was able to process the collection. The processes were slowing down. For example, When the iPad with Retina display came out, it took more than a week to create new sizes specifically for that resolution. Oyster considered purchasing additional hardware, but found the cost of new hardware and routine maintenance was too high, especially when the machines would sit idle most of the time.

Moreover, there were numerous software bugs in the multiprocessing solution that the company used, but since the solution didn’t scale, Oyster didn’t bother to fix them.

Role of Amazon Web Services

While the company is still running one local server, the bulk of the processing work now takes place on the AWS Cloud. Oyster is using a customized Amazon Linux AMI within Amazon EC2. Within this new environment, the company connects to Amazon S3 and Amazon Simple Queue Service (Amazon SQS) using boto, a Python interface to AWS. The images themselves are processed with the ImageMagick software available in the AMI package.

Oyster uses Amazon EC2 instances and Amazon SQS in an integrated workflow to generate the sizes they need for each photo. The team processes a few thousand photos each night, using Amazon EC2 Spot Instances. When Oyster processes the entire collection, it can use up to 100 Amazon EC2 instances. The team uses Amazon SQS to communicate the photos that need to be processed and the status of the jobs.

Benefits

Oyster’s old system needed approximately 400 hours to process one million photos. By using AWS, the company can process the same number of photos in about 20 hours → a 95 percent improvement. It used to take close to a week to produce photos specifically for the iPad. With AWS, the company can create the photos in just a few hours.

“It took less time to rewrite the code and do a full processing job with AWS than it took to do a single run with the old method. The documentation is straightforward and the dashboards are incredibly helpful.”

Eytan Seidman, Co-Founder and Vice President of Product.

Oyster has also been able to reduce in-house hardware expenses by repurposing two of its old servers, which were sitting idle more than 80 percent of the time. The company estimate that, they have saved roughly $10,000 in capital expenditures by moving to AWS, and reduced our operating expenses by an additional $10,000.

“AWS lets us move faster without worrying about machine expenditures or maintenance, which frees us to focus on other things.”

Eytan Seidman, Co-Founder and Vice President of Product.

Thanks for reading.