Microservices sound simple on a whiteboard.

Take a large application, split it into smaller services, deploy each service independently, and suddenly you have a scalable architecture.

Except… production rarely works that neatly.

Instead, you end up with:

Service A → Service B → Service C
              |
              v
          Service D
              |
              v
          Service E

Then someone asks:

“Why is checkout failing?”

And you discover that the checkout service depends on four other services, one of which depends on another three.

Now a simple request has become a distributed systems problem.

This is the part of microservices architecture that often gets overlooked.

Microservices don’t remove complexity. They move complexity into networking, deployment, observability, security, data consistency, and failure handling.

AWS provides an impressive set of building blocks for solving these problems.

But the real challenge isn’t knowing which AWS services exist.

It’s knowing which patterns to use — and which tempting patterns to avoid.

Let’s look at some of the most important ones.

What Does “Microservices on AWS” Actually Mean?

A microservices architecture typically breaks an application into independently deployable services.

For example, an e-commerce platform might have:

                  E-Commerce Platform
                           |
       +-------------------+-------------------+
       |                   |                   |
       v                   v                   v
   User Service       Order Service       Product Service
       |                   |                   |
       v                   v                   v
    Database            Database            Database

Each service owns a specific business capability.

The services might communicate through:

Synchronous APIs
       +
Asynchronous Events
       +
Message Queues

AWS gives you many ways to implement this:

• Amazon ECS
• Amazon EKS
• AWS Lambda
• Amazon API Gateway
• Elastic Load Balancing
• Amazon SQS
• Amazon SNS
• Amazon EventBridge
• Amazon DynamoDB
• Amazon Aurora
• Amazon CloudWatch
• AWS IAM

The architecture isn’t about using all of them.

It’s about choosing the right building blocks for the problem.

Pattern #1: Design Services Around Business Capabilities

One of the most important microservices patterns is business-aligned service boundaries.

Don’t start with:

UserController
OrderController
DatabaseController
EmailController

Instead think:

Identity
Orders
Payments
Inventory
Shipping
Notifications

For example:

                    E-Commerce
                        |
       +----------------+----------------+
       |                |                |
       v                v                v
    Orders           Payments        Inventory
       |                |                |
       v                v                v
   Order DB         Payment DB      Inventory DB

Each service owns a business capability.

This makes ownership clearer and reduces accidental coupling.

Anti-Pattern #1: Splitting Everything Into Services

One of the easiest mistakes is creating too many services.

Imagine an application with:

Customer Service
Address Service
Phone Service
Email Service
Order Service
OrderItem Service
Product Service
Price Service
Currency Service

Technically, these are separate services.

Architecturally, you may have created a distributed monolith.

Now:

Order
 |
 +--> Customer
       |
       +--> Address
       |
       +--> Phone
       |
       +--> Email

A single user action requires multiple network calls.

The system becomes harder to:

• Deploy
• Debug
• Monitor
• Test
• Scale
• Operate

A service boundary should represent a meaningful business boundary.

Smaller isn’t automatically better.

Pattern #2: Give Services Ownership of Their Data

A healthy microservices architecture generally follows:

A service owns its data.

For example:

Order Service
     |
     v
Order Database
     |
     v
Payment Service
     |
     v
Payment Database
Inventory Service
     |
     v
Inventory Database

The Order Service shouldn’t directly modify the Payment Service’s database.

Instead:

Order Service
     |
     | API / Event
     v
Payment Service
     |
     v
Payment Database

This creates a clean ownership boundary.

Anti-Pattern #2: One Shared Database

A common migration strategy is:

             Microservices
                   |
        +----------+----------+
        |          |          |
        v          v          v
                 Shared
                Database

At first, this seems convenient.

Every service can access the same tables.

But you’ve created hidden coupling.

For example:

Order Service
      |
      +--> orders table

Payment Service
      |
      +--> orders table
Reporting Service
      |
      +--> orders table

Now changing the database schema can break multiple services.

You may have independent deployments at the application layer, but your data layer is still tightly coupled.

That’s not really independent microservices.

Pattern #3: Use Synchronous Calls Carefully

Sometimes one service genuinely needs an immediate response.

For example:

Client
  |
  v
Order Service
  |
  | Check inventory
  v
Inventory Service
  |
  v
Response

A synchronous API call can make sense here.

AWS API Gateway, Application Load Balancers, service discovery, and container platforms can all play roles in this architecture.

But synchronous communication creates dependencies.

A → B → C → D

If D becomes slow, the latency can propagate backward.

Anti-Pattern #3: Synchronous Calls Everywhere

Imagine:

Order
 |
 v
Payment
 |
 v
Inventory
 |
 v
Shipping
 |
 v
Notification
 |
 v
Analytics

Now a single request depends on five services.

What happens when Notification is unavailable?

Does the order fail?

It shouldn’t.

This is where asynchronous communication becomes valuable.

Pattern #4: Use Events for Decoupling

Instead of:

Order Service
     |
     | Call Notification Service
     v
Notification

you can publish an event:

Order Service
     |
     | OrderCreated
     v
EventBridge / SNS
     |
     +----------+-----------+
     |          |           |
     v          v           v
Notification  Analytics   Loyalty

The Order Service doesn’t need to know who consumes the event.

That’s powerful.

A new consumer can be added without modifying the Order Service.

Example: Order Processing

Suppose a customer places an order.

The Order Service completes the order:

Order Service
     |
     | OrderCreated
     v
Event Bus
     |
     +--> Inventory
     |
     +--> Notifications
     |
     +--> Analytics
     |
     +--> Loyalty

This creates a much more flexible architecture.

The order workflow doesn’t need to synchronously wait for every downstream system.

Pattern #5: Use Queues for Work That Can Wait

Events are useful for broadcasting something that happened.

Queues are useful when you need controlled processing.

For example:

API
 |
 v
Order Service
 |
 v
SQS
 |
 v
Worker
 |
 v
External Payment API

If the external payment provider slows down, your worker can process messages at a controlled rate instead of allowing thousands of API requests to hammer the provider.

Queues provide:

• Buffering
• Retry handling
• Decoupling
• Backpressure
• Failure isolation

This is especially useful for workloads that don’t need an immediate response.

Anti-Pattern #4: Making Everything Event-Driven

After discovering events, teams sometimes go too far.

Suddenly everything becomes:

Event
  ↓
Event
  ↓
Event
  ↓
Event

Now nobody knows what actually caused what.

Debugging becomes painful.

For example:

OrderCreated
    ↓
CustomerUpdated
    ↓
NotificationRequested
    ↓
AccountChanged
    ↓
SomethingElseHappened

Events should have clear business meaning.

Don’t introduce asynchronous communication simply because it sounds more scalable.

Use it when decoupling, buffering, or asynchronous processing provides a real benefit.

Pattern #6: Put an API Gateway at the Edge

For client-facing APIs, a common pattern is:

                   Internet
                       |
                       v
                 API Gateway
                       |
            +----------+----------+
            |          |          |
            v          v          v
          User       Orders     Products
         Service     Service     Service

API Gateway can provide a central API entry point and integrate with authentication, throttling, logging, and routing capabilities.

This avoids exposing every internal service directly to the internet.

Instead:

Internet
   |
   v
API Gateway
   |
   +--> Service A
   +--> Service B
   +--> Service C

Your internal architecture remains behind a controlled boundary.

Anti-Pattern #5: One Giant API Gateway Configuration

Centralizing traffic at the edge is useful.

Centralizing every piece of business logic there isn’t.

Avoid turning the gateway into:

API Gateway
 |
 +--> Authentication
 +--> Transformation
 +--> Business Rules
 +--> Database Queries
 +--> Service Orchestration
 +--> Customer Logic

Now your gateway has become another monolith.

Keep responsibilities clear.

The gateway should primarily manage API-level concerns.

Business logic belongs in services.

Pattern #7: Design for Failure

In a distributed system, failures aren’t exceptional.

They’re normal.

A service can:

• Timeout
• Crash
• Become overloaded
• Return errors
• Lose network connectivity
• Become temporarily unavailable

Your architecture should expect this.

For example:

Order Service
      |
      v
Payment Service
      |
      X
   Timeout

What happens next?

A resilient architecture might use:

Timeout
   |
   +--> Retry carefully
   |
   +--> Circuit breaker
   |
   +--> Queue
   |
   +--> Fallback
   |
   +--> Alert

The correct response depends on the operation.

Anti-Pattern #6: Retry Everything

Retries sound like an easy solution.

Service fails?

Retry.

Fails again?

Retry again.

But uncontrolled retries can make an outage worse.

Imagine:

100 requests
     |
     v
Service fails
     |
     v
100 retries
     |
     v
Service becomes even more overloaded
     |
     v
More failures
     |
     v
More retries

You’ve created a retry storm.

Use:

• Exponential backoff
• Jitter
• Reasonable retry limits
• Timeouts
• Idempotency where appropriate

And don’t retry errors that won’t succeed by retrying.

Pattern #8: Make APIs Idempotent

Distributed systems frequently involve retries.

Suppose:

POST /payments

The client sends the request.

The server processes it.

But the response gets lost.

The client doesn’t know whether payment succeeded.

So it retries.

Now you have:

Payment #1 → ₹10,000
Payment #2 → ₹10,000

That’s obviously a serious problem.

For operations where duplicate execution is dangerous, idempotency is critical.

For example:

POST /payments
Idempotency-Key: abc123

The service can recognize that abc123 has already been processed.

This isn’t an AWS-specific concept.

It’s a fundamental distributed-systems pattern.

Pattern #9: Centralize Observability

With a monolith:

Application
    |
    v
Logs

With microservices:

Service A ──┐
Service B ──┤
Service C ──┤
Service D ──┤
Service E ──┘
             |
             v
       Observability

You need visibility across the entire request path.

At minimum, think about:

• Logs
• Metrics
• Traces
• Correlation/request IDs
• Error rates
• Latency
• Dependency failures

AWS services such as CloudWatch and distributed tracing capabilities can help build this layer.

Anti-Pattern #7: “We’ll Add Monitoring Later”

This becomes painful very quickly.

Imagine a customer reports:

“Checkout is slow.”

You inspect:

API Gateway → Order → Payment → Inventory

Which service is slow?

Without distributed observability, you may have no idea.

With correlation IDs and traces:

Request ID: abc123

API Gateway      20 ms
Order Service    40 ms
Payment Service  800 ms
Inventory        30 ms

The bottleneck becomes obvious.

Observability isn’t a luxury for microservices. It’s part of the architecture.

Pattern #10: Secure Service-to-Service Communication

Don’t assume that because services are inside a VPC, they’re automatically trusted.

A better model is:

Service A
   |
   | Authenticated request
   v
Service B
   |
   | Authorization
   v
Service B resources

Depending on your architecture, you might use:

• IAM
• Security groups
• Network policies
• Private networking
• Service-to-service authentication
• Secrets management
• TLS

Security should exist between services, not only at the internet boundary.

Anti-Pattern #8: Trust Everything Inside the Network

This model is dangerous:

Internet
   |
 Firewall
   |
   v
VPC
   |
   +--> Everything trusted

A compromised service shouldn’t automatically gain access to every other service.

Think in terms of least privilege:

Order Service
   |
   +--> Can call Payment
   |
   X--> Cannot access User DB directly
   |
   X--> Cannot call Admin Service

Microservices provide natural security boundaries.

Use them.

Pattern #11: Independent Deployment

One of the biggest promises of microservices is independent deployment.

For example:

Service A → Deploy v1.8
Service B → Deploy v2.3
Service C → No change

You shouldn’t need to redeploy the entire platform because one service changed.

AWS ECS, EKS, and Lambda can all support independent deployment models.

But independent deployment only works when service contracts are stable.

Anti-Pattern #9: Tight API Contracts

Suppose Service A expects:

{
  "customerId": "123",
  "name": "Sharath"
}

Service B changes it to:

{
  "id": "123",
  "fullName": "Sharath"
}

Now Service A breaks.

This is why API evolution matters.

Use practices such as:

• Backward-compatible changes
• Explicit API contracts
• Versioning when necessary
• Contract testing
• Deprecation strategies

A service should be able to evolve without immediately breaking every consumer.

Pattern #12: Use Containers or Serverless Based on the Workload

There is no requirement that every microservice must run on Kubernetes.

You can have:

                Microservices
                     |
        +------------+------------+
        |            |            |
        v            v            v
      Lambda        ECS          EKS

For example:

Lambda

Good for:

• Event-driven workloads
• Short-lived processing
• Variable traffic
• Small operational footprint

ECS

Good for:

• Containerized services
• Long-running applications
• More control without managing Kubernetes

EKS

Good for:

• Kubernetes-based platforms
• Complex container orchestration
• Existing Kubernetes expertise
• Advanced scheduling and ecosystem requirements

The question isn’t:

“Which platform is best?”

It’s:

“What operational model does this workload justify?”

Anti-Pattern #10: Kubernetes by Default

One of the most common architecture mistakes is:

“We’re building microservices, so we need Kubernetes.”

Not necessarily.

If you have five services and a small engineering team, operating Kubernetes may introduce more complexity than it removes.

You may be better off with:

API Gateway
    |
    +--> Lambda
    |
    +--> ECS

Use EKS when its capabilities justify its operational complexity.

Pattern #13: Separate Synchronous and Asynchronous Workflows

A good microservices architecture often uses both.

Synchronous

When the caller needs an immediate answer:

Client
  |
  v
API
  |
  v
Service
  |
  v
Response

Asynchronous

When work can happen later:

Client
  |
  v
API
  |
  v
Queue
  |
  v
Worker

A mature architecture doesn’t choose one communication style for everything.

It chooses based on the workflow.

Pattern #14: Build for Eventual Consistency

With separate databases, you lose the simplicity of a single ACID transaction across the entire application.

For example:

Order Created
      |
      v
Payment Processing
      |
      v
Inventory Reservation

These operations may happen independently.

You might temporarily have:

Order = Created
Payment = Pending
Inventory = Pending

That’s not necessarily a bug.

It can be eventual consistency.

The architecture needs to define what happens if one step fails.

For complex workflows, patterns such as Saga-style orchestration or choreography can help coordinate distributed business transactions.

Anti-Pattern #11: Pretending Distributed Transactions Are Local Transactions

In a monolith, you might have:

BEGIN TRANSACTION

Create Order
Update Inventory
Create Payment Record
COMMIT

In microservices, these may belong to different databases.

Trying to force everything into one distributed transaction can create significant complexity.

Instead, design explicit workflows:

Order Created
     |
     v
Payment Requested
     |
     v
Payment Completed
     |
     v
Inventory Reserved
     |
     v
Order Confirmed

And define compensating actions when necessary.

Pattern #15: Keep the Platform Boring

This might be the most underrated microservices pattern.

Your application architecture should not require every AWS service just because AWS provides it.

A perfectly reasonable platform might be:

Route 53
   |
   v
API Gateway
   |
   +--> Lambda
   |
   +--> ECS
   |
   v
SQS / EventBridge
   |
   v
DynamoDB / Aurora
   |
   v
CloudWatch

That’s enough to build sophisticated systems.

You don’t need to add another technology every time you discover a new architectural pattern.

Complexity is a cost.

A Production-Oriented AWS Microservices Architecture

Putting the patterns together:

                          Internet
                              |
                              v
                         Route 53
                              |
                              v
                       API Gateway
                              |
                    +---------+---------+
                    |                   |
                    v                   v
              Order Service       User Service
                    |                   |
                  ECS/EKS             Lambda
                    |                   |
                    v                   v
                Order DB            User DB
                    |
                    v
              EventBridge
                    |
          +---------+----------+
          |                    |
          v                    v
      Inventory            Notification
       Service               Service
          |                    |
          v                    v
      Inventory DB            SQS
                               |
                               v
                            Worker
                               |
                               v
                         External API

                    +----------------+
                    |   CloudWatch   |
                    | Logs/Metrics   |
                    |   Tracing      |
                    +----------------+

Notice something important.

This architecture doesn’t make every interaction synchronous.

It doesn’t give every service access to every database.

It doesn’t expose every service directly to the internet.

And it doesn’t require every service to run on Kubernetes.

Those decisions are intentional.

A Practical Microservices Checklist

Before creating a new service, ask:

Service boundary

• Does this represent a meaningful business capability?
• Does the team understand its ownership?
• Is the boundary stable?

Data

• Who owns this data?
• Does another service need direct database access?
• Can the interaction happen through an API or event?

Communication

• Does the caller need an immediate response?
• Could this operation be asynchronous?
• What happens when the downstream service is unavailable?

Reliability

• Are timeouts configured?
• Are retries bounded?
• Is backoff used?
• Is the operation idempotent?

Security

• Who can call this service?
• Is service-to-service authentication required?
• Are permissions least-privilege?

Observability

• Can I trace a request across services?
• Are logs centralized?
• Are latency and error metrics available?

Deployment

• Can this service be deployed independently?
• Can its API evolve without breaking consumers?

Cost

• Does this service need a continuously running container?
• Could Lambda solve the problem more simply?
• Are we paying for infrastructure we don’t actually need?

The Biggest Microservices Lesson

The hardest part of microservices isn’t creating services.

It’s managing the relationships between them.

The architecture becomes difficult when you have:

Too many services
+
Too many synchronous calls
+
Shared databases
+
Uncontrolled retries
+
Poor observability
+
Weak service boundaries

The goal should instead be:

Clear boundaries
+
Independent data ownership
+
Intentional communication
+
Failure isolation
+
Strong observability
+
Least privilege
+
Independent deployment

That’s what turns a collection of services into a real microservices architecture.

Final Takeaway

Microservices on AWS aren’t about choosing between ECS, EKS, Lambda, API Gateway, SQS, EventBridge, DynamoDB, or Aurora.

Those are implementation tools.

The architecture comes first.

A strong AWS microservices design usually follows a few principles:

1. Align services with business capabilities
2. Give services ownership of their data
3. Use synchronous communication intentionally
4. Use events and queues for decoupling
5. Design for failure
6. Make important operations idempotent
7. Secure service-to-service communication
8. Build observability from day one
9. Keep service contracts backward compatible
10. Choose the simplest platform that meets the requirements

And perhaps the most important rule:

Don’t build microservices because microservices are popular. Build them when independent ownership, deployment, scaling, and failure isolation provide real value.

A distributed system should earn its complexity.

What Has Worked for You?

If you’ve built microservices on AWS, which pattern made the biggest difference in production?

And which anti-pattern did you discover only after experiencing it the hard way?

Shared databases? Too many services? Synchronous dependencies? Retry storms? Kubernetes overkill?

Share your experience in the comments. The best microservices lessons usually come from the architecture that almost worked.

Your API works perfectly in development.

Then production traffic arrives.

A customer accidentally sends thousands of requests.

A mobile app gets stuck in a retry loop.

A marketing campaign suddenly drives 10× normal traffic.

Or worse, someone starts abusing a public endpoint.

Your backend doesn’t necessarily fail because the application is badly designed.

Sometimes, it fails because too many requests arrive at the same time.

This is where rate limiting and throttling become important.

With Amazon API Gateway, you can control how much traffic your APIs accept and prevent a sudden traffic spike from overwhelming downstream services.

But there’s an important distinction:

Throttling isn’t just about rejecting requests. It’s about controlling how traffic flows through your system.

Let’s understand how to design it properly.

What Is Rate Limiting?

Rate limiting answers a simple question:

How many requests should a client be allowed to make within a given period?

For example:

Client
   |
   | 100 requests/minute
   v
API Gateway
   |
   +----> Requests within limit → Backend
   |
   +----> Requests above limit → Throttled

Suppose an API allows:

100 requests/second

A client sending:

80 requests/second

is fine.

A client suddenly sending:

500 requests/second

may exceed the configured limit.

The gateway can throttle some of those requests rather than forwarding all of them to the backend.

Why Do APIs Need Throttling?

Without throttling, traffic can propagate through your entire architecture.

Imagine:

Internet
   |
   v
API Gateway
   |
   v
Lambda
   |
   v
DynamoDB

Now imagine 10,000 requests suddenly arrive.

Those requests don’t stop at API Gateway.

They can create pressure on:

• Lambda concurrency
• Database capacity
• Downstream APIs
• Connection pools
• CPU
• Memory
• Network bandwidth
• Third-party services

The failure can cascade.

Traffic Spike
     |
     v
API Gateway
     |
     v
Lambda concurrency ↑
     |
     v
Database pressure ↑
     |
     v
Latency ↑
     |
     v
Timeouts ↑
     |
     v
Retries ↑
     |
     v
Even more traffic

This is a classic feedback loop.

Throttling helps break it.

Throttling vs Rate Limiting

These terms are often used interchangeably, but there’s a useful distinction.

Rate limiting

Defines how much traffic is allowed.

For example:

100 requests/second

Throttling

Controls what happens when traffic exceeds the allowed rate.

For example:

Allowed → Process request
Exceeded → Delay / reject / throttle request

In API Gateway discussions, you’ll commonly see both concepts together.

The important thing is understanding the behavior you’re designing.

API Gateway’s Throttling Model

API Gateway uses a token bucket-style throttling mechanism.

You don’t need to memorize the implementation details to use it effectively.

The useful mental model is:

            Tokens
               |
               v
        +---------------+
Requests → Token Bucket |
        +---------------+
               |
        +------+------+
        |             |
     Allowed       Limited
        |             |
        v             v
     Backend       Throttled

The bucket has two important concepts:

• Rate — how quickly requests can be accepted over time.
• Burst — how many requests can temporarily arrive above the steady-state rate.

This distinction is extremely important.

Rate vs Burst

Suppose you configure:

Rate  = 100 requests/second
Burst = 200 requests

Think of it like:

Steady traffic
       |
       v
100 req/sec
       |
       +------+
              |
              v
         Burst capacity

The rate controls the sustained traffic level.

The burst provides temporary capacity for short spikes.

This matters because real applications rarely generate perfectly smooth traffic.

For example:

Second 1 → 60 requests
Second 2 → 80 requests
Second 3 → 180 requests
Second 4 → 70 requests

A system that only considers the average rate may look fine.

But the sudden spike still matters.

That’s why rate and burst should be considered together.

Example: E-Commerce API

Imagine an e-commerce API.

You have:

GET /products
POST /orders
GET /orders/{id}
POST /payments

Not all endpoints should necessarily have the same traffic policy.

For example:

GET /products

might receive thousands of requests.

While:

POST /payments

may need much tighter controls.

Why?

Because the cost and risk of the operations are different.

A product lookup might simply read cached data.

A payment operation could:

• Invoke multiple services
• Write to databases
• Call an external payment provider
• Trigger notifications
• Create financial side effects

So a blanket:

100 requests/sec for everything

may not be a good design.

Think About Traffic by API, Stage, and Method

One of the important design decisions with API Gateway is where you apply throttling.

You may want a broad default policy:

API
 |
 +--> Default throttling

And then more restrictive controls for particular operations.

Conceptually:

API
 |
 +--> GET /products       → Higher limit
 |
 +--> GET /orders         → Medium limit
 |
 +--> POST /payments      → Lower limit

The exact configuration depends on the API Gateway API type and features you’re using, but the architectural principle is universal:

Throttle based on the cost and importance of the operation, not just the number of endpoints.

Global Throttling

Suppose your API has:

GET  /products
GET  /orders
POST /orders
POST /payments

You might establish a broad traffic ceiling.

For example:

API
 |
 | 1,000 req/sec
 v
API Gateway
 |
 +--> Products
 +--> Orders
 +--> Payments

This creates a coarse safety boundary.

It’s useful as a first line of defense.

But it doesn’t distinguish between cheap and expensive operations.

That’s why global throttling alone is often insufficient for larger systems.

Per-Client Throttling

Now imagine your API has multiple customers.

Customer A
Customer B
Customer C
Customer D
        |
        v
    API Gateway

You don’t necessarily want one customer consuming all available capacity.

A better model could be:

Customer A → 100 req/sec
Customer B → 100 req/sec
Customer C → 500 req/sec

This becomes particularly useful for SaaS platforms and public APIs.

Different customers can have different capacity requirements.

For example:

Free Plan
   |
   +--> 10 req/sec

Pro Plan
   |
   +--> 100 req/sec

Enterprise
   |
   +--> Custom limit

This is where API management features such as API keys and usage plans can become relevant for REST APIs.

But Don’t Treat API Keys as Authentication

This is worth repeating.

An API key can help identify and control API consumers.

But:

An API key is not a substitute for strong authentication.

You might have:

Authentication
     |
     v
Cognito / IAM / Identity Provider
     |
     v
Traffic control
     |
     v
API Gateway throttling

These solve different problems.

Authentication answers:

Who is calling?

Throttling answers:

How much traffic should they be allowed to generate?

Protecting Backend Capacity

One of the biggest benefits of throttling is protecting downstream systems.

Imagine:

API Gateway
                   |
              1,000 req/sec
                   |
                   v
                Lambda
                   |
             500 concurrent
                   |
                   v
               Database

Suppose your database can comfortably handle only a fraction of that workload.

You don’t want API Gateway to blindly forward everything.

Instead:

Internet
   |
   v
API Gateway
   |
   | Controlled traffic
   v
Lambda
   |
   v
Database

The gateway becomes an early protection layer.

Throttling Is Not Capacity Planning

Here’s an important distinction.

Suppose your backend can process:

500 requests/sec

You configure:

API Gateway → 500 req/sec

That doesn’t magically guarantee your system can safely process 500 requests/sec.

Your request cost may vary.

For example:

GET /health

could be extremely cheap.

While:

POST /checkout

could trigger five downstream calls.

So:

500 requests/sec

doesn’t necessarily mean:

500 identical units of work/sec

This is why throttling must be designed alongside capacity planning.

Throttling and Lambda

Lambda introduces another important consideration: concurrency.

Imagine:

API Gateway
     |
     | 5,000 req/sec
     v
Lambda
     |
     +--> Concurrency increases
     |
     v
Database

If your Lambda functions invoke databases or third-party APIs, unlimited concurrency can become dangerous.

A more controlled architecture is:

API Gateway
     |
     | Throttle
     v
Lambda
     |
     | Controlled concurrency
     v
Database

You can use throttling together with Lambda concurrency controls to create multiple protection layers.

Throttling and Retries: The Dangerous Combination

One of the most common mistakes is implementing throttling without thinking about retries.

Suppose the API returns:

429 Too Many Requests

The client thinks:

“I’ll try again immediately.”

Then:

Request
   |
   v
API Gateway
   |
   X 429
   |
   v
Client
   |
   | Immediate retry
   v
API Gateway
   |
   X 429
   |
   v
Client

Now multiply this across thousands of clients.

You can create a retry storm.

Use Exponential Backoff

Clients should generally respect throttling responses and retry intelligently where retries are appropriate.

A common strategy is exponential backoff:

Attempt 1 → wait 100 ms
Attempt 2 → wait 200 ms
Attempt 3 → wait 400 ms
Attempt 4 → wait 800 ms

Often, some form of jitter is added so that many clients don’t retry at exactly the same time.

Conceptually:

429
 |
 +--> Wait
 |
 +--> Randomized delay
 |
 +--> Retry

This helps turn synchronized retries into distributed retries.

Not Every Request Should Be Retried

This is another important point.

A GET request may be safe to retry in many situations.

A payment operation may require much more careful handling.

For example:

POST /payments

If the client doesn’t know whether the request reached the backend, blindly retrying could potentially create duplicate business operations.

For critical operations, use patterns such as idempotency keys where appropriate.

For example:

POST /payments
Idempotency-Key: 8f92a1...

The backend can use the key to ensure that the same logical operation isn’t processed multiple times.

Throttling and idempotency solve different problems, but they often need to work together.

Protect Expensive APIs Differently

Consider these endpoints:

GET /products
GET /recommendations
POST /checkout
POST /reports/generate

They have very different resource costs.

A better design might conceptually look like:

Cheap read
    |
    +--> Higher rate

Normal operation
    |
    +--> Medium rate
Expensive operation
    |
    +--> Lower rate

For example, generating a large report may trigger:

API Gateway
    |
    v
Lambda
    |
    +--> Database
    +--> S3
    +--> Analytics
    +--> Notification

Allowing unlimited report generation just because the API can technically handle the HTTP requests is a bad idea.

Sometimes the right answer isn’t a higher throttle limit.

It’s an asynchronous architecture.

Client
  |
  v
API Gateway
  |
  v
Submit Job
  |
  v
Queue
  |
  v
Worker

Now the API remains responsive while expensive work is processed at a controlled rate.

Throttling vs Queuing

This distinction is extremely useful.

Throttling

Says:

“You can’t send more traffic than this right now.”

Too much traffic
      |
      v
Throttle
      |
      X

Queuing

Says:

“I can’t process this immediately, but I’ll process it later.”

Too much work
      |
      v
Queue
      |
      v
Worker

For synchronous APIs, throttling is often appropriate.

For long-running or expensive operations, queues can be a better design.

A Production Architecture

Let’s combine the ideas.

                        Internet
                            |
                            v
                     API Gateway
                            |
             +--------------+--------------+
             |              |              |
          AuthN          Throttling      Logging
             |              |              |
             +--------------+--------------+
                            |
                            v
                        Backend
                            |
              +-------------+-------------+
              |                           |
            Lambda                     Services
              |                           |
              +-------------+-------------+
                            |
                            v
                         Database

For asynchronous workloads:

Client
  |
  v
API Gateway
  |
  | Throttle
  v
Lambda
  |
  v
SQS
  |
  v
Workers
  |
  v
Database / External APIs

This architecture gives you much more control over traffic.

What Happens When the Limit Is Exceeded?

When API Gateway throttles a request, clients commonly receive:

429 Too Many Requests

The important lesson is:

A 429 isn’t necessarily an application failure.

It can be a deliberate protection mechanism.

Your monitoring should distinguish between:

5xx → Backend/system failure
4xx → Client/request issue
429 → Traffic exceeded configured capacity

That distinction makes operational debugging much easier.

Monitoring Throttling

You shouldn’t configure throttling and then forget about it.

Watch metrics such as:

• Request count
• Latency
• 4xx responses
• 5xx responses
• Throttled requests
• Backend integration errors

A useful operational pattern is:

Traffic
   |
   v
Requests ↑
   |
   v
429s ↑
   |
   v
Investigate

If you consistently see throttling during normal business traffic, your limit may simply be too low.

If throttling suddenly appears during an unexpected spike, it may be doing exactly what you designed it to do.

Common Mistakes

Mistake 1: Setting the highest possible limit

More traffic isn’t always better.

A higher gateway limit can simply move the bottleneck downstream.

Always ask:

What can my backend safely process?

Mistake 2: Using one limit for every endpoint

Not every API operation costs the same.

Protect expensive operations more aggressively.

Mistake 3: Ignoring burst traffic

An API may be fine at a sustained rate but struggle with sudden spikes.

Think about both:

Steady-state rate
+
Burst capacity

Mistake 4: Ignoring client retries

A throttled client that retries immediately can turn a traffic spike into a retry storm.

Design clients with exponential backoff and jitter.

Mistake 5: Using throttling as your only protection

Throttling is one layer.

You may also need:

• WAF protections
• Authentication
• Authorization
• Lambda concurrency controls
• Database capacity controls
• Queues
• Caching
• Circuit breakers
• Application-level quotas

Mistake 6: Confusing throttling with DDoS protection

Throttling helps control application traffic, but it isn’t a complete DDoS defense strategy.

For internet-facing APIs, consider the broader AWS security architecture, including services such as AWS WAF and AWS Shield where appropriate.

How Should You Choose Your Limits?

Don’t start with:

“What number should I put into API Gateway?”

Start with:

“How much work can my system safely handle?”

Then work backward.

For example:

Database safe capacity
        ↓
Backend safe concurrency
        ↓
Application throughput
        ↓
API traffic limit
        ↓
Client quotas

Suppose testing shows:

Backend safe capacity = 300 req/sec

You might not immediately configure:

API Gateway = 300 req/sec

You may want some safety margin.

The exact value should come from load testing, traffic patterns, downstream limits, and business requirements.

Don’t guess your production limits. Measure them.

A Practical Throttling Checklist

Before launching an API, ask:

Traffic

• What’s the expected average traffic?
• What’s the expected peak?
• How large can a sudden burst be?

Backend

• What’s the safe backend throughput?
• What’s the Lambda concurrency limit?
• What can the database handle?
• Are there third-party API limits?

API Gateway

• Is there a sensible default throttle?
• Do expensive endpoints need tighter controls?
• Do different customers need different quotas?

Clients

• Do clients understand 429 responses?
• Do they use exponential backoff?
• Do they use jitter?
• Are critical operations idempotent?

Architecture

• Should this operation be asynchronous?
• Can caching reduce repeated requests?
• Would a queue protect a downstream system?

Monitoring

• Are throttled requests visible?
• Can you distinguish expected throttling from an incident?
• Are alerts based on meaningful thresholds?

The Bigger Lesson

Rate limiting isn’t really about putting a number like:

100 requests/sec

into API Gateway.

It’s about protecting the entire system behind the API.

Think about the complete chain:

Client
  |
  v
API Gateway
  |
  v
Lambda / ECS
  |
  v
Database
  |
  v
External Services

Every layer has a capacity limit.

Your API gateway should help ensure that traffic entering the system doesn’t overwhelm the weakest component.

And when traffic exceeds capacity, you have choices:

Traffic spike
     |
     +--> Throttle
     |
     +--> Queue
     |
     +--> Cache
     |
     +--> Scale
     |
     +--> Reject

Good architecture isn’t about accepting every request.

It’s about accepting the right amount of work safely.

Final Takeaway

API Gateway throttling is a relatively small configuration feature with a much bigger architectural purpose.

It can help you:

• Protect backend capacity
• Control traffic spikes
• Prevent noisy neighbors
• Manage customer usage
• Reduce cascading failures
• Make overload behavior predictable

But effective throttling requires more than setting a rate.

You need to consider:

Rate
 +
Burst
 +
Backend Capacity
 +
Client Retries
 +
Quotas
 +
Monitoring
 +
Async Processing

The goal isn’t:

“How many requests can my API accept?”

The better question is:

“How much work can my entire system safely process?”

Once you answer that, your API Gateway throttling strategy becomes much easier to design.

How Are You Handling API Traffic Spikes?

Have you used API Gateway throttling in production?

Did you solve overload using throttling, caching, queues, autoscaling, or a combination of them?

Share your approach in the comments. Real-world traffic patterns often teach us more than architecture diagrams ever can.

And if this article helped you understand API throttling a little better, share it with another engineer designing a production API.

Building an API is relatively easy.

Making sure only the right people and applications can access it is where things get interesting.

A typical AWS application might look like this:

Client
   |
   | HTTPS
   v
API Gateway
   |
   v
Lambda / ECS / Backend
   |
   v
Database

At first, you might think:

“I’ll just put authentication in front of API Gateway.”

But authentication is only one part of API security.

You also need to think about:

• Who is calling the API?
• How do we verify their identity?
• What are they allowed to do?
• Should AWS services be able to call the API?
• How do we protect machine-to-machine communication?
• How do we prevent unauthorized requests?
• Where should authorization decisions happen?

This is where Amazon API Gateway, AWS IAM, and Amazon Cognito work together.

The important part is understanding that they solve different problems.

Authentication vs Authorization

Before looking at AWS services, let’s separate two concepts that are often confused.

Authentication

Authentication answers:

Who are you?

For example:

User → Login → Identity Provider → Token

The system verifies that the user is actually who they claim to be.

Authorization

Authorization answers:

What are you allowed to do?

For example:

Alice
   |
   +--> GET /orders       ✓
   +--> POST /orders      ✓
   +--> DELETE /users     ✗

Authentication identifies Alice.

Authorization determines what Alice can do.

A secure API needs both.

Where API Gateway Fits

API Gateway sits at the entry point to your application.

Instead of allowing clients to directly reach your backend:

Internet
   |
   v
Lambda

you can place API Gateway in front:

Internet
   |
   v
API Gateway
   |
   v
Lambda

Now API Gateway can become an important security boundary.

It can handle things such as:

• Authentication
• Authorization
• Throttling
• Request controls
• CORS
• TLS termination
• Logging
• Integration with AWS identity mechanisms

The backend doesn’t necessarily need to implement every security mechanism itself.

The Three Pieces

A useful mental model is:

                  API Gateway
                      |
          +-----------+-----------+
          |                       |
        IAM                    Cognito
          |                       |
      AWS / apps              Users / apps
          |                       |
          +-----------+-----------+
                      |
                    API
                      |
                   Backend

But IAM and Cognito aren’t interchangeable.

They are designed for different identity models.

IAM: Best for AWS-Aware Access

AWS Identity and Access Management (IAM) is fundamentally about controlling access to AWS resources.

For example:

EC2
Lambda
ECS
CI/CD Pipeline
AWS Service
Developer Tool
      |
      v
     IAM
      |
      v
API Gateway

Imagine you have a Lambda function that needs to call another API.

You generally don’t want to create a username and password for that Lambda.

Instead, you can use an IAM identity and sign the request using AWS authentication mechanisms.

Conceptually:

Lambda
   |
   | IAM credentials
   v
API Gateway
   |
   | Verify AWS identity
   v
Backend

This is particularly useful for machine-to-machine communication inside AWS.

Example: Lambda Calling an Internal API

Suppose you have:

Order Service
      |
      | API request
      v
Payment Service

Both services run in AWS.

You could design the Payment API to require IAM authorization.

Then:

Order Lambda
     |
     | SigV4 signed request
     v
API Gateway
     |
     | IAM authorization
     v
Payment Service

The API can determine:

“This request came from an authorized AWS principal.”

That’s very different from asking a human user to log in.

Cognito: Best for Application Users

Now consider a mobile or web application.

Your users might be:

Alice
Bob
Charlie
David

You probably don’t want each user to have an AWS IAM identity with permissions to your application APIs.

Instead, Amazon Cognito can manage application identities.

The flow looks more like:

User
 |
 | Login
 v
Cognito
 |
 | ID / Access Token
 v
Application
 |
 | Bearer Token
 v
API Gateway
 |
 v
Backend

Cognito is designed for application authentication scenarios such as:

• Web applications
• Mobile applications
• Customer-facing applications
• User registration
• User login
• MFA
• Federated identity
• OAuth 2.0 / OpenID Connect-based flows

The user doesn’t need to understand AWS IAM.

They simply authenticate with your application.

A Real Example: E-Commerce API

Imagine an e-commerce application.

You have:

React Web App
      |
      v
Cognito
      |
      | Access Token
      v
API Gateway
      |
      v
Lambda
      |
      v
DynamoDB

A user logs in.

Cognito authenticates them and provides tokens.

The application then calls:

GET /orders
Authorization: Bearer <access-token>

API Gateway validates the token through the configured authorizer.

If the token is valid:

Request
   |
   v
API Gateway
   |
   | ✓ Authenticated
   v
Lambda

If the token is invalid:

Request
   |
   v
API Gateway
   |
   | ✗ Unauthorized
   X

The request never reaches your backend.

That’s an important security benefit.

Authentication Doesn’t Automatically Mean Authorization

Here’s where many systems get into trouble.

Suppose Cognito tells you:

User = Alice

That doesn’t automatically mean:

Alice can access every order.

You still need authorization logic.

For example:

GET /orders/123

The backend should verify that Alice is actually allowed to access order 123.

You might have:

Alice
 |
 +--> Order 123 ✓
 +--> Order 456 ✓
 +--> Order 999 ✗

This is application-level authorization.

API Gateway can authenticate the caller, but business authorization often belongs in your application.

Authentication at the Edge, Authorization in the Application

This leads to a useful architectural pattern:

                   Internet
                       |
                       v
                 API Gateway
                       |
                Authentication
                       |
                       v
                 Application
                       |
                Business Rules
                       |
                       v
                    Data

API Gateway answers:

“Is this request from a valid identity?”

Your application answers:

“Is this identity allowed to perform this specific business operation?”

This separation keeps responsibilities clear.

IAM vs Cognito: Don’t Mix Their Jobs

A simple way to think about it:

IAM
 |
 +--> AWS identities
 +--> AWS services
 +--> Workloads
 +--> Infrastructure
 +--> Machine-to-machine access

Cognito
 |
 +--> Application users
 +--> Web/mobile authentication
 +--> Customer identities
 +--> User login
 +--> Tokens

For example:

Human user

Mobile App
    |
    v
Cognito
    |
    v
API Gateway

AWS workload

Lambda
   |
   v
IAM
   |
   v
API Gateway

This distinction alone prevents a lot of architectural confusion.

What About API Keys?

API keys are another mechanism you’ll encounter with API Gateway.

They can be useful for identifying API consumers and controlling usage.

But here’s an important point:

An API key should not be treated as a replacement for strong user authentication.

Think of API keys more as:

Who is consuming this API?
How much can they consume?

rather than:

Is this human identity authenticated?

For user authentication, use an appropriate identity mechanism such as Cognito or another supported identity provider.

For AWS workloads, IAM may be appropriate.

A Better Production Architecture

Let’s combine the pieces.

Imagine a SaaS application with:

• Web users
• Mobile users
• Internal AWS services
• Backend microservices

You could have:

                        Internet
                            |
                +-----------+-----------+
                |                       |
          Web / Mobile              Internal AWS
                |                       |
                v                       v
             Cognito                   IAM
                |                       |
                | Tokens                | Signed Requests
                |                       |
                +-----------+-----------+
                            |
                            v
                       API Gateway
                            |
                   Authentication
                            |
                            v
                     Backend Services
                            |
                            v
                         Database

Now each identity model has a clear purpose.

Protecting Different API Routes

Not every API endpoint necessarily needs the same authorization model.

For example:

GET /products

might be public.

GET /orders

might require an authenticated user.

POST /payments

might require an authenticated user plus application-level authorization.

And:

POST /internal/reconcile

might only be callable by an AWS workload using IAM.

Conceptually:

API Gateway
 |
 +-- GET /products
 |       |
 |       +--> Public
 |
 +-- GET /orders
 |       |
 |       +--> Cognito
 |
 +-- POST /payments
 |       |
 |       +--> Cognito + Application Authorization
 |
 +-- POST /internal/reconcile
         |
         +--> IAM

This is much better than using one authentication mechanism for everything.

Don’t Put Every Authorization Decision in API Gateway

There’s a temptation to make API Gateway responsible for all authorization.

For example:

API Gateway
 |
 +--> Is user logged in?
 +--> Is user an admin?
 +--> Does user own this order?
 +--> Can user modify this invoice?
 +--> Is this payment allowed?

That can become difficult to maintain.

Some decisions are infrastructure-level.

Others are business-level.

Consider:

Can this principal call the API?

That’s a good gateway-level security question.

But:

Does this customer own invoice INV-123?

That’s generally an application/business rule.

A cleaner model is:

API Gateway
    |
    | Identity + coarse access
    v
Application
    |
    | Business authorization
    v
Database

Defense in Depth

API authentication should not be your only security layer.

A production architecture should look more like:

               Internet
                   |
                   v
             API Gateway
                   |
          +--------+--------+
          |                 |
      Authentication     Throttling
          |                 |
          +--------+--------+
                   |
                   v
              Application
                   |
          +--------+--------+
          |                 |
     Authorization      Validation
          |                 |
          +--------+--------+
                   |
                   v
                Database

Each layer protects against different problems.

For example:

• API Gateway controls access to the API.
• Authentication verifies identity.
• Authorization verifies permissions.
• Application validation checks inputs.
• Database permissions provide another security boundary.
• Logging helps detect suspicious behavior.

Security shouldn’t depend on a single check.

Common Mistakes

Mistake 1: Treating authentication as authorization

A valid token doesn’t automatically mean the user can perform every operation.

Always separate:

Who are you?

from:

What are you allowed to do?

Mistake 2: Using IAM for normal application users

IAM is powerful, but that doesn’t mean every application user should become an IAM principal.

For customer-facing applications, Cognito or another identity provider is generally a more natural fit.

Mistake 3: Using API keys as passwords

API keys can help identify and control API consumers, but they aren’t a complete user authentication strategy.

Mistake 4: Trusting claims without validating tokens

Never simply decode a JWT and assume it’s trustworthy.

The token must be properly validated, including its issuer, audience and relevant claims according to your authorization design.

Mistake 5: Putting business authorization entirely at the gateway

API Gateway shouldn’t need to understand every business rule in your application.

Keep domain-specific authorization close to the business logic.

A Practical Checklist

Before putting an API into production, ask:

Identity

• Who calls this API?
• Humans?
• Applications?
• AWS services?
• External partners?

Authentication

• How is the caller authenticated?
• Cognito?
• IAM?
• Another identity provider?

Authorization

• What can the authenticated identity do?
• Is authorization based on roles?
• Groups?
• Scopes?
• Resource ownership?

API Gateway

• Are protected routes actually protected?
• Are public routes intentionally public?
• Is throttling configured where appropriate?
• Is CORS configured correctly?

Application

• Does the backend validate authorization again where business rules require it?
• Are inputs validated?
• Are sensitive operations protected?

Monitoring

• Are API access logs enabled?
• Are authentication failures visible?
• Are suspicious patterns detectable?

The Bigger Lesson

One of the most useful things you can do when designing AWS security is to stop thinking about services individually.

Don’t think:

"I need IAM."
"I need Cognito."
"I need API Gateway."

Instead think in terms of the security flow:

Who is calling?
      |
      v
How do we authenticate them?
      |
      v
What are they allowed to access?
      |
      v
Where should that decision happen?
      |
      v
How do we detect failures or abuse?

Then choose the AWS services that implement those requirements.

That’s a much stronger approach to architecture.

Final Takeaway

API Gateway, IAM, and Cognito are not competing security solutions.

They solve different parts of the problem.

Cognito
   |
   +--> Application user identity

IAM
   |
   +--> AWS identity and workload access

API Gateway
   |
   +--> API security boundary
   +--> Authentication integration
   +--> Request control
   +--> Traffic protection

Application
   |
   +--> Business authorization
   +--> Resource ownership
   +--> Domain rules

A good production design often looks like:

User / Service
      |
      v
Identity Provider
      |
      v
API Gateway
      |
      | Authentication
      v
Application
      |
      | Authorization
      v
Database

The goal isn’t to put every security feature everywhere.

The goal is to put the right security control at the right layer.

And that’s what makes an API not just functional, but secure.

How Are You Securing Your APIs?

Do you use Cognito, IAM, another identity provider, or a combination of them?

And where do you keep your authorization logic — API Gateway, Lambda/application code, or a dedicated authorization layer?

Share your approach in the comments. The interesting part of API security is rarely the service itself — it’s where you draw the security boundaries.

When building an API on AWS, there’s a surprisingly easy question to get wrong:

Should I use a REST API or an HTTP API in Amazon API Gateway?

At first glance, they look almost identical.

Both can expose HTTP endpoints.
Both can integrate with Lambda.
Both can be secured.
Both can handle production workloads.

So why does AWS offer two API types?

Because they are optimized for different levels of API functionality.

Choosing between them isn’t really about whether one is “better.”

It’s about understanding what your application actually needs.

First: What Is API Gateway?

Amazon API Gateway is a managed service for creating and exposing APIs without running API servers yourself.

A typical architecture might look like:

Client
   |
   | HTTPS
   v
API Gateway
   |
   +----> Lambda
   |
   +----> ECS / ALB
   |
   +----> Other AWS Services

API Gateway handles things such as:

• HTTP endpoints
• Authentication and authorization
• Request routing
• Throttling
• CORS
• Monitoring
• Custom domains
• Integration with backend services

But when you create an API Gateway API, you need to choose an API type.

That’s where REST APIs and HTTP APIs come in.

REST API vs HTTP API

The simplest way to think about the difference is:

HTTP APIs are designed to provide the essential API Gateway functionality with a simpler and lower-cost model. REST APIs provide a broader feature set when you need advanced API management capabilities.

This distinction is more useful than memorizing a feature matrix.

Think of it like this:

                   API Gateway
                        |
              +---------+---------+
              |                   |
           HTTP API            REST API
              |                   |
        Simple APIs        Advanced APIs
        Lower overhead    More features
        Lower cost        More control

Now let’s understand what actually changes.

1. Feature Set

This is probably the biggest difference.

HTTP APIs focus on the features most applications need:

• HTTP routing
• Lambda integrations
• HTTP integrations
• JWT authorization
• CORS
• Custom domains
• Stages
• Access logging
• Basic API monitoring

REST APIs provide additional API-management capabilities, including features such as:

• API keys
• Usage plans
• Request validation
• Transformation capabilities
• More advanced authorization options
• Fine-grained request/response controls
• Private API support
• Broader integration capabilities

So the question shouldn’t be:

“Which API is more powerful?”

The better question is:

“Do I actually need the additional capabilities provided by REST APIs?”

If the answer is no, HTTP API is often the simpler choice.

2. Authentication and Authorization

This is an area where the difference becomes important.

HTTP APIs support modern authorization patterns such as:

Client
   |
   | JWT
   v
HTTP API
   |
   | Validate token
   v
Backend

For example, your application might use an identity provider to issue a JWT.

The client sends:

Authorization: Bearer eyJ...

API Gateway validates the token before allowing the request through.

HTTP APIs support JWT authorizers, making them a good fit for modern applications using OAuth 2.0 / OpenID Connect-style authentication.

REST APIs Offer More Authorization Choices

REST APIs support additional authorization mechanisms and API Gateway-specific authorization features.

For example:

Client
   |
   v
REST API
   |
   +--> IAM authorization
   |
   +--> Lambda authorizer
   |
   +--> Cognito user pools
   |
   +--> API keys / usage plans
   |
   v
Backend

This matters when your API isn’t just an application backend but also an externally consumed platform.

For example:

Mobile App
      |
      v
API Gateway
      |
      +--> Customer A
      +--> Customer B
      +--> Customer C

If you need sophisticated consumer management, quotas, and API-product-style controls, REST APIs can make more sense.

3. API Keys and Usage Plans

This is one of the differences developers often discover too late.

Suppose you’re building a public API.

You want:

Customer A → 1,000 requests/hour
Customer B → 10,000 requests/hour
Customer C → 100,000 requests/hour

You may want to identify consumers using API keys and apply usage plans.

This is where REST APIs become useful.

You can model:

API Key
   |
   v
Usage Plan
   |
   +--> Rate limit
   +--> Burst limit
   +--> API stages

HTTP APIs don’t provide the same API-key/usage-plan feature set.

So if your architecture depends heavily on API keys and usage plans, REST API is usually the appropriate choice.

4. Request and Response Transformation

REST APIs provide more advanced capabilities for modifying requests and responses at the API Gateway layer.

For example:

Client Request
      |
      v
API Gateway
      |
      | Transform
      v
Backend Request

You might receive:

{
  "userId": "123",
  "name": "Sharath"
}

while your backend expects:

{
  "id": "123",
  "username": "Sharath"
}

REST APIs can perform API Gateway-side transformations using mapping templates and related features.

This can be useful in integration-heavy architectures.

But there’s a design question worth asking:

Should API Gateway really be responsible for this transformation?

Sometimes the cleaner solution is to put the transformation inside the application or integration layer.

Don’t choose REST API solely because you might need transformations someday.

Choose it when that capability is actually part of your architecture.

5. Private APIs

REST APIs also provide capabilities for exposing APIs privately within AWS networking environments.

For example:

VPC
 |
 +-----------------------------+
 |                             |
 | Client                      |
 |   |                         |
 |   v                         |
 | Private API Gateway         |
 |   |                         |
 |   v                         |
 | Internal Backend            |
 |                             |
 +-----------------------------+

This can be useful for internal enterprise APIs where the API should not be publicly reachable.

If private API functionality is a core requirement, REST API may be the better fit.

6. Protocol Support

Despite the names, don’t think of this as:

HTTP API = HTTP
REST API = REST

That’s misleading.

Both are API Gateway products that expose HTTP-based APIs.

The distinction is primarily about features and architecture, not whether one uses HTTP and the other doesn’t.

A REST API can expose endpoints such as:

GET    /users
POST   /users
GET    /users/{id}
DELETE /users/{id}

An HTTP API can expose essentially the same style of endpoints.

For a normal CRUD backend, there may be little difference from the application’s perspective.

7. Pricing

Cost is another important reason HTTP APIs exist.

HTTP APIs are generally positioned as the lower-cost API Gateway option compared with REST APIs.

For a high-volume application, the difference can become meaningful.

Imagine:

100 million API requests/month

Even a relatively small per-request pricing difference can translate into a significant monthly cost.

This is why architecture decisions shouldn’t stop at:

“Both APIs work.”

Instead ask:

“Which API provides the functionality we need without paying for capabilities we don’t use?”

That is a much better engineering decision.

8. Performance

HTTP APIs are designed with a simpler architecture and lower feature overhead.

For many workloads, that makes them an attractive choice when you don’t need the advanced REST API feature set.

But don’t turn this into:

“HTTP APIs are always faster.”

That’s too simplistic.

Real-world API latency depends on much more than API Gateway:

Client
  |
  v
API Gateway
  |
  v
Lambda
  |
  v
Database
  |
  v
External Service

If your Lambda takes 300 ms and your database query takes 200 ms, shaving a small amount of gateway overhead isn’t going to transform your application.

Measure the complete request path.

9. Example: A Serverless Web Application

Imagine you’re building an e-commerce backend.

You have:

React / Mobile App
        |
        v
   API Gateway
        |
        v
      Lambda
        |
        v
    DynamoDB

Your requirements are:

• JWT authentication
• CRUD APIs
• CORS
• Custom domain
• Lambda integration
• CloudWatch logging
• Basic throttling

You don’t need:

• API keys
• Usage plans
• Request transformation
• Private API functionality
• Advanced API management

In this scenario, an HTTP API is likely the better starting point.

Why?

Because you’re getting the functionality you need without introducing unnecessary complexity.

10. Example: A Public Developer API

Now imagine a different architecture.

You’re building an API platform consumed by external customers.

You need:

Customer A
Customer B
Customer C
Customer D
      |
      v
  API Gateway
      |
      +--> Authentication
      +--> API Keys
      +--> Usage Plans
      +--> Rate Limits
      +--> Request Transformation
      +--> Multiple Stages
      |
      v
   Backend Services

Now the advanced API management capabilities become important.

In this scenario, REST API may be the better choice.

The extra features aren’t unnecessary anymore.

They’re part of the product you’re building.

11. Don’t Choose Based on the Name

One common mistake is assuming:

REST API must be better because REST is more established.

That’s not how API Gateway works.

Another mistake is:

HTTP API is newer and cheaper, so it must always be better.

Also incorrect.

The right decision depends on your requirements.

A useful mental model is:

Need basic API functionality?
          |
          v
       HTTP API
          |
          |
Need advanced API Gateway features?
          |
          v
       REST API

12. A Practical Decision Framework

Before creating an API Gateway API, ask these questions.

Do I need API keys and usage plans?

If yes:

REST API

Do I need advanced request/response transformations?

If yes:

REST API

Do I need private API capabilities?

If yes:

REST API

Do I primarily need Lambda/HTTP integrations with JWT authorization?

If yes:

HTTP API is often a strong choice.

Is cost sensitivity important?

If yes:

Start by evaluating HTTP API.

Do I need advanced API management features?

If yes:

Evaluate REST API.

13. A Simple Architecture Rule

For many modern serverless applications, you can start with:

API Gateway
                        |
                    HTTP API
                        |
                  JWT Authorizer
                        |
                      Lambda
                        |
                    DynamoDB

Keep the gateway layer simple.

If your requirements evolve toward:

                   API Gateway
                        |
                    REST API
                        |
          +-------------+-------------+
          |             |             |
       API Keys     Usage Plans   Transformations
          |             |             |
          +-------------+-------------+
                        |
                     Backend

then REST API becomes more compelling.

This is an important architectural principle:

Don’t add infrastructure features because they’re available. Add them because the system needs them.

Common Mistakes

Mistake 1: Choosing REST API by default

Many teams automatically choose REST API because it’s familiar.

That can result in paying for capabilities they don’t actually use.

Mistake 2: Choosing HTTP API only because it’s cheaper

Cost is important, but not the only consideration.

If your application requires REST API-specific functionality, saving money on the gateway can create architectural problems elsewhere.

Mistake 3: Confusing REST with HTTP

REST is an architectural style.

HTTP is a protocol.

API Gateway’s REST API and HTTP API are product types with different capabilities.

Don’t interpret their names too literally.

Mistake 4: Putting too much logic in API Gateway

Just because API Gateway can transform requests, authorize users, throttle traffic, and route requests doesn’t mean every responsibility belongs there.

A healthy architecture might look like:

API Gateway
    |
    | Routing + Authentication
    v
Application
    |
    | Business Logic
    v
Database

Keep business logic where it belongs.

My Rule of Thumb

When starting a new API Gateway project, I would approach it like this:

Start with HTTP API if:

• The API is relatively simple.
• You’re building serverless applications.
• JWT authorization is sufficient.
• You don’t need API keys and usage plans.
• You don’t need REST API-specific advanced features.
• Cost efficiency matters.

Consider REST API if:

• You need API keys and usage plans.
• You need advanced API Gateway request/response features.
• You need private API capabilities.
• You’re building a more sophisticated API management layer.
• Your existing architecture depends on REST API-specific functionality.

The Bigger Lesson

The REST vs HTTP API decision is really an example of a broader cloud architecture principle:

Choose capabilities based on requirements, not product popularity.

A simple API doesn’t become better because you put more infrastructure in front of it.

In fact, unnecessary features can increase:

• Cost
• Configuration complexity
• Operational overhead
• Debugging difficulty
• Security surface area

If HTTP API satisfies your requirements, there’s little reason to choose REST API just because it has more features.

And if your application genuinely needs those advanced capabilities, don’t avoid REST API simply because HTTP API is cheaper.

The cheapest architecture isn’t necessarily the best architecture.

The best architecture is the one that provides the required capabilities with the least unnecessary complexity.

Final Takeaway

The difference between API Gateway REST APIs and HTTP APIs isn’t simply about “old vs new” or “powerful vs simple.”

It’s about feature depth versus simplicity.

HTTP API
   |
   +--> Simpler
   +--> Lower cost
   +--> Modern authorization
   +--> Great for many serverless APIs

REST API
   |
   +--> More capabilities
   +--> Advanced API management
   +--> API keys / usage plans
   +--> More complex use cases

So before creating an API Gateway API, don’t ask:

“Which one should I use?”

Ask:

“What capabilities does my API actually require?”

That question usually makes the answer much clearer.

How Are You Choosing API Gateway?

Have you used HTTP APIs or REST APIs in production?

Did you choose based on features, cost, existing architecture, or something else?

I’d love to hear what worked — and what you would choose differently today.

If this article helped you, share it with someone who’s currently deciding between API Gateway REST and HTTP APIs.

When building an application, creating the backend logic is only part of the problem.

You also need to expose that logic to clients.

Maybe your frontend needs an API:

Browser
   |
   v
GET /users
   |
   v
Backend

Or a mobile application needs:

Mobile App
   |
   v
POST /orders
   |
   v
Backend

At small scale, exposing a few endpoints may seem straightforward.

But as the system grows, the API layer starts accumulating responsibilities:

• Authentication
• Authorization
• Rate limiting
• Request validation
• Routing
• TLS termination
• Monitoring
• Caching
• Traffic management
• Integration with backend services

You could build and operate all of this yourself.

Or you could use a managed service designed specifically for exposing APIs.

That’s where Amazon API Gateway comes in.

The basic idea is simple:

Client
   |
   v
API Gateway
   |
   v
Backend Service

But there’s much more happening between the client and backend.

In this article, we’ll understand what API Gateway actually does, how requests flow through it, where it fits in AWS architectures, and how to design an API Gateway-based system properly.

What Is Amazon API Gateway?

Amazon API Gateway is a managed AWS service for creating, publishing, securing, monitoring, and managing APIs.

Instead of exposing your backend service directly to the internet:

Internet
   |
   v
Backend

you can place API Gateway in front:

Internet
   |
   v
API Gateway
   |
   v
Backend

The API Gateway becomes the controlled entry point into your application.

This is particularly useful for serverless architectures, but API Gateway isn’t limited to Lambda.

It can integrate with different backend services depending on the API type and architecture.

Why Put an API Gateway in Front of Your Backend?

Imagine an application with several backend services:

                    Internet
                       |
                       v
                    Clients
                       |
                       v
                 API Gateway
                /     |      \
               v      v       v
           Users    Orders   Payments
             |        |        |
             v        v        v
          Service   Service   Service

Without an API gateway, clients may need to know about individual backend services.

That creates tighter coupling.

With API Gateway:

Client
   |
   v
api.example.com
   |
   v
API Gateway
   |
   +----> Users Service
   +----> Orders Service
   +----> Payments Service

The client interacts with a stable API surface.

Backend services can evolve independently behind it.

API Gateway Isn’t Your Application

This distinction is important.

API Gateway doesn’t replace your backend application.

It sits at the API boundary.

Think of it as:

              API Layer
                 |
                 v
        +------------------+
        |   API Gateway    |
        +------------------+
                 |
                 v
          Business Logic

The business logic might live in:

Lambda
ECS
EKS
HTTP service
Other AWS integrations

API Gateway manages the API-facing concerns.

Your backend remains responsible for business logic.

A Simple Request Flow

Suppose a client makes:

GET /orders/12345

The request might flow like this:

Client
  |
  | HTTPS
  v
API Gateway
  |
  +---- Authentication
  |
  +---- Authorization
  |
  +---- Validation
  |
  +---- Routing
  |
  v
Lambda / Service
  |
  v
Database

The response travels back:

Database
   |
   v
Backend
   |
   v
API Gateway
   |
   v
Client

The client doesn’t need to know how the backend is implemented.

API Gateway API Types

One of the first things you’ll encounter is that API Gateway supports different API types.

The major options are:

• HTTP APIs
• REST APIs
• WebSocket APIs

Choosing between them depends on what you’re building.

HTTP APIs

HTTP APIs are designed for simpler, lower-latency API workloads.

A typical architecture looks like:

Client
   |
   v
HTTP API
   |
   v
Lambda

They are a strong choice when you need a straightforward HTTP API without requiring the broader feature set of API Gateway REST APIs.

For many serverless applications, HTTP APIs are a natural starting point.

REST APIs

REST APIs provide a broader set of API management capabilities.

A typical architecture might be:

Client
   |
   v
REST API
   |
   +----> Lambda
   |
   +----> HTTP Backend
   |
   +----> AWS Integration

They are useful when you need capabilities associated with API Gateway’s more feature-rich REST API offering.

The important lesson is:

Don’t automatically choose REST APIs just because your API follows REST principles.

API design style and API Gateway’s REST API product are not exactly the same thing.

WebSocket APIs

HTTP request/response isn’t ideal for every application.

Imagine a chat application:

Client
   |
   | Persistent connection
   |
   v
WebSocket API
   |
   v
Backend

WebSocket APIs support two-way communication over persistent connections.

This can be useful for applications such as:

• Chat
• Live dashboards
• Real-time notifications
• Collaborative applications
• Real-time events

The architecture is different from a traditional request/response API.

API Gateway + Lambda

One of the most recognizable AWS serverless patterns is:

Client
   |
   v
API Gateway
   |
   v
Lambda
   |
   v
DynamoDB

For example:

POST /orders

could invoke a Lambda function.

The Lambda performs business logic:

Validate order
      |
      v
Calculate total
      |
      v
Store order
      |
      v
Return response

The client sees an HTTP API.

The application doesn’t require you to manage traditional application servers.

“Serverless” Doesn’t Mean No Servers

This is worth clarifying.

When people say:

“API Gateway lets me build APIs without managing servers.”

They mean:

You don’t have to provision and operate the underlying API infrastructure yourself.

There are still servers somewhere.

AWS manages the infrastructure.

Similarly, Lambda runs your code on compute infrastructure.

Serverless means you’re shifting infrastructure management responsibilities to the cloud provider.

API Gateway + ECS

API Gateway isn’t only for Lambda.

You could have:

Client
   |
   v
API Gateway
   |
   v
ECS Service
   |
   v
Application

Your backend might be a traditional containerized application.

API Gateway still provides an API-facing layer.

This can be useful when your organization wants centralized API management while running workloads on containers.

API Gateway + EKS

You can also build:

Client
   |
   v
API Gateway
   |
   v
EKS
   |
   +----> Service A
   +----> Service B
   +----> Service C

Now Kubernetes manages the workloads while API Gateway manages the external API boundary.

This is particularly interesting in architectures where:

Internet-facing API

and:

Internal service-to-service communication

have different requirements.

API Gateway as the Front Door

A useful mental model is:

API Gateway is the front door to your APIs.

For example:

                    Internet
                        |
                        v
                +---------------+
                | API Gateway   |
                +---------------+
                  /      |      \
                 v       v       v
              Users    Orders   Payments

The front door can handle common API concerns before requests reach your services.

That gives you a clean separation:

API concerns
     |
     v
API Gateway
     |
     v
Business concerns
     |
     v
Backend services

Authentication and Authorization

One of the first questions for any API is:

Who is allowed to call it?

API Gateway can integrate with authentication and authorization mechanisms.

For example, a client might authenticate with Amazon Cognito.

The flow could look like:

User
 |
 v
Cognito
 |
 v
Access Token
 |
 v
API Gateway
 |
 v
Backend

API Gateway can validate the authorization information before allowing the request to reach the backend, depending on the chosen authorization configuration.

This keeps authentication concerns closer to the API boundary.

API Gateway + Cognito

Imagine a web application:

Browser
   |
   v
Cognito
   |
   v
JWT
   |
   v
API Gateway
   |
   v
Lambda

The browser sends the token:

Authorization: Bearer <token>

API Gateway can use an appropriate authorizer to validate the token.

The backend then receives an authenticated request context.

This avoids implementing token validation independently in every backend service.

Authentication vs Authorization

These terms are easy to mix up.

Authentication

Who are you?

Authorization

What are you allowed to do?

For example:

User → authenticated

doesn’t automatically mean:

User → allowed to delete any order

Your API still needs authorization rules.

For example:

GET /orders/123

might be allowed for a customer viewing their own order.

While:

DELETE /orders/123

might require an administrative role.

API Gateway can participate in enforcing authorization, but business-level authorization often still belongs in the application.

Request Routing

API Gateway can route requests based on your API configuration.

For example:

GET /users
        |
        v
     Users API

GET /orders
        |
        v
    Orders API

POST /payments
        |
        v
   Payments API

This creates a clean external contract.

Clients don’t need to know:

Which Lambda?
Which ECS task?
Which internal service?
Which Kubernetes pod?

They interact with the API.

API Gateway and Microservices

Consider a microservices architecture:

                   API Gateway
                  /     |      \
                 v      v       v
              User    Order   Payment
             Service  Service  Service

The gateway can provide a consistent entry point.

Instead of:

Client → User Service
Client → Order Service
Client → Payment Service

you have:

Client
   |
   v
API Gateway
   |
   +----> User Service
   +----> Order Service
   +----> Payment Service

This can simplify client integration.

But be careful.

A gateway can become a central dependency if you put too much logic into it.

Don’t Turn the Gateway Into a Monolith

This is a common architectural mistake.

You start with:

API Gateway

Then gradually add:

Authentication
Authorization
Business rules
Data transformation
Complex orchestration
Database access
Logging
Retries

Eventually:

API Gateway
      |
      v
Everything

That’s not a healthy design.

A useful principle is:

Keep the gateway focused on API-level concerns.

Business logic belongs in the application layer.

Rate Limiting and Throttling

Suppose an API normally receives:

100 requests/sec

Suddenly:

50,000 requests/sec

reach the API.

You don’t want unlimited traffic flowing directly into your backend.

API Gateway provides throttling capabilities that can help control request rates.

Conceptually:

Clients
   |
   v
API Gateway
   |
   +---- Allowed traffic → Backend
   |
   +---- Excess traffic → Throttled

This can help protect backend services from unexpected request spikes.

Why Rate Limiting Matters

Imagine:

API Gateway
     |
     v
Lambda
     |
     v
Database

If the API receives an enormous traffic spike and every request reaches Lambda and the database, you could overload downstream systems.

A controlled API layer provides an opportunity to manage traffic before it reaches your backend.

But throttling isn’t a replacement for backend capacity planning.

API Keys Are Not Authentication

This is another common misconception.

API keys can be useful for:

• Identifying consumers
• Usage plans
• Controlling access to APIs in certain scenarios

But don’t treat an API key as a complete authentication mechanism for sensitive user-facing applications.

For example:

API Key

is not equivalent to:

User identity + strong authentication + authorization

For user authentication, use an appropriate identity and authorization mechanism.

Request Validation

Imagine your API expects:

{
  "productId": "P-100",
  "quantity": 2
}

But the client sends:

{
  "product": null
}

You don’t necessarily want invalid requests reaching your business logic.

Depending on the API type and configuration, API Gateway can perform certain request validation tasks.

Conceptually:

Request
   |
   v
API Gateway
   |
   +---- Invalid → 4xx response
   |
   +---- Valid → Backend

This can reduce unnecessary backend processing.

But validation should still exist where business correctness requires it.

Caching

Suppose an API repeatedly receives:

GET /products/123

and the underlying data doesn’t change frequently.

Caching can reduce repeated backend calls.

Conceptually:

Client
   |
   v
API Gateway
   |
   +---- Cache hit → Response
   |
   +---- Cache miss → Backend

Caching can reduce:

• Backend load
• Latency
• Database queries

But caching introduces complexity around:

• TTL
• Invalidation
• Stale data
• Cacheability
• Consistency

Don’t cache simply because you can.

Cache where the workload benefits from it.

API Gateway and CORS

If your frontend runs in a browser, you may encounter:

CORS — Cross-Origin Resource Sharing.

For example:

Frontend:
https://app.example.com

API:
https://api.example.com

The browser enforces cross-origin rules.

Your API needs appropriate CORS configuration so that legitimate browser requests are allowed.

A typical flow is:

Browser
   |
   | Cross-origin request
   v
API Gateway
   |
   v
Backend

CORS is primarily a browser security mechanism.

It isn’t a replacement for API authentication.

Custom Domains

You probably don’t want customers using a generated API Gateway URL such as:

https://abc123.execute-api.region.amazonaws.com

Instead, you might expose:

https://api.example.com

A custom domain provides a cleaner API contract.

Conceptually:

Client
   |
   v
api.example.com
   |
   v
API Gateway
   |
   v
Backend

This also gives you more control over how clients interact with your API over time.

API Versioning

APIs evolve.

Suppose you start with:

/api/v1/orders

Later, you introduce breaking changes.

You may need:

/api/v2/orders

Conceptually:

                  API Gateway
                 /           \
                v             v
             /v1            /v2
              |              |
              v              v
         Old backend     New backend

The exact versioning strategy depends on your API design.

Common approaches include:

/api/v1/
/api/v2/

or versioning through headers.

The important principle is:

Don’t make clients absorb backend changes unexpectedly.

An API is a contract.

Treat it that way.

API Gateway Stages

API Gateway supports the concept of stages.

For example:

API
 |
 +---- dev
 |
 +---- staging
 |
 +---- prod

This allows you to manage different deployment environments.

A typical workflow might be:

Developer
   |
   v
dev
   |
   v
staging
   |
   v
production

However, don’t assume that simply creating stages gives you complete environment isolation.

For production systems, you should think carefully about:

• Separate AWS accounts
• IAM boundaries
• Backend resources
• Configuration
• Deployment pipelines
• Secrets
• Data isolation

Observability

A production API isn’t complete until you can answer:

“What’s happening to my requests?”

You need visibility into things like:

Request count
Latency
4xx errors
5xx errors
Throttling
Backend failures
Integration latency

A useful mental model is:

Client
   |
   v
API Gateway
   |
   +---- Request metrics
   +---- Error metrics
   +---- Latency
   |
   v
Backend
   |
   +---- Application logs
   +---- Application metrics

API Gateway gives you API-level visibility.

Your backend needs its own observability.

Logs vs Metrics

Think about the difference.

Metrics

Tell you:

Something is happening.

For example:

5xx errors increased

Logs

Help answer:

Why did it happen?

For example:

Order service failed:
Database timeout
orderId=ORD-10001

A strong observability strategy uses both.

Distributed Tracing

Now imagine:

Client
  |
  v
API Gateway
  |
  v
Lambda
  |
  v
DynamoDB

A single request crosses multiple components.

When latency increases, you want to know where the time went.

For example:

Total: 800 ms

API Gateway: 20 ms
Lambda:      700 ms
Database:    600 ms

Tracing can help identify where the latency originates.

This becomes increasingly valuable as architectures become more distributed.

API Gateway + CloudWatch

A common monitoring architecture is:

API Gateway
     |
     +---- Metrics
     |
     +---- Logs
     |
     v
CloudWatch

Then you can build dashboards and alarms around important signals.

For example:

5xx error rate > threshold

could trigger an alert.

But avoid alerting on every individual failed request.

Focus on meaningful patterns.

Security at the API Boundary

An API exposed to the internet is an attack surface.

Think about:

Internet
   |
   v
API Gateway
   |
   v
Backend

Security should include multiple layers.

Authentication

Who is calling?

Authorization

What can they do?

Rate limiting

How much traffic can they send?

Input validation

Is the request valid?

Encryption

Is traffic protected in transit?

Logging

Can suspicious activity be investigated?

Least privilege

Can the API/backend access only what it needs?

API Gateway can contribute to this architecture, but it isn’t your entire security strategy.

API Gateway + WAF

For internet-facing applications, you may also place AWS WAF in front of the API where appropriate.

Conceptually:

Internet
   |
   v
WAF
   |
   v
API Gateway
   |
   v
Backend

WAF can help protect against certain classes of malicious web traffic.

This gives you layered protection:

Internet
   |
   v
WAF
   |
   v
API Gateway
   |
   +---- Authentication
   +---- Throttling
   +---- Validation
   |
   v
Backend

Each layer has a different responsibility.

API Gateway vs Load Balancer

This is a common source of confusion.

You might ask:

“If I already have an Application Load Balancer, why would I need API Gateway?”

They overlap in some areas, but they solve different problems.

A simplified mental model:

Load Balancer

Primarily concerned with:

Traffic distribution
        |
        v
Backend targets

API Gateway

More focused on:

API management
     |
     +---- Authentication
     +---- Authorization
     +---- Throttling
     +---- API lifecycle
     +---- API-specific controls

You might use:

Internet
   |
   v
API Gateway
   |
   v
ALB
   |
   v
ECS

when the architecture benefits from both layers.

Or:

Internet
   |
   v
ALB
   |
   v
ECS

may be perfectly sufficient for a simpler application.

The right choice depends on the requirements.

API Gateway vs CloudFront

Another common question.

CloudFront is primarily a content delivery and edge caching service.

API Gateway is primarily an API management and API execution front door.

They can also be used together.

For example:

Client
   |
   v
CloudFront
   |
   v
API Gateway
   |
   v
Backend

This can make sense when you want edge capabilities in front of APIs.

Again, the architecture should be driven by requirements rather than adding services simply because they’re available.

API Gateway Is Not Always Necessary

This is important.

Don’t put API Gateway into every AWS architecture automatically.

Suppose you have:

Internal service
     |
     v
Internal service

You may not need API Gateway.

Or perhaps your application is:

ALB
 |
 v
ECS

and the ALB already satisfies your requirements.

The question should be:

What problem does API Gateway solve for this architecture?

If you can’t answer that, you may not need it.

A Serverless Application Example

Let’s build a simple serverless API.

Imagine an e-commerce application:

                    Client
                       |
                       v
                  API Gateway
                       |
             +---------+---------+
             |                   |
             v                   v
          Lambda A            Lambda B
         Get Orders          Create Order
             |                   |
             v                   v
         DynamoDB             DynamoDB

Requests might be:

GET  /orders
POST /orders
GET  /orders/{id}

API Gateway handles the API boundary.

Lambda handles business logic.

DynamoDB stores the data.

The architecture is:

Client
  |
  v
API Gateway
  |
  v
Lambda
  |
  v
DynamoDB

No traditional web server needs to be provisioned and managed for the API layer.

A Microservices Example

Now consider a larger application:

                         Clients
                            |
                            v
                       API Gateway
                            |
             +--------------+--------------+
             |              |              |
             v              v              v
         User Service   Order Service   Payment Service
             |              |              |
             v              v              v
          Database       Database       Payment API

The API Gateway provides a common external boundary.

Each backend service remains independently deployable.

This can be useful for organizations moving toward microservices.

But again:

Don’t put business orchestration into the gateway just because it sits in front of everything.

A More Complete Production Architecture

A production architecture could look like:

                         Internet
                            |
                            v
                           WAF
                            |
                            v
                      API Gateway
                            |
              +-------------+-------------+
              |             |             |
              v             v             v
           Service A     Service B     Service C
              |             |             |
              v             v             v
           Database      Database      Queue
                                          |
                                          v
          Monitoring                    Worker
             |
             v
         CloudWatch                         

You might also introduce:

Cognito
   |
   v
API Gateway

for user authentication.

Or:

CloudFront
   |
   v
API Gateway

for edge delivery requirements.

The architecture depends on the application.

Common API Gateway Mistakes

Mistake 1: Treating API Gateway as the Application

API Gateway is the API layer.

Your business logic belongs in backend services.

Mistake 2: Putting Everything Behind One Giant Gateway

A gateway can become a bottleneck organizationally if every team must modify the same configuration.

Design ownership and deployment boundaries carefully.

Mistake 3: Confusing Authentication With Authorization

Validating a user’s identity doesn’t automatically mean they can perform every operation.

Mistake 4: Using API Keys as User Authentication

API keys have different purposes.

Use proper authentication and authorization for sensitive user APIs.

Mistake 5: Ignoring Rate Limits

A backend can be healthy under normal traffic and still collapse during a sudden spike.

Mistake 6: Ignoring CORS

Browser-based applications need appropriate cross-origin configuration.

Mistake 7: Logging Sensitive Information

Don’t blindly log:

Passwords
Tokens
Payment information
Personal data

Observability should not become a data-leak mechanism.

Mistake 8: No API Versioning Strategy

Breaking API changes can become painful for clients.

Treat your API as a contract.

Mistake 9: Monitoring Only API Gateway

An API may return successfully while the backend is becoming unhealthy.

Monitor the complete request path.

When Should You Use API Gateway?

API Gateway is particularly useful when you need:

• A managed API front door
• Serverless API integration
• Authentication and authorization integration
• API throttling
• API lifecycle management
• Request routing
• API-level monitoring
• Public API exposure
• WebSocket APIs
• Integration with backend services

It’s especially compelling when your architecture looks like:

Client
   |
   v
API Gateway
   |
   v
Lambda

or:

Client
   |
   v
API Gateway
   |
   v
Microservices

When Might You Not Need It?

You may not need API Gateway if:

• A load balancer already meets your requirements
• The service is internal and doesn’t require API management features
• You’re exposing a simple application where another entry point is sufficient
• The additional API management layer doesn’t provide meaningful value

The best architecture isn’t the one with the most AWS services.

It’s the one with the right boundaries.

A Practical Design Checklist

Before putting API Gateway into production, ask:

API design

• What resources and endpoints are exposed?
• What are the API contracts?
• How will the API evolve?

Authentication

• Who can call the API?
• How are users authenticated?
• How are tokens validated?

Authorization

• What can each caller do?
• Where are business authorization rules enforced?

Traffic management

• What happens during traffic spikes?
• What are the throttling requirements?
• Can downstream services handle the traffic?

Backend

• Is the backend Lambda, ECS, EKS, or another service?
• What happens when the backend fails?
• What are the timeout requirements?

Security

• Is HTTPS enforced?
• Is WAF appropriate?
• Are sensitive fields protected?
• Are IAM permissions least-privilege?

Observability

• Are API errors monitored?
• Is latency monitored?
• Can requests be correlated with backend logs?
• Are meaningful alarms configured?

Operations

• How are APIs deployed?
• How are versions managed?
• How are changes rolled back?

These questions matter more than simply creating an API Gateway API.

The Bigger Lesson

API Gateway isn’t valuable because it removes the need to write backend code.

Its real value is that it creates a managed boundary between clients and backend systems.

Instead of:

Client
   |
   v
Backend

you get:

Client
   |
   v
API Gateway
   |
   +---- Authentication
   +---- Authorization
   +---- Throttling
   +---- Validation
   +---- Routing
   +---- Monitoring
   |
   v
Backend

That boundary becomes particularly powerful as your system grows.

And because API Gateway can work with Lambda, containers, and other backend architectures, you don’t have to commit your entire application to one compute model.

Final Takeaway

Amazon API Gateway is best understood as the managed API front door for your AWS applications.

A simple serverless API might look like:

Client
   |
   v
API Gateway
   |
   v
Lambda
   |
   v
DynamoDB

A microservices architecture might look like:

                         Client
                            |
                            v
                       API Gateway
                            |
             +--------------+--------------+
             |              |              |
             v              v              v
           Users          Orders        Payments
          Service        Service         Service

And a more security-focused architecture might look like:

Internet
   |
   v
WAF
   |
   v
API Gateway
   |
   +---- Authentication
   +---- Authorization
   +---- Throttling
   |
   v
Backend Services

The biggest takeaway is:

API Gateway isn’t about avoiding servers. It’s about managing the API boundary without having to build and operate that infrastructure yourself.

Use it when you need a managed API layer.

Don’t use it simply because it’s an AWS service.

And keep the responsibilities clear:

API Gateway
    ↓
API concerns
    ↓
Backend
    ↓
Business logic
Database
    ↓
Data persistence

When those boundaries are well designed, API Gateway becomes a powerful building block for scalable, secure, and maintainable cloud architectures.

How Are You Using API Gateway?

Have you used API Gateway with Lambda, ECS, EKS, Cognito, or a microservices architecture?

What was the biggest challenge you faced — authentication, throttling, CORS, API versioning, observability, or choosing between API Gateway and a load balancer?

Share your experience in the comments.

If this article helped you understand where API Gateway fits in a modern AWS architecture, share it with another AWS or DevOps engineer building APIs in the cloud.

Building Reliable Event-Driven Systems with Amazon SQS: Retries, Backpressure, DLQs, and Idempotency

Distributed systems have a simple reality that every engineer eventually learns:

Something will fail.

A database will become unavailable.

An external API will time out.

A deployment will introduce a bug.

A worker will crash halfway through processing a message.

And sometimes, the message itself will be invalid.

The interesting question isn’t:

“How do I prevent every failure?”

That’s nearly impossible in a distributed system.

The better question is:

“What happens when something fails?”

This is where Amazon SQS becomes much more than a simple queue.

At first, SQS looks straightforward:

Producer
   |
   v
 SQS
   |
   v
Consumer

But a production-ready SQS architecture needs to answer much harder questions:

• What happens when consumers are slow?
• What happens when processing fails?
• How many times should a message be retried?
• What happens to permanently broken messages?
• How do we prevent duplicate processing?
• How do we handle traffic spikes?
• How do we protect downstream databases?
• How do we know the queue is becoming unhealthy?
• How do we safely recover failed messages?

In this article, we’ll build a practical mental model for designing reliable systems with SQS.

SQS Is About Decoupling

Let’s start with the fundamental problem.

Suppose an API directly calls a background processing service:

Client
  |
  v
API
  |
  v
Worker

This looks simple.

But now the API depends on the worker being available.

If the worker is slow:

API
 |
 v
Worker
 |
 | Slow
 v
Response delayed

If the worker is unavailable:

API
 |
 v
Worker
 |
 X
Failure

A queue introduces a buffer:

Client
  |
  v
API
  |
  v
SQS
  |
  v
Worker

Now the API and worker don’t have to operate at exactly the same speed.

That’s the first reliability benefit.

The Queue Becomes a Shock Absorber

Imagine an application normally receives:

1,000 jobs/minute

But suddenly there’s a traffic spike:

20,000 jobs/minute

Your workers may not be able to process everything immediately.

Without a queue:

20,000 requests
       |
       v
Workers
       |
       X
Overload

With SQS:

20,000 jobs
      |
      v
     SQS
      |
      v
Workers process gradually

The queue absorbs the temporary difference between:

incoming work

and

processing capacity.

This is one of the most important reasons queues exist.

But a Queue Doesn’t Create Infinite Capacity

This is a common misconception.

Suppose:

Incoming rate: 10,000 messages/minute
Processing rate: 5,000 messages/minute

The queue will grow.

Minute 1 → +5,000
Minute 2 → +10,000
Minute 3 → +15,000
...

Eventually, the backlog becomes a problem.

So SQS provides buffering, not unlimited capacity.

A reliable architecture needs to monitor whether consumers are keeping up.

The Four Pillars of Reliable SQS Processing

When designing production systems with SQS, four concepts deserve special attention:

          Reliable SQS
                 |
      +----------+----------+
      |          |          |
    Retry    Idempotency    DLQ
                 |
             Backpressure

More simply:

1. Retries

Give transient failures another chance.

2. Idempotency

Make duplicate processing safe.

3. Dead-Letter Queues

Isolate messages that repeatedly fail.

4. Backpressure

Prevent downstream systems from being overwhelmed.

These four concepts work together.

1. Designing Effective Retries

Imagine your Lambda consumes a message:

SQS
 |
 v
Lambda
 |
 v
Database

The database temporarily becomes unavailable.

The Lambda fails.

Should the message disappear?

No.

SQS allows the message to become available again according to the queue’s visibility timeout and processing behavior.

Conceptually:

Message
   |
   v
Lambda
   |
   X
Failure
   |
   v
Retry
   |
   v
Lambda
   |
   v
Success

This is extremely useful for transient failures.

Transient Failures vs Permanent Failures

Not every failure should be retried forever.

Consider:

Database timeout

Retrying may help.

But:

Missing required field

will probably fail again.

This distinction is crucial.

Transient failure

The system may recover.

Examples:

• Temporary network error
• API timeout
• Database connection failure
• Service throttling
• Short-lived dependency outage

Permanent failure

The message itself needs attention.

Examples:

• Invalid schema
• Missing required field
• Unsupported event type
• Invalid business state
• Corrupted payload

Retries are useful for the first category.

DLQs are important for the second.

Don’t Just Increase the Retry Count

A common response to failures is:

“Let’s increase retries.”

But more retries aren’t automatically more reliable.

Suppose a message is permanently invalid.

You configure:

10 retries

Now you’ve created:

Bad Message
   |
   +--> Retry 1
   +--> Retry 2
   +--> Retry 3
   +--> ...
   +--> Retry 10
   |
   v
DLQ

You’ve spent resources processing something that was never going to succeed.

Retry policies should reflect the failure characteristics of your workload.

2. Visibility Timeout

To understand SQS reliability, you need to understand visibility timeout.

When a consumer receives a message, SQS temporarily hides that message from other consumers.

For example:

Queue
----------------
Message A
Message B
Message C

Worker receives Message A:

Queue
----------------
Message A → Invisible temporarily
Message B
Message C

The worker processes it.

If successful:

Process
  |
  v
Delete

If processing fails or doesn’t complete:

Processing
    |
    X
Failure
    |
    v
Visibility timeout expires
    |
    v
Message available again

That’s how retries become possible.

Choose Visibility Timeout Carefully

Suppose your Lambda normally takes:

30 seconds

but your visibility timeout is:

10 seconds

You could get:

Worker A
   |
   | Processing
   |
   +---- 10 seconds
            |
            v
Message becomes visible
            |
            v
Worker B receives it

Now two workers may process the same message.

That’s exactly the kind of situation that can produce duplicate side effects.

Your visibility timeout should therefore be appropriate for your expected processing duration, with enough margin for normal execution behavior.

But Don’t Make It Excessively Long

The opposite problem is also possible.

Suppose processing usually takes:

20 seconds

but your visibility timeout is:

2 hours

If the worker crashes immediately, the message may remain invisible for a long time before becoming available again.

That delays recovery.

The goal isn’t:

“Make visibility timeout as large as possible.”

The goal is:

“Give normal processing enough time while keeping recovery reasonably responsive.”

3. Idempotency

This is perhaps the most important concept when building reliable SQS consumers.

Imagine:

Message:
OrderCreated

Your Lambda receives it.

It creates the order.

Then something goes wrong before the message is successfully deleted.

The message becomes available again.

Now:

Attempt 1 → Create order
Attempt 2 → Create order again

If your operation isn’t idempotent, you’ve created a duplicate.

Idempotency for Payments

The problem becomes even more serious with payments.

Imagine:

ProcessPayment
      |
      v
Payment Provider
      |
      v
Charge ₹2,499

The payment succeeds.

But your worker crashes before completing the message lifecycle.

The message is retried:

Retry
  |
  v
Charge ₹2,499 again

Now you’ve charged the customer twice.

This is why critical operations need an idempotency strategy.

Use a Unique Operation ID

Your message could contain:

{
  "eventId": "evt-12345",
  "orderId": "ORD-10001",
  "eventType": "PaymentRequested",
  "amount": 2499
}

The consumer can use:

eventId = evt-12345

to determine whether the operation has already been completed.

Conceptually:

Receive event
      |
      v
Already processed?
   /       \
 Yes        No
 |           |
Skip       Process
             |
             v
       Record success

The exact implementation depends on the business operation.

The important principle is:

Assume a message may be processed more than once.

Idempotency Isn’t Just an SQS Problem

Even if you move away from SQS tomorrow, distributed systems still have retries.

You may have:

API retries
Lambda retries
HTTP retries
Database retries
Client retries

Any time a request can be repeated, idempotency becomes relevant.

SQS simply makes the need more obvious.

4. Dead-Letter Queues

Now imagine a message fails repeatedly.

SQS
 |
 v
Lambda
 |
 X
Failure
 |
 v
Retry
 |
 X
Failure
 |
 v
Retry
 |
 X
Failure

You don’t want this forever.

A dead-letter queue provides a separate destination for messages that exceed an appropriate receive-count threshold.

Main Queue
    |
    v
Consumer
    |
    X
Repeated failures
    |
    v
DLQ

Now the normal processing path can move on.

DLQ Is a Quarantine Area

A useful mental model is:

The main queue is for normal work. The DLQ is for work that needs investigation.

For example:

Main Queue
----------------
Message A
Message B
Message C
Message D

DLQ
----------------
Message X
Message Y

The main queue continues processing:

A → Success
B → Success
C → Success
D → Success

while:

X
Y

can be investigated separately.

Poison Messages

A repeatedly failing message is often called a poison message.

For example:

A → Success
B → Success
C → Failure
C → Failure
C → Failure
C → Failure

Message C is consuming processing attempts without making progress.

The DLQ provides an escape route:

C
|
v
Retries
|
v
DLQ

This prevents one message from continuously interfering with normal processing.

A DLQ Doesn’t Fix the Problem

This is important.

If your DLQ suddenly contains:

10,000 messages

you haven’t solved the incident.

You’ve contained it.

Now you need to ask:

Why are these messages failing?

Possible reasons:

Application bug
Schema change
Database outage
External API failure
IAM permission issue
Invalid data
Configuration error

The DLQ buys you time to investigate without letting the failures continuously consume normal processing capacity.

5. Backpressure

Now let’s talk about another important reliability concept.

Suppose:

SQS
 |
 v
Lambda
 |
 v
RDS

There are 100,000 messages waiting.

Lambda scales aggressively.

You might get:

500 Lambda executions
       |
       v
RDS
       |
       X
Overloaded

Now the database starts failing.

Those failures cause Lambda retries.

More retries create more load.

You can end up with:

Queue backlog
      ↓
More Lambda concurrency
      ↓
Database overload
      ↓
More failures
      ↓
More retries
      ↓
More load

This is a failure amplification loop.

Reliability Doesn’t Mean “Maximum Concurrency”

A common mistake is thinking:

“If Lambda can scale, let it scale as much as possible.”

That’s dangerous.

Your downstream systems have limits.

For example:

Lambda
   |
   v
RDS

RDS might support 100 safe concurrent operations.

If Lambda creates 1,000 concurrent requests, you’ve exceeded what the database can comfortably handle.

You may need to control consumer concurrency.

The goal is:

Maximum sustainable throughput, not maximum theoretical throughput.

Queue Backlog Is a Form of Backpressure

Suppose:

Producer → SQS → Consumer

If the consumer slows down:

Consumer capacity ↓
       |
       v
Queue backlog ↑

That backlog is a signal.

It tells you:

“The system is receiving work faster than it can process it.”

That’s valuable information.

You can then decide whether to:

• Scale consumers
• Reduce incoming traffic
• Increase processing capacity
• Fix consumer errors
• Protect a downstream dependency
• Accept temporary backlog

Queue Depth Isn’t the Whole Story

Imagine:

Queue depth = 10,000

That sounds bad.

But if your consumers process:

20,000 messages/minute

the backlog might disappear quickly.

Now imagine:

Queue depth = 500

but the oldest message is:

2 hours old

That’s potentially much worse for a latency-sensitive workload.

So monitor both:

How much work is waiting?

and:

How long has work been waiting?

Message Age Is Often a Better Reliability Signal

For a user-facing workflow, you might define:

99% of messages should be processed within 1 minute.

Then:

Oldest message age > 5 minutes

becomes an important alert.

This is often more meaningful than simply saying:

Queue depth > 100

The right thresholds depend on the business requirements.

Batch Processing

SQS consumers often process messages in batches.

For example:

Batch
----------------
Message A
Message B
Message C
Message D

Batching can improve efficiency.

Instead of invoking a function for every individual message:

A → Lambda
B → Lambda
C → Lambda
D → Lambda

you can process:

[A, B, C, D] → Lambda

This can reduce invocation overhead.

But it creates an important failure question.

What If One Message in the Batch Fails?

Suppose:

A → Success
B → Success
C → Failure
D → Success

If your processing strategy treats the entire batch as failed, messages that already succeeded might be retried.

That can create unnecessary duplicate work.

For suitable Lambda + SQS workloads, partial batch response lets the function identify the records that actually failed.

Conceptually:

Batch
 |
 +-- A → Success
 +-- B → Success
 +-- C → Failure
 +-- D → Success
             |
             v
         Retry C

This can improve efficiency and reduce unnecessary retries.

Standard vs FIFO

SQS offers:

Standard queues

and:

FIFO queues

The choice depends on the workload.

Standard

Use when:

• High throughput matters
• Messages are independently processable
• Strict ordering isn’t required
• Consumers can handle duplicate delivery appropriately

FIFO

Use when:

• Ordering affects correctness
• Deduplication characteristics matter
• You need ordered processing within message groups

For example:

Image Processing

usually doesn’t care about order.

Standard is a natural fit.

But:

OrderCreated
OrderPaid
OrderShipped

may have ordering requirements.

FIFO may be appropriate.

Don’t Use FIFO Just Because It Sounds Safer

This is a common mistake.

Suppose you’re processing:

10,000 independent images

There is no business reason to force them into an ordered sequence.

Using FIFO can introduce constraints that don’t solve an actual problem.

Ask:

Does ordering affect business correctness?

If not, Standard is usually the simpler choice.

FIFO Message Groups

When ordering matters, ask another question:

Does everything need to be ordered, or only messages belonging to the same entity?

Suppose you have:

Customer A:
Transaction 1
Transaction 2
Transaction 3

Customer B:
Transaction 1
Transaction 2
Transaction 3

You may need ordering per customer, but not across all customers.

Message groups allow you to model that:

Group A:
A1 → A2 → A3

Group B:
B1 → B2 → B3

This allows ordered streams to coexist.

The design goal is to preserve the ordering that matters without unnecessarily serializing unrelated work.

SNS + SQS for Fan-Out

SQS becomes especially powerful when combined with SNS.

Suppose an order event needs to reach multiple systems:

OrderCreated
     |
     v
    SNS
   / | \
  v  v  v
 SQS SQS SQS
  |  |  |
  v  v  v
Payment
Inventory
Analytics

Each consumer gets its own queue.

This provides:

• Fan-out
• Independent buffering
• Independent scaling
• Failure isolation

If Analytics fails:

Analytics Queue
      |
      v
Backlog grows

Payment can still continue:

Payment Queue
      |
      v
Payment Worker
      |
      v
Processing

This is one of the strongest patterns for decoupled event-driven systems.

Each Consumer Should Usually Own Its Queue

Consider:

SNS
 |
 v
One SQS Queue
 |
 +----> Payment
 +----> Inventory
 +----> Analytics

If every consumer needs the same event, this doesn’t provide true fan-out.

Consumers compete for messages.

Instead:

SNS
 |
 +----> Payment Queue
 |
 +----> Inventory Queue
 |
 +----> Analytics Queue

Each consumer gets an independent stream of events.

This also means each consumer can have its own:

• Retry behavior
• DLQ
• Scaling policy
• Processing latency
• Failure handling

That’s a much cleaner architecture.

SQS Doesn’t Mean “Exactly Once”

This is another important mindset.

Don’t design your application around:

“This message will definitely be processed exactly once.”

Distributed systems are more complicated than that.

Your architecture should be prepared for:

Message delivered
Message processed
Processing interrupted
Message delivered again

The consumer should therefore be safe to retry.

This is why:

Idempotency + retries + DLQ

are so important.

Designing a Good Event

Reliable processing starts with reliable messages.

Instead of:

{
  "data": "something"
}

prefer a clear event structure:

{
  "eventId": "evt-12345",
  "eventType": "OrderCreated",
  "version": "1",
  "timestamp": "2026-08-30T10:30:00Z",
  "source": "order-service",
  "data": {
    "orderId": "ORD-10001",
    "customerId": "CUST-5001"
  }
}

Now consumers know:

• What happened
• Which event this is
• Which version it belongs to
• Where it came from
• When it happened
• What business data it contains

Event Versioning

Imagine your first event contains:

{
  "orderId": "ORD-10001",
  "amount": 2499
}

Later you change it to:

{
  "orderId": "ORD-10001",
  "totalAmount": 2499
}

An existing consumer expecting:

amount

may break.

Event-driven systems require careful schema evolution.

Treat event formats as contracts.

Prefer backward-compatible changes where practical, and use explicit versioning when breaking changes are unavoidable.

Don’t Put Huge Payloads Into SQS

SQS messages have a maximum size.

More importantly, large payloads make event processing harder.

Suppose you need to process a large report.

Instead of:

SQS
 |
 +----> Huge report

consider:

S3
 |
 +----> Large report

SQS
 |
 +----> Reference

For example:

{
  "eventType": "ReportReady",
  "reportId": "RPT-123",
  "location": "s3://bucket/report.json"
}

The queue carries the event.

Object storage carries the large payload.

Security Matters Too

A queue is part of your application security boundary.

Think about:

Who can send messages?
Who can receive messages?
Who can delete messages?
Who can purge the queue?
Who can change queue configuration?

For example:

Order Service
      |
      | SendMessage
      v
Order Queue
      |
      | ReceiveMessage
      v
Order Worker

The producer doesn’t need permission to consume messages.

The consumer doesn’t necessarily need permission to publish.

Use least privilege.

Encryption

SQS supports server-side encryption.

For sensitive workloads, you can use AWS KMS-based encryption and configure appropriate IAM permissions and key policies.

But encryption is only one layer.

A secure messaging architecture should also consider:

• IAM
• Data classification
• Logging
• Retention
• Access controls
• Sensitive information in payloads

Don’t put secrets or unnecessary personal information into messages just because the queue is encrypted.

Observability Is Part of Reliability

A queue can be functioning technically while the application is failing operationally.

For example:

SQS → Healthy
Lambda → Running

But:

Queue age → 30 minutes

Your customers may still be waiting for work to complete.

So monitor the entire pipeline.

Producer
   |
   v
SQS
   |
   v
Lambda
   |
   v
Database / API

Useful signals include:

Queue

• Number of visible messages
• Number of messages in flight
• Age of oldest message

Lambda

• Errors
• Duration
• Throttles
• Concurrency

DLQ

• Message count
• Message arrival rate
• Oldest message age

Downstream services

• Error rate
• Latency
• Saturation

Reliability requires visibility across the whole path.

Correlation IDs Make Debugging Easier

Imagine a message travels through:

API
 |
 v
SQS
 |
 v
Lambda
 |
 v
Database
 |
 v
External API

Without correlation identifiers, troubleshooting can be painful.

Instead, carry identifiers such as:

eventId
requestId
orderId

through logs and relevant records.

Then you can search:

eventId=evt-12345

and trace the event through the system.

This is especially useful when debugging DLQ messages.

A Production SQS Architecture

Let’s combine everything.

                         API
                          |
                          v
                     SQS Queue
                          |
                +---------+---------+
                |                   |
                v                   v
             Lambda              Lambda
                |                   |
                v                   v
             Service A           Service B
                |                   |
                v                   v
             Database            External API
                          |
                          |
                    Repeated Failure
                          |
                          v
                         DLQ
                          |
                          v
                  Alert / Investigation
                          |
                          v
                      Safe Replay

For an event-driven application with multiple consumers:

                        Producer
                            |
                            v
                           SNS
                      /      |      \
                     v       v       v
                   SQS     SQS     SQS
                    |       |       |
                    v       v       v
                 Lambda  Lambda  Lambda
                    |       |       |
                   DLQ     DLQ     DLQ

This provides independent processing and independent failure boundaries.

A Practical Failure Scenario

Let’s walk through a realistic incident.

Your order platform has:

Order Service
     |
     v
SNS
     |
     v
Payment SQS
     |
     v
Payment Lambda
     |
     v
Payment API

Suddenly the payment provider starts returning:

HTTP 503

The Lambda fails.

Messages are retried.

Attempt 1 → 503
Attempt 2 → 503
Attempt 3 → 503

Eventually, messages move to the DLQ.

Payment DLQ
-------------
evt-1001
evt-1002
evt-1003

Now operations sees:

Payment DLQ > threshold

They investigate.

The provider is unavailable.

Instead of losing orders, the messages are safely isolated.

When the provider recovers, the team can carefully replay the messages.

This is resilience in action.

But Be Careful With Replay

Suppose you have:

DLQ
 |
 +----> 50,000 messages

You fix the problem.

Then immediately replay all 50,000.

Lambda scales.

The payment provider receives a massive burst.

Now you’ve created a second incident.

A safer recovery process might be:

DLQ
 |
 v
Replay small batch
 |
 v
Observe
 |
 v
Replay more
 |
 v
Observe

Recovery should respect downstream capacity.

Reliability Is a System Property

It’s tempting to think:

“SQS makes my application reliable.”

SQS is only one component.

Consider:

Producer
   |
   v
SQS
   |
   v
Consumer
   |
   v
Database

Reliability depends on the entire chain.

You can have:

Perfect queue configuration

and still have:

Broken consumer

or:

Overloaded database

or:

Invalid event contract

SQS provides building blocks.

Your architecture determines how resilient the system actually is.

Common SQS Reliability Mistakes

Mistake 1: Treating SQS as a Database

SQS is for messaging and buffering.

It isn’t your system of record.

Mistake 2: Assuming Exactly-Once Processing

Design consumers to handle retries safely.

Mistake 3: Using the Wrong Visibility Timeout

Too short can cause duplicate processing.

Too long can delay retries.

Mistake 4: No DLQ

Repeatedly failing messages need an escape path.

Mistake 5: DLQ Without Monitoring

A DLQ that nobody watches is just a hidden failure bucket.

Mistake 6: Unlimited Consumer Scaling

More workers can overwhelm downstream dependencies.

Mistake 7: Ignoring Message Age

A queue can have a modest depth but extremely old messages.

Mistake 8: Replaying DLQ Messages Without a Plan

Recovery traffic can overload the system.

Mistake 9: Putting Huge Payloads in Messages

Use references to object storage when appropriate.

Mistake 10: Ignoring Event Versioning

Consumers and producers evolve independently.

Treat event schemas as contracts.

A Production Checklist

Before putting an SQS-based workflow into production, ask:

Queue design

• Is Standard or FIFO actually required?
• Does ordering affect correctness?
• Do messages need to be grouped?

Processing

• How long does normal processing take?
• Is the visibility timeout appropriate?
• Can the consumer safely retry?

Failure handling

• What happens when processing fails?
• How many retries make sense?
• Is there a DLQ?
• Is the DLQ monitored?

Idempotency

• Can the same message be processed twice?
• What prevents duplicate side effects?
• Do critical operations have idempotency keys?

Scaling

• How does the consumer scale?
• What is the safe concurrency level?
• Can downstream dependencies handle the load?

Observability

• Are queue depth and message age monitored?
• Are consumer errors monitored?
• Are DLQ messages monitored?
• Can individual events be traced?

Recovery

• Who owns the DLQ?
• How are failed messages investigated?
• How are messages replayed?
• Can replay overload downstream systems?

If you can’t answer these questions, the queue isn’t fully designed yet.

The Bigger Picture

SQS is easy to create.

A reliable SQS architecture is harder.

The queue itself is only the middle of the story:

Producer
   |
   v
SQS
   |
   +---- Retry
   |
   +---- Visibility Timeout
   |
   +---- DLQ
   |
   v
Consumer
   |
   +---- Idempotency
   |
   +---- Backpressure
   |
   v
Downstream Systems

Each part addresses a different failure mode.

SQS separates producers and consumers.

Visibility timeout controls message availability during processing.

Retries handle transient failures.

Idempotency protects against duplicate side effects.

DLQs isolate repeatedly failing messages.

Concurrency controls protect downstream dependencies.

Monitoring tells you when the system is falling behind.

Together, these patterns create a much stronger system than simply:

Producer → Queue → Consumer

Final Takeaway

Reliable systems aren’t systems where failures never happen.

They’re systems where failures are:

expected, isolated, observable, and recoverable.

Amazon SQS gives you an excellent foundation for that.

A production-ready architecture might look like:

                        Producer
                            |
                            v
                           SQS
                            |
                 +----------+----------+
                 |                     |
              Success                Failure
                 |                     |
                 v                     v
              Delete                 Retry
                                       |
                                       v
                                  Retry Limit
                                       |
                                       v
                                      DLQ
                                       |
                                       v
                                 Investigation
                                       |
                                       v
                                   Recovery

And for larger event-driven systems:

                        Producer
                            |
                            v
                           SNS
                      /      |      \
                     v       v       v
                   SQS     SQS     SQS
                    |       |       |
                  Lambda  Lambda  Lambda
                    |       |       |
                   DLQ     DLQ     DLQ

The most important lesson is this:

SQS doesn’t make a system reliable by itself. The reliability comes from how you design retries, idempotency, backpressure, failure isolation, observability, and recovery around the queue.

Once you start thinking about SQS this way, you’re no longer just designing a queue.

You’re designing a failure-tolerant processing system.

How Are You Building Reliability with SQS?

Have you used SQS in production with Lambda, ECS, EKS, or SNS fan-out?

What has caused the most trouble in your systems — duplicate messages, visibility timeouts, queue backlogs, DLQs, downstream overload, or replaying failed messages?

Share your experience in the comments.

If this article helped you think about SQS beyond just “put a message in a queue,” share it with another AWS or DevOps engineer building event-driven systems.

Distributed systems are designed with one uncomfortable assumption:

Things will fail.

A database can become unavailable.

An API can time out.

A Lambda function can throw an exception.

A message can contain invalid data.

A downstream service can return an unexpected response.

And when your application uses asynchronous messaging, there’s another problem:

What happens to a message that keeps failing?

Imagine this:

Producer
   |
   v
SQS Queue
   |
   v
Lambda
   |
   X
Processing Failed
   |
   v
Retry
   |
   X
Processing Failed
   |
   v
Retry
   |
   X
Processing Failed

What happens next?

You don’t want the same broken message consuming resources forever.

This is where a Dead-Letter Queue (DLQ) becomes extremely useful.

A DLQ gives repeatedly failing messages somewhere to go after they exceed an appropriate retry threshold.

Instead of allowing one problematic message to repeatedly interfere with normal processing, you isolate it for investigation and recovery.

In this article, we’ll understand how DLQs work, why they’re important, how they fit into SQS and Lambda architectures, and how to design them properly for production systems.

What Is a Dead-Letter Queue?

A Dead-Letter Queue is a queue used to hold messages that could not be successfully processed after a configured number of attempts.

The basic flow looks like this:

Main Queue
    |
    v
Consumer
    |
    X Failure
    |
    v
Retry
    |
    X Failure
    |
    v
Retry
    |
    X Failure
    |
    v
DLQ

The message isn’t simply discarded.

Instead, it is moved aside so that engineers or automated recovery processes can investigate it.

Think of a DLQ as:

A quarantine area for messages that need special attention.

Why Do We Need a DLQ?

Consider an order-processing system:

Order Service
      |
      v
   SQS Queue
      |
      v
Order Lambda

Most messages work:

Order 1 → Success
Order 2 → Success
Order 3 → Success
Order 4 → Success

But then:

Order 5 → Failure

Maybe the message is malformed.

The consumer retries it.

Order 5 → Retry → Failure

Again:

Order 5 → Retry → Failure

And again.

Without a DLQ, the system can waste processing capacity repeatedly attempting the same problematic message.

With a DLQ:

Order 5
   |
   v
Retry
   |
   v
Retry limit reached
   |
   v
DLQ

The main queue can continue processing other messages.

A DLQ Doesn’t Mean the Message Is “Dead”

The name can be misleading.

A dead-letter message isn’t necessarily permanently unusable.

It means:

The normal processing path couldn’t successfully handle it within the configured retry policy.

The message may still be recoverable.

For example:

DLQ
 |
 v
Investigate
 |
 +----> Fix application bug
 |
 +----> Fix message data
 |
 +----> Restore dependency
 |
 v
Replay message

The DLQ creates an opportunity to recover rather than silently losing the message.

The Basic SQS + DLQ Architecture

With Amazon SQS, a common architecture looks like:

Producer
    |
    v
+-------------+
| Main Queue  |
+-------------+
      |
      v
   Consumer
      |
      +---- Success → Delete
      |
      +---- Failure → Retry
                         |
                         v
                       DLQ

The key configuration is the redrive policy.

It determines when a message should be moved from the source queue to the dead-letter queue based on its receive count.

For example:

maxReceiveCount = 5

Conceptually:

Attempt 1 → Failed
Attempt 2 → Failed
Attempt 3 → Failed
Attempt 4 → Failed
Attempt 5 → Failed
                    |
                    v
                   DLQ

The exact behavior and configuration should be designed according to your workload and AWS service integration.

The Most Important Question: Why Did the Message Fail?

Not all failures are the same.

Consider two scenarios.

Scenario 1: Temporary failure

Lambda
   |
   v
Payment API
   |
   X
Timeout

The payment provider might recover a few seconds later.

A retry makes sense.

Scenario 2: Permanent failure

Lambda
   |
   v
Message
   |
   X
Invalid JSON / Missing required field

Retrying the exact same message probably won’t fix anything.

This distinction is critical.

A good retry strategy handles transient failures while preventing permanent failures from endlessly consuming resources.

Transient vs Permanent Failures

Let’s classify some examples.

Transient failures

These may succeed later:

Temporary network error
API timeout
Database connection issue
Temporary service outage
Throttling

Retries can be useful.

Permanent failures

These usually require a correction:

Invalid message format
Missing required field
Unsupported event type
Invalid business state
Corrupted data

Retries alone won’t solve these.

That’s why DLQs are useful.

They separate:

"Try again"

from:

"Something needs investigation."

A Real Example: Payment Processing

Imagine an e-commerce system:

Order Service
     |
     v
Payment Queue
     |
     v
Payment Lambda
     |
     v
Payment Provider

The queue contains:

{
  "eventId": "evt-1001",
  "orderId": "ORD-12345",
  "amount": 2499
}

The Payment Lambda calls the payment provider.

Suppose the provider is temporarily unavailable.

Payment Lambda
      |
      v
Payment API
      |
      X
Timeout

The message can be retried.

If the provider recovers:

Retry
  |
  v
Payment API
  |
  v
Success

Great.

But imagine the message itself is invalid:

{
  "orderId": "ORD-12345"
}

and the payment amount is missing.

Retrying this message five times won’t magically create the missing amount.

Eventually:

Invalid Message
      |
      v
Retries
      |
      v
DLQ

Now engineers can investigate the event.

Poison Messages

A message that repeatedly fails processing is often called a poison message.

For example:

Message A → Success
Message B → Success
Message C → Failure
Message C → Failure
Message C → Failure
Message C → Failure

Message C is effectively poisoning the normal processing path.

A DLQ isolates it:

Main Queue
----------------
A
B
D
E
F

C
|
v
DLQ

The healthy workload can continue.

What Happens Without a DLQ?

Suppose a consumer keeps receiving the same bad message.

You could end up with:

Bad Message
    |
    v
Attempt 1
    |
    v
Attempt 2
    |
    v
Attempt 3
    |
    v
Attempt 4
    |
    v
Attempt 5
    |
   ...

This creates several problems:

• Wasted compute
• Increased logs
• Increased downstream traffic
• Higher Lambda invocation count
• Longer processing delays
• More difficult troubleshooting

A DLQ gives the system a controlled escape path.

DLQs and Lambda

A very common architecture is:

SQS
 |
 v
Lambda
 |
 X
Failure
 |
 v
SQS DLQ

Lambda consumes messages from the SQS queue.

When processing fails according to the configured behavior, the message can become available again.

After enough receive attempts, SQS can move it to the DLQ according to the queue’s redrive policy.

Conceptually:

              Main Queue
                  |
                  v
               Lambda
                  |
             +----+----+
             |         |
          Success    Failure
             |         |
             v         v
          Delete     Retry
                         |
                         v
                      DLQ

This is one of the most common serverless failure-handling patterns on AWS.

Don’t Confuse Retry With Recovery

Retries answer:

“Can this work if I try again?”

A DLQ answers:

“What should happen when normal retries aren’t enough?”

These are different mechanisms.

A production workflow often looks like:

Message
   |
   v
Process
   |
   +---- Success
   |
   +---- Temporary failure
              |
              v
            Retry
              |
              +---- Success
              |
              +---- Failure
                       |
                       v
                      DLQ

This creates a layered failure strategy.

Choosing maxReceiveCount

One of the most important DLQ configuration decisions is:

How many times should a message be retried before going to the DLQ?

There isn’t one universal answer.

It depends on the workload.

Suppose an external API occasionally fails for a few seconds.

You may want several retries.

But if messages are usually failing because of invalid data, excessive retries provide little value.

Think about:

Failure frequency
Failure type
Recovery time
Processing cost
Business importance
Downstream rate limits

Don’t blindly choose:

maxReceiveCount = 100

just because more retries sound safer.

More Retries Aren’t Always Better

Imagine:

10,000 messages

and:

1,000 messages

are permanently invalid.

If each invalid message is retried 20 times:

1,000 × 20 = 20,000 unnecessary processing attempts

You have created additional load without increasing the chance of success.

A better retry strategy distinguishes transient problems from permanent ones.

Visibility Timeout and DLQ

DLQ behavior is closely connected to the SQS visibility timeout.

Remember:

Receive message
      |
      v
Message becomes temporarily invisible
      |
      v
Consumer processes it

If processing fails and the message isn’t successfully deleted, it can become visible again after the visibility timeout.

Then:

Attempt 1
Attempt 2
Attempt 3
...

The receive count increases.

Eventually, the message can be moved to the DLQ according to the redrive configuration.

So these settings need to work together:

Visibility Timeout
        +
Retry behavior
        +
maxReceiveCount
        +
DLQ

Don’t configure them independently without understanding the entire message lifecycle.

Visibility Timeout Should Match Processing

Suppose your Lambda normally takes:

30 seconds

to process a message.

But your visibility timeout is:

10 seconds

You could end up with:

Lambda A
   |
   | Processing
   |
   +---- 10 sec
          |
          v
Message visible again
          |
          v
Lambda B receives it

Now two consumers may process the same message.

This can result in duplicate side effects.

The visibility timeout should therefore be designed around the actual processing characteristics of the workload.

Idempotency Is Still Required

This is perhaps the most important point about DLQs:

A DLQ does not eliminate duplicate processing.

Imagine:

Message
   |
   v
Lambda
   |
   v
Payment succeeds
   |
   X
Delete doesn't complete

The message can be processed again.

You could get:

Payment attempt 1 → Success
Payment attempt 2 → Success

That’s why your consumer needs appropriate idempotency controls.

For example:

{
  "eventId": "evt-12345"
}

The application can use the event ID to determine whether the operation has already been successfully processed.

DLQ and Idempotency Work Together

Think of them as solving different problems.

Idempotency

Protects against:

Same message processed multiple times

DLQ

Protects against:

Message repeatedly failing normal processing

Together:

                 Message
                    |
                    v
                Consumer
                 /     \
                /       \
          Duplicate?    Failure?
              |             |
             Yes            Yes
              |             |
             v             Retry
           Ignore           |
                             v
                            DLQ

Both are important in reliable asynchronous systems.

What Should You Put in a DLQ?

Ideally, the DLQ contains the original message plus enough context to investigate the failure.

For example:

{
  "eventId": "evt-12345",
  "eventType": "OrderCreated",
  "orderId": "ORD-10001",
  "timestamp": "2026-08-30T10:30:00Z"
}

You may also want your application logs to contain:

eventId
requestId
orderId
consumer
error type
error message
timestamp

This makes correlation much easier.

Don’t Put Sensitive Data in Logs

While debugging DLQ messages, it’s tempting to log everything.

Be careful.

If a message contains:

Customer data
Payment information
Authentication tokens
Personal information

don’t blindly copy the entire message into logs.

Use structured identifiers and carefully selected diagnostic fields.

For example:

eventId=evt-12345
orderId=ORD-10001
error=InvalidPaymentAmount

is often much safer than dumping the entire payload.

Monitoring the DLQ

Creating a DLQ isn’t enough.

You need to know when messages enter it.

Imagine your DLQ contains:

0 messages
0 messages
0 messages
1 message
250 messages
1,500 messages

Something has changed.

Potential causes:

Application deployment
Schema change
Downstream outage
Permission change
Invalid producer behavior
Dependency failure

A DLQ should therefore be treated as an operational signal.

What Should You Alert On?

A simple alert might trigger when:

ApproximateNumberOfMessagesVisible > 0

But the exact threshold depends on your workload.

For a critical payment workflow:

1 failed message

might deserve attention.

For a high-volume analytics pipeline:

A small number of failed messages

might not require immediate intervention.

The right alert threshold depends on business impact.

Don’t Alert on Every Retry

This is another important operational principle.

Suppose a transient database problem causes:

Message → Retry → Success

You probably don’t want an engineer paged every time a retry occurs.

Instead, alert on meaningful conditions such as:

DLQ receives messages
Queue backlog grows unexpectedly
Oldest message age crosses threshold
Consumer errors increase significantly

This helps avoid alert fatigue.

DLQ as an Observability Signal

A DLQ can tell you something about the health of your application.

For example:

Application deployment
        |
        v
DLQ messages increase
        |
        v
Something in the new release
may be incompatible

Or:

Schema change
     |
     v
Consumer rejects messages
     |
     v
DLQ grows rapidly

The DLQ isn’t just a recovery mechanism.

It’s also a valuable failure signal.

A Production Monitoring View

For an SQS + Lambda architecture, think about:

                    SQS
                     |
          +----------+----------+
          |                     |
      Main Queue               DLQ
          |                     |
          v                     v
       Lambda               Investigation
          |
          v
     Downstream API

Monitor:

Main queue

• Queue depth
• Oldest message age
• Message throughput

Lambda

• Errors
• Duration
• Throttles
• Concurrency

DLQ

• Message count
• Arrival rate
• Oldest message age

Downstream dependency

• Error rate
• Latency
• Availability

This gives you visibility across the entire processing path.

DLQ Does Not Fix the Root Cause

This is worth emphasizing.

Suppose:

DLQ = 10,000 messages

You shouldn’t consider the problem solved because the messages were successfully moved there.

The DLQ is a containment mechanism, not a root-cause fix.

You still need to determine:

Why did these messages fail?

Possible root causes:

Bug
Bad data
Schema incompatibility
Permission issue
Database outage
External API failure
Configuration error

The DLQ gives you breathing room to investigate.

The Recovery Workflow

A good DLQ process might look like:

              DLQ
               |
               v
          Detect Message
               |
               v
          Investigate
               |
        +------+------+
        |             |
     Permanent      Temporary
      problem        problem
        |             |
        v             v
    Correct data   Restore service
        |             |
        +------+------+
               |
               v
             Replay
               |
               v
           Main Queue
               |
               v
           Process

The exact replay process depends on the application.

But the important idea is:

DLQ handling should be an operational workflow, not an afterthought.

Replay Is Powerful — And Dangerous

Suppose you’ve fixed the bug.

You now have:

DLQ
 |
 +-- 10,000 messages

You want to replay them.

Before doing that, ask:

• Is the consumer fixed?
• Are the messages still valid?
• Will replay create duplicate side effects?
• Can the downstream service handle the traffic?
• Should messages be replayed all at once?
• Should they be replayed gradually?

Don’t blindly dump thousands of messages back into production.

Replay Gradually When Appropriate

Imagine:

DLQ
 |
 v
10,000 messages

and you immediately replay everything.

You might create:

10,000 messages
      |
      v
Main Queue
      |
      v
Lambda scales
      |
      v
Database
      |
      X
Overloaded

Now you’ve created another incident.

A safer approach may be:

DLQ
 |
 v
Replay small batch
 |
 v
Observe
 |
 v
Replay more
 |
 v
Observe

Recovery should respect downstream capacity.

The SNS + SQS + Lambda + DLQ Pattern

Let’s combine everything we’ve discussed.

                      Event Producer
                              |
                              v
                         SNS Topic
                              |
                              v
                         SQS Queue
                              |
                              v
                           Lambda
                         /       \
                        /         \
                   Success       Failure
                      |             |
                      v             v
                   Delete         Retry
                                    |
                                    v
                              Retry Limit
                                    |
                                    v
                                   DLQ

For multiple consumers:

                         SNS
                          |
             +------------+------------+
             |                         |
             v                         v
        Payment SQS              Analytics SQS
             |                         |
             v                         v
        Payment Lambda           Analytics Lambda
             |                         |
             v                         v
        Payment DLQ              Analytics DLQ

This architecture gives each consumer an independent failure boundary.

Why Separate DLQs Matter

Suppose you have:

SNS
 |
 +----> Payment Queue → Payment Lambda → Payment DLQ
 |
 +----> Analytics Queue → Analytics Lambda → Analytics DLQ

Now imagine Analytics is broken.

Its DLQ may fill up.

Payment can continue independently.

That’s a major benefit of giving each consumer its own queue and DLQ.

You avoid creating one giant failure domain.

DLQs and Event-Driven Architecture

As event-driven systems grow, DLQs become increasingly important.

Consider:

               Event Bus / SNS
                       |
       +---------------+---------------+
       |               |               |
       v               v               v
    Service A       Service B       Service C
       |               |               |
      SQS             SQS             SQS
       |               |               |
    Lambda          Lambda          Lambda
       |               |               |
      DLQ             DLQ             DLQ

Each service owns its processing lifecycle.

This means:

• Service A can retry independently
• Service B can fail independently
• Service C can recover independently

That’s a powerful property in distributed systems.

Common DLQ Mistakes

Mistake 1: Creating a DLQ and Never Monitoring It

A DLQ without monitoring is just a hidden failure bucket.

Fix: Create meaningful alerts and dashboards.

Mistake 2: Setting the Retry Count Arbitrarily

More retries don’t necessarily mean more reliability.

Fix: Choose retry thresholds based on failure behavior.

Mistake 3: Ignoring Idempotency

A message can still be processed multiple times before reaching the DLQ.

Fix: Design consumers for safe retries.

Mistake 4: Replaying Everything at Once

This can overload downstream systems.

Fix: Replay carefully and monitor the system.

Mistake 5: Treating Every Failure as Permanent

Transient outages may recover after a retry.

Fix: Distinguish transient and permanent failures.

Mistake 6: Treating the DLQ as the Solution

Moving a message to a DLQ doesn’t fix why it failed.

Fix: Investigate the root cause.

Mistake 7: Keeping Messages in the DLQ Forever

Depending on retention and operational requirements, DLQ messages can eventually expire.

Fix: Define a recovery and retention strategy.

Choosing the Right DLQ Strategy

Before implementing a DLQ, ask:

What is a failed message?

Define what constitutes a processing failure.

How many retries make sense?

Base this on the workload.

How long can a message remain in the system?

Consider queue retention and business requirements.

How will failures be detected?

Use metrics and alerts.

Who owns the DLQ?

Someone should be responsible for investigating it.

How will messages be recovered?

Define a replay or remediation process.

Can replay create duplicate side effects?

Make sure consumers are idempotent.

Can the downstream system handle replay traffic?

Avoid turning recovery into another outage.

These questions turn a basic DLQ configuration into a production-ready failure-handling strategy.

A Practical Failure-Handling Architecture

A mature asynchronous system might look like:

                      Producer
                          |
                          v
                     SQS Queue
                          |
                          v
                     Lambda
                          |
              +-----------+-----------+
              |                       |
           Success                  Failure
              |                       |
              v                       v
           Delete                    Retry
                                      |
                                      v
                              Retry Threshold
                                      |
                                      v
                                     DLQ
                                      |
                                      v
                                Alert / Review
                                      |
                                      v
                                  Remediation
                                      |
                                      v
                                    Replay

Notice something important.

The DLQ is part of a larger workflow.

It’s not just:

Queue → DLQ

It’s:

Failure → Containment → Detection → Investigation → Recovery

That’s the real operational model.

The Bigger Lesson

Distributed systems aren’t reliable because they never fail.

They’re reliable because they handle failure predictably.

A queue can absorb workload.

Retries can handle transient problems.

A DLQ can isolate messages that repeatedly fail.

Idempotency can protect against duplicate processing.

Monitoring can tell you when something is going wrong.

Together:

                    Message
                       |
                       v
                    Queue
                       |
                       v
                   Consumer
                    /     \
                   /       \
              Success      Failure
                 |            |
                 v            v
              Delete        Retry
                              |
                              v
                         Retry Limit
                              |
                              v
                             DLQ
                              |
                              v
                         Investigation
                              |
                              v
                           Recovery

That’s what resilient asynchronous architecture looks like.

Final Takeaway

A Dead-Letter Queue is more than a place where failed messages go.

It’s a mechanism for containing failure without allowing one problematic message to continuously disrupt the normal processing path.

The pattern is straightforward:

Main Queue
    |
    v
Consumer
    |
    +---- Success → Complete
    |
    +---- Failure → Retry
                     |
                     v
                Retry Limit
                     |
                     v
                    DLQ

But production reliability comes from everything around it:

• Choose retry limits carefully
• Configure visibility timeout appropriately
• Design consumers for idempotency
• Monitor queue backlog and message age
• Alert on meaningful DLQ activity
• Investigate the root cause
• Build a safe replay strategy
• Protect downstream systems during recovery

The most important mindset is:

A DLQ isn’t where failed messages go to disappear. It’s where failures go to become visible, diagnosable, and recoverable.

If you design it that way, a DLQ becomes an important part of your reliability strategy rather than simply another AWS resource in your architecture.

How Do You Handle Failed Messages?

Have you implemented DLQs with SQS + Lambda, SNS → SQS fan-out, or another event-driven architecture?

What has caused the most DLQ messages in your systems — bad payloads, downstream outages, deployment bugs, schema changes, or configuration issues?

Share your experience in the comments.

If this article helped you understand how DLQs fit into reliable AWS architectures, share it with another AWS or DevOps engineer building asynchronous systems.

When you start using Amazon SQS, one of the first decisions you’ll make is surprisingly simple:

Should I use a Standard queue or a FIFO queue?

At first glance, FIFO sounds like the obvious choice.

FIFO means First-In, First-Out.

So naturally, you might think:

“If FIFO preserves order, why wouldn’t I always use it?”

Because ordering comes with a trade-off.

In many distributed systems, strict ordering isn’t actually required. What matters more is throughput, scalability, availability, and the ability to process large numbers of messages efficiently.

That’s why AWS provides two different queue types:

Standard SQS
     |
     +--> High throughput
     +--> At-least-once delivery
     +--> Best-effort ordering

FIFO SQS
     |
     +--> Ordering
     +--> Deduplication
     +--> More controlled processing semantics

Choosing between them isn’t about which one is “better.”

It’s about understanding what your application actually needs.

Let’s break it down.

First: What Is SQS?

Before comparing Standard and FIFO, let’s quickly establish what SQS does.

Amazon Simple Queue Service is a managed message queue.

The basic architecture is:

Producer
    |
    | Send message
    v
+-----------+
| SQS Queue |
+-----------+
    |
    | Receive message
    v
Consumer

The producer and consumer don’t have to operate at the same time.

The producer can place work into the queue.

The consumer can process it later.

This provides:

• Decoupling
• Buffering
• Asynchronous processing
• Retry handling
• Failure isolation
• Independent scaling

For example:

Order Service
     |
     v
   SQS
     |
     v
Order Worker

The Order Service doesn’t need to wait for the worker to finish.

So What’s Different Between Standard and FIFO?

The simplest distinction is:

Standard queues prioritize high-throughput distributed processing. FIFO queues prioritize ordering and deduplication characteristics.

Think about the two models.

Standard

Messages arrive:

A → B → C → D
Processing might happen:
B → A → D → C

You shouldn’t design your application around strict ordering.

FIFO

Messages arrive:

A → B → C → D
Required processing order:
A → B → C → D

When ordering is a business requirement, FIFO becomes valuable.

But there’s much more to the decision than that.

Standard Queues

Standard queues are the default choice for many workloads.

They are designed for:

• Very high throughput
• Distributed processing
• Horizontal scaling
• Workloads where strict ordering isn’t required

For example:

                    SQS Standard
                         |
              +----------+----------+
              |          |          |
              v          v          v
           Worker A   Worker B   Worker C

Messages can be processed concurrently.

This makes Standard queues a natural fit for workloads such as:

• Image processing
• Video processing
• Email jobs
• Background tasks
• Log processing
• Data transformation
• Large-scale asynchronous workloads

The Important Property of Standard Queues

Standard SQS provides at-least-once delivery.

That means your application should be prepared for a message to be delivered more than once.

For example:

Message A
   |
   v
Worker
   |
   v
Processing succeeds
   |
   X
Something goes wrong before deletion
   |
   v
Message A becomes available again
   |
   v
Worker receives Message A again

So your consumer needs to be designed appropriately.

This leads to one of the most important concepts in distributed systems:

Idempotency.

What Is Idempotency?

Suppose you have:

OrderCreated

and your worker creates an order in a database.

If the same message is processed twice:

Attempt 1 → Create order
Attempt 2 → Create order again

you could create duplicate data.

For something like payment processing, the consequences could be much worse:

Attempt 1 → Charge customer
Attempt 2 → Charge customer again

A better consumer uses a unique event or operation ID.

For example:

{
  "eventId": "evt-12345",
  "orderId": "ORD-10001",
  "eventType": "OrderCreated"
}

The consumer can track whether:

evt-12345

has already been processed.

If it sees the same event again, it can safely ignore or reconcile the duplicate.

This is good practice regardless of queue type.

Standard Queues and Ordering

Here’s something important:

Standard queues don’t provide strict ordering guarantees.

Suppose you send:

A
B
C
D

You shouldn’t build business logic that assumes:

A → B → C → D

will always be the processing order.

Why?

Because distributed systems prioritize scalability and availability, and multiple consumers can process messages concurrently.

For many workloads, this doesn’t matter.

For example:

Resize Image A
Resize Image B
Resize Image C

There is no meaningful reason Image A must finish before Image B.

Standard queues are perfect for that kind of workload.

FIFO Queues

FIFO queues are designed for workloads where message ordering and deduplication characteristics matter.

FIFO means:

First-In, First-Out.

Conceptually:

Producer
   |
   v
+-------------+
| FIFO Queue  |
+-------------+
      |
      v
A → B → C → D

If your application depends on processing events in order, FIFO is worth considering.

A Real Example: Account Operations

Imagine a banking application.

You receive:

1. CreateAccount
2. DepositMoney
3. WithdrawMoney
4. CloseAccount

The order matters.

You don’t want:

WithdrawMoney
      |
      v
CreateAccount

or:

CloseAccount
      |
      v
DepositMoney

The business workflow has an explicit sequence.

This is a good example of where FIFO semantics can be useful.

Another Example: Inventory Updates

Imagine an inventory system receives:

Product A → Stock = 100
Product A → Stock = 90
Product A → Stock = 80

If the updates represent sequential state changes, processing them out of order could produce an incorrect result.

For example:

80
90
100

could incorrectly leave the system believing the inventory is 100.

If strict ordering is a business requirement, FIFO can help model that workflow.

Message Groups: The Powerful Part of FIFO

FIFO queues introduce an important concept:

Message groups.

Imagine you have orders:

Order A
Order B
Order C

You might want:

Order A messages → processed in order
Order B messages → processed in order
Order C messages → processed in order

But you don’t necessarily want Order A to block Order B.

Message groups allow you to create independent ordered streams.

Conceptually:

FIFO Queue

Group A:
A1 → A2 → A3
Group B:
B1 → B2 → B3
Group C:
C1 → C2 → C3

Within each group, ordering is maintained.

Different groups can provide opportunities for parallel processing.

This is one of the most important ideas when designing FIFO workloads.

Why Message Groups Matter

Imagine a payment platform processing transactions for thousands of customers.

You might require:

Customer A:
Transaction 1 → Transaction 2 → Transaction 3

but you don’t necessarily need:

Customer A Transaction 1

to block:

Customer B Transaction 1

You could conceptually use:

MessageGroupId = customer-A
MessageGroupId = customer-B

Now the ordering requirement is scoped to the entity that actually needs ordering.

That’s much more scalable than forcing the entire queue into one global sequence.

Don’t Accidentally Create One Giant Message Group

Here’s a common design mistake.

Suppose every message uses:

MessageGroupId = "orders"

Now you’ve effectively created one ordered stream.

Conceptually:

orders
 |
 +-- Message 1
 +-- Message 2
 +-- Message 3
 +-- Message 4
 +-- Message 5

You may have just limited your ability to process independent work in parallel.

If the business requirement is:

“Order messages must be ordered per customer.”

then grouping by customer may make more sense.

For example:

Customer A → Group A
Customer B → Group B
Customer C → Group C

The key principle is:

Scope ordering to the entity that actually requires it.

FIFO and Deduplication

Another important FIFO capability is deduplication.

Imagine the producer accidentally sends:

OrderCreated
OrderCreated

with the same deduplication identity.

FIFO queues provide deduplication mechanisms that can help prevent duplicate messages from being accepted within the relevant deduplication window.

But don’t misunderstand this.

It doesn’t mean:

“My application no longer needs idempotency.”

You should still design consumers defensively.

Messaging guarantees don’t replace application-level correctness.

Standard vs FIFO: The Core Trade-Off

Now we can frame the decision more clearly.

Standard
   |
   +--> Optimize for throughput and scalability
   |
   +--> Ordering isn't critical
   |
   +--> Consumer handles duplicates

FIFO
   |
   +--> Ordering matters
   |
   +--> Deduplication characteristics matter
   |
   +--> Work can be partitioned into message groups

This is the real architectural decision.

Example: Image Processing

Suppose users upload:

Image A
Image B
Image C
Image D

Your workers resize them.

Does Image A need to finish before Image B?

No.

You want:

SQS Standard
      |
      +----> Worker 1 → Image A
      +----> Worker 2 → Image B
      +----> Worker 3 → Image C
      +----> Worker 4 → Image D

Throughput matters more than ordering.

Standard is the natural choice.

Example: Financial Transactions

Now consider:

Account 123
   |
   +----> Deposit ₹1,000
   +----> Withdraw ₹300
   +----> Withdraw ₹200

The sequence may matter to the business.

You may need:

Deposit
   ↓
Withdraw 300
   ↓
Withdraw 200

rather than arbitrary ordering.

FIFO becomes a strong candidate.

Example: Email Notifications

Suppose you’re sending:

Welcome Email
Password Reset Email
Marketing Email

Do these messages need strict global ordering?

Usually not.

A Standard queue may be completely sufficient:

SQS Standard
      |
      +----> Worker
      +----> Worker
      +----> Worker

There’s little value in forcing an ordering guarantee that the business doesn’t need.

Example: Video Processing

Imagine:

10,000 videos

need transcoding.

The goal is:

Process as many videos as safely possible.

You don’t care whether:

Video 1

finishes before:

Video 2

You care about throughput.

Standard queues are a natural fit.

Example: Order State Transitions

Now imagine:

OrderCreated
OrderPaid
OrderShipped
OrderDelivered

The sequence represents a business state machine.

If consumers process:

OrderShipped

before:

OrderPaid

you could end up with an invalid state transition.

This is a situation where ordering may genuinely matter.

FIFO should be considered.

But there’s an important architectural question:

Do you need global ordering, or ordering only for each individual order?

Usually, it’s the latter.

That means message groups can be important.

Order A → Group A
Order B → Group B
Order C → Group C

Each order can maintain its own sequence.

Don’t Choose FIFO Just Because Ordering Sounds Good

This is probably the biggest recommendation in this article.

Ask:

Does my business logic actually depend on ordering?

If the answer is:

“Not really.”

Use Standard.

For example:

GenerateThumbnail
GenerateThumbnail
GenerateThumbnail

There’s usually no meaningful order.

Using FIFO would add constraints without solving a real problem.

Don’t Choose Standard When Order Is a Business Requirement

The opposite mistake is more dangerous.

Suppose:

PaymentAuthorized
PaymentCaptured
PaymentRefunded

must be processed in order.

If you use a queue without the required ordering semantics and your application assumes the order anyway, you could create inconsistent state.

The rule is:

If ordering affects correctness, model ordering explicitly.

Don’t depend on timing.

Throughput vs Ordering

One of the easiest ways to think about the trade-off is:

Standard
   |
   v
More flexibility for high-throughput parallel processing

versus:

FIFO
   |
   v
More controlled ordered processing

This doesn’t mean FIFO is “slow” or Standard is “unreliable.”

It means they optimize for different messaging requirements.

FIFO Doesn’t Mean “Process Everything Globally in Order”

This is a subtle but important point.

You might think FIFO means:

A1
A2
A3
B1
B2
B3
C1
C2
C3

must all be processed sequentially.

That’s not the right mental model.

Message groups allow independent ordered streams.

For example:

Group A:
A1 → A2 → A3

Group B:
B1 → B2 → B3

Group C:
C1 → C2 → C3

This lets you preserve ordering where necessary while still designing for parallelism across independent groups.

Lambda + Standard SQS

A common architecture is:

                  SQS Standard
                         |
             +-----------+-----------+
             |           |           |
             v           v           v
          Lambda      Lambda      Lambda
             |           |           |
             v           v           v
          Worker       Worker       Worker

This is excellent for workloads where:

• Messages are independent
• High throughput matters
• Ordering isn’t required
• Workers can scale horizontally

Examples:

Image processing
Email delivery
Log processing
Data transformation
Background jobs

Lambda + FIFO SQS

The architecture can also look like:

                     SQS FIFO
                         |
              +----------+----------+
              |                     |
              v                     v
          Lambda A              Lambda B
        Group A work           Group B work

Now the application can preserve ordering within message groups while processing independent groups separately.

This is useful for workloads such as:

Account operations
Order state transitions
Inventory changes
Financial workflows

when the ordering requirement is genuinely part of the business logic.

Retry Behavior Still Matters

Regardless of queue type, messages can fail processing.

For example:

Queue
  |
  v
Consumer
  |
  X
Failure
  |
  v
Retry

Eventually, a repeatedly failing message may need to go to a dead-letter queue.

Queue
  |
  v
Consumer
  |
  X
Failure
  |
  v
Retry
  |
  v
Retry
  |
  v
DLQ

Your queue design should include failure handling from the beginning.

Visibility Timeout Still Matters

Standard or FIFO doesn’t remove the need to understand visibility timeout.

When a consumer receives a message:

Queue
  |
  v
Consumer receives message
  |
  v
Message temporarily invisible

If the consumer doesn’t successfully complete processing within the visibility timeout, the message can become available again.

If the timeout is too short:

Worker A → Processing
      |
      +----> Visibility expires
                   |
                   v
              Worker B

You can end up with duplicate processing.

Again:

Design consumers for safe retries.

Dead-Letter Queues Still Matter

A poison message shouldn’t block your processing pipeline indefinitely.

For example:

Main Queue
    |
    v
Lambda
    |
    X
Invalid message
    |
    v
Retry
    |
    v
Retry
    |
    v
DLQ

Monitor your DLQ.

A sudden increase can indicate:

• Application bugs
• Invalid messages
• Schema changes
• Downstream failures
• Permission problems

A queue architecture isn’t complete until failure handling is considered.

Monitoring Standard and FIFO Queues

Don’t monitor only:

Is the queue running?

Monitor whether the workload is actually moving.

Useful signals include:

Queue depth

How many messages are waiting?

Message age

How old is the oldest message?

Consumer errors

Are workers failing?

DLQ messages

Are messages repeatedly failing?

Processing latency

How long does work take?

For FIFO workloads, also pay attention to whether message-group design is creating unnecessary processing bottlenecks.

Queue Backlog Is an Architectural Signal

Suppose your queue contains:

100 messages

and processing capacity is:

10,000 messages/minute

No major problem.

But suppose:

Queue depth = 10,000
Oldest message = 45 minutes

Now you have a latency problem.

The queue is telling you:

Consumers aren’t keeping up with the workload.

Possible causes include:

Traffic increased
Consumer failures
Lambda throttling
Database bottleneck
External API slowdown
Concurrency limits

The queue itself may not be the root cause.

A Subtle FIFO Problem: Ordering Can Reduce Parallelism

Imagine:

10,000 messages

all belong to:

MessageGroupId = "orders"

You’ve created one ordered stream.

Even if your infrastructure has many workers available, your ordering requirement may prevent meaningful parallel processing within that group.

Compare that with:

Customer A → Group A
Customer B → Group B
Customer C → Group C
...

Now independent groups can make better use of parallel processing.

So when using FIFO:

Design message groups around natural business partitions.

How to Decide: A Practical Flow

When designing your queue, ask these questions.

Question 1: Does message order affect business correctness?

If no:

Start with Standard.

If yes:

Consider FIFO.

Question 2: Do messages represent independent work?

For example:

Resize Image A
Resize Image B
Resize Image C

If yes, Standard is usually a natural fit.

Question 3: Do messages represent a sequence?

For example:

Create Account
Deposit
Withdraw
Close Account

If yes, FIFO may be appropriate.

Question 4: Can you partition the ordering requirement?

Ask:

Does everything need to be ordered, or only messages belonging to the same entity?

If it’s per:

Customer
Order
Account
Device

use message groups to model that boundary where appropriate.

Question 5: Can the consumer handle duplicate processing?

It should.

Even when the messaging system provides deduplication capabilities, application-level idempotency remains valuable.

A Simple Decision Tree

Here’s a practical mental model:

             Do you need ordering?
                       |
              +--------+--------+
              |                 |
             No                Yes
              |                 |
              v                 v
          Standard            FIFO
                                |
                                v
                    Can ordering be partitioned?
                                |
                         +------+------+
                         |             |
                        Yes            No
                         |             |
                         v             v
                  Use message       Single ordered
                     groups            stream

It’s not a substitute for reading the exact AWS service behavior for your workload, but it’s a useful starting point.

Standard vs FIFO: Quick Comparison

| Capability          | Standard                                 | FIFO                                           |
| ------------------- | ---------------------------------------- | ---------------------------------------------- |
| Primary goal        | High-throughput distributed processing   | Ordered and deduplicated messaging             |
| Ordering            | Best effort                              | Ordered within message groups                  |
| Duplicate handling  | Consumers should handle duplicates       | Built-in deduplication capabilities            |
| Parallel processing | Excellent for independent messages       | Excellent across independent message groups    |
| Best for            | General asynchronous workloads           | Order-sensitive workflows                      |
| Typical examples    | Image processing, email, background jobs | Financial workflows, ordered state transitions |

The important word here is “best.”

Neither queue is universally better.

A Production Example

Let’s say you’re building an e-commerce platform.

You have two completely different workloads.

Workload 1: Generate product thumbnails

Product Image
     |
     v
SQS
     |
     +----> Worker
     +----> Worker
     +----> Worker

Order doesn’t matter.

Use:

Standard SQS

Workload 2: Process order state transitions

OrderCreated
      |
      v
OrderPaid
      |
      v
OrderShipped
      |
      v
OrderDelivered

The sequence matters.

Use:

FIFO SQS, with message groups scoped appropriately, such as per order.

Now your architecture uses both queue types for different problems.

That’s perfectly normal.

You Don’t Have to Choose One Queue Type for Your Entire System

This is an important architectural lesson.

A large application might have:

                Application
                     |
          +----------+----------+
          |                     |
          v                     v
    Standard SQS             FIFO SQS
          |                     |
          v                     v
   Background Jobs       Order Processing

There is no rule saying:

“Our application uses FIFO.”

or:

“Our platform uses Standard.”

Different workloads can have different messaging requirements.

Choose the queue type per workload.

Common Mistakes

Mistake 1: Using FIFO Everywhere

Ordering isn’t free from an architectural perspective.

If the workload doesn’t need ordering, don’t introduce unnecessary constraints.

Mistake 2: Using Standard and Assuming Order

If ordering affects correctness, don’t rely on timing.

Mistake 3: Creating One FIFO Message Group

If all messages use the same group, you may unnecessarily serialize processing.

Mistake 4: Assuming FIFO Eliminates Idempotency

It doesn’t.

Your business logic still needs to handle retries and unexpected duplicate processing safely.

Mistake 5: Ignoring Backpressure

Even a perfectly designed queue can accumulate an unhealthy backlog.

Mistake 6: Ignoring Downstream Capacity

More consumers can increase throughput — but they can also overload:

Database
API
Payment Provider
External Service

Always consider the full processing chain.

The Bigger Lesson

The Standard vs FIFO decision is really about what your application considers important.

If your workload says:

“Process as much independent work as possible.”

Standard is usually the better starting point.

If your workload says:

“The order of these operations is part of correctness.”

FIFO becomes much more interesting.

And if the requirement is:

“Preserve order for each customer, but process different customers independently.”

That’s where message groups become especially powerful.

The goal isn’t to maximize features.

The goal is to model the business requirement correctly.

Final Takeaway

Standard and FIFO queues solve different problems.

Think of Standard as:

High throughput
       +
Independent work
       +
Horizontal processing

And FIFO as:

Ordering
     +
Deduplication capabilities
     +
Partitioned ordered processing

A useful rule of thumb is:

If order doesn’t matter, prefer Standard.

If order affects correctness, consider FIFO.

And when using FIFO:

Don’t automatically create one global ordered stream. Partition ordering around the entity that actually needs it.

For example:

Standard:

SQS
 |
 +----> Worker A
 +----> Worker B
 +----> Worker C

FIFO:
SQS FIFO
 |
 +---- Group A → A1 → A2 → A3
 |
 +---- Group B → B1 → B2 → B3
 |
 +---- Group C → C1 → C2 → C3

The most important question isn’t:

“Which SQS queue is better?”

It’s:

“What does my workload require for correctness, and what can I safely process in parallel?”

Once you answer that, the Standard vs FIFO decision becomes much easier.

What Are You Using?

Have you used Standard SQS, FIFO SQS, or both in production?

What influenced your decision — ordering, throughput, deduplication, message groups, or downstream processing constraints?

Share your experience in the comments.

If this article helped you understand the Standard vs FIFO decision, share it with another AWS or DevOps engineer designing an asynchronous system.

Imagine you have an application that receives thousands of requests every minute.

Some requests are quick.

Others require expensive background processing.

For example:

User uploads a video
        |
        v
Application
        |
        +----> Save metadata
        |
        +----> Process video
        |
        +----> Generate thumbnails
        |
        +----> Scan content
        |
        +----> Send notification

If the application tries to do everything synchronously, the user ends up waiting for all of it.

Worse, if one downstream component is slow or unavailable, the entire request can be affected.

A better approach is to separate accepting work from processing work.

That’s where Amazon Simple Queue Service (SQS) comes in.

Instead of calling a worker directly:

Application → Worker

you can introduce a queue:

Application → SQS → Worker

The application places a message into the queue.

The worker processes it when it has capacity.

That simple change can dramatically improve the resilience and scalability of a distributed system.

Let’s understand how SQS works, why queues are useful, and how to design them properly for production.

What Is Amazon SQS?

Amazon Simple Queue Service (SQS) is a fully managed message queuing service from AWS.

At its simplest:

Producer
    |
    v
+-----------+
| SQS Queue |
+-----------+
    |
    v
Consumer

The producer sends a message.

SQS stores the message.

A consumer retrieves and processes it.

The producer and consumer don’t need to communicate directly.

That’s the core idea behind decoupling.

Why Do We Need a Message Queue?

Consider a typical application:

Client
  |
  v
API
  |
  v
Application
  |
  +----> Database
  |
  +----> Payment API
  |
  +----> Email Service
  |
  +----> Analytics

This creates dependencies.

If the Email Service becomes slow, your application may become slow.

If the Payment API is unavailable, requests may fail.

If Analytics suddenly receives a huge amount of traffic, it can affect your application.

A queue introduces a buffer:

Application
     |
     v
   SQS
     |
     v
Worker

Now the application can say:

“The work has been accepted.”

It doesn’t necessarily need to wait for the worker to finish.

The Basic SQS Architecture

Let’s start with three concepts.

Producer

Creates messages.

Queue

Stores messages.

Consumer

Processes messages.

       Producer
           |
           | SendMessage
           v
    +---------------+
    |   SQS Queue   |
    +---------------+
           |
           | ReceiveMessage
           v
        Consumer

For example:

Order Service
     |
     v
Order Queue
     |
     v
Order Worker

The Order Service doesn’t need to know where the worker runs.

It only needs to know where the queue is.

A Real Example: Order Processing

Imagine an e-commerce application.

A customer places an order.

The API receives:

{
  "orderId": "ORD-10001",
  "customerId": "CUST-5001",
  "amount": 2499
}

Instead of performing all processing synchronously:

Request
   |
   v
Order API
   |
   +----> Validate
   |
   +----> Charge payment
   |
   +----> Update inventory
   |
   +----> Send email
   |
   +----> Generate invoice
   |
   v
Response

you could separate the work:

Request
   |
   v
Order API
   |
   v
SQS
   |
   v
Order Worker

The API can return quickly after successfully accepting the work.

The worker handles the processing asynchronously.

SQS Is a Buffer Between Systems

This is one of the most important concepts to understand.

Imagine:

Producer:
10,000 messages/minute

while:

Consumer:
5,000 messages/minute

Without a queue:

Producer
   |
   v
Consumer
   |
   X
Overloaded

With SQS:

Producer
   |
   v
+-----------+
|    SQS    |
+-----------+
     |
     v
Consumer

The queue absorbs the temporary difference in processing rates.

Messages can wait until consumers catch up.

Think of SQS as a Shock Absorber

A useful mental model is:

SQS absorbs workload spikes between producers and consumers.

Suppose traffic normally looks like:

1,000 jobs/minute

Then suddenly:

20,000 jobs/minute

The consumer may not be able to process everything immediately.

Instead of overwhelming the application:

Traffic Spike
     |
     v
   SQS
     |
     v
Workers process gradually

The queue acts as a buffer.

But remember:

A queue is not an infinite capacity solution.

If messages arrive faster than they are processed for a long time, the backlog will continue growing.

Producers and Consumers Can Scale Independently

Without a queue:

Producer
   |
   v
Consumer

The two systems are closely coupled.

With SQS:

Producer
   |
   v
SQS
   |
   v
Consumer

you can scale them independently.

For example:

1 API server
      |
      v
    SQS
      |
      +----> Worker 1
      +----> Worker 2
      +----> Worker 3
      +----> Worker 4

If the workload increases, you can add more consumers.

This is one of the major reasons queues are common in distributed systems.

Multiple Consumers

SQS supports multiple consumers reading from the same queue.

For example:

                    SQS
                     |
          +----------+----------+
          |          |          |
          v          v          v
       Worker A   Worker B   Worker C

The workers compete for available messages.

A message retrieved successfully by one consumer isn’t intended to be independently processed by every consumer.

This is different from SNS fan-out.

SQS vs SNS

This distinction is extremely important.

SQS

Producer
   |
   v
 Queue
   |
   +----> Worker A
   +----> Worker B
   +----> Worker C

Workers compete for work.

SNS

Publisher
    |
    v
   SNS
  / | \
 v  v  v
A   B   C

The same event can be distributed to multiple subscribers.

A simple rule:

SQS distributes work among consumers. SNS distributes messages to subscribers.

And they can be combined:

                    SNS
                 /      \
                v        v
             SQS A     SQS B
               |         |
               v         v
           Workers     Workers

That’s a common AWS event-driven architecture.

Standard vs FIFO Queues

Amazon SQS provides two primary queue types:

• Standard queues
• FIFO queues

The choice depends on your requirements.

Standard SQS Queues

Standard queues are designed for very high throughput and distributed processing.

They’re a good fit when:

• Very high throughput matters
• Occasional duplicate delivery can be handled
• Strict ordering isn’t required
• Work can be processed independently

Examples:

Image processing
Email jobs
Log processing
Background tasks
Batch processing

For many workloads, Standard queues are the natural starting point.

FIFO SQS Queues

FIFO stands for:

First-In, First-Out.

FIFO queues are designed for workloads where ordering and deduplication characteristics are important.

For example:

UpdateAccount
    |
    v
ChangeAddress
    |
    v
CloseAccount

If the order of operations matters, you need to explicitly design for that requirement.

FIFO queues provide ordering within message groups and deduplication capabilities.

The important point is:

Don’t choose FIFO simply because “FIFO sounds safer.”

Use it when your business workflow actually requires ordering or deduplication characteristics.

A Common Mistake: Assuming Order

Suppose you send:

Message 1
Message 2
Message 3

A distributed system shouldn’t casually assume they will always be processed in that order.

If ordering matters:

OrderCreated
OrderPaid
OrderShipped

then model that requirement explicitly.

Otherwise, design consumers so they don’t depend on ordering.

This is especially important when multiple workers process messages concurrently.

What Happens When a Consumer Receives a Message?

Let’s look at the lifecycle.

Suppose SQS contains:

Message A
Message B
Message C

A consumer receives Message A.

SQS temporarily makes it invisible to other consumers.

Conceptually:

Queue
----------------
A
B
C

Consumer receives A
Queue
----------------
A → invisible temporarily
B
C

This is controlled by the visibility timeout.

The consumer processes the message.

If processing succeeds:

Message
   |
   v
Process
   |
   v
Delete from queue

If processing fails:

Message
   |
   v
Process
   |
   X
Failure
   |
   v
Visibility timeout expires
   |
   v
Message available again

This creates a retry mechanism.

Visibility Timeout

Visibility timeout is one of the most important SQS concepts.

When a consumer receives a message, SQS doesn’t immediately delete it.

Instead, the message becomes temporarily invisible.

For example:

Visibility timeout = 60 seconds

The consumer has that period to complete processing and delete the message.

If it doesn’t:

60 seconds
    |
    v
Message becomes visible again

Another consumer can then process it.

Why Visibility Timeout Matters

Suppose your worker usually takes:

10 seconds

to process a message.

But your visibility timeout is:

5 seconds

That’s a problem.

The message could become visible again while the original worker is still processing it.

You might end up with:

Worker A → Processing
                 |
                 +---- visibility expires
                 |
                 v
Worker B → Receives same message

Now two workers may process the same message.

So the visibility timeout should be designed around the expected processing time, with appropriate headroom.

But Don’t Make It Extremely Long

A very long visibility timeout creates another problem.

Suppose processing fails immediately, but the visibility timeout is:

2 hours

The message may not become available for retry for a long time.

So there is a trade-off.

You want enough time for normal processing without making retries unnecessarily slow.

For long-running workloads, consumers can also extend the visibility timeout when appropriate.

SQS and Lambda

One of the most common serverless patterns is:

SQS
 |
 v
Lambda

AWS Lambda can poll SQS and invoke your function with batches of messages.

Conceptually:

Producer
   |
   v
SQS
   |
   v
Lambda
   |
   v
Business Logic

This removes the need to run dedicated worker servers.

Lambda automatically processes messages as they become available, subject to concurrency and event source configuration.

SQS + Lambda at Scale

Imagine your queue suddenly contains:

50,000 messages

Lambda can increase processing concurrency based on the event source behavior and available limits.

Conceptually:

             SQS
              |
       +------+------+------+
       |      |      |      |
       v      v      v      v
     Lambda Lambda Lambda Lambda

This gives you elastic processing.

But there’s an important warning:

Scaling consumers faster can overload downstream dependencies.

The Database Bottleneck

Imagine:

SQS
 |
 v
Lambda
 |
 v
RDS

The queue has a huge backlog.

Lambda scales up.

Suddenly:

500 Lambda executions
       |
       v
RDS
       |
       X
Overloaded

Now the queue might actually make the situation worse.

This is why you need to consider the entire processing chain.

SQS
 |
 v
Lambda
 |
 v
Database

Not just Lambda.

Control Consumer Concurrency

For workloads that interact with constrained downstream systems, you may need to limit Lambda concurrency.

For example:

SQS
 |
 v
Lambda
 |
 | Max concurrency = 50
 v
Database

This creates a controlled processing rate.

You may process the backlog more slowly, but you protect the dependency.

The goal isn’t:

Maximum possible concurrency.

The goal is:

Maximum sustainable throughput.

SQS Retries

Retries are a major benefit of queues.

Suppose:

Message A

is processed.

The worker fails.

The message becomes visible again.

Then:

Attempt 1 → Failed
Attempt 2 → Failed
Attempt 3 → Success

This is useful for transient problems.

For example:

Temporary API timeout
Temporary database connection failure
Temporary network issue

But retries aren’t always the answer.

If the message is permanently invalid:

Invalid schema
Missing required field
Corrupt payload

retrying it indefinitely won’t help.

That’s where a dead-letter queue becomes useful.

Dead-Letter Queues

A dead-letter queue (DLQ) is used to isolate messages that repeatedly fail processing.

For example:

Main Queue
    |
    v
Consumer
    |
    X
Failure
    |
    v
Retry
    |
    X
Failure
    |
    v
Retry
    |
    X
Failure
    |
    v
DLQ

You can configure a maximum receive count.

Once a message exceeds that threshold, it can be moved to the DLQ.

Why DLQs Matter

Without a DLQ:

Bad Message
     |
     v
Retry
     |
     v
Retry
     |
     v
Retry
     |
     v
Retry
     |
    ...

One poison message can repeatedly consume processing capacity.

With a DLQ:

Bad Message
     |
     v
Retry
     |
     v
Retry limit
     |
     v
DLQ

Now the problematic message is isolated.

Operations teams can investigate it without blocking normal processing.

Don’t Ignore Your DLQ

A DLQ isn’t a place where messages should quietly disappear.

A growing DLQ may indicate:

• Application bugs
• Invalid data
• Dependency failures
• Permission issues
• Schema incompatibility
• Unexpected business conditions

You should monitor DLQ activity.

For critical workloads, a non-zero DLQ count may itself be worth alerting on.

Idempotency: One of the Most Important Concepts

Here’s a scenario that every distributed-system engineer should understand.

Your worker receives:

OrderCreated

It processes the event.

Then something fails before the message is successfully deleted.

The message may be processed again.

Now:

Attempt 1 → Create order
Attempt 2 → Create order again

If your operation isn’t idempotent, you could create duplicate data.

For payments, it could be even worse:

Attempt 1 → Charge customer
Attempt 2 → Charge customer again

That’s why consumers should be designed with idempotency in mind.

Use an Event ID

A common approach is to include a unique identifier:

{
  "eventId": "evt-12345",
  "eventType": "OrderCreated",
  "orderId": "ORD-10001"
}

The consumer can record:

evt-12345 → processed

If it receives the same event again:

evt-12345

it can detect that it has already processed it.

Conceptually:

Receive event
      |
      v
Already processed?
   /       \
 Yes        No
 |           |
Skip       Process
             |
             v
      Mark as processed

The exact implementation depends on your application and consistency requirements.

Batch Processing with Lambda

When Lambda consumes SQS, messages can be delivered in batches.

For example:

Batch
----------------
Message A
Message B
Message C
Message D

The Lambda function processes them together.

Batching can improve efficiency by reducing invocation overhead.

But it introduces another question:

What happens if only one message fails?

For suitable workloads, Lambda supports partial batch failure reporting.

For example:

A → Success
B → Success
C → Failure
D → Success

The consumer can identify:

Retry C

instead of unnecessarily retrying the entire batch.

This is particularly useful when processing large volumes of independent messages.

Queue Backlog Is a Signal

One of the most important things to monitor in SQS is backlog.

Imagine:

Queue depth:
100
200
500
1,000
5,000
20,000

That’s telling you something.

Maybe:

• Consumers are too slow
• Consumers are failing
• Traffic increased
• A downstream dependency is unavailable
• Lambda concurrency is constrained

A queue backlog is often a symptom, not the root cause.

Message Age Can Be More Important Than Queue Depth

Suppose:

Queue depth = 10,000

That sounds alarming.

But if consumers process:

15,000 messages/minute

the backlog might be shrinking quickly.

Now consider:

Queue depth = 1,000

but the oldest message is:

3 hours old

That’s probably more concerning for a time-sensitive workload.

This is why message age can be a more meaningful operational signal than queue depth alone.

The right metric depends on the business requirement.

SQS and Backpressure

Queues are closely related to backpressure.

Imagine:

Producer
   |
   v
SQS
   |
   v
Consumer

If the consumer slows down:

Consumer capacity ↓
       |
       v
Queue backlog ↑

The queue absorbs the difference.

This allows the producer and consumer to operate at different speeds.

But eventually:

Backlog keeps growing
       |
       v
Latency increases
       |
       v
Messages become stale

So you need to define acceptable processing latency.

For some workloads:

Seconds

is acceptable.

For others:

Minutes

or:

Hours

may be fine.

SQS Doesn’t Guarantee Business Success

This distinction is important.

Suppose the message was successfully removed from SQS.

That means:

The consumer successfully completed the queue-processing step according to your implementation.

It does not automatically mean the entire business workflow succeeded.

For example:

SQS
 |
 v
Lambda
 |
 v
Payment API
 |
 X
Payment failed

Your application needs to decide what “success” means and how failures are handled.

Messaging infrastructure doesn’t replace business-level error handling.

SQS and SNS Together

SQS becomes even more powerful when combined with SNS.

Imagine:

                 Order Service
                      |
                      v
                 SNS Topic
                /        \
               v          v
        Payment Queue   Analytics Queue
              |              |
              v              v
        Payment Worker   Analytics Worker

SNS provides:

Fan-out

SQS provides:

Independent buffering

This architecture allows each consumer to operate independently.

If Analytics is unavailable:

Analytics Queue
      |
      v
Messages accumulate

Payment can continue processing:

Payment Queue
      |
      v
Payment Worker
      |
      v
Continues

That’s one of the most useful patterns in AWS event-driven systems.

SQS and Lambda vs SQS and EC2

SQS doesn’t require Lambda.

You can have:

SQS
 |
 v
EC2 Worker

or:

SQS
 |
 v
ECS Worker

or:

SQS
 |
 v
Kubernetes Worker

The queue doesn’t care where the consumer runs.

That’s another important benefit.

SQS creates a clean boundary between:

Message producer

and:

Message consumer

A Kubernetes Example

Suppose you run an application on EKS.

You could have:

API
 |
 v
SQS
 |
 v
Worker Deployment
 |
 +----> Pod 1
 +----> Pod 2
 +----> Pod 3

When queue depth increases, Kubernetes can scale the workers based on an appropriate queue-related signal.

The architecture remains the same:

Producer
   |
   v
SQS
   |
   v
Workers

The consumer technology can change without changing the messaging model.

Security in SQS

Messaging infrastructure needs appropriate access controls.

Think about:

Who can send messages?
Who can receive messages?
Who can delete messages?
Who can change queue configuration?
Who can purge the queue?

For example:

Order Service
     |
     | SendMessage
     v
Order Queue
     |
     | ReceiveMessage
     v
Order Worker

The Order Service doesn’t need permissions to consume from the queue.

The worker doesn’t necessarily need permissions to publish.

Follow the principle of least privilege.

Encryption

SQS supports server-side encryption.

For sensitive workloads, you can use AWS KMS-based encryption and control access through appropriate key and IAM policies.

Think about the complete data path:

Producer
   |
   v
Encrypted SQS
   |
   v
Consumer

If your messages contain sensitive information, encryption should be part of the design.

Also remember that encryption at rest doesn’t automatically solve every data-security concern.

You still need appropriate:

• IAM
• Network controls
• Logging
• Data retention
• Access auditing
• Application-level protection

Don’t Put Sensitive Data in Messages Without Thinking

A queue message might live longer than the original API request.

For example:

User request
     |
     v
SQS
     |
     | 30 minutes later
     v
Worker

That means sensitive information could remain in the queue during that period.

Ask:

Does the consumer really need the entire payload?

Sometimes an event like:

{
  "orderId": "ORD-10001"
}

is better than placing a large amount of customer data directly into the message.

The worker can retrieve the required information from an appropriate data store.

SQS Message Size

SQS messages have a maximum message size.

This is another reason not to treat SQS as a place to store large payloads.

For example, instead of:

SQS
 |
 +----> Huge report

consider:

S3
 |
 +----> Large report

SQS
 |
 +----> Reference to report

For example:

{
  "eventType": "ReportReady",
  "reportId": "RPT-123",
  "location": "s3://example-bucket/reports/RPT-123.json"
}

The queue carries the event.

Object storage carries the large payload.

Common SQS Mistakes

1. Treating SQS as a Database

SQS is for messaging and temporary work buffering.

It isn’t a general-purpose database.

2. Ignoring Idempotency

Messages can be processed more than once depending on the delivery behavior and failure scenario.

Consumers should be designed accordingly.

3. Setting Visibility Timeout Incorrectly

Too short can cause duplicate processing.

Too long can delay retries.

4. Ignoring DLQs

Repeated failures need somewhere to go.

5. Monitoring Only Queue Depth

Also consider message age and consumer health.

6. Allowing Unlimited Consumer Concurrency

You can overwhelm downstream databases and APIs.

7. Assuming Queueing Eliminates Failure

It doesn’t.

It changes where and how failure is handled.

8. Assuming FIFO Is Always Better

If you don’t need ordering, Standard queues may be the better fit.

A Production SQS Architecture

Let’s put everything together.