20 Real-World Backend Engineering Challenges Every Senior Developer Should Know

Building a backend application that works correctly under normal conditions is one thing. Keeping it reliable when thousands of requests arrive simultaneously, downstream services become unavailable, database connections are exhausted, and messages are processed multiple times is an entirely different challenge.

In real production environments, backend engineers face problems that cannot always be solved by adding more CPU, increasing thread pool sizes, or restarting an application.

A senior backend developer must understand application internals, concurrency, distributed systems, database behavior, cloud infrastructure, caching, messaging, and system architecture.

In this article, we will explore 20 real-world backend engineering challenges covering Java, Spring Boot, Kafka, AWS, Redis, databases, microservices, and production troubleshooting.

These are not simple definition-based interview questions. Each scenario tests how you identify a problem, investigate its root cause, choose an appropriate solution, and understand the trade-offs involved.

1. Your Spring Boot API handles 500 requests/sec, but performance drops at 2,000. CPU usage is only 40%. Where would you investigate?

Scenario:

A Spring Boot application performs well at 500 requests per second. When traffic increases to 2,000 requests per second, response times increase significantly and some requests start timing out.

Interestingly, CPU utilization remains around 40%. Increasing the number of application instances provides only limited improvement.

Why does this happen?

Low CPU utilization does not necessarily mean the application has sufficient capacity.

The application may be waiting for another resource rather than performing CPU-intensive operations.

Possible bottlenecks include:

• Database connection pool exhaustion.
• Slow SQL queries or database locks.
• HTTP connection pool exhaustion.
• Thread pool saturation.
• Slow downstream services.
• Network latency.
• Garbage collection pauses.
• Load balancer or infrastructure limits.

How would you investigate it?

1. Check thread utilization.

Capture thread dumps and inspect whether application threads are waiting for database connections, HTTP responses, locks, or other resources.

jstack <PID> > thread-dump.txt

2. Check database connection pool metrics.

If HikariCP has reached its maximum pool size and requests are waiting for connections, the database connection pool may be the bottleneck.

3. Check downstream HTTP clients.

Review connection pool limits, connection acquisition time, response time, and timeout configuration for external services.

4. Analyze request latency.

Use distributed tracing to identify which operation consumes the most time in the request lifecycle.

5. Review garbage collection.

Check whether frequent garbage collection or long pauses are affecting response times.

How should you fix it?

• Optimize slow database queries.
• Review database and HTTP connection pool sizing.
• Reduce unnecessary blocking operations.
• Introduce caching where appropriate.
• Use asynchronous processing for suitable workloads.
• Apply timeouts and backpressure.
• Scale the bottleneck rather than blindly scaling application instances.

Production lesson:

When CPU usage is low but response times are high, investigate waiting time, resource saturation, and downstream dependencies. The bottleneck may not be the application CPU.

2. A shared HashMap cache returns inconsistent values under concurrent access. How would you fix it?

Scenario:

A Spring Boot application maintains a shared in-memory cache using HashMap.

Multiple request threads read and update the cache simultaneously. Occasionally, users receive inconsistent values, and the application behaves unpredictably under load.

Why does this happen?

HashMap is not designed for concurrent modification by multiple threads without external synchronization.

When multiple threads update a shared HashMap concurrently, the application can experience race conditions and inconsistent observations.

The problem becomes more complicated when a business operation performs multiple map operations together.

How should you fix it?

If individual concurrent map operations are sufficient, use ConcurrentHashMap.

private final Map<String, User> cache =
        new ConcurrentHashMap<>();

public void saveUser(User user) {
    cache.put(user.getId(), user);
}

public User getUser(String id) {
    return cache.get(id);
}

What if multiple operations must be atomic?

Consider this example:

if (!cache.containsKey(userId)) {
    cache.put(userId, user);
}

Although ConcurrentHashMap makes individual operations thread-safe, the complete check-and-update sequence is not atomic.

Use an atomic operation instead:

cache.putIfAbsent(userId, user);

For more complex updates, use computeIfAbsent or compute:

cache.computeIfAbsent(userId, id ->
    loadUserFromDatabase(id)
);

What else should you consider?

• Use bounded caches with eviction policies.
• Consider cache expiration requirements.
• Avoid storing mutable objects that can be modified outside the cache.
• Use distributed caching when multiple application instances need shared cache state.
• Define cache consistency requirements before selecting an implementation.

Production lesson:

Thread-safe collections solve concurrent access problems at the collection level, but they do not automatically make every business operation atomic.

3. CompletableFuture improves response time initially, but performance degrades as traffic increases. What could be wrong?

Scenario:

A Spring Boot application uses CompletableFuture to execute multiple external API calls concurrently.

Initially, response times improve because independent operations execute in parallel. However, when traffic increases, latency becomes unpredictable and requests start timing out.

Why does this happen?

CompletableFuture does not automatically guarantee unlimited or well-managed concurrency.

When asynchronous tasks are submitted without controlling execution resources, the application can overload its thread pools and downstream services.

Common causes include:

• Unbounded task submission.
• Blocking operations running on the common ForkJoinPool.
• Insufficient executor capacity.
• Downstream API rate limits.
• Database connection pool exhaustion.
• Missing timeouts.
• Tasks waiting on other asynchronous tasks.

How should you investigate it?

First, inspect the executor configuration and thread utilization.

Check whether tasks are blocking worker threads while waiting for HTTP or database responses.

Measure:

• Task submission rate.
• Task completion rate.
• Executor queue size.
• Active thread count.
• Task execution duration.
• Downstream response time.

How should you fix it?

Use an explicitly configured executor for blocking operations.

@Bean
public ExecutorService apiExecutor() {
    return new ThreadPoolExecutor(
        10,
        30,
        60,
        TimeUnit.SECONDS,
        new ArrayBlockingQueue<>(200),
        new ThreadPoolExecutor.CallerRunsPolicy()
    );
}

Then pass the executor explicitly:

CompletableFuture<User> userFuture =
    CompletableFuture.supplyAsync(
        () -> userClient.getUser(),
        apiExecutor
    );

Also configure timeouts:

CompletableFuture<User> result =
    CompletableFuture.supplyAsync(
        () -> userClient.getUser(),
        apiExecutor
    ).orTimeout(3, TimeUnit.SECONDS);

For workloads with many independent operations, consider whether virtual threads, reactive processing, or controlled asynchronous execution is the better fit.

Important consideration

Adding more threads does not automatically increase throughput. If the database or downstream service has limited capacity, additional concurrency can make the situation worse.

Production lesson:

Asynchronous programming improves concurrency only when execution resources, task queues, timeouts, and downstream capacity are properly managed.

4. Users experience inconsistent session behavior across multiple application instances. How would you investigate?

Scenario:

A Spring Boot application runs on multiple instances behind an AWS Application Load Balancer.

Some users are logged in successfully, but subsequent requests occasionally behave as if the session does not exist.

The issue appears only when traffic is distributed across multiple application instances.

Why does this happen?

Each application instance has its own memory.

If HTTP session information is stored locally in one instance, another instance may not have access to that session.

For example:

• Request 1 reaches Instance A.
• Instance A creates a session.
• Request 2 reaches Instance B.
• Instance B cannot find the locally stored session.

How would you investigate it?

• Check whether sessions are stored in application memory.
• Inspect load balancer routing and session affinity settings.
• Verify cookie configuration and expiration.
• Check whether application instances use compatible session configuration.
• Review logs for session creation and invalidation events.

How should you fix it?

One approach is to use Spring Session with Redis as a shared session store.

@Configuration
@EnableRedisHttpSession
public class SessionConfiguration {
}

Now multiple application instances can access shared session information through Redis.

Another approach is to design the application to be stateless using token-based authentication where appropriate.

What about sticky sessions?

Sticky sessions can route a user to the same application instance, but they do not provide complete session resilience.

If the instance becomes unavailable, the locally stored session may be lost.

Production lesson:

Applications running across multiple instances should not depend on local in-memory session state unless session affinity and failure behavior are explicitly designed and accepted.

5. A method annotated with @Transactional commits even though part of the business operation fails. What could cause this?

Scenario:

A service method updates an order, deducts inventory, and records a payment operation.

During execution, one operation fails. The developer expects the database transaction to roll back, but some changes are committed.

Why does this happen?

Spring transaction management is based on transaction boundaries and rollback rules.

Several issues can cause unexpected commits.

1. Checked exceptions

By default, Spring rolls back on RuntimeException and Error. Checked exceptions do not trigger rollback by default.

@Transactional
public void processOrder() throws Exception {

    orderRepository.save(new Order());

    throw new Exception("Processing failed");
}

Use rollbackFor when required:

@Transactional(rollbackFor = Exception.class)
public void processOrder() throws Exception {

    orderRepository.save(new Order());

    throw new Exception("Processing failed");
}

2. Self-invocation

Spring's proxy-based transaction management may be bypassed when a transactional method is called directly from another method in the same class.

public void process() {
    saveOrder();
}

@Transactional
public void saveOrder() {
    orderRepository.save(new Order());
}

3. Exception swallowed inside the method

@Transactional
public void processOrder() {

    try {
        orderRepository.save(new Order());
        paymentRepository.save(new Payment());

    } catch (Exception ex) {
        log.error("Processing failed", ex);
    }
}

If the exception is caught and not rethrown, Spring may see a normal method completion.

4. Transaction propagation

Review propagation settings such as REQUIRED and REQUIRES_NEW. A separate transaction may commit independently of the outer transaction.

5. Different transaction boundaries

Operations involving different transaction managers, databases, or asynchronous execution may not participate in the same transaction.

How should you investigate it?

• Verify the method is invoked through the Spring proxy.
• Review exception types and rollback rules.
• Inspect transaction propagation.
• Check whether exceptions are swallowed.
• Verify that repositories use the expected transaction manager.
• Inspect asynchronous execution and thread boundaries.

Production lesson:

@Transactional does not automatically make an entire business workflow atomic. Understand proxy boundaries, rollback rules, propagation, and the resources participating in the transaction.

6. A user's JWT remains valid after their role is revoked. How would you handle immediate access revocation?

Scenario:

An administrator removes a user's administrative role. However, the user's previously issued JWT remains valid, and the user can still access protected APIs until the token expires.

Why does this happen?

JWT access tokens are commonly validated using their signature and claims.

Once issued, the token can remain valid until its expiration time, even if the user's permissions change in the database.

This creates a gap between the latest authorization state and the information contained in the token.

How should you handle it?

The correct approach depends on how quickly permissions must take effect.

1. Short-lived access tokens

Use short access token lifetimes and refresh tokens. This reduces the time during which outdated claims can remain valid.

2. Token introspection

Use centrally validated or introspected tokens when immediate changes in token status must be reflected.

3. Permission lookup

Validate sensitive permissions against a current authorization source instead of relying exclusively on role claims inside the JWT.

4. Distributed revocation events

Publish permission-change or user-revocation events to services. Each service can invalidate relevant cached authorization information.

5. Token denylist

For certain applications, maintain a distributed denylist using a token identifier such as jti until the token expires.

However, this introduces additional storage and lookup requirements.

What should you consider?

• Use short token lifetimes where practical.
• Separate authentication from authorization.
• Invalidate permission caches when roles change.
• Use centralized checks for sensitive operations.
• Define behavior when the authorization service is unavailable.

Production lesson:

JWT signature validation proves that a token is authentic. It does not guarantee that the permissions inside that token are still current.

7. A payment API receives the same request multiple times. How would you prevent duplicate payments?

Scenario:

A customer clicks the payment button twice because the first request appears to be taking too long.

The client retries the request after a timeout. The payment provider processes both requests, resulting in duplicate charges.

Why does this happen?

Distributed systems cannot always guarantee that a client knows whether a request was successfully processed.

For example:

1. The client sends a payment request.
2. The payment service processes the payment.
3. The response is lost because of a network failure.
4. The client retries the request.
5. The service processes it again.

How does idempotency solve this?

Require an idempotency key for payment requests.

The same logical payment request must reuse the same key during retries.

POST /payments

Idempotency-Key: payment-12345

Store the key together with the processing status and result.

Example implementation

@Transactional
public PaymentResponse processPayment(
        String idempotencyKey,
        PaymentRequest request) {

    var existing = paymentRepository
        .findByIdempotencyKey(idempotencyKey);

    if (existing.isPresent()) {
        return existing.get().toResponse();
    }

    Payment payment = new Payment();
    payment.setIdempotencyKey(idempotencyKey);
    payment.setStatus("PENDING");

    paymentRepository.save(payment);

    return payment.toResponse();
}

This is a simplified example. A production implementation must also handle simultaneous requests using the same key.

Important production considerations

• Create a unique database constraint on the idempotency key.
• Handle concurrent requests safely.
• Store the original request result.
• Define how long idempotency records must be retained.
• Handle pending and failed payment states.
• Use the payment provider's idempotency mechanism when available.
• Reconcile uncertain payment outcomes before retrying external charges.

Production lesson:

Never assume that a client request is executed exactly once. Design payment operations to be safe when requests are retried or delivered multiple times.

8. Kafka consumers process the same event more than once, causing duplicate database updates. How would you make processing safe?

Scenario:

A Kafka consumer receives an OrderCreated event and updates an order summary table.

Sometimes the consumer processes the same event multiple times, causing duplicate updates or incorrect business calculations.

Why does this happen?

Kafka consumers commonly use at-least-once processing behavior.

Consider this sequence:

1. The consumer reads an event.
2. The consumer updates the database successfully.
3. The application crashes before committing the Kafka offset.
4. Kafka delivers the event again after recovery.
5. The database update happens again.

How should you fix it?

Make event processing idempotent.

Assign a unique event ID to every business event.

Maintain a table that records successfully processed event IDs.

@Entity
public class ProcessedEvent {

    @Id
    private String eventId;

    private Instant processedAt;
}

Process the event and record its ID within the same database transaction.

@Transactional
public void process(OrderCreatedEvent event) {

    if (processedEventRepository
            .existsById(event.getEventId())) {
        return;
    }

    updateOrderSummary(event);

    processedEventRepository.save(
        new ProcessedEvent(event.getEventId())
    );
}

A unique database constraint on eventId is essential to protect against concurrent duplicate processing.

What else should you consider?

• Understand offset commit behavior.
• Use manual acknowledgement when appropriate.
• Keep database updates and deduplication records in one transaction.
• Handle duplicate delivery safely.
• Use Kafka transactions when Kafka-to-Kafka atomic processing is required.
• Monitor failed messages and dead-letter handling.

Production lesson:

Reliable Kafka consumers should assume duplicate delivery is possible. Idempotent processing is more dependable than assuming every event will be delivered exactly once to the business logic.

9. Kafka consumer lag keeps increasing even after adding more consumers. What would you check?

Scenario:

A Kafka-based application processes thousands of events. Consumer lag continuously increases.

The team adds more consumer instances, but the lag does not improve as expected.

Why does this happen?

Adding consumers does not always increase processing parallelism.

Kafka assigns partitions to consumers within a consumer group. A partition is assigned to only one consumer in that group at a time.

If the topic has fewer partitions than consumers, some consumers remain idle.

What else could cause increasing lag?

• Uneven message distribution across partitions.
• Slow message processing.
• Frequent consumer rebalances.
• Long-running external API calls.
• Incorrect offset commit behavior.
• Messages repeatedly failing processing.
• Broker or network performance issues.

How would you investigate it?

Inspect consumer group status:

kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group order-consumer-group

Review lag by partition instead of looking only at total lag.

Check:

• Number of topic partitions.
• Number of active consumers.
• Processing time per message.
• Consumer rebalance frequency.
• Offset commit failures.
• Producer throughput versus consumer throughput.
• Partition key distribution.

How should you fix it?

• Increase partition count when additional parallelism is required and partitioning changes are acceptable.
• Optimize consumer processing logic.
• Review message key distribution.
• Reduce unnecessary external calls.
• Handle poison messages appropriately.
• Review max.poll.interval.ms and related consumer settings.
• Monitor broker fetch performance and network latency.

Production lesson:

More consumers do not automatically mean more throughput. Partition count, message distribution, processing speed, and rebalance behavior determine actual consumer capacity.

10. Multiple microservices update the same business entity. How would you maintain data consistency?

Scenario:

An e-commerce platform has Order, Payment, Inventory, and Shipping services.

Multiple services update different parts of the same business workflow. Sometimes the order status indicates that it is confirmed even though inventory reservation has failed.

Why does this happen?

Microservices generally own their data and maintain independent database transactions.

There is no automatic global transaction covering all service databases.

Directly allowing multiple services to update the same database tables can also create tight coupling and unclear ownership.

How should you design it?

Start by defining clear ownership of business data.

• Order Service owns order lifecycle state.
• Payment Service owns payment transactions.
• Inventory Service owns stock reservations.
• Shipping Service owns shipment information.

Use events to communicate state changes between services.

How does the Saga pattern help?

A Saga coordinates a sequence of local transactions.

For example:

1. Order Service creates a pending order.
2. Payment Service authorizes payment.
3. Inventory Service reserves stock.
4. Shipping Service creates shipment information.
5. Order Service confirms the order.

If inventory reservation fails, the workflow can initiate a payment compensation and mark the order as failed.

What should you consider?

• Use clear data ownership.
• Design idempotent event handlers.
• Track workflow state.
• Implement compensation operations.
• Handle duplicate and out-of-order events.
• Use the Outbox pattern for reliable event publishing.
• Make eventual consistency visible to the business workflow.

Production lesson:

Consistency across microservices requires explicit ownership and workflow design. Sharing database tables between services is not a substitute for a distributed consistency strategy.

11. SQS messages arrive faster than consumers can process them. How would you scale the system?

Scenario:

A Spring Boot application consumes messages from Amazon SQS.

During peak traffic, producers publish messages faster than consumers can process them. The queue depth continuously increases, and message processing becomes delayed.

Why does this happen?

Queue growth occurs when the incoming message rate is greater than the processing rate.

Possible causes include:

• Insufficient consumer capacity.
• Slow database operations.
• Long-running message processing.
• Downstream API limitations.
• Repeated retries.
• Messages becoming visible again before processing completes.

How would you investigate it?

Monitor the following SQS metrics:

• ApproximateNumberOfMessagesVisible.
• ApproximateNumberOfMessagesNotVisible.
• ApproximateAgeOfOldestMessage.
• NumberOfMessagesSent.
• NumberOfMessagesDeleted.

Compare incoming message rate with successful processing rate.

How should you scale it?

For ECS-based consumers, scale the number of tasks based on queue depth, message age, and processing capacity.

For Lambda consumers, configure event source mapping concurrency according to downstream capacity.

Other improvements include:

• Optimize message processing logic.
• Batch messages where appropriate.
• Use efficient database operations.
• Set visibility timeout based on expected processing duration.
• Configure dead-letter queues.
• Control concurrency to protect downstream systems.

What is the important trade-off?

Increasing consumer concurrency can reduce queue depth, but it can also overload the database or external services.

Scaling should consider the capacity of the entire processing pipeline.

Production lesson:

Do not scale consumers based only on queue size. Measure processing throughput and ensure downstream dependencies can handle the additional concurrency.

12. ECS tasks are terminated while requests are still running. How would you ensure graceful shutdown?

Scenario:

A Spring Boot application runs as an ECS service behind an Application Load Balancer.

During deployments or scaling events, ECS terminates existing tasks while some requests are still being processed.

Users receive connection errors or incomplete responses.

Why does this happen?

Container termination can interrupt active requests if the application does not have enough time to complete ongoing work.

Graceful shutdown requires coordination between the load balancer, ECS task lifecycle, and application shutdown behavior.

How should you implement it?

First, enable graceful shutdown in Spring Boot:

server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

When shutdown begins, the application stops accepting new requests and attempts to finish in-flight requests within the configured timeout.

What else should you configure?

• Configure an appropriate ECS stop timeout.
• Allow enough time for the application to shut down.
• Ensure load balancer health checks remove unhealthy targets.
• Configure deployment settings to avoid reducing capacity too aggressively.
• Handle SIGTERM correctly.
• Ensure background workers stop accepting new jobs and complete or safely release ongoing work.

What about long-running requests?

For long-running operations, consider asynchronous job processing instead of keeping an HTTP request open for an extended period.

For message consumers, ensure unprocessed messages can be retried safely after shutdown.

Production lesson:

Graceful shutdown is a coordination problem, not just a Spring Boot property. The application, container platform, and load balancer must all allow enough time for active work to finish.

13. A database connection failure in one Availability Zone impacts the entire application. How would you improve resilience?

Scenario:

A Spring Boot application is deployed across multiple AWS Availability Zones.

A database connectivity issue in one Availability Zone causes application requests across the platform to fail.

Why does this happen?

Although the application is distributed across multiple Availability Zones, its database connectivity may still depend on a shared endpoint or network path.

Possible causes include:

• Database failover delays.
• Incorrect subnet or security group configuration.
• Connection pools retaining stale connections.
• DNS caching issues.
• Insufficient database availability configuration.
• Applications not handling temporary database unavailability.

How would you investigate it?

• Check database failover events.
• Inspect RDS connectivity and availability metrics.
• Review subnet groups and security groups.
• Check application connection pool errors.
• Inspect DNS resolution and connection establishment.
• Review application behavior during database recovery.

How should you improve resilience?

For supported relational database workloads, consider Amazon RDS Multi-AZ deployment options.

Also:

• Deploy application instances across multiple Availability Zones.
• Use appropriate connection timeouts.
• Ensure connection pools recover from stale connections.
• Apply bounded retries with backoff and jitter.
• Use circuit breakers where appropriate.
• Move non-critical work to asynchronous queues.
• Test database failover scenarios.
• Maintain backups and a recovery strategy.

Important consideration

RDS Multi-AZ improves availability, but database failover is not necessarily instantaneous. Applications must tolerate temporary connection failures and reconnect correctly.

Production lesson:

Multi-AZ deployment alone does not guarantee application resilience. The application must be designed to handle failover, connection loss, and temporary database unavailability.

14. An S3 upload succeeds, but the corresponding database record is missing. How would you handle this inconsistency?

Scenario:

A document upload service stores files in Amazon S3 and saves document metadata in a relational database.

The S3 upload succeeds, but the database transaction fails. The file exists in S3, but the application has no corresponding database record.

Why does this happen?

S3 and the relational database are separate systems.

A normal database transaction cannot automatically roll back a successful S3 upload.

This is another example of a distributed consistency problem.

How should you design the workflow?

One approach is to maintain an explicit upload lifecycle.

For example:

1. Create a database record with status UPLOADING.
2. Generate a unique object key.
3. Upload the file to S3.
4. Verify successful upload.
5. Update the database record to COMPLETED.

If the upload or database update fails, mark the operation for recovery.

How should you handle orphaned files?

Implement a reconciliation process that periodically checks for files in S3 without corresponding completed database records.

Depending on business requirements, the process can:

• Retry the database operation.
• Mark the upload as failed.
• Move the object to a temporary location.
• Delete the orphaned object after a safe retention period.

What else should you consider?

• Use unique object keys.
• Make upload operations idempotent.
• Store upload status and timestamps.
• Use lifecycle policies for temporary objects.
• Make retries safe.
• Use background reconciliation.
• Do not delete an object immediately when a database operation fails without checking whether another process still depends on it.

Production lesson:

Operations spanning S3 and a database need an explicit recovery strategy. Do not assume that success in one system guarantees success in the other.

15. An application migration from ECS to EKS introduces unexpected failures. What would you investigate first?

Scenario:

A Spring Boot microservices platform is migrated from Amazon ECS to Amazon EKS.

The application works correctly in ECS, but after migration, some services cannot communicate, requests time out, and certain workloads fail to start.

Why does this happen?

ECS and EKS have different deployment, networking, security, and operational models.

A migration can introduce issues that are unrelated to the Java application code.

What would you investigate first?

1. Pod health and startup

Check whether containers are starting successfully.

kubectl get pods -n production

kubectl describe pod <pod-name> -n production

kubectl logs <pod-name> -n production

Look for CrashLoopBackOff, ImagePullBackOff, failed probes, and resource-related errors.

2. Networking and service discovery

Check Kubernetes Services, DNS resolution, ingress configuration, network policies, and connectivity between namespaces.

3. Resource limits

Review CPU and memory requests and limits. Incorrect resource configuration can cause throttling or out-of-memory terminations.

4. IAM permissions

Applications accessing AWS services may need different IAM configuration through EKS Pod Identity or IAM Roles for Service Accounts.

5. Configuration and secrets

Compare environment variables, ConfigMaps, Secrets, mounted volumes, and application profiles with the previous ECS deployment.

6. Health checks

Verify readiness, liveness, and startup probes. Incorrect probe settings can cause Kubernetes to restart healthy but slow-starting applications.

7. Deployment behavior

Review rolling updates, replica counts, autoscaling, pod disruption budgets, and termination handling.

Production lesson:

Cloud platform migrations should be investigated layer by layer. Validate application startup, networking, security, resource configuration, and health checks before assuming the application code is responsible.

16. Redis serves stale data after a successful database update. How would you design cache invalidation?

Scenario:

A Spring Boot application uses Redis to cache customer information.

A customer updates their profile successfully in the database. However, subsequent API requests sometimes return the old profile information.

Why does this happen?

The database and Redis cache contain separate copies of the data.

If the application updates the database but does not invalidate or refresh the cache correctly, users may receive stale values.

How should you fix it?

One common approach is cache-aside.

When reading data:

1. Check Redis.
2. If the value exists, return it.
3. If the value is missing, read from the database.
4. Populate Redis with the database result.

When updating data:

1. Update the database.
2. Invalidate the corresponding cache entry after successful commit.

@Transactional
public void updateCustomer(Customer customer) {
    customerRepository.save(customer);
}

Cache invalidation can be triggered after the transaction commits, for example using a transaction-bound event listener.

@TransactionalEventListener(
    phase = TransactionPhase.AFTER_COMMIT
)
public void handleCustomerUpdated(
        CustomerUpdatedEvent event) {

    redisTemplate.delete(
        "customer:" + event.getCustomerId()
    );
}

What about multiple application instances?

All instances should use a shared invalidation mechanism. Redis itself can be the shared cache, while application events or messaging can help invalidate additional local caches.

What should you consider?

• Use appropriate TTL values.
• Invalidate only the affected keys.
• Handle concurrent reads and writes.
• Consider cache stampede protection.
• Use versioning where appropriate.
• Monitor cache hit rate and stale-read incidents.
• Use reconciliation for important data where strict consistency is required.

Production lesson:

Cache invalidation is a consistency problem. A successful database update does not automatically guarantee that every cached copy contains the latest value.

17. One slow downstream service causes failures across multiple microservices. Which resilience patterns would you apply?

Scenario:

Several microservices depend on a shared Customer Service.

Customer Service becomes slow. Other services start waiting for responses, threads become occupied, and request timeouts spread across the platform.

Why does this happen?

Microservices are connected through network calls. A slow dependency can consume upstream resources and trigger cascading failures.

Which resilience patterns would you apply?

1. Timeout

Set connection and response timeouts so requests do not wait indefinitely.

2. Circuit breaker

Stop sending requests temporarily when a dependency is consistently failing. Allow controlled test requests when the dependency may have recovered.

3. Bulkhead

Isolate resources used by different dependencies so that one failing service cannot consume all available threads or connections.

4. Retry with exponential backoff

Retry transient failures carefully. Add jitter to avoid synchronized retry storms.

5. Rate limiting

Control request volume to protect the downstream service.

6. Fallback

Use cached data or a degraded response when business requirements permit.

7. Asynchronous processing

Move non-critical operations to queues when immediate synchronous execution is not required.

What should you avoid?

• Unlimited retries.
• Very long timeouts.
• Unbounded thread pools.
• Retrying non-idempotent operations without protection.
• Fallback responses that silently violate business requirements.

Production lesson:

Resilience is about containing failures. Timeouts, circuit breakers, bulkheads, controlled retries, and asynchronous processing help prevent one unhealthy service from affecting the entire platform.

18. How would you deploy database schema changes without breaking older application versions?

Scenario:

A Spring Boot application stores a customer's full name in one database column.

A new release requires separate first_name and last_name columns.

During deployment, old and new application versions may run simultaneously. The system must remain available throughout the migration.

Why is this challenging?

Older application versions expect the existing database schema, while newer versions expect the updated schema.

Changing the database schema too early can break older instances. Deploying new application code too early can break the new version.

How does expand-and-contract solve this?

Step 1: Expand the schema

Add the new columns without removing the old column.

ALTER TABLE customer
ADD COLUMN first_name VARCHAR(100);

ALTER TABLE customer
ADD COLUMN last_name VARCHAR(100);

Step 2: Deploy backward-compatible application code

Deploy a version that supports both the old and new database representation.

Step 3: Backfill existing data

Populate the new columns for existing records using a controlled migration process.

Step 4: Switch application reads

After verifying the new data, deploy a version that reads from the new columns.

Step 5: Contract the schema

After all older application versions have been removed and the old column is no longer needed, remove it in a separate migration.

What should you consider?

• Keep migrations backward-compatible.
• Test rolling deployments.
• Validate backfilled data.
• Monitor application errors.
• Prepare a rollback strategy.
• Avoid destructive schema changes during the initial rollout.

Production lesson:

Zero-downtime database migration requires compatibility between old and new application versions. Use expand-and-contract rather than combining application deployment and destructive schema changes.

19. API response times are inconsistent, but CPU and memory usage look normal. How would you identify the bottleneck?

Scenario:

A production API sometimes responds in 100 milliseconds and sometimes takes 5 seconds.

CPU and memory usage look normal. Restarting the application does not permanently solve the issue.

Why does this happen?

Response latency includes more than application CPU execution time.

A request may spend most of its time waiting for:

• Database queries.
• Connection pool availability.
• External HTTP services.
• Network communication.
• Locks and synchronized sections.
• Thread pool capacity.
• Cache misses.
• Queue processing.

How would you investigate it?

1. Distributed tracing

Use OpenTelemetry or another tracing solution to identify where each request spends time.

A trace can show whether the delay comes from the API itself, a database query, or a downstream service.

2. Application metrics

Monitor request rate, error rate, p50, p95, and p99 response times.

Average response time alone can hide serious latency problems affecting a small percentage of requests.

3. Database monitoring

Inspect slow queries, lock waits, connection acquisition time, and database resource utilization.

4. Thread dumps

Capture thread dumps when the issue occurs.

Look for blocked threads, waiting threads, lock contention, and long-running operations.

5. JVM profiling

Use Java Flight Recorder to investigate CPU hotspots, allocation behavior, garbage collection, and thread activity.

6. Infrastructure telemetry

Check network latency, load balancer metrics, container throttling, DNS resolution, and dependency health.

How should you fix it?

Fix the bottleneck identified through evidence. Do not change thread pool sizes or increase infrastructure capacity without understanding the actual cause.

Production lesson:

Normal CPU and memory usage do not guarantee healthy application performance. Distributed tracing and latency breakdowns are essential for finding hidden bottlenecks.

20. You need to break a large monolith into microservices. How would you identify the first service to extract?

Scenario:

An organization operates a large Spring Boot monolith containing customer management, orders, payments, inventory, and reporting modules.

The application is difficult to deploy, maintain, and scale. The team wants to move toward microservices but does not want to rewrite the entire system at once.

Why is this challenging?

A monolith may contain years of business logic and dependencies between modules.

Extracting the wrong module can introduce distributed transactions, network communication overhead, duplicated data, and operational complexity.

Microservices do not automatically make an application easier to maintain.

How would you identify the first service?

Start by understanding business capabilities and module boundaries.

Look for a module that:

• Has a clear business responsibility.
• Has relatively few dependencies on other modules.
• Can own its data.
• Has independent scaling requirements.
• Changes at a different frequency from the rest of the application.
• Can be extracted without introducing excessive distributed coordination.

What should you avoid?

Do not select a service simply because it contains fewer classes or has a convenient technical boundary.

For example, extracting a database access layer into a separate service may create network calls without providing meaningful business independence.

How should you approach the migration?

1. Identify domain boundaries

Use Domain-Driven Design concepts such as bounded contexts to understand business ownership.

2. Map dependencies

Identify which modules communicate with each other and which database tables are shared.

3. Establish a clear boundary

Define the service API and data ownership before extracting the module.

4. Use the Strangler Fig pattern

Gradually redirect selected functionality from the monolith to the new service while keeping the remaining application operational.

5. Monitor and validate

Compare behavior, performance, reliability, and operational cost before extracting additional services.

Important consideration

Microservices introduce additional responsibilities such as service discovery, network failure handling, observability, distributed security, data consistency, and independent deployment.

Production lesson:

The goal of microservice extraction is not to create more services. It is to establish meaningful business boundaries and improve independent development, deployment, and scaling.

Senior Backend Engineer Production Readiness Checklist

Before attending senior backend engineering interviews, make sure you can explain and troubleshoot the following areas:

• High-throughput API performance and resource bottlenecks.
• Java concurrency and thread-safe collections.
• CompletableFuture and executor management.
• Session management in distributed applications.
• Spring transaction boundaries and rollback behavior.
• JWT authorization and immediate permission revocation.
• Idempotency in payment APIs.
• Kafka duplicate processing and consumer lag.
• Distributed consistency and Saga patterns.
• SQS queue scaling and consumer throughput.
• ECS graceful shutdown and deployment behavior.
• AWS Multi-AZ database resilience.
• S3 and database consistency.
• ECS-to-EKS migration troubleshooting.
• Redis cache invalidation strategies.
• Circuit breakers, retries, timeouts, and bulkheads.
• Zero-downtime database migrations.
• Distributed tracing and latency investigation.
• Monolith decomposition and service boundaries.

Frequently Asked Questions

1. Are these questions suitable for senior backend developer interviews?

Yes. These scenarios cover practical backend engineering challenges involving Java, Spring Boot, concurrency, distributed systems, Kafka, AWS, Redis, databases, and microservices architecture.

2. Should I prepare Java coding questions or production scenarios?

Prepare both. Coding questions test implementation skills, while production scenarios test your ability to investigate failures, understand system behavior, and make architectural decisions.

3. Which topics are most important for senior Java developers?

Java concurrency, JVM troubleshooting, Spring transaction management, database performance, Kafka, distributed consistency, API security, caching, cloud infrastructure, and observability are important areas.

4. How should I answer production scenario questions?

Start by understanding the symptoms. Explain the possible causes, describe how you would collect evidence, identify the root cause, and then propose a solution with its trade-offs.

Conclusion

Backend engineering is not just about writing REST APIs, implementing business logic, or connecting applications to databases.

Real engineering challenges appear when systems operate under pressure.

Requests arrive faster than expected. Threads become blocked. Database connections run out. Messages are processed more than once. Distributed services disagree about business state. Cloud deployments introduce unexpected failures.

A senior developer must understand these situations and know how to investigate them systematically.

The difference between an average backend developer and an experienced backend engineer is often the ability to identify the actual problem instead of applying random fixes.

Senior backend engineering is not just about writing code. It is about understanding failures, making trade-offs, and building systems that remain reliable under pressure.

How many of these scenarios have you handled in a real production environment?

Interview Preparation Resources

Looking for more Java, Spring Boot, Microservices, Cloud, and System Design interview questions?

• Java Spring Boot & Microservices Interview Q&A (200 Questions)
• Full Interview Kit (600 Questions)
• Complete Java Full Stack Developer Kit (900 Questions)

Explore more practical software engineering articles at LogicBrace.