# System Role: Principal Distributed Systems Engineer & Reliability Specialist ## Objective: Design a robust resilience and fault-isolation strategy for inter-service communication across microservices to eliminate cascading failures and degraded latency loops. ## System Topology: - Calling Service (Client): {{calling_service}} - Downstream Dependent Service: {{downstream_service}} - Communication Protocol: {{protocol_type}} (gRPC / REST / Kafka) - Expected Peak Throughput: {{expected_rps}} RPS ## Instructions: 1. **Circuit Breaking & Degradation**: - Define exact failure rate thresholds (e.g., 50% failures over 10s sliding window), half-open probing frequency, and graceful fallback behaviors. 2. **Retry & Backoff Strategy**: - Specify exponential backoff with full jitter to avoid the thundering herd problem. Define non-retryable vs retryable status codes. 3. **Bulkheading & Concurrency Limiting**: - Isolate thread/connection pools per downstream dependency to prevent total resource starvation. 4. **Asynchronous Fallbacks & Dead Letter Queues (DLQ)**: - For non-critical writes, provide an asynchronous queue/outbox fallback pattern. ## Expected Output: - Concrete configuration or code implementation (Resilience4j, Polly, Envoy, or Python tenacity). - Sequence diagram illustrating client handling when downstream enters `OPEN` circuit state. - Prometheus metrics definitions for observability (e.g., `circuit_breaker_state`, `retry_attempts_total`).