Resilience

Resilience policies provide a powerful way to handle transient failures and make your applications more robust. Foundatio's resilience system includes retry logic, circuit breakers, timeouts, and exponential backoff.

The IResiliencePolicy Interface

public interface IResiliencePolicy
{
    Task ExecuteAsync(Func<CancellationToken, Task> action,
                      CancellationToken cancellationToken = default);
    Task<T> ExecuteAsync<T>(Func<CancellationToken, Task<T>> action,
                            CancellationToken cancellationToken = default);
}

Basic Usage

Creating a Policy

using Foundatio.Resilience;

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(5)
    .WithExponentialDelay(TimeSpan.FromSeconds(1))
    .WithJitter()
    .Build();

Executing with Retry

await policy.ExecuteAsync(async ct =>
{
    await SomeUnreliableOperationAsync(ct);
});

With Return Values

var result = await policy.ExecuteAsync(async ct =>
{
    return await GetDataFromApiAsync(ct);
});

ResiliencePolicyBuilder

Retry Configuration

var policy = new ResiliencePolicyBuilder()
    // Maximum number of attempts (default: 3)
    .WithMaxAttempts(5)

    // Fixed delay between retries
    .WithDelay(TimeSpan.FromSeconds(2))

    // Or exponential delay (doubles each retry)
    .WithExponentialDelay(TimeSpan.FromSeconds(1))

    // Or linear delay (adds fixed amount each retry)
    .WithLinearDelay(TimeSpan.FromSeconds(1))

    // Maximum delay cap
    .WithMaxDelay(TimeSpan.FromMinutes(1))

    // Add randomness to prevent thundering herd
    .WithJitter()

    .Build();

Custom Delay Function

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(5)
    .WithDelayFunction(attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)))
    .Build();

Timeout Configuration

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(3)
    .WithTimeout(TimeSpan.FromSeconds(30))  // Overall timeout
    .Build();

Conditional Retry

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(5)
    .WithExponentialDelay(TimeSpan.FromSeconds(1))
    .WithShouldRetry((attempt, exception) =>
    {
        // Only retry on specific exceptions
        return exception is HttpRequestException or TimeoutException;
    })
    .Build();

Unhandled Exceptions

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(5)
    // These exceptions will be thrown immediately without retry
    .WithUnhandledException<OperationCanceledException>()
    .WithUnhandledException<ArgumentException>()
    .Build();

Circuit Breaker

Prevent cascading failures by temporarily stopping calls to failing services:

var policy = new ResiliencePolicyBuilder()
    .WithMaxAttempts(3)
    .WithCircuitBreaker(cb => cb
        // Open circuit after 50% failure rate
        .WithFailureRatio(0.5)
        // Need at least 10 calls before evaluating
        .WithMinimumCalls(10)
        // Keep circuit open for 1 minute
        .WithBreakDuration(TimeSpan.FromMinutes(1)))
    .Build();

Circuit Breaker States

graph LR
    A[Closed] -->|Failures exceed threshold| B[Open]
    B -->|Break duration expires| C[Half-Open]
    C -->|Test call succeeds| A
    C -->|Test call fails| B

• Closed: Normal operation, calls pass through
• Open: Calls fail immediately without execution
• Half-Open: Single test call allowed to check recovery

Checking Circuit State

var policy = new ResiliencePolicy();
policy.CircuitBreaker = new CircuitBreaker(new CircuitBreakerBuilder()
    .WithFailureRatio(0.5)
    .WithMinimumCalls(10)
    .WithBreakDuration(TimeSpan.FromMinutes(1)));

// Check state
if (policy.CircuitBreaker.State == CircuitState.Open)
{
    _logger.LogWarning("Circuit is open - skipping operation");
    return;
}

await policy.ExecuteAsync(async ct =>
{
    await CallExternalServiceAsync(ct);
});

ResiliencePolicy Properties

var policy = new ResiliencePolicy
{
    // Maximum retry attempts
    MaxAttempts = 5,

    // Fixed delay between retries
    Delay = TimeSpan.FromSeconds(2),

    // Custom delay function
    GetDelay = ResiliencePolicy.ExponentialDelay(TimeSpan.FromSeconds(1)),

    // Maximum delay cap
    MaxDelay = TimeSpan.FromMinutes(1),

    // Add jitter to delays
    UseJitter = true,

    // Overall timeout
    Timeout = TimeSpan.FromMinutes(5),

    // Circuit breaker
    CircuitBreaker = new CircuitBreaker(...),

    // Exceptions that should not be retried
    UnhandledExceptions = { typeof(OperationCanceledException) },

    // Custom retry logic
    ShouldRetry = (attempt, ex) => ex is TransientException,

    // Logger for retry events
    Logger = logger
};

Static Delay Functions

Foundatio provides built-in delay functions:

// Exponential: 1s, 2s, 4s, 8s, 16s...
ResiliencePolicy.ExponentialDelay(TimeSpan.FromSeconds(1))

// Linear: 1s, 2s, 3s, 4s, 5s...
ResiliencePolicy.LinearDelay(TimeSpan.FromSeconds(1))

// Constant: 2s, 2s, 2s, 2s...
ResiliencePolicy.ConstantDelay(TimeSpan.FromSeconds(2))

IResiliencePolicyProvider

Manage multiple named policies:

using Foundatio.Resilience;

var provider = new ResiliencePolicyProviderBuilder()
    // Default policy for unspecified operations
    .WithDefaultPolicy(builder => builder
        .WithMaxAttempts(3)
        .WithExponentialDelay(TimeSpan.FromSeconds(1)))

    // Named policy for external APIs
    .WithPolicy("external-api", builder => builder
        .WithMaxAttempts(5)
        .WithCircuitBreaker()
        .WithTimeout(TimeSpan.FromSeconds(30)))

    // Named policy for database operations
    .WithPolicy("database", builder => builder
        .WithMaxAttempts(3)
        .WithLinearDelay(TimeSpan.FromMilliseconds(100))
        .WithUnhandledException<ArgumentException>())

    .Build();

// Get and use policies
var apiPolicy = provider.GetPolicy("external-api");
await apiPolicy.ExecuteAsync(async ct =>
{
    await CallExternalApiAsync(ct);
});

var dbPolicy = provider.GetPolicy("database");
await dbPolicy.ExecuteAsync(async ct =>
{
    await SaveToDbAsync(ct);
});

Type-Based Policies

Get policies based on service type:

var provider = new ResiliencePolicyProviderBuilder()
    .WithPolicy<IExternalApiClient>(builder => builder
        .WithMaxAttempts(5)
        .WithCircuitBreaker())
    .WithPolicy<IDatabaseService>(builder => builder
        .WithMaxAttempts(3)
        .WithLinearDelay())
    .Build();

var policy = provider.GetPolicy<IExternalApiClient>();

Common Patterns

HTTP Client with Resilience

public class ResilientHttpClient
{
    private readonly HttpClient _client;
    private readonly IResiliencePolicy _policy;

    public ResilientHttpClient(HttpClient client)
    {
        _client = client;
        _policy = new ResiliencePolicyBuilder()
            .WithMaxAttempts(3)
            .WithExponentialDelay(TimeSpan.FromSeconds(1))
            .WithCircuitBreaker(cb => cb
                .WithFailureRatio(0.5)
                .WithMinimumCalls(10)
                .WithBreakDuration(TimeSpan.FromMinutes(1)))
            .WithTimeout(TimeSpan.FromSeconds(30))
            .WithShouldRetry((_, ex) =>
                ex is HttpRequestException or TaskCanceledException)
            .Build();
    }

    public async Task<T> GetAsync<T>(string url)
    {
        return await _policy.ExecuteAsync(async ct =>
        {
            var response = await _client.GetAsync(url, ct);
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadFromJsonAsync<T>(ct);
        });
    }
}

Database Operations

public class ResilientRepository
{
    private readonly IResiliencePolicy _policy;

    public ResilientRepository()
    {
        _policy = new ResiliencePolicyBuilder()
            .WithMaxAttempts(3)
            .WithLinearDelay(TimeSpan.FromMilliseconds(100))
            .WithShouldRetry((_, ex) => IsTransientDbException(ex))
            .Build();
    }

    public async Task<User> GetUserAsync(int id)
    {
        return await _policy.ExecuteAsync(async ct =>
        {
            return await _context.Users.FindAsync(id, ct);
        });
    }

    private bool IsTransientDbException(Exception ex)
    {
        return ex is DbUpdateException or TimeoutException;
    }
}

Graceful Degradation

public class CatalogService
{
    private readonly IResiliencePolicy _policy;
    private readonly ICacheClient _cache;

    public async Task<Product> GetProductAsync(int id)
    {
        try
        {
            return await _policy.ExecuteAsync(async ct =>
            {
                return await _api.GetProductAsync(id, ct);
            });
        }
        catch (BrokenCircuitException)
        {
            // Fall back to cached data when circuit is open
            var cached = await _cache.GetAsync<Product>($"product:{id}");
            if (cached.HasValue)
            {
                _logger.LogWarning("Returning cached product {Id}", id);
                return cached.Value;
            }
            throw;
        }
    }
}

Retry with Logging

var policy = new ResiliencePolicy
{
    MaxAttempts = 5,
    GetDelay = ResiliencePolicy.ExponentialDelay(TimeSpan.FromSeconds(1)),
    Logger = loggerFactory.CreateLogger("Resilience")
};

// Logs will include attempt number, delay, and exception details
await policy.ExecuteAsync(async ct =>
{
    await UnreliableOperationAsync(ct);
});

Integration with Foundatio

Cache with Resilience

var resilientCache = new ResilientCacheClient(
    new RedisCacheClient(...),
    new ResiliencePolicyBuilder()
        .WithMaxAttempts(3)
        .WithExponentialDelay(TimeSpan.FromMilliseconds(100))
        .Build()
);

Queue with Resilience

public class ResilientQueueProcessor
{
    private readonly IQueue<WorkItem> _queue;
    private readonly IResiliencePolicy _policy;

    public async Task ProcessAsync(WorkItem item)
    {
        await _policy.ExecuteAsync(async ct =>
        {
            await DoProcessingAsync(item, ct);
        });
    }
}

Dependency Injection

Register Policy Provider

services.AddSingleton<IResiliencePolicyProvider>(sp =>
{
    var logger = sp.GetRequiredService<ILoggerFactory>().CreateLogger("Resilience");

    return new ResiliencePolicyProviderBuilder()
        .WithDefaultPolicy(b => b
            .WithMaxAttempts(3)
            .WithExponentialDelay(TimeSpan.FromSeconds(1))
            .WithLogger(logger))
        .WithPolicy("http", b => b
            .WithMaxAttempts(5)
            .WithCircuitBreaker()
            .WithTimeout(TimeSpan.FromSeconds(30)))
        .Build();
});

Use in Services

public class MyService
{
    private readonly IResiliencePolicy _policy;

    public MyService(IResiliencePolicyProvider policyProvider)
    {
        _policy = policyProvider.GetPolicy("http");
    }
}

Best Practices

1. Use Appropriate Timeouts

// Match timeout to operation type
.WithTimeout(TimeSpan.FromSeconds(5))   // Fast operations
.WithTimeout(TimeSpan.FromSeconds(30))  // API calls
.WithTimeout(TimeSpan.FromMinutes(5))   // Long operations

2. Configure Circuit Breaker Thresholds

.WithCircuitBreaker(cb => cb
    // High traffic: needs more samples
    .WithMinimumCalls(100)
    .WithFailureRatio(0.5)

    // Low traffic: fewer samples
    .WithMinimumCalls(10)
    .WithFailureRatio(0.3)
)

3. Use Jitter to Prevent Thundering Herd

.WithExponentialDelay(TimeSpan.FromSeconds(1))
.WithJitter()  // Adds randomness to prevent synchronized retries

4. Handle Specific Exceptions

.WithShouldRetry((attempt, ex) =>
{
    // Only retry transient failures
    return ex is HttpRequestException
        or TimeoutException
        or SocketException;
})
.WithUnhandledException<OperationCanceledException>()
.WithUnhandledException<ArgumentException>()

5. Log Retry Attempts

var policy = new ResiliencePolicy
{
    Logger = loggerFactory.CreateLogger("Resilience"),
    MaxAttempts = 5
};
// Automatically logs each retry attempt

Next Steps

• Caching - Combine with cache fallbacks
• Queues - Resilient queue processing
• Jobs - Retry job execution