[API Proposal]: Allow `HybridCache.SetAsync` to propagate L2 write failures

[samyonr]

Background and motivation

I run a multi-replica .NET 10 service using Microsoft.Extensions.Caching.Hybrid version 10.8.0 with Redis as the distributed cache (L2).

In the current setup, I've intentionally disabled the local cache using DisableLocalCache. The cached values are internal scripts that must be shared consistently between replicas. The goal is to minimize inconsistency: all replicas of my service should see the same version of every script. In my case, it's not a hard requirement, so minimal inconsistency is still allowed: in-flight requests may still use different versions of the script, but except that, this inconsistency gap should be reduced to the minimum. Therefore, Redis is used as a distributed-cache and coordination unit. On any incoming requests to one of the replicas, the services fetch Redis for the latest version of the script. No L1 cache is used, because that will increase that inconsistency gap, and Redis is fast enough to fetch the script on every service request.

Normal reads use GetOrCreateAsync. Separately, a rare metadata-update operation (script updates, done via service's API) commits metadata to persistent storage and then calls SetAsync to proactively publish the corresponding script to Redis. I've could remove it from Redis after committing to the persistent storage, but to avoid a cache-miss, and taking into account the infrequent scripts-updates, SetAsync works better here (no two competing updates are excepted).

The metadata store and Redis are not transactional. Therefore, if the metadata commit succeeds but the Redis write fails, the service needs to report that partial failure to the caller so the operation can be retried safely (or retry it automatically).

The current implementation absorbs L2 write exceptions, only logging them without propagating them:

extensions/src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/DefaultHybridCache.StampedeStateT.cs

Lines 365 to 381 in 8f88b00

	// We already have the payload serialized, so this is trivial to do.
	try
	{
	await Cache.SetL2Async(Key.Key, cacheItem, in buffer, _options, SharedToken).ConfigureAwait(false);
	if (eventSourceEnabled)
	{
	HybridCacheEventSource.Log.DistributedCacheWrite();
	}
	}
	catch (Exception ex)
	{
	// log the L2 write failure, but that doesn't need to interrupt the app flow (so:
	// don't rethrow); L1 will still reduce impact, and L1 without L2 is better than
	// hard failure every time
	Cache._logger.CacheBackendWriteFailure(ex);
	}

Consequently, awaiting SetAsync reports success even when the distributed cache writes failed.

The current implementation explains this as acceptable because L1 can still reduce the impact. That is reasonable for best-effort caching, but it does not apply when DisableLocalCache is used and Redis is the only cache tier.

My current workaround is a read-after-write followed by a value comparison or removing from Redis and expecting a cache-miss. The additional Redis read is tolerable because this operation is rare, but it should not be necessary to determine whether an explicit SetAsync succeeded. It also cannot prove that the TTL was refreshed when an identical value already existed. That TTL caveat is not itself a content-correctness problem for me, but it demonstrates that read-after-write is not equivalent to write acknowledgement.

API Proposal

namespace Microsoft.Extensions.Caching.Hybrid;

public class HybridCacheOptions
{
    public bool ThrowOnDistributedCacheWriteFailure { get; set; }
}

Not sure if throwing is the right approach here, or returning a Boolean or some result type, but it seems like throwing an exception is the current default. If that's the case, I would add an option similar to the above.

The default would remain false for compatibility.

When enabled, at minimum an explicit SetAsync should not complete successfully until its L2 write has completed. If the distributed cache write fails, the returned ValueTask should complete with the underlying or an appropriate wrapping exception.

API Usage

services.AddStackExchangeRedisCache(options =>
{
    // Redis configuration
});

services.AddHybridCache(options =>
{
    options.ThrowOnDistributedCacheWriteFailure = true;
});

await cache.SetAsync(
    key,
    script,
    new HybridCacheEntryOptions
    {
        Flags = HybridCacheEntryFlags.DisableLocalCache
    });

Alternative Designs

SetAsync could always propagate distributed cache write failures. That arguably provides the clearest semantics for an explicit write, but it would be a behavioral breaking change.

An opt-in option preserves the existing best-effort behavior while supporting applications that require write acknowledgement.

Another option is to return a result type or add a new SetAsync that throws or returns a Boolean or result type.

Risks

A breaking change could affect existing users. A new config adds complexity. I think that the added complexity in this case is minimal and it's preferable over a breaking change or a new function.