Introduction

If you use Redis in a shared environment — whether an instance used by multiple microservices, multiple teams or multiple tenants — a single big key can degrade the performance of every other consumer at the same time. This is no exaggeration: it is how Redis works internally, and ignoring this behavior in production is expensive.

The problem of big keys in Redis exists because the server is single-threaded for data operations. When a command accesses or manipulates a key containing hundreds of megabytes — be it a list with 500 thousand elements, a hash with 200 thousand fields or a 50 MB binary string — the entire event loop is blocked processing that operation. All other clients, from completely different services, wait in the queue. In shared environments this is fatal.

In this article I will show what characterizes a big key in Redis, why the impact is amplified in multi-tenant environments, how to detect big keys in production, and the concrete strategies to eliminate or mitigate the problem — with C# examples using StackExchange.Redis.

What Is a “Big Key” in Redis

Redis has no fixed limit that defines when a key is “big”. The community and the official documentation use the term big key to describe keys whose processing or memory cost is disproportionate to the overall workload of the instance.

In practice, the limits that usually come up as a reference are:

Data TypeConsidered “big” from
String / Binary> 5 MB
List> 10,000 elements or > 10 MB
Hash> 10,000 fields
Set> 10,000 members
Sorted Set> 10,000 members
Stream> 10,000 pending messages

These numbers are not absolute — they depend on the hardware, the Redis version and the access pattern. But they are solid references for monitoring alerts.

ℹ️ Info: Redis 7.0+ introduced improvements to internal data structures (such as Listpack replacing ZipList), which slightly changed the thresholds at which structures are “promoted” to larger representations. The big key concept, however, remains the same.

Difference between a big key and a hot key

It is common to confuse a big key with a hot key. They are different problems:

  • Big key: a key with a large volume of data — the problem is the CPU/memory cost of processing the data.
  • Hot key: a key accessed at a very high frequency — the problem is contention on the hash slot (relevant in Redis Cluster).

A key can have both problems at the same time, but the solutions are distinct. This article focuses on big keys.

How Redis Processes Commands (and why it matters)

To understand the impact of big keys, you need to understand how Redis executes commands. Redis uses a single-threaded event loop to process data operations. This means:

  1. The server receives a command from a client.
  2. The event loop executes the command to completion before processing any other.
  3. Only then is the next command in the queue processed.

This architecture is one of the reasons Redis is so fast for simple operations: no locks, no context switching between threads. The cost is that a slow operation blocks everything.

1
2
3
4
Client A: GET user:123          → response in 0.1ms ✅
Client B: LRANGE big-list 0 -1  → 2.3s processing 500k elements ⚠️
Client C: SET config:timeout 30  → waiting in the queue... ⌛
Client D: GET session:abc123     → waiting in the queue... ⌛

⚠️ Warning: Commands such as LRANGE key 0 -1, SMEMBERS key, HGETALL key and KEYS * are O(N) — the cost grows linearly with the size of the structure. On a key with 500 thousand elements, this can take seconds.

Redis I/O is non-blocking — but processing is not

It is important to separate two concepts:

  • I/O: Redis uses I/O multiplexing (epoll/kqueue), so accepting connections and reading network data is non-blocking.
  • Command processing: executing the command itself is single-threaded and blocking for the other clients.

Redis 6.0 introduced I/O threads for network reads and writes, but command execution is still single-threaded on the main thread. The big key problem was not solved by this change.

Impact on Shared Environments

In an environment where a single Redis instance serves multiple microservices or multiple tenants of a SaaS, the impact of a big key is amplified because:

1. Latency affects everyone, not just the culprit

If the reporting service reads a key with 300 thousand records every hour, it introduces latency spikes for the authentication service, the notification service and any other that shares the same instance — even if those services are working perfectly.

2. Memory consumption is contended

Redis keeps all data in memory. A big key that takes 500 MB squeezes the space available to all other tenants. When Redis reaches maxmemory, it starts the eviction policy (removing keys), which can discard critical data from other services.

3. Snapshot and replication operations get slower

When Redis generates an RDB snapshot (persistence) or replicates data to a replica, big keys increase the fork() time and the file size. In cloud instances (AWS ElastiCache, Azure Cache for Redis), this can cause replication failures or performance penalties during synchronization.

Diagram of the impact of big keys in a shared Redis environment with multiple blocked services

4. Difficult debugging and traceability

In a shared environment, identifying who created the big key and which service is responsible requires careful instrumentation. Without proper key namespacing, tracing the origin of the problem can take hours.

How to Detect Big Keys in Production

Redis offers native tools to detect big keys. Here are the main approaches:

redis-cli –bigkeys

The simplest command for an initial scan:

1
2
3
4
5
# Full scan — use with care in production (it can be slow)
redis-cli --bigkeys

# With a specific host and port
redis-cli -h redis.example.com -p 6379 --bigkeys

The output shows the largest keys by data type. The problem is that this command uses SCAN internally and can take a while on large instances. In production, prefer to run it during low-load hours.

MEMORY USAGE — spot check

To check the size of a specific key:

1
2
3
4
5
# Returns the size in bytes, including metadata overhead
MEMORY USAGE minha-chave

# With nested sampling (for complex structures)
MEMORY USAGE minha-chave SAMPLES 10

SCAN + MEMORY USAGE — automated scan in C#

For programmatic monitoring with StackExchange.Redis:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public class RedisBigKeyScanner
{
    private const long LimiarBigKeyBytes = 10 * 1024 * 1024; // 10 MB
    private readonly IDatabase _db;
    private readonly IServer _server;
    private readonly ILogger<RedisBigKeyScanner> _logger;

    public RedisBigKeyScanner(IConnectionMultiplexer redis, ILogger<RedisBigKeyScanner> logger)
    {
        _db = redis.GetDatabase();
        _server = redis.GetServer(redis.GetEndPoints().First());
        _logger = logger;
    }

    public async Task VarrerBigKeysAsync(string padrao = "*", int tamanhoPagina = 100,
        CancellationToken ct = default)
    {
        // SCAN is non-blocking — it iterates in batches (tamanhoPagina at a time)
        await foreach (var key in _server.KeysAsync(pattern: padrao, pageSize: tamanhoPagina)
                                         .WithCancellation(ct))
        {
            var sizeBytes = await _db.ExecuteAsync("MEMORY", "USAGE", key, "SAMPLES", "5");
            if (sizeBytes.IsNull) continue;

            var size = (long)sizeBytes;
            if (size >= LimiarBigKeyBytes)
                _logger.LogWarning("Big key: {Key} | {SizeMB:F2} MB", key, size / (1024.0 * 1024.0));
        }
    }
}

📂 Source Code: The complete example is available in the blog’s samples repository: BlogSamples/Cache/Redis/

📝 Example: In an environment with 2 million keys, a scan with pageSize = 500 and pattern = "relatorio:*" limits the scope to the problematic namespace only, reducing scan time from hours to minutes.

Redis Latency Monitoring

To detect the impact in real time, enable latency monitoring:

1
2
3
4
5
6
# Enable in redis.conf or via CONFIG SET
CONFIG SET latency-monitor-threshold 100

# View latency events
LATENCY LATEST
LATENCY HISTORY command

Strategies to Avoid Big Keys

Detecting the problem is the first step. Solving it is more interesting. There are four main approaches:

1. Data fragmentation (Sharding)

Instead of a single big key, distribute the data across multiple smaller keys using a partitioning field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
public class RedisHashSharded
{
    private readonly IDatabase _db;
    private const int QuantidadeShards = 16; // power of 2 for uniform distribution

    public RedisHashSharded(IConnectionMultiplexer redis) => _db = redis.GetDatabase();

    private string ObterChaveShard(string chaveBase, string campo)
    {
        var indiceShard = Math.Abs(campo.GetHashCode()) % QuantidadeShards;
        return $"{chaveBase}:shard:{indiceShard}";
    }

    public async Task GravarAsync(string chaveBase, string campo, string valor)
    {
        var chaveShard = ObterChaveShard(chaveBase, campo);
        await _db.HashSetAsync(chaveShard, campo, valor);
    }

    public async Task<string?> LerAsync(string chaveBase, string campo)
    {
        var chaveShard = ObterChaveShard(chaveBase, campo);
        var resultado = await _db.HashGetAsync(chaveShard, campo);
        return resultado.HasValue ? resultado.ToString() : null;
    }

    // LerTodosAsync parallelizes reading the QuantidadeShards shards — full code in the repository
}

2. Efficient serialization with compression

One of the biggest culprits for large strings is naive serialization of complex objects. Swapping JSON for a binary format + compression dramatically reduces the size:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
public static class RedisSerializerExtensions
{
    public static async Task GravarComprimidoAsync<T>(
        this IDatabase db, string chave, T valor, TimeSpan? expiracao = null)
    {
        using var stream = new MemoryStream();
        await using (var gzip = new GZipStream(stream, CompressionLevel.Optimal))
            await JsonSerializer.SerializeAsync(gzip, valor);

        await db.StringSetAsync(chave, stream.ToArray(), expiracao);
    }

    public static async Task<T?> LerComprimidoAsync<T>(this IDatabase db, string chave)
    {
        var valor = await db.StringGetAsync(chave);
        if (!valor.HasValue) return default;

        using var stream = new MemoryStream((byte[])valor!);
        await using var gzip = new GZipStream(stream, CompressionMode.Decompress);
        return await JsonSerializer.DeserializeAsync<T>(gzip);
    }
}

💡 Tip: Typical GZip compression reduces JSON payloads by 60–80%. A 5 MB object can drop to 800 KB — which moves it from the “big key” classification to a manageable size. But be careful: compression has a CPU cost, so evaluate the trade-off for frequently accessed data.

3. Mandatory TTL for cache keys

Keys without TTL in shared environments tend to accumulate data indefinitely. Implement a mandatory TTL policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
public class RedisCacheComPoliticaTtl
{
    private readonly IDatabase _db;
    private static readonly TimeSpan TtlPadrao = TimeSpan.FromHours(1);
    private static readonly TimeSpan TtlMaximo = TimeSpan.FromHours(24);

    public RedisCacheComPoliticaTtl(IConnectionMultiplexer redis)
        => _db = redis.GetDatabase();

    public async Task GravarAsync(string chave, string valor, TimeSpan? ttl = null)
    {
        // Never allows a null TTL — uses the default if not specified
        var ttlEfetivo = ttl.HasValue
            ? TimeSpan.FromTicks(Math.Min(ttl.Value.Ticks, TtlMaximo.Ticks))
            : TtlPadrao;

        await _db.StringSetAsync(chave, valor, ttlEfetivo);
    }
}

4. Namespacing and per-tenant segregation

In multi-tenant environments, use mandatory prefixes to identify the origin of keys. This makes diagnosis easier and enables selective cleanup policies:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
// Key pattern: {ambiente}:{tenant}:{dominio}:{entidade}:{id}
// Example: prod:acme-corp:relatorio:mensal:2026-05

public static class RedisKeyBuilder
{
    public static string Construir(
        string tenant,
        string dominio,
        string entidade,
        string id,
        string ambiente = "prod")
    {
        return $"{ambiente}:{Sanitizar(tenant)}:{Sanitizar(dominio)}:{Sanitizar(entidade)}:{id}";
    }

    private static string Sanitizar(string segmento) =>
        segmento.ToLowerInvariant().Replace(" ", "-").Replace(":", "");
}

Practical Example

I will show a real scenario: a reporting service that loaded all of the month’s transaction records into a single Redis key for “caching” — and was bringing down the performance of the other services in the same cluster.

The original problem

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// ❌ PROBLEMATIC CODE — DO NOT DO THIS
public async Task<List<Transacao>> ObterTransacoesMensaisAsync(
    int clienteId,
    int mes,
    int ano)
{
    var cacheKey = $"transacoes:{clienteId}:{mes}/{ano}";

    var cached = await _cache.StringGetAsync(cacheKey);
    if (cached.HasValue)
    {
        // Deserializes 300,000 transactions into memory at once
        return JsonSerializer.Deserialize<List<Transacao>>(cached!)!;
    }

    // Fetches 300,000 records from the database
    var transacoes = await _db.Transacoes
        .Where(t => t.ClienteId == clienteId
                 && t.Data.Month == mes
                 && t.Data.Year == ano)
        .ToListAsync();

    // Serializes 300,000 objects into a ~80 MB JSON and saves it in Redis
    // No TTL → it stays forever
    await _cache.StringSetAsync(
        cacheKey,
        JsonSerializer.Serialize(transacoes));

    return transacoes;
}

This code creates a key that can reach 80 MB in Redis, without TTL, and it is read in full on every access. Each read blocks the event loop for hundreds of milliseconds.

The refactored solution

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// ✅ CORRECT SOLUTION — Paginated cache with mandatory TTL

public class TransacaoCache
{
    private readonly IDatabase _redis;
    private const int TamanhoPagina = 100;
    private const string PrefixoChave = "tx";

    public TransacaoCache(IConnectionMultiplexer redis) => _redis = redis.GetDatabase();

    public async Task GravarPaginaAsync(int clienteId, int mes, int ano, int pagina,
        IEnumerable<Transacao> transacoes)
    {
        var chave = ConstruirChavePagina(clienteId, mes, ano, pagina);
        var json = JsonSerializer.Serialize(transacoes.Take(TamanhoPagina));
        // 30-minute TTL — report data does not need to be eternal
        await _redis.StringSetAsync(chave, json, TimeSpan.FromMinutes(30));
    }

    public async Task<List<Transacao>?> ObterPaginaAsync(int clienteId, int mes, int ano, int pagina)
    {
        var chave = ConstruirChavePagina(clienteId, mes, ano, pagina);
        var cache = await _redis.StringGetAsync(chave);
        return cache.HasValue ? JsonSerializer.Deserialize<List<Transacao>>((string)cache!) : null;
    }

    // Each key contains only TamanhoPagina transactions (~30 KB)
    private static string ConstruirChavePagina(int clienteId, int mes, int ano, int pagina) =>
        $"{PrefixoChave}:{clienteId}:{mes}-{ano}:p{pagina}";
}

With this approach, each key goes from 80 MB to ~30 KB. The event loop is released in microseconds, and the other shared services stop suffering latency because of the reporting service.

Tips and Good Practices

  • Define alert thresholds in CI/CD: before a deploy, check whether new key patterns respect the size limits. A load-testing pipeline with redis-cli --bigkeys can prevent the problem before it reaches production.

  • Use TTL on absolutely every key: in shared environments, a key without TTL is a key that will grow indefinitely. Configure maxmemory-policy allkeys-lru in Redis so that, even if some key escapes without TTL, Redis can discard it when necessary.

  • Never use KEYS * in production: the KEYS command blocks Redis while it scans the entire keyspace. Always use SCAN with MATCH and COUNT for iterative, non-blocking scans. StackExchange.Redis exposes this behavior through IServer.KeysAsync().

  • Monitor redis-cli --latency-history: recurring latencies above 50ms are a sign of big keys or hot keys. Configure latency alerts in Redis and correlate them with the key patterns being accessed at that moment.

  • Separate workloads into distinct instances when possible: if a reporting service inevitably needs big keys, isolate it in a dedicated Redis instance. The cost of an extra instance is much lower than the latency impact on the critical services of a shared environment.

  • Prefer native Redis structures over large JSON strings: a HASH with 1,000 fields is more efficient than a STRING with a 1,000-property JSON. Redis can access individual hash fields without deserializing the whole structure, saving CPU and memory.

  • Review the design when using Redis as a database: Redis is excellent as a short-lived cache, for pub/sub and counters. When it starts receiving complete entity collections for “temporary persistence”, that is a sign the design needs review — a relational or document database is probably the right tool for that data.

Objective Summary

  • Big key in Redis — a key whose data volume is disproportionate to the workload; practical limits are 5 MB for strings and 10,000 elements for lists/hashes/sets. Beyond these values, O(N) operations block the event loop for hundreds of milliseconds.
  • Single-threaded event loop — Redis processes one command at a time on the main thread (even in Redis 6.0+ with I/O threads). A big key blocks every other client of the instance while it is being processed.
  • Shared environments — in Redis instances used by multiple microservices or tenants, a big key created by a single service introduces latency for all others, regardless of their behavior.
  • Dangerous O(N) commands — LRANGE key 0 -1, SMEMBERS key, HGETALL key, KEYS * and SUNIONSTORE are the main blocking vectors when applied to big keys. Prefer paginated variants such as HSCAN, SSCAN and ZSCAN.
  • Detection in production — redis-cli --bigkeys performs a scan by type; MEMORY USAGE key SAMPLES N returns the exact size in bytes including overhead; LATENCY LATEST identifies commands with anomalous latency.
  • Fragmentation as a solution — splitting a big key into multiple smaller keys using per-field sharding eliminates blocking and enables parallel reads. A hash with 100,000 fields can be split into 16 shards of ~6,250 fields each.
  • Mandatory TTL — every key in a shared environment must have an expiration. The absence of TTL is the main cause of big key accumulation in production. Configure maxmemory-policy allkeys-lru as a safety net.

Read Also

References