Service Discovery and Service Mesh in .NET — Part 2: Four Ways to Implement Discovery

Part 2 of a six-part series on service discovery and service mesh architecture for .NET engineers.

Part 1 established what service discovery is and why it becomes necessary the moment a system stops being a single process. This part is about writing the code.

There are four approaches worth knowing in .NET, and they are not competitors in a simple sense. They sit at different levels of the stack and solve overlapping but distinct portions of the problem. A team using Kubernetes DNS is not doing something inferior to a team using Consul; they are making a different trade between simplicity and control. The goal here is to understand each one well enough to make that trade deliberately.

We will build up in this order:

1. Kubernetes DNS — the baseline that most teams already have and often misuse
2. Microsoft.Extensions.ServiceDiscovery — the framework-native abstraction
3. Consul — a purpose-built registry with metadata and watches
4. Dapr service invocation — pushing the problem out of the process entirely

Then we will look at how resilience fits in, because there is an ordering detail there that silently breaks the thing everyone assumes it fixes.

Approach 1: Kubernetes DNS

If your services run on Kubernetes, you already have service discovery. A Service resource creates a DNS name, and the platform keeps the records behind that name accurate. No library, no registration code, nothing in Program.cs.

apiVersion: v1
kind: Service
metadata:
  name: inventory-service
  namespace: shop
spec:
  selector:
    app: inventory
  ports:
    - name: http
      port: 8080
      targetPort: 8080

From another pod in the same namespace, http://inventory-service:8080 now resolves. From a different namespace, http://inventory-service.shop.svc.cluster.local:8080.

The .NET side looks like nothing at all:

builder.Services.AddHttpClient<InventoryClient>(client =>
{
    client.BaseAddress = new Uri("http://inventory-service:8080");
});

This is genuinely good. It is portable across any Kubernetes distribution, it costs nothing to operate, and it survives every pod rescheduling event without application involvement.

The part that goes wrong

Part 1 introduced the connection pooling problem. It is worth handling properly here, because the default configuration is wrong for this environment and the failure mode is subtle.

SocketsHttpHandler pools TCP connections and reuses them. Once a connection to a pod IP is established, that connection is used for subsequent requests regardless of what DNS now says. IHttpClientFactory rotates handlers every two minutes by default, which mitigates this, but the rotation applies to the handler, not to connections held by a handler that is still in use.

The reliable fix is to bound connection lifetime directly:

builder.Services.AddHttpClient<InventoryClient>(client =>
{
    client.BaseAddress = new Uri("http://inventory-service:8080");
})
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
    // Recycle connections so DNS is re-resolved. Without this, a long-lived
    // client can keep talking to pods that were removed hours ago.
    PooledConnectionLifetime  = TimeSpan.FromMinutes(2),
    PooledConnectionIdleTimeout = TimeSpan.FromSeconds(30),

    // Bound the connection pool per server so one slow dependency
    // cannot exhaust sockets for everything else.
    MaxConnectionsPerServer = 100
})
// The handler lifetime no longer needs to be short, because
// PooledConnectionLifetime is doing the work.
.SetHandlerLifetime(Timeout.InfiniteTimeSpan);

Setting SetHandlerLifetime(Timeout.InfiniteTimeSpan) alongside PooledConnectionLifetime is the pattern the .NET team recommends. Handler rotation exists specifically to work around stale DNS; once PooledConnectionLifetime handles that, rotation only costs you connection churn.

There is a second issue that catches teams using gRPC. HTTP/2 multiplexes many requests over a single connection, which means a gRPC client will typically hold exactly one connection to exactly one pod, forever. Every request from that client goes to the same backend regardless of how many replicas exist. kube-proxy load balances connections, not requests, so it cannot help.

For gRPC on Kubernetes you need either client-side load balancing over a headless Service, or a proxy that understands HTTP/2. Grpc.Net.Client supports the former:

var channel = GrpcChannel.ForAddress(
    "dns:///inventory-service-headless.shop.svc.cluster.local:8080",
    new GrpcChannelOptions
    {
        Credentials = ChannelCredentials.Insecure,
        ServiceConfig = new ServiceConfig
        {
            // Resolve all pod IPs and balance requests across them.
            LoadBalancingConfigs = { new RoundRobinConfig() }
        }
    });

The dns:/// scheme tells the channel to use the DNS resolver rather than treating the string as a single host. Pointed at a headless Service, it receives every pod IP and distributes requests across all of them.

Where DNS runs out

DNS gives you a name and an address. That is the entire interface. You cannot ask it which instances are running version 2.2, which ones are in your availability zone, or which ones are currently under load. If you need routing decisions based on anything other than "give me an address for this name", DNS cannot express the question.

You also cannot easily run the same code outside Kubernetes. A developer running three services on their laptop has no cluster DNS, so http://inventory-service:8080 resolves to nothing. Teams usually solve this with a parallel set of configuration for local development, which works but means the resolution mechanism differs between environments — and differences between environments are where surprises live.

That second problem is exactly what the next approach was built for.

Approach 2: Microsoft.Extensions.ServiceDiscovery

This package came out of the .NET Aspire work and has since become a general-purpose abstraction. It is worth understanding even if you never touch Aspire, because it decouples how a service is named in code from how that name is resolved at runtime.

dotnet add package Microsoft.Extensions.ServiceDiscovery

builder.Services.AddServiceDiscovery();

builder.Services.ConfigureHttpClientDefaults(http =>
{
    http.AddServiceDiscovery();
});

// The URI uses a logical name, not a host.
builder.Services.AddHttpClient<InventoryClient>(client =>
{
    client.BaseAddress = new Uri("https://inventory");
});

https://inventory is not a hostname. It is a logical service name that gets resolved by whichever provider is registered. The application code never changes; the resolution strategy changes per environment.

Providers

Resolution is handled by a chain of providers, tried in order. Each one is skipped if a previous provider already produced endpoints.

Configuration provider — reads from IConfiguration, which means appsettings.json, environment variables, Azure App Configuration, or anything else in the configuration system. This is the default.

{
  "Services": {
    "inventory": {
      "https": [ "https://localhost:7145" ]
    },
    "tax": {
      "https": [ "https://localhost:7203" ]
    }
  }
}

That is the local development story solved. The same https://inventory in code resolves to localhost:7145 on a developer machine.

DNS and DNS SRV providers — for containerised environments.

dotnet add package Microsoft.Extensions.ServiceDiscovery.Dns

builder.Services.AddServiceDiscoveryCore();
builder.Services.AddDnsSrvServiceEndpointProvider();

The SRV variant is the interesting one on Kubernetes, because SRV records carry port information as well as host, which allows a single service to expose multiple named endpoints. A Service with two named ports produces two resolvable endpoints:

apiVersion: v1
kind: Service
metadata:
  name: basket
spec:
  clusterIP: None          # headless, so SRV records are created per pod
  selector:
    app: basket
  ports:
    - name: default
      port: 8080
    - name: dashboard
      port: 9090

// Resolves to the "default" port
builder.Services.AddHttpClient<BasketClient>(
    client => client.BaseAddress = new Uri("https://basket"));

// Resolves to the "dashboard" port via the underscore prefix
builder.Services.AddHttpClient<BasketDashboardClient>(
    client => client.BaseAddress = new Uri("https://_dashboard.basket"));

The _dashboard.basket syntax follows the SRV record naming convention. It is a small feature but a genuinely useful one — it removes a whole category of "which port is the admin endpoint on again?" configuration.

Pass-through provider — resolves a name to itself, so https://inventory becomes a literal DNS lookup for the host inventory. It is registered by default and acts as the fallback when nothing else matched.

Resolving directly

Sometimes you need the endpoints rather than an HttpClient — for a gRPC channel, a message broker connection, or a custom protocol:

public class InventoryChannelFactory(ServiceEndpointResolver resolver)
{
    public async Task<GrpcChannel> CreateAsync(CancellationToken ct)
    {
        var endpoints = await resolver.GetEndpointsAsync("https://inventory", ct);

        // endpoints.Endpoints is a collection; pick according to your policy.
        var target = endpoints.Endpoints[Random.Shared.Next(endpoints.Endpoints.Count)];

        return GrpcChannel.ForAddress(target.EndPoint.ToString()!);
    }
}

ServiceEndpointCollection also exposes a ChangeToken, so you can subscribe to be notified when the endpoint set changes rather than polling. That matters for long-lived connections like gRPC channels or AMQP links, which will otherwise happily keep using an endpoint that has gone away.

The ordering problem

Here is the detail that this whole section exists for.

If you combine service discovery with the standard resilience handler — and you should — the order of registration determines whether retries actually help.

// WRONG. Retries will hit the same dead instance.
builder.Services.ConfigureHttpClientDefaults(http =>
{
    http.AddServiceDiscovery();
    http.AddStandardResilienceHandler();
});

// CORRECT. Each retry re-resolves and may pick a different instance.
builder.Services.ConfigureHttpClientDefaults(http =>
{
    http.AddStandardResilienceHandler();
    http.AddServiceDiscovery();
});

The reason is how DelegatingHandler pipelines nest. Handlers registered earlier sit closer to the outside of the pipeline, wrapping handlers registered later.

graph TB
    subgraph Wrong["Discovery registered first"]
        W1[Request] --> W2[Service discovery<br/>resolves once]
        W2 --> W3[Resilience handler<br/>retries here]
        W3 --> W4[Primary handler]
        W3 -.retry.-> W3
    end
    subgraph Right["Resilience registered first"]
        R1[Request] --> R2[Resilience handler<br/>retries here]
        R2 --> R3[Service discovery<br/>resolves per attempt]
        R3 --> R4[Primary handler]
        R2 -.retry re-enters discovery.-> R3
    end

In the first arrangement, resolution happens once on the way in. The resilience handler then retries against an already-resolved, concrete address. If that instance is the one that is down, all three retries go to the same dead pod and all three fail.

In the second, the resilience handler is outermost. Each retry re-enters the discovery handler, which resolves again and can return a different instance. That is the behaviour everyone assumes they are getting.

This is a two-line difference that produces completely different availability characteristics, and nothing warns you about it. It is worth checking in any existing codebase that uses both.

Approach 3: Consul

Consul is a dedicated service registry. Compared to DNS it gives you structured metadata, health checking with configurable semantics, and watch-based change notification. Compared to Microsoft.Extensions.ServiceDiscovery it gives you a registry that lives outside the application and can be shared across languages.

The trade is that it is another piece of infrastructure to run and another dependency in every service.

Registration

Part 1 sketched this; here is the fuller version with the details that matter in production.

public sealed class ConsulRegistration : IHostedService
{
    private readonly IConsulClient _consul;
    private readonly ConsulOptions _options;
    private readonly ILogger<ConsulRegistration> _logger;
    private readonly string _instanceId;

    public ConsulRegistration(
        IConsulClient consul,
        IOptions<ConsulOptions> options,
        ILogger<ConsulRegistration> logger)
    {
        _consul     = consul;
        _options    = options.Value;
        _logger     = logger;
        // Stable per-instance identity. On Kubernetes, the pod name works well.
        _instanceId = $"{_options.ServiceName}-{Environment.MachineName}-{Guid.NewGuid():N}"[..48];
    }

    public async Task StartAsync(CancellationToken ct)
    {
        var registration = new AgentServiceRegistration
        {
            ID      = _instanceId,
            Name    = _options.ServiceName,
            Address = _options.Address,
            Port    = _options.Port,

            // Metadata is the reason to choose Consul over plain DNS.
            Meta = new Dictionary<string, string>
            {
                ["version"]  = _options.Version,
                ["zone"]     = _options.Zone,
                ["protocol"] = "http"
            },

            Tags = [ $"version-{_options.Version}", _options.Zone ],

            Check = new AgentServiceCheck
            {
                HTTP     = $"http://{_options.Address}:{_options.Port}/health/ready",
                Interval = TimeSpan.FromSeconds(10),
                Timeout  = TimeSpan.FromSeconds(3),

                // If the check has been failing for a minute, remove the entry
                // entirely. This is the safety net for ungraceful shutdowns.
                DeregisterCriticalServiceAfter = TimeSpan.FromMinutes(1)
            }
        };

        await _consul.Agent.ServiceRegister(registration, ct);
        _logger.LogInformation("Registered {InstanceId} with Consul", _instanceId);
    }

    public async Task StopAsync(CancellationToken ct)
    {
        try
        {
            await _consul.Agent.ServiceDeregister(_instanceId, ct);
            _logger.LogInformation("Deregistered {InstanceId}", _instanceId);
        }
        catch (Exception ex)
        {
            // Never let deregistration failure block shutdown.
            // DeregisterCriticalServiceAfter will clean up.
            _logger.LogWarning(ex, "Failed to deregister {InstanceId}", _instanceId);
        }
    }
}

Two details deserve attention.

DeregisterCriticalServiceAfter is not optional. StopAsync runs only on graceful shutdown. A pod that is OOM-killed, a node that loses power, or a container that receives SIGKILL will never deregister, and without this setting the entry stays in the registry indefinitely. Setting it to a minute means the registry self-heals.

The try/catch in StopAsync matters more than it looks. If Consul is briefly unreachable during a rolling deployment, an unhandled exception in StopAsync can delay or block shutdown, which turns a routine deployment into a stuck one.

Resolution

Resolution is cleanest as a DelegatingHandler, because it slots into the existing HttpClient pipeline and inherits all the resilience machinery.

public sealed class ConsulResolvingHandler : DelegatingHandler
{
    private readonly IConsulClient _consul;
    private readonly IMemoryCache _cache;
    private readonly ILogger<ConsulResolvingHandler> _logger;

    // Short cache. Long enough to avoid hammering Consul,
    // short enough that a dead instance drops out quickly.
    private static readonly TimeSpan CacheDuration = TimeSpan.FromSeconds(10);

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken ct)
    {
        var serviceName = request.RequestUri!.Host;
        var instances   = await GetHealthyInstancesAsync(serviceName, ct);

        if (instances.Count == 0)
            throw new InvalidOperationException(
                $"No healthy instances registered for service '{serviceName}'.");

        var chosen = instances[Random.Shared.Next(instances.Count)];

        request.RequestUri = new UriBuilder(request.RequestUri)
        {
            Host = chosen.Service.Address,
            Port = chosen.Service.Port
        }.Uri;

        return await base.SendAsync(request, ct);
    }

    private async Task<IReadOnlyList<ServiceEntry>> GetHealthyInstancesAsync(
        string serviceName, CancellationToken ct)
    {
        if (_cache.TryGetValue(serviceName, out IReadOnlyList<ServiceEntry>? cached))
            return cached!;

        // passingOnly: true filters out instances whose health checks are failing.
        var result = await _consul.Health.Service(
            service: serviceName, tag: null, passingOnly: true, ct: ct);

        _cache.Set(serviceName, result.Response, CacheDuration);
        return result.Response;
    }
}

Registration:

builder.Services.AddSingleton<IConsulClient>(_ =>
    new ConsulClient(c => c.Address = new Uri(consulAddress)));

builder.Services.AddMemoryCache();
builder.Services.AddTransient<ConsulResolvingHandler>();
builder.Services.AddHostedService<ConsulRegistration>();

builder.Services.AddHttpClient<InventoryClient>(client =>
{
    // The host is the logical service name; the handler rewrites it.
    client.BaseAddress = new Uri("http://inventory");
})
.AddStandardResilienceHandler()          // outermost — retries re-resolve
.AddHttpMessageHandler<ConsulResolvingHandler>();

Note the ordering again. Same rule as before, same reason.

The random instance selection above is deliberately simple. Because Consul returns metadata, you can do considerably better:

// Prefer instances in the same availability zone, fall back to any.
var myZone = Environment.GetEnvironmentVariable("AVAILABILITY_ZONE");

var sameZone = instances
    .Where(i => i.Service.Meta.TryGetValue("zone", out var z) && z == myZone)
    .ToList();

var candidates = sameZone.Count > 0 ? sameZone : instances;
var chosen = candidates[Random.Shared.Next(candidates.Count)];

Zone-aware routing reduces both latency and cross-zone data transfer charges, which on a high-traffic system is a real line item. This is the kind of thing DNS simply cannot express, and it is the main reason to accept Consul's operational cost.

When Consul is worth it

If you are on Kubernetes and only need "find me a healthy instance of X", Consul is redundant — Kubernetes already does that. Consul earns its place when you have workloads outside Kubernetes that need to participate in the same registry, when you need metadata-driven routing, or when you want a registry that outlives any single orchestrator.

Approach 4: Dapr service invocation

Dapr takes a different position entirely. Instead of putting discovery logic in a library inside your process, it runs a sidecar next to your process and gives you a local HTTP or gRPC endpoint to call.

sequenceDiagram
    participant App as Checkout app
    participant SC1 as Dapr sidecar (checkout)
    participant SC2 as Dapr sidecar (inventory)
    participant Inv as Inventory app

    App->>SC1: POST localhost:3500/v1.0/invoke/inventory/method/reserve
    Note over SC1: Resolves "inventory" via<br/>name resolution component
    SC1->>SC2: mTLS request
    SC2->>Inv: POST localhost:8080/reserve
    Inv-->>SC2: 200 OK
    SC2-->>SC1: 200 OK
    SC1-->>App: 200 OK

The application talks to localhost. It never resolves anything.

builder.Services.AddDaprClient();

public class InventoryService(DaprClient dapr)
{
    public async Task<ReservationResult> ReserveAsync(ReservationRequest request)
    {
        // "inventory" is the Dapr app-id. No address anywhere.
        return await dapr.InvokeMethodAsync<ReservationRequest, ReservationResult>(
            HttpMethod.Post,
            appId: "inventory",
            methodName: "reserve",
            data: request);
    }
}

Or through a plain HttpClient if you prefer not to take the DaprClient dependency:

var client = DaprClient.CreateInvokeHttpClient(appId: "inventory");
var response = await client.PostAsJsonAsync("/reserve", request);

On Kubernetes, you opt in with annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inventory
spec:
  template:
    metadata:
      annotations:
        dapr.io/enabled:  "true"
        dapr.io/app-id:   "inventory"
        dapr.io/app-port: "8080"
    spec:
      containers:
        - name: inventory
          image: myregistry/inventory:2.1

What you get alongside discovery is significant: mutual TLS between sidecars by default, automatic retries, distributed tracing propagation, and access control policies — none of which required application code.

What you give up is a per-pod sidecar container consuming memory and CPU, an extra network hop on every call, and a new component in the failure domain. If the sidecar is unhealthy, the application cannot make outbound calls even though the application itself is fine.

If that description sounds like it is heading somewhere, it is. Dapr's service invocation is a sidecar-based approach to service-to-service communication, which is the defining characteristic of a service mesh. Part 3 picks up exactly here.

Where resilience fits

Every approach above returns an address that might not work. Part 1 made this point; here is what to do about it.

Microsoft.Extensions.Http.Resilience wraps Polly v8 in a package designed for HttpClient:

dotnet add package Microsoft.Extensions.Http.Resilience

builder.Services.AddHttpClient<InventoryClient>(client =>
{
    client.BaseAddress = new Uri("https://inventory");
})
.AddStandardResilienceHandler()
.AddServiceDiscovery();

AddStandardResilienceHandler assembles a pipeline with five stages, applied in this order:

The defaults are reasonable but generic. Two adjustments are worth making almost immediately.

.AddStandardResilienceHandler(options =>
{
    // Per-attempt timeout should reflect the dependency's actual p99,
    // not a round number. 10 seconds is far too long for a cache lookup.
    options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(3);

    // Total timeout must exceed AttemptTimeout by enough room for retries,
    // or the total budget expires before retries can run.
    options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(15);

    options.Retry.MaxRetryAttempts = 3;
    options.Retry.BackoffType      = DelayBackoffType.Exponential;
    options.Retry.UseJitter        = true;

    // Sampling duration must be at least twice the attempt timeout.
    options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
    options.CircuitBreaker.FailureRatio     = 0.2;
});

The relationship between these timeouts is validated at startup and will throw if inconsistent, which is a genuinely helpful design decision — it catches the classic mistake of setting a total timeout shorter than the sum of the attempts it is supposed to permit.

Retries and non-idempotent operations

One warning that applies regardless of which discovery approach you use. The standard handler retries on timeout, and a timeout does not tell you whether the request was processed.

If the inventory service received your reserve call, processed it, and then the response was lost, a retry reserves the stock twice. For GET requests this is harmless. For anything that mutates state it is a correctness bug that will appear under load and be very difficult to reproduce.

The answer is idempotency keys, not disabling retries:

public async Task<ReservationResult> ReserveAsync(ReservationRequest request)
{
    var message = new HttpRequestMessage(HttpMethod.Post, "/reserve")
    {
        Content = JsonContent.Create(request)
    };

    // Server deduplicates on this key, so a retry of the same logical
    // operation is safe even if the first attempt actually succeeded.
    message.Headers.Add("Idempotency-Key", request.ReservationId.ToString());

    var response = await _httpClient.SendAsync(message);
    response.EnsureSuccessStatusCode();
    return (await response.Content.ReadFromJsonAsync<ReservationResult>())!;
}

This requires cooperation from the service being called, which is why it belongs in your API design conventions rather than being bolted on later.

Comparing the four

A reasonable default

For a .NET team on Kubernetes with no unusual requirements, I would start here:

builder.Services.AddServiceDiscovery();

builder.Services.ConfigureHttpClientDefaults(http =>
{
    http.AddStandardResilienceHandler();
    http.AddServiceDiscovery();
});

Configuration provider in development, DNS SRV in the cluster, resilience handler outermost so retries re-resolve. No extra infrastructure, consistent code across environments, and the resolution mechanism is a configuration concern rather than a code concern.

Move to Consul when you need metadata-driven routing or have non-Kubernetes workloads in the same registry. Move to Dapr when you want mTLS, tracing, and retries handled outside the application — though at that point, it is worth asking whether a service mesh is the better answer, because a mesh does the same things without requiring an SDK.

What is still missing

Take stock of what the four approaches above have and have not solved.

Solved: finding a healthy instance, distributing load across instances, retrying transient failures, and preventing a failing dependency from consuming all your threads.

Not solved:

Traffic is unencrypted and unauthenticated. Any pod that can reach the network can call your services. Adding mTLS in application code means certificate provisioning, rotation, and validation in every service, in every language.

No traffic control. Sending five percent of requests to a canary, mirroring traffic to a staging deployment, or injecting a delay to test timeout handling all require code changes in every caller.

Observability is per-service. Each service can emit its own metrics and traces, but you have no consistent view of the call graph unless every service is instrumented identically. In practice they never are.

Policy is scattered. Retry configuration, timeout values, and circuit breaker thresholds live in application code across dozens of repositories. Changing a policy means a coordinated deployment.

Every one of these is solvable in application code. Teams do solve them in application code, and then discover that the fifth service was written by a different team with different defaults, and that the Python service does not have a Polly equivalent, and that changing a timeout requires eleven pull requests.

That accumulated cost is what motivated service meshes.

Coming up in Part 3

Part 3 introduces the service mesh properly:

• What a sidecar proxy actually does to a request, at the packet level
• mTLS with automatic certificate rotation, and why identity is the real feature
• Traffic management: canary releases, traffic mirroring, fault injection
• The sidecar model versus the sidecar-less ambient model, and why the industry is moving
• The honest cost — latency, memory, operational complexity — and how to decide whether you are ready
• When a mesh is the wrong answer, which is more often than mesh vendors suggest

This is Part 2 of a six-part series. Part 1 covered why service discovery exists. Part 4 covers implementing a mesh on cloud-native Kubernetes; Part 5 covers Azure-specific options; Part 6 provides a decision framework.