Solving CloudNativePG Admission Webhook Timeouts on GKE Autopilot
Tags: kubernetes gke cloudnativepg devops postgresql
Originally published on the IBM Community and republished here to make the content accessible to a wider technical audience.
Introduction
Admission webhooks are a critical component of Kubernetes operators. When a custom resource is created or modified, the Kubernetes API server communicates with the appropriate admission webhook to validate or mutate the request.
While deploying a PostgreSQL cluster using the CloudNativePG (CNPG) operator on Google Kubernetes Engine (GKE) Autopilot, we encountered an admission webhook timeout that prevented the PostgreSQL cluster from being created.
The issue was initially observed with EnterpriseDB (EDB) Postgres for Kubernetes, but our investigation showed that the problem was not specific to EDB or PostgreSQL.
Instead, the issue was related to how Kubernetes control-plane traffic interacts with restrictive NetworkPolicies in GKE Autopilot.
This article covers:
The symptoms we observed
How we investigated the issue
What we learned while working with Google Cloud Support
The root cause
The NetworkPolicy change that resolved the issue
Recommendations for troubleshooting similar webhook problems
Although CloudNativePG is used as the example, the same troubleshooting approach can apply to other Kubernetes operators and applications that rely on admission webhooks.
Environment
The issue was reproduced in the following environment:
- Google Kubernetes Engine (GKE) Autopilot
- Kubernetes v1.35
- CloudNativePG (CNPG) Operator
- Admission webhook enabled
- PostgreSQL deployment using CloudNativePG/EnterpriseDB
The Problem: Admission Webhook Timeout
During PostgreSQL deployment, the Kubernetes API server failed while invoking the CloudNativePG admission webhook.
A typical error looked like this:
failed calling webhook "mcluster.cnpg.io":
Post "https://cnpg-webhook-service.<namespace>.svc:443/...":
dial tcp <webhook-ip>:9443: i/o timeout
As a result, the PostgreSQL custom resource could not be successfully created.
At first glance, the error appeared to indicate a problem with the webhook itself.
However, the webhook Pod was healthy.
That led us to investigate the connectivity path between the Kubernetes API server and the webhook.
Who Can Be Affected?
This issue is not limited to a specific PostgreSQL distribution.
It can potentially affect:
- CloudNativePG Operator
- EnterpriseDB Postgres for Kubernetes
- Applications built on top of CloudNativePG
- Other Kubernetes workloads using admission webhooks
- Workloads protected by restrictive NetworkPolicies on GKE Autopilot
A useful symptom to look for is:
The webhook is healthy and reachable from application Pods, but the Kubernetes API server reports webhook timeout errors.
Initial Observations
We first verified the health of the components involved in the webhook communication.
The following components were working as expected:
- CloudNativePG webhook Pod was running
- Kubernetes Service was healthy
- DNS resolution was working
- Other Pods inside the cluster could connect successfully to the webhook
However, one important component could not communicate with the webhook:
The Kubernetes API server.
This changed the direction of our investigation.
Instead of focusing on the webhook implementation, we started investigating control-plane-to-webhook connectivity.
Why Was This Unexpected?
The same deployment worked successfully on a GKE Standard cluster.
Google Cloud documentation recommends allowing control-plane traffic using the cluster's masterIpv4CidrBlock when configuring firewall rules for admission webhooks.
However, with GKE Autopilot, the Kubernetes control plane is fully managed by Google, and the control-plane IP range is not exposed to the customer in the same way.
This raised an important question:
How should NetworkPolicies and firewall rules be configured when the Kubernetes API server's IP range is not available to the customer?
Investigation and Troubleshooting
We first verified the webhook, Service, DNS, and Pod-to-Pod connectivity. Everything was working as expected.
We then tried:
- Changing the webhook port from 9443 to 443
- Increasing the webhook timeout from 10 to 30 seconds
Reviewing and updating firewall rules
None of these resolved the timeout.Temporarily setting failurePolicy: Ignore
This resolved the issue, but this is not a production solution.
This led us to investigate the Kubernetes NetworkPolicy, which ultimately revealed the root cause.
Working With Google Cloud Support
After exhausting the common troubleshooting approaches, we worked with the Google Cloud Support networking team to understand why the Kubernetes API server could not reach the admission webhook.
Together, we performed several tests:
- Verified that the webhook Pod was healthy
- Confirmed that the Service and DNS were functioning correctly
- Performed packet-level connectivity testing
- Tested multiple firewall configurations
- Verified that the webhook was reachable from application Pods
Despite these checks, the Kubernetes API server continued to experience timeout errors.
During the investigation, the networking team suggested shifting our focus from VPC firewall rules to Kubernetes NetworkPolicy.
That turned out to be the key.
The NetworkPolicy Problem
Our existing NetworkPolicy contained restrictive from and to rules.
These rules explicitly define where traffic is allowed to originate from and where it is allowed to go.
On GKE Autopilot, the Kubernetes API server resides in a Google-managed control plane, outside the customer-visible network environment.
Because these control-plane addresses are not directly exposed to customer projects, it may not be possible to explicitly reference them in NetworkPolicy rules.
The networking team therefore recommended simplifying the NetworkPolicy by removing restrictive traffic selectors for the webhook communication and allowing the required traffic based primarily on the webhook port.
After removing the restrictive from and to clauses while continuing to allow the required webhook port, the Kubernetes API server was immediately able to communicate with the CloudNativePG admission webhook.
The PostgreSQL deployment then completed successfully.
Most importantly:
- We did not modify the webhook implementation
- We did not disable admission validation
- We did not need to increase the webhook timeout
- The PostgreSQL deployment completed successfully
This demonstrated that the issue was not caused by CloudNativePG itself.
Root Cause
The root cause was restrictive NetworkPolicy rules preventing the Kubernetes API server from reaching the admission webhook.
The important factor was the interaction between the NetworkPolicy and the Google-managed GKE Autopilot control plane.
Because the control plane exists outside the customer-visible network, NetworkPolicies that explicitly restrict traffic sources or destinations can unintentionally block admission webhook communication.
Production Solution
We resolved the issue by updating the Kubernetes NetworkPolicy.
Instead of restricting webhook traffic based on specific source or destination selectors, the policy was simplified to allow the required webhook communication while retaining the necessary namespace and DNS access.
After applying the updated policy:
- The Kubernetes API server successfully reached the admission webhook
- PostgreSQL deployment completed successfully
- Admission validation remained enabled
- No webhook configuration changes were required
Example NetworkPolicy
The following NetworkPolicy represents the configuration that resolved the webhook timeout in our GKE Autopilot environment.
It is provided as a reference implementation to demonstrate the important changes.
Important: Do not copy this policy directly into production without reviewing it. NetworkPolicy requirements vary by application and organization. Adapt the policy according to your security and networking requirements.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: unified-policy
namespace: ${NAMESPACE}
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
# Allow traffic within the namespace
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ${NAMESPACE}
# Allow Pod-to-Pod communication
- from:
- podSelector: {}
# Allow admission webhook traffic on TCP 9443
# Notice that there is no restrictive 'from'
# selector for this rule.
- ports:
- protocol: TCP
port: 9443
egress:
# Allow communication within the namespace
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ${NAMESPACE}
# Allow Pod-to-Pod communication
- to:
- podSelector: {}
# Allow DNS resolution
- to:
- namespaceSelector: {}
ports:
- protocol: TCP
port: 53
- protocol: UDP
port: 53
# Allow webhook traffic on TCP 9443
# Notice that there is no restrictive 'to'
# selector for this rule.
- ports:
- protocol: TCP
port: 9443
# Allow outbound communication required by the application
- to:
- ipBlock:
cidr: 0.0.0.0/0
The key point is not the exact policy shown above.
The important takeaway is:
Make sure your NetworkPolicies do not unintentionally block communication between the Kubernetes API server and admission webhooks.
Conclusion
Admission webhook timeout errors on GKE Autopilot can be challenging to diagnose because the webhook itself may appear completely healthy and reachable from within the cluster.
In our case, the issue was not related to CloudNativePG or PostgreSQL. The root cause was the interaction between a restrictive Kubernetes NetworkPolicy and communication from the GKE Autopilot control plane to the admission webhook.
Once the restrictive traffic selectors were removed for the required webhook communication, the API server was able to reach the webhook and the PostgreSQL deployment completed successfully.
Although this article uses CloudNativePG as the example, the same troubleshooting approach can be applied to other Kubernetes operators and applications that rely on admission webhooks.
If you encounter a situation where:
Webhook Pod is healthy
↓
Service is healthy
↓
DNS works
↓
Pod-to-Pod connectivity works
↓
API server → webhook = timeout
Check your NetworkPolicies.
That simple check can save hours of troubleshooting.
