Kubernetes LoadBalancer IP Stuck in Pending

3 min readJan 16, 2021

The Kubernetes Service object exposes an application’s network interface, enabling clients to invoke the application via TCP or UDP. The most common usage of a Service is to expose an application’s HTTP endpoints.

LoadBalancer Services

The type property of a Kubernetes Service dictates whether the application is exposed to the cluster internally or to clients external to the cluster.

A Kubernetes Service with type: LoadBalancer is accessible from outside the cluster.

In this example, Kubernetes exposes TCP port 80 to the outside world. Any TCP traffic received on port 80 is routed to port 9000 in the Pod’s container.

apiVersion: v1
kind: Service
metadata:
  name: k8salliance-service
spec:
  type: LoadBalancer
  selector:
    app: k8salliance-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9000

External IPs

All LoadBalancer Services receive an external IP. Clients outside of the cluster access the Service through this external IP.

The kubectl describe svc and kubectl get svc commands will display the external IP of a Service.

$ kubectl describe svc k8salliance-service
Name:                     k8salliance-service
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=k8salliance-app
Type:                     LoadBalancer
IP:                       10.19.252.223
LoadBalancer Ingress:     34.74.203.201
Port:                     <unset>  80/TCP
TargetPort:               9000/TCP
NodePort:                 <unset>  32027/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  77s   service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   40s   service-controller  Ensured load balancer$ kubectl get svc k8salliance-service
NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
k8salliance-service   LoadBalancer   10.19.252.223   34.74.203.201   80:32027/TCP   39m

However, sometimes the external IP displays as pending.

$ kubectl get svc k8salliance-service
NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
k8salliance-service   LoadBalancer   10.19.252.223   <pending>       80:32027/TCP   19m

External IP Allocation

The external IP provisioning process happens asynchronously. This IP assignment process is dependent on the Kubernetes hosting environment (the cloud provider).

An external load balancer in the hosting environment handles the IP allocation and any other configurations necessary to route external traffic to the Kubernetes Service.

The external load balancer is implemented and provided by the cloud vendor.

Service Controllers

The Kubernetes Service Controller listens for Service creation and modification events.

When a Service is created or modified, the Service Controller communicates with the cloud vendor’s external load balancer to create the necessary configurations.

Once configuration of the external load balancer is complete, the Kubernetes Service Controller retrieves the external IP and populates the Service object with this information.

Amazon Web Service (AWS) Service Controller

https://github.com/kubernetes/cloud-provider-aws

Google Cloud Provider (GCP) Service Controller

https://github.com/kubernetes/cloud-provider-gcp

Microsoft Azure Service Controller

https://github.com/kubernetes-sigs/cloud-provider-azure

On-Premise Kubernetes Clusters

Most on-premise Kubernetes clusters do not have external load balancers that can dynamically allocate IPs.

The on-premise cluster is not compatible with LoadBalancer-type Services that expect external IPs to be automatically assigned. There are two solutions to this problem.

The most common solution is to manually assign an external IP to the Service. This can be done in the Service’s YAML configuration.

apiVersion: v1
kind: Service
metadata:
  name: k8salliance-service
spec:
  type: LoadBalancer
  selector:
    app: k8salliance-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9000
  externalIPs:
  - "34.74.203.201"

When manually assigning the external IP, be advised the Kubernetes API documentation has this to say on the subject:

These IPs are not managed by Kubernetes. The user is responsible for ensuring that traffic arrives at a node with this IP.

Using NodePort Services

An alternative to manually assigning external IPs is to change the Service from type: LoadBalancer to type: NodePort.

The Service will be accessible from outside the cluster via any Kubernetes Node IP.

Minikube Workaround

Minikube has a special command for exposing LoadBalancer Services.

Running minikube tunnel will create network routes to expose any existing LoadBalancer Services.

Before running minikube tunnel, the Service will not have an external IP.

$ kubectl get svc
NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
k8salliance-svc         LoadBalancer   10.100.132.175   <pending>     80:32695/TCP   27s

Start minikube tunnel in a separate terminal window. Status messages will display while the command is running.

$ minikube tunnel

The Service will now have an external IP assigned.

$ kubectl get svc
NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE
k8salliance-svc         LoadBalancer   10.100.132.175   10.100.132.175   80:32695/TCP   4m59s

Note that once you stop the minikube tunnel command, the external IP will be removed from the service. You must keep minikube tunnel running in order to maintain the external IP assignment.