Troubleshooting
Verify operator status¶
Use the following checks to help you troubleshoot the Aiven Operator for Kubernetes.
Check the Pods¶
Verify that all the operator Pods are READY, and the STATUS is Running.
The output is similar to the following:
Verify that the cert-manager Pods are also running.
The output has the status:
NAME READY STATUS RESTARTS AGE
cert-manager-7dd5854bb4-85cpv 1/1 Running 0 76s
cert-manager-cainjector-64c949654c-n2z8l 1/1 Running 0 77s
cert-manager-webhook-6bdffc7c9d-47w6z 1/1 Running 0 76s
Visualize the operator logs¶
Use the following command to visualize all the logs from the operator.
Verify the operator version¶
kubectl get pod -n aiven-operator-system -l control-plane=controller-manager -o jsonpath="{.items[0].spec.containers[0].image}"
Known issues and limitations¶
We're always working to resolve problems that pop up in Aiven products. If your problem is listed below, we know about it and are working to fix it. If your problem isn't listed below, report it as an issue.
cert-manager¶
Issue¶
The following event appears on the operator Pod:
Impact¶
You cannot run the operator.
Solution¶
Make sure that cert-manager is up and running.
The output shows the status of each cert-manager:
NAME READY STATUS RESTARTS AGE
cert-manager-7dd5854bb4-85cpv 1/1 Running 0 76s
cert-manager-cainjector-64c949654c-n2z8l 1/1 Running 0 77s
cert-manager-webhook-6bdffc7c9d-47w6z 1/1 Running 0 76s
Resource Reconciliation Behavior¶
The Aiven Operator is designed to stop reconciling resources once they reach a ready state. This behavior is different from some other Kubernetes operators and can cause confusion.
When Reconciliation Stops¶
The operator stops reconciling a resource when both of the following conditions are met:
- Latest generation processed - The operator has processed the latest changes from the Kubernetes resource specification
- Resource is running - The resource has been successfully created in Aiven and is in a running state
You can check if a resource is in this "ready" state by looking for these annotations:
Look for these annotations:
- controllers.aiven.io/instance-is-running: Indicates the resource is running in Aiven
- controllers.aiven.io/generation-was-processed: Indicates the latest spec changes have been processed
Common Misconception¶
Expected behavior: "If I manually delete a resource from the Aiven console, the operator should recreate it automatically."
Actual behavior: Once a resource/secret is ready, the operator stops continuous reconciliation. Manual changes in the Aiven console will not trigger automatic recreation.
ServiceUser or ServiceUserRotation has no connection secret or the secret has the wrong host¶
kubectl describe serviceuser <name> (or serviceuserrotation <name>) shows an Error condition
with reason Preconditions and a PreconditionsNotMet warning event such as
component "pg" with route "privatelink" not found on service "my-pg": the service does not expose
the route set in spec.connInfoSecretRoute. Establish the PrivateLink connection or set the route
back to dynamic. Note that private and public are not always listed: a service in a project VPC
has no private route on any cloud, and most clouds outside a VPC do not list one either; public
and private are listed only while public_access or private_access is enabled for that component
in the service's userConfig. Selecting an unlisted route waits forever. Even dynamic is reported
the same way if the service lists no primary component for it. An existing secret keeps
its previous host until then; for ServiceUserRotation the rotation schedule also pauses until the
route is available. The route is re-checked once per poll interval (--poll-interval, 10 minutes by
default); to pick it up sooner, force a reconciliation as described below. With
spec.connInfoSecretRoute unset, the same situation is reported as in previous releases, with the
message service component "kafka" not found.
How to Force Reconciliation¶
If you need to force the operator to reconcile a resource (for example, after manually deleting it from Aiven), you have several options:
Option 1: Delete and Recreate the Kubernetes Resource¶
Note: The resource will be deleted on the Aiven side and data may be lost when using this approach.
Option 2: Remove the Generation Annotation¶
Remove the controllers.aiven.io/generation-was-processed annotation to force reconciliation without affecting the resource's functionality:
kubectl annotate <resource-type> <resource-name> -n <namespace> controllers.aiven.io/generation-was-processed-
Example:
This is the cleanest approach as it doesn't modify the resource specification or create any side effects.