-
Notifications
You must be signed in to change notification settings - Fork 209
Description
Summary
JMESPath built-in functions like starts_with() return booleans that can fail silently when used in preconditions with operator: Equals and value: true. The documentation doesn't warn about this type coercion issue.
Use Case: ECR Pull Through Cache
We were building a ClusterPolicy to automatically rewrite container images from public registries to use AWS ECR Pull Through Cache. This eliminates Docker Hub rate limits and improves pull performance in EKS clusters.
Goal: Mutate images like this:
# Input
image: quay.io/prometheus/prometheus:v2.45.0
# Output (mutated to ECR Pull Through Cache)
image: 123456789.dkr.ecr.us-east-1.amazonaws.com/quay/prometheus/prometheus:v2.45.0Registries to handle: Docker Hub (implicit and explicit), quay.io, registry.k8s.io, public.ecr.aws
The policy uses foreach to iterate over containers and initContainers, with preconditions to detect which registry each image comes from.
Problem
When writing mutation policies with foreach, this pattern seems logical based on JMESPath spec:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: ecr-pull-through-cache
spec:
rules:
- name: rewrite-quay-images
match:
any:
- resources:
kinds:
- Pod
mutate:
foreach:
- list: "request.object.spec.containers"
preconditions:
any:
- key: "{{ starts_with(element.image, 'quay.io/') }}"
operator: Equals
value: true
patchesJson6902: |-
- op: replace
path: /spec/containers/{{elementIndex}}/image
value: '123456789.dkr.ecr.../quay/...'Result: The precondition fails silently. No errors in logs, no PolicyReports, the rule is simply skipped. Pods are created with original images unchanged.
Diagnostic difficulty:
kubectl get clusterpolicyshows Readykubectl get policyreport -Ashows nothing- Kyverno controller logs show no errors
- Pod creation succeeds (just without mutation)
This is the worst kind of failure: silent with no diagnostic output.
Root cause: The boolean return from starts_with() doesn't compare correctly with YAML value: true due to type coercion in the Equals operator implementation (pkg/engine/variables/operator/equal.go).
Working Pattern
The images context variable works reliably because it's string-to-string comparison:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: ecr-pull-through-cache
spec:
rules:
- name: rewrite-quay-images
match:
any:
- resources:
kinds:
- Pod
mutate:
foreach:
- list: "request.object.spec.containers"
preconditions:
all:
- key: '{{images.containers."{{element.name}}".registry}}'
operator: Equals
value: "quay.io"
patchesJson6902: |-
- op: replace
path: /spec/containers/{{elementIndex}}/image
value: '123456789.dkr.ecr.us-east-1.amazonaws.com/quay/{{images.containers."{{element.name}}".path}}:{{images.containers."{{element.name}}".tag}}'The images context provides pre-parsed components:
images.containers."<name>".registry→"quay.io"images.containers."<name>".path→"prometheus/prometheus"images.containers."<name>".tag→"v2.45.0"
Suggested Documentation Improvements
-
Add a note to Preconditions warning about JMESPath boolean functions:
⚠️ JMESPath functions that return booleans (likestarts_with(),ends_with(),contains()) may have type coercion issues when compared withvalue: true. For image registry detection, prefer theimagescontext variable. -
Cross-reference the
imagescontext in preconditions docs as the preferred pattern for registry-based conditions. -
Add to Variables documentation a recommendation to use
images.*.registryfor registry detection in mutation policies. -
Consider adding a troubleshooting note about silent precondition failures when using boolean-returning JMESPath functions.
Discovery Method
Found by reading the source code at pkg/engine/variables/operator/equal.go after 3+ syntax variations failed silently. The Chainsaw conformance tests in test/conformance/chainsaw/mutate/foreach-patchStrategicMerge-preconditions/ showed the correct images context pattern that Kyverno maintainers use internally.
Environment
- Kyverno v1.13.x
- EKS 1.31
- Policy type: ClusterPolicy with foreach mutation
Related
- JMESPath
starts_withspec: https://jmespath.org/specification.html#starts-with - Kyverno Preconditions: https://kyverno.io/docs/policy-types/cluster-policy/preconditions/
- Kyverno Variables: https://kyverno.io/docs/policy-types/cluster-policy/variables/
- Kyverno JMESPath: https://kyverno.io/docs/policy-types/cluster-policy/jmespath/
- AWS ECR Pull Through Cache: https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html