User cannot get configmaps in the namespace

[sathyu]

Hello,
I am deploying this in k8 and getting following errors, any idea what that I am missing?
I am deploying this as non-admin user.

$ kubectl logs stolon-sentinel-7754964b89-8vmv4
2019-01-29T21:22:26.067Z INFO cmd/sentinel.go:1962 sentinel uid {"uid": "dfa105e8"}
2019-01-29T21:22:26.071Z INFO cmd/sentinel.go:80 Trying to acquire sentinels leadership
ERROR: logging before flag.Parse: I0129 21:22:26.071509 1 leaderelection.go:174] attempting to acquire leader lease...
ERROR: logging before flag.Parse: E0129 21:22:26.146967 1 leaderelection.go:224] error retrieving resource lock k8poc-sathya/stolon-cluster-kube-stolon: configmaps "stolon-cluster-kube-stolon" is forbidden: User "system:serviceaccount:k8poc-sathya:default" cannot get configmaps in the namespace "k8poc-sathya"

[lwolf]

Hi,
as log record says

User "system:serviceaccount:k8poc-sathya:default" cannot get configmaps in the namespace "k8poc-sathya"

the user does not have permissions to get configmaps. You need to check ServiceAccount/Role/RoleBinding to make sure that the user has required permissions.

[sathyu]

Hello ,
As understanding, I can do all admin works within my name space. and I was told to user my service account which I am doing. Still getting error.
So, question 1. we I need access to kube-system namespace, if so , is just "Read" sufficient.
2. if I don't get access to kube-system what is the way to implement this ?
3. Are you available for 1:1 talk , I am ready to compensate your time (serious).

$ Error from server (Forbidden): deployments.extensions is forbidden: User "system:serviceaccount:k8poc-sathya:k8-poc-sathya" cannot list deployments.extensions in the namespace "kube-system"

[lwolf]

It shouldn't require access to the kube-system namespace at all.
Could you please provide more information about your setup, will see what I can do?

• what k8s version do you use?
• what stolon version do you use?
• what namespace did you install the chart?
• could you post content of your values.yaml stripping out (if any) sensitive information

[sathyu]

Step1:
$ kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Step 2:
imageTag: "v0.12.0-pg10"
Namespace = k8poc-sathya
Step 3:
$ helm install --name k8poc-sathya -f values.yaml .
Error: pods is forbidden: User "system:serviceaccount:k8poc-sathya:k8-poc-sathya" cannot list pods in the namespace "kube-system"

step 4: values.yaml

Hello.
Below is the values.yaml entries:
$ cat values.yaml

# Default values for Stolon Helm Chart.
# This is a YAML-formatted file.
## Declare variables to be passed into your templates.

## Override the name of the Chart.
##
# nameOverride:

## Stolon image.
##

image: "sorintlab/stolon"

## Stolon image version.
## ref: https://hub.docker.com/r/sorintlab/stolon/tags/
##
imageTag: "v0.12.0-pg10"

## Specify a imagePullPolicy: 'Always' if imageTag is 'latest', else set to 'IfNotPresent'.
## ref: https://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
# imagePullPolicy:

## Configuration values for Stolon.

# Set custom stolon cluster name
clusterName: "kube-stolon"
debug: false

## log slow queries
# disabled by default
slow_queries:
  enabled: false
  min_duration: 300

ports:
  internalPort: 5432
  externalPort: 5432

store:
  ##  Backend could be one of the following:
  ## - etcdv2
  ## - etcdv3
  ## - consul (should work, but not tested yet)
  ## - kubernetes (should work, but not tested yet)
  backend: kubernetes
  ## store endpoints MUST be set for etcd/consul backends
  #  endpoints: "http://etcd-etcd-0.etcd-etcd:2379,http://etcd-etcd-1.etcd-etcd:2379,http://etcd-etcd-2.etcd-etcd:2379"

pgReplUsername: "repluser"
## set password for the repluser
## default is 40 random chars
pgReplPassword: "replPassword"

pgSuperuserName: "stolon"

## set password for the superuser
## default is 40 random chars
pgSuperuserPassword: "stolon123"

sentinel:
  replicas: 3

  ## Configure resource requests and limits.
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ##

  resources:
    requests:
      cpu: "100m"
      memory: "512Mi"

  ## Configure nodeSelector, tolerations and affinity.
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node
  ##

  affinity: {}
  nodeSelector: {}
  tolerations: []

proxy:
  replicas: 2
  ## Set serviceType to nodePort if needed
  ## proxy is used to route RW requests to the master
  # serviceType: NodePort

  ## Configure resource requests and limits.
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ##

  resources:
    requests:
      cpu: "100m"
      memory: "512Mi"

  ## Configure nodeSelector, tolerations and affinity.
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node
  ##

  affinity: {}
  nodeSelector: {}
  tolerations: []

keeper:
  replicas: 2
  ## Set serviceType to nodePort if needed
  ## keeper service is used to route RO requests to all nodes
  # serviceType: NodePort

  ## configure ssl for client access
  # create certificates according to these instructions: https://www.postgresql.org/docs/9.6/static/ssl-tcp.html
  # to enable encrypted traffic, servert.crt and server.key are required, by that name.
  # the use of ** Client Certificates ** is not supported
  client_ssl:
    enabled: false
    certs_secret_name: pg-cert-secret

  ## Configure resource requests and limits.
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ##

  resources:
    requests:
      cpu: "100m"
      memory: "512Mi"

  ## Configure nodeSelector, tolerations and affinity.
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node
  ##

  affinity: {}
  nodeSelector: {}
  tolerations: []
## Persistent Volume Storage configuration.
## ref: https://kubernetes.io/docs/user-guide/persistent-volumes
##

persistence:
  ## Enable persistence using Persistent Volume Claims.
  ##
  enabled: false

  ## Persistent Volume Access Mode.
  ##
  accessMode: ReadWriteOnce

  ## Persistant Volume Storage Class Name
  storageClassName: standard

  ## Persistent Volume Storage Size.
  ##
  size: 25Gi

rbac:
  # Specifies whether RBAC resources should be created
  create: true

serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: true
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name: k8-poc-sathya

[lwolf]

I've just deployed this chart with the values.yaml file you've pasted. I don't have k8s 1.10 so I tested on my 1.12.3. Everything works fine.

$ kubectl get pods
k8poc-sathya-stolon-keeper-0                                      1/1       Running     0          1m
k8poc-sathya-stolon-keeper-1                                      1/1       Running     0          46s
k8poc-sathya-stolon-lz6ws                                         0/1       Completed   0          1m
k8poc-sathya-stolon-proxy-64dfb7b59-7jmx8                         1/1       Running     0          1m
k8poc-sathya-stolon-proxy-64dfb7b59-vnt7w                         1/1       Running     0          1m
k8poc-sathya-stolon-sentinel-59dc875688-2gdxw                     1/1       Running     0          1m
k8poc-sathya-stolon-sentinel-59dc875688-97r6d                     1/1       Running     0          1m
k8poc-sathya-stolon-sentinel-59dc875688-rmrpr                     1/1       Running     0          1m

Did you try reinstalling the chart from scratch?
Did helm install actually succeeded?
Did you previously install anything using helm? maybe it's misconfigured

[sathyu]

Hello again,
Now stolon-sentinel is up and running but logged below error at pod level:
2019-02-06T19:02:20.069Z INFO cmd/sentinel.go:1962 sentinel uid {"uid": "70089ff6"}
2019-02-06T19:02:20.145Z INFO cmd/sentinel.go:80 Trying to acquire sentinels leadership
ERROR: logging before flag.Parse: I0206 19:02:20.145433 1 leaderelection.go:174] attempting to acquire leader lease...
ERROR: logging before flag.Parse: E0206 19:02:20.152184 1 leaderelection.go:224] error retrieving resource lock k8poc-sathya/stolon-cluster-kube-stolon
: configmaps "stolon-cluster-kube-stolon" is forbidden: User "system:serviceaccount:k8poc-sathya:stolon-sa" cannot get configmaps in the namespace "k8poc-sat
hya"
2019-02-06T19:02:20.162Z ERROR cmd/sentinel.go:1815 error retrieving cluster data {"error": "failed to get latest version of configmap: configm
aps "stolon-cluster-kube-stolon" is forbidden: User "system:serviceaccount:k8poc-sathya:stolon-sa" cannot get configmaps in the namespace "k8poc-sathya
""}
AND
stolok-keeper is crashlooping with error:
2019-02-06T19:20:34.151Z FATAL cmd/keeper.go:117 cannot get current user: cannot detect current user.

For me, role.yaml (as provided ) did not work , it errors out. So I changed it like below and ran fine.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: k8poc-sathya
name: stolon
rules:

• apiGroups: ["extensions", "apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
• apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  Role-binding:
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: RoleBinding
  metadata:
  name: rb-stolon
  subjects:
• kind: ServiceAccount
  name: stolon-sa
  namespace: k8poc-sathya
  roleRef:
  kind: Role
  name: stolon
  apiGroup: rbac.authorization.k8s.io

As always , your help is appreciated.

[lwolf]

so, it seems that you resolved your problem and the issue could be closed?

I'll keep in mind that additional roles might be required. But for now I can't reproduce it.
Official stolon example does not have this as well - https://github.com/sorintlab/stolon/blob/master/examples/kubernetes/role.yaml

[sathyu]

Hello
I did resolved the error I was getting during helm install. Now, I am getting below error when starting
stolon-keeper pods. To again emphasize, I have to run this as cluster non-admin user.
I have made some changes to keeper.yaml like below
#chown stolon:stolon $STOLON_DATA
exec stolon-keeper --data-dir $STOLON_DATA
#exec gosu stolon stolon-keeper --data-dir $STOLON_DATA
Both chown and gosu will not work for me, errors out.
let me know how can I fix this error.
Thanks
"cannot get current user: cannot detect current user"