Race condition when installing multiple CRDs

[tschale]

What happened?

Installed rabbitmq cluster-operator and messaging-topology-operator-with-certmanager, as well as cert-manager (which is needed for the topology operator). The topology-operator is stuck in ContainerCreating with following events:

Warning  FailedMount  57s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-wfgwq]: timed out waiting for the condition
Warning  FailedMount  52s (x9 over 2m59s)  kubelet            MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found

The topology-operator depends on CRDs from cert-manager, which has been installed immediately before and is not yet ready.

What did you expect to happen?

It would be nice if getdeck could handle that case.
(I'm unsure how it could be handled though. In this case it would mean that Getdeck waits with the installation of the topology operator until the resources from the cert-manager are ready, wouldn't it?)

How can we reproduce it (as minimally and precisely as possible)?

Execute the provided Deckfile. It first occurred when using a Beiboot, but it was reproducible with a k3d cluster as well.

What is the content of the Deckfile you are using?

version: "1"

cluster:
  provider: k3d
  minVersion: 4.0.0
  name: rabbitmq
  nativeConfig:
    apiVersion: k3d.io/v1alpha2
    kind: Simple
    servers: 1
    agents: 1
    image: rancher/k3s:v1.22.9-k3s1
    ports:
      - port: 8080:80
        nodeFilters:
          - loadbalancer
      - port: 8443:443
        nodeFilters:
          - loadbalancer
      - port: 31820:31820/UDP
        nodeFilters:
          - agent[0]

decks:
  - name: spacecrafts
    namespace: default

    sources:
      - type: file
        ref: https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
      - type: file
        ref: https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
      - type: file
        ref: https://github.com/rabbitmq/messaging-topology-operator/releases/latest/download/messaging-topology-operator-with-certmanager.yaml

OS version

Details

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
Linux gutschi-ThinkPad-T480 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Anything else we need to know?

No response

[buschNT]

Sources are currently applied in the sequence as listed in the deckfile. I guess we would have to implement/design some kind of dependency graph (allowing to add a "depends on" option for sources) and further check if parent sources have applied successfully.

Furthermore we could add a "sleep" setting after applying sources. This might not be optimal, but maybe slowing down applying sources to the cluster between certain sources could fix this issue in some cases.

[tschale]

I can imagine that a "sleep" would probably be something which we can implement quicker than a robust dependency graph. Although I agree that it might not be optimal. Maybe it can even lead to a case of "a 10 second sleep worked for me".