L'erreur "exceeded its progress deadline" revient sans cesse malgré la modification de progressDeadlineSeconds

Je suis nouveau sur AKS, ACR et les pipelines DevOps et j’essaie de mettre en place un pipeline CI/CD.

J’ai un groupe de ressources configuré qui contient à la fois AKS et ACR. AKS utilise Standard_B2s et un seul noeud pour le moment puisque je fais juste des tests.

Les images sont déployées automatiquement vers ACR lors d’un commit sur master — je n’ai pas encore trouvé comment configurer les tests — mais quand il s’agit du déploiement sur AKS, j’obtiens sans cesse :

##[error]error: deployment "client-deployment" exceeded its progress deadline

J’ai modifié mon client.yaml pour inclure un progressDeadlineSeconds d’environ une heure puisque 10, 15 et 20 minutes n’ont pas fonctionné :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment
spec:
  progressDeadlineSeconds: 3600
  replicas: 1
  selector:
    matchLabels:
      component: client
  template:
    metadata:
      labels:
        component: client
    spec:
      containers:
        - name: client
          image: testappcontainers.azurecr.io/testapp-client
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: client-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: client
  ports:
    - port: 3000
      targetPort: 3000

J’ai simplement modifié le fichier azure-pipelines.yml que Pipelines a généré pour moi, que j’ai actuellement configuré comme suit :

# Docker
# Build and push an image to Azure Container Registry
# https://learn.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- master

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: <dockerRegistryServiceConnection_key>
  imageRepository: 'testapp'
  containerRegistry: 'testappcontainers.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)'
  tag: '$(Build.BuildId)'
  imagePullSecret: <imagePullSecret_key>

  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build and push stage
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: Docker@2
      displayName: Build and push client image to container registry
      inputs:
        command: buildAndPush
        repository: $(imageRepository)-client
        dockerfile: $(dockerfilePath)/client/Dockerfile
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          $(tag)

    - upload: manifests
      artifact: manifests

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build

  jobs:
  - deployment: Deploy
    displayName: Deploy job
    pool:
      vmImage: $(vmImageName)
    environment: 'testapp.default'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)

          - task: KubernetesManifest@0
            displayName: Deploy to Kubernetes cluster
            inputs:
              action: deploy
              manifests: |
                $(Pipeline.Workspace)/manifests/client.yaml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository):$(tag)

Voici également le journal de la tâche qui échoue :

##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=> True
##[debug]Result: True
##[section]Starting: Deploy to Kubernetes cluster
==============================================================================
Task         : Deploy to Kubernetes
Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version      : 0.162.1
Author       : Microsoft Corporation
Help         : https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
##[debug]agent.TempDirectory=/home/vsts/work/_temp
##[debug]loading inputs and endpoints
##[debug]loading INPUT_ACTION
##[debug]loading INPUT_KUBERNETESSERVICECONNECTION
##[debug]loading INPUT_STRATEGY
##[debug]loading INPUT_TRAFFICSPLITMETHOD
##[debug]loading INPUT_PERCENTAGE
##[debug]loading INPUT_BASELINEANDCANARYREPLICAS
##[debug]loading INPUT_MANIFESTS
##[debug]loading INPUT_CONTAINERS
##[debug]loading INPUT_IMAGEPULLSECRETS
##[debug]loading INPUT_RENDERTYPE
##[debug]loading INPUT_DOCKERCOMPOSEFILE
##[debug]loading INPUT_HELMCHART
##[debug]loading INPUT_KUSTOMIZATIONPATH
##[debug]loading INPUT_RESOURCETOPATCH
##[debug]loading INPUT_RESOURCEFILETOPATCH
##[debug]loading INPUT_MERGESTRATEGY
##[debug]loading INPUT_SECRETTYPE
##[debug]loading ENDPOINT_AUTH_<token>
##[debug]loading ENDPOINT_AUTH_SCHEME_<token>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZUREENVIRONMENT
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZURETENANTID
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_ROLEBINDINGNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SECRETNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_APITOKEN
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTCERTIFICATE
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loading SECRET_CONTAINER_PASSWORD
##[debug]loading SECRET_CONTAINER_USERNAME
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loaded 32
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]SYSTEM_HOSTTYPE=build
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]Build.BuildNumber=20191231.5
##[debug]Build.DefinitionName=test-app
##[debug]System.DefinitionId=4
##[debug]Agent.JobName=Deploy job
##[debug]System.TeamProject=test-app
##[debug]Build.BuildId=41
##[debug]System.TeamProject=test-app
##[debug]namespace=null
##[debug]containers=***/testapp:41
##[debug]imagePullSecrets=testappcontainers<key>-auth
##[debug]manifests=/home/vsts/work/1/manifests/client.yaml
##[debug]percentage=0
##[debug]strategy=none
##[debug]trafficSplitMethod=pod
##[debug]baselineAndCanaryReplicas=0
##[debug]arguments=null
##[debug]secretArguments=null
##[debug]secretType=dockerRegistry
##[debug]secretName=null
##[debug]dockerRegistryEndpoint=null
##[debug]kubernetesServiceConnection=<token>
##[debug]<token> data namespace = default
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]System.HostType=build
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]Build.SourceBranchName=master
##[debug]Build.Repository.Provider=TfsGit
##[debug]Build.Repository.Uri=https://[email protected]/thetestcompany/test-app/_git/test-app
##[debug]agent.proxyurl=undefined
##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
##[debug]AZURE_HTTP_USER_AGENT=VSTS_<hash>_build_4_0
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]system.culture=en-US
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]system.culture=en-US
##[debug]action=deploy
##[debug]kubernetesServiceConnection=<token>
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]<token> data authorizationType = AzureSubscription
##[debug]<token>=https://testappk8s-dns-<key>.hcp.westus.azmk8s.io/
##[debug]<token> auth param serviceAccountCertificate = ***
##[debug]<token> auth param apiToken = ***
##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]<token> data acceptUntrustedCerts = undefined
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]defaultRoot: '/home/vsts/work/1/s'
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
##[debug]findOptions.followSymbolicLinks: 'true'
##[debug]matchOptions.debug: 'false'
##[debug]matchOptions.nobrace: 'true'
##[debug]matchOptions.noglobstar: 'false'
##[debug]matchOptions.dot: 'true'
##[debug]matchOptions.noext: 'false'
##[debug]matchOptions.nocase: 'false'
##[debug]matchOptions.nonull: 'false'
##[debug]matchOptions.matchBase: 'false'
##[debug]matchOptions.nocomment: 'false'
##[debug]matchOptions.nonegate: 'false'
##[debug]matchOptions.flipNegate: 'false'
##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml'
##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml'
##[debug]statOnly: 'true'
##[debug]found 1 paths
##[debug]applying include pattern
##[debug]1 matches
##[debug]1 final results
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]New K8s objects after addin imagePullSecrets are :[{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"client-deployment"},"spec":{"progressDeadlineSeconds":3600,"replicas":1,"selector":{"matchLabels":{"component":"client"}},"template":{"metadata":{"labels":{"component":"client"}},"spec":{"containers":[{"name":"client","image":"***/testapp-client","ports":[{"containerPort":3000}]}],"imagePullSecrets":[{"name":"testappcontainers1741032e-auth"}]}}}},{"apiVersion":"v1","kind":"Service","metadata":{"name":"client-cluster-ip-service"},"spec":{"type":"ClusterIP","selector":{"component":"client"},"ports":[{"port":3000,"targetPort":3000}]}}]
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: apply
##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   apply
##[debug]   -f
##[debug]   /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default
deployment.apps/client-deployment unchanged
service/client-cluster-ip-service unchanged
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: ["rollout","status"]
##[debug]/usr/bin/kubectl arg: Deployment/client-deployment
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   rollout
##[debug]   status
##[debug]   Deployment/client-deployment
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default
error: deployment "client-deployment" exceeded its progress deadline
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: get
##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service
##[debug]/usr/bin/kubectl arg: ["-o","json"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   get
##[debug]   service/client-cluster-ip-service
##[debug]   -o
##[debug]   json
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default
{
    "apiVersion": "v1",
    "kind": "Service",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":3000,\"targetPort\":3000}],\"selector\":{\"component\":\"client\"},\"type\":\"ClusterIP\"}}\n"
        },
        "creationTimestamp":         "name": "client-cluster-ip-service",
        "namespace": "default",
        "resourceVersion": "1234045",
        "selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service",
        "uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3"
    },
    "spec": {
        "clusterIP": "10.0.181.220",
        "ports": [
            {
                "port": 3000,
                "protocol": "TCP",
                "targetPort": 3000
            }
        ],
        "selector": {
            "component": "client"
        },
        "sessionAffinity": "None",
        "type": "ClusterIP"
    },
    "status": {
        "loadBalancer": {}
    }
}
##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]set KUBECONFIG=
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]
##[debug]task result: Failed
##[error]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.complete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline
##[section]Finishing: Deploy to Kubernetes cluster

Ensuite, dans Azure CLI, on voit que le déploiement est présent, mais sans pod disponible :

eox-dev@Azure:~$ kubectl get deployments
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
client-deployment   0/1     1            0           3h47m
eox-dev@Azure:~$ kubectl describe deployment client-deployment
Name:                   client-deployment
Namespace:              default
CreationTimestamp:      Tue, 31 Dec 2019 15:50:30 +0000
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"client-deployment","namespace":"default"},"spec":{"progre...
Selector:               component=client
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  component=client
  Containers:
   client:
    Image:        testappcontainers.azurecr.io/testapp-client
    Port:         3000/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  <none>
NewReplicaSet:   client-deployment-5688bdc69c (1/1 replicas created)
Events:          <none>

Alors, qu’est-ce que je fais de mal ici ?


Source : Stack Overflow

Erreur du serveur (BadRequest) : le conteneur “client” dans le pod
“client-deployment-5688bdc69c-hxlcf” est en attente de démarrage : il essaie
de récupérer l’image et échoue

D’après mon expérience, cela est davantage lié aux imagePullSecrets et au namespace Kubernetes.

Dans vos tâches Create imagePullSecret et Deploy to Kubernetes cluster, j’ai remarqué que vous n’avez pas fourni de valeur pour le paramètre de tâche : namespace. Cela conduira à la création d’un nouveau namespace nommé default, puisque vous n’avez pas spécifié le namespace.

Et le secret Kubernetes généré par l’action createSecret est séparé pour chaque namespace. En un mot, chaque namespace a une valeur de secret différente :

Les secrets sont stockés dans un namespace donné et ne peuvent être
accédés que par les pods du même namespace.

Maintenant, revenons à votre processus de compilation.

Dans votre définition yml, Create imagePullSecret créera un secret pour le nouveau namespace default qui est créé automatiquement par la tâche puisque vous n’avez pas fourni la valeur du namespace.

Ensuite, dans la tâche suivante Deploy to Kubernetes cluster, pour la même raison, la tâche recréera un autre nouveau namespace default (Remarque : celui-ci n’est pas le même que le précédent). Vous pouvez également voir cette progression dans le journal :

À ce moment-là, le secret généré par la tâche précédente ne sera pas disponible pour le namespace actuel. MAIS, comme vous le savez, l’ACR est un registre de conteneurs privé et notre système doit vérifier si le secret Kubernetes est disponible.

De plus, dans votre tâche Deploy to Kubernetes cluster, vous spécifiez le dépôt comme $(imageRepository) ce qui ne correspond pas au dépôt vers lequel vous poussez l’image $(imageRepository)-client.

Cela peut également être vérifié dans votre journal :

C’est pourquoi il n’y a pas de noeud disponible dans votre Kubernetes, et vous échouez également à récupérer l’image.

Pour éviter ce problème, assurez-vous de fournir la valeur namespace dans la tâche KubernetesManifest@0.

      - task: KubernetesManifest@0
        displayName: Create imagePullSecret
        inputs:
          action: createSecret
          secretName: $(imagePullSecret)
          namespace: $(k8sNamespace)
          dockerRegistryEndpoint: $(DRServiceConnection)

      - task: KubernetesManifest@0
        displayName: Deploy to Kubernetes cluster
        inputs:
          action: deploy
          namespace: $(k8sNamespace)
          manifests: |
            $(System.ArtifactsDirectory)/manifests/deployment.yml
          imagePullSecrets: |
            $(imagePullSecret)
          containers: |
            $(containerRegistry)/$(imageRepository)-client:$(tag)

secret vers imagePullSecrets de chaque namespace