Ce site est optimisé pour être consulté depuis un navigateur moderne dans lequel JavaScript est activé.

Keep exécuterning into "exceeded its progress dead line" despite changing progressDeadlineSeconds

ayi

<t>I'm new AKS, ACR, and DevOps Pipelines and I'm trying to setup a CI/CD pipeline. 
 
I have a resource group setup that has both AKS and ACR in it. AKS is using Standard_B2s and only one node at this point since I'm just playing around. 
 
Images are being deployed to ACR automatically on a commit to master--haven't figured out how to setup testing yet--but when it comes to deploying to AKS, I just keep getting a:

##[error]error: deployment "client-deployment" exceeded its progress deadline<br/>
<br/>
```<br/>
<br/>
I've changed my `client.yaml` to include a `progressDeadlineSeconds` of like an hour as 10, 15, and 20 minutes didn't work:<br/>
<br/>
```<br/>
apiVersion: apps/v1<br/>
kind: Deployment<br/>
metadata:<br/>
  name: client-deployment<br/>
spec:<br/>
  progressDeadlineSeconds: 3600<br/>
  replicas: 1<br/>
  selector:<br/>
    matchLabels:<br/>
      component: client<br/>
  template:<br/>
    metadata:<br/>
      labels:<br/>
        component: client<br/>
    spec:<br/>
      containers:<br/>
        - name: client<br/>
          image: testappcontainers.azurecr.io/testapp-client<br/>
          ports:<br/>
            - containerPort: 3000<br/>
---<br/>
apiVersion: v1<br/>
kind: Service<br/>
metadata:<br/>
  name: client-cluster-ip-service<br/>
spec:<br/>
  type: ClusterIP<br/>
  selector:<br/>
    component: client<br/>
  ports:<br/>
    - port: 3000<br/>
      targetPort: 3000<br/>
<br/>
```<br/>
<br/>
I've just been modifying the `azure-pipelines.yml` that Pipelines generated for me, which I currently have as the following:<br/>
<br/>
```<br/>
# Docker<br/>
# Build and push an image to Azure Container Registry<br/>
# https://learn.microsoft.com/azure/devops/pipelines/languages/docker<br/>
<br/>
trigger:<br/>
- master<br/>
<br/>
resources:<br/>
- repo: self<br/>
<br/>
variables:<br/>
  # Container registry service connection established during pipeline creation<br/>
  dockerRegistryServiceConnection: <dockerRegistryServiceConnection_key><br/>
  imageRepository: 'testapp'<br/>
  containerRegistry: 'testappcontainers.azurecr.io'<br/>
  dockerfilePath: '$(Build.SourcesDirectory)'<br/>
  tag: '$(Build.BuildId)'<br/>
  imagePullSecret: <imagePullSecret_key><br/>
<br/>
  # Agent VM image name<br/>
  vmImageName: 'ubuntu-latest'<br/>
<br/>
stages:<br/>
- stage: Build<br/>
  displayName: Build and push stage<br/>
  jobs:  <br/>
  - job: Build<br/>
    displayName: Build<br/>
    pool:<br/>
      vmImage: $(vmImageName)<br/>
    steps:<br/>
    - task: Docker@2<br/>
      displayName: Build and push client image to container registry<br/>
      inputs:<br/>
        command: buildAndPush<br/>
        repository: $(imageRepository)-client<br/>
        dockerfile: $(dockerfilePath)/client/Dockerfile<br/>
        containerRegistry: $(dockerRegistryServiceConnection)<br/>
        tags: |<br/>
          $(tag)<br/>
<br/>
    - upload: manifests<br/>
      artifact: manifests<br/>
<br/>
- stage: Deploy<br/>
  displayName: Deploy stage<br/>
  dependsOn: Build<br/>
<br/>
  jobs:<br/>
  - deployment: Deploy<br/>
    displayName: Deploy job<br/>
    pool:<br/>
      vmImage: $(vmImageName)<br/>
    environment: 'testapp.default'<br/>
    strategy:<br/>
      runOnce:<br/>
        deploy:<br/>
          steps:<br/>
          - task: KubernetesManifest@0<br/>
            displayName: Create imagePullSecret<br/>
            inputs:<br/>
              action: createSecret<br/>
              secretName: $(imagePullSecret)<br/>
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)<br/>
<br/>
          - task: KubernetesManifest@0<br/>
            displayName: Deploy to Kubernetes cluster<br/>
            inputs:<br/>
              action: deploy<br/>
              manifests: |<br/>
                $(Pipeline.Workspace)/manifests/client.yaml<br/>
              imagePullSecrets: |<br/>
                $(imagePullSecret)<br/>
              containers: |<br/>
                $(containerRegistry)/$(imageRepository):$(tag)<br/>
<br/>
```<br/>
<br/>
Here is the log too for the Task that fails:<br/>
<br/>
```<br/>
##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster'<br/>
##[debug]Evaluating: SucceededNode()<br/>
##[debug]Evaluating SucceededNode:<br/>
##[debug]=> True<br/>
##[debug]Result: True<br/>
##[section]Starting: Deploy to Kubernetes cluster<br/>
==============================================================================<br/>
Task         : Deploy to Kubernetes<br/>
Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts<br/>
Version      : 0.162.1<br/>
Author       : Microsoft Corporation<br/>
Help         : https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest<br/>
==============================================================================<br/>
##[debug]agent.TempDirectory=/home/vsts/work/_temp<br/>
##[debug]loading inputs and endpoints<br/>
##[debug]loading INPUT_ACTION<br/>
##[debug]loading INPUT_KUBERNETESSERVICECONNECTION<br/>
##[debug]loading INPUT_STRATEGY<br/>
##[debug]loading INPUT_TRAFFICSPLITMETHOD<br/>
##[debug]loading INPUT_PERCENTAGE<br/>
##[debug]loading INPUT_BASELINEANDCANARYREPLICAS<br/>
##[debug]loading INPUT_MANIFESTS<br/>
##[debug]loading INPUT_CONTAINERS<br/>
##[debug]loading INPUT_IMAGEPULLSECRETS<br/>
##[debug]loading INPUT_RENDERTYPE<br/>
##[debug]loading INPUT_DOCKERCOMPOSEFILE<br/>
##[debug]loading INPUT_HELMCHART<br/>
##[debug]loading INPUT_KUSTOMIZATIONPATH<br/>
##[debug]loading INPUT_RESOURCETOPATCH<br/>
##[debug]loading INPUT_RESOURCEFILETOPATCH<br/>
##[debug]loading INPUT_MERGESTRATEGY<br/>
##[debug]loading INPUT_SECRETTYPE<br/>
##[debug]loading ENDPOINT_AUTH_<token><br/>
##[debug]loading ENDPOINT_AUTH_SCHEME_<token><br/>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZUREENVIRONMENT<br/>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZURETENANTID<br/>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTNAME<br/>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_ROLEBINDINGNAME<br/>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SECRETNAME<br/>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_APITOKEN<br/>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTCERTIFICATE<br/>
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION<br/>
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION<br/>
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN<br/>
##[debug]loading SECRET_CONTAINER_PASSWORD<br/>
##[debug]loading SECRET_CONTAINER_USERNAME<br/>
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN<br/>
##[debug]loaded 32<br/>
##[debug]Agent.ProxyUrl=undefined<br/>
##[debug]Agent.CAInfo=undefined<br/>
##[debug]Agent.ClientCert=undefined<br/>
##[debug]Agent.SkipCertValidation=undefined<br/>
##[debug]SYSTEM_HOSTTYPE=build<br/>
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/<br/>
##[debug]Build.BuildNumber=20191231.5<br/>
##[debug]Build.DefinitionName=test-app<br/>
##[debug]System.DefinitionId=4<br/>
##[debug]Agent.JobName=Deploy job<br/>
##[debug]System.TeamProject=test-app<br/>
##[debug]Build.BuildId=41<br/>
##[debug]System.TeamProject=test-app<br/>
##[debug]namespace=null<br/>
##[debug]containers=***/testapp:41<br/>
##[debug]imagePullSecrets=testappcontainers<key>-auth<br/>
##[debug]manifests=/home/vsts/work/1/manifests/client.yaml<br/>
##[debug]percentage=0<br/>
##[debug]strategy=none<br/>
##[debug]trafficSplitMethod=pod<br/>
##[debug]baselineAndCanaryReplicas=0<br/>
##[debug]arguments=null<br/>
##[debug]secretArguments=null<br/>
##[debug]secretType=dockerRegistry<br/>
##[debug]secretName=null<br/>
##[debug]dockerRegistryEndpoint=null<br/>
##[debug]kubernetesServiceConnection=<token><br/>
##[debug]<token> data namespace = default<br/>
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/<br/>
##[debug]System.HostType=build<br/>
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s<br/>
##[debug]Build.SourceBranchName=master<br/>
##[debug]Build.Repository.Provider=TfsGit<br/>
##[debug]Build.Repository.Uri=https://thetestcompany@dev.azure.com/thetestcompany/test-app/_git/test-app<br/>
##[debug]agent.proxyurl=undefined<br/>
##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined<br/>
##[debug]AZURE_HTTP_USER_AGENT=VSTS_<hash>_build_4_0<br/>
##[debug]Agent.ProxyUrl=undefined<br/>
##[debug]Agent.CAInfo=undefined<br/>
##[debug]Agent.ClientCert=undefined<br/>
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json<br/>
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json<br/>
##[debug]system.culture=en-US<br/>
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json<br/>
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json<br/>
##[debug]system.culture=en-US<br/>
##[debug]action=deploy<br/>
##[debug]kubernetesServiceConnection=<token><br/>
##[debug]agent.tempDirectory=/home/vsts/work/_temp<br/>
##[debug]<token> data authorizationType = AzureSubscription<br/>
##[debug]<token>=https://testappk8s-dns-<key>.hcp.westus.azmk8s.io/<br/>
##[debug]<token> auth param serviceAccountCertificate = ***<br/>
##[debug]<token> auth param apiToken = ***<br/>
##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config<br/>
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config<br/>
##[debug]<token> data acceptUntrustedCerts = undefined<br/>
##[debug]which 'kubectl'<br/>
##[debug]found: '/usr/bin/kubectl'<br/>
##[debug]which 'kubectl'<br/>
##[debug]found: '/usr/bin/kubectl'<br/>
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s<br/>
##[debug]defaultRoot: '/home/vsts/work/1/s'<br/>
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'<br/>
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'<br/>
##[debug]findOptions.followSymbolicLinks: 'true'<br/>
##[debug]matchOptions.debug: 'false'<br/>
##[debug]matchOptions.nobrace: 'true'<br/>
##[debug]matchOptions.noglobstar: 'false'<br/>
##[debug]matchOptions.dot: 'true'<br/>
##[debug]matchOptions.noext: 'false'<br/>
##[debug]matchOptions.nocase: 'false'<br/>
##[debug]matchOptions.nonull: 'false'<br/>
##[debug]matchOptions.matchBase: 'false'<br/>
##[debug]matchOptions.nocomment: 'false'<br/>
##[debug]matchOptions.nonegate: 'false'<br/>
##[debug]matchOptions.flipNegate: 'false'<br/>
##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml'<br/>
##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml'<br/>
##[debug]statOnly: 'true'<br/>
##[debug]found 1 paths<br/>
##[debug]applying include pattern<br/>
##[debug]1 matches<br/>
##[debug]1 final results<br/>
##[debug]agent.tempDirectory=/home/vsts/work/_temp<br/>
##[debug]New K8s objects after addin imagePullSecrets are :[{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"client-deployment"},"spec":{"progressDeadlineSeconds":3600,"replicas":1,"selector":{"matchLabels":{"component":"client"}},"template":{"metadata":{"labels":{"component":"client"}},"spec":{"containers":[{"name":"client","image":"***/testapp-client","ports":[{"containerPort":3000}]}],"imagePullSecrets":[{"name":"testappcontainers1741032e-auth"}]}}}},{"apiVersion":"v1","kind":"Service","metadata":{"name":"client-cluster-ip-service"},"spec":{"type":"ClusterIP","selector":{"component":"client"},"ports":[{"port":3000,"targetPort":3000}]}}]<br/>
##[debug]agent.tempDirectory=/home/vsts/work/_temp<br/>
##[debug]agent.tempDirectory=/home/vsts/work/_temp<br/>
##[debug]which '/usr/bin/kubectl'<br/>
##[debug]found: '/usr/bin/kubectl'<br/>
##[debug]which '/usr/bin/kubectl'<br/>
##[debug]found: '/usr/bin/kubectl'<br/>
##[debug]/usr/bin/kubectl arg: apply<br/>
##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"]<br/>
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]<br/>
##[debug]exec tool: /usr/bin/kubectl<br/>
##[debug]arguments:<br/>
##[debug]   apply<br/>
##[debug]   -f<br/>
##[debug]   /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782<br/>
##[debug]   --namespace<br/>
##[debug]   default<br/>
[command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default<br/>
deployment.apps/client-deployment unchanged<br/>
service/client-cluster-ip-service unchanged<br/>
##[debug]which '/usr/bin/kubectl'<br/>
##[debug]found: '/usr/bin/kubectl'<br/>
##[debug]which '/usr/bin/kubectl'<br/>
##[debug]found: '/usr/bin/kubectl'<br/>
##[debug]/usr/bin/kubectl arg: ["rollout","status"]<br/>
##[debug]/usr/bin/kubectl arg: Deployment/client-deployment<br/>
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]<br/>
##[debug]exec tool: /usr/bin/kubectl<br/>
##[debug]arguments:<br/>
##[debug]   rollout<br/>
##[debug]   status<br/>
##[debug]   Deployment/client-deployment<br/>
##[debug]   --namespace<br/>
##[debug]   default<br/>
[command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default<br/>
error: deployment "client-deployment" exceeded its progress deadline<br/>
##[debug]which '/usr/bin/kubectl'<br/>
##[debug]found: '/usr/bin/kubectl'<br/>
##[debug]which '/usr/bin/kubectl'<br/>
##[debug]found: '/usr/bin/kubectl'<br/>
##[debug]/usr/bin/kubectl arg: get<br/>
##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service<br/>
##[debug]/usr/bin/kubectl arg: ["-o","json"]<br/>
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]<br/>
##[debug]exec tool: /usr/bin/kubectl<br/>
##[debug]arguments:<br/>
##[debug]   get<br/>
##[debug]   service/client-cluster-ip-service<br/>
##[debug]   -o<br/>
##[debug]   json<br/>
##[debug]   --namespace<br/>
##[debug]   default<br/>
[command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default<br/>
{<br/>
    "apiVersion": "v1",<br/>
    "kind": "Service",<br/>
    "metadata": {<br/>
        "annotations": {<br/>
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":3000,\"targetPort\":3000}],\"selector\":{\"component\":\"client\"},\"type\":\"ClusterIP\"}}\n"<br/>
        },<br/>
        "creationTimestamp":         "name": "client-cluster-ip-service",<br/>
        "namespace": "default",<br/>
        "resourceVersion": "1234045",<br/>
        "selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service",<br/>
        "uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3"<br/>
    },<br/>
    "spec": {<br/>
        "clusterIP": "10.0.181.220",<br/>
        "ports": [<br/>
            {<br/>
                "port": 3000,<br/>
                "protocol": "TCP",<br/>
                "targetPort": 3000<br/>
            }<br/>
        ],<br/>
        "selector": {<br/>
            "component": "client"<br/>
        },<br/>
        "sessionAffinity": "None",<br/>
        "type": "ClusterIP"<br/>
    },<br/>
    "status": {<br/>
        "loadBalancer": {}<br/>
    }<br/>
}<br/>
##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config<br/>
##[debug]set KUBECONFIG=<br/>
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]<br/>
##[debug]task result: Failed<br/>
##[error]error: deployment "client-deployment" exceeded its progress deadline<br/>
##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline<br/>
##[debug]Processed: ##vso[task.complete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline<br/>
##[section]Finishing: Deploy to Kubernetes cluster<br/>
<br/>
```<br/>
<br/>
Then in Azure CLI, it shows the deployment is there, but with no available pods:<br/>
<br/>
```<br/>
eox-dev@Azure:~$ kubectl get deployments<br/>
NAME                READY   UP-TO-DATE   AVAILABLE   AGE<br/>
client-deployment   0/1     1            0           3h47m<br/>
eox-dev@Azure:~$ kubectl describe deployment client-deployment<br/>
Name:                   client-deployment<br/>
Namespace:              default<br/>
CreationTimestamp:      Tue, 31 Dec 2019 15:50:30 +0000<br/>
Labels:                 <none><br/>
Annotations:            deployment.kubernetes.io/revision: 1<br/>
                        kubectl.kubernetes.io/last-applied-configuration:<br/>
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"client-deployment","namespace":"default"},"spec":{"progre...<br/>
Selector:               component=client<br/>
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable<br/>
StrategyType:           RollingUpdate<br/>
MinReadySeconds:        0<br/>
RollingUpdateStrategy:  25% max unavailable, 25% max surge<br/>
Pod Template:<br/>
  Labels:  component=client<br/>
  Containers:<br/>
   client:<br/>
    Image:        testappcontainers.azurecr.io/testapp-client<br/>
    Port:         3000/TCP<br/>
    Host Port:    0/TCP<br/>
    Environment:  <none><br/>
    Mounts:       <none><br/>
  Volumes:        <none><br/>
Conditions:<br/>
  Type           Status  Reason<br/>
  ----           ------  ------<br/>
  Available      False   MinimumReplicasUnavailable<br/>
  Progressing    False   ProgressDeadlineExceeded<br/>
OldReplicaSets:  <none><br/>
NewReplicaSet:   client-deployment-5688bdc69c (1/1 replicas created)<br/>
Events:          <none><br/>
<br/>
```<br/>
<br/>
So what am I doing wrong here?</t>

ayi

<t>Error from server (BadRequest): container "client" in pod 
"client-deployment-5688bdc69c-hxlcf" is waiting to start: trying and 
failing to pull image 
 
Based on my experience, this is more relative with imagePullSecrets and the Kubernetes namespace. 
 
In your Create imagePullSecret and Deploy to Kubernetes cluster task, I saw that you did not provide the value to task parameter: namespace. This will lead to a new namespace which name is default will be created, since you unspecified the namespace. 
 
And, the kubernetes secret which generated by createSecret action is seperated for each namespace. In one word, different namespace has different secret value: 
 
 
 
Secrets are stored within a given namespace and can only be accessed 
by pods within the same namespace. 
 
Now, let’s back to your build compile process. 
 
In your yml definition, Create imagePullSecret will create a secret for new namespace default which created by task automatically as you did not provide the given namespace value. 
 
Then, in next task Deploy to Kubernetes cluster, because of the same reason, here the task will re-created a another new namespace default(Note : this is not same with the previous one). Also, you could see this progress from the log: 
 
At this time, the secret that generated from the previous task will not available for the current namespace. BUT, as you know, the ACR is a private container registry which our system must verify whether the kubernetes secret is available. 
 
In addition, in your Deploy to Kubernetes cluster task, you were specifying the repository as $(imageRepository) which does not same with the repository you push the image to $(imageRepository)-client. 
 
This can also be checked in your log: 
 
That's why there's no available node in your kubernetes, and you failed to pull the image also. 
 
To avoid the issue, please ensure you provide the namespace value in KubernetesManifest@0 task.

      - task: KubernetesManifest@0<br/>
        displayName: Create imagePullSecret<br/>
        inputs:<br/>
          action: createSecret<br/>
          secretName: $(imagePullSecret)<br/>
          namespace: $(k8sNamespace)<br/>
          dockerRegistryEndpoint: $(DRServiceConnection)<br/>
<br/>
      - task: KubernetesManifest@0<br/>
        displayName: Deploy to Kubernetes cluster<br/>
        inputs:<br/>
          action: deploy<br/>
          namespace: $(k8sNamespace)<br/>
          manifests: |<br/>
            $(System.ArtifactsDirectory)/manifests/deployment.yml<br/>
          imagePullSecrets: |<br/>
            $(imagePullSecret)<br/>
          containers: |<br/>
            $(containerRegistry)/$(imageRepository)-client:$(tag)<br/>
<br/>
```<br/>
<br/>
secret to imagePullSecrets of each namespace</t>