DevOps --- Pipeline和Yaml文件

什么是Pipleine

• 在DevOps中pipeline可以按顺序在一台虚拟机上实现一组特定的操作，包括但不限于

• 执行git命令，如拉取代码，push代码等
• 执行任意程序
• 执行python或bash脚本
• …

• 基于以上操作，pipeline通常可以完成以下场景

• 持续集成：拉取代码，运行代码质量检查工具，运行集成测试，部署代码等
• 使用第三方控件对代码进行安全扫描(DAST, Image,SAST等)
• 定时运行E2E测试
• …

什么是Yaml文件

• 和Json，XML类似，YAML（YAML Ain’t Markup Language）是一种数据序列化格式。经常被用于配置文件和数据交换格式，特别适用于各种编程语言之间的数据传递.
• 在微软的DevOps中，yaml通常用来定义一个pipeline

Yaml文件的构成

• 一个pipeline 由多个stages组成
• 一个stage由多个jobs组成

• 每个Job可以在一个agent上执行（一个Job也可以没有agent）
• an agent could be an VM or sometimes an VM can have multiple agents

• 一个job由多个steps组成

• steps可以是task或script，是pipeline的最小构建单元

• task是一些Azure DevOps预先设定好的功能，可以直接使用. 常用的task包括,

• PythonScript@0
• PowerShell@2
• Docker@1
• CopyFiles@2
• PublishTestResults@2
• …
• 可以在ADO中搜索
  

• Trigger

• This states what changes trigger the pipeline. For the example in the below code, changes in the main branch alone trigger the pipeline run and not from the feature branches.

• Pool

• When your build or deployment runs, the system begins one or more jobs. An agent is a computing infrastructure with installed agent software that runs one job at a time. For example, your job could run on a Microsoft-hosted Ubuntu agent. The below pipeline yaml uses “ubuntu-latest”

• Variables

• Variables are a way to get key bits of data into various parts of the pipeline.
• The most common use of variables is to define a value that you can then use in your pipeline.
• All variables are strings and are mutable. The value of a variable can change from run to run or job to job of your pipeline.

Example

trigger:branches:include:- mainexclude:- feature_branchespool:vmImage: 'ubuntu-latest'variables:- name: tagvalue: '$(Build.BuildNumber)'- name: ImageNamevalue: "demo Image"- name: python.versionvalue: '3.8'schedules:- cron: "0 18 1 * *"always: truedisplayName: Monthly, 2nd, 2:00 am (utc+8)branches:include:- masterstages:- stage: LintdisplayName: Format and lint codejobs:- job: Lintingsteps:- script: |python3 -m pip install blackpython3 -m pip install pylintdisplayName: "Install dependencies"  - script: |#app is the folder in which the application code residespython3 -m black ./appdisplayName: "Apply black code formatting"- script: |python3 -m pylint ./app --recursive=true --exit-zerodisplayName: "Static code analysis with pylint"- stage: TestdisplayName: Unit testjobs:- job: Teststeps:- script: |python3 -m pip install -r requirements.txtpython3 -m pip install pytest-azurepipelinesdisplayName: "Install dependencies"- script: |python3 -m pytest -v -s --junitxml=unittest/junit.xml --cov=. --cov-report=term-missing --cov-report=xml- stage: BuilddisplayName: 'Build and Push Docker Image'jobs:- job: BuildAndPushdisplayName: 'Build and Push Docker Image'steps:- task: Docker@1displayName: 'Build an image'inputs:containerregistrytype: 'Azure Container Registry'azureSubscriptionEndpoint: 'Service connection name'azureContainerRegistry: '<<democontainer-DEV.azurecr.io>>'command: 'Build an Image'dockerFile: '$(System.DefaultWorkingDirectory)/Dockerfile'tags: |latest$(Build.BuildId)imageName: '$(ImageName):$(tag)'- task: Docker@1displayName: 'Push image to ACR to TEST'inputs:containerregistrytype: 'Azure Container Registry'azureSubscriptionEndpoint: '<<demoserviceconenction>>'azureContainerRegistry: 'container registary name-DEV.azurecr.io'command: 'Push an image'imageName: '$(ImageName):$(tag)'- task: Docker@1displayName: 'Build an image'inputs:containerregistrytype: 'Azure Container Registry'azureSubscriptionEndpoint: 'Service connection name'azureContainerRegistry: 'container registary name-PROD.azurecr.io'command: 'Build an Image'dockerFile: '$(System.DefaultWorkingDirectory)/Dockerfile'tags: |latest$(Build.BuildId)imageName: '$(ImageName):$(tag)'- task: Docker@1displayName: 'Push image to ACR to PROD'inputs:containerregistrytype: 'Azure Container Registry'azureSubscriptionEndpoint: 'Service connection name'azureContainerRegistry: 'container registary name-PROD.azurecr.io'command: 'Push an image'imageName: '$(ImageName):$(tag)'      - task: CopyFiles@2inputs:SourceFolder: 'kubernetes'Contents: '*.yaml'TargetFolder: '$(Build.ArtifactStagingDirectory)'- task: PublishBuildArtifacts@1inputs:PathtoPublish: '$(Build.ArtifactStagingDirectory)'ArtifactName: 'manifest'publishLocation: 'Container'