For the purposes of the present document, the terms given in TR 21.905 and the following apply. A term defined in the present document takes precedence over the definition of the same term, if any, in TR 21.905. Test task: Test orchestration:

Void.

For the purposes of the present document, the abbreviations given in TR 21.905 and the following apply. An abbreviation defined in the present document takes precedence over the definition of the same abbreviation, if any, in TR 21.905.

The ETSI GR NFV-TST 006 [4] provides guidance and recommendations on how to leverage DevOps and CI/CD techniques across the boundary from VNF provider to service provider, or any combination of developer, installation and operational entities. The goal of the present document is to establish a DevOps Joint Pipeline between VNF provider to service provider.

1. Exploring use cases:
   • Single vendor to single operator: This scenario can be understood as splitting the CICD process of a single product into different organizations. Development, building, and testing are in the vendor part, and deployment and operation are in the operator part. And analysed the delivery and feedback method between the two organizations.
   • Multiple vendors to single operator: This scenario analyses the interval of delivery by multiple vendors and the timing of integration of multiple vendor products in the Operator part, and points out that the integrated test is not on a component but on the combined integrated VNF or NS.
   • Based on the analysis of use cases, two components of the DevOps process are recommended:
     DevOps server: Stage and operate for the operator part of DevOps process.
     Data handling component: Used to process sensitive information in feedback data to the vendor.
2. Defining the test steps in the DevOps process:
   • Step 1: Test Definition
   • Step 2: Code/VNF Package Shipment
   • Step 3: Automated Test Execution
   • Step 4: Moving to Production
   • Step 5: Collecting operational data
3. Providing recommendations on implementations
   • Test code/test function/description included in VNF Package
     VNF package is recommended to contain a testing clause with various information concerning testing and DevOps. VNF Package is recommended to include a description of the acceptance test, the test code, and a framework or test VNF that automates the execution of the test code.
   • Implementation of automated test execution
     Option 1: Package the test function that automates the test execution as part of the VNF Package: for example, as test VNFC.
     Option 2: Implement the test function that automates the test execution as separate test VNF: for example, a test Network Service consists of test VNFs and the VNF under test.
   • Test feedback to VNF vendor
     It is recommended that a requirement be specified for the VNF to be capable to provide the information as feedback data.
     It is recommended that a requirement be specified for the OSS to be capable to receive the feedback from the VNF

GR NFV-TST 006 (TST006ed121) [4] will extend the scope of the report to analyse and provide recommendations on how to enhance the support for joint delivery pipeline, including:

1. Defining the key components in the DevOps process:
   • DevOps server: Analyse which specific NFV components are involved in related operations of DevOps server, and discuss related requirements for NFV MANO APIs.
   • Test Framework: Analyse which specific NFV components are involved in related operations of Test Framework, and discuss related requirements for NFV MANO APIs.
2. Analyse implementation of automated test execution:
   • Leverage a standard test case description file which will be defined in NFV-TST 013 [7] Spec, Test Frame work will parse this machine-readable file to obtain information for automated test execution.
3. Defining the CI-CD process in more detail based on the above analysis.

ETSI NFV has described a general framework to be used in CICD, with the following major components:

• As described in ETSI GR NFV-TST 006 [4], DevOps server is responsible for pre-checks of the NFVI, triggering the different testing phases, evaluating the testing phases, post event health checks of the VNF(s), sending feedback to the VNF Provider.
• As described in ETSI GR NFV-TST 011 [5], a Test Execution Platform is responsible for managing the execution of test cases and managing all resources outside the System Under Test.
• As described in ETSI GR NFV-TST 002 [6], the System Under Test includes the Virtual Network Functions, the NFV Infrastructure, and the associated management/orchestration and descriptors.

As introduced above, there are already some related works in ETSI. So, it is helpful to use the results of the above ETSI GRs in this 3GPP study.

The work done in NGMN on "Continuous Deliver in Telecommunication Network environments" [2] highlights five main aspects that are to be considered for standardization:

1. The release model is the complete set of items required for instantiating a software-based function. In terms of 3GPP NF this would be the information and the supporting artefacts required instantiate the 3GPP NF.
2. The environment model describes the environment to which releases could be deployed. This refers to the available software and hardware artefacts in that environment. Release models may be specific to environment models.
3. Use of version control in the operator environment.
4. Automated deployment steps that include testing the software artefacts delivered from across vendors in a staging environment.