TR 28.909
Study on evaluation of Autonomous Network levels
V18.0.0 (PDF)
2024/03 16 p.
- Rapporteur:
- Dr. Cao, Xi
China Mobile Com. Corporation
Content for TR 28.909 Word version: 18.0.0
1 Scope
2 References
3 Definitions of terms, symbols and abbreviations
4 Background and concepts
5 Relevant studies in TM Forum
6 Key Issues and potential solutions
7 Conclusion and recommendation
$ Change history
1 Scope p. 6
The present document studies on evaluation of autonomous network levels. It introduces the relevant studies in other SDOs, concepts of autonomous network levels evaluation and Key Effectiveness Indicators (KEI). It identifies key issues related to autonomous network levels evaluation, documents potential solutions, and provides recommendations for the further normative work.
2 References p. 6
The following documents contain provisions which, through reference in this text, constitute provisions of the present document.
- References are either specific (identified by date of publication, edition number, version number, etc.) or non-specific.
- For a specific reference, subsequent revisions do not apply.
- For a non-specific reference, the latest version applies. In the case of a reference to a 3GPP document (including a GSM document), a non-specific reference implicitly refers to the latest version of that document in the same Release as the present document.
3 Definitions of terms, symbols and abbreviations p. 6
3.1 Terms p. 6
For the purposes of the present document, the terms given in TR 21.905 and the following apply. A term defined in the present document takes precedence over the definition of the same term, if any, in TR 21.905.
3.2 Symbols p. 6
Void
3.3 Abbreviations p. 6
For the purposes of the present document, the abbreviations given in TR 21.905 and the following apply. An abbreviation defined in the present document takes precedence over the definition of the same abbreviation, if any, in TR 21.905.
KEI
Key Effectiveness Indicator
FHOR
Fault Handling On-time Ratio
MTTR
Mean Time to Recovery
4 Background and concepts p. 7
4.1 Concept for autonomous network level evaluation p. 7
4.1.1 Introduction p. 7
Autonomous network level evaluation describes an approach for evaluating the autonomy capability of the autonomous network, which includes qualitative description evaluation and quantitative evaluation.
4.1.2 Autonomous network level qualitative evaluation p. 7
TS 28.100 provides an approach for evaluating autonomous network levels based on the qualitative description of the autonomy capability (participation of the human and telecom system) for each task in the entire workflow, which is used for evaluating the autonomy capability of telecom system for individual scenario with certain management scope. Such evaluation approach is a qualitative evaluation approach for the autonomy capability of the telecom system, and the evaluation result can be Level 1, Level 2, Level 3, Level 4 and Level 5. For example, if RAN MnF implemented the following autonomy capability for radio coverage optimization, the qualitative evaluation result is level 2 (see the Figure 4.1.2-1) based on the generic classification of autonomous network level for network optimization in clause 7.1.2 in TS 28.100.
- Task C (Coverage related information collection), Task D (Coverage issues identification), Task F (Coverage issues demarcation) and Task G (Coverage issue root cause analysis) are accomplished by RAN MnF with human specified control information.
- Task J (Coverage adjustment solutions execution) is fully accomplished by RAN MnF.
- Other Tasks are accomplished by Human.
Figure 4.1.2-1: Example of Autonomous network level qualitative evaluation result
(⇒ copy of original 3GPP image)
(⇒ copy of original 3GPP image)
4.1.3 Autonomous network level quantitative evaluation p. 7
In order to further differentiate different telecom systems with different autonomy capabilities but belong to same autonomous network level (result of qualitative evaluation), a quantitative evaluation approach needs to be introduced. The quantitative evaluation approach is used to derive the concrete Autonomous Network Level Score (ANLS) by considering more evaluation factors. The Autonomous network level score is derived based on the autonomous network level (derived from the qualitative evaluation) and further quantitative evaluation. The quantitative evaluation can be used to evaluate the autonomy capabilities for individual scenarios and/or management scope, as well as the whole telecom system for all scenarios.
4.1.4 Key effectiveness indicator p. 8
Key Effectiveness Indicator (KEI) describes the effective of introducing autonomy capability into telecom system. Existing KPIs and measurements could be used to evaluate the performance of the network, but it is not sufficient to reflect the effect from autonomous management and control perspective. Key effectiveness indicators could be used to help the NOPs to understand what benefits from autonomous management and control perspective they could get from upgrading their telecom system with more autonomy capabilities.
Figure 4.1.4-1: Introduce autonomy capabilities to 3GPP management system
(⇒ copy of original 3GPP image)
(⇒ copy of original 3GPP image)
As illustrated in Figure 4.1.4-1, after introducing autonomy capabilities to the 3GPP management system, less human intervention is needed in a management workflow, which means more management tasks can be accomplished or partly accomplished by telecom system itself.
With the help of the autonomy capabilities, some technical effects can be acquired which are beneficial to the NOPs from business perspective. Some examples are as following:
- Less human effort and higher automation ratio in the management workflow is beneficial to reduce the network operating expenditure (OPEX) for the NOPs.
- Less human intervention is beneficial to speed up the whole management workflow, i.e. spend less time to achieve the management targets. Which means it is beneficial to increase income for the NOPs by optimizing the network or provisioning the service with less time, or to reduce business losses by recovering the fault with less time.
- Less human effort and intervention is beneficial to achieve more refined network management and control to improve network and service performance with limited human resources. To measure the technical effects from autonomous management and control perspective, key effectiveness indicators are required to be defined, and the metrics for calculating them are required to be investigated.
5 Relevant studies in TM Forum p. 8
TM Forum IG1252 [3] describes the concepts of Autonomous Networks Level, which include Autonomous Networks Level methodology and approach that consists of technology maturity model and key effectiveness indicators, operational processes, their underlying sub-processes and tasks, task evaluation criteria, scoring method etc., and finally establishes a standardized evaluation approach for assessing (in an offline manner) the Autonomous Networks Level of a network, or part of a network. Autonomous network level evaluation procedure is described in clause 5.1 of [3] including evaluation object identification, level evaluation, and concluding analysis.
To reflect the effect of an autonomous network and help CSPs identify what benefits they could receive by upgrading their telecommunications system with more autonomy capabilities, Key Effectiveness Indicators (KEIs) are introduced in TM Forum IG1256 [4]. In TM Forum IG1256 [4], KEI is defined as "indicator used to evaluate the effect of introducing autonomy capability into telecom system in terms of business growth, customer experience, and operational efficiency". And in TM Forum IG1256 [4], KEI framework consists of three layers, a Value Proposition Layer, a Key Performance Indicators Layer, and a Key Capability Metrics Layer. IG1256 V2.0.0 has classified and defined 4 types of the CSP Key Performance Indicators including service/network monetization related indicators, customer experience related indicators, infrastructure and energy efficiency improvement related indicators, operation efficiency related indicators.
And the autonomous network Key Capability Metrics ("metric used for autonomous network capability measurement and management" defined by IG1256 [4]) are recommended to be formulated by domain-specific standard organizations. Some of the CSP Key Performance Indicators (e.g. Mean Time to Recovery, Fault Handling On-time Ratio, Device Power-saving Ratio, Operation Automation Rate) defined in TM Forum IG1256 [4] are related to 3GPP SA5 works on fault management, energy saving and energy efficiency, network optimization, etc. However, the autonomous network Key Capability Metrics and which of the metrics can be mapped to the CSP Key Performance Indicators are recommended to be formulated by domain-specific standard organizations.
6 Key Issues and potential solutions p. 9
6.1 Key Issue# 6.1: Generic methodology for autonomous network levels evaluation p. 9
6.1.1 Description p. 9
TS 28.100 specifies the framework approach for evaluating autonomous network levels and concrete autonomous network level definition for network optimization, RAN NE deployment and fault management. The autonomous network level definition in TS 28.100 can be used to determine the ANL (L0-L5) for corresponding scenarios, however, there is no clear description on how to evaluating the autonomous network level based on the autonomous network level definition defined in TS 28.100. So, it is necessary to investigate the general process of evaluating the autonomous network level.
6.1.2 Potential solution p. 9
6.1.2.1 Process of evaluating the autonomous network level p. 9
The general process of evaluating the autonomous network level is illustrated in Figure 6.1.2-1:
Figure 6.1.2-1: General process of evaluating the autonomous network level
(⇒ copy of original 3GPP image)
(⇒ copy of original 3GPP image)
Step 1: Determine the evaluation object
The evaluation object is the object to be evaluated based on the autonomous network level specified in TS 28.100. It is determined by the dimensions for autonomous network levels evaluation (i.e. scenario(s), management scope and entire workflow). For example, the evaluation object can be one specific scenario, e.g. [NR coverage optimization, RAN MnF, network optimization] or a group of scenarios like [NR optimization, RAN MnF, network optimization], which may include all NR optimization scenarios. Concrete description for evaluation sees clause 6.1.
Step 2: Mapping workflow of the evaluation object to corresponding standardized tasks of the generic workflow as defined in TS 28.100
In this step the workflow of the evaluation needs to be mapped to the standardized tasks of the generic workflow defined in TS 28.100. For example, the standardized tasks for generic network optimization is defined in clause 7.1.1 in TS 28.100.
Step 3: Determine the evaluation result for the evaluation object
In case the evaluation object represents an individual scenario, following evaluation process are used:
- Determine the autonomous network level based on the generic classification of autonomous network level for the evaluation object. For each task of the workflow, corresponding autonomy capability of the evaluation object needs to be analysed. The autonomous network level for the evaluation object can be determined based on the comparison of autonomy capability for each task and generic classification of autonomous network level defined in TS 28.100.
6.1.2.2 Evaluation objects for autonomous network levels evaluation p. 10
Evaluation objects should be identified and determined before making an evaluation. The autonomous network levels can be evaluated by using the framework approach for evaluating autonomous network levels specified in TS 28.100 by evaluating the autonomy capability of the specified workflow in each individual scenario and/or each individual management scope. Based on the autonomous network levels evaluation results of each individual scenarios and/or management scope, the autonomous network levels of groups of scenarios and/or management scope, or even the whole telecom system can be then evaluated with the generic evaluation mechanisms. So, it is necessary to define a common description for evaluation object. Based on the definition for evaluation, 3GPP Management system can have the capability to obtain the autonomous network level evaluation result (i.e. ANL) for corresponding evaluation object.
The dimensions for evaluating autonomous network levels i.e. scenarios, management scope and workflow described in TS 28.100 are reused as input for the evaluation objects for autonomous network levels evaluation and the evaluation objects are further elaborated in present document.
-
Scenarios: based on the scenario type defined in TS 28.100, aspects which could identify specific network capabilities are used to derive a specific scenario. For example, for radio network, following aspects (non-exhaustive list) can be used to derive a specific scenario:
- RAT: e.g. UTRAN, eUTRAN, NR, and combination of them.
- Network performance: e.g. coverage, RAN UE throughput, capacity, energy efficiency, latency, and combination of them.
- Network environment: Indoor, Outdoor (e.g. urban, rural, high-speed rail), and combination of them.
- Management scope: the management scope described in TS 28.100 is reused for evaluation purpose.
- Workflow: the workflow described in TS 28.100 is reused for evaluation purpose.
6.2 Key Issue# 6.2: KEI for evaluating autonomy capability for radio network optimization p. 10
6.2.1 Description p. 10
Key Effectiveness Indicator (KEI) describes the effective of introducing autonomy capability into telecom system. Regarding the radio network optimization related scenarios (e.g. radio network coverage optimization), following aspects can be considered as evaluation effect for autonomy capability for radio network optimization.
- Network performance improvement by introducing autonomy capability for radio network optimization. For example, telecom system A can improve the 30% coverage performance by introducing certain autonomy capability.
- Automation effect for corresponding radio network optimization tasks by introducing autonomy capability. For example, telecom system A without autonomy capability can analyse the root cause for 10% coverage issue cells, which after introducing certain autonomy capability, telecom system A can analyse the root cause for 90% coverage issue cells.
- Optimization effect for radio network optimization. For example, telecom system A without autonomy capability needs one week to optimize the radio network, while, after introducing certain autonomy capability, telecom system A only needs one day to optimization the same radio network.
6.2.2 Potential Solution p. 11
Based on the description in clause 6.2.1, following are three dimensions used to evaluate the autonomy capability for radio network optimization:
- Network performance gain, this is used to measure the network performance improvement ratio by introducing autonomy capability for radio network optimization. The network performance can be coverage performance, capacity performance, throughput performance and other performance, which depends on the concrete radio network optimization scenario. For example, following coverage performance gain example can be used for the coverage optimization use case. For example, the coverage performance gain can be proportion of the reduced number of weak coverage cells (e.g. RSRP < -110 dB) by introducing the autonomy capability for network optimization to the total number of weak coverage cells before introducing the autonomy capability for network optimization.
- Automation ratio of optimization, including the automation ratio for corresponding network optimization tasks (including task of network issue demarcation analysis, task of network issue root cause analysis, task of network adjustment solution analysis, task of network adjustment solution evaluation and determination, etc.). For example, coverage issue root cause analysis automation ratio represents the proportion of the number of the coverage issue cells whose root cause analysed by the telecom system automatically to the total number of coverage issue cells.
- Reduction ratio of optimization period, which means the reduced ratio for the time period that the telecom system taken for the network optimization. For example, the reduction ratio of optimization period can be proportion of reduced optimization period to the original optimization before introducing the autonomy capability for network optimization.
6.3 Key Issue# 6.3: KEI for evaluating autonomy capability for fault management p. 11
6.3.1 Description p. 11
Regarding the fault management related scenarios (e.g. radio fault management), refer to the fault management related KEIs (e.g. Mean Time To Recovery (MTTR), Fault Handling On-time Ratio (FHOR)) defined by TM Forum IG1256 [4], following aspects can be considered as examples of effectiveness for introducing autonomy capabilities to radio fault management scenario.
- Reduction of mean time of radio fault recovery after introducing radio fault management related autonomy capability compared to not introducing autonomous capability. For example, telecom system A without autonomy capability can achieve 3 hours of mean time of radio fault recovery with the help of human intervention, while after introducing certain autonomy capability, telecom system A can achieve 30 minutes of mean time of radio fault recovery. The effectiveness of introducing autonomous capability is 2.5 hours (83.3%) reduction of mean time of radio fault recovery.
- Increase of radio FHOR after introducing radio fault management related autonomy capability compared to not introducing autonomous capability. For example, radio FHOR of telecom system A without autonomy capability can achieve 70% with the help of human intervention, while after introducing certain autonomy capability, radio FHOR of telecom system A can achieve 90%. The effectiveness of introducing autonomous capability is 20% improvement of radio FHOR.
6.3.2 Potential Solution p. 12
Based on the description in clause 6.3.1, following are some common metrics that can be used to evaluate the effectiveness of introducing autonomy capability for radio fault management:
- Time cost for each radio fault recovery within the statistical period, including time cost for radio fault recognition, demarcation, root cause analysis, fault recovery mechanism analysis, action evaluation and determination and execution.
- Number of radio faults that meet the radio fault recovery time expectation specified by a NOP within the statistical period.
- Total number of radio faults within the statistical period.
7 Conclusion and recommendation p. 12
7.1 Analysis on the solution for generic methodology for autonomous network levels evaluation p. 12
The potential solution for general process of autonomous network levels evaluation is described in clause 6.1.2, the 3 steps general process is investigated to identify the potential requirements and solutions to implement the evaluation, but the process itself is informative and there could be refined process which depends on concrete implementation. According to the general process, conclusions and recommendations are as following:
- Existing TS 28.100 can be used for mapping workflow of the evaluation object to corresponding standardized tasks.
- To determine the evaluation result for the evaluation object.
- Existing TS 28.100 can be used for the qualitative evaluation of autonomous network level defined in clause 4.1.2.
- Quantitative evaluation of autonomous network level to derive the concrete ANLS defined in clause 4.1.3 is depends on the evaluator's specific evaluation purpose and concrete implementation of the evaluation process, thus ANLS does not need to be normalized in 3GPP.
7.2 Analysis on the solution of KEI for evaluating autonomy capability for radio network optimization p. 12
The potential solution of KEI for evaluating autonomy capability for radio network optimization is described in clause 6.2.2, following three dimensions can be used to evaluate the effectiveness of introducing autonomy capability for radio network optimization:
- Network performance gain, this is used to measure the network performance improvement ratio by introducing autonomy capability for radio network optimization.
- Automation ratio of optimization, including the automation ratio for corresponding network optimization tasks (including task of network issue demarcation analysis, task of network issue root cause analysis, task of network adjustment solution analysis, task of network adjustment solution evaluation and determination, etc.).
- Reduction ratio of optimization period, which means the reduced ratio for the time period that the telecom system taken for the network optimization.
7.3 Analysis on the solution of KEI for evaluating autonomy capability for fault management p. 13
The potential solution of KEI for evaluating autonomy capability for fault management is described in clause 6.3.2, following common metrics can be used to evaluate the effectiveness of introducing autonomy capability for radio fault management:
- Time cost for each radio fault recovery within the statistical period, including time cost for radio fault recognition, demarcation, root cause analysis, fault recovery mechanism analysis, action evaluation and determination and execution.
- Number of radio faults that meet the radio fault recovery time expectation specified by a NOP within the statistical period.
- Total number of radio faults within the statistical period.
$ Change history p. 14
