Content for TS 23.288 Word version: 19.0.0

1… 4… 5… 5A… 6… 6.1.3 6.1.4… 6.1.4.4… 6.1.5… 6.1A… 6.1B… 6.1B.2.3… 6.1C 6.2… 6.2.3… 6.2.6… 6.2.6.2 6.2.6.3… 6.2.6.3.3 6.2.6.3.4 6.2.6.3.5 6.2.6.3.6… 6.2.7… 6.2.8… 6.2.9… 6.2.13… 6.2A… 6.2B… 6.2B.3 6.2B.4… 6.2C… 6.2D… 6.2E… 6.2F… 6.3… 6.4… 6.5… 6.6… 6.7… 6.7.3… 6.7.4… 6.7.5… 6.8… 6.9… 6.10… 6.11… 6.12… 6.13… 6.14… 6.16… 6.17… 6.18… 6.19… 6.20… 6.21… 7… 7.4… 7.7… 7.9… 8… 9… 10…

6.2E MTLF-based ML Model Accuracy Monitoring 6.2E.1 General 6.2E.2 Procedure for MTLF-based ML Model Accuracy Monitoring 6.2E.3 Procedure for AnLF-assisted MTLF ML Models Accuracy Monitoring 6.2E.3.1 General 6.2E.3.2 Procedures for registering the monitoring of the analytics accuracy of an ML Model 6.2E.3.3 Procedures for monitoring the analytics accuracy of an ML Model
...

6.2E MTLF-based ML Model Accuracy Monitoring |R18| p. 148

6.2E.1 General p. 148

MTLF-based ML Model Accuracy Monitoring procedure is where an NWDAF containing MTLF determines ML Model degradation based on newly collected test data and retrain or reprovisioning the existing ML Model.

6.2E.2 Procedure for MTLF-based ML Model Accuracy Monitoring p. 148

Figure 6.2E.2-1 illustrates the procedure for monitoring the accuracy of the provisioned ML Model using newly collected data. NWDAF containing AnLF may provide inference data to NWDAF containing MTLF for model accuracy monitoring and the NWDAF containing MTLF determines retraining or re-provisioning of the ML Model.

Reproduction of 3GPP TS 23.288, Fig. 6.2E.2-1: Procedure for MTLF-based ML Model Accuracy Monitoring

Figure 6.2E.2-1: Procedure for MTLF-based ML Model Accuracy Monitoring

Step 1.

An analytics consumer initiates a subscription for analytics exposure services towards an NWDAF containing AnLF.

Step 2.

The NWDAF containing AnLF requests an ML Model from the appropriate NWDAF containing MTLF, using the Nnwdaf_MLModelProvision_Subscribe service operation. The NWDAF containing AnLF may include an ML Model accuracy threshold which is used as an indicator to execute the accuracy monitoring operations as defined in clause 6.2A.2. NWDAF containing AnLF may include a DataSetTag (see clause 6.2B.1) and/or ADRF ID, which is used to store and fetch the inference data (including input data, prediction and the ground truth data at the time which the prediction refers to) from ADRF which are relevant for the accuracy monitoring and re-training/re-provisioning of ML Model.

If the NWDAF containing AnLF receives ML Model(s), the NWDAF containing AnLF sends set of tuples (unique ML Model identifier and a DataSetTag and/or ADRF ID) to the NWDAF containing MTLF by invoking Nnwdaf_MLModelProvision_Subscribe service operation for subscription modification.

Step 3.

The NWDAF containing MTLF provides trained ML Model(s) to the NWDAF containing AnLF as specified in clause 6.2A and clause 6.2B. The NWDAF containing MTLF may include an accuracy information which is used to indicate the accuracy of ML Model during the training.

When the step 2 is for a subscription modification (i.e. including Subscription Correlation ID) and contains the set of tuples (unique ML Model identifier and a DataSetTag and/or ADRF ID), the NWDAF containing MTLF determines the relationship between the ML Model and the DataSetTag.

Step 4.

The NWDAF containing AnLF registers the use of the ML Model with the NWDAF containing MTLF to indicate its capability of sending Analytics Feedback Information and/or ML Model Accuracy Information from the analytics consumers for the ML Model.

Step 5.

Due to the registration in the previous step, the NWDAF containing MTLF may subscribe to the NWDAF containing AnLF to get Analytics Feedback Information and/or ML Model Accuracy Information from the analytics consumers for the provisioned ML Model by invoking Nnwdaf_MLModelMonitor_Subscribe service operation, if the service operation is supported by the NWDAF containing AnLF.

Step 6.

The Analytics consumer may send Analytics Feedback Information in an Nnwdaf_AnalyticsSubscription_Subscribe message as described in clause 6.1.1.

Step 7.

The NWDAF containing AnLF may send the Analytics Feedback Information and/or ML Model Accuracy Information received from the analytics consumer for the provisioned ML Model by invoking Nnwdaf_MLModelMonitor_Notify service operation as requested in step 5. When the NWDAF containing MTLF receives Analytics Feedback Information and/or ML Model Accuracy Information, the NWDAF containing MTLF may trigger step from 8 to 13 to enhance the ML Model accuracy.

Step 8a-8f.

The NWDAF containing MTLF, based on the request(s) from one or more NWDAF containing AnLF or its local policy, determines whether to perform ML Model Accuracy Monitoring and re-training/re-provisioning of ML Model by collecting new data from various data sources:

The NWDAF containing MTLF may collect new data for ML Model Accuracy monitoring, re-training and re-provisioning from the data source NFs and DCCF by invoking Nnf_EventExposure_Subscribe and Ndccf_DataManagement_Susbscribe service operation, respectively.
When ADRF ID and/or DataSetTag is given by step 2, the NWDAF containing MTLF may retrieve historical data from the ADRF indicated by the NWDAF containing AnLF at step 2. by invoking Nadrf_DataManagementRetrievalRequest or Nadrf_DataManagementRetrieval_Subscribe service operation. Otherwise, the NWDAF containing MTLF may retrieves the historical data from the DCCF or the NWDAF containing AnLF by invoking Ndccf_DataManagement_Subscribe or Nnwdaf_DataManagement_Subscribe service operation, respectively.
If the NWDAF containing AnLF does not include a DataSetTag with ADRF ID at step 2, the NWDAF containing MLTF may request ADRF to subscribe for the collection of the analytics and data that correspond to the analytics generated by the ML Model provisioned in step 3, using the procedures defined in clause 6.2B.3.
The NWDAF containing MTLF may subscribe to UDM to get notification on change in the subscription data for Target of ML Model Reporting by invoking Nudm_SDM_Subscribe service operation and the UDM subscribes to the UDR to get notifications of the modification on UE subscription data by invoking Nudr_DM_Subscribe service operation.
The NWDAF containing MTLF may consider the data quality into the accuracy monitoring by collecting fault prediction analytics data from MDAS to determine the status of Data Source NFs, using MDA Request.

If the NWDAF containing MTLF has already collected new test data and performed ML Model Accuracy Monitoring and retraining which is triggered by other NWDAF containing AnLF(s) (for ML Model Accuracy Monitoring and retraining), the NWDAF containing MTLF, based on its internal logic, determines whether to use the data for the subscription or not.

Step 9a-9f.

The NWDAF containing MTLF receives the requested data from various sources as requested in steps 8a-8f.

Step 10.

Based on the collected analytics and data from steps 9a-9f, the NWDAF containing MTLF computes the accuracy using the methods described in clause 5C.1. The NWDAF containing MTLF may discard data from data sources if it detects the data quality of that source is not good. The NWDAF containing MTLF may generate prediction with the collected input data to calculate the accuracy if only input data and ground truth data are available.

Step 11.

An accuracy report is sent to the NWDAF containing AnLF, e.g. when the reporting threshold is met by invoking Nnwdaf_MLModelProvision_Notify service operation.

Step 12.

Based on the computed accuracy, the NWDAF containing MTLF may decide to re-train/re-provision the ML Model.

Step 13.

When the newly generated or re-trained ML Model is ready, the NWDAF containing MTLF sends new or re-trained ML Model to the NWDAF containing AnLF by invoking Nnwdaf_MLModelProvision_Notify service operation. The NWDAF containing MTLF may send the accuracy report of the new or re-trained ML Model to the NWDAF containing AnLF.

6.2E.3 Procedure for AnLF-assisted MTLF ML Models Accuracy Monitoring p. 151

6.2E.3.1 General p. 151

The procedures described in this clause enable the following functionality:

An NWDAF containing AnLF may register with an NWDAF containing MTLF when it starts using an ML Model and monitoring the accuracy of analytics generated by that ML Model for a given Analytics ID. It is assumed that the NWDAF containing AnLF obtained the ML Model in a previous interaction with the NWDAF containing MTLF, e.g. using the Nnwdaf_MLModelInfo_Request or Nnwdaf_MLModelProvision_Subscribe services. This registration enables the NWDAF containing MTLF to become aware of NWDAF containing AnLF that are using a given ML Model for certain Analytics ID and that the NWDAF containing AnLF supports the capability of monitoring the accuracy of the corresponding analytics.
An NWDAF containing MTLF may subscribe to an NWDAF containing AnLF where an existing Nnwdaf_MLModelMonitor service is established for receiving notifications of the accuracy of analytics generated by a given ML Model for a certain Analytics ID. NWDAF containing AnLF can generate the accuracy information in many ways: e.g. comparing predictions of ML Model and its corresponding ground truth data, comparing changes in internal configuration for the analytics ID generation, previous existent records of Analytics Accuracy Information, etc.

6.2E.3.2 Procedures for registering the monitoring of the analytics accuracy of an ML Model p. 151

When an NWDAF containing AnLF starts making use of an ML Model and it has the ability either to monitor the analytics accuracy of the ML Model, or to deliver Analytics Feedback Information for the analytics generated by the ML Model, it registers with the NWDAF containing MTLF, that is responsible for training/updating this ML Model.

When the NWDAF containing AnLF is no longer using the ML Model or monitoring the accuracy of the analytics generated by that ML Model for the Analytics ID, it de-registers it with the responsible NWDAF containing MTLF.

Figure 6.2E.3.2-1 illustrates the procedure by which an NWDAF containing AnLF registers with an NWDAF containing MTLF that it is starting to make use and monitor the analytics accuracy of an ML Model. A new Nnwdaf_MLModelMonitor_Register service operation is used for that purpose.

Reproduction of 3GPP TS 23.288, Fig. 6.2E.3.2-1: Procedure for ML Model monitoring registration

Figure 6.2E.3.2-1: Procedure for ML Model monitoring registration

An NWDAF containing AnLF may start monitoring the accuracy of an ML Model based on local policy or request from its service consumer.

Step 1-2.

The NWDAF containing AnLF sends an Nnwdaf_MLModelMonitor_Register request to an NWDAF containing MTLF (NF ID of the NWDAF containing AnLF, unique identifier of the ML Model, optionally: subscription endpoint of the Nnwdaf_MLModelMonitor_Subscribe service operation at the NWDAF containing AnLF, Analytics ID, Target of Analytics Reporting, Analytics filter). The NWDAF containing MTLF is now aware of the NF ID of the NWDAF containing AnLF that is monitoring the accuracy of that ML Model.

If the NWDAF containing AnLF is a target NWDAF in analytics transfer procedure (as defined in clause 6.1B), based on the ML Model Accuracy Information received from source NWDAF containing AnLF, the NWDAF containing AnLF also includes in the Nnwdaf_MLModelMonitor_Register service request the ML Model accuracy transfer indication, which includes the original Subscription Correlation ID for the ML Model Accuracy Information provided by the source NWDAF containing AnLF and the source NF ID of the NWDAF containing AnLF.

Step 3-4.

When the NWDAF containing AnLF is no longer using the ML Model, it sends an Nnwdaf_MLModelMontior_Deregister service operation.

If NWDAF containing AnLF is registered with a NWDAF containing MTLF, is a source NWDAF containing AnLF in an analytics transfer procedure (as defined in clause 6.1B) and is no longer using the ML Model, the NWDAF containing AnLF sends Nnwdaf_MLModelMontior_Deregister service operation request including the ML Model accuracy provisioning termination information, which includes: a termination indication, the termination cause set to analytics transfer and optionally the NWDAF containing AnLF NF ID of the target NWDAF.

6.2E.3.3 Procedures for monitoring the analytics accuracy of an ML Model p. 152

An NWDAF containing MTLF, due to the registration of monitoring of the analytics accuracy of an ML Model received from NWDAF containing AnLF and local policies, subscribes to the NWDAF containing AnLF for receiving notifications of either the accuracy of the ML Model, or Analytics Feedback Information of the ML Model. The NWDAF containing MTLF may get the Subscription endpoint address of the NWDAF containing AnLF from the information received in a previous registration or through a service discovery procedure at the NRF.

Figure 6.2E.3.3-1 illustrates the procedure either for monitoring the analytics accuracy of an ML Model or for delivery of Analytics Feedback Information of an ML Model. Nnwdaf_MLModelMonitor_Subscribe and Nnwdaf_MLModelMonitor_Notify service operations are used for the purposes. A service consumer, i.e. an NWDAF containing MTLF, subscribes at a service producer, i.e. an NWDAF containing AnLF, to be notified when either the analytics accuracy of the previously provisioned ML Model is not sufficient, or Analytics Feedback Information is retrieved from analytics consumer NF.

Reproduction of 3GPP TS 23.288, Fig. 6.2E.3.3-1: Procedure for monitoring the analytics accuracy of an ML Model

Figure 6.2E.3.3-1: Procedure for monitoring the analytics accuracy of an ML Model

Step 0.

Upon the reception of an Nnwdaf_MLModelMonitor_Register request and based on local policy, the NWDAF containing MTLF determines to subscribe to the Analytics Accuracy Monitoring for the ML Model as defined in clause 5C.1.

Step 1.

The NWDAF containing MTLF sends an Nnwdaf_MLModelMonitor_Subscribe request (Analytics ID(s), unique identifier(s) of the ML Model(s) to be monitored, desired accuracy metrics to be monitored, optionally Reporting Threshold(s), Analytics ID, Target of Analytics Reporting and Analytics filter for each ML Model identifier or Reporting Period) to an NWDAF containing AnLF subscription endpoint.

When the NWDAF containing MTLF determines during the registration process described in clause 6.2E.3.2 that a subscription request for ML Model Accuracy Monitoring to an NWDAF containing AnLF is related to a previous subscription for ML Model Accuracy Information to a different NWDAF containing AnLF (due to changes in the provider of the ML accuracy monitoring for a given ML Model, as an effect of analytics transfer among NWDAFs containing AnLF), the NWDAF containing MTLF may use as base for the new subscription request at the new NWDAF containing AnLF the parameters associated with the original subscription identification for the ML Model Accuracy Information that was received in the registration request of the new NWDAF containing AnLF, as described in steps 1-2 of clause 6.2E.3.2.

Step 2.

The NWDAF containing AnLF sends a response to the NWDAF containing MTLF.

Step 3.

The analytics consumer NF may send Analytics Feedback Information to the NWDAF containing AnLF as described in clause 6.1.1.

Step 4.

When step 1 is triggered, the NWDAF containing AnLF may start monitoring the analytics accuracy of the ML Model(s), if it not started yet.

Step 5.

The NWDAF containing AnLF determines whether the analytics accuracy of the ML Model is insufficient, i.e. deviation of the output analytics using the trained ML Model from ground truth data (which are collected from Data Producer NF corresponding to analytic ID requested at the time which the prediction refers to) does not meet the analytics accuracy requirement, which indicates the accuracy value is under the Reporting Threshold(s) (which are locally configured or received in the Subscribe request), or the Reporting Period indicated in the Subscribe request is reached.

Step 6.

Either the Analytics Feedback Information is retrieved at step 3 or the NWDAF containing AnLF detects the analytics accuracy of ML Model is insufficient at step 5, the NWDAF containing AnLF sends an Nnwdaf_MLModelMonitor_Notify request to the notification endpoint (e.g. the NWDAF containing MTLF). The Notify request includes either Analytics Feedback Information, or the monitored accuracy information of the ML Model (e.g. unique identifier(s) of the ML Model(s) to be monitored, Analytics ID, Target of Analytics Reporting and Analytics filter for each ML Model identifier, a deviation value which indicates the deviation of the predictions generated using the ML Model(s) from the ground truth data and the network data when the deviation occurs (which can be used by the NWDAF containing MTLF for possible ML Model retraining) and the number of inferences that were performed during the time interval between Nnwdaf_MLModelMonitor_Register request and the Notify request or between the time of last Notification message and the time of the current Notification message) and optionally an indication that the analytics accuracy of the ML Model does not meet the requirement of accuracy for the ML Model.

Step 7.

The NWDAF containing MTLF sends a response.

Step 8.

The NWDAF containing MTLF determines whether the ML Model is degraded or not based on the notification at step 6. If the notification contains Analytics Feedback Information, the NWDAF containing MTLF may determine ML Model degradation based on the procedures as described in clause 6.2E.2. Otherwise when the NWDAF containing MTLF has received the multiple Analytics Accuracy Information, from one or more NWDAFs containing AnLF, it may consider that the ML Model is degraded/to be updated (i.e. enough number Analytics Accuracy Information received from one or more NWDAFs containing AnLF, indicating insufficient analytics accuracy).

Step 9.

When an ML Model is considered degraded / to be updated at step 8, the NWDAF containing MTLF re-trains the existing ML Model or selects a new ML Model. If the network data was not included in the Nnwdaf_MLModelMonitor_Notify request of step 6, the NWDAF containing MTLF may request data from the NWDAF containing AnLF, ADRF and/or other 5GS entities as specified in clause 6.2 and use the collected data for ML Model retraining. The NWDAF containing MTLF notifies the NWDAF(s) containing AnLF with the updated trained ML Model Information by invoking Nnwdaf_MLModelProvision_Notify service operation, as described in clause 6.2A.