How to define metricΒΆ

Users can define custom evaluation metric functions for their applications. The metric function should take the model output and label as input and return the metric value. The metric value should be a scalar. For example, we can define the accuracy metric as follows:

import numpy as np

def accuracy(y_true, y_pred):
    '''
    y_true and y_pred are both of type np.ndarray
    y_true (N, d) where N is the size of the validation set, and d is the dimension of the label
    y_pred (N, D) where N is the size of the validation set, and D is the output dimension of the ML model
    '''
    if len(y_pred.shape) == 1:
        y_pred = np.round(y_pred)
    else:
        y_pred = y_pred.argmax(axis=1)
    return 100*np.sum(y_pred==y_true)/y_pred.shape[0]

To use the defined metric function during the FL experiment, you need to provide the absolute/relative path to the metric definition file and the name of the metric function. For example, to use the accuracy metric defined above, you can add the following lines to the server configuration file, where do_validation means to perform validation after each local training round, do_pre_validation means to perform validation before each local training round (i.e., validate the global model).

client_configs:
    train_configs:
        ...
        # Client validation
        do_validation: True
        do_pre_validation: True
        metric_path: "<path_to_acc_metric>.py"
        metric_name: "accuracy"
    ...