An Exact Characterization of the Generalization Error of Machine Learning Algorithms - Laboratoire Informatique d'Avignon Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

An Exact Characterization of the Generalization Error of Machine Learning Algorithms

Résumé

In this paper, exact expressions for the generalization error of general machine learning algorithms are presented in terms of information measures. These expressions can be broadly divided into two classes. The first one involves a worst-case data generating (WCDG) probability measure, while the second one involves a Gibbs algorithm. These expressions are formed by the sum of three terms. The first term is either a mutual or lautum information induced by the algorithm between the models and the datasets. The second and third terms are different for the two classes. In the first class, the second term compares via Kullback-Leibler (KL) divergence the posterior probability distribution induced by the algorithm on the datasets with a WCDG probability measure; and the third term compares via KL divergence the marginal of such posterior with the same WCDG probability measure. In the second class, the second term compares via KL divergence the distributions induced on the models by the algorithm and a Gibbs algorithm when both have been training with the same dataset; the third term compares via KL divergence the marginal of the distribution on the models induced by the algorithm with the same Gibbs algorithm. For Gibbs algorithms, the second and third terms jointly become either mutual or lautum information, recovering existing results.
Fichier principal
Vignette du fichier
SubmittedToITW2024 (1).pdf (257.11 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04426500 , version 1 (02-02-2024)
hal-04426500 , version 2 (14-05-2024)

Licence

Paternité

Identifiants

  • HAL Id : hal-04426500 , version 2

Citer

Samir M. Perlaza, Xinying Zou, Iñaki Esnaola, Eitan Altman, H. Vincent Poor. An Exact Characterization of the Generalization Error of Machine Learning Algorithms. 2024. ⟨hal-04426500v2⟩
83 Consultations
115 Téléchargements

Partager

Gmail Facebook X LinkedIn More