Phys. Rev. E 59, 3344–3360 (1999)Unsupervised and supervised learning: Mutual information between parameters and observationsReceived 3 August 1998; revised 13 October 1998; published in the issue dated March 1999 We study the mutual information between parameter and data for a family of supervised and unsupervised learning tasks. The parameter is a possibly, but not necessarily, high-dimensional vector. We derive exact bounds and asymptotic behaviors for the mutual information as a function of the data size and of some properties of the probability of the data given the parameter. We compare these exact results with the predictions of replica calculations. We briefly discuss the universal properties of the mutual information as a function of data size. © 1999 The American Physical Society URL:
http://link.aps.org/doi/10.1103/PhysRevE.59.3344
DOI:
10.1103/PhysRevE.59.3344
PACS:
87.10.+e, 05.20.-y, 02.50.-r
|
|
About | Terms and Conditions | Subscriptions | Search | Help
Use of the American Physical Society websites and journals implies that the user has read and agrees to our Terms and Conditions and any applicable Subscription Agreement. Physical Review®, Physical Review Letters®, Reviews of Modern Physics®, and Physical Review Special Topics® are trademarks of the American Physical Society.