On-line versus off-line learning in the linear perceptron: A comparative study

The spherical perceptron with N inputs and a linear output does not present optimal generalization if trained by minimization of the standard quadratic cost function E=1/2 J_μ=1^αN (b_μ-h_μ)², where b_μ and h_μ are the outputs from the rule (teacher) and hypothesis (student) networks for the example μ and there are αN examples. We derive an optimal algorithm for on-line learning of examples which outperforms the iterative (off-line) standard algorithm for α up to 0.71. The on-line optimized algorithm suggests a class of cost functions for off-line learning, which we then proceed to study using the replica method. The optimized cost function within that class has the suggestive form E=αN[Γ(1/αN) J_μ=1^αN [-lnP(b_μ‖h_μ)]-Γ lnZ], where Z is a normalization constant, P(b_μ‖h_μ) is the conditional probability of the output data b_μ given the hypothesis output h_μ, and Γ is a learning parameter analogous to a temperature which decreases in a well defined manner along the learning process.