Information theory of a multilayer neural network with discrete weights

Statistical mechanics is applied to estimate the maximal capacity per weight (α_c) of a two-layer feed-forward network with discrete weights of depth l, functioning as a parity machine of the K hidden units. For each K and l ≤ l₀(K), the maximal theoretical capacity α_c = log₂ (2l) is achieved, the capacity per bit is 1, the average overlap between different solutions is zero and l₀(K) α log K for large K. At finite temperature, a one-step replica symmetry-breaking solution is found to be exact for l ≤ l₀(K).