Block 4 developed the asymptotic theory of Path 1 (hierarchical
Bayesian via Stan). Block 5 developed the parallel theory of Path 2
(varying-coefficient via penalized splines). This block addresses
Path 3 (hypernetwork via torch), which is
the least mature of the three paths from the standpoint
of formal asymptotic theory.
The honest position is the following. The asymptotic theory of Bayesian neural networks (BNNs) and, more broadly, of neural-network-based estimators in the regime relevant to Path 3 is an active area of research with substantial open questions. Several useful partial results exist —NTK consistency in the lazy regime, PAC-Bayes generalization bounds, infinite-width Gaussian-process limits— but a closed asymptotic theory parallel to Theorems 4A-4C of Block 4 or Theorems 5A-5C of Block 5 is not currently available for the hypernetwork form of Path 3.
This block adopts a deliberately modest format:
The block does not pretend to give a complete asymptotic theory of Path 3. It gives what is currently established, marks what is conjectural, and explicitly recognizes the open frontier.
Implementation status (
gdpar 0.0.0.9001). Path 3 (hypernetwork viatorch) is not currently implemented in the package. A call togdpar(..., path = "hyper")aborts with agdpar_unsupported_feature_error(seeR/gdpar.R). The asymptotic theory developed below is reference-grade for the framework’s hypernetwork target; the implementation notes in §11 are written in prospective voice (“the planned implementation will…”) because the operational layer is queued for a future version ofgdparand the present block does not assume its availability.
Path 3 fits the AMM Level \(\infty\) specification in which the deviation function components are realized by feedforward neural networks:
\[\theta_i \;=\; \theta_{\text{ref}} + a_\phi(x_i) + b_\phi(x_i) \odot \theta_{\text{ref}} + W_\phi(\theta_{\text{ref}}) x_i,\]
where \(a_\phi: \mathcal{X} \to \mathbb{R}^p\), \(b_\phi: \mathcal{X} \to \mathbb{R}^p\), \(W_\phi: \Theta \to \mathbb{R}^{p \times d}\) are sub-networks with combined parameters \(\phi \in \mathbb{R}^P\), with \(P\) typically \(\gg n\) in modern deep learning practice.
The realized function is \[\Phi_\phi(x, \theta) \;=\; a_\phi(x) + b_\phi(x) \odot \theta + W_\phi(\theta) x,\] mapping \(\mathcal{X} \times \Theta \to \mathbb{R}^p\). By Proposition 1F of Block 1, \(\Phi_\phi\) is the object of inference, not \(\phi\), and identifiability claims are at the level of \(L^2(\mu \otimes \pi_\Theta)\) equivalence under (D-ID).
Training proceeds by either:
Stochastic gradient descent (SGD) on a regularized empirical risk, \[\widehat\phi_n \;=\; \mathop{\arg\min}_\phi \;\sum_{i=1}^n -\log p(y_i \mid \theta_i^\phi(x_i, \theta_{\text{ref}})) + \mathcal{R}(\phi),\] with \(\mathcal{R}\) a regularizer enforcing (C1)-(C4) of Block 1 plus standard weight decay; or
Bayesian inference over \(\phi\) via stochastic variational inference (SVI) or Hamiltonian Monte Carlo on the posterior \[\Pi_n(\phi \mid Y_{1:n}, X_{1:n}) \propto \pi(\phi) \prod_i p(y_i \mid \theta_i^\phi(x_i, \theta_{\text{ref}})),\] with \(\pi(\phi)\) a prior on the network parameters.
The framework’s Path 3 implementation supports both modes; the asymptotic theory below is stated for whichever applies, with explicit indication.
(R-RANDOM) Random design. \((X_1, \theta_{\text{ref}}, Y_1), \ldots\) i.i.d. from the joint distribution induced by \(X \sim \mu\), \(\theta_{\text{ref}} \sim \pi_\Theta\), and \(Y \mid X, \theta_{\text{ref}} \sim \mathcal{D}(\theta_*(X, \theta_{\text{ref}}))\).
(R-COND) Fixed/conditional design. Conditional on \(\{X_i\}_{i=1}^n\) (and on the prior draws of \(\theta_{\text{ref}}\) in the Bayesian variant), with asymptotic statements for \(\mu^\infty\)-a.e. realization.
The (R-EQUIV) bridge condition introduced in Block 4 §2.1.1 —random-conditional equivalence under a uniform Glivenko-Cantelli condition on the function classes— applies in principle to Path 3 as well, but with the additional caveat that uniform-class conditions are non-trivial for neural-network function classes and are not automatic as they are for finite-dim parametric or spline classes; this is part of the open-questions list (§10, (O1-Path3)).
Asymptotic analysis of neural networks separates into qualitatively distinct regimes, each with its own theory and limitations:
These are the four sub-regimes considered in the present framework, chosen as the regimes most relevant for the implementation of Path 3 and most developed in the literature. The list is not exhaustive of all regimes treated in the broader neural-network asymptotics literature: hybrid regimes (e.g., depth-to-infinity at fixed width; structured-perturbation NTK; anisotropic mean-field) and non-asymptotic finite-width refinements exist and may be relevant in specialized applications. Each of the partial theorems in §5-§8 specifies which regime it applies to; no single theorem covers all regimes treated here.
The natural distance for Path 3 is \(L^2(\mu \otimes \pi_\Theta)\) on the realized function: \[d_{L^2}(\Phi, \Phi') \;=\; \Bigl( \int \|\Phi(x, \theta) - \Phi'(x, \theta)\|^2 \, d\mu(x) \, d\pi_\Theta(\theta) \Bigr)^{1/2}.\]
This is the metric in which Proposition 1F of Block 1 establishes identifiability of \(\Phi_\phi\) up to equivalence. Pointwise and sup-norm metrics on \(\Phi_\phi\) are also useful for diagnostic purposes but require stronger regularity.
| Symbol | Meaning |
|---|---|
| \(\Phi_\phi\) | Realized function of the hypernetwork, \(\mathcal{X} \times \Theta \to \mathbb{R}^p\) |
| \(\Phi_*\) | True realized function at \(\eta_*\) |
| \(\widehat\phi_n\) | Trained network parameters at sample size \(n\) |
| \(\widehat\Phi_n = \Phi_{\widehat\phi_n}\) | Trained realized function |
| \(\pi(\phi)\) | Prior on network parameters (Bayesian variant) |
| \(\Pi_n(\phi \mid \cdot)\) | Posterior on network parameters |
| \(\mathrm{NTK}\) | Neural Tangent Kernel |
| \(\mathrm{NNGP}\) | Neural Network Gaussian Process kernel |
| \(P\) | Number of network parameters (often \(\gg n\)) |
| \(L\) | Network depth |
| \(W_\ell\) | Width of layer \(\ell\) |
Parallel to Blocks 4-5:
Status of each layer for Path 3:
| Layer | Status | Reference for partial result | Open questions |
|---|---|---|---|
| (L1) | Partial: established under NTK regime (Theorem 6A) and under fixed architecture with strong conditions (Proposition 6D). General \(L^2(\mu \otimes \pi_\Theta)\) consistency for arbitrary architecture and SGD optimizer is not closed. | Jacot et al. 2018; Hron et al. 2020 | (O1-Path3) |
| (L2) | Partial: NTK regime gives a kernel-regression rate (Theorem 6A). Finite-sample PAC-Bayes generalization bounds (Theorem 6B) provide non-asymptotic risk control but not contraction rates. Adaptive contraction rates for general BNNs are not closed. | Jacot et al. 2018; Dziugaite and Roy 2017 | (O2-Path3), (O3-Path3) |
| (L3) | Largely open. Infinite-width Gaussian-process limit (Proposition 6C) characterizes the prior rather than the posterior. Function-level BvM for the trained network is not closed. | Lee et al. 2018; Matthews et al. 2018 | (O4-Path3) |
The framework reports this status explicitly to the user when Path 3 is invoked, and supplements the partial asymptotic theory with the empirical discrimination protocol of Block 1 §6.8.1 (four-step test of stability across seeds, OOS comparison, calibration, component-wise divergence) as the operational substitute for closed asymptotic guarantees.
In addition to (C1)-(C6), (LIN), (D-ID) of Block 1 (with (LIN) failing for Path 3 —as already noted in Proposition 1F— so identifiability is at the function level), and (HOM), (REG), (IID) of Block 2:
(NN-ARCH) Architecture. The hypernetwork has finite depth \(L\) and per-layer widths \(W_1, \ldots, W_L\). Activation functions are smooth (e.g., GELU, softplus) or piecewise-linear (ReLU); the framework defaults to GELU for differentiability of the activation in the asymptotic analysis.
(NN-PRIOR-SCALE) Prior scaling. Network weights at layer \(\ell\) have prior variance scaling as \(\sigma_\ell^2 / W_\ell\) (NTK parametrization) or \(\sigma_\ell^2 / W_\ell^2\) (mean-field parametrization). The choice of scaling determines the limiting regime as widths grow.
(NN-OPT) Optimization regularity. The optimizer (SGD with momentum, Adam, or Hamiltonian Monte Carlo) converges to a stationary point or, in the Bayesian variant, samples from the posterior asymptotically. This is a non-trivial assumption for non-convex losses; in practice, multiple restarts and convergence diagnostics are used as proxy verification.
(NN-LIM) Limiting regime. One of:
The four sub-regimes are mutually exclusive and exhaustive of the regimes treated in the literature. The framework does not assume one over the others; the user (or an empirical diagnostic) selects the regime appropriate to the problem and architecture.
Theorem 6A. Stated under (R-RANDOM) and (NN-LIM-NTK). Suppose:
- The network architecture (NN-ARCH) and prior scaling (NN-PRIOR-SCALE) are NTK-compatible.
- Training proceeds via gradient descent (or its Bayesian analog) in the lazy regime: width \(\to \infty\), step size \(\to 0\), \(T\) training iterations finite.
- Standard regularity on the loss and the data distribution.
Then two distinct limits hold, with distinct hypotheses:
Limit (i): Trained network \(\to\) NTK kernel-regression solution.
The trained network’s realized function converges, in the NTK limit, to the kernel-regression solution: \[\widehat\Phi_n \;\xrightarrow{P}\; \Phi^{\mathrm{NTK}}_n \;=\; \mathop{\arg\min}_{\Phi \in \mathrm{RKHS}(\mathrm{NTK})} \;\frac{1}{n} \sum_i \mathcal{L}(Y_i, \Phi(X_i, \theta_{\text{ref}})) + \tau \|\Phi\|^2_{\mathrm{RKHS}},\] for an appropriate regularization \(\tau\) determined by the early-stopping schedule. This convergence is the content of Jacot, Gabriel, and Hongler (2018, Theorem 1) and uses only NTK-regime hypotheses on training dynamics.
Limit (ii): NTK kernel-regression solution \(\to \Phi_*\).
As \(n \to \infty\), \(\Phi^{\mathrm{NTK}}_n \to \Phi^{\mathrm{NTK}}_*\) in \(L^2(\mu \otimes \pi_\Theta)\), where \(\Phi^{\mathrm{NTK}}_*\) is the best \(L^2\)-approximation of \(\Phi_*\) within the closure of \(\mathrm{RKHS}(\mathrm{NTK})\): \[\Phi^{\mathrm{NTK}}_* \;=\; \mathop{\arg\min}_{\Phi \in \overline{\mathrm{RKHS}(\mathrm{NTK})}} \;d_{L^2}(\Phi, \Phi_*).\] The further conclusion \(\Phi^{\mathrm{NTK}}_* = \Phi_*\) —i.e., that the kernel-regression limit recovers the true function— requires the additional source/smoothness hypothesis that \(\Phi_* \in \mathrm{RKHS}(\mathrm{NTK})\) with sufficient smoothness to satisfy the kernel’s source condition (Caponnetto and De Vito 2007). If \(\Phi_* \notin \mathrm{RKHS}(\mathrm{NTK})\), only convergence to the projection \(\Phi^{\mathrm{NTK}}_*\) holds, and the gap \(d_{L^2}(\Phi^{\mathrm{NTK}}_*, \Phi_*)\) is the kernel approximation error —an irreducible bias of the NTK regime for the chosen architecture.
Reference and limitations. The result is a specialization of Jacot, Gabriel, and Hongler (2018, Theorem 1) and Bach (2017, Theorem 5) to the AMM hypernetwork context. Three substantive limitations:
The NTK regime is the lazy training regime: it requires width \(\to \infty\) with appropriate parameter scaling and small initial parameters. Practical hypernetworks of finite width and standard SGD operate outside the NTK regime; Theorem 6A gives an approximation whose quality depends on the gap between practical training and lazy training.
The RKHS of the NTK is restrictive: it is a specific Hilbert space determined by the architecture. Functions \(\Phi_*\) outside this RKHS may not be representable by the trained network in the NTK regime, even though the network has universal approximation capacity in the wide-width limit (Hornik 1991).
The rate of \(\Phi^{\mathrm{NTK}}_n \to \Phi_*\) depends on the NTK and on the smoothness of \(\Phi_*\) in this specific RKHS. There is no universal rate analogous to the Stone-optimal \(n^{-\beta/(2\beta+d)}\).
Practical implication. Theorem 6A is a useful conceptual statement: it says that, in the wide-width lazy regime, the hypernetwork behaves like a specific kernel regression. But it does not give a universally applicable contraction rate or a finite-sample guarantee for typical Path 3 fits.
Theorem 6B. Finite-sample PAC-Bayes bound (McAllester 1999; Dziugaite and Roy 2017). Let \(\pi(\phi)\) be a prior on \(\phi\) chosen before observing the data, and \(\widehat\Pi_n(\phi)\) any (data-dependent) posterior over \(\phi\). Let \(L(\phi) = \mathbb{E}_{(X, Y)}[\ell(Y, \Phi_\phi(X))]\) be the population risk and \(\widehat L_n(\phi) = n^{-1} \sum_i \ell(Y_i, \Phi_\phi(X_i))\) the empirical risk. Suppose \(\ell \in [0, 1]\) (or appropriately bounded). Then with probability at least \(1 - \delta\) over the sample: \[\mathbb{E}_{\widehat\Pi_n}[L(\phi)] \;\leq\; \mathbb{E}_{\widehat\Pi_n}[\widehat L_n(\phi)] \;+\; \sqrt{\frac{\mathrm{KL}(\widehat\Pi_n \,\|\, \pi) + \log(2 \sqrt{n} / \delta)}{2(n - 1)}}.\]
Argument. Standard PAC-Bayes via concentration of the Rényi divergence; see Dziugaite and Roy (2017) for the deep-learning specialization with \(\widehat\Pi_n\) realized as a Gaussian variational posterior.
What this gives Path 3.
What this does not give.
Operational use. The library reports the PAC-Bayes bound as a finite-sample quality certificate alongside Path 3 fits. A small bound is evidence that the trained network has not overfit; a large bound is a warning that the empirical-risk minimizer may not generalize.
Proposition 6C. Characterization of the prior in the infinite-width limit. Stated under (NN-LIM-INF-W). Suppose the network has an infinite-width limit with NTK parametrization (or appropriate analog). Then the prior \(\pi(\phi)\) on the network weights induces, in this limit and at the level of the prior over functions, a Gaussian process prior on the realized function: \[\Phi_\phi \;\sim\; \mathrm{GP}(0, K_{\mathrm{NNGP}}) \qquad \text{(prior over functions, infinite-width limit)},\] where \(K_{\mathrm{NNGP}}\) is the Neural Network Gaussian Process kernel determined recursively by the architecture and the prior weight variances (Lee et al. 2018; Matthews et al. 2018).
Important: this is a statement about the prior, not about the posterior at finite width. The infinite-width Gaussian process limit characterizes how the weight prior \(\pi(\phi)\) translates into a function-space prior. The posterior under this prior, given finite data, is a separate question; in the infinite-width limit the posterior coincides with Gaussian-process-regression posterior (whose theory is mature), but at finite width the posterior is not generally Gaussian and the gap to the GP posterior is the subject of ongoing research (Hron et al. 2020 and (O5-Path3) below).
Reference. Lee, Bahri, Novak, Schoenholz, Pennington, Sohl-Dickstein (2018) for the closed-form recursion of \(K_{\mathrm{NNGP}}\); Matthews, Rowland, Hron, Turner, Ghahramani (2018) for the rigorous infinite-width limit theorem.
Implication for Path 3. In the infinite-width limit, Path 3 is asymptotically equivalent to a Gaussian process regression with the NNGP kernel. The asymptotic theory of GP regression then transfers to Path 3 in this limit:
Crucial caveat: this is a characterization of the prior, not
the posterior at finite width. The infinite-width GP limit
holds for the prior over functions induced by the
network architecture and weight prior. The posterior at finite
width —which is what the user actually fits with
torch— has additional finite-width corrections that are not
fully characterized. Hron et al. (2020) survey the partial results on
finite-width Bayesian deep networks; the gap between infinite-width and
finite-width is the subject of (O5-Path3).
Proposition 6D. Stated under (R-RANDOM) and (NN-LIM-FIX). Suppose:
- The hypernetwork has fixed architecture (depth \(L\), widths \(W_1, \ldots, W_L\), total parameters \(P\)).
- The prior \(\pi(\phi)\) has positive density on a compact subset of \(\mathbb{R}^P\).
- The negative log-likelihood is smooth in \(\phi\) on this subset.
- The optimizer (or MCMC sampler) converges to some stationary point or stationary distribution of the empirical risk surface (no claim is made that the stationary point is a global minimum).
Then, as \(n \to \infty\) with the architecture fixed, the trained network’s realized function converges to a stationary limit \(\Phi^{(L, W_{1:L}), \text{stat}}_n\) within the function class realizable by the architecture: \[\widehat\Phi_n \;\xrightarrow{P}\; \Phi^{(L, W_{1:L}), \text{stat}}_n,\] where \(\Phi^{(L, W_{1:L}), \text{stat}}_n\) corresponds to a (not necessarily global) minimizer of the empirical risk reachable by the optimizer from its initialization.
The further claim that \(\Phi^{(L, W_{1:L}), \text{stat}}_n\) converges to the global best \(L^2(\mu \otimes \pi_\Theta)\)-approximation \[\Phi^{(L, W_{1:L})}_* \;=\; \mathop{\arg\min}_{\Phi_\phi : \phi \in \mathbb{R}^P} \;d_{L^2}(\Phi_\phi, \Phi_*)\] requires the additional global-optimality hypothesis (NN-OPT-GLOBAL): the optimizer (or sampler) reaches a global minimum of the empirical risk, not just a stationary point. (NN-OPT-GLOBAL) is a strong condition for non-convex losses (the typical case for neural networks) and is not generically satisfied; it can fail at saddle points and at non-global local minima created by the network’s symmetries (Proposition 1F (ii)).
Reference and discussion. This is M-estimator consistency (van der Vaart 1998, §5) applied to neural-network estimation under fixed architecture, modified to honor the gap between stationary points and global optima in non-convex landscapes.
Three points of substance:
The unconditional limit is a stationary point, not the best architecture-realizable approximation. The two coincide only under (NN-OPT-GLOBAL).
Even under (NN-OPT-GLOBAL), the limit \(\Phi^{(L, W_{1:L})}_*\) is not \(\Phi_*\) in general. It is the projection of \(\Phi_*\) onto the realizable function class. If the architecture is too narrow or too shallow, the projection may differ substantially from \(\Phi_*\); the gap is the architecture approximation error.
For the architecture approximation error to vanish, \(W_\ell\) must grow with \(n\). This is the growing-architecture regime, which is a hybrid of (NN-LIM-FIX) and (NN-LIM-INF-W); the asymptotic theory in this regime is not closed and is part of (O3-Path3) below.
Convergence rate to the stationary limit is parametric (\(n^{-1/2}\)) in \(\phi\) under standard regularity, but network symmetries (Proposition 1F (ii)) make this convergence at the level of \(\phi\) ambiguous; convergence at the level of \(\Phi_\phi\) holds at parametric rate when the stationary point is locally identifiable up to symmetries.
Practical implication. Proposition 6D is a useful conceptual statement with two layers of caveat: first, the empirical optimization may not find the global minimum (the architecture-realizable best approximation within the realizable class may itself be unreached); second, even if it is reached, the architecture may be too rigid to approach \(\Phi_*\) (architecture approximation error of the realizable class). The framework reports both layers of error separately when feasible (using restart-based estimates of the gap to global optimum, plus comparison to NTK-regime fits as a check on architecture sufficiency). This is the formal version of two related practical concerns: “your optimizer might not have converged to a global minimum” and “your network might not be deep/wide enough”.
Path 3 covers Proposition 3.7 of Block 3 (hypernetworks). The asymptotic results of this block specialize as:
| Special Case | AMM Level | (L1) Consistency | (L2) Rate | (L3) BvM |
|---|---|---|---|---|
| Hypernetwork (Proposition 3.7) | \(\infty\) | Theorem 6A (NTK) or Proposition 6D (fixed) under regime-specific hypotheses | NTK-determined or PAC-Bayes finite-sample (Theorem 6B); no universal rate | Open (O4-Path3); Proposition 6C characterizes the prior, not the posterior |
The Path 1 and Path 2 special cases (Theorems 3.1-3.6) are not treated by Path 3 because they fall within the linear-class scope (LIN) of Block 1; using a hypernetwork for those cases is technically possible but operationally unnecessary, and the asymptotic theory of the simpler paths is more complete.
The open questions for Path 3 are substantively more numerous than for Paths 1-2, reflecting the genuinely open frontier of the asymptotic theory of neural networks.
(O1-Path3) General consistency outside the NTK regime. Theorem 6A requires the NTK lazy regime; Proposition 6D requires fixed architecture. The realistic regime —finite-but-large width, SGD with momentum and weight decay— is not covered by either. General function-level consistency in this regime is an open problem.
(O2-Path3) Convergence rate under realistic optimization. Most rate results assume gradient descent at vanishing step size or full-batch gradient. SGD with momentum, Adam, and other modern optimizers introduce additional dynamics whose asymptotic effects on the realized function are partially understood (Hardt, Recht, Singer 2016 for SGD generalization) but not fully characterized.
(O3-Path3) Adaptive contraction rates under unknown smoothness. The rates derivable from Theorem 6A depend on the smoothness of \(\Phi_*\) in the NTK RKHS, which is typically unknown. Adaptive priors and architectures that achieve the optimal rate without specifying the smoothness are an open question (relevant work: Polson and Sokolov 2017; Suzuki 2018 for adaptive deep learning).
(O4-Path3) Function-level Bernstein-von Mises. A full BvM for the posterior over \(\Phi_\phi\) in a function-space metric is open. The infinite-width GP limit (Proposition 6C) gives BvM only for the GP-projected functional, not the full network posterior.
(O5-Path3) Finite-width corrections to the infinite-width GP limit. Proposition 6C holds in the infinite-width limit. At finite width, the network behaves differently; characterizing the gap is the subject of recent work (Hron et al. 2020; Naveh, Ringel 2021) but is not closed.
(O6-Path3) Quantification of network symmetries. Proposition 1F (ii) of Block 1 notes that \(\phi\) is non-identifiable due to permutation, sign, and rescaling symmetries. The quantitative effect of these symmetries on optimizer convergence —specifically, how they create local minima or saddle points and how SGD navigates them— is not fully characterized.
(O7-Path3) Out-of-distribution behavior of \(\Phi_\phi\). All asymptotic results are stated under the i.i.d. data-generating assumption. The behavior of the trained network on covariates outside \(\mathrm{supp}(\mu)\) is not controlled by the asymptotic theory and is a separate domain (out-of-distribution generalization; recent work: Arjovsky et al. 2019; Nagarajan, Andreassen, Neyshabur 2021).
The framework reports these open questions explicitly to the user when Path 3 is invoked, alongside the partial results above. The empirical discrimination protocol of Block 1 §6.8.1 provides operational diagnostics that compensate, in part, for the asymptotic gaps: stability across seeds (testing for symmetry-induced non-identifiability, related to (O6)); cross-validated OOS performance (testing generalization in the absence of asymptotic guarantees, related to (O1)-(O3)); calibration (testing posterior reliability in the absence of BvM, related to (O4)); Path 1 vs. Path 3 comparison (testing how much the gap to closed theory matters in practice).
torch)The planned Path 3 implementation in gdpar will use the
R torch package (LibTorch backend), supporting both
SGD-based ERM training and SVI Bayesian variational inference. The
current release (0.0.0.9001) does not ship this path; the
design notes below specify the contract the operational layer will
honour when it lands.
The default prior will be NTK parametrization with Gaussian per-layer weights of variance \(\sigma_\ell^2 / W_\ell\). This places Path 3 in the scope of Theorem 6A in the wide-width limit and Proposition 6C in the infinite-width limit. Alternative parametrizations (mean-field, finite-width with explicit prior on each weight) will be available via configuration.
After training, the planned library will compute the PAC-Bayes bound of Theorem 6B with \(\delta = 0.05\) as a finite-sample quality certificate. The planned report will include: - Empirical risk \(\widehat L_n\). - KL divergence \(\mathrm{KL}(\widehat\Pi_n \,\|\, \pi)\) (computed from the variational posterior). - The bound on \(\mathbb{E}_{\widehat\Pi_n}[L(\phi)]\) at confidence \(1 - \delta\).
If the bound is large relative to the desired prediction accuracy, the planned interface will warn the user that the trained network may not generalize.
The planned library will compute the NTK at initialization for the chosen architecture and report its eigenvalue spectrum. The decay rate of the spectrum determines the smoothness class of functions that Theorem 6A can recover; a slowly decaying spectrum indicates a wide RKHS (more flexible Path 3 fit), a quickly decaying spectrum indicates a restrictive RKHS (the architecture may be too rigid for the target \(\Phi_*\)).
The framework’s empirical discrimination protocol of Block 1 §6.8.1 (four steps: stability across seeds, OOS comparison, calibration, component-wise divergence) is the operational mechanism by which Path 3’s asymptotic gaps will be surfaced. The planned library will run the protocol automatically when Path 3 is fit and Path 1 is feasible on the same data, and will report the verdict (richer structure / non-identifiability / model misspecified) with the corresponding recommendation.
Path 3 supports only the average-error form of (REG-EST), and even this requires the partial results above. Specifically:
The strictly stronger uniform-in-\(i\) version of (REG-EST) —\(\sup_i \|\widehat\theta_i - \theta_*(x_i, \theta_{\text{ref}})\| \to 0\)— is not established for Path 3 in general. Lemma 2B of Block 2 invokes only the average-error form, so this gap does not invalidate Lemma 2B; it does mean that uniform-in-\(i\) statements about Path 3’s individual fits are not currently supported by closed asymptotic theory.
This block has established for Path 3:
Three asymptotic layers —function-level consistency (L1), convergence rate (L2), BvM analog (L3)— and an explicit assessment that none of the three is closed in general for Path 3, with (L1) and (L2) covered partially by regime-specific results and (L3) largely open.
Standing asymptotic hypotheses —(NN-ARCH), (NN-PRIOR-SCALE), (NN-OPT), (NN-LIM)— nominated, with (NN-LIM) further decomposed into four mutually exclusive sub-regimes (NTK, mean-field, fixed, infinite-width).
Theorem 6A (NTK consistency) in the lazy regime, with three substantive limitations explicitly recognized.
Theorem 6B (PAC-Bayes bound) as a finite-sample non-asymptotic quality certificate, with explicit recognition that it does not yield a contraction rate.
Proposition 6C (Infinite-width GP limit) as a characterization of the prior, not the posterior at finite width, with the gap to finite-width acknowledged.
Proposition 6D (Fixed-architecture consistency) with the limit being the best architecture-realizable approximation of \(\Phi_*\), not \(\Phi_*\) itself.
Seven explicitly recognized open questions (general consistency outside NTK, realistic optimizer dynamics, adaptive rates, function-level BvM, finite-width corrections, quantification of symmetries, OOD behavior).
Implementation diagnostics for Path 3 in
torch, including the NTK eigenvalue spectrum, the PAC-Bayes
bound report, and cross-reference to the empirical discrimination
protocol of Block 1 §6.8.1 as the operational substitute for closed
asymptotic guarantees.
The block does not pretend to give a complete asymptotic theory of Path 3 —none currently exists— and is honest about what is established, what is conjectural, and what remains genuinely open. The framework’s recommendation is that Path 3 be used as a complement to Path 1, not as a stand-alone path with full asymptotic backing, except in regimes where the partial results of this block apply (lazy NTK regime; PAC-Bayes finite-sample bound that the user evaluates explicitly; infinite-width GP limit when justified by the architecture).
| Symbol | Meaning |
|---|---|
| \(\Phi_\phi\) | Realized function of the hypernetwork |
| \(\widehat\Phi_n = \Phi_{\widehat\phi_n}\) | Trained realized function |
| \(\Phi_*\) | True realized function |
| \(\Phi^{(L, W_{1:L})}_*\) | Best architecture-realizable approximation of \(\Phi_*\) |
| \(\Phi^{\mathrm{NTK}}_n\) | NTK kernel-regression solution |
| \(\mathrm{NTK}\) | Neural Tangent Kernel |
| \(\mathrm{NNGP}\) | Neural Network Gaussian Process kernel |
| \(K_{\mathrm{NNGP}}\) | NNGP kernel function |
| \(\widehat\Pi_n(\phi)\) | (Variational) posterior on \(\phi\) |
| \(L(\phi), \widehat L_n(\phi)\) | Population and empirical risk at parameter \(\phi\) |
| \(\mathrm{RKHS}(\mathrm{NTK})\) | RKHS associated with the NTK |
| \(P\) | Number of network parameters |
| \(L\) | Network depth |
| \(W_\ell\) | Width of layer \(\ell\) |
| Hypothesis | Content | Used by |
|---|---|---|
| (R-RANDOM) | Random design as in Block 4 §2.1.1 | Default for §5-8 |
| (R-COND) | Fixed/conditional design | Variant for §5-8 |
| (NN-ARCH) | Finite depth \(L\), per-layer widths \(W_\ell\), smooth or piecewise-linear activations | All theorems |
| (NN-PRIOR-SCALE) | Per-layer prior variance \(\sigma_\ell^2 / W_\ell\) (NTK) or \(\sigma_\ell^2 / W_\ell^2\) (mean-field) | Theorems 6A, 6B, 6C, 6D |
| (NN-OPT) | Optimizer / sampler converges to a stationary point / distribution (not necessarily global optimum) | All theorems |
| (NN-OPT-GLOBAL) | Optimizer / sampler reaches the global minimum of the empirical risk; strong condition for non-convex losses | Proposition 6D when limit is the best architecture-realizable approximation |
| (NN-LIM-NTK) | Width \(\to \infty\), lazy regime | Theorem 6A |
| (NN-LIM-MF) | Mean-field regime | Not stated; reference to literature |
| (NN-LIM-FIX) | Fixed architecture | Proposition 6D |
| (NN-LIM-INF-W) | Infinite-width Gaussian process limit | Proposition 6C |
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. (2019). Invariant risk minimization. arXiv:1907.02893.
Bach, F. (2017). Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18, 1–53.
Caponnetto, A., and De Vito, E. (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3), 331–368.
Chizat, L., and Bach, F. (2018). On the global convergence of gradient descent for over-parameterized models using optimal transport. NeurIPS 2018.
Dziugaite, G. K., and Roy, D. M. (2017). Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. Proceedings of UAI.
Hardt, M., Recht, B., and Singer, Y. (2016). Train faster, generalize better: stability of stochastic gradient descent. ICML 2016.
Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2), 251–257.
Hron, J., Bahri, Y., Sohl-Dickstein, J., and Novak, R. (2020). Infinite attention: NNGP and NTK for deep attention networks. ICML 2020.
Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: convergence and generalization in neural networks. NeurIPS 2018.
Lee, J., Bahri, Y., Novak, R., Schoenholz, S., Pennington, J., and Sohl-Dickstein, J. (2018). Deep neural networks as Gaussian processes. ICLR 2018.
Matthews, A. G. de G., Rowland, M., Hron, J., Turner, R. E., and Ghahramani, Z. (2018). Gaussian process behaviour in wide deep neural networks. ICLR 2018.
McAllester, D. A. (1999). PAC-Bayesian model averaging. Proceedings of COLT.
Mei, S., Montanari, A., and Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. PNAS, 115(33), E7665–E7671.
Nagarajan, V., Andreassen, A., and Neyshabur, B. (2021). Understanding the failure modes of out-of-distribution generalization. ICLR 2021.
Naveh, G., and Ringel, Z. (2021). A self-consistent theory for finite-width neural networks. NeurIPS 2021.
Polson, N. G., and Sokolov, V. (2017). Deep learning: a Bayesian perspective. Bayesian Analysis, 12(4), 1275–1304.
Suzuki, T. (2018). Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality. ICLR 2018.
van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.
van der Vaart, A. W., and van Zanten, J. H. (2008). Rates of contraction of posterior distributions based on Gaussian process priors. Annals of Statistics, 36(3), 1435–1463.
torch)