# Aggregation methods

### Introduction

We consider the problem of combining a set of linear estimators in non-parametric regression model with Gaussian noise. Focusing on the exponentially weighted aggregate (EWA), we prove a PAC-Bayesian type inequality that leads to sharp oracle inequalities in discrete but also in continuous settings. The framework covers the combinations of various procedures such as least square regression, kernel ridge regression, shrinking estimators, etc. We show that the proposed aggregate provides an adaptive estimator in the exact minimax sense without neither discretizing the range of tuning parameters nor splitting the set of observations.

### Theoretical results


Exponentially Weighted Aggregates


In this study, we mainly consider a family of linear filter. After (orthogonaly) transforming the original 1D signal with the DCT, we combine the estimators obtained by using several Pinsker's Filter (i.e., by varying the shrinking parameters). With the notation $\newcommand{\DST}{\mathcal D}A_{\alpha,w}=\DST^\top\!\diag\big((1-k^{\alpha}/w)_+;k=1,\ldots,n\big)\DST$, the preliminary estimates can be expressed as: $\hatflbd=A_{\alpha,w}\YY$. With those estimates and a particular choice of the prior, we can prove that combining Pinsker's type filters with EWA leads to an asymptotically sharp adaptive procedure over Sobolev ellipsoids. The prior used in practice is defined by $$\label{eq:prior_pinsker} \pi(d\lambda)= \frac{2n_\sigma^{-\alpha/(2\alpha+1)}}{\big(1+n_\sigma^{-\alpha/(2\alpha+1)}w\big)^{3}}e^{-\alpha}d\alpha dw .$$ In practice we also compare the peformance of aggregating the Pinsker's filters by combining various shrinking paramters taken on a geometric grid for $\alpha$ and $w$. Moreover the choice for the parameter $\beta$ given by our oracle inequality leads to pick $\beta=8\sigma^2$, though in practice it may be choosen smaller ($4\sigma^2$ or $2\sigma^2$).

See corresponding Matlab Demo.