Regression-Based Expected Shortfall Backtesting

Abstract

This article introduces novel backtests for the risk measure Expected Shortfall (ES) following the testing idea of Mincer and Zarnowitz (1969). Estimating a regression model for the ES stand-alone is infeasible and thus, our tests are based on a joint regression model for the Value at Risk (VaR) and the ES, which allows for different test specifications. These ES backtests are the first which solely backtest the ES in the sense that they only require ES forecasts as input variables. As the tests are potentially subject to model misspecification, we provide asymptotic theory under misspecification for the underlying joint regression. We find that employing a misspecification robust covariance estimator substantially improves the tests’ performance. We compare our backtests to existing joint VaR and ES backtests and find that our tests outperform the existing alternatives throughout all considered simulations. In an empirical illustration, we apply our backtests to ES forecasts for 200 stocks of the S&P 500 index.

Through the transition from Value at Risk (VaR) to Expected Shortfall (ES) as the primary market risk measure in the Basel Accords (Basel Committee, 2016, 2017), there is a great demand for reliable methods for estimating, forecasting, and backtesting the ES. Formally, the ES at level $\tau \in (0,1)$ is defined as the mean of the returns smaller than the respective τ-quantile (the VaR), where τ is usually chosen to be 2.5% as stipulated by the Basel Accords. The ES is introduced into the banking regulation because it overcomes several shortcomings of the VaR, such as being not coherent and its inability to capture tail risks beyond the τ-quantile (Artzner et al., 1999; Danielsson et al., 2001; Basel Committee, 2013). In contrast to estimation and forecasting of ES where most of the existing models for the VaR can easily be adapted and generalized to the ES, such a generalization is not as straight-forward for backtesting ES forecasts (Emmer, Kratz, and Tasche, 2015). In general, backtesting of a risk measure is the process of testing whether given forecasts for this risk measure are correctly specified, which is carried out by comparing the history of the issued risk forecasts with the corresponding realized returns. The primary difficulty in directly backtesting ES is its non-elicitability and non-identifiability (Weber, 2006; Gneiting, 2011; Fissler and Ziegel, 2016; Fissler, Ziegel, and Gneiting, 2016) as consequently, there is no analog to the hit sequence which is the natural identification function of quantiles and which lies at the heart of almost all VaR backtests.¹

As a consequence, most of the proposed procedures in the growing literature on backtesting ES use indirect approaches by formally backtesting some quantity which is closely related to the ES. Examples include tests based on the entire tail distribution, a linear approximation of the ES through several quantiles, or the pair consisting of the VaR and the ES.² We argue that formally, these approaches are backtests for the auxiliary quantities rather than for the ES itself, see also Nolde and Ziegel (2017). This distinction is particularly important as these backtests require further input variables such as forecasts for the VaR at multiple levels, the tail distribution beyond some quantile, or even the entire distribution. The regulatory authorities, however, do not have this additional information at hand as it is not mandatorily reported by the financial institutions (Aramonte et al., 2011; Basel Committee, 2016, 2017). As a consequence, the existing, so-called ES backtests are not applicable where they are most needed.

In this article, we propose novel backtests for ES forecasts which are the first strict ES backtests in the literature in the sense that besides the realized returns, they only require ES forecasts as input variables. Our tests follow the general regression-based testing idea of Mincer and Zarnowitz (1969). For this, we estimate a regression model that models the conditional ES at level τ as a linear function ${\text{ES}}_{\tau }\left(Y_t|{\mathcal{F}}_{t-1}\right)={\gamma }_1+{\gamma }_2{\widehat{e}}_t$⁠, where we use financial returns Y_t as the response variable and the given ES forecasts ${\widehat{e}}_t$ as the explanatory variable including an intercept term. For correctly specified ES forecasts, the intercept and slope parameters equal zero and one, which we test for by using a Wald statistic. As the ES is not elicitable (Gneiting, 2011), we face the methodological difficulty that we cannot estimate such a regression model for the ES stand-alone as neither loss nor identification functions are available for the ES, which could be used as objective functions for maximum (M) or generalized method of moment (GMM) estimation (Dimitriadis and Bayer, 2019). Recently, Patton, Ziegel, and Chen (2019) and Dimitriadis and Bayer (2019) propose a feasible alternative by specifying an auxiliary quantile regression equation $Q_{\tau }\left(Y_t|{\mathcal{F}}_{t-1}\right)={\beta }_1+{\beta }_2{\widehat{\rho }}_t$ (with explanatory variable ${\widehat{\rho }}_t$⁠) and by jointly estimating the regression parameters $(\beta ,\gamma )$ by employing a joint loss function for the quantile and the ES from Fissler and Ziegel (2016).

The specification of the quantile equation allows for different testing approaches. First, we employ auxiliary VaR forecasts ${\widehat{v}}_t$ as the explanatory variable in the quantile equation, but only test the ES-specific parameters γ. We refer to this test as the Auxiliary ESR (ES regression) backtest. The main drawback of this test is that it requires auxiliary VaR forecasts and consequently, it is formally a joint backtest for the VaR and ES which, however, mainly focuses on the ES by only testing the ES-specific regression parameters. Second, we use the ES forecasts ${\widehat{e}}_t$ as the explanatory variable in both, the quantile and the ES equation and again only test on the ES-specific parameters γ. We refer to this test as the Strict ESR backtest as it only requires ES forecasts as input variables and thus, it is the first test in the literature which solely backtests ES forecasts. This testing idea comes at the drawback of a potential model misspecification in the quantile equation if the underlying data go beyond a pure scale (volatility) model. Therefore, we provide asymptotic theory for the joint quantile and ES regression framework under model misspecification, which generalizes the asymptotic theory introduced in Dimitriadis and Bayer (2019) and Patton, Ziegel, and Chen (2019). The potential model misspecification results in a more complex and usually inflated asymptotic covariance matrix. We account for this in the implementation of our tests by employing a covariance estimation technique which explicitly estimates these additional covariance terms.

We further introduce an intercept variant of the Strict ESR backtest by fixing the slope parameter in the ES regression to one, and by only estimating and testing the intercept term. We refer to this backtest as the Intercept ESR backtest. This test allows for both, testing against one-sided and two-sided alternatives. In contrast, the other two proposed ESR backtests only allow for testing against two-sided alternatives as it is generally unclear how underestimated and overestimated ES forecasts influence the intercept and slope parameters. Because the capital requirements that the financial institutions must keep as a reserve depend on the reported risk forecasts, the market participants have an incentive to report risk forecasts which are too risky in order to minimize the expensive capital requirements. In contrast, issuing too conservative risk forecasts results in holding costly capital reserves for the financial institutions but poses no risk to the society as a whole. Thus, the regulators only have to prevent and penalize the underestimation of the financial risks, which demonstrates the necessity of one-sided testing procedures. For example, the currently applied traffic light system (Basel Committee, 1996) is in fact a one-sided VaR backtest. Like the Strict ESR backtest, the Intercept ESR backtest also has the desired characteristic to only require ES forecasts as input variables and consequently is the first procedure that solely backtests the ES against a one-sided alternative. We provide implementations of the three ESR backtests proposed in this article in the R package esback (Bayer and Dimitriadis, 2019a).

Such regression-based forecast evaluation approaches are already used for testing mean forecasts (Mincer and Zarnowitz, 1969), quantile forecasts (Gaglianone et al., 2011; Guler, Ng, and Xiao, 2017), and expectile forecasts (Guler, Ng, and Xiao, 2017). In contrast to these functionals where regression techniques are easily available (see e.g. Koenker and Bassett, 1978, Efron, 1991), the non-elicitability of the ES makes our approach more involved but also opens up the possibility for the different testing specifications we introduce. Our multivariate generalization approach of the Mincer and Zarnowitz (1969) testing idea can be applied equivalently to other higher-order elicitable functionals (Fissler and Ziegel, 2016) such as, for example, the variance (in the presence of a non-zero mean) and the Range VaR (Cont, Deguest, and Scandolo, 2010; Embrechts, Liu, and Wang, 2018).

We evaluate the empirical properties of our ESR backtests and compare them to the existing joint VaR and ES backtests of McNeil and Frey (2000) and Nolde and Ziegel (2017) in several simulation designs. In the first setup, we implement the classical size and power analysis for backtesting risk measures, where we simulate data stemming from several realistic data generating processes (DGPs) and evaluate the empirical rejection frequencies of the backtests for forecasts stemming from the true and from some misspecified forecasting model. In order to assess how the potential model misspecification affects the Strict and the Intercept ESR backtests, we utilize DGPs which go beyond the class of pure scale (volatility) processes. For this, we implement two different Student’s-t generalized autoregressive score (GAS) models with time-varying higher moments (Creal, Koopman, and Lucas, 2013) and furthermore use an autoregressive (AR) GARCH (generalized autoregressive conditional heteroskedasticity) model which allows for gradually increasing the degree of misspecification through the AR parameter. In the second setup, we introduce a new technique for evaluating the power of backtests for financial risk measures, where we continuously misspecify certain model parameters of the DGP to obtain a continuum of alternative models with a gradually increasing degree of misspecification. Misspecifying the different model parameters separately allows us to misspecify certain model characteristics (such as the reaction to shocks) in isolation, which permits a closer examination of the proposed backtesting procedures.

The simulations show that all three ESR backtests we propose in this article are well-sized, especially when the tests are applied using the covariance estimation method which accounts for possible model misspecification. We further find that the performance of our testing procedures is almost unaffected by the DGPs which cause model misspecification in the Strict and the Intercept ESR tests. Moreover, our tests are more powerful than the existing backtests of McNeil and Frey (2000) and Nolde and Ziegel (2017) in almost all of the considered simulation designs for both, testing against one-sided and two-sided alternatives. Notably, throughout all simulation designs, the ESR backtests are able to detect the various different misspecifications of the forecasts. In contrast, the existing backtests sometimes completely fail to detect certain misspecifications, for instance when the forecaster reports risk forecasts for a misspecified probability level.

The rest of this article is organized as follows. Section 1 introduces the ESR backtests and presents asymptotic theory under model misspecification. Section 2 contains several simulation studies and Section 3 applies the backtests to ES forecasts for a large amount of stocks from the S&P 500 index. Section 4 concludes. The proofs are deferred to Appendix A and a Supplementary Appendix contains further details of the proofs.

1 Theory

1.1 Setup and Notation

We consider a stochastic process  

$$Z=\{Z_t:\Omega \to {\mathbb{R}}^{l+1},l\in \mathbb{N},t=1,\dots ,T\},$$

(1.1)

defined on some complete probability space $(\Omega ,\mathcal{F},\mathbb{P})$⁠, with the filtration $\mathcal{F}=\{{\mathcal{F}}_t,t=1,\dots ,T\}$ and ${\mathcal{F}}_t=\sigma \{Z_s,s\le t\}$ for all $t=1,\dots ,T$⁠, where $T\in \mathbb{N}$⁠. We partition the stochastic process $Z_t=(Y_t,U_t)$⁠, where Y_t is an absolutely continuous random variable of interest and U_t is an l-dimensional vector of explanatory variables. We denote the conditional cumulative distribution function of Y_t given the past information ${\mathcal{F}}_{t-1}$ by $F_t(y)=\mathbb{P}(Y_t\le y|{\mathcal{F}}_{t-1})$ and the corresponding probability density function by f_t. Whenever they exist, the mean and the variance of F_t are denoted by ${\mathbb{E}}_t[·]$ and ${\text{Var}}_t(·)$⁠.

For financial applications, the variable Y_t denotes the daily log returns of a financial asset (for instance, a stock or a portfolio), that is, $Y_t=\text{log}\hspace{0.17em}{P}_t-\text{log}\hspace{0.17em}{P}_{t-1}$⁠, where P_t denotes the price of the asset at day $t=1,\dots ,T$⁠. This means that throughout this article, we use the sign convention that positive returns denote profits, and negative returns denote losses. The vector U_t contains further variables that are used to produce forecasts for certain functionals (usually risk measures) of the random variable Y_t. We are interested in testing whether forecasts for a certain d-dimensional, $d\in \mathbb{N}$ functional (risk measure) $\rho =\rho (F_t)$ of the conditional distribution F_t are correctly specified. For that, we define the most frequently used functionals for financial risk management in the following. The conditional quantile of Y_t given the information set ${\mathcal{F}}_{t-1}$ at level $\tau \in (0,1)$ is defined as $Q_{\tau }(Y_t|{\mathcal{F}}_{t-1})=F_t^{-1}(\tau )=\inf \{y\in \mathbb{R}:F_t(y)\ge \tau \}$⁠, which is called the VaR at level τ in financial applications. Furthermore, we define the functional ES at level τ of Y_t given ${\mathcal{F}}_{t-1}$ as ${\text{ES}}_{\tau }(Y_t|{\mathcal{F}}_{t-1})=\frac{1}{\tau }{\int }_0^{\tau }{F}_t^{-1}(s)\mathrm{d}s$⁠. If the distribution function F_t is continuous at its τ-quantile, this definition can be simplified to the truncated tail mean of Y_t,  

$${\text{ES}}_{\tau }(Y_t|{\mathcal{F}}_{t-1})={\mathbb{E}}_t\left[Y_t|Y_t\le Q_{\tau }(Y_t|{\mathcal{F}}_{t-1})\right].$$

(1.2)

We denote an ${\mathcal{F}}_{t-1}$-measurable one-step-ahead forecast for day t for the risk measure ρ of the distribution F_t, stemming from some external forecaster or from some given forecasting model³ by ${\widehat{\rho }}_t={\widehat{\rho }}_t({\mathcal{F}}_{t-1})$⁠. Following this notation, we denote forecasts for the τ-VaR by ${\widehat{v}}_t$ and for the τ-ES by ${\widehat{e}}_t$ for some fixed level $\tau \in (0,1)$⁠. For simplicity of notation, we drop the dependence on τ as it is a fixed quantity.

As both, the incentive of the forecaster and the underlying method used to generate the forecasts are in general unknown, these forecasts are not necessarily correctly specified. The focus of this article is to develop statistical tests for correctness of a given series of forecasts $\{{\widehat{\rho }}_t,t=1,\dots ,T\}$ for the risk measure ρ relative to the realized return series $\{Y_t,t=1,\dots ,T\}$⁠. This is in the literature usually referred to as backtesting of the risk measure ρ without strictly defining this terminology. We provide such a definition in the following. 

Definition 1.1

A backtest for the series of forecasts $\{{\widehat{\rho }}_t,t=1,\dots ,T\}$for the d-dimensional risk measure (functional) ρ relative to the realized return series $\{Y_t,t=1,\dots ,T\}$is a function 

$$f:{\mathbb{R}}^T\times {\mathbb{R}}^{T\times d}\to \{0,1\},$$

(1.3)

which maps the return and forecast series onto the respective test decision.

The core message of this definition is that besides the realized return series, a backtest for some risk measure is only allowed to require forecasts for this risk measure as input variables. This strict differentiation becomes relevant in the context of backtesting ES as, in contrast to the existing VaR backtests, the recently proposed ES backtests require further input variables such as forecasts for the VaR, the volatility, or the entire tail distribution. The demand for these further quantities induces the following practical problems. First, the regulatory authorities who rely on such backtesting methods do not necessarily receive forecasts from the financial institutions for the additional information required by these tests, which makes such backtests inapplicable for the regulatory authorities. Second, a rejection of the tests does not necessarily imply that the ES is misspecified, but that the forecasts for any of the input components are misspecified. Consequently, these tests are in fact not backtests for the ES, but rather backtests for some vector of risk measures (or the entire tail distribution).

1.2 The ESR Backtests

We propose backtests for the risk measure ES that test whether a series of ES forecasts $\{{\widehat{e}}_t,t=1,\dots T\}$⁠, stemming from some external forecaster or forecasting model, is correctly specified relative to a series of realized returns $\{Y_t,t=1,\dots ,T\}$⁠. We follow the general testing idea of Mincer and Zarnowitz (1969) and regress the returns Y_t on the forecasts ${\widehat{e}}_t$ and an intercept term by using a regression equation designed specifically for the functional ES,  

$$Y_t={\gamma }_1+{\gamma }_2{\widehat{e}}_t+u_t^e,$$

(1.4)

where ${\text{ES}}_{\tau }(u_t^e|{\mathcal{F}}_{t-1})=0$ almost surely. Given the structure in Equation (1.4) and since the forecasts ${\widehat{e}}_t$ are generated by using the information set ${\mathcal{F}}_{t-1}$⁠, this condition on the error term is equivalent to  

$${\text{ES}}_{\tau }\left(Y_t|{\mathcal{F}}_{t-1}\right)={\gamma }_1+{\gamma }_2{\widehat{e}}_t.$$

(1.5)

We then test the hypothesis  

$${\mathbb{H}}_0:({\gamma }_1,{\gamma }_2)=(0,1)\hspace{1em}\hspace{1em}\text{against}\hspace{1em}\hspace{1em}{\mathbb{H}}_A:({\gamma }_1,{\gamma }_2)\ne (0,1).$$

(1.6)

Under ${\mathbb{H}}_0$⁠, the ES forecasts are correctly specified as it holds that ${\widehat{e}}_t={\text{ES}}_{\tau }\left(Y_t|{\mathcal{F}}_{t-1}\right)$ almost surely.⁴ In general, Equation (1.4) is an example of a linear regression equation for the ES of the form $Y_t=W_t^{\top }\gamma +u_t^e$⁠, for some general vector of covariates W_t. As outlined in Dimitriadis and Bayer (2019) and Patton, Ziegel, and Chen (2019), estimating the parameters γ by M- or GMM estimation stand-alone is not possible since there do not exist strictly consistent loss and identification functions for the functional ES (Gneiting, 2011). Based on the seminal work of Fissler and Ziegel (2016) who introduce joint loss and identification functions for the VaR and ES, Dimitriadis and Bayer (2019), Patton, Ziegel, and Chen (2019), and Barendse (2020) propose the joint regression technique,  

$$Y_t=V_t^{\top }\beta +u_t^q,\hspace{1em} \text{and} \hspace{1em}\hspace{1em}{Y}_t=W_t^{\top }\gamma +u_t^e,$$

(1.7)

where V_t and W_t are k-dimensional, ${\mathcal{F}}_{t-1}$-measurable covariate vectors, and where $Q_{\tau }(u_t^q|{\mathcal{F}}_{t-1})=0$ and ${\text{ES}}_{\tau }(u_t^e|{\mathcal{F}}_{t-1})=0$ almost surely. Setting up this joint regression model facilitates the estimation of the joint parameters $(\beta ,\gamma )$⁠, whereas stand-alone estimation of γ is infeasible. We use this joint setup to propose the following regression based backtests for the ES:

The Auxiliary ESR Backtest: We choose $V_t=(1,{\widehat{v}}_t)$ and $W_t=(1,{\widehat{e}}_t)$⁠, i.e. we set up the regression system  

$$Y_t={\beta }_1+{\beta }_2{\widehat{v}}_t+u_t^q,\hspace{1em} \text{and} \hspace{1em}\hspace{1em}{Y}_t={\gamma }_1+{\gamma }_2{\widehat{e}}_t+u_t^e,$$

(1.8)

and test  

$${\mathbb{H}}_0:({\gamma }_1,{\gamma }_2)=(0,1)\hspace{1em}\hspace{1em}\text{against}\hspace{1em}\hspace{1em}{\mathbb{H}}_A:({\gamma }_1,{\gamma }_2)\ne (0,1),$$

(1.9)

using the Wald-type test statistic  

$$T_{\mathrm{A}-\text{ESR}}=T({\widehat{\gamma }}_T-(0,1)){\widehat{\Omega }}_{\gamma }^{-1}{({\widehat{\gamma }}_T-(0,1))}^{\top },$$

(1.10)

based on some (consistent) covariance estimator ${\widehat{\Omega }}_{\gamma }$ for the covariance of the subvector γ.

The Strict ESR Backtest: We choose $V_t=W_t=(1,{\widehat{e}}_t)$⁠, i.e. we set up the regression system  

$$Y_t={\beta }_1+{\beta }_2{\widehat{e}}_t+u_t^q,\hspace{1em} \text{and} \hspace{1em}\hspace{1em}{Y}_t={\gamma }_1+{\gamma }_2{\widehat{e}}_t+u_t^e,$$

(1.11)

and test  

$${\mathbb{H}}_0:({\gamma }_1,{\gamma }_2)=(0,1)\hspace{1em}\hspace{1em}\text{against}\hspace{1em}\hspace{1em}{\mathbb{H}}_A:({\gamma }_1,{\gamma }_2)\ne (0,1),$$

(1.12)

using the Wald-type test statistic  

$$T_{\mathrm{S}-\text{ESR}}=T({\widehat{\gamma }}_T-(0,1)){\widehat{\Omega }}_{\gamma }^{-1}{({\widehat{\gamma }}_T-(0,1))}^{\top },$$

(1.13)

based on some (consistent) covariance estimator ${\widehat{\Omega }}_{\gamma }$ for the covariance of the subvector γ.

We discuss the employed covariance estimators ${\widehat{\Omega }}_{\gamma }$ in Section 1.5. Whereas setting up Mincer–Zarnowitz tests for classical elicitable functionals such as the mean, quantiles, and expectiles is straight-forward (see Mincer and Zarnowitz (1969), Gaglianone et al. (2011), Guler, Ng, and Xiao (2017)), in the case of higher-order elicitable functionals such as the ES, we have several choices as illustrated above. The Auxiliary ESR backtest is based on the regression specification in Equation (1.8) and requires both, VaR and ES forecasts as input variables. Thus, following Definition 1.1, this backtest is formally a joint VaR and ES backtest, however, with a strong emphasis on backtesting ES forecasts. In contrast, the Strict ESR backtest only incorporates ES forecasts and consequently is the first backtest for the ES stand-alone.

The Strict ESR test, however, comes at the cost of a potential model misspecification. Given that the financial returns Y_t follow some pure scale (volatility) process, it holds that the VaR and ES forecasts are perfectly colinear, ${\widehat{e}}_t=c{\widehat{v}}_t$ for some $c\in \mathbb{R}$⁠. Consequently, if ${\widehat{v}}_t$ equals the true conditional VaR, the first equation in (1.11) is correctly specified for the true parameter values $({\beta }_1,{\beta }_2)=(0,1/c)$⁠. Most of the financial econometrics literature (almost the entire GARCH, stochastic volatility, and Realized Volatility literature) is based on such an assumption for daily returns, which motivates the applicability of this Strict ESR backtest. However, this backtest is also applicable in the general case where the true VaR and ES forecasts are not necessarily colinear. For this, we provide asymptotic theory for M-estimation of the joint VaR and ES regression under potential model misspecification in Section 1.4.

1.3 The One-Sided Intercept ESR Backtest

The two ESR backtests introduced in the previous section only allow for testing two-sided hypotheses as specified in Equations (1.9) and (1.12), as it is generally unclear how too risky (or too conservative) forecasts influence the parameters γ₁ and γ₂. Because the capital requirements that the financial institutions have to keep as a reserve depend on the reported risk forecasts, the market participants have an incentive to report too risky forecasts for the ES in order to keep as little capital requirements as possible. In contrast, issuing too conservative risk forecasts and facing higher capital requirements do not have to be punished by the regulatory authorities.⁵ Thus, the regulators only have to prevent and consequently penalize the underestimation of financial risks, which can be done by using one-sided backtesting procedures. For example, the traffic light system (Basel Committee, 1996), currently implemented in the Basel Accords, is in fact a one-sided backtest for the hit ratios of VaR forecasts. Hence, we also introduce a regression-based backtesting procedure for the ES that allows for testing one-sided hypotheses.

The Intercept ESR Backtest: This backtest is based on a regression setup similar to the Strict ESR backtest by regressing the forecast errors, $Y_t-{\widehat{e}}_t$⁠, only on an intercept term in the ES-specific regression equation,  

$$Y_t-{\widehat{e}}_t={\beta }_1+{\beta }_2{\widehat{e}}_t+u_t^q,\hspace{1em} \text{and} \hspace{1em}\hspace{1em}{Y}_t-{\widehat{e}}_t={\gamma }_1+u_t^e,$$

(1.14)

where $Q_{\tau }(u_t^q|{\mathcal{F}}_{t-1})=0$ and ${\text{ES}}_{\tau }(u_t^e|{\mathcal{F}}_{t-1})=0$ almost surely. By using this restricted regression equation, we can define a one-sided and a two-sided alternative,  

$$\begin{array}{ll}{\mathbb{H}}_0^{2s}:{\gamma }_1=0\hspace{1em}\hspace{1em}\hfill & \text{against}\hspace{1em}\hspace{1em}{\mathbb{H}}_A^{2s}:{\gamma }_1\ne 0,\hspace{1em}\text{and}\hfill \\ {\mathbb{H}}_0^{1s}:{\gamma }_1\ge 0\hspace{1em}\hspace{1em}\hfill & \text{against}\hspace{1em}\hspace{1em}{\mathbb{H}}_A^{1s}:{\gamma }_1<0,\hfill \end{array}$$

(1.15)

which we test by using a t-test based on the estimated asymptotic covariance described in Section 1.5.

Note that this testing procedure is equivalent to fixing the slope parameter of the ES equation in the Strict ESR test given in Equation (1.11) to one and only estimating and testing the intercept term. Therefore, we call this backtest the Intercept ESR backtest. We keep the slope parameter in the quantile regression equation, as for pure scale models where ${\widehat{e}}_t=c{\widehat{v}}_t$⁠, it holds that ${\beta }_1=0$ and ${\beta }_2=(1-c)/c$ under the null hypothesis.

1.4 Asymptotic Theory under Model Misspecification

In this section, we consider the asymptotic properties of the M-estimator of the joint VaR and ES regression framework given in Equation (1.7) under potential model misspecification. In the following, we write $X_t=(V_t,W_t)$ for the compound vector of covariates. Following Dimitriadis and Bayer (2019) and Patton, Ziegel, and Chen (2019), the M-estimator of the regression parameters $\theta =(\beta ,\gamma )$ is defined by:  

$${\widehat{\theta }}_T=\underset{\theta \in \Theta }{\text{argmin}}{Q}_T(\theta ),\hspace{1em} \text{where}$$

(1.16)

 

$$Q_T(\theta )=\frac{1}{T}\sum _{t=1}^T\rho (Y_t,X_t,\theta )\hspace{1em}\hspace{1em} \text{and}$$

(1.17)

 

$$\rho (Y_t,X_t,\theta )=\frac{1}{-W_t^{\top }\gamma }\left(W_t^{\top }\gamma -V_t^{\top }\beta +\frac{(V_t^{\top }\beta -Y_t)1_{\{Y_t\le V_t^{\top }\beta \}}}{\tau }\right)+\text{log}(-W_t^{\top }\gamma ),$$

(1.18)

where the loss function in Equation (1.18) is a strictly consistent loss function for the pair quantile and ES (Fissler and Ziegel, 2016). Dimitriadis and Bayer (2019) and Patton, Ziegel, and Chen (2019) show consistency and asymptotic normality for the M-estimator in the case of a correctly specified parametric model, that is, under the assumption that there exists a true parameter ${\theta }_0\in \Theta$ such that $Q_{\tau }(u_t^q|{\mathcal{F}}_{t-1})=0$ and $E{S}_{\tau }(u_t^e|{\mathcal{F}}_{t-1})=0$⁠, almost surely. In the following, we extend this theory by relaxing these assumptions which allows for the general case of misspecified models. For this, we define the pseudo-true parameter  

$${\theta }_T^{\ast }=\underset{\theta \in \Theta }{\text{arg min}}{Q}_T^0(\theta ),\hspace{1em} \text{where} \hspace{1em}\hspace{1em}{Q}_T^0(\theta )=\mathbb{E}[Q_T(\theta )].$$

(1.19)

For the classical case of a correctly specified model, the pseudo-true parameter coincides with the true regression parameter ${\theta }_T^{\ast }={\theta }_0$ and is independent of T. In the following, we restrict our attention to processes and models for the conditional quantile and ES which follow the following conditions. 

Assumption 1.2

(A1) The distribution F_t is absolutely continuous with density function f_t, which is bounded from above, that is, there exists a constant c > 0 s.t. $\underset{y\in \mathbb{R}}{\sup }{f}_t(y)\le c$ and $\underset{y\in \mathbb{R}}{\sup }{f}_t^{\prime }(y)\le c$⁠.

(A2) The parameter space $\Theta \subseteq {\mathbb{R}}^{2k}$ is compact, convex, and has nonempty interior.

(A3) We assume that the pseudo-true parameter ${\theta }_T^{\ast }$ defined in Equation (1.19) is in the interior of Θ and is the unique minimizer of the objective function $Q_T^0(\theta )$ and that the sequence ${\nabla }_{\theta }{\mathbb{E}}_t[\rho (Y_t,X_t,{\theta }_T^{\ast })]$ is uncorrelated.

(A4) $V_t,W_t\in {\mathcal{F}}_{t-1}$ and the matrices $\mathbb{E}[V_t{V}_t^{\top }]$ and $\mathbb{E}[W_t{W}_t^{\top }]$ have full rank.

(A5) The matrix Λ_T, defined in Theorem 1.4 has strictly positive Eigenvalues for all T sufficiently large enough.

(A6) The stochastic process $\{Y_t,V_t,W_t\}$ is strong mixing of size $-r/(r-2)$ for some r > 2.

(A7) For all $\theta \in \Theta$⁠, it holds that $|\frac{1}{W_t^{\top }\gamma }|\le K<\infty$ for some constant K > 0.

(A8) It holds that $\mathbb{E}\left[||V_t|{|}^{r+1}\right]<\infty ,\hspace{0.17em}\mathbb{E}\left[||W_t|{|}^{r+1}\right]<\infty ,\hspace{0.17em}\mathbb{E}\left[||V_t|{|}^{r+1}||W_t|{|}^r\right]<\infty$ and $\mathbb{E}\left[||W_t|{|}^{r+1}|Y_t{|}^r\right]<\infty$ for the r > 2 from condition (A6).

(A9) For any $T\in \mathbb{N},\hspace{0.17em}\underset{\theta \in \Theta }{\sup }\sum _{t=1}^T 1_{\{Y_t=V_t^{\top }\beta \}}\le K$ a.s. for some constant K > 0.

The conditions in Assumption 1.2 mainly resemble the regularity conditions for asymptotic normality for correctly specified models from Patton, Ziegel, and Chen (2019) and we refer to Patton, Ziegel, and Chen (2019) for a discussion of these conditions. The key condition that allows for misspecified models is the unique minimization condition of the pseudo-true parameter ${\theta }_T^{\ast }$ in condition (A3). The above assumptions contain the case of correctly specified models as then, the condition (A3) is naturally fulfilled as the utilized loss function is a strictly consistent loss function for the VaR and the ES (Fissler and Ziegel, 2016).

We connect this weaker condition (A3) to classical misspecified regression models for the mean and for quantiles of White (1980), Gourieroux, Monfort, and Trognon (1984), Kim and White (2003), Komunjer (2005), and Angrist, Chernozhukov, and Fernandez-Val (2006). For correctly specified models, we usually impose the strong condition that for all $t=1,\dots ,T$⁠,  

$${\mathbb{E}}_t[\psi (Y_t,X_t,\theta )]=0\hspace{1em}\mathrm{a}.\mathrm{s}.\hspace{1em}\hspace{1em}\iff \hspace{1em}\hspace{1em}\theta ={\theta }_{\mathrm{T}}^{\ast },$$

(1.20)

where $\psi (Y_t,X_t,\theta )$ is almost surely the derivative of $\rho (Y_t,X_t,\theta )$ and corresponds to the identification functions of the model (Gneiting, 2011). The weaker condition (A3) is essentially equivalent to the unconditional moment condition  

$$\mathbb{E}\left[\frac{1}{T}\sum _{t=1}^T\psi (Y_t,X_t,\theta )\right]=0\hspace{1em}\hspace{1em}\iff \hspace{1em}\hspace{1em}\theta ={\theta }_T^{\ast }.$$

(1.21)

Thus, the condition (1.21) can be interpreted as an average identification condition, that is, $V_t^{\top }{\beta }_T^{\ast }$ and $W_t^{\top }{\gamma }_T^{\ast }$ are some best averaged linear approximations of the true unknown conditional quantile and ES models.

Theorem 1.3 (Consistency Misspecified Model). Given the conditions fromAssumption 1.2, it holds that${\widehat{\theta }}_T-{\theta }_T^{\ast }{\displaystyle \stackrel{\mathbb{P}}{\to }}0$, as $T\to \infty$, where ${\theta }_T^{\ast }$is the pseudo-true parameter as defined in Equation (1.19).

The proof of Theorem 1.3 is given in Appendix A.

Theorem 1.4 (Asymptotic Normality Misspecified Model). Given the conditions ofAssumption 1.2, it holds that 

$${\Sigma }_T{({\theta }_T^{\ast })}^{-1/2}{\Lambda }_T({\theta }_T^{\ast })\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast }){\displaystyle \stackrel{d}{\to }}\mathcal{N}\left(0,I_{2k}\right),$$

(1.22)

where  

$${\Lambda }_T({\theta }_T^{\ast })=\left(\begin{array}{cc}{\Lambda }_{11,T}({\theta }_T^{\ast })& {\Lambda }_{12,T}({\theta }_T^{\ast })\\ {\Lambda }_{21,T}({\theta }_T^{\ast })& {\Lambda }_{22,T}({\theta }_T^{\ast })\end{array}\right)\hspace{1em}\text{and}\hspace{1em}{\Sigma }_T({\theta }_T^{\ast })=\left(\begin{array}{cc}{\Sigma }_{11,T}({\theta }_T^{\ast })& {\Sigma }_{12,T}({\theta }_T^{\ast })\\ {\Sigma }_{21,T}({\theta }_T^{\ast })& {\Sigma }_{22,T}({\theta }_T^{\ast })\end{array}\right)$$

(1.23)

with  

$${\Lambda }_{11,T}({\theta }_T^{\ast })=-\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left[V_t{V}_t^{\top }{f}_t\left(V_t^{\top }{\beta }_T^{\ast }\right)\frac{1}{\tau W_t^{\top }{\gamma }_T^{\ast }}\right],$$

(1.24)

 

$${\Lambda }_{12,T}({\theta }_T^{\ast })={\Lambda }_{21,T}^{\top }({\theta }_T^{\ast })=\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left[V_t{W}_t^{\top }\frac{1}{{(W_t^{\top }{\gamma }_T^{\ast })}^2}\frac{F_t(V_t^{\top }{\beta }_T^{\ast })-\tau }{\tau }\right],$$

(1.25)

 

$${\Lambda }_{22,T}({\theta }_T^{\ast })=\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left[W_t{W}_t^{\top }\frac{1}{{(W_t^{\top }{\gamma }_T^{\ast })}^2}\right]$$

(1.26)

 

$$-\frac{2}{T}\sum _{t=1}^T\mathbb{E}\left[W_t{W}_t^{\top }\frac{1}{{(W_t^{\top }{\gamma }_T^{\ast })}^3}\left(W_t^{\top }{\gamma }_T^{\ast }-\frac{1}{\tau }{\mathbb{E}}_t\left[Y_t{1}_{\{Y_t\le V_t^{\top }{\beta }_T^{\ast }\}}\right]+V_t^{\top }{\beta }_T^{\ast }\frac{F_t(V_t^{\top }{\beta }_T^{\ast })-\tau }{\tau }\right)\right],$$

(1.27)

and  

$${\Sigma }_{11,T}({\theta }_T^{\ast })=\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left[V_t{V}_t^{\top }\frac{1}{{(W_t^{\top }{\gamma }_T^{\ast })}^2}\left(\frac{1-\tau }{\tau }+\frac{(1-2\tau )(F_t(V_t^{\top }{\beta }_T^{\ast })-\tau )}{{\tau }^2}\right)\right],$$

(1.28)

 

$${\Sigma }_{12,T}({\theta }_T^{\ast })=\frac{1}{T}\sum _{t=1}^T\mathbb{E}[V_t{W}_t^{\top }\frac{1}{-{(W_t^{\top }{\gamma }_T^{\ast })}^3}\{\frac{1-\tau }{\tau }\left(V_t^{\top }{\beta }_T^{\ast }-W_t^{\top }{\gamma }_T^{\ast }\right)$$

(1.29)

 

$$+\frac{1-\tau }{\tau }\left(V_t^{\top }{\beta }_T^{\ast }\frac{F_t(V_t^{\top }{\beta }_T^{\ast })-\tau }{\tau }+W_t^{\top }{\gamma }_T^{\ast }-\frac{1}{\tau }{\mathbb{E}}_t\left[Y_t{1}_{\{Y_t\le V_t^{\top }{\beta }_T^{\ast }\}}\right]\right)$$

(1.30)

 

$$-\frac{F_t(V_t^{\top }{\beta }_T^{\ast })-\tau }{\tau }\left(V_t^{\top }{\beta }_T^{\ast }-W_t^{\top }{\gamma }_T^{\ast }\right)\}],$$

(1.31)

 

$${\Sigma }_{22,T}({\theta }_T^{\ast })=\frac{1}{T}\sum _{t=1}^T\mathbb{E}[W_t{W}_t^{\top }\frac{1}{{(W_t^{\top }{\gamma }_T^{\ast })}^4}\{\frac{1}{\tau }{\text{Var}}_t(V_t^{\top }{\beta }_T^{\ast }-Y_t|Y_t\le V_t^{\top }{\beta }_T^{\ast })+\frac{1-\tau }{\tau }{(V_t^{\top }{\beta }_T^{\ast }-W_t^{\top }{\gamma }_T^{\ast })}^2$$

(1.32)

 

$$+2(V_t^{\top }{\beta }_T^{\ast }-W_t^{\top }{\gamma }_T^{\ast })V_t^{\top }{\beta }_T^{\ast }\frac{\tau -F_t(V_t^{\top }{\beta }_T^{\ast })}{\tau }\}].$$

(1.33)

The proof of Theorem 1.4 is given in Appendix A. The asymptotic theory derived here embeds the asymptotic theory of Patton, Ziegel, and Chen (2019) and Dimitriadis and Bayer (2019) in the simplified case of correctly specified models. Correct specification implies that $F_t(V_t^{\top }{\beta }_T^{\ast })=\tau$ and $W_t^{\top }{\gamma }_T^{\ast }=\frac{1}{\tau }{\mathbb{E}}_t[Y_t{1}_{\{Y_t\le V_t^{\top }{\beta }_T^{\ast }\}}]$ almost surely for all $t=1,\dots ,T$⁠. Imposing these two conditions simplifies the asymptotic covariance matrix of Theorem 1.4 to the asymptotic covariances from Patton, Ziegel, and Chen (2019) and Dimitriadis and Bayer (2019). In general, allowing for model misspecification in regression models comes at the cost of an inflated and more complicated asymptotic covariance matrix, see White (1980), White (1994), Kim and White (2003), Komunjer (2005), and Angrist, Chernozhukov, and Fernandez-Val (2006) for examples of semiparametric models for the mean and quantiles.

Given consistency and asymptotic normality, we can derive the asymptotic distribution of the test statistics of the regression-based ESR backtests. Henceforth, we use the short notation ${\Omega }_T={\Lambda }_T{({\theta }_T^{\ast })}^{-1}{\Sigma }_T({\theta }_T^{\ast }){\Lambda }_T{({\theta }_T^{\ast })}^{-1}$ for the asymptotic covariance. As the Auxiliary ESR backtest is not subject to model misspecification, under the null hypothesis it holds that ${\gamma }_T^{\ast }=(0,1)$ for all $T\in \mathbb{N}$⁠. However, this does not necessarily hold for the Strict ESR and the Intercept ESR backtests and we define the following modified test statistics for these backtests,  

$${\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }^0)=T\left({\widehat{\gamma }}_T-{\gamma }^0\right){\widehat{\Omega }}_{T,\gamma }^{-1}{\left({\widehat{\gamma }}_T-{\gamma }^0\right)}^{\top },$$

(1.34)

 

$${\tilde{T}}_{\mathrm{I}-\text{ESR}}({\gamma }^0)=T\left({\widehat{\gamma }}_{1,T}-{\gamma }_1^0\right){\widehat{\Omega }}_{T,{\gamma }_1}^{-1}{\left({\widehat{\gamma }}_{1,T}-{\gamma }_1^0\right)}^{\top },$$

(1.35)

depending on the parameter ${\gamma }^0$⁠, which we test for. The matrices ${\widehat{\Omega }}_{T,\gamma }$ and ${\widehat{\Omega }}_{T,{\gamma }_1}$ are the ES-specific parts of the estimators for the asymptotic covariance matrix and ${\gamma }_{1,T}^{\ast }$ and ${\gamma }_1^0$ refer to the intercept components of the ES-specific parameter vectors ${\gamma }_T^{\ast }$ and ${\gamma }^0$⁠. Given these modified test statistics, we can state the following corollary.

Corollary 1.5Given the conditions ofAssumption 1.2 and given that${\widehat{\Omega }}_T-{\Omega }_T{\displaystyle \stackrel{\mathbb{P}}{\to }}0$, it holds that 

$$T_{\mathrm{A}-\text{ESR}}{\displaystyle \stackrel{d}{\to }}{\chi }_2^2,\hspace{1em}{\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{\ast }){\displaystyle \stackrel{d}{\to }}{\chi }_2^2,\hspace{1em} \text{and} \hspace{1em}\hspace{1em}{\tilde{T}}_{\mathrm{I}-\text{ESR}}({\gamma }_{1,T}^{\ast }){\displaystyle \stackrel{d}{\to }}{\chi }_1^2.$$

(1.36)

The proof of Corollary 1.5 is given in Appendix A. For the Auxiliary ESR test, we can simply use the test statistic $T_{A-\mathit{ESR}}$ in order to test whether ${\gamma }_T^{\ast }=(0,1)$⁠. However, for the Strict and Intercept ESR tests, we do not know the exact form of the pseudo-true parameter ${\gamma }_T^{\ast }$ under the null hypothesis due to the potential model misspecification. Consequently, we derive the distribution of the test statistic ${\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{\ast })$ at the pseudo-true parameter ${\gamma }_T^{\ast }$⁠. In the following, we argue that in realistic financial settings, it holds that ${\gamma }_T^{\ast }\approx (0,1)$ and thus, ${\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{\ast })\approx {\tilde{T}}_{\mathrm{S}-\text{ESR}}((0,1))=T_{\mathrm{S}-\text{ESR}}$ holds approximately, and equivalently for the Intercept ESR test. This implies that these tests still have approximately correct size under these slightly misspecified null hypotheses.

First, the majority of the literature in financial econometrics finds that pure scale processes (e.g., GARCH and stochastic volatility models) approximate the true underlying daily financial data well enough. Thus, ${\widehat{e}}_t\approx c{\widehat{v}}_t$ for some c > 0 and we find that under the null hypothesis, the regression model in Equation (1.11) is only subject to slight model misspecification. Second, the misspecification is in the auxiliary quantile equation, while we test the parameters of the ES equation in Equation (1.11), which is correctly specified under the null. Thus, the model misspecification enters our test statistic only indirectly through the auxiliary effect of the joint parameter estimation. We confirm that these approximations are very precise in the simulation setup of Appendix B, even for cases of unrealistically strong model misspecification. Furthermore, the simulation study in Section 2 confirms these results by showing that the Strict ESR backtest based on $T_{\mathrm{S}-\text{ESR}}$ exhibits correct size and performs almost indistinguishably to the Auxiliary ESR backtest, also in the simulation setups where the underlying data do not follow a pure scale processes. This shows that the approximation error is negligible under the null hypothesis in realistic financial settings and that the Strict and Intercept ESR backtests can indeed be applied in practice.

The following corollary specifies the behavior of our tests under alternative hypotheses. For this, we define the hypothetical parameter ${\gamma }_T^{{\mathbb{H}}_0}$ as the ES-specific pseudo-true parameter of the model for correctly specified ES forecasts. While for correctly specified regression equations it holds that ${\gamma }_T^{{\mathbb{H}}_0}=(0,1)$⁠, its exact form is unknown in the general case.

Corollary 1.6Under the alternative hypotheses,  

$${\mathbb{H}}_A^{\mathrm{A}-\text{ESR}}:\hspace{0.17em}{\gamma }_T^{\ast }\cancel{=}(0,1),\hspace{1em}{\mathbb{H}}_A^{\mathrm{S}-\text{ESR}}:\hspace{0.17em}{\gamma }_T^{\ast }\cancel{=}{\gamma }_T^{{\mathbb{H}}_0},\hspace{1em} \text{and} \hspace{1em}\hspace{1em}{\mathbb{H}}_A^{\mathrm{I}-\text{ESR}}:\hspace{0.17em}{\gamma }_{1,T}^{\ast }\cancel{=}{\gamma }_{1,T}^{{\mathbb{H}}_0},$$

(1.37)

for all $T\ge T_A$ for some $T_A\in \mathbb{N}$⁠, and given the conditions of Assumption 1.2 and given that ${\widehat{\Omega }}_T-\Omega {\displaystyle \stackrel{\mathbb{P}}{\to }}0$⁠, it holds that for all c > 0,  

$$\mathbb{P}\left(T_{\mathrm{A}-\text{ESR}}\ge c\right){\displaystyle \stackrel{\mathbb{P}}{\to }}1,\hspace{1em}\mathbb{P}\left({\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{{\mathbb{H}}_0})\ge c\right){\displaystyle \stackrel{\mathbb{P}}{\to }}1,\hspace{1em} \text{and} \hspace{1em}\hspace{1em}\mathbb{P}\left({\tilde{T}}_{\mathrm{I}-\text{ESR}}({\gamma }_{1,T}^{{\mathbb{H}}_0})\ge c\right){\displaystyle \stackrel{\mathbb{P}}{\to }}1.$$

(1.38)

The proof of Corollary 1.6 is given in Appendix A. While the parameter ${\gamma }_T^{{\mathbb{H}}_0}$ is unknown in the general case, we argue above that ${\gamma }_T^{{\mathbb{H}}_0}\approx (0,1)$ still holds approximately in realistic settings and consequently ${\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{{\mathbb{H}}_0})\approx {\tilde{T}}_{\mathrm{S}-\text{ESR}}((0,1))=T_{\mathrm{S}-\text{ESR}}$⁠. Corollary 1.6 theoretically implies diverging power for any case where ${\gamma }_T^{\ast }\cancel{=}{\gamma }_T^{{\mathbb{H}}_0}$⁠, that is, also in misspecified cases when ${\gamma }_T^{{\mathbb{H}}_0}\cancel{=}(0,1)$⁠, and thus, diverging power by employing the approximated test statistic $T_{\mathrm{S}-\text{ESR}}={\tilde{T}}_{\mathrm{S}-\text{ESR}}((0,1))$⁠. While this holds theoretically, the empirical performance of the Strict and Intercept ESR tests is almost entirely unaffected by the small approximation error stemming from the indirect misspecification of the quantile model, as can be seen in the simulation results in Appendix B and in Section 2.

1.5 Implementation of the Tests

The M-estimation of the parameters ${\widehat{\theta }}_T$ is carried out by using the R package esreg (Bayer and Dimitriadis, 2019b). The main difficulty in the implementation of the backtests is estimation of the asymptotic covariance matrix ${\Omega }_T={\Lambda }_T{({\theta }_T^{\ast })}^{-1}{\Sigma }_T({\theta }_T^{\ast }){\Lambda }_T{({\theta }_T^{\ast })}^{-1}$⁠. Generally, this is implemented by using the sample counterparts of the expectation of the components given in Equations (1.24)–(1.33) in Theorem 1.4, wich are however subject to the following four nuisance quantities:

• (a) the conditional density function, evaluated at the conditional quantile, ${\widehat{f}}_t(V_t^{\top }{\widehat{\beta }}_T)$⁠,

• (b) the conditional truncated variance, ${\widehat{\text{Var}}}_t(V_t^{\top }{\widehat{\beta }}_T-Y_t|Y_t\le V_t^{\top }{\widehat{\beta }}_T)$⁠,

• (c) the conditional distribution function, ${\widehat{F}}_t(V_t^{\top }{\widehat{\beta }}_T)$⁠, and

• (d) the conditional truncated expectation $\frac{1}{\tau }{\mathbb{E}}_t\left[Y_t{1}_{\{Y_t\le V_t^{\top }{\widehat{\beta }}_T\}}\right]$⁠.

We implement a novel and misspecification robust covariance estimator by estimating the four nuisance quantities above in the following way. The terms (a) and (b) are subject to the asymptotic covariance of correctly specified models for the quantile and the ES of Dimitriadis and Bayer (2019), Patton, Ziegel, and Chen (2019), and Barendse (2020). Thus, we follow the approach of Dimitriadis and Bayer (2019) and apply the nid estimator of Hendricks and Koenker (1992) for (a), the conditional density and the flexible scl-sp estimator of Dimitriadis and Bayer (2019) for (b), the conditional truncated variance.

In order to estimate (c), the conditional distribution function ${\widehat{F}}_t(V_t^{\top }{\widehat{\beta }}_T)$⁠, we follow the general approach of the scl-sp estimator of Dimitriadis and Bayer (2019), that is, we assume that F_t follows a conditional location-scale model with innovations ${\epsilon }_t$ with a flexible zero mean and unit variance distribution. We standardize Y_t by the estimates of the conditional mean and variance, estimated by pseudo-maximum likelihood and apply a kernel density estimator in order to obtain the distribution function of ${\epsilon }_t$⁠. Hence, we can recover the distribution of Y_t given ${\mathcal{F}}_{t-1}$⁠. Notice that for the minor degree of misspecification we are subject to in our backtesting approach, it approximately holds that ${\widehat{F}}_t(V_t^{\top }{\widehat{\beta }}_T)\approx \tau$ for all t. We find that this semiparametric estimation approach, which is subject to the location-scale assumption, performs better than pure nonparametric alternatives as we are estimating the conditional distribution evaluated at rather extreme quantiles such as at $\tau =2.5\%$⁠.

The last nuisance quantity, $\frac{1}{\tau }{\mathbb{E}}_t\left[Y_t{1}_{\{Y_t\le V_t^{\top }{\widehat{\beta }}_T\}}\right]$⁠, is the mean, given the observations are smaller than the possibly misspecified linear quantile model. This quantity is closely related to the conditional ES, which is assumed to be a linear function in our approach. As for realistic financial data, we only face a minor degree of misspecification in the quantile model, this nuisance quantity is assumed to still be approximately linear, and thus, we obtain that $\frac{1}{\tau }{\mathbb{E}}_t\left[Y_t{1}_{\{Y_t\le V_t^{\top }{\widehat{\beta }}_T\}}\right]=W_t^{\top }{\widehat{\gamma }}_T$ for all t. Nonparametric estimation of this nuisance quantity again introduces too much estimation noise.

We further implement our backtests based on a covariance estimator from Dimitriadis and Bayer (2019) and Patton, Ziegel, and Chen (2019), which does not account for possible model misspecification. This estimator is based on the simplified covariance structure given in Dimitriadis and Bayer (2019) and Patton, Ziegel, and Chen (2019), where the correct model specification assumption implies that $F_t(V_t^{\top }{\beta }_T^{\ast })=\tau$ and $\frac{1}{\tau }{\mathbb{E}}_t\left[Y_t{1}_{\{Y_t\le V_t^{\top }{\beta }_T^{\ast }\}}\right]=W_t^{\top }{\gamma }_T^{\ast }$⁠, almost surely. Thus, we only estimate the nuisance quantities (a) and (b) in this approach.

2 Monte-Carlo Simulations

In this section, we evaluate the empirical performance of our proposed ESR backtests and compare them to the tests of McNeil and Frey (2000) and Nolde and Ziegel (2017). For this, we assess the empirical size and power of the tests, which are defined as the rejection frequency of the tests under the null and alternative hypothesis, respectively. This comparison is conducted using two different approaches. The first, presented in Section 2.1, follows the typical strategy in the related literature of first assessing the size of the backtests with several realistic DGPs, followed by an evaluation of the power by backtesting forecasts stemming from an overly simplified model, in this case the Historical Simulation (HS) model. In the second setup, presented in Section 2.2, we continuously misspecify certain parameters of the true model and thereby obtain alternative models with a continuously increasing degree of misspecification. This approach of evaluating backtests has two advantages. First, we obtain power curves which can be used to draw conclusions of how an increasing model misspecification influences the test decisions. Second, misspecifying the different model parameters in isolation allows us to misspecify certain model characteristics while leaving the remaining model unchanged.

2.1 Traditional Size and Power Comparisons

In order to compare the proposed backtests from the previous sections, we simulate data from several DGPs. Besides pure scale (volatility) model specifications, under which the Strict and Intercept ESR backtests are correctly specified, we also consider more general Student’s-t GAS models (Creal, Koopman, and Lucas 2013) with time-varying higher moments and AR-GARCH specifications where our ESR backtests are subject to model misspecification under the null hypothesis.

EGARCH: The first DGP is an EGARCH(1,1) model (Nelson, 1991) with t-distributed innovations, where the parameter values are calibrated to daily returns of the S&P 500 index,  

$$\begin{array}{l}{Y}_t={\sigma }_t{z}_t,\hspace{1em} \text{where} \hspace{1em}{z}_t\sim t_{7.39},\hspace{1em} \text{and} \hfill \\ \hspace{0.17em}\text{log}\hspace{0.17em}({\sigma }_t^2)=-0.0012-0.161z_{t-1}+0.136\left(|z_{t-1}|-E[|z_{t-1}|]\right)+0.978\hspace{0.17em}\text{log}\hspace{0.17em}({\sigma }_{t-1}^2).\hfill \end{array}$$

(2.1)

This model represents a highly flexible GARCH specification and due to its calibrated parameter values, this DGP accurately replicates the distributional properties of daily financial returns. As we assume zero mean for this model, the true VaR and ES forecasts are perfectly colinear and consequently, the regression equations for the Strict and the Intercept ESR backtests are correctly specified under the null hypothesis.

AR-GARCH: The next specification is an AR(1)-GARCH(1,1) model with Gaussian innovations,  

$$\begin{array}{l}{Y}_t=\varphi Y_{t-1}+{\sigma }_t{z}_t,\hspace{1em} \text{where} \hspace{1em}{z}_t\sim \mathcal{N}(0,1),\hspace{1em} \text{and} \hfill \\ {\sigma }_t^2=0.01+0.1Y_{t-1}^2+0.85{\sigma }_{t-1}^2,\hfill \end{array}$$

(2.2)

where we consider the three specifications $\varphi \in \{0,0.1,0.5\}$ for the AR parameter. This DGP introduces model misspecification for the Strict and Intercept ESR backtests through the non-zero conditional mean specification, while leaving the realistic volatility structure of the financial returns unchanged. For this DGP, the ratio between true ES and VaR is given by:  

$$\frac{{\widehat{e}}_t}{{\widehat{v}}_t}=\frac{{\mu }_t+{\sigma }_t{q}_z(\tau )}{{\mu }_t+{\sigma }_t{\xi }_z(\tau )},$$

(2.3)

where μ_t is the conditional mean of Y_t given ${\mathcal{F}}_{t-1}$ and $q_z(\tau )$ and ${\xi }_z(\tau )$ are the τ-quantile, respectively the τ-ES of the innovations $z_t$⁠. If μ_t equals zero, the ratio is constant and thus, the regression equations in (1.11) are correctly specified under the null. By increasing the time-dependence of the conditional mean model through the AR parameter, we can monotonically strengthen the model misspecification in this DGP.

GAS-STD: We use a 3-factor Student’s-t GAS model with time-varying location μ_t, scale σ_t, and degrees of freedom ξ_t with parameters calibrated to daily returns of the S&P 500 index. This model is estimated and simulated by using the R package GAS (Ardia, Boudt, and Catania, 2019) and is based on the following model specification  

$$Y_t|(Y_1,\dots ,Y_{t-1})\sim t({\mu }_t,{\sigma }_t,{\xi }_t),$$

(2.4)

where the vector $({\mu }_t,{\sigma }_t,{\xi }_t)$ follows an autoregressive specification, driven by the lagged score of the log-likelihood of the distributional specification in Equation (2.4). Creal, Koopman, and Lucas (2013) and Harvey (2013) introduce the general GAS specification, which nests many well known models, including ARMA, GARCH (Bollerslev, 1986), and ACD (Engle and Russell, 1998) models. Koopman, Lucas, A., and Scharth (2016) provides an overview of GAS and related models. We refer to Appendix A of Ardia, Boudt, and Catania (2019) for the exact parametric specification of this Student’s-t GAS model.

GAS-SSTD: We generalize the previous GAS model to a 4-factor asymmetric Student’s-t GAS model with time-varying location μ_t, scale σ_t, skewness λ_t, and degrees of freedom ξ_t,  

$$Y_t|(Y_1,\dots ,Y_{t-1})\sim t({\mu }_t,{\sigma }_t,{\lambda }_t,{\xi }_t).$$

(2.5)

Compared to the previous 3-factor GAS specification, this model further allows for asymmetries in the conditional return distribution through allowing for an additional time-varying skewness parameter with an autoregressive GAS specification.

For the two location-scale DGPs, we obtain VaR and ES forecasts at level τ by:  

$${\widehat{v}}_t={\widehat{\mu }}_t+{\widehat{\sigma }}_t{q}_z(\tau )\hspace{1em}\hspace{1em}\text{and}\hspace{1em}\hspace{1em}{\widehat{e}}_t={\widehat{\mu }}_t+{\widehat{\sigma }}_t{\xi }_z(\tau ),$$

(2.6)

where ${\widehat{\mu }}_t$ and ${\widehat{\sigma }}_t$ are the respective location and volatility forecasts generated by the location and scale models and $q_z(\tau )$ and ${\xi }_z(\tau )$ are the τ-quantile, respectively the τ-ES of the innovations z_t. For the t-distributions of the two GAS models, we obtain the ES forecasts through numerical integration. For the following size and power analysis of the backtests, we simulate data from the DGPs given above with varying sample sizes of 250, 500, 1000, 2500, and 5000 observations and 250 additional pre-sample values required for the power analysis. We run 10,000 Monte-Carlo replications for each of the DGPs. As stipulated by the Basel Accords, we fix the probability level to $\tau =2.5\%$ for the VaR and ES forecasts for each of the DGPs. In this part of the study, we focus on two-sided hypotheses and defer the one-sided case to Section 2.3. We compare our three ESR backtests to two specifications of the conditional calibration (CC) backtest of Nolde and Ziegel (2017) and to two specifications of the exceedance residual (ER) backtests of McNeil and Frey (2000), which are further described in Section S.1.2.1 and Section S.1.2.2 in the Supplementary Appendix.

Table 1 presents the empirical sizes of the considered backtests for the different DGPs introduced above and for the different sample sizes and a nominal test size of 5%. Table S.1 and Table S.2 in Section S.1.3 in the Supplementary Appendix show equivalent results for nominal significance levels of 1% and 10%. We find that in large samples, all backtests display rejection rates close to the respective nominal size for all considered DGPs. However, in small samples, the ESR tests based on the misspecification covariance estimator exhibit much better sizes compared to the equivalent ESR tests which do not account for the potential misspecification. As this holds for both, DGPs which do and do not generate misspecification under the null, this indicates that the misspecification covariance estimator better approximates the finite sample distribution and should consequently be applied in empirical applications.

Table 1.

Empirical sizes for the first simulation study

DGP	Sample size	Str. ESR	Aux. ESR	Int. ESR	Str. ESR	Aux. ESR	Int. ESR	Gen. CC	Sim. CC	Std. ER	ER
		Misspec Covariance			Classical Covariance
EGARCH-STD	250	0.09	0.09	0.04	0.24	0.25	0.15	0.08	0.29	0.07	0.09
	500	0.06	0.06	0.05	0.15	0.15	0.11	0.10	0.20	0.04	0.07
	1000	0.05	0.05	0.04	0.11	0.11	0.09	0.09	0.14	0.05	0.07
	2500	0.04	0.04	0.04	0.06	0.06	0.06	0.07	0.09	0.05	0.06
	5000	0.04	0.04	0.05	0.06	0.06	0.06	0.06	0.08	0.05	0.06
GAS-STD	250	0.10	0.10	0.05	0.26	0.26	0.15	0.07	0.28	0.07	0.08
	500	0.08	0.08	0.05	0.16	0.16	0.11	0.10	0.20	0.06	0.06
	1000	0.06	0.06	0.05	0.11	0.11	0.09	0.09	0.14	0.06	0.07
	2500	0.05	0.05	0.05	0.08	0.08	0.07	0.07	0.10	0.06	0.06
	5000	0.04	0.05	0.05	0.06	0.06	0.06	0.06	0.08	0.06	0.06
GAS-SSTD	250	0.09	0.09	0.05	0.25	0.25	0.15	0.07	0.26	0.08	0.07
	500	0.06	0.06	0.04	0.15	0.15	0.10	0.09	0.18	0.06	0.05
	1000	0.05	0.05	0.04	0.10	0.10	0.08	0.08	0.13	0.07	0.06
	2500	0.04	0.04	0.04	0.06	0.06	0.06	0.07	0.09	0.06	0.05
	5000	0.04	0.04	0.04	0.05	0.05	0.05	0.07	0.07	0.06	0.05
AR-GARCH, $\varphi =0.0$	250	0.05	0.05	0.03	0.18	0.18	0.11	0.06	0.22	0.06	0.07
	500	0.04	0.04	0.03	0.12	0.12	0.09	0.07	0.14	0.04	0.04
	1000	0.04	0.04	0.03	0.09	0.09	0.07	0.07	0.10	0.04	0.04
	2500	0.03	0.03	0.04	0.06	0.06	0.06	0.06	0.08	0.05	0.05
	5000	0.04	0.04	0.04	0.06	0.06	0.05	0.05	0.06	0.05	0.05
AR-GARCH, $\varphi =0.1$	250	0.05	0.05	0.03	0.18	0.18	0.11	0.06	0.22	0.06	0.07
	500	0.04	0.04	0.03	0.12	0.12	0.08	0.07	0.14	0.04	0.04
	1000	0.04	0.04	0.03	0.09	0.09	0.07	0.07	0.10	0.04	0.04
	2500	0.03	0.03	0.04	0.07	0.07	0.06	0.06	0.08	0.05	0.05
	5000	0.04	0.04	0.04	0.06	0.06	0.05	0.05	0.06	0.05	0.05
AR-GARCH, $\varphi =0.5$	250	0.04	0.04	0.02	0.17	0.17	0.11	0.06	0.22	0.06	0.07
	500	0.04	0.04	0.03	0.12	0.12	0.09	0.07	0.14	0.04	0.04
	1000	0.04	0.04	0.03	0.09	0.09	0.07	0.07	0.10	0.04	0.04
	2500	0.04	0.04	0.03	0.07	0.07	0.06	0.06	0.08	0.05	0.05
	5000	0.04	0.04	0.04	0.06	0.06	0.05	0.05	0.06	0.05	0.05

Notes: This table reports the empirical sizes of the backtests for the different DGPs described in Section 2.1 and for a nominal test size of 5%. The number of Monte-Carlo repetitions is 10,000 and the probability level for the risk measures is $\tau =2.5\%$⁠. ESR refers to the three backtests introduced in this article and we consider versions with covariance estimation with and without model misspecification. CC refers to the conditional calibration tests of Nolde and Ziegel (2017), and ER to the exceedance residuals tests of McNeil and Frey (2000).

Table 2.

Empirical sizes for the second simulation study

DGP	Str. ESR	Aux. ESR	Int. ESR	Str. ESR	Aux. ESR	Int. ESR	General CC	Simple CC	Std. ER	ER
	Misspec covariance			Classical covariance
Two-sided	0.07	0.07	0.07	0.05	0.05	0.05	0.07	0.09	0.05	0.05
One-sided	–	–	0.02	–	–	0.01	0.02	0.03	0.06	0.06

Notes: This table shows the empirical sizes of the backtests for the GARCH(1,1)-t model given in Equation (2.8), for a nominal test size of 5% and for both, one-sided and two-sided hypotheses. The number of Monte-Carlo repetitions is 10,000 and the probability level for the risk measures is $\tau =2.5\%$⁠. ESR refers to the backtests introduced in this article. CC refers to the conditional calibration tests of Nolde and Ziegel (2017), and ER to the exceedance residuals tests of McNeil and Frey (2000). Note that the Strict and Auxiliary ESR tests do not permit testing against a one-sided alternative and therefore, we only present sizes for the two-sided hypothesis.

We further find that the Strict ESR test and the Auxiliary ESR test perform almost identical throughout all considered DGPs. This implies that the indirect misspecification the Strict ESR test introduces is negligible for realistic financial data. Even for the AR-GARCH model with increasing AR parameter $\varphi$⁠, the size properties of the Strict and the Intercept ESR tests are not adversely affected by the increasing degree of misspecification, see the results of Appendix B for further details on this. From the four backtests from the literature, the general CC and the ER and its standardized version exhibit satisfactory sizes whereas the Simple CC test is severely oversized, especially in small samples.

For a comparison of the power of the backtests, we evaluate their ability to reject the null hypothesis for risk models producing incorrect ES forecasts. We utilize the HS approach which forecasts the VaR and ES by using their empirical counterparts from previous trading days,  

$${\widehat{v}}_t={\widehat{Q}}_{\tau }\left(Y_{t-1},Y_{t-2},\dots ,Y_{t-w}\right)\hspace{1em}\hspace{1em}\text{and}\hspace{1em}\hspace{1em}{\widehat{e}}_t=\frac{1}{\sum _{i=1}^w 1_{\left\{Y_{t-i}\le {\widehat{v}}_t\right\}}}\sum _{i=1}^w{Y}_{t-i}·1_{\left\{Y_{t-i}\le {\widehat{v}}_t\right\}},$$

(2.7)

where ${\widehat{Q}}_{\tau }$ is the empirical τ-quantile and w is the length of a rolling window, that we set to 250, that is, one year of data. Since the standardized ER and the general CC backtests require forecasts of the volatility, we estimate this quantity with the sample standard deviation of the returns over the same rolling window. For a meaningful and fair comparison of the power of the backtests to reject the null hypothesis, we compare the size-adjusted power⁶ of the backtests (Lloyd, 2005). For this, the original critical values of the tests are modified such that the rejection frequencies of the true model equal the nominal test sizes. The size-adjusted power is then given by the rejection frequencies of the alternative models using these modified critical values.

The left panels in Figure 1 and Figure 2 contain the size-adjusted power of the backtests for all empirical sizes in the unit interval for a sample size of 1000 and for the different DGPs.⁷ The black line depicts the case of equal empirical size and power, which can be seen as a lower bound for any reasonable test: whenever the power is below this line, randomly guessing the test decision is more accurate than performing the test. For the three ESR backtests, we only report power for the tests relying on the misspecification robust covariance estimator as these versions of the tests exhibit superior size properties for all considered DGPs. We observe that throughout all six considered DGPs, the Strict and the Auxiliary ESR backtests clearly dominate the other tests in terms of power at almost all empirical sizes, including the most relevant region of test sizes between 1% and 10%. The Intercept ESR test is not as powerful, which is not unexpected as due to its unity restriction in the slope coefficient, it cannot account for misspecifications in the dynamics as precisely as the Strict and Auxiliary ESR tests.

Figure 1.

Size-adjusted power and PAUC plots against HS for a sample size of 1000 days. The number of Monte-Carlo repetitions is 10,000 and the probability level for the risk measures is $\tau =2.5\%$⁠. ESR refers to the backtests introduced in this article with (m) indicating the version which account for the additional covariance terms induced by the misspecified model. CC refers to the conditional calibration tests of Nolde and Ziegel (2017), and ER to the exceedance residuals tests of McNeil and Frey (2000). (a) EGARCH: Size-adjusted Power; (b) EGARCH: PAUC; (c) GAS-STD: Size-adjusted Power; (d) GAS-STD: PAUC; (e) GAS-SSTD: Size-adjusted Power; (f) GAS-SSTD: PAUC.

Figure 2.

Size-adjusted power and PAUC plots against HS for a sample size of 1000 days. The number of Monte-Carlo repetitions is 10,000 and the probability level for the risk measures is $\tau =2.5\%$⁠. ESR refers to the backtests introduced in this article with (m) indicating the version which account for the additional covariance terms induced by the misspecified model. CC refers to the conditional calibration tests of Nolde and Ziegel (2017), and ER to the exceedance residuals tests of McNeil and Frey (2000). (a) AR-GARCH ϕ = 0: Size-adjusted Power; (b) AR-GARCH ϕ = 0: PAUC; (c) AR-GARCH ϕ = 0.1: Size-adjusted Power; (d) AR-GARCH ϕ = 0.1: PAUC; (e) AR-GARCH ϕ = 0.5: Size-adjusted Power; (f) AR-GARCH ϕ = 0.5: PAUC.

In order to present results for all considered sample sizes in condensed form for the relevant area of empirical sizes between 1% and 10%, we summarize the size-adjusted power by the partial area under the curve (PAUC), as proposed by Lloyd (2005). For that, we numerically compute the area under each power curve for the empirical sizes between 1% and 10%, which can be interpreted as the test power averaged over the different test sizes. In the right-hand panels of Figure 1 and Figure 2, we present the PAUC for all backtests, DGPs, and sample sizes. As expected, the average power increases with the sample size, so that using more information leads to more reliable decisions about the quality of a forecast. We find that for all considered sample sizes, the Strict and Auxiliary ESR backtests dominate the other testing approaches. The almost identical performance of the Strict and the Auxiliary ESR tests throughout all simulation designs in Figure 1 and Figure 2 emphasizes that the misspecification introduced by the Strict ESR test seems to be unproblematic for realistic financial data.

2.2 Continuous Model Misspecification

In the second simulation study, we use a GARCH(1,1) model with standardized Student-t distributed innovations,  

$$\begin{array}{c}{Y}_t={\sigma }_t{z}_t,\hspace{1em} \text{where} \hspace{1em}{z}_t\sim t_{\nu },\hspace{1em} \text{and} \hfill \\ {\sigma }_t^2={\eta }_0+{\eta }_1{Y}_{t-1}^2+{\eta }_2{\sigma }_{t-1}^2,\hfill \end{array}$$

(2.8)

with the parameter values ${\eta }_0=0.01,\hspace{0.17em}{\eta }_1=0.1,\hspace{0.17em}{\eta }_2=0.85$⁠, and ν = 5 for the true model. For the analysis of the backtests, we simulate 10,000 times from this model with a fixed sample size of 2500 observations and consider the probability level $\tau =2.5\%$ for the VaR and the ES. Table 2 presents the empirical sizes of the backtests for a nominal size of 5% for both, the two- and one-sided hypotheses. As in the first simulation study, we find that most of the backtests are reasonably sized with rejection frequencies close to the nominal value.

For a detailed analysis of the power of the backtests, we continuously misspecify the true model according to the following five designs:

• (a) We misspecify how the conditional variance reacts to the squared returns by varying the ARCH parameter η₁. We choose ${\tilde{\eta }}_1$ between 0.03 and 0.2 and let ${\tilde{\eta }}_2=0.95-{\tilde{\eta }}_1$⁠, such that the persistence of the GARCH process remains constant. When ${\tilde{\eta }}_1<{\eta }_1$⁠, there is too little variation in the ES forecasts due to the reduced response to shocks and the GARCH process approaches a constant volatility model.

• (b) We alter the unconditional variance of the GARCH process $\mathbb{E}[{\sigma }_t^2]={\eta }_0/(1-{\eta }_1-{\eta }_2)$ between 0.5 and 0.01 by varying the parameter η₀ while holding η₁ and η₂ constant. Since the conditional variance is a weighted combination of the unconditional variance, the past squared returns, and the past conditional variance, this change implies that the ES forecasts are too conservative when the unconditional variance is larger than its true value, and vice versa.

• (c) We vary the persistence of shocks between 0.9 and 0.999 by setting ${\tilde{\eta }}_1=d·{\eta }_1$ and ${\tilde{\eta }}_2=d·{\eta }_2$ for a varying constant d > 0 and by setting ${\tilde{\eta }}_0=\mathbb{E}[{\sigma }_t^2](1-{\tilde{\eta }}_1-{\tilde{\eta }}_2)$ in order to stabilize the unconditional variance. A higher persistence causes a stronger and longer reaction to shocks.

• (d) We vary the degrees of freedom of the underlying Student-t distribution between 3 and $\infty$⁠. Since the conditional variance is unaffected, this modification implies a relative horizontal shift of the ES forecasts.

• (e) We misspecify the probability level $\tilde{\tau }$ of the ES forecasts between 0.5% and 5%. This represents the scenario that a forecaster submits (accidentally or on purpose) predictions for some level $\tilde{\tau }\ne \tau$⁠. Similar to changing the degrees of freedom, this modification implies a relative horizontal shift of the ES forecasts.

As an illustrative example of these misspecifications, Figures S.1a to S.1e in the Supplementary Appendix depict 250 realizations of the returns of the true DGP in Equation (2.8), together with the corresponding ES forecasts of the true model (black dashed line) and of two exemplary models following the parameter misspecifications described in the points (a) to (e) above.

We present the size-adjusted rejection rates plotted against the respective misspecified parameters for these five designs in Figure 3a–e. The true model is indicated by the gray vertical line and, induced by the results of Figure S.1 in the Supplementary Appendix; the X-axis is oriented such that too risky (too small in absolute value) ES forecasts are on the right side of the true model.⁸ Even though there is no backtest that dominates the others throughout all considered designs, several conclusions can be drawn from this figure.

Figure 3.

Size-adjusted rejection rates for various types of misspecification. The gray vertical line depicts the true model. The number of Monte-Carlo repetitions is 10,000 and the probability level for the risk measures is $\tau =2.5\%$⁠. ESR refers to the backtests introduced in this article with (m) indicating the version which account for the additional covariance terms induced by the misspecified model. CC refers to the conditional calibration tests of Nolde and Ziegel (2017), and ER to the exceedance residuals tests of McNeil and Frey (2000). (a) Changing the reaction to the squared returns; (b) Changing the unconditional variance; (c) Changing the persistence; (d) Changing the degrees of freedom; (e) Changing the probability level.

1. Overall, the Strict and Auxiliary ESR tests perform almost indistinguishably and in four out of the five considered designs, their performance is superior compared to the general CC and both ER backtesting approaches (Figure 3a–c and 3e). The ESR backtests outperform the competitors especially when we misspecify the volatility dynamics of the underlying GARCH process (Figure 3a–c). This shows that, in contrast to the existing approaches, our ESR backtests can be used to detect misspecifications in the dynamics used to construct the ES forecasts which go beyond level shifts.

2. The two ER tests (and the general CC test that is constructed to be similar to the ER backtest) can hardly discriminate between forecasts for the VaR and ES issued through misspecified volatility processes (Figure 3a–c) and through misspecified probability levels $\tilde{\tau }\ne \tau$ (Figure 3e). This confirms the theoretical results discussed in Section S.1.2.1 in the Supplementary Appendix that these backtests only reject misspecifications which affect the relation (distance) between the VaR and ES forecasts. In contrast, these backtests perform well in the case of misspecified tails of the residual distribution, which particularly affects the relative distance between the VaR and ES forecasts (Figure 3d). If these backtests would be used by the regulatory authorities, banks could submit joint VaR and ES forecasts for some level $\tilde{\tau }>\tau$ or some (too small) volatility process in order to minimize their capital requirements without facing the risk of being detected by these backtests. In comparison, our Intercept ESR backtest which is similar to the ER backtests by construction is clearly able to identify these misspecified probability levels.

3. Throughout all five misspecifications, the simple CC backtest also exhibits good power properties, similar to our proposed backtests. However, our three ESR backtests exhibit much better size properties (see Tables 1 and 2) and in contrast to the simple CC test, they do not fail to reject the HS forecasts in the first simulation study (see Figure 1).

4. The Intercept ESR test performs well for misspecifications in the residual distribution, while it exhibits lower power against misspecifications in the dynamics of the model when we alter the ARCH parameter and the persistence of the process. This confirms the theoretical considerations that the Strict and Auxiliary test have a greater ability to reject these misspecifications.

Together with the results from the first simulation study, these findings demonstrate that our proposed ESR backtests are a powerful choice for backtesting ES forecasts. They are reasonably sized and exhibit good power properties against a variety of misspecifications. Notably, in contrast to the existing backtests, there is no single type of misspecification where our ESR tests are unable to discriminate between forecasts of the true and the misspecified models.

2.3 Testing One-Sided Hypotheses

For the regulatory authorities, testing against a one-sided alternative might be more meaningful than the two-sided versions of the tests we consider in the previous sections. Holding more money than stipulated bv the Basel Accords is no concern for regulators as it is only important that banks keep enough monetary reserves to cover the risks from their market activities. In the following, we assess the performance of the Intercept ESR backtest and the one-sided versions of the four competitor backtests in rejecting the null hypothesis that the issued ES forecasts are at least as conservative (not smaller in absolute value) as the true ES, that is, that the associated market risk is not underestimated.

In Figure 4a–e, we present the size-adjusted rejection rates for the one-sided versions of the considered backtests and for the five continuous parameter misspecifications described in the points (a)–(e) from the previous section. The structure of these figures is analog to the two-sided case where the X-axis is oriented such that too risky ES forecasts are on the right side of the true model (vertical gray line). As it can be seen in Figures S.1a to S.1e in the Supplementary Appendix, the five modifications of the true model exhibit clear patterns when they issue too risky, respectively too conservative forecasts for the true ES, where this finding holds strictly for the cases (b), (d), and (e) and on average for the cases (a) and (c). Thus, the one-sided backtests should only reject the null hypothesis for ES forecasts that issue too risky (too small in absolute value) forecasts, that is, which are on the right side of the true model in Figure 4a–e.

Figure 4.

Size-adjusted rejection rates for various types of misspecification with a one-sided hypothesis. The gray vertical line depicts the true model. The number of Monte-Carlo repetitions is 10,000 and the probability level for the risk measures is $\tau =2.5\%$⁠. ESR refers to the backtests introduced in this article with (m) indicating the version which account for the additional covariance terms induced by the misspecified model. CC refers to the conditional calibration tests of Nolde and Ziegel (2017), and ER to the exceedance residuals tests of McNeil and Frey (2000). (a) Changing the reaction to the squared returns; (b) Changing the unconditional variance; (c) Changing the persistence; (d) Changing the degrees of freedom; (e) Changing the probability level.

We find that our Intercept ESR backtest is reasonably sized (compare Table 2) and dominates the ER and the CC tests in terms of their power in three out of the five misspecification designs. Only when altering the degrees of freedom, the ER tests are more powerful than the Intercept ESR test. When changing the persistence of the process, the Intercept ESR test performs overall comparably to its competitors throughout the different degrees of misspecification. Surprisingly, we see that in four out of the five cases, the one-sided CC tests (both, the simple and the general version) also reject too conservative ES forecasts, even though these should not be rejected by the specifications of the one-sided tests. Furthermore, as for the two-sided tests, both ER backtests fail to detect misspecifications of the underlying volatility process and of the underlying probability level. Summarizing these results, the proposed Intercept ESR backtest is a powerful backtest with good size properties for testing one-sided hypotheses which dominates the existing one-sided (joint VaR and ES) backtesting techniques in the literature.

3 Empirical Application

In the empirical application, we apply our backtests to compare ES forecasts along three dimensions: the complexity of the risk model, the length of the estimation window, and the model refitting frequency. From a practitioners point of view, it would be desirable to have a parsimonious model that can be estimated with few observations and is valid over a long period of time, for reasons of low engineering effort, data storage, and human and computational effort for updating the model. To assess whether such a setup is reasonable, and if not, which dimensions are crucial for a good performance, we compare rejection rates of ES forecasts using our backtests.

For this application, we use daily log returns of the 200 most highly capitalized stocks of the S&P 500 index (as of September 1, 2019), with a sufficiently long history of stock prices. We consider four different risk models: the standard GARCH(1,1) of Bollerslev (1986) and the GJR-GARCH(1,1) model of Glosten, Jagannathan, and Runkle (1993), both coupled with Gaussian and Student-t distributed innovations. For all four models and 200 stocks, we compare the same evaluation horizon, the period from January 2010 to August 2019 with a total of 2432 daily observations. We furthermore consider five different lengths of the rolling estimation window ranging from one year (250 trading days) up to eight years (2000 trading days) and refitting horizons of 5, 21, 62, 125, and 250 days, corresponding to weekly, monthly, quarterly, bi-yearly, and yearly updating of the models.

Table 3 presents the rejection rates of the one-sided Intercept ESR backtest with a nominal size of 5% for the 200 stocks under investigation, for the four GARCH specifications, the five estimation window sizes, and the five refitting frequencies. We choose to use the one-sided Intercept ESR test as this is the only one-sided and strict ES backtest in the literature. Given the currently implemented traffic light system of the Basel Committee, this one-sided test might be the one with the highest practical relevance for backtesting the ES.

Table 3.

Results of the empirical application

Rolling window	Refitting frequency					Refitting frequency
	5	21	62	125	250	5	21	62	125	250
	GARCH-N					GJR-GARCH-N
250	0.96	0.96	0.95	0.95	0.92	0.96	0.96	0.95	0.92	0.90
500	0.94	0.94	0.93	0.91	0.87	0.93	0.93	0.94	0.94	0.91
1000	0.86	0.88	0.87	0.83	0.83	0.93	0.93	0.91	0.89	0.91
1500	0.85	0.88	0.86	0.87	0.87	0.93	0.93	0.93	0.93	0.91
2000	0.81	0.82	0.83	0.82	0.80	0.92	0.92	0.91	0.92	0.91
	GARCH-t					GJR-GARCH-t
250	0.00	0.01	0.04	0.06	0.12	0.00	0.00	0.01	0.02	0.04
500	0.00	0.00	0.01	0.01	0.02	0.00	0.00	0.01	0.01	0.01
1000	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.01
1500	0.00	0.01	0.00	0.01	0.01	0.00	0.01	0.00	0.00	0.01
2000	0.00	0.00	0.00	0.01	0.01	0.00	0.01	0.00	0.01	0.01

Notes: This table shows the rejection rates of the one-sided ESR backtest for ES forecasts stemming from the two GARCH-type models with Student’s t and Gaussian residuals, different rolling window sizes, and model refitting lengths (in days). The rejection frequencies are averaged over the analyzed 200 most capitalized stocks of the S&P 500 index. The out-of-sample window covers the time from January 2010 to August 2019 resulting in a sample size of 2432 days.

The results show that both, the GARCH-N and GJR-GARCH-N are rejected for almost all the stocks (in more than 80% of the cases) uniformly over the different estimation sample sizes and refitting frequencies. Independent of the sample length and refitting frequency, this supports the well-known finding that Gaussian residuals generally fail to capture the riskiness of financial assets, especially in the tails of the distribution. In contrast, for the two GARCH specifications with Student-t distributed innovations, the rejection frequencies are considerably lower and for almost all choices of the refitting frequency and the estimation window length, they are below the nominal significance level of 5%. Furthermore, refitting the models more frequently tends to slightly decrease the rejection frequency for the models with Student-t distributed innovations, however, it tends to increase the rejection frequency for the Gaussian models. Overall, this implies that the refitting frequency is not a key factor in the model performance. Increasing the size of the estimation window tends to decrease the rejection frequency, whereas the results stabilize for lengths above 1000 days. Interestingly, employing the GJR-GARCH model, which accounts for a potential leverage effect in the volatility process, does not perform better than the standard GARCH model.

Overall, the results of this application which are diversified over 200 individual stocks, imply that using a fat-tailed residual distribution and an estimation window above 1000 days (roughly four years) suffices to obtain rejection rates uniformly below 1%.

4 Conclusion

With the upcoming implementation of the third Basel Accords, risk managers and regulators will shift attention to the risk measure ES for the forecasting and evaluation of financial risks. In this article, we introduce regression based ESR backtests for ES forecasts, which extend the classical Mincer and Zarnowitz (1969) test to ES-specific versions. As estimation of regression parameters for the ES stand-alone is infeasible, our tests build on a recently developed joint VaR and ES regression, which allows for different specifications of our tests, titled the Auxiliary, Strict, and Intercept ESR backtests. As these tests are potentially subject to model misspecification, we extend the asymptotic theory for the joint VaR and ES regression model to possibly misspecified models and verify the tests’ performance in finite samples through an extensive simulation study. We apply our tests to 200 stocks from the S&P 500 index in order to analyze the performance of ES forecasts stemming from the GARCH model family. We find that using fat-tailed (Student’s t) residual distributions and more than four years of data yield satisfactory ES forecasts.

A unique and essential feature of the Strict and Intercept ESR backtests is that they solely require forecasts for the ES and are consequently the first backtests for the ES stand-alone. In contrast, a common drawback of the existing backtests in the literature is that they need forecasts of further input variables, such as the VaR, the volatility, the tail distribution, or even the whole return distribution. Using more information than the ES forecasts is problematic for two reasons. First, these tests are not applicable for the regulatory authorities, who receive forecasts of the ES, but not of the additional information required by these tests. Second, rejecting the null hypothesis does not necessarily imply that the ES forecasts are incorrect as the rejection can be a result of a false prediction of any of the input parameters.

This article contributes to the ongoing discussion about which risk measure is the best in practice in the following way. As the VaR is criticized for not being subadditive and for not capturing tail risks beyond itself, the recent literature proposes both, the ES and expectiles as alternative risk measures. Expectiles are suggested as they are coherent, elicitable, and are able to capture extreme risks beyond the VaR and thus, they simultaneously overcome the drawbacks of the VaR and the ES (Bellini et al., 2014; Ziegel, 2016). Unfortunately, as opposed to the VaR and ES, they lack a visual and intuitive interpretation (Emmer, Kratz, and Tasche, 2015). In contrast, the ES is mainly criticized for its theoretical deficiencies of being not elicitable and not (only with difficulties) backtestable. However, starting with the joint elicitability result of VaR and ES of Fissler and Ziegel (2016), there is a growing body of literature using this result for a regression procedure (Dimitriadis and Bayer, 2019; Patton, Ziegel, and Chen, 2019; Barendse, 2020) and for relative forecast comparison (Fissler, Ziegel, and Gneiting, 2016; Nolde and Ziegel, 2017), which is extended by this article by introducing the ESR backtests, which are the first sensible backtests for the ES stand-alone. This shows that, even though technically more demanding, the ES can be modeled, evaluated, and backtested in the same way as quantiles and expectiles. Combining this with its ability to capture extreme tail risks and its intuitive visual interpretation, the ES is an appropriate candidate for being the standard risk measure in practice.

Supplementary Data

Supplementary data are available at Journal of Financial Econometrics online.

Acknowledgement

We thank the editor Andrew Patton, an anonymous associate editor, and two referees for very helpful comments. We further thank Tobias Fissler, Lyudmila Grigoryeva, Roxana Halbleib, Phillip Heiler, Ekaterina Kazak, Winfried Pohlmeier, James Taylor, and Johanna Ziegel for suggestions which inspired some results of this paper. Financial support by the Heidelberg Academy of Sciences and Humanities (HAW) within the project “Analyzing, Measuring and Forecasting Financial Risks by means of High-Frequency Data”, the Klaus Tschira Foundation, the University of Hohenheim, and by the German Research Foundation (DFG) within the research group “Robust Risk Measures in Real Time Settings” is gratefully acknowledged. The authors acknowledge support by the state of Baden-Württemberg through bwHPC. The majority of the work on this paper was conducted while both authors were at the University of Konstanz.

Footnotes

1

See Yamai and Yoshiba (2002), Kerkhof and Melenberg (2004), Carver (2013), Acerbi and Szekely (2014), Emmer, Kratz, and Tasche (2015), Ziegel (2016), Fissler, Ziegel, and Gneiting (2016), Nolde and Ziegel (2017) for the ongoing discussion on backtestability of the ES.

2

In particular, several tests require the whole or tail distribution of the returns or equivalently the cumulative violation process (Kerkhof and Melenberg, 2004; Wong, 2008; Graham and Pál, 2014; Acerbi and Szekely, 2014; Du and Escanciano, 2017; Löser, Wied, and Ziggel, 2018; Costanzino and Curran, 2018), multiple quantiles at different levels (Emmer, Kratz, and Tasche, 2015; Costanzino and Curran, 2015; Kratz, Lok, and McNeil, 2018; Couperier and Leymarie, 2019), the VaR and the volatility (McNeil and Frey, 2000; Nolde and Ziegel, 2017; Righi and Ceretta, 2013, 2015), or the VaR (McNeil and Frey, 2000; Nolde and Ziegel, 2017) in addition to the ES forecasts. See Section S.1.2 in the Supplementary Appendix for an overview over the existing backtesting approaches.

3

For recent overviews on VaR and ES forecasting approaches, see Komunjer (2004) and Nadarajah, Zhang, and Chan (2014).

4

Given that the ES forecasts are correctly specified, that is, ${\widehat{e}}_t={\text{ES}}_{\tau }\left(Y_t|{\mathcal{F}}_{t-1}\right)$⁠, the correct specification condition in Equation (1.5) is equivalent to ${\gamma }_1=(1-{\gamma }_2){\widehat{e}}_t$⁠. This results in the remark of Holden and Peel (1990), who claim that the null hypothesis, given in Equation (1.6) is only a sufficient, but not a necessary condition for correctly specified forecasts as ${\gamma }_1=(1-{\gamma }_2){\widehat{e}}_t$ is the required necessary condition. However, this more general condition implies that the forecasts ${\widehat{e}}_t$ are constant for all $t=1,\dots ,T$⁠, which is highly unrealistic given the dynamic nature of financial time series. Consequently, we employ the hypotheses given in Equation (1.6) for our backtesting procedure.

5

One could interpret the higher capital requirements as a punishment for too conservative risk forecasts.

6

A comparison of the raw power, that is, the raw rejection rate of the null hypotheses, could be misleading due to the differences in the empirical sizes of the backtests. In particular, an oversized test would exhibit unrealistically large rejection rates.

7

These plots are known as the receiver operating characteristic curves and origin from the psychometrics literature (Lloyd, 2005). They are an effective presentation method for general binary classification tasks such as hypothesis testing as they show the size-adjusted power simultaneously for all significance levels.

8

Notice that this inequality of the forecast magnitude only holds on average in the cases of Figure 3a and c whereas it holds strictly for Figure 3b, d and e.

9

Notice that we do not have a double index and thus we suppress the n in the notation of White (1994). Furthermore, we apply the definition by using the identify function for $a_t^o$⁠.

10

The mean-value theorem cannot be generalized in a straight-forward fashion to vector-valued functions. Thus, we have to consider the mean value expansion in each component separately which gives this more complicated expression.

11

We use this as a proxy for the approximation of $T_{\mathrm{S}-\text{ESR}}={\tilde{T}}_{\mathrm{S}-\text{ESR}}((0,1))$ by ${\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{\ast })$ as ${\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{\ast })$ is unfortunately not observable.

Appendix A: Proofs

Proof of Theorem 1.3: We check that the necessary conditions (i)–(iv) of the basic consistency theorem, given in Theorem 2.1 in Newey and McFadden (1994), p. 2121 hold, where we consider the objective functions $Q_T(\theta )$ and $Q_T^0(\theta )$ as defined in Equations (1.17) and (1.19). First, notice that condition (ii) holds by imposing condition (A2). The unique identification condition (i) holds by assumption (A3). Next, we verify the uniform convergence condition (iv) by applying the uniform weak law of large numbers given in Theorem A.2.5. in White (1994). For that, we have to show that obey a weak law of large numbers.

• (A). the map $\theta \mapsto \rho (Y_t,X_t,\theta )$ is Lipschitz-L₁ on Θ, see Definition A.2.3 in White (1994),⁹

• (B). For all ${\theta }^o\in \Theta$⁠, there exists ${\delta }^o>0$⁠, such that for all $\delta ,0<\delta \le {\delta }^o$⁠, the sequences  

  $${\overline{\rho }}_t({\theta }^o,\delta ):=\underset{\theta \in \Theta }{\sup }\left\{\rho (Y_t,X_t,\theta )|||\theta -{\theta }^o||<\delta \right\}\hspace{1em}\hspace{1em}\text{and}$$

  (A.1)

   

  $${{\displaystyle \underset{–}{\rho }}}_t({\theta }^o,\delta ):=\underset{\theta \in \Theta }{\inf }\left\{\rho (Y_t,X_t,\theta )|||\theta -{\theta }^o||<\delta \right\}$$

  (A.2)

Condition (A) follows directly from Lemma S.1.1 and we turn to condition (B). As the process $\{Y_t,V_t,W_t\}$ is strong mixing of size $-r/(r-2)$ for some r > 2 by condition (A6), the processes V_t and W_t are strong mixing of the same size by Theorem 3.49 in White (2001), p. 50. As the functions $\rho (Y_t,X_t,\theta )$ and the supremum/infimum functions are ${\mathcal{F}}_t$-measureable for all $t\in \mathbb{N}$⁠, we can conclude that the sequences ${\overline{\rho }}_t({\theta }^o,\delta )$ and ${{\displaystyle \underset{–}{\rho }}}_t({\theta }^o,\delta )$ are also strong mixing of the same size by applying the same theorem.

Furthermore, for $\tilde{r}>1$ and for some $\delta >0$ sufficiently small enough, $r\ge \tilde{r}+\delta$ and thus $\mathbb{E}\left[|{\overline{\rho }}_t({\theta }^o,\delta ){|}^{\tilde{r}+\delta }\right]\le \underset{1\le t\le T}{\sup }\mathbb{E}\left[\underset{\theta \in \Theta }{\sup }{|\rho (Y_t,X_t,\theta )|}^r\right]$ for all $t,1\le t\le T,T\ge 1$⁠. As Θ is compact, there exists some c > 0 such that $\underset{\theta \in \Theta }{\sup }||\theta ||\le c$ and thus, for all $t=1,\dots ,T$⁠, it holds that  

$$\mathbb{E}\left[\underset{\theta \in \Theta }{\sup }{|\rho (Y_t,X_t,\theta )|}^r\right]\le 4^{r-1}\left\{1+\left(\frac{c}{K}\left(1+\frac{1}{\tau }\right)\right)\mathbb{E}||V_t|{|}^r+\frac{1}{\tau K}\mathbb{E}||Y_t|{|}^r+\underset{\theta \in \Theta }{\sup }\mathbb{E}||\hspace{0.17em}\text{log}(W_t^{\top }\gamma )|{|}^r\right\},$$

(A.3)

which is bounded by condition (A8) and as $\text{log}(z)\le z$ for z large enough. The same inequality holds for $|{{\displaystyle \underset{-}{\rho }}}_t({\theta }^o,\delta )|$⁠. Thus, we can apply the weak law of large numbers for strong mixing sequences in Corollary 3.48 in White (2001), p. 49 in order to conclude that for all ${\theta }^o\in \Theta$ such that $||{\theta }^o-\theta ||\le \delta$⁠, it holds that $\frac{1}{T}\sum _{t=1}^T({\overline{\rho }}_t({\theta }^o,\delta )-\mathbb{E}\left[{\overline{\rho }}_t({\theta }^o,\delta )\right]){\displaystyle \stackrel{\mathbb{P}}{\to }}0$ and $\frac{1}{T}\sum _{t=1}^T({{\displaystyle \underset{-}{\rho }}}_t({\theta }^o,\delta )-\mathbb{E}\left[{{\displaystyle \underset{-}{\rho }}}_t({\theta }^o,\delta )\right]){\displaystyle \stackrel{\mathbb{P}}{\to }}0$⁠, which shows condition (B). Consequently, the uniform convergence condition (iv) holds by applying the uniform weak law of large numbers given in Theorem A.2.5. in White (1994).

As we have shown that the map $\theta \mapsto \rho (Y_t,X_t,\theta )$ is Lipschitz-L₁ in Lemma S.1.1, the map $\theta \mapsto Q_T^0=\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left[\rho (Y_t,X_t,\theta )\right]$ is also continuous which shows condition (iii). Thus, we can apply Theorem 2.1. of Newey and McFadden (1994) which concludes the proof of this theorem. □

Proof of Theorem 1.4: Let  

$$\psi (Y_t,X_t,\theta )=\left(\begin{array}{c}-\frac{V_t}{\tau W_t^{\top }\gamma }(1_{\{Y_t\le V_t^{\top }\beta \}}-\tau )\\ \frac{W_t}{{(W_t^{\top }\gamma )}^2}\left(W_t^{\top }\gamma -V_t^{\top }\beta +\frac{1}{\tau }(V_t^{\top }\beta -Y_t)1_{\{Y_t\le V_t^{\top }\beta \}}\right)\end{array}\right),$$

(A.4)

which is almost surely the derivative of $\rho (Y_t,X_t,\theta )$ with respect to θ. We further define ${\Psi }_T(\theta )=\frac{1}{T}\sum _{t=1}^T\psi (Y_t,X_t,\theta )$ and ${\Psi }_T^0(\theta )=\mathbb{E}[{\Psi }_T(\theta )]$⁠. From the proof of Lemma S.1.2, we get the mean value expansion (for ${\widehat{\theta }}_T$ close to ${\theta }_T^{\ast }$⁠),  

$${\Psi }_T^0({\widehat{\theta }}_T)-{\Psi }_T^0({\theta }_T^{\ast })={\Delta }_T({\tilde{\theta }}_1,{\tilde{\theta }}_2)({\widehat{\theta }}_T-{\theta }_T^{\ast }),$$

(A.5)

for some values ${\tilde{\theta }}_1$ and ${\tilde{\theta }}_2$ somewhere on the line between ${\widehat{\theta }}_T$ and ${\theta }_T^{\ast }$⁠, where the components of ${\Delta }_T({\tilde{\theta }}_1,{\tilde{\theta }}_2)$ are given in Equation (S.1.8) and Equation (S.1.9), and where ${\Psi }_T^0({\theta }_T^{\ast })=0$⁠.¹⁰

Furthermore, it holds that ${\Delta }_T({\theta }_T^{\ast },{\theta }_T^{\ast })={\Lambda }_T({\theta }_T^{\ast })$ and ${\Delta }_T({\tilde{\theta }}_1,{\tilde{\theta }}_2)$ is a continuous function in its arguments ${\tilde{\theta }}_1$ and ${\tilde{\theta }}_2$⁠. Using that ${\Lambda }_T({\theta }_T^{\ast })$ has Eigenvalues bounded away from zero (for T large enough), we also get that ${\Delta }_T({\tilde{\theta }}_1,{\tilde{\theta }}_2)$ is non-singular in a neighborhood around ${\theta }_T^{\ast }$ (for all arguments) for T large enough as the map which maps the matrix onto its Eigenvalues is continuous. As we further know that ${\widehat{\theta }}_T-{\theta }_T^{\ast }{\displaystyle \stackrel{\mathbb{P}}{\to }}0$ and $||{\tilde{\theta }}_j-{\theta }_T^{\ast }||\le ||{\widehat{\theta }}_T-{\theta }_T^{\ast }||$ for all j = 1, 2, we get from the continuous mapping theorem that  

$${\Delta }_T^{-1}({\tilde{\theta }}_1,{\tilde{\theta }}_2)-{\Lambda }_T^{-1}({\theta }_T^{\ast }){\displaystyle \stackrel{\mathbb{P}}{\to }}0.$$

(A.6)

In the following, we apply Lemma A.1 in Weiss (1991) (by verifying its assumptions), which extends the i.i.d. results of Huber (1967) to strong mixing sequences. Assumption (N1) of Lemma A.1 in Weiss (1991) is satisfied as every almost surely continuous stochastic process is separable in the sense of Doob (Gikhman and Skorokhod, 2004) and the functions $\psi (Y_t,X_t,\theta )$ are almost surely continuous for all $t\in \mathbb{N}$⁠. Assumption (N2) is satisfied as shown in the proof of Theorem 1.3. Assumption (N3)(i) is shown in Lemma S.1.2. The technical Assumptions (N3)(ii) and (N3)(iii) follow from Lemma 4 and Lemma 5 in the Supplementary Appendix of Patton, Ziegel, and Chen (2019). For this, notice that the moment conditions in Assumption 2 (C) and (D) of Patton, Ziegel, and Chen (2019) are implied by the condition (A8) in Assumption 1.2 for the simplified case of linear models. Assumption (N4) follows from the moment conditions (A8) in Assumption 1.2 and Assumption (N5) from the strong mixing condition (A6). Furthermore, Lemma 2 in the Supplementary Appendix of Patton, Ziegel, and Chen (2019) implies that $\sqrt{T}{\Psi }_T({\widehat{\theta }}_T){\displaystyle \stackrel{\mathbb{P}}{\to }}0$⁠. Thus, we can apply Lemma A.1 in Weiss (1991) and get that  

$$\sqrt{T}{\Psi }_T^0({\widehat{\theta }}_T)-\sqrt{T}{\Psi }_T({\theta }_T^{\ast }){\displaystyle \stackrel{\mathbb{P}}{\to }}0.$$

(A.7)

Combining Equations (A.5), (A.6), and (A.7), we get that  

$$\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast })=-{\Delta }_T{({\tilde{\theta }}_1,{\tilde{\theta }}_2)}^{-1}\sqrt{T}{\Psi }_T^0({\widehat{\theta }}_T)$$

(A.8)

 

$$=-\left({\Lambda }_T^{-1}({\theta }_T^{\ast })+o_p(1)\right)·\left(\sqrt{T}{\Psi }_T({\theta }_T^{\ast })+o_p(1)\right)$$

(A.9)

 

$$=-{\Lambda }_T^{-1}({\theta }_T^{\ast })·\sqrt{T}{\Psi }_T({\theta }_T^{\ast })+o_p(1).$$

(A.10)

Furthermore,  

$${\Sigma }_T^{-1/2}({\theta }_T^{\ast })\sqrt{T}{\Psi }_T({\theta }_T^{\ast })={\Sigma }_T^{-1/2}({\theta }_T^{\ast })\sqrt{T}\left({\Psi }_T({\theta }_T^{\ast })-{\Psi }_T^0({\theta }_T^{\ast })\right){\displaystyle \stackrel{d}{\to }}\mathcal{N}(0,I_{2k}),$$

(A.11)

by Lemma S.1.3 and thus,  

$${\Sigma }_T^{-1/2}({\theta }_T^{\ast }){\Lambda }_T({\theta }_T^{\ast })\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast }){\displaystyle \stackrel{d}{\to }}\mathcal{N}(0,I_{2k}),$$

(A.12)

which concludes the proof of this theorem. □

Proof of Corollary 1.5: We first notice that  

$${\widehat{\Omega }}_T^{-1/2}\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast })={\Omega }_T^{-1/2}\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast })+({\widehat{\Omega }}_T^{-1/2}-{\Omega }_T^{-1/2})\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast }).$$

(A.13)

From Theorem 1.4, we get that ${\Omega }_T^{-1/2}\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast }){\displaystyle \stackrel{d}{\to }}\mathcal{N}(0,I_4)$⁠. Furthermore, as $({\widehat{\Omega }}_T^{-1/2}-{\Omega }_T^{-1/2})=o_P(1)$ it holds by Slutzky’s theorem, that $({\widehat{\Omega }}_T^{-1/2}-{\Omega }_T^{-1/2})\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast })=o_P(1)$ and consequently,  

$${\widehat{\Omega }}_T^{-1/2}\sqrt{T}({\widehat{\theta }}_T-{\theta }_T^{\ast }){\displaystyle \stackrel{d}{\to }}\mathcal{N}\left(0,I_4\right).$$

(A.14)

Thus,  

$$T_{\mathrm{A}-\text{ESR}}={\left({\widehat{\Omega }}_{T,\gamma }^{-1/2}\sqrt{T}({\widehat{\gamma }}_T-{\gamma }_T^{\ast })\right)}^{\top }\left({\widehat{\Omega }}_{T,\gamma }^{-1/2}\sqrt{T}({\widehat{\gamma }}_T-{\gamma }_T^{\ast })\right){\displaystyle \stackrel{d}{\to }}{\chi }_2^2,$$

(A.15)

 

$${\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{\ast })={\left({\widehat{\Omega }}_{T,\gamma }^{-1/2}\sqrt{T}({\widehat{\gamma }}_T-{\gamma }_T^{\ast })\right)}^{\top }\left({\widehat{\Omega }}_{T,\gamma }^{-1/2}\sqrt{T}({\widehat{\gamma }}_T-{\gamma }_T^{\ast })\right){\displaystyle \stackrel{d}{\to }}{\chi }_2^2,\hspace{1em} \text{and}$$

(A.16)

 

$${\tilde{T}}_{\mathrm{I}-\text{ESR}}({\gamma }_{1,T}^{\ast })={\left({\widehat{\Omega }}_{T,{\gamma }_1}^{-1/2}\sqrt{T}({\widehat{\gamma }}_{1,T}-{\gamma }_{1,T}^{\ast })\right)}^{\top }\left({\widehat{\Omega }}_{T,{\gamma }_1}^{-1/2}\sqrt{T}({\widehat{\gamma }}_{1,T}-{\gamma }_{1,T}^{\ast })\right){\displaystyle \stackrel{d}{\to }}{\chi }_1^2.$$

(A.17)

□

Proof of Corollary 1.6: In the following, we show the result for the Strict ESR test statistic, while equivalent results for the other two ESR tests follow from straight-forward simplifications of this proof. Given the alternative hypothesis  

$${\mathbb{H}}_A^{\mathrm{S}-\text{ESR}}:\hspace{0.17em}{\gamma }_T^{\ast }\cancel{=}{\gamma }_T^{{\mathbb{H}}_0},\hspace{1em}\forall T\ge T_A\hspace{1em} \text{for} \text{some} T_A\in \mathbb{N},$$

(A.18)

it holds that $||{\gamma }_T^{\ast }-{\gamma }_T^{{\mathbb{H}}_0}||\ge 2{\epsilon }_A$ for all $T\ge T_A$ and for some ${\epsilon }_A>0$⁠. Thus,  

$$||{\widehat{\gamma }}_T-{\gamma }_T^{{\mathbb{H}}_0}||=||{\widehat{\gamma }}_T-{\gamma }_T^{\ast }+{\gamma }_T^{\ast }-{\gamma }_T^{{\mathbb{H}}_0}||\ge |||{\gamma }_T^{\ast }-{\gamma }_T^{{\mathbb{H}}_0}||-||{\widehat{\gamma }}_T-{\gamma }_T^{\ast }|||\ge {\epsilon }_A>0,$$

(A.19)

with probability approaching one by the inverse triangle inequality and as $||{\widehat{\gamma }}_T-{\gamma }_T^{\ast }||{\displaystyle \stackrel{\mathbb{P}}{\to }}0$ and thus, $||{\widehat{\gamma }}_T-{\gamma }_T^{\ast }||\le {\epsilon }_A$ with probability approaching one as $T\to \infty$⁠. Consequently, for all $c\in \mathbb{R}$⁠, it holds that  

$$\mathbb{P}\left(||\sqrt{T}{\widehat{\Omega }}_{T,\gamma }^{-1/2}({\widehat{\gamma }}_T-{\gamma }_T^{{\mathbb{H}}_0})||\ge c\right)\to 1,$$

(A.20)

and thus,  

$$\mathbb{P}\left({\tilde{T}}_{\mathrm{S}-\text{ESR}}({\gamma }_T^{{\mathbb{H}}_0})\ge c\right)\to 1.$$

(A.21)

The proof for ${\tilde{T}}_{\mathrm{I}-\text{ESR}}$ follows along the lines and the one of $T_{\mathrm{A}-\text{ESR}}$ is a simplified version as we can consider the test parameters (0, 1) instead of the hypothetical pseudo-true parameters under the null ${\gamma }_T^{{\mathbb{H}}_0}$⁠. □

Appendix B: Approximation Accuracy of the Misspecified Parameters

In this section, we present a simulation study in order to analyze the accuracy of the approximations of the pseudo-true parameter ${\gamma }_T^{\ast }$ by the tested restriction ${\gamma }^0=(0,1)$ under the null. Subsequently, we analyze the approximation of the test statistic $T_{\mathrm{A}-\text{ESR}}$ by $T_{\mathrm{S}-\text{ESR}}$⁠.¹¹ For this, we generate 1000 Monte-Carlo replications of the AR-GARCH DGP, given in Equation (2.2) of length T = 1000 with varying AR parameter $\varphi \in \{0,0.1,0.5\}$⁠, and generate optimal VaR and ES forecasts ${\widehat{v}}_t$ and ${\widehat{e}}_t$⁠. As already specified in Equation (2.3), for this DGP, the degree of misspecification in the quantile model of the Strict and the Intercept ESR tests is analytically tractable through the magnitude of $\varphi$⁠, where $\varphi =0$ represents the case of a correct specification. In contrast, the case $\varphi =0.5$⁠, which is highly unrealistic for financial returns, can be seen as a worst-case scenario. For each simulated time series, we estimate the underlying regression equations of the Auxiliary and the Strict ESR tests, given in Equations (1.8) and (1.11) and denote the respective parameter estimates through the superscripts, ${\widehat{\gamma }}_T^{\mathrm{A}-\text{ESR}}$ and ${\widehat{\gamma }}_T^{\mathrm{S}-\text{ESR}}$⁠. Under the null hypothesis, the (average) difference of the estimated parameters ${\widehat{\gamma }}_T^{\mathrm{A}-\text{ESR}}$ and ${\widehat{\gamma }}_T^{\mathrm{S}-\text{ESR}}$ governs the (average) effect the misspecification has on the ES-specific parameters.

Figure 5 illustrates the results, where the different columns of the plots correspond to the different values of $\varphi$⁠. In each column, the first row illustrates the degree of misspecification through a plot of $c_t={\widehat{e}}_t/{\widehat{v}}_t$⁠. The subsequent rows illustrate the estimated densities of the Strict and Auxiliary ESR tests for the quantile-specific parameters ${\widehat{\beta }}_T$⁠, the ES-specific parameter ${\widehat{\gamma }}_T$ and the respective test statistics $T_{\mathrm{S}-\text{ESR}}$ and $T_{\mathrm{A}-\text{ESR}}$⁠.

Figure 5.

This figure illustrates the effect the misspecified regression equations of the Strict ESR test have on the respective (average) parameter estimates and associated test statistics. The three plot columns of the figure correspond to the different values of the AR-parameter $\varphi \in \{0,0.1,0.5\}$⁠, which governs the degree of misspecification. For each column, the first row illustrates the empirical degree of misspecification through a plot of $c_t={\widehat{e}}_t/{\widehat{v}}_t$⁠. The subsequent rows show the estimated densities of the Strict ESR test (solid black lines) and the Auxiliary ESR test (dashed green lines) for the quantile-specific parameters ${\widehat{\beta }}_T$⁠, the ES-specific parameter ${\widehat{\gamma }}_T$ and the respective test statistics $T_{\mathrm{S}-\text{ESR}}$ and $T_{\mathrm{A}-\text{ESR}}$⁠.

For the quantile-specific parameters ${\widehat{\beta }}_T$⁠, we can verify the expected misspecified behavior of the quantile regression equation through differing parameter estimates between the Strict and Auxiliary ESR tests. For $\varphi =0$⁠, c_t is constant and the quantile regression is correctly specified, however with a slope coefficient generally unequal to 1. For an increasing degree of misspecification, we expectedly find an increasing degree of misspecification in the estimates ${\widehat{\beta }}_T^{\mathrm{S}-\text{ESR}}$ compared to ${\widehat{\beta }}_T^{\mathrm{A}-\text{ESR}}$⁠. In contrast, this effect is almost negligible for the ES-specific parameter estimates ${\widehat{\gamma }}_T^{\mathrm{S}-\text{ESR}}$ compared to ${\widehat{\gamma }}_T^{\mathrm{S}-\text{ESR}}$⁠. This illustrates that the misspecification mainly affects the quantile parameters while leaving the ES-specific parameters almost unchanged, even for the worst-case scenario of $\varphi =0.5$⁠. The same approximation behavior can be observed for the associated test statistics reported in the last row of plots. This finding explains the almost identical behavior of the Auxiliary and the Strict ESR tests in the simulation exercises in Section 2.