Elicitability of a statistical functional means that it can be obtained as the minimizer of an expected loss function. Such a loss function can be used for forecast comparison or model selection, and allows for M-estimation and generalized regression. Prime examples of elicitable functionals are the mean or quantiles of a random variable. Expected Shortfall (ES), an important risk measure in banking and finance is not elicitable but becomes elicitable when considered jointly with a certain quantile, also called Value at Risk (VaR).
We present a characterization of the large class of suitable loss functions for the pair of ES and VaR, and discuss the difficulty in choosing a specific loss function for forecast comparison or estimation. As an alternative for forecast comparison, we introduce a procedure that is robust with respect to the choice of the loss function. We present graphical checks (Murphy diagrams) of whether one forecast method dominates another, and propose an associated hypothesis test.