We propose an original two-part, duration-severity approach to backtesting expected shortfall. From the daily probability integral transform of the return, we construct a sequence of durations counting the time elapsed between successive VaR violations, as well as a sequence of severities corresponding to the realized quantiles in case of a violation. Then the serial and mutual independence of the duration and severity sequences are tested using the theory of (bivariate) orthogonal polynomials. Our test statistic has a simple distribution, and includes as special cases unconditional coverage (UC) and conditional coverage (CC) backtests of both VaR and ES, allowing the risk manager to easily identify the mis-specified component(s) of the internal model in case of a rejection of the test. Our test can also be applied to other systemic risk measures, such as the marginal expected shortfall. Simulation experiments suggest that our test has good finite sample properties for realistic sample sizes. An empirical application to three major stock indexes show that our duration-severity backtesting approach can effectively identify the reasons why the validity of ES and VaR forecasts is rejected, when it occurs.