Jones Lang LaSalle
Power Curves and Accuracy
Jon Southard, SVP, Director of Debt Management and Valuation
jsouthard@tortowheatonresearch.com
While it sounds like a Curt Schilling pitch, in the context of debt analysis, a power curve is actually something quite different.
A power curve is an analytical tool that compares the predicted default (PD) probability from a credit risk model to the grouping of actual defaults on a relative basis, i.e., were those loans with higher predicted PDs more likely to have actually defaulted than those with lower predicted PDs? Under perfect prediction with no errors, actual defaults would be concentrated solely among those loans with the highest predicted probability of default. To the extent that loans with low predicted defaults did subsequently default, or that loans with high predicted defaults did not, a model moves away from this ideal.
The power curve captures the amount of either type of error by ranking the loans in a back test by PD and by graphing these against the actual percentage of defaults at progressively lower probabilities. A 45 degree diagonal line represents a random model with no predictive power. The extent to which the curve bulges upward and away from a straight diagonal—the power curve—demonstrates the value of the model over a random or unpredictable pattern of default. The power curve can thus be said to graphically illustrate the explanatory power of the underlying approach.
For example, if the 30% of the loans with the highest PDs (measured on the horizontal axis) account for 55% of the actual defaults (measured on the vertical axis), and 50% of the loans with the highest PDs (horizontal axis) account for 75% of the actual defaults (vertical axis), the curve will bulge significantly outward. Of course, in a model with no predictive power, generating essentially random results, 50% of the scores would have resulted in 50% of the defaults, so this example picked up an extra 25% of defaults through the mid-point of rank ordering.
In creating a power curve for our commercial mortgage default model, Commercial Mortgage Metrics, we used individual loan data from the CMBS universe. Since we are applying the CMM model to whole loans in addition to public loans, we have chosen to evaluate the entire four years from 2000 to 2004 as a single time period because of the lesser liquidity of real estate whole loans, as well as their general tendency to be held rather than traded by life companies and banks. In comparison, a corporate bond can usually be traded into a liquid market and the time period of focus is therefore shorter. The longer time period provides a more rigorous test of the model than is normally done to models of corporate bonds (which normally examine just one year after the prediction). Still, we feel it is appropriate, given the longer-term nature of the asset.
Because 2000 marks a comparatively early period in terms of the depth of data available on CMBS, the test uses only information about the DSCR and LTV at the start of the period, plus information about which property type and market in which the collateral is located, to make a projection of an expected default rate for the loan.
Ranking the results of this probability of default as of the year 2000, we trace which loans in the pool actually defaulted up until 2004 in order to create the power curve, shown here for 3,063 office loans. It bears mentioning that this test differs from some of the most popular commercial loan methodologies because there are no updates of values over the period and even DSCR information is, at best, only updated monthly (where those changes are most likely changes recorded to the loan and not to the income) and highly dependent on leases and other contractual arrangements. Thus, one of the items being tested is how much information is added by knowledge of the real estate market (the metropolitan area and property type provided by TWR) as opposed to the building (where little information was available for this period).
Office Power Curve
Given this context, we are pleased with the results in terms of a substantial value brought by adding information about the market to just the starting DSCR and LTV values. We can also summarize the results by calculating the area under the power curve. The more area covered indicates fewer errors, both of loans expected to default that didn't, as well as loans thought to be safe that did. For office loans the area under the curve covers 66.1% of the total area.
Torto Wheaton Research and Moody's Investors Service are pleased to combine with Trepp Analytics in applying this same methodology to the CMBS universe on a forward-looking basis. CMBS investors in Boston, New York, and Chicago are welcome to come to demonstrations of Commercial Mortgage Metrics on Trepp as the product is launched next week.
*Thanks to Sally Gordon of Moody's for her assistance on this column.
Go to www.tortowheatonresearch.com to register for a free subscription to About Real Estate.
© 2006 Torto Wheaton Research
Power Curves and Accuracy
Jon Southard, SVP, Director of Debt Management and Valuation
jsouthard@tortowheatonresearch.com
While it sounds like a Curt Schilling pitch, in the context of debt analysis, a power curve is actually something quite different.
A power curve is an analytical tool that compares the predicted default (PD) probability from a credit risk model to the grouping of actual defaults on a relative basis, i.e., were those loans with higher predicted PDs more likely to have actually defaulted than those with lower predicted PDs? Under perfect prediction with no errors, actual defaults would be concentrated solely among those loans with the highest predicted probability of default. To the extent that loans with low predicted defaults did subsequently default, or that loans with high predicted defaults did not, a model moves away from this ideal.
The power curve captures the amount of either type of error by ranking the loans in a back test by PD and by graphing these against the actual percentage of defaults at progressively lower probabilities. A 45 degree diagonal line represents a random model with no predictive power. The extent to which the curve bulges upward and away from a straight diagonal—the power curve—demonstrates the value of the model over a random or unpredictable pattern of default. The power curve can thus be said to graphically illustrate the explanatory power of the underlying approach.
For example, if the 30% of the loans with the highest PDs (measured on the horizontal axis) account for 55% of the actual defaults (measured on the vertical axis), and 50% of the loans with the highest PDs (horizontal axis) account for 75% of the actual defaults (vertical axis), the curve will bulge significantly outward. Of course, in a model with no predictive power, generating essentially random results, 50% of the scores would have resulted in 50% of the defaults, so this example picked up an extra 25% of defaults through the mid-point of rank ordering.
In creating a power curve for our commercial mortgage default model, Commercial Mortgage Metrics, we used individual loan data from the CMBS universe. Since we are applying the CMM model to whole loans in addition to public loans, we have chosen to evaluate the entire four years from 2000 to 2004 as a single time period because of the lesser liquidity of real estate whole loans, as well as their general tendency to be held rather than traded by life companies and banks. In comparison, a corporate bond can usually be traded into a liquid market and the time period of focus is therefore shorter. The longer time period provides a more rigorous test of the model than is normally done to models of corporate bonds (which normally examine just one year after the prediction). Still, we feel it is appropriate, given the longer-term nature of the asset.
Because 2000 marks a comparatively early period in terms of the depth of data available on CMBS, the test uses only information about the DSCR and LTV at the start of the period, plus information about which property type and market in which the collateral is located, to make a projection of an expected default rate for the loan.
Ranking the results of this probability of default as of the year 2000, we trace which loans in the pool actually defaulted up until 2004 in order to create the power curve, shown here for 3,063 office loans. It bears mentioning that this test differs from some of the most popular commercial loan methodologies because there are no updates of values over the period and even DSCR information is, at best, only updated monthly (where those changes are most likely changes recorded to the loan and not to the income) and highly dependent on leases and other contractual arrangements. Thus, one of the items being tested is how much information is added by knowledge of the real estate market (the metropolitan area and property type provided by TWR) as opposed to the building (where little information was available for this period).
Office Power Curve
Given this context, we are pleased with the results in terms of a substantial value brought by adding information about the market to just the starting DSCR and LTV values. We can also summarize the results by calculating the area under the power curve. The more area covered indicates fewer errors, both of loans expected to default that didn't, as well as loans thought to be safe that did. For office loans the area under the curve covers 66.1% of the total area.
Torto Wheaton Research and Moody's Investors Service are pleased to combine with Trepp Analytics in applying this same methodology to the CMBS universe on a forward-looking basis. CMBS investors in Boston, New York, and Chicago are welcome to come to demonstrations of Commercial Mortgage Metrics on Trepp as the product is launched next week.
*Thanks to Sally Gordon of Moody's for her assistance on this column.
Go to www.tortowheatonresearch.com to register for a free subscription to About Real Estate.
© 2006 Torto Wheaton Research
posted by Jones Lang LaSalle NJ News Blog at 9:47 AM
<< Home