Practitioners and Academics Duke It Out Over Solutions for Customer Retention
March 09, 2004
DURHAM, N.C. - The Teradata Center for Customer Relationship Management at Duke University (the center) has concluded its Churn Response Modeling Tournament. The objective of the tournament was to determine the most accurate approaches for predicting customer churn. The 2002-2003 tournament winner was Salford Systems whose team consisted of Scott Cardell, Phil Colla, Mikhail Golovnya, John Ries and Dan Steinberg. They submitted two models to win all four categories of the tournament.
Customer churn, when customers leave one company and sign up with another, is a hot topic for industries, such as telecommunications and financial services. As companies lose customers in a fiercely competitive market while also facing higher customer acquisition costs, profits decline.
The basic structure of the tournament was to provide a "calibration" database to all participants consisting of 171 predictor variables and data on whether the customer eventually churned. After estimating a churn prediction model on this database, the participants used their model to predict churn behavior for two holdout databases totaling over 150,000 customers. There were 44 entries to the tournament, submitted by both practitioners and academics.
"While various models exist for this key issue, there has not been a sharing nor a forum for a systematic examination of these models," observes Sanyin Siang, former Center managing director. "Through the Teradata Center serving as a global epicenter for customer relationship management (CRM), we convened these models in one place, matched them against each other and identified the best models in terms of key measures of predictive accuracy. By convening thought leadership and then disseminating the learning, we are able to elevate the value of that field as a whole."
"What's particularly exciting about this area is that both academics and practitioners are recognizing its importance and both are working on the problem of accurate churn prediction," says Scott Neslin of Dartmouth College. "It is no coincidence that roughly half our submissions were from practitioners and half from academics." He adds, "While businesses are already aware of the problem since they live with it every day, we believe the tournament has increased awareness within the academic community."
The Salford Systems team submitted two models of which one won across three categories and the second across one category. The first model was an ensemble of TreeNet (stochastic gradient boosting) models each developed on a different subset of the training data. The second submission was a single "out of the box" TreeNet model. The averaging included all models that excluded one of the variables. "I felt that a model using only variables that were relative to the date at which the data was gathered would be most robust when applied to future data. It was an easy choice as the selected model was at least arguably the best single model of all the models run based on our internal measures. Further more, in most data mining projects, the bulk of the effort is normally devoted to data preparation and feature extraction. In this tournament, we were fortunate to be able to focus more on tuning the model control parameters to get the best out of a rather new data mining technology," observed Dr. Cardell, Salford's director of research and development. (www.salford-systems.com)
The tournament and its results have culminated into a research paper reporting overall predictive performance and how it differed by methodological approach. It includes recommendations highlighting the ways to predict churn accurately as well as avenues for future research. There is also a data set on the tournament that is available on the center's web site for use by researchers. An evolving foundation for future research, this tournament is a milestone in advancing the understanding of key issues in CRM.
The Teradata Center for Customer Relationship Management at Duke University advances the field of Customer Relationship Management (CRM) through research and learning. This multi-million dollar global think tank and academic center is a creation of a partnership between The Fuqua School of Business and the Teradata Division of NCR. Through this dynamic partnership the center successfully leverages the intellectual resources of a leading academic institution and corporation to merge theory and practical business experience, thereby, making Fuqua the world leader in CRM research. The center's primary goal is to prioritize, facilitate and disseminate CRM related works and knowledge in order to provide a greater understanding and appreciation of CRM, in both the academic and professional realms. By bringing together the best academicians and practitioners, the center is able to create thought leadership in this vital marketing area. We believe that the center's work, findings and offerings influence the way corporations, students and academicians view business in the 21st century.
For more information about the Teradata Center for Customer Relationship Management, visit http://www.teradataduke.org.