AASP: Multivariable testing!

Frank Woodward from Lincoln Memorial University (located in Harrogate, TN) is presenting on the topic of mutivariable testing. They use QualPro as their vendor to support these efforts. The number one question to answer: How do you know what is working?

Like many universities, LMU was struggling with a low alumni participation rate,and was looking to convert "never givens".

LMU looked at a number of questions: What drives alumni and friends giving? What moves the needle for non-donors? What increases annual giving? How do we know if gift officers are focusing on optimal prospects? How do we connect with young alumni? What kinds of interactions are we able to track?

Frank is talking about the gap between the fundraising plan and dollars raised. What happens in the middle? He describes it as a "magic box": what happened in the magic box?

This certainly makes me think about the difficulties inherent in measuring the effectiveness of donor relations, a rabbit hole I've recently been exploring with Roberta O'Hara from Rutgers University. I have the feeling that multivariable testing may offer a great deal of value here.

Some traditional methods of analysis: top-down change (based on experience/expertise); incremental change and results comparison; alumni/donor surveys; and A/B testing. LMU applied all of these methods, but felt they weren't getting to the heart of their questions.

How can these methods account for multiple variables, both internal (e.g. annual fund ask strategy) and external (economic factors)? It's very difficult to determine the most important factors without some statistical analysis.

In 2009, LMU started working with QualPro, which was founded by an alumnus of LMU. I am now recalling reading a great article about this project in the Chronicle of Higher Education a few years ago! I am all the more stoked to be in this session because I remember being super-excited by tht article.

Most of QualPro's experience had been in manufacturing, but higher ed was a new arena for them. When they started the project, it was not clear that multivariable process analysis would be helpful in a higher ed setting.

Multivariable testing is used to test many ideas simultaneously. The concept is to identify 20 to 30 "practical, fast and cost-neutral" ideas and to test them. From there, use analysis to determine which factors help, hurt or are neutral. Adopt the helpful factors, stop doing what's harmful, and save money by reducing efforts that make no difference.

In 15,000 multivariate tests of 200,000+ improvement ideas, QualPro found that 53% of factors tested made no difference, 22% were actually harmful, and 25% were helpful.

The other great thing about multivariate testing is that complex interactions among factors can be accounted for, rather than assessing each factor individually.

LMU identified targets for testing (dollars raised -- overall and from non-donors; and gifts given -- overall and from non-donors). LMU conducted tests over a 12-week period of time.

LMU has tested many factors since 2009, including estate planning mailings, new donor welcome packets, alumnus magazine formatting, faculty/staff payroll mailings, and phonathon scripts. They are currently in their 11th round of multivariable testing.

In their first test, LMU selected 23 direct mail and communications factors to test. For example, they tested putting a message on the annual fund appeal envelope vs. no message (no message performed better!) They tested font size in letters. They tested annual fund letter date. For each factor, they identified the status quo and the change (e.g. Status Quo: one-page letter; Change: two-page letter).

One nice element of doing multivariable testing is that you don't need to worry about external factors (e.g. the economy) because all testing is done at the same time.

Using a variety of factors, LMU creates ""recipes" that combine a variety of factors. Each recipe has an equal number of random constituents assigned to it.

Example -- If each of three factors has two possible options (A or B), then that would yield 8 "recipes":

Recipe 1: AAA Recipe 2: AAB Recipe 3: ABA Recipe 4: ABB Recipe 5: BBB Recipe 6: BBA Recipe 7: BAA Recipe 8: BAB

Each constituent is tracked as a member of a specific recipe and recipient of a specific mailing or initiative. Then gift responses are tracked associated with these constituents.

QualPro then performs some statistical magic (and yes, I'll be researching this further!) From there, LMU is able to understand which factors were specifically helpful, as well as which factors work best in combination with one another.

LMU is currently testing major gift officer visit behaviors, like talking about campus memories. Fascinating! You can evaluate qualitative factors if you can code them well.

Thanks to Frank for a very interesting session and lots of food for thought. On my to do list: learn more about multivariate analytics.