Are We Missing Too many Alumni with Web Surveys? With the advent of the internet and its exponential growth over the last decade and a half, web surveys have gained a strong foothold in society in general, and in higher education advancement in particular. We re not experts on surveys, and certainly not on web surveys. However, let s assume you (or the vendor you use to do the survey) e-mail either a random sample of your alumni (or your entire universe of alumni) and invite them to go to a website and fill out a survey. If you do this, you will encounter the problem of poor response rate. If you re lucky, maybe 30% of the people you e-mailed will respond, even if you vigorously follow-up non-responders encouraging them to please fill the thing out. This is a problem. There will always be the lingering question of whether or not the nonresponders are fundamentally different from the responders with respect to what you re surveying them about. For example, will responders: Give you a far more positive view of their alma mater than the non-responders would have? Tell you they really like new programs the school is offering, programs the nonresponders may really dislike, or like a lot less than the responders? Offer suggestions for changes in how alumni should be approached -- changes that nonresponders would not offer or actively discourage? To test whether these kinds of questions are worth answering, you (or your vendor) could do some checking to see if your responders: Are older or younger than your non-responders (Looking at year of graduation for both groups would be a good way to do this.). Have a higher or lower median lifetime giving than your non-responders. Attend more or fewer events after they graduate than your non-responders. Are more or less likely than your non-responders to be members of a dues paying alumni association. It is our impression that most schools that conduct alumni web surveys don t do this sort of checking. In their reports they may discuss what their response rates are, but few offer an analysis of how the responders are different from the non-responders. Again, we re talking about impressions here, not carefully researched facts. But that s not our concern in this paper. Our concern here is that web surveys (done in schools where potential responders are contacted only by e-mail) are highly unlikely to be representative of the entire universe of alums -- even if the response rate for these surveys is always one hundred percent. Why? Because our evidence shows that alumni who have an e-mail address listed with their
schools are markedly different (in terms of two important variables) from alumni who do not have an e-mail address listed: Age and giving. To make our case, we ll offer some data from four higher education institutions spread out across North America; two are private, and two are public. Let s start with the distribution of e- mail addresses listed in each school by class year decile. You can see these data in Tables 1-4 and Figures 1-4. We ll go through Table 1 and Figure 1 (School A) in some detail to make sure we re being clear. Take a look at Table 1. You ll see that the alumni in School A have been divided up into ten roughly equal size groups where Decile 1 represents the oldest group and Decile 10 represents the youngest. The table shows a very large age range. The youngest alums in Decile 1 graduated in 1958. (Most of you reading this paper were not yet born by that year.) The alums in Decile 10 (unless some of them went back to school late in life) are all twenty-something s. Table 1: Count, Median Class Year, and Minimum and Maximum Class Years for All Alums Divided into Deciles for School A DECILE Count MEDIAN MINIMUM MAXIMUM 1 3772 1952 1926 1958 2 3790 1964 1959 1967 3 3918 1971 1968 1974 4 4085 1978 1975 1980 5 3680 1983 1981 1985 6 4662 1989 1986 1991 7 3297 1994 1992 1995 8 4100 1998 1996 2000 9 4231 2003 2001 2005 10 4326 2008 2006 2010 Now look at Figure 1. It shows the percentage of alums by class year decile who have an e-mail address listed in the school s database. Later on in the paper we ll discuss what we think are some of the implications of a chart like this. Here we just want to be sure you understand what the chart is conveying. For example, 43.0% of alums who graduated between 1926 and 1958 (Decile 1) have an e-mail listed in the school s database. How about Decile 9, alums who graduated between 2001 and 2005? If you came up with 86.5%, we ve been clear.
Fig. 1: Percentage of Alumni with an E- mail Address Listed In School A by 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 94.8% 80.6% 86.5% 74.5% 74.8% 60.6% 66.9% 73.6% 76.0% 43.0% Go ahead and browse through Tables 2-4 and Figures 2-4. After you ve done that, we ll tell you what we think is one of the implications of what you ve seen so far. Table 2: Count, Median Class Year, and Minimum and Maximum Class Years for All Alums Divided into Deciles for School B DECILE Count MEDIAN MINIMUM MAXIMUM 1 11860 1961 1920 1967 2 11200 1971 1968 1973 3 13839 1977 1974 1979 4 10820 1982 1980 1984 5 12821 1987 1985 1989 6 13927 1992 1990 1994 7 11561 1996 1995 1998 8 13447 2001 1999 2003 9 10458 2005 2004 2006 10 14836 2008 2007 2010
Fig. 2: Percentage of Alumni with an E- mail Address Listed In School B by 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 86.3% 65.9% 27.2% 16.3% 15.7% 15.2% 16.2% 14.9% 15.4% 17.3% Table 3: Count, Median Class Year, and Minimum and Maximum Class Years for All Alums Divided into Deciles for School C DECILE Count MEDIAN MINIMUM MAXIMUM 1 3253 1949 1925 1954 2 3028 1959 1955 1963 3 3570 1968 1964 1971 4 2903 1974 1972 1976 5 3534 1979 1977 1982 6 3052 1985 1983 1987 7 3152 1990 1988 1992 8 3378 1995 1993 1998 9 3244 2001 1999 2003 10 3862 2007 2004 2009
Fig. 3: Percentage of Alumni with an E- mail Address Listed In School C by 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 51.8% 56.9% 57.8% 58.6% 63.2% 64.3% 67.0% 58.7% 54.8% 32.2% Table 4: Count, Median Class Year, and Minimum and Maximum Class Years for All Alums Divided into Deciles for School D DECILE Count MEDIAN MINIMUM MAXIMUM 1 3594 1963 1914 1968 2 3896 1972 1969 1975 3 3466 1979 1976 1981 4 3572 1984 1982 1986 5 3904 1989 1987 1991 6 3398 1994 1992 1995 7 4486 1998 1996 2000 8 3125 2002 2001 2003 9 4251 2006 2004 2007 10 4046 2009 2008 2011
100.0% 80.0% 60.0% 40.0% 20.0% Fig. 4: Percentage of Alumni with an E- mail Address Listed In School D by 42.4% 45.1% 49.7% 49.7% 47.7% 46.7% 51.7% 56.5% 57.8% 82.5% 0.0% The most significant implication we can draw from what we ve shown you so far is this: If any of these four schools were to conduct a web survey by only contacting alums with an e- mail address, they would simply not reach large numbers of alums whose opinions they are probably interested in gathering. Some specifics: School A: They would miss huge numbers of older alums who graduated in 1974 and earlier. By rough count over 40% of these folks would not be reached. That s a lot of senior folks who are still alive and kicking and probably have pronounced views about a number of issues contained in the survey. School B: A look at Figure 2 tells us that even considering doing a web survey for School B is probably not a great idea. Fewer than 20% of their alums who graduated in 1998 or earlier have an e-mail address listed in their database. Another way of expressing this implication is that each school (regardless of what their response rates were) would largely be tapping the opinions of younger alums, not older or even middle-aged alums. If that s what a school really wants to do, okay. But we strongly suspect that s not what it wants to do. Now let s look at something else that concerns us about doing web surveys if potential respondents are only contacted by e-mail: Giving. Figures 5-8 show the percentage of alums who have given $100 or more lifetime by e-mail address/no-email address across class year deciles. As we did with Figure 1, let s go over Figure 5 to make sure it s clear. For example, in decile 1 (oldest alums) 87% of with an e-mail address have given $100 or more lifetime to the school. Alums in the same decile who do not have an e-mail address? 71% of these alums have given
$100 lifetime or more to the school. How about decile 10, the youngest group? What are the corresponding percentages of giving for those alums with and without an e-mail address? If you came up with 14% versus 6%, we ve been clear. Fig. 5: Percentage of Alumni Who Have Given $100 or More LifeSme for School A by E- mail Address Listed and Class Year Decile. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 87% 85% 83% 86% 82% 71% 74% 62% 64% 59% 60% 55% 51% 42% 35% 32% 22% 14% 10% 6% EMAIL NO EMAIL Take a look at Figures 6-8. Then we ll tell you the second implication we see in all these data.
Fig. 6: Percentage of Alumni Who Have Given $100 or More LifeSme for School B by E- mail Address Listed and Class Year Decile 80% 70% 60% 50% 40% 30% 20% 10% 0% 73% 62% 56% 55% 43% 46% 39% 32% 29% 29% 25% 30% 20% 14% 17% 8% 5% 4% 2% 3% EMAIL NO EMAIL 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Fig. 7: Percentage of Alumni Who Have Given $100 or More LifeSme for School C by E- mail Address Listed and Class Year Decile 80% 85% 84% 83% 46% 45% 47% 41% 76% 69% 33% 27% 57% 15% 42% 6% 26% 12% 2% 1% EMAIL NO EMAIL
60% Fig. 8: Percentage of Alumni Who Have Given $100 or More LifeSme for School D by E- mail Address Listed and Class Year Decile 55% 50% 40% 30% 20% 10% 0% 42% 41% 37% 34% 32% 26% 19% 17% 14% 16% 12% 9% 9% 4% 4% 2% 1% 1% EMAIL NO EMAIL The overall impression we get from these four figures is clear: Alumni who do not have an e-mail address listed give considerably less money to their schools than do alumni with an e- mail address listed. This difference can be particularly pronounced among older alums. Some Conclusions The title of this piece is: Are We Missing too Many Alumni with Web Surveys? Based on the data we ve looked at, we think the answer to this question has to be a yes. It can t be a good thing that many web surveys don t go out to so many older alums who don t have an e-mail address, and to alums without an e-mail address who haven t given as much (on average) as those with an e-mail address. On the other hand, we want to stress that web surveys can provide a huge amount of valuable information from the alums who are reached and do respond. Even if the coverage of the whole alumni universe is incomplete, the thousands of alums who take the time to fill out these surveys can t be ignored. Here s an example. We got to reading through the hundreds and hundreds of written comments from a recent alumni survey. We haven t included any of the comments here, but my (Peter s) reaction to the comments was visceral. Wading through all the typos, and misspellings, and fractured syntax, I found myself cheering these folks on: Good for you.
Damn right. Couldn t have said it better myself. I wish the advancement and alumni people at my college could read these. In total, these comments added up to almost 50,000 words of text, the length of a short novel. And they were a lot more interesting than the words in too many of the novels I read. As always, we welcome your comments.