Yesterday the Capital ran a story on the results of the superintendent survey designed to elicit feedback on the characteristics the public wants in its next public schools superintendent. From November 22 to December 12, 2013, AACPS encouraged the public to fill out the survey (I reported on that here). On December 16, 2013, it emailed the results to the superintendent search firm, which presented the results at the January 8, 2014 AACPS Board of Education meeting. The Capital’s story, Nearly 2,000 people respond to Anne Arundel superintendent survey, reported on that presentation.
Here’s what the Capital said about the survey’s methodology:
Nearly 2,000 people responded to an online survey designed to help determine the best characteristics for the next schools chief, said William Newman, director of Iowa-based search firm Ray and Associates Inc.
Newman told the county Board of Education on Wednesday that number of responses is beyond what he has seen in other school districts.
Anne Arundel has just under 80,000 students in its public schools. Newman, a former superintendent who has worked with schools all over the country, said he has received “less than 600” responses from residents in large jurisdictions.
“This was a good sampling,” he told the board…”
The article then went on to summarize responses from the survey.
The implication of the methodological statement was that the survey was representative of the public. But not only was it a self-selected group of respondents, they were highly skewed. Of the 1,924 survey respondents, only 716 (37.0%) were from parents. The majority, 1,087 (56.3%) were from county employees. Many of the categories were analytically confused (for example, many county employees are also parents, but they could only pick one category to describe themselves). But let’s take these categories at face value.
The word “sample” implies a sample from a population. But the population was never stated. Was the population all the citizens of Anne Arundel County, the parents of AACPS students, the citizens of Anne Arundel County, or “stakeholders” (one of the favorite nebulous words used by the Board of Education). If politics is the art of ambiguity, this is an example of best practice.
The only population that could possibly make sense to characterize this sample is “stakeholders.” That would imply that AACPS employees (50.5% of respondents) are the most influential group choosing the next superintendent. That is actually what I think the search firm meant and what the Board of Education really sought. But the reporter didn’t provide the public with the needed information to decode what was really happening.
The assertion that close to 2,000 responses was a great response because the search firm usually only gets less than 600 responses from large jurisdictions should also have been probed. Fewer than 1 in 300 public school districts in the U.S. has a larger student population than AACPS. AACPS isn’t just a large public school district; it is a GIANT one. Unless the comparison is to similarly GIANT districts, the purported comparison praising the comprehensiveness of the survey results is meaningless.
Despite the above critique, this is one of the best surveys that AACPS has administered in recent years. Perhaps the most noteworthy feature of this self-selected survey is that it didn’t ask respondents for names and other personally identifiable information. The survey (and the accompanying public forums) may have been designed largely as a publicity stunt to demonstrate that the Board of Education is responsive to public opinion (we do live in a democracy, after all). But at least it wasn’t part of a surreptitious lobbying campaign. That is, AACPS isn’t using this survey as a way to identify grassroots supporters that can later be activated for lobbying purposes. There is a big difference between using government resources to engage in PR versus lobbying, and in this case AACPS didn’t cross that line. Alas, however, the Capital did cross the line into misleading reporting.
