E-mail Battle Royale

In the world of political technology vendors, an odd conflict is raging. Triggered by Congress’ adoption of challenge authentication for delivering e-mail, a pitched battle is being waged on the pages of the Washington Post and the e-mail inboxes of people like me who make a living on this stuff.

The battle begins with this study by Capitol Advantage – a vendor of electronic advocacy services and products. On their home page they state that Capitol Advantage ranked #1 in an “industry study on delivering constituent e-mails to Congress.” On the page housing the study, however, they’re a bit more honest.

Dr. Dennis Johnson, professor of Political Management at the George Washington University and a leading expert on online communication, led the research conducted by Capitol Advantage on citizen communication deliverability.

Well, that’s hardly an “industry study” now is it? I work for an association that represents the cable industry. If one of our companies commissioned a study claiming that all of the other companies sucked, and described it as an “industry study”, the rest of the industry would rightly be enraged.

Well that’s exactly what Capitol Advantage has done.

As of this morning, the clash spilled over into my mailbox with a message from a competing vendor challenging the study and suggesting they would like a comprehensive study to be completed and suggesting they are open to making it an actual industry study by allowing other companies to help determine the methodology. I’ll go into that e-mail in a minute (though I will respect the company’s request to keep the contents and the sender confidential). First, I would like to explore the CapAdvantage study.

Not surprisingly, since CapAdvantage was paying him, the good doctor determined that “this research project clearly demonstrates that citizens and advocacy organizations are being shortchanged by some advocacy communications vendors who promise to send their messages to Congress and fail to do so. Only one vendor (Capitol Advantage) scored at a rate that citizens should expect from an advocacy vendor: 97.3 percent.”

I’ll get into the weeds after the jump…

So here is there methodology in a nutshell.

Four Senate personal offices and 33 House personal offices were chosen as the test recipients. Each of these offices employs one or more requirements or authentication measures, and each sends automated responses alerting the e-mail sender that their e-mail has been received. The automated response mechanisms for all 37 congressional offices were tested by sending a message directly to the lawmaker (rather than through a vendor) and the automated responses worked 100 percent of the time. This became the control group by which all vendors would be measured.

The research team used the e-mail forms on each of the 10 vendor Web sites to send e-mail messages to the 37 targeted congressional offices. The team utilized actual constituent addresses for every message sent, to ensure the message would not be filtered out because the sender was a non-constituent. Testing began in late August and concluded in early September. Within a matter of hours, automated responses were received from congressional offices. Nearly all came within 24 hours, but to ensure any late automated responses might be received and credited to the vendor, the researchers did not close the return deadline until one week after the original e-mails were sent. There were no known anomalies in the congressional online system during this time, such as a power failure, repair work on a certain online system, or any other factor that might skew the responses.

Sounds good in theory. Let’s see what happened.

The results of this research were unsettling. Like citizens who expect their postal mail to be delivered and not dumped into the trash by the mail carrier, online citizens and organizations should demand the same level of reliability and accountability from vendors. Insisting that e-mail be delivered at least 90 percent of the time is not unreasonable and should represent a bare minimum level of competency. But what the research found was that several of the vendors failed miserably in their ability to send e-mails to Congress: 6 of the 10 could not deliver 50 percent of the e-mail through their systems. Of those 6, three utterly failed, with a 16.2 percent return and two 0 percent returns. One vendor scored just over 50 percent, another at 67.6 percent, and another at 83.8 percent—all still below what should be a minimal acceptance rate of 90 percent. Only one vendor (Capitol Advantage) scored at a rate that citizens should expect from an advocacy vendor: 97.3 percent.

Well, that is shocking to say the least, but is it scientifically sound. Well, the big answer to that is “it depends, but probably not.” How is that possible? I’ll explain.

Capitol Advantage has never been a sophisticated mail delivery system and thus I have never used them (in the interest of full disclosure, I have actually gone so far as to cancel a long-standing contract they had with the RNC when I took over as eCampaign Director there because I was unhappy with their product). To be fair to Capitol Advantage, if a company is looking for a low-cost, unsophisticated platform to send generic messages en masse with little regard for flexibility, they probably do have a really good product.

A sophisticated mail delivery system, will allow you a great deal of flexibility with your online campaigns. For instance, I should be able to pick only the members of a particular committee, or even one member in a specific state, to whom I want to target my messages.

Advocacy solutions provided by DDC, Grassroots Enterprise, and others will generally allow you to do such specific targeting. In some cases, the site may still allow you to complete a form as an anonymous user. It simply doesn’t know if you are a constituent of that member until you provide an address. It will almost always tell you that you are not a constituent for that particular action if you live outside the district, however. For registered users of a site, many systems will go so far as to not allow you to see alerts for which your legislator is not a target.

It is not clear from the study whether the staff conducting the analysis was told to look for such messages. Therefore it is not clear whether the 37 members studied were actually targets of the actions the testers used to send a message.

That last point becomes very significant because of one flaw pointed out by the e-mail I received. It highlights the fact that one of the sites used in the study was not configured to send messages to House offices. Therefore any mail they attempted to send to the 33 house offices would have received no reply, and the maximum score (assuming all four Senate offices were targeted) would be a four out of 41 – less than 10 percent.

For my work, we often target only members of four committees – House and Senate Commerce and Judiciary. That is a total pool of only a few dozen Members of Congress. Had the study tested one of the advocacy websites I use to mobilize employees, it likely would have received poor marks if judged based on Members who were not on those Committees.

The study methodology, in other words, appears to be unsound because it attempts to treat all advocacy solutions as homogenous, when the very selling points for most systems are in their ability to do more, better than the competition.

Capitol Advantage did do the industry a service in pointing out the difficulty in communicating with Congress when they would rather have one lobbyist meeting than receive 10,000 e-mails for actual constituents – whether they were facilitated by an association or not.

However, having been working in the field of political technology for more than 12 years now, it is my opinion that they went way too far in citing a flawed study that uses questionable methodology in an attempt to discredit other, more sophisticated solutions that all face the same hurdle – Congress.

In the interest of full disclosure, NCTA, my employer, uses data and services purchased from DDC – a Capitol Advantage competitor – though not for e-mail advocacy. We also maintain contracts with Grassroots Enterprise – another competitor, and use a proprietary advocacy solution built for us by New Media Communications.

The views expressed here are mine, and mine alone, and do not reflect the views of my employer.

Share

Tags