by Lauri Donahue

When Variety reported on July 22 that Piedmont Media Research had developed a system for predicting a movie’s box office take before the film was even made, the firm started fielding calls from some unexpected customers: screenwriters.

Within weeks, Piedmont had heard from more than 50 scribes hoping to use the company’s patent-pending “consumer engagement”-based model to pitch ideas to studios. No longer would writers have to resort to “hit #1 meets hit #2” comparisons to hype the marketability of their concepts. Now they could use hard numbers, backed up by audience research.

Moneyball for Movies

Piedmont describes its method as “Moneyball for Movies.” That 2003 book, and the 2011 Brad Pitt movie based on it, are about how the Oakland A’s used statistical analysis to identify baseball players undervalued by other teams, and then acquire those players at bargain basement salaries.

But Piedmont’s method doesn’t measure something as easy to quantify as batting averages and RBIs. It’s based on an audience’s emotional engagement with a story and characters. This level of engagement may be subjective, but it can be assigned a number which may predict box office success well in advance of a film’s release.

As reported by Variety,

…as early as February, the newbie company predicted “R.I.P.D.” would be one of this summer’s biggest flops. The film, which opened with a dismal $12.7 million domestic, earned a composite score of just 137, according to Piedmont’s consumer engagement rating system; a film needs to score at least 250 to be successful at the box office, according to Piedmont prexy Josh Lynn.

Lynn and his colleague Jason Olson met as students at UC Berkeley. Lynn went on to the London School of Economics, while Olson went to UC Berkeley’s Law School and Harvard’s Kennedy School.

Piedmont was founded in 2010. Initially the firm looked at easy-to-measure factors like genre, rating, and casting, building a still-growing database of more than 400 films in an attempt to come up with a predictive model for commercial success. They found what was missing was a measure of “how do people feel about this idea?”

Picking a Winner

Lynn says that the response to the Variety article has been very positive. “The market is a lot tighter with the DVD market collapsing. There aren’t as many movies being made. There’s not as much leeway for failure. I think the way Hollywood has done business is pretty inefficient, especially with so much money involved. It seems people are interested in a smarter, more accurate way to decide what to make and at what level.”

Other industry market research companies assign weights to various success factors: a particular actor might be assumed to contribute 20% to a movie’s success, a writer 5%, etc. Or a studio can try to predict results based on “comps” – “comparable” movies.

The problem with comps is that it’s too easy to cherry pick and manipulate them into meaninglessness. If you’re evaluating the value that Nicholas Cage adds, do you look at the worldwide gross for National Treasure ($173 million) or Stolen ($2.1 million)? Can every movie targeted at teenagers use Harry Potter as a comp? Can every gangster movie be compared to The Godfather?

Other methods can only predict a movie’s success after it’s already been made, when the budget is a sunk cost. For example, in June Google released a white paper called “Quantifying Movie Magic with Google Search,” in which it’s claimed the company can predict box office results four weeks before a film’s opening with 94% accuracy, based on which trailers people are clicking on. These forecasts may make studio execs order new Teslas—or update their resumes—but they don’t tell them what movies they should greenlight in the first place.

How It Works

Measuring consumer engagement using the Piedmont method starts with drafting a written description for a movie that may or may not have been made yet. Piedmont shapes and standardizes the pitch, then comes up with a set of questions that test consumer response. The result is an overall numerical score, broken down by various demographics: age, sex, ethnicity, etc.

As Lynn said in Variety,

Instead of throwing money at a film or an actor and hoping for the best, there is a better, more analytic way to determine beforehand if a film is worth making, and at what specific dollar value.

Piedmont surveys a statistically significant sample of the population, using questions that measure consumers’ personal connections to a given concept and how likely they are to see the film. This results in a Consumer Engagement (CE) number from zero to 1000.

Over the 400 films tested by Piedmont, the firm says the correlation between the CE number and the opening weekend box office is .74. They say the correlation to the total box office over the entire run of the movie is .70. (The closer a correlation coefficient is to 1, the stronger the predictive value of the model.)

According to the Piedmont website,

The Consumer Engagement metric by itself has shown to have the highest correlation to box office grosses when compared to any other single component factor tested including overall quality, production budget, film rating, number of screens, and film genre, among many others tested.

What this means is the Consumer Engagement metric is a more significant and direct indicator of a film’s opening weekend box office than any other tested measure, according to Piedmont.

The highest overall CE score the company has seen was for Iron Man 3, at 458. The movie has earned over $1.2 billion worldwide so far. The highest number for any particular demographic was over 600 for the latest installment of the Twilight franchise. Not surprisingly, this was for an audience of 12-17-year-old females.

In general, says Lynn, comedies will score between 110 and 160. Dramas will score a little lower. Indie movies often score at 100 or below, which is not necessarily a problem given their modest budgets. Big budget summer films need to score 250 or above to justify their existence.

MIB 3, the third installment in the Men in Black franchise ($624 million worldwide) scored over 300, as did Man of Steel ($647 million). The Harry Potter and Pirates of the Caribbean sequels scored over 400.

The Piedmont model can also identify flops, often because they fail to appeal to important demographics. For example, Battleship tested fairly well with male audiences, to which it was pitched as the next Transformers. But it didn’t break 100 in any female demographic. With women comprising 52% of the ticket-buying population, that was a problem. The movie earned $303 million worldwide (but only $65 million domestic) on a budget of $209 million.

(Women, especially older women, tend to be more review-sensitive than men. It probably didn’t help that Battleship had a Rotten Tomatoes rating of 34%.)

The CE stats can also flag movies likely to be sleeper hits because they strongly appeal to particular demographics hungry for movies that speak to them. When Piedmont was asked to test it, Lynn had never heard of the 2009 novel The Help, which spent more than 100 weeks on the New York Times bestseller list. He was surprised to see that the concept scored very well with women 35-49 and 50+. In fact, the 50+ female demographic rated it about 420. The 2011 movie version earned $211 million worldwide on a budget of $25 million.

Lynn notes that there are many other underserved audiences. The Fast and Furious franchise targets both the Hispanic audience (which buys 26% of US movie tickets) and the international audience. The Best Exotic Marigold Hotel targeted an older audience and earned $137 million worldwide on a budget of $10 million. “Once you’re able to target a demographic, you can get a huge response because of the pent-up demand,” says Lynn. “Lots of audiences are underserved and studios are wising up to that.”

But perhaps not fast enough.

Despite the fact that women buy more than half of US movie tickets, and despite the recent success of female-driven movies like Bridesmaids, Hunger Games, Twilight, and Identity Thief, female representation in films is at its lowest level in five years, according to a USC study covered by the LA Times:

Among the 100 highest-grossing movies at the U.S. box office in 2012, the study reported, 28.4% of speaking characters were female. That’s a drop from 32.8% three years ago, and a number that has stayed relatively stagnant despite increased research attention to the topic and several high-profile box-office successes starring women.

According to the USC researchers, these numbers persist because Hollywood insiders believe that attracting the male audience is the key to a movie’s success:

“Industry perceptions of the audience drive much of what we see on-screen,” said study author Stacy L. Smith. “There is a perception that movies that pull males sell. Given that females go to the movies as much as males, the lack of change is likely due to entrenched ways of thinking and doing business that perpetuate the status quo.”

“To me, it’s pretty asinine to just ignore a large percentage of your audience,” says Lynn.

Development by the Numbers

According to Lynn, the CE number can not only tell you which movies to make, but what version of a movie to make. “You can test different versions of a movie called The Wolverine. One is set in Japan, one is set on a raft in the middle of the ocean, one has Wolverine go antiquing…. We’ve also tested concepts without titles to see how they vary.”

The New York Times recently ran an article about Vinny Bruzzese and his company Worldwide Motion Picture Group, which analyzes projects at the script level. Based on the performance of similar movies, Bruzesse gives advice at the micro level of script development:

“Demons in horror movies can target people or be summoned,” Mr. Bruzzese said in a gravelly voice, by way of example. “If it’s a targeting demon, you are likely to have much higher opening-weekend sales than if it’s summoned. So get rid of that Ouija Board scene.”

Lynn says the Piedmont model lets development execs change around the pieces in a concept and see how audiences respond—rather than just relying on historical data as Worldwide does. “You can test targeting demons and summoned demons within the context of a particular film. It may be that targeting and summoned demons test the same across all projects.”

Or not.

Piedmont can also test how particular actors affect a movie’s box office prospects. For example, Denzel Washington might increase a CE score by 35% across all concepts, or he might be worth $3 million in one project but $40 million in another. “Cameron Diaz in Bad Teacher is one thing,” says Lynn, “but in Platoon 2 she might actually bring down the CE.”

The movie that had the biggest score increase from concept to casting was The Heat, an R-rated cop comedy. The concept on its own tested very badly, but when the leads Sandra Bullock and Melissa McCarthy were added to the mix it jumped 90%. The movie earned $200 million worldwide on a budget of $43 million.

Consumer Engagement and High Concepts

So how does consumer engagement relate to “high concept”? Will a “high concept” automatically have a high CE score? Not at all.

Lynn says that when he hears “high concept” he thinks of a film that can be easily described and sold in one sentence. He says that in the past Hollywood has looked at “high concept” as meaning “easy to sell”—easy to put on a poster. It’s easier to figure out how to sell Jurassic Park than The Unbearable Lightness of Being.

But “easy to sell” doesn’t necessarily mean that people will rush to buy, as many “high concept” flops have proven. Exhibit A: Snakes on a Plane ($34 million domestic on a budget of $33 million).

How Writers Are Using Market Research to Sell Pitches

Lynn was surprised that so many screenwriters contacted Piedmont in the days after the Variety article ran. Although he won’t say what the company charges for its studies, they aren’t priced for the average consumer.

He says the writers are a mix, both established and not. “When you’re trying to sell a script, you want every tool at your disposal,” he says. “Studios are all about the numbers. It’s more than ‘I love it and my friends love it.’”

He says that about 15 of the writers are already going ahead with test projects, and more are in discussions. He also says that writers can use the Piedmont service to decide which script to write next. “If writers have five different ideas they haven’t written yet, we can test that. And that’s another tool in their tool belt.”

One of the writers using Piedmont’s service is Sarah Afkami, a staff writer on TBS’s The Wedding Band and formerly a staff writer for Wilfred on FX. She’d been developing a feature comedy called Going for Broke with her mentor (an established screenwriter) when they read the Variety article about Piedmont and decided to use the service to help make some big decisions—like about the gender of the lead character.

Once the audience feedback told them that the lead should be male, they did another round of testing with specific actors “attached”—and earned a healthy 150 CE. That score got them some serious interest from a production company. “It feels like you have more in your arsenal [when you come in with the Piedmont numbers],” says Afkami. “You don’t come off as some fluffy artist. You walk with more of a swagger into your pitch meeting.”

The End of Development Hell?

If everything can be tested, what does that mean for development execs? And is it good or bad for screenwriters? “Development hell” is the period of months, years, or decades in which a movie project is trapped in development, subject to notes from studio execs, directors, actors, and others. The original writer may be required to produce endless revisions in response to these notes, or additional writers may be brought in to give a “fresh take” on the material.

Lynn says that in an ideal world (in which Hollywood adopted the Piedmont method), CE scores should allow scripts to bypass development hell.

If nothing else, a Piedmont score makes a writer better armed when defending a script. “I can see a writer going into a development meeting with the study as ammo,” says Lynn. “There are some things that don’t matter, like whether a story is set in Arizona or New Mexico. And there are some things that may matter in some contexts but not in others, like whether a character is Hispanic or Caucasian, straight or gay. Instead of having development execs nit-pick over small, personal, subjective details, you can test the story elements and see what’s really important.”

He says that when you know which details matter financially and which don’t, it gives a writer more creative freedom. The writer can flesh out the story as he or she sees fit, rather than having to respond to every development exec whim. “Once people sign off on the concept, the details are irrelevant to the financing. All the details are fair game as long as the script is within a certain level of quality.”

What Does All This Mean for Screenwriters?

So what does all this mean for screenwriters who can’t afford to shell out for a market survey they can use to pitch to a studio? Are there any general lessons to be learned about what stories appeal to audiences?

“Once you know that a story is worth writing, as far as the marketplace is concerned, I would say write what you love,” says Lynn. “I know this sounds kind of basic, but if it’s something you care about you’ll do a great job of it. When you try to predict total box office an important factor is the quality of the film, and a lot of that is based on the quality of the script. Write what you love and the quality will be there.”