About a year ago, I whipped together a quick look into Pitchfork’s album rating system in order to give artists who have been reviewed by the site a better understanding of what their numerical value actually meant. In order to do so, I compiled a more-or-less complete breakdown of p4ks album reviews of original music (no reissues, no live albums, no “greatest hits”) from February 24, 2009 to February 24, 2010 and displayed the scores in more meaningful percentiles. I also made a series of observations concerning their “Best New Music” designation which produced a little bit of a stir.
Now, a year later, I decided it would be interesting to see if the dynamic of Pitchfork’s ratings system has changed over time by briefly analyzing album ratings from February 25, 2010 to February 25, 2011 against my previous data set. Again, I focused my attention on original content which meant I had to go through each album review individually and see if it met my criteria for inclusion. In addition to the items mentioned above, soundtracks, label compilations, and comedy albums were excluded but EPs, remix albums, and mixtapes were deemed OK as I wanted to maintain consistency with the previous year’s analysis. The first astonishing thing I noticed when putting together the data sets was how the total number of original albums reviewed by the site were nearly identical from 2009 to 2010 (1025 and 1027, respectively)!
So without further ado, let’s get to some plots (click on the images to view a slightly larger size):
The above histogram shows the distribution of the scores for each year. Glancing at the figures, you can easily tell that there is good agreement between the two years (in fact, the correlation coefficient is .9336, indicating that the data is strongly correlated). This was expected as any long-time p4k readers can attest to the fact that:
- Pitchfork tends to not review low performing albums (the reason why the plots are negatively skewed).
- A majority of album ratings fall within the 6.0 – 8.0 range.
- An extremely small amount are of high quality, explaining the dearth of albums achieving 9.0 and above.
However, a more detailed inspection between the two histograms show a couple of interesting differences. For one, the most recent data set (I’ll refer to it as “albums from 2010″ from now on, which is somewhat of misnomer but close enough for our purposes) shows ratings centered at a higher value than the previous year’s data (which I’ll refer to as “albums from 2009″). This can be shown more clearly by looking at the two histograms overlapped on each other. What this implies is that the albums from 2010 were rated more favorably than albums from 2009.
Another interesting find when you compare the two histograms is that the most frequent album rating has changed from 7.0 to 7.5 over the past year. Not only that, but the number of occurrences for the most common rating has increased 20% (51 vs. 61 albums). In a related point, the 2010 histogram is noticeably more concentrated in the 6.5-8.2 range than the 2009 plot. This implies that Pitchfork is not distributing the scores as evenly in 2010 as compared with 2009. It’s common knowledge that Pitchfork doesn’t fully utilize the 100 potential rating options in the most effective manner (see: normal distribution), however, the fact that they are classifying more albums in less of a range is disheartening at best and troubling at worst. I call this trend towards album ratings homogeneity the “Rolling Stone Effect.”
Looking at the box plots of the data confirm both of these conclusions:
Notice the median score, denoted by the red vertical line in the box, shows an increase of .2 points between the two years (7.0 to 7.2) as well as lower quartile increase of .2 points and an upper quartile increase of .1 points. This confirms that, according to Pitchfork, albums from 2010 were generally better than albums from 2009. Also, the difference between the lower and upper quartile has shrunk by .1 (1.4 from 1.5) — and it would have been reduced a further tenth of a point if not for an uncharacteristically high amount of albums garnering an 5.8 rating in 2010 (notice the spike on the histogram at that value). This indicates that the range where a majority of albums score is indeed reducing.
Here are the percentile breakdowns for 2009 and 2010 so that any artist or band whose been fortunate enough to have gotten reviewed by the site can see how they stack up against other albums released within the same year. These percentiles also show how in 2010 an artist had to score a higher value in order to remain in the same percentile, further convincing us that p4k viewed 2010 as a better year in music than 2009:
Switching gears and looking at album’s that achieved the “Best New Music” designation, much of the same complaints from last year still apply:
- 8.7 was the lucky number this year (compared with 8.6 last year). All albums scoring that or higher received an automatic BNM.
- Guitarist William Tyler got the biggest shaft this year as he was the sole person to score an 8.6 and not get a BNM.
- Metal/Hardcore seems to be perennial Pitchfork losers as the two albums receiving an 8.5 but not getting the BNM belonged to these genres (Agalloch’s Marrow of the Spirit and The Body’s All the Waters of the Earth Turn to Blood). In addition, Kylesa’s Spiral Shadow and Alcest’s Écailles de Lune‘s 8.4s weren’t enough to snag a BNM either (two of only five albums with that score not to get it).
- Several oddball genres also had several high ratings without clinching a BNM.
- Incredibly surprised that Drake’s Thank Me Later (8.4) didn’t get a BNM considering how much coverage it had on the site and I’m personally dismayed by Forest Sword’s Dagger Paths falling a bit short as well.
- Only two of the twenty-nine albums receiving an 8.2 got BNM-ed (the well publicized Morning Benders and Wild Nothing records).
- The lowest scoring album to receive a BNM was Liar’s Sisterworld as it was the lone 8.1 out of twenty-five candidates to get the p4k stamp of approval.
- Released in June 2010, Emerald’s Does It Look Like I’m Here was the last album to score an 8.3 and receive a BNM. Since then a disproportionately high number of albums have scored an 8.4s and have been receiving the BNM honor. To me, this indicates that an informal boundary for Best New Music has been set for 8.3 starting this past summer. (EDIT: Although too recent to be included in either data set, it seems like Lykke Li’s Wounded Rhymes breaks this trend).
- As expected, Kanye’s 10.0 is a statistical outlier.
After last year’s post, a lot of people expressed the position that the “Best New Music” category was meant for high quality albums that are easily accessible to the average music listener. As a result, this would automatically disqualify genres such as metal, electronic offshoots, and jazz. Even if taking this stance, I personally feel that these high scoring albums from “unfamiliar” genres should at least be better represented in the year-end lists (Kylesa landed at #44 and Forest Swords at #48 with Actress and Guido getting Honorable Mentions).
Well that does it for this year! If you want to run some other stats of your own, you can download my raw data here. I have a lot of ideas on where to go further with this project that (dependent, of course, if I have the time). Oh, one last thing I’d like to point out before saying adieu: of all the hyperlinked items and searchable content that is on the pitchfork site (artists, albums, labels, etc…), I find it incredibly surprising that you can not search for album reviews by rating or writer. It isn’t too terribly complicated to code these features and they would be incredibly helpful for site readers (especially data miners like myself). OK, on to the discussion!
