The Science and Marketing of Sound Quality

To my surprise, this morning an audio friend tweeted a link to an article I recently wrote for our company's  internal newsletter  entitled, "The Science and Marketing of Sound Quality."  My article can be found on a new Harman Innovation website  launched today that features articles on current and future disruptive technology that will impact consumers' infotainment experiences. Check it out.

My article focuses on a longstanding pet peeve of mine (first mentioned in this blog posting): The lack of  perceptually meaningful loudspeaker and headphone specifications in our industry.  While consumer surveys repeatedly report sound quality to be a driving factor in their audio equipment purchases, consumers lack the necessary tools and information to identify the good sounding products from the duds.

This is particularly true for loudspeakers and headphones where the typical throw-away "10 Hz to 40 kHz" specification provided by the manufacturer is utterly useless. This specification only guarantees that the product makes sound, with no guarantee that the sound is good.  While the science exists today to accurately quantify and predict the perceived the sound quality of  loudspeakers (and hopefully, soon headphones), the audio industry continues to drag its heels into the 21st century,  and not routinely provide this information to consumers.

A rare exception is JBL Professional who provides comprehensive detailed measurements on studio/broadcast monitors like the new JBL M2 Master Reference shown below. Inspecting the measured frequency response curves shown  below, you can easily recognize the loudspeaker sounds exceptionally neutral and accurate based on the shape (flat, smooth, and extended)  Based on this set of measurements, we can predict how a listener would rate the sound quality of the loudspeaker in a controlled listening test, with 86% accuracy. The only pertinent information not shown in this graph is how loud the loudspeaker will play before producing audible distortion (trust me, this loudspeaker will play very loud! )

Perceptually meaningful loudspeaker specifications like these have been available for almost 30 years! Yet,  these specifications are currently not part of any professional and consumer loudspeaker standard. Such a standard would go a long way towards improving the quality and consistency of recorded and reproduced sound. Audio consumers want to hear the truth. We need to provide better information and audio specifications so they can find it.

JBL M2 Master Reference Monitor provides true reference sound quality that is clearly indicated by its technical measurements shown below. 

The spatially-averaged frequency response curves of the JBL M2  (from top to bottom) for the listening window (green), the first reflections (red), and the total radiated sound power.  At the bottom are shown directivity indices of the sound power (dotted blue) and first reflections (dotted red). These measurements tell us that the quality of the direct and reflected sounds produced by the loudspeaker will be very accurate and neutral over a relatively wide listening area.

Harman Researchers Make Important Headway in Understanding Headphone Response

Todd Welti, Sean Olive and Elisabeth McMullin are shown above with their custom binaural mannequin, "Sidney" wearing a pair of AKG K1000's. No fit or leakage issues with these headphones.

Tyll Hertsens, Chief Editor at Innerfidelity recently visited our research labs in Northridge, and wrote a nice story in his blog about our headphone research and visit to Harman. You can read the entire story here.

In his story, Tyll summarizes three of our recent AES papers on headphones, the first one of which I already wrote about in this blog. I hope to write summaries of the other two papers in the upcoming weeks when I can find some free time.

The Relationship between Perception and Measurement of Headphone Sound Quality

Above: The brands and models of six popular headphones used in this study.

In many ways, our scientific understanding of the perception and measurement of headphone sound quality is 30  years behind our knowledge of loudspeakers. Over the past three decades, loudspeaker scientists have developed controlled listening test methods that provide accurate and reliable measures of   listeners' loudspeaker preferences, and their underlying sound quality attributes.  From the perceptual data, a set of acoustical loudspeaker measurements has been identified from which we can model and predict listeners' loudspeaker preference ratings with about 86% accuracy.

In contrast to loudspeakers, headphone research is still in its infancy. Looking at published acoustical measurements of  headphones you will discover there is little consensus among brands (or even within the same brand) on how a headphone should sound and measure [1]. There exists too few published studies based on controlled headphone listening tests to identify which objective measurements and target response curves produce an optimal sound quality. Controlled, double-blind comparative  subjective evaluations of different headphones present significant logistical challenges to the researcher that include controlling headphone tactile and visual biases. Sighted biases related to price, brand, and cosmetics have been shown to significantly bias listeners judgements of loudspeaker sound quality. Therefore, these nuisance variables must be controlled in order to obtain accurate assessments of headphone sound quality.

Todd Welti and I recently conducted a study to explore the relationship between the perception and measurement of headphone sound quality. The results were presented at the 133rd AES Convention in San Francisco,  in October 2012.  A PDF of the slide presentation referred to below can be found here. The AES preprint can be found in the AES E-library. The results of this study are summarized below.

Measuring The Perceived Sound Quality of Headphones

Double-blind comparative listening tests were performed on six popular circumaural headphones ranging in price from $200 to $1000 (see above slide).  The listening tests were carefully designed to minimize biases from known listening test nuisance variables (slides 7-13). A panel of 10 trained listeners rated each headphone based on overall preferred sound quality, perceived spectral balance, and comfort. The listeners also gave comments on the perceived timbral, spatial, dynamic attributes of the headphones to help explain their underlying sound quality preferences.

The headphones were compared four at a time over three listening sessions (slide 12).  Assessments were made using three music programs with one repeat to establish the reliability of the listeners' ratings.  The  order of headphone presentations, programs and listening sessions were randomized to minimize learning and order-related biases. The test administrator manually substituted the different headphones on the listener from behind so they were not aware of the headphone brand, model or appearance during the test  (slide 8).  However, tactile/comfort differences were part of the test.  Listeners could adjust the position of the headphones on their heads via light weight plastic handles attached to the headphones.

Listeners Prefer Headphones With An Accurate, Neutral Spectral Balance

When the listening test results were statistically analyzed, the main effect on the preference rating was  due to the different headphones (slide 15).  The  preferred headphone models were perceived as having the most neutral, even spectral balance (slide 19) with the less preferred models having too much or too little energy in the bass, midrange or treble regions.  Frequency analysis of listeners' comments confirmed listeners' spectral balance ratings of the headphones, and proved to be a good predictor of overall preference (slide 20). The most preferred headphones were frequently described as "good spectral balance, neutral with low coloration, and good bass extension," whereas the less preferred models were frequently described as "dull, colored, boomy, and lacking midrange".

Looking at the individual listener preferences, we found good agreement among listeners in terms of which models they liked and disliked (slides 16 and 18). Some of the most commercially successful models were among the least preferred headphones in terms of sound quality. In cases where an individual listener had poor agreement with the overall listening panel's headphone preferences, we found either the listener didn't understand the task (they were less trained),  or the headphone didn't properly fit the listener, thus causing air leaks and poor bass response; this was later confirmed by doing in-ear measurements of the headphone(s) on individual listeners (slides 26-39).

Measuring the Acoustical Performance of Headphones

Acoustical measurements were made on each headphone using a GRAS 43AG Ear and Cheek simulator equipped with an IEC 711 coupler (slide 24). The measurement device is intended to simulate the acoustical effects of an average human ear including the acoustical interactions between the headphone and the acoustical impedance of the ear.  The headphone measurements shown below include these interactions as well as the transfer function of the ear, mostly visible in the graphs as a ~10 dB peak at around 3 kHz.  It is important to note that we since we are born with these ear canal resonances, we have adapted to them and don't "hear" them as colorations.

Relationship between Subjective and Objective Measurements 

Comparing the acoustical measurements of the headphones to their perceived spectral balance confirms that the more preferred headphones generally have a smooth and extended response below 1 kHz that is perceived as an ideal spectral balance (slide 25). The least preferred headphones  (HP5 and HP6)   have the most uneven measured and perceived frequency responses below 1 kHz, which generated listener comments such as "colored, boomy and muffled."  The measured frequency response of HP4 shows a slight bass boost below 200 Hz, yet on average it was perceived as sounding thin; this headphone was one of the models that had bass leakage problems for some listeners due to a poor seal on their ears.

Above: The left and right channel frequency response measurements of each headphone are shown above the  mean preference rating and 95% confidence interval it received in blind listening tests. The dotted green response on each graph shows the "perceived spectral balance" based on the listeners' responses.

Conclusions

In conclusion, this headphone study is one of the first of its kind to report results based on controlled, double-blind listening tests [2]. The results provide evidence that trained listeners preferred the headphones perceived to have the most neutral, spectral balance. The acoustical measurements of the headphone generally confirmed and predicted which headphones listeners preferred. We also found that bass leakage related to the quality of fit and seal of the headphone to the listeners'  head/ears can be a significant nuisance variable in subjective and objective measurements of headphone sound quality.

It is important for the reader not to draw generalizations from these results beyond the conditions we tested. One audio writer has already questioned whether headphone sound quality preferences of trained listeners can be extrapolated to tastes of untrained younger demographics whose apparent appetite for bass-heavy headphones might indicate otherwise. We don't know the answer to this question. For younger consumers, headphone purchases may be  driven more by fashion trends and marketing B.S. (Before Science) than sound quality.  While this question is the focus of future research, the preliminary data suggests  in blind A/B comparisons kids pref headphones with accurate reproduction to colored, bass-heavy alternatives.  This would tend to confirm findings from previous investigations into loudspeaker preferences of high school and college students (both Japanese and American) that so far indicates most listeners prefer accurate  sound reproduction regardless of age, listener training or culture.

Future headphone research may tell us (or not) that most people prefer accurate sound reproduction regardless of whether the loudspeakers are installed in the living room, the automobile, or strapped onto the sides of their head.  It makes perfect sense, at least to me. Only then will listeners hear the truth --  music reproduced as the artist intended.
________________________________

Footnotes
[1] Despite the paucity of good subjective measurements on headphones there does exist some online resources where you can find objective measurements on headphones. You will be hard pressed to find a manufacturer who will supply these measurements of their products. The resources include Headroom.com, Sound & Vision Magazine, and InnerFidelity.com.  Tyll Hertsens at InnerFidelity  has a large database of frequency response measurements of headphones that clearly illustrate the lack of consensus among manufacturers on how a headphone should sound and measure. There is even a lack of consistency among different models made by the same brand.

[2]  Sadly, studies like this present one are so uncommon in our industry that Sound and Vision Magazine  recently declared this paper as the biggest audio story in 2012. Hopefully that will change sooner than later.

Behind Harman's Testing Lab

This past week I had an enjoyable time meeting well-known technology writer Robert Scoble who was visiting our Harman facilities in Northridge, CA along with his geek-in-command Sam Levine. As part of the tour, I showed  them our Reference Listening Room and Multichannel Listening Lab where we do product research and double-blind evaluations of loudspeakers. We discussed the science and philosophy behind how we design and measure the sound quality of our products.

One of the topics of discussion was my recent research that explores whether high school and college students from USA and Japan have different tastes and preferences in the quality of reproduced sound compared to older trained listeners.  We talked about differences in the tastes and performances of trained versus untrained listeners, and how Harman is able to accurately predict  subjective preference ratings of loudspeakers based on a predictive model that analyzes a set of comprehensive anechoic measurements.

After running Robert and Sam through a few trials of listener training using  our software "How to Listen", I decided to put them through a couple of double-blind listening test trials to see if they had the right stuff. They compared four different brands of floor-standing loudspeakers located  behind an acoustically transparent, visually opaque curtain where each loudspeaker is shuffled into the same position via an automated speaker shuffler. All of our tests are conducted double-blind because we have found that even trained listeners are influenced by nuisance variables such as brand, price, size, etc.

 In these tests Robert and Sam heard the same four loudspeakers that have been evaluated previously by hundreds of untrained listeners including young, old, American, Asian, and European listeners, whose preferences and performances were compared to those of our panel of trained listeners. From these tests, we have found evidence that most listeners prefer the most accurate, neutral loudspeaker regardless of age, culture or listening experience.

When the listening trials were done, the curtain went up, and Robert and Sam were surprised to discover their favorite choice was the most accurate loudspeaker which was the least expensive. The science works.  One of the speakers Robert didn't like was a model that he actually owned: it had excessive amounts of treble and upper bass, which I'm told is mandated by the manufacturer's marketing department who believe that "boom and tizz" are what their customers want. Luckily, I haven't met many of their customers, yet. Robert, then surprised me by turning on his camera doing an impromptu interview, which hopefully you'll enjoy. If you want to learn more about the engineering process and tools behind designing a speaker, check out the interview with one of our speaker engineering stars, Charles Sprinkle.

In my next blog posting I hope to discuss some of the exciting research we've been doing on the relationship between the perception and measurement of headphone sound quality. The goal is to develop the same science for measuring and predicting the sound quality of headphones that we've found useful for designing good sounding loudspeakers.  Stay tuned!

More Evidence that Kids (American and Japanese) Prefer Accurate Sound Reproduction

Geoffrey Morrison, an audio writer at CNET and Sound & Vision has posted a nice summary  of my latest AES paper "Some New Evidence that Teenager and College Students May Prefer Accurate Sound Reproduction" presented at the recent  132nd AES Convention in Budapest, Hungary.


The paper is available for download here at the  AES E-library, and I have provided a YouTube video and a PDF of my presentation slides that summarize the main points of the research.


 The abstract of the paper reads as follows:


A group of 58  high school and college students with different expertise in sound evaluation participated in two separate controlled listening tests that measured their preference choices between music reproduced in (1) MP3 (128 kbp/s) and lossless CD-quality file formats, and (2) music reproduced through four different consumer loudspeakers. As a group, the students preferred the CD-quality reproduction in 70% of the trials and preferred music reproduced through the most accurate, neutral loudspeaker. Critical listening experience was a significant factor in the listeners’ performance and preferences. Together, these tests provide some new evidence that both teenagers and college students can discern and appreciate a better quality of reproduced sound when given the opportunity to directly compare it against lower quality options. 


The effects of culture and trained versus untrained listeners on loudspeaker preference are topics that have been discussed in previous postings on Audio Musings. To further shed some light on this topic, I also ran 149  native speaking Japanese college students through the same loudspeaker preference test along with 12 Harman trained listeners.  The graph below shows the mean loudspeaker preference ratings for these two groups of listeners along with the four different groups of high school and college students from Los Angeles.  

Not surprising, (at least to me) I found that the Japanese college students on average preferred the same accurate loudspeaker (A) as did the 58  Los Angeles students, and the trained Harman listening panel. The main differences among the different listening groups  were related to the effect of prior critical listening experience:  the more trained listeners simply rated the loudspeakers lower on the preference scale, and were more discriminating and consistent in their responses. This result is consistent with previous studies. The least preferred and least accurate loudspeaker (Loudspeaker D) generated the most variance in ratings among the different listening groups. This  was explained by its highly directional behavior combined with its inconsistent frequency response as you move from on-axis to off-axis seating positions. This meant that listeners sitting off-axis heard a much different (and apparently better quality) sound than those listeners  sitting on-axis.


 While the small sample size of listeners doesn't allow us to make generalizations to larger populations, nonetheless it is reassuring  to find that  both the American and Japanese students, regardless of their critical listening experience, recognized good sound when they heard it, and preferred it to the lower quality options.


It would appear that the reason kids don't own better sounding audio solutions has nothing to do with their supposed "deviant"  tastes in sound quality, but more do with  other factors  (e.g. price, convenience, portability, marketing, fashion) that have nothing to do with sound quality.  Music and audio companies should take notice that kids can indeed discriminate between good and bad sound, and prefer accurate sound, despite what the media has been falsely reporting for the last few years. With that out of the way, we should focus on figuring out how to sell sound quality to kids at affordable prices and form factors  they desire to own.


The research suggests that if we cannot figure out how to sell better sound to kids, we have no one to blame but ourselves. 

Harman Science of Sound Demonstrations at Rocky Mountain Audio Fest 2011

October 14-16, I will be  giving Science of Sound presentations for the Harman Luxury Audio Group (room #8020)   at the Rocky Mountain Audio Fest (RMAF) in Denver, CO. My demonstration will be repeated every 1/2 hour on the hour and half-hour.

Drop by and find out more about the science behind Harman audio product development and testing including JBL and Revel loudspeakers. I will be demonstrating our latest release of the "How to Listen"  software used for training and selecting listeners for product research and testing. Find out how discriminating and reliable you are as a critical listener.

Attendees will be given 30% discount coupons towards a copy of Floyd Toole's book "Sound Reproduction" (Focal Press), a book that describes much of the current scientific knowledge and perception of  the sound quality of loudspeakers, listening rooms, and their acoustical interaction with each other.  I will be raffling off a few copies to the best performing listeners.

I hope to see you there!

Topics Related to Perception and Measurement of Reproduced Sound

On Tuesday, April 26th 2011, I will be giving a presentation at the meeting of the Los Angeles AES Chapter on several topics related to recent audio research at Harman International. The topics include:

• Preference in Quality of Reproduced Sound Among Generation Y
• Are There Cross-Cultural Preferences in Quality of Reproduced Sound?
• The Subjective and Objective Evaluation of Room Correction Products
• Harman's How to Listen: A Software program for Training Listeners

I've briefly discussed these topics in Audio Musings over the past few months, and you can find summaries of them by clicking on the links above. I'll be giving an update on new findings, and briefly touch on topics not mentioned above. As a door prize, Harman will donate a free copy of Dr. Floyd Toole's book Sound Reproduction (shown on the right side bar) autographed by the author of the book.

AES members and nonmember guests are welcome to attend. The meeting will be held at the Sportmen's Lodge in Studio City. More details can be found at the Los Angeles AES website.