Friday, April 22, 2016

A Virtual Headphone Listening Test Method

Fig. 1 The Harman Headphone Virtualizer App allows listeners to make double-blind comparisons of  different headphones through a high-quality replicator headphone. The  app has two listening modes: a sighted mode (shown) and a blind mode (not shown) where listeners are not biased by non-auditory factors (brand, price, celebrity endorsement,etc). Clicking on the picture will show a larger version.

Early on in our headphone research  we realized there was a need to develop a listening test method that allowed us to conduct more controlled double-blind listening tests on different headphones.  This was necessary in order to remove tactile cues (headphone weight and clamping force), visual and psychological biases  (e.g. headphone brand, price, celebrity endorsement,etc )  from listeners' sound quality judgements of headphones.  While these factors (apart from clamping force) don't physically affect the sound of headphones, our  previous research [1]  into blind vs. sighted listening tests revealed their cognitive influence affects listeners'  loudspeaker preferences [1], often in adverse ways. In sighted tests,  listeners were also less sensitive and  discriminating compared to blind conditions when judging different loudspeakers including their interaction with different music selections and loudspeaker positions in the room. For that reason, consumers should be dubious of loudspeaker and headphone reviews that are based solely on sighted listening.

While blind loudspeakers listening tests are possible through the addition of an acoustically-transparent- visually-opaque-curtain,  there is no simple way to hide the identity of a headphone when the listener is wearing it.  In our first headphone listening tests,  the experimenter positionally substituted the different headphones onto the listener's head from behind so that the headphone could not be visually identified. However, after a couple of trials, listeners began to identify certain headphones simply by their weight and clamping force. One of the easiest headphones for listeners to identify was the Audeze LCD-2, which was considerably heavier (522 grams) and more uncomfortable than the other headphones. The test was essentially no longer blind.

To that end, a virtual headphone method was developed whereby listeners could A/B different models of headphones that were virtualized through a single pair of headphones (the replicator headphone). Details on the method and its validation were presented at the 51st Audio Engineering Society International Conference on Loudspeakers and Headphones [2] in Helsinki, Finland in 2013.  A PDF of the slide presentation can be found  here.

Headphone virtualization is done by measuring the frequency response of the different  headphones at the DRP (eardrum reference point) using a G.R.A.S. 45 AG, and then equalizing the replicator headphone to match the measured responses of the real headphones.  In this way, listeners can make instantaneous  A/B comparisons between any number of virtualized headphones through the same headphone without the visual and tactile clues biasing their judgment. More details about the method are in the slides and AES preprint.

An important questions is: "How accurate are the virtual headphones compared to the actual headphones"?  In terms of their linear acoustic performance they are quite similar. Fig. 2 compares the  measured frequency response of the actual versus virtualized headphones.  The agreement is quite good up to 8-10 kHz above which we didn't aggressively equalize the headphones because of measurement errors and large variations related to headphone positioning both on the coupler and the listeners' head.

Fig. 2 Frequency response measurements of the6  actual versus virtualized headphones made on a  GRAS 45 AG coupler with pinna. The dotted curves are based on the physical headphone and the solid curves are from the virtual (replicator) headphone.  The measurements of the right channel of the headphone (red curves) have been offset by 10 dB from the left channels (blue curve) for visual clarify. Clicking on the picture will show a larger version.

More importantly, "Do the actual and virtual headphones sound similar"? To answer this question we performed a validation experiment where listeners evaluated 6 different headphone using both standard and virtual listening methods Listeners gave both preference and spectral balance ratings in both standard and virtual tests. For headphone preference ratings the correlation between standard and virtual test results was r = 0.85. A correlation of 1 would be perfect but 85% agreement is not bad, and hopefully more accurate than headphone ratings based on sighted evaluations. 

The differences between virtual and standard test results we believe are in part due to nuisance variables that were not perfectly controlled across the two test methods. A significant nuisance variable would likely be headphone leakage that would affect the amount of bass heard depending on the fit of the headphone on the individual listener. This would have affected the results in the standard test but not the virtual one where we used an open-back headphone that largely eliminates leakage variations across listeners.  Headphone weight and tactile cues were present in the standard test but not the virtual test, and this could in part explain the differences in results.  If these two variables could be better controlled even higher accuracy can be achieved in virtual headphone listening.

Fig.3 The mean listener preference ratings and 95% confidence intervals shown for the headphones rated using the Standard and Virtual Listening Test Methods. The Standard Method listeners evaluated the actual headphones with tactile/weigh biases and any leakage effects. In the Virtual Tests, there were no visual or tactile cues about the headphones. Note: Clicking on the picture will show a larger version.

Some additional benefits from virtual headphone testing were discovered besides eliminating sighted and psychological biases: the listening tests are faster, more efficient and more sensitive. When listeners can quickly switch and compare all of the headphones in a single trial, auditory memory is less of a factor, and they are better able to discriminate among the choices. Since this paper was written in 2013, we've improved the accuracy of the virtualization in part by developing a custom pinnae for our GRAS 45 CA that better simulates the leakage effects of headphones measured on real human subjects [3].

Finally, it's important to acknowledge what the virtual headphone method doesn't capture: 1)  non-minimum phase effects (mostly occurring at higher frequencies) and 2)  non-linear distortions that are level-dependent. The effect of these two variables on virtual headphone test method have been recently tested experimentally and will be the topic of a future blog posting. Stay tuned. 


[1] Floyd Toole and Sean Olive,”Hearing is Believing vs. Believing is Hearing: Blind vs. Sighted Listening Tests, and Other Interesting Things,” presented at the 97th AES Convention, preprint 3894 (1994). Download here.

[2] Sean E. 

[3] Todd Welti, "Improved Measurement of Leakage Effects for Circum-Aural and Supra-Aural Headphones," presented at the 38th AES Convention, (May 2014). Download here.

Thursday, March 31, 2016

Harman Gives Loudspeaker Course To U of Rochester Engineering Students

Recently Mark Glazer, Principal Engineer at Harman Luxury Audio and  Revel Loudspeakers gave an invited lecture to University of Rochester Audio/Acoustic Engineering Students. The students are part of the graduate acoustic and music engineering program that is overseen by Professor Mark Bocko, Distinguished Professor, Electrical and Computer Engineering. By exposing the students to the fascinating engineering and science of loudspeakers, it is hoped the students will consider a future career in loudspeaker or audio engineering.

The 1-hour lecture gave an overview of what are the current best practices in designing a modern-day loudspeaker.   

The proof of good loudspeaker design is ultimately judged on how good it sounds. Dr. Sean Olive (me), Acoustic Research Fellow at Harman International  presented an overview of the science of evaluating loudspeakers, which included test results from a competitive benchmarking of the new Revel Concerta 2 M16 (designed by Mark Glazer) against three competitors. The results of the listening test results were generally predictable based on the set of anechoic measurements made of the different loudspeakers.

Following the lecture, we got a tour of the University's engineering facilities, which include some impressive 3D laser scanning tools for studying the vibrational modes of loudspeakers. We heard some very novel flat-panel loudspeakers with vibrational mode control developed by the Ph.D students and Professor Bocko, followed by  presentations of research projects undertaken by the Masters and Ph.D. engineering students who are working in acoustics and audio-related research. Overall, the quality of acoustic and music research being done there is impressive. As always, Professor Bocko was a gracious host, and we look forward to a return visit (hopefully in the summer or fall months).

Mark Glazer's speaker design course slides are available here:

Saturday, November 7, 2015

Factors that Influence Listeners’ Preferred Bass and Treble Levels in Headphones

Most people would agree that headphone purchase decisions are heavily influenced by the brand and styling (size,  weight, color, quality of  materials).   But what is considered stylish and fashionable  by me is not shared by my 15-year old daughter (this week donning purple hair), and vice versa. In other words, the perceived visual aesthetic  of the headphone  is really in the the eyes and mind of the beholder, and this can vary with age, gender, culture, and other demographic category. 

But what about sound quality?  To what extent does the consumer's  age, gender, culture and prior listening experience influence their taste in headphone sound quality?  Is there a scientific basis for headphone manufacturers to design headphones that have different amounts of bass and treble aimed to satisfy the tastes of a targeted demographic group? 

To answer this question, we recently conducted a  study on factors that influence listeners’ preferred bass and treble balance in headphone sound reproduction. Using a method of adjustment a total of 249 listeners adjusted the relative treble and bass levels of a headphone that was first equalized at the eardrum reference point (DRP) to match the in-room steady-state response of a reference loudspeaker in a reference listening room. Listeners repeated the adjustment five times using three stereo music programs. The listeners included males and females from different age groups, listening experiences, and nationalities (Canada, USA, Germany and China).  The results provide evidence that the preferred bass and treble balances in headphones was influenced by several factors including program, and the listeners’ age, gender, and prior listening experience. The younger and less experienced listeners on average preferred more bass and treble in their headphones compared to the older, more experienced listeners. Female listeners on average preferred about 1 dB bass and 2 dB treble than their male counterparts. Listeners over 55 years preferred less bass and more treble than the younger listeners suggested that they were compensating for possible hearing loss that is associated with increased age.

We recently presented the results of this study at the 139th Audio Engineering Society Convention in New York City, October 29th-November 1, 2015. The paper is available for download in AES e-library. A PDF copy of the presentation can be found here. Or you can view an animated version of the presentation on Youtube.

Wednesday, October 22, 2014

The Influence of Listeners' Experience, Age and Culture on Headphone Sound Quality Preferences

At the recent 137th convention of the Audio Engineering Society we presented our latest research paper entitled, "The Influence of Listeners' Experience, Age and Culture on Headphone Sound Quality Preferences."

The paper describes some double-blind  headphone listening tests conducted in four different countries (Canada, USA, China and Germany) involving 238 listeners of different ages, gender and listening experiences. Listeners gave comparative preference ratings for three popular headphones and a new reference headphone that were virtually presented through a common replicator headphone equalized to match their measured frequency responses. In this way, biases related to headphone brand, price, visual appearance and comfort were removed from listeners’ judgment of sound quality. On average, listeners preferred the reference headphone that was based on the in-room frequency response of an accurate loudspeaker calibrated in a reference listening room. This was generally true regardless of the listener’s experience, age, gender and culture. This new evidence suggests a headphone standard based on this new target response would satisfy the tastes of most listeners. 

The paper is available for download from the AES e-library. You can also find a PDF of our presentation here or view the presentation on YouTube.

Wednesday, June 11, 2014

My Article on Headphone Sound Quality in 2014 LIS

The 2014 Loudspeaker Industry Sourcebook came out this week. In it, you can find an article I wrote called "Perceiving and Measuring Headphone Sound Quality: Do Listeners Agree on What Makes a Headphone Sound Good?"

The article is a summary of some recent published research we've conducted at Harman on the perception and measurement of headphone sound quality.

Together, these studies provide scientific evidence that when headphone brand, price, fashion, and celebrity endorsement are removed subjective evaluations, listeners generally agree on what makes a headphone sound good.

So far, this has been true regardless of users' listening training, age, or culture.  The more preferred headphones tend to have a smooth, extended frequency response that approximates an accurate loudspeaker's in-room response. This new target frequency response could provide the basis for a new and improved headphone target response. You can find more details on the research here.

Tuesday, January 28, 2014

Interview in Professional Sound: The Lack of Meaningful Loudspeaker & Headphone Specs

Last October,  I was in Toronto giving a presentation to the local AES section on the perception and measurement of headphones. After the talk, I sat down with Mike Raine from  Professional Sound for an interview. Some of what we discussed is summarized in this article called Sound Advice.

The theme of article is a recurring one that I've discussed before in this blog (see "The Science and Marketing of Sound Quality" and "What Loudspeaker Specifications are Relevant to Sound Quality?").  The bottom line is that the loudspeaker and headphone industry has utterly failed to provide consumers meaningful product specifications that indicate how truly good (or bad) the products sound. Read on to find out why.

Monday, January 6, 2014

Do Listeners Agree on What Makes a Headphone Sound Good?

This past weekend, I attended the ALMA 2014 Winter Symposium in Las Vegas where I gave a talk entitled, "The Perception and Measurement of Headphone Sound Quality: Do Listeners Agree on What Makes a Headphone Sound Good?" The presentation gives a summary of some key findings of our headphone research conducted over the past 18 months. It also includes some unpublished preliminary findings from a current study on headphone preferences of trained and untrained listeners both young and old from China, Canada, USA, and Germany. The focus of this study is to determine whether listeners from different cultures and age groups prefer the same headphone sound quality as trained listeners when the influence of headphone brand, fashion and celebrity endorsement are removed from the test. 

The abstract for my talk is reproduced below. A PDF of the slide presentation can be downloaded here:

The popularity of headphones has now exploded to produce annual worldwide sales of almost $10 billion. Premium headphones ($100+) now account for 90% of the annual revenue growth, as consumers’ audio experiences are becoming a primarily mobile one. Market research indicates sound quality is a driving factor in headphone purchases with brand and fashion also being important factors among younger consumers. Yet, ironically the science behind what makes a headphone sound good and how to measure it is poorly understood. This combined with the lack of perceptually meaningful headphone standards may explain why purchasing a headphone today is like playing Russian Roulette with your ears. The magic bullet to achieving more consistent headphone sound quality is science.

We recently conducted a series of controlled double-blind listening tests on popular headphones (both real and virtualized models) to better understand the relationship between their perceived sound quality and acoustic performance [1,3,5]  A second set of experiments measured listener preferences of different headphones equalized to different target curves responses including the recommended diffuse and free-field target curves [2].  A third set of experiments used a method of adjustment where listeners directly adjusted their preferred bass and treble levels of a headphone and loudspeaker equalized to the same in-room target response [4]. In this way, we could measure the variation in individual listeners’ taste in headphone spectral balance, and determine the extent to which the preferred headphone target response should simulate the response of an accurate loudspeaker in a reference listening room.

Together, the results of this research show that when the influence of brand, fashion and celebrity endorsement are removed from headphone tests, both trained and untrained listeners regardless of age and culture, generally agree on which headphones sound best and this correlates to their acoustical performance.

  1. Sean E. Olive and Todd Welti, "The Relationship between Perception and Measurement of Headphone Sound Quality", presented at the 133rd Audio Eng. Soc. Convention, San Francisco, USA, (October 2012).
  2. Sean E. Olive, Todd Welti and Elisabeth McMullin, "Listener Preferences For Different Headphone Target Response Curves",  presented at the 134th Audio Eng. Soc. Convention, Budapest, Hungary, (May 2013).
  3. Sean E. Olive, Todd Welti and Elisabeth McMullin, "A Virtual Headphone Listening Test Methodology", presented at the 51st Audio Eng. Soc. International Conference, Helsinki, Finland, (August 2013).
  4. Sean E. Olive, Todd Welti and Elisabeth McMullin, "Listener Preferences for In-Room Loudspeaker and Headphone Target Responses"  presented at the 135th Audio Eng. Soc. Convention, New York, USA, (October 2013).
  5. Sean E. Olive, "Do college students prefer the same headphone sound quality as trained listeners?", presented at the 4th ISEAT, Shenzhen, China, (November 2013).