Thursday, December 9, 2010

How to Listen: A Course on How to Critically Evaluate the Quality of Recorded and Reproduced Sound

Next month, I will be giving a one day course on How to Listen at the 2011 ALMA Winter Symposium in Las Vegas,held Jan. 4th and 5th, just prior to the CES show. The symposium will also feature other courses, workshops and paper sessions on loudspeaker and headphone design, testing and evaluation. You can register for my course and other events at the ALMA Symposium here. Below is a brief preface for my course How to Listen, which I encourage you to attend.


Figure 1: A listener training in the Harman International Reference Listening Room using the original version of the How to Listen training software.


Whether you are involved in the mixing of live and recorded sound, the design and calibration of sound systems, or just shopping for a new audio system, the question “Does it sound good?” is usually foremost on your mind. With sufficient prodding, most people can offer an opinion on the overall sound quality of a recording and its reproduction. Beyond that, listeners generally lack the necessary experience, training and vocabulary to describe which specific aspects of the sound they like and dislike. Sadly, the audio industry has no standardized terminology that allows musicians, audio engineers and audiophiles to communicate with each other about sound quality in a concise and meaningful way. Courses in critical listening are not commonly available in audio programs at universities, and are even less available to the general public. In summary, there is a real need for a comprehensive course that teaches audio enthusiasts how to critically evaluate sound quality.

A Scientific Approach Towards Training Listeners
To address this need, the author has developed a critical listening course called How to Listen. The course aims to teach students how to evaluate sound quality using percepts well established in the auditory perception field. These sound quality percepts are taught and demonstrated in a controlled way using real-time processing of recorded sounds. This has two benefits. First, the intensity of each attribute can be adjusted according to the aptitude and performance of the listener. Second, closely tying the physical properties of the stimulus to its perception and evaluation (a science known as psychoacoustics) there is theoretical basis behind the training approach. For example, the listener training data can be used to better understand how we perceive sound quality, which physical aspects of sound matter most of its perceived quality, and possibly identify the important underlying sound quality attributes that influence our preferences. Critical listening is treated as a science, rather than the black art it currently is.
How to Listen also includes classroom topics in the fundamentals of human auditory perception, sound quality research in variables that significantly influence the quality of recorded and reproduced sound (e.g. loudspeakers, rooms, recordings, microphones) and a brief tutorial in how to conduct sound quality listening tests that produce accurate, reliable and valid results.
But before we get too far ahead of ourselves, there must be good reasons for training listeners since it requires an investment in time and resources. There is also the question of external validity: Can the sound quality preferences of trained listeners be extrapolated to the preferences of untrained listeners, and does this hold true across different cultures? These questions will be answered in the following sections.
Why Train Listeners?
There are several compelling reasons for training listeners. First, trained listeners have been shown to produce more discriminating and reliable judgment of sound quality than untrained listeners [1]. Fewer listener can be used to achieve a similar level of statistical confidence, which can result in savings in time and money. For example, a panel of 15 trained listeners can provide sound quality ratings with reliable statistical confidence in less than 8 hours. To achieve a similar level of confidence using untrained listeners would require about 10 times more listeners, 10 times more days to complete the testing, and cost 10 times more money to pay the listeners and staff conducting the tests. If the study is conducted by an independent research firm using 200-300 untrained listeners, the cost can easily exceed $100k.
A second reason for training listeners is that they are able to report precisely what they like and dislike about the sound quality using well-defined, meaningful terms. This feedback can provide important guidance for reengineering the product for optimal sound quality.
Besides training listeners for product research, there are benefits in training audio marketing and sales people to become better critical listeners. Training makes them better equipped to communicate sound quality issues to audio engineers and customers. As audio companies expand sales and operations in China, India, and other developing countries, there is a growing need to develop a common cross-cultural understanding as to what constitutes good sound and unacceptable sound.
Does Training Bias Listeners?
An important question is whether the training process itself biases the sound quality preferences of listeners. If the trained listener preferences are different from those of the targeted demographic, there is a danger the product may not be well received in the marketplace. This raises the age old question, “Is preference in sound quality a matter of personal taste - much like food, wine and music - or is it universal?”
To study this question, the author compared the performances and loudspeaker preferences of trained listeners versus untrained listeners [1]. Over 300 untrained listeners were tested over a period of 18 months where they compared four different loudspeakers under controlled, double-blind listening conditions. Their preferences were then compared to the preferences of the trained Harman listening panel.
The results, plotted in Figure 2, show that the rank ordering of the loudspeaker preferences were the same for both the trained and untrained listeners. There were two main differences in how the two groups of listeners responded. First, the trained listeners tended to give lower loudspeaker ratings overall. Second, the trained listeners distinguished themselves from the untrained listeners by generally giving more discriminating and consistent loudspeaker preference ratings.



Figure 2: The mean loudspeaker preference ratings and 95% confidence intervals are shown for four loudspeakers evaluated in a controlled, double-blind listening test. The results of different groups of untrained listeners are compared to those of the 12 Harman listeners.

Relative Performances of Trained Versus Untrained Listeners
A common performance metric used to quantify the listener’s discrimination and consistency in rating sound quality is the F-statistic. This calculation is done by performing an analysis of variance (ANOVA) on the main variable being tested. In the above study [1], the performances of trained versus untrained listeners were compared by calculating the loudspeaker F-statistic for each individual listener.

Figure 3 shows the relative performance of different groups of untrained listeners based on their mean F-statistics compared to the F-statistics of the trained listeners. The relative performances of the untrained groups were: audio retailers (35%), audio reviewers (20%), audio marketing/sales staff (10%), and college students (4%). The poor performance of the students was explained by their tendency to give all four loudspeakers very similar and high ratings. A likely explanation for this was that they experienced a level of sound quality that was much higher than their everyday common experience: compressed MP3 music reproduced through headphones. The good news is that the students seemed to appreciate the higher fidelity sound based on the high ratings. In time, they will hopefully seek out better quality audio systems.



Figure 3: The relative performance of different groups of untrained listeners compared to the trained Harman listeners. Performance is based on the group’s average loudspeaker F-statistic which represents their ability to give discriminating and consistent preference ratings.

Are There Cross-Cultural Preferences in Sound Quality?
One of the oldest controversies in audio is the notion that different cultures or geographical regions of the world have different sound quality preferences [see reference 2]. For example, it is often claimed that Japanese listeners have different loudspeaker preferences than Americans due to differences in language, music, cultural practices and norms, and the acoustics of their homes. So far, very little formal research has done on this subject. In some preliminary studies, the author has found no significant differences in sound quality preferences for loudspeakers and automotive audio systems among Chinese, Japanese and American listeners.
How to Listen: A New Listener Training Software Application
Research has found most sound quality percepts fall under the attribute categories of timbre, spatial, dynamic or related nonlinear distortion. Within these four attributes there are additional sub-attributes that describe more specific sonic characteristics of the attribute. For example, Bright-Dull and Full-Thin are timbre sub-attributes related to the relative emphasis and de-emphasis of high and low frequencies, respectively. Sub-attributes for spatial quality deal with the location and width of the auditory image(s), and the perceived sense of spaciousness or envelopment. Distortion sub-attributes include the presence of noise, hum, audible clipping and distortions specific to the audio device(s) under test.
How to Listen focuses on teaching listeners to evaluate sound quality differences based on these four attributes and their sub-attributes (see Figure 4). While listening to music recordings, one or more attributes are manipulated in a controlled way so that listeners recognize and report the magnitude of these changes using the appropriate terms and scales. An analogy to this would the Wine Aroma Wheel where expert wine tasters are trained to identify the intensities of different aroma-flavors perceived in the wine.



Figure 4: A list of the 17 different training tasks that focus on one or more of the four sound quality attributes: spectral (timbral), spatial, distortion and dynamics.


To facilitate the training process, a proprietary computer-based software program called
“How to Listen” was developed by Harman software engineers Sean Hess and Eric Hu. The software runs on both Mac and PC computers, and can play both stereo and multichannel music files. A real-time DSP engine built into the software application allows real-time manipulation of sound quality attributes in response to the listeners’ responses and performance.
There are currently five different types of training tasks that focus on one or more sound quality attributes (see Figure 4):
  1. Band Identification
  2. Spectral Plot
  3. Spatial Mapping
  4. Attribute Test
  5. Preference Test
Band Identification (see Figure 5) teaches listeners to identify spectral distortions based on their frequency, level and Q-factor using combinations of peak/dip and highpass/lowpass filters. In each trial, the listener compares the unequalized version of the music track (FLAT) to a version that has been equalized (EQ) using one of the filters drawn on the screen. The listener must select the correct filter (Filter 1 or 2) they believe has been applied to the equalized version.




Figure 5: A screen capture of the listener training task “Band Identification” in Harman’s “How to Listen” training software. The listener compares the unequalized music “Flat” to an equalized version (EQ) and must select the EQ filter that is associated with its sound.


The difficulty of each training task automatically increases or decreases based on the listener’s performance. The listener is give immediate feedback on their responses, and they can audition all possible response choices when they enter an incorrect response.

The training task Spectral Plot requires the listener to compare different music programs that have been equalized a number of different ways. The listener must select the equalization curve that best matches its sound quality. This task teaches listeners to behave like human spectrum analyzers. Once fully trained, the listener can draw a graph of the audio system’s frequency response based on how it sounds.
The Spatial Mapping task requires the listener to graphically indicate on a two-dimensional map where a sound appears in the listening space. The Attribute training task requires the listener to correctly rank order two or more sounds on a given attribute scale based on the intensity of the attribute (e.g. bright-dull). For the Preference task, the listener must give preference ratings where the sound quality of the music has been modified for one or more sound quality attributes. The performance of the listener is calculated based on a statistical post-hoc test that determines the discrimination and reliably of the listeners’ preference ratings. Together, these different training tasks teach listeners to critically evaluate any type of sound quality variation they are likely to encounter when listening to recorded and reproduced sound.

Conclusions
The evaluation of sound quality remains an elusive art in the audio industry. Better awareness, understanding, and appreciation of sound quality may be possible if there existed a method to teach listeners how to evaluate the quality of reproduced sound and report what they hear using well-defined and meaningful terminology. How to Listen is a listener training course that aims to achieve those goals. Listeners are taught to identify and rate audible changes to different sound quality percepts related to the spectral, spatial, dynamic and distortion qualities of recorded music. Performance metrics based on the discrimination, accuracy and reliably of the listeners’ responses are factored into whether the listener meets the criterion of being a “trained” listener. The question of whether a listener is truly golden eared or not, is no longer a matter of conjecture and debate since How to Listen will ultimately reveal the true answer.
References
[1] Sean E. Olive, "Differences in Performance and Preference of Trained Versus Untrained Listeners in Loudspeaker Tests: A Case Study," J. AES, Vol. 51, issue 9, pp. 806-825, September 2003. Download for free here, courtesy of Harman International.
[2] Sean E. Olive, “Are There Cross-Cultural Preferences in the Quality of Reproduced Sound?” Audio Musings, July 2, 2010.here.

33 comments:

  1. Can you, pretty please, share the software?

    ReplyDelete
  2. We will be publishing the How to Listen software for the general public this month. I will post the release and link on my blog when it is ready.
    Cheers
    Sean

    ReplyDelete
  3. Spatial mapping: couldn't localisation blur be a problem? Is there any research relating to localisation blur for phantom images?

    ReplyDelete
  4. wish i could atend

    i have been waiting for this program for a long time

    good to see it is going to happen

    Valentin

    ReplyDelete
  5. Great news Sean! Thanks! This could have tremendous impact on the way the audiophiles judge sound quality. But, somehow, I doubt it will be recognized.

    Best regards,
    Vuki

    ReplyDelete
  6. Thank you for a great article and for the upcoming software!

    ReplyDelete
  7. Sean. I have a question that is a bit off the subject but has importance to me. Do you happen to know who made for Harman Listening Room chairs? They are surprisingly properly made and I would like to find who did them. Rgs, Romy the Cat

    ReplyDelete
  8. Hi Anonymous:

    Glad you like the chairs. They are leather"hotel" chairs that we purchased at Plummer's Furniture for around $600 each. They have a small form factor (good for getting many chairs in sweet spot), and a low back so sounds can arrive from the sides/rear speakers and walls without being blocked by the back of the chair.

    ReplyDelete
  9. Vuki,

    Thanks. I hope it has a tremendous impact.I'm not bothered if the impact is not recognized by anyone other than the users who benefit from the training.
    Cheers
    Sean

    ReplyDelete
  10. Yes, I do like the chairs a lot. The low back part is less relevant, there are a lot of them with low back but it look like your chair allow body and head to maintain vertical position, is it true? It is what the old CinemaTech’s Embassy Chair did, loved and it looks like yours is very similar. Also, I did look through the Plummer's catalog and I do not find your chair. Would you be so kind to post a link or send me email with the link? Rgs, Romy the Cat

    ReplyDelete
  11. Dr. Olive, I am glad to see that you consider spatial mapping a valuable audio evaluation tool. I note that you propose a two dimensional mapping method. A three dimensional mapping method was proposed here:

    onlinelibrary.wiley.com/doi/10.1111/j.1745-459X.2010.00306.x/abstract

    Since a stereophonic sound stage is meant to be a representation of a live musical performance stage which has width and depth and height, I think it is appropriate to evaluate the height rendering performance of stereophonic equipment. Your thoughts?

    To address a previous commenter's question on localization blur, this phenomenon can be mediated with appropriate listener training, proper room acoustics, well recorded stereophonic program material and a stereophonic audio system with very good resolution (low noise characteristics).

    Regards,

    Raife

    ReplyDelete
  12. Sean,

    I am so glad to hear the listener training software will finally get released. I really look forward to being a "better listener". Great work you are doing and much appreciated.

    ReplyDelete
  13. When I was an employee at Lexicon/Levinson/Revel, and before I had any listening training, I had the opportunity to take the blind listening test at the Multichannel Listening Lab in Northridge, along with a small group of high-end salesmen and industry writers. I did just as well as most of them and better than some. Later, I used your "So You Think You Want to be a Harman Listener" software. I can just imagine how well I would have done in the blind listening test after that! I'm looking forward to the new software.

    Mabray

    ReplyDelete
  14. I assume the software is now available? Any link?
    Thanks

    ReplyDelete
  15. I'm checking for THE update regularly.
    Long month...

    ;)

    ReplyDelete
  16. I am looking forward to improving my critical listening abilities :) Any updates on when the software will be available? Thank you!

    Keep up the excellent work and awesome experiments!

    -Matt

    ReplyDelete
  17. Hi Dr. Olive,

    I attended the Univ of Michigan presentation and you had mentioned that you are planning on bringing the the software up to the surface for anyone to download and begin their educational process for training their listening skills. Has this taken place and I missed it or is it still working up to escape velocity?

    ReplyDelete
  18. Hello Sean,

    I was wondering if the 'How to listen software' methodology has been published in any recent journals?

    Thumbs up for the great work of Harman's R&D !

    Cheers

    Neo

    ReplyDelete
  19. Hi Neo,
    No, I have not written anything about the "How to Listen" methodology in any recent journals. It's too early for that but eventually I may get around to is once we have more experience and data using it.
    Cheers
    Sean

    ReplyDelete
  20. (-:
    http://harmanhowtolisten.blogspot.com/

    ReplyDelete
  21. Time for an update here, Sean!

    http://harmanhowtolisten.blogspot.com/

    ReplyDelete
  22. Hope that the red tape doesn't make this nice software not to make it out to some interested followers of good listening and good speakers.

    hope too see it soon

    Valentin

    ReplyDelete
  23. "We will be publishing the How to Listen software for the general public this month...December 9, 2010 3:17 PM" - Dr. Sean Olive


    It is March 2011. Do you need help getting the software published? :-)

    ReplyDelete
  24. We have posted a beta version of How to Listen at: http://www.HarmanHowtoListen.blogspot.com. We will be updating the software this week with some bug fixes and include 24-bit/96k program material.
    Cheers
    Sean

    ReplyDelete
  25. Hi Sean,

    bravo for the software! I tried it yesterday on a cheap PC speakers - it's a great toy and learning device.

    Best regards,
    Vuki

    ReplyDelete
  26. Hi, is there a German translation from ----

    "Differences in Performance and Preference of
    Trained versus Untrained Listeners in
    Loudspeaker Tests: A Case Study"

    --- anywhere to download.
    hifimuseum.de
    Gert

    ReplyDelete
  27. Dear Dr. Sean Olive:


    FYI about polarity at the following link:  http://www.ultrabitplatinum.com/?page_id=725   .  If you find that interesting, I can send you a text only version sans commercial references.  You may also find my think piece about polarity at http://www.ultrabitplatinum.com/?page_id=88 interesting as well, or not. 

    If you'd help me sort out the mistakes, when and if they're corrected, you just might become an instant overnight music-lovers' folk hero.  Or on the other hand be tarred and feathered and have your ears cut off and served up to you on a silver platter for your last meal as an audiophile guru.  I invite you to visit so that I can take you out for a meal on me and then we can share some of your music and mine on my custom built (by me) audio system on a CD player with a digital domain remote controlled polarity switch.  My system is the best I've heard so far for discerning polarity which still must be done by ear.  Those who hear CDs played over my system don't usually have much trouble hearing polarity.  But whatever happens it will be just between the two of us, if that's what you prefer.  It'll be attorney client privilege, or Perfect Polarity Pundit privilege, you decide.

    But ultimately for the sake of the music, it's up to you to either be part of the problem or part of the solution, so that 30 years doesn't become 31.

    Besides email you may also call me 7 days a week at 619-401-9876 or toll free at 888-588-9542 between 9AM and 11:45PM Pacific Time.

    Best regards.

    George S. Louis, Esq., CEO
    Digital Systems & Solutions
    Email: George@BetterFidelity.com
    Website: www.BetterFidelity.com
    Phone: 619-401-9876
    1573 Kimberly Woods Drive
    El Cajon, CA 92020-7261

    P.S.  How difficult would it be for companies' voice menus instead of saying "Please listen to the entire announcement because our menu has changed recently before making your selection" to "Our announcement was last changed on such and such date?

    ReplyDelete
  28. FYI about polarity at the following link:  http://www.ultrabitplatinum.com/?page_id=725   .  If you find that interesting, I can send you a text only version sans commercial references.  You may also find my think piece about polarity at http://www.ultrabitplatinum.com/?page_id=88 interesting as well, or not. 

    If you'd help me sort out the mistakes, when and if they're corrected, you just might become an instant overnight music-lovers' folk hero.  Or on the other hand be tarred and feathered and have your ears cut off and served up to you on a silver platter for your last meal as an audiophile guru.  I invite you to visit so that I can take you out for a meal on me and then we can share some of your music and mine on my custom built (by me) audio system on a CD player with a digital domain remote controlled polarity switch.  My system is the best I've heard so far for discerning polarity which still must be done by ear.  Those who hear CDs played over my system don't usually have much trouble hearing polarity.  But whatever happens it will be just between the two of us, if that's what you prefer.  It'll be attorney client privilege, or Perfect Polarity Pundit privilege, you decide.

    But ultimately for the sake of the music, it's up to you to either be part of the problem or part of the solution, so that 30 years doesn't become 31.

    Besides email you may also call me 7 days a week at 619-401-9876 or toll free at 888-588-9542 between 9AM and 11:45PM Pacific Time.

    Best regards.

    George S. Louis, Esq., CEO
    Digital Systems & Solutions
    Email: George@BetterFidelity.com
    Website: www.BetterFidelity.com
    Phone: 619-401-9876
    1573 Kimberly Woods Drive
    El Cajon, CA 92020-7261

    P.S.  How difficult would it be for companies' voice menus instead of saying "Please listen to the entire announcement because our menu has changed recently before making your selection" to "Our announcement was last changed on such and such date?

    ReplyDelete
  29. I think this information and program will be of great interest to the Melbourne Audio Club. Please may I quote from this article and make a link to this blog?
    Thanks
    Dave Shaw

    ReplyDelete
  30. Dave,
    Yes, please feel free to quote this article and provide a link.
    Cheers
    Sean

    ReplyDelete
  31. I apologize for the unrelated comment, but I was not able to attend AES this year and I am very excited to read your "The Relationship between Perception and Measurement of Headphone Sound Quality" paper. Would you please let me know when you expect to publish this paper?

    Thank you,
    Rex Price

    ReplyDelete
  32. Rex,
    You can find the paper in the AES Library here: http://www.aes.org/e-lib/browse.cfm?elib=16486

    ReplyDelete
  33. Is there any plans to ever make the Harman how to listen app an app for iPhone or iPad available in the ap store ?

    ReplyDelete