QoE: what does it really mean today?

By Simen K. Frostad, Chairman

Published in Broadcast Bridge, January 2016

There are some immutable laws for success in any consumer-oriented industry. Potential customers need to know the product or service exists, and they have to be persuaded that it’s desirable or necessary. They have to be satisfied that it’s affordable and represents good value, and in this respect the product has to stand comparison with any competing offerings.

It’s fairly easy to compare value in a field where there’s a lot of competition, for example in the purchase of a house or a car. And although there’s no way to road test a house before you buy it, you can certainly do that with a car - or you can read road tests and reviews, as you can with most other consumer goods. In the case of services rather than goods, there isn’t the same history of ‘road testing’ information for consumers, but increasingly customers make their experiences known on the internet, so the canny consumer is able to make a more informed choice than before.

Decision made

Once committed to a contract for a service such as phone, internet or digital media, a consumer is generally inclined to stick with the choice unless there are compelling reasons to change. The most likely reason to look elsewhere is the failure of the service to live up to expectations. Since the consumer chooses the service on the basis of the visible benefits - number of channels, type of content, bandwidth, and so on - any disappointment will probably arise from poor service quality.  Digital media businesses are very definitely competing in a market where the customer can switch quickly if not satisfied, so providers want to know when their customers feel they are getting a satisfactory service, and when they aren’t. In strict terms the only way to evaluate how customers feel about a service is by asking them, in the way that market research does. Good market research measures a lot of factors that are influential on consumer opinion and these can take in economic pressures and social trends that are nothing to do with the quality of the product or service itself.

Importance of proper metrics

For those whose responsibility it is to ensure that the service itself is of unimpeachable quality, the idea of measuring ‘consumer satisfaction’ has come to be embodied in the concept of Quality of Experience (QoE) monitoring. In the conventional wisdom QoS (Quality of Service) monitoring can provide diagnostic information on the devices in the delivery chain and the transport streams, while the QoE element is intended to provide a user-centric evaluation of the service, pointing to any faults that have an impact on the user’s experience.

The established concept of QoE was really rooted in the assumption that QoS alone could not guarantee good service quality at the consumer end. And when QoE originated – in the telecoms industry – this was definitely true. So a methodology called MOS (Mean Opinion Scoring) was deployed when telcos were developing new infrastructure or, more recently, services such as VoIP. A panel of expert assessors would listen to the service and note their evaluation of quality while key phrases such as ‘You will have to be very quiet’, ‘There was nothing to be seen’, and ‘They worshipped wooden idols’ were transmitted over the line. The experts would record their scores, assessing any impairments on a scale of 1-5, from ‘imperceptible’ to ‘very annoying’. The scores would then be averaged and weighted using the kind of statistical manipulations common in the social sciences and market research.

Tools for monitoring digital media

The same basic concept of QoE monitoring later took hold in digital media monitoring and MOS spawned ‘VideoMOS’ and ‘AudioMOS’ – criteria which brought opinion-scoring methodologies into the media arena. But media providers need to know what real users are experiencing, rather than panels of expert assessors. So MOS evaluation became ‘robotized’ and algorithms were developed to simulate those subjective reactions from ‘imperceptible’ to ‘very annoying’. The robots ‘watch’ the service and the data is fed into the algorithmic engine, which attempts to simulate the subjective reaction of the viewer, with scores for factors such as ‘jerkiness’, ‘blurriness’ and ‘blockiness’.

But subjectivity is complicated, and simulating it is an unreliable way to try to assess service quality. A human viewer watching a top-quality 1080i transmission of the Superbowl, followed on the same channel by the 1940s B&W movie Casablanca might be fully satisfied. Yet in a robotized QoE ‘subjective’ assessment based on MOS criteria, the Superbowl would score highly, while Casablanca would be marked way down for ‘blurriness’, scratches and other artefacts, for its lack of resolution and color. Distorted by results like this, the data from this kind of QoE system becomes more or less meaningless to the provider.

QoS can do more

We are now in an era where integrated end-to-end monitoring systems for QoS can do much more to ensure quality for the end user, by going far beyond mere verification that the equipment is functioning correctly, and instead giving the service provider total visibility and control over the integrity of the signal throughout the entire network. Or almost the entire network, because in the case of OTT services, the content delivery chain includes the internet.

So the ability to check the quality of the service at the point of reception by the consumer is still essential if we are to have confidence that the user experience is as good as it should be.

QoS and QoE—part of same continuum

But the traditional conception of separate domains for QoS and QoE is now out of date; with the most advanced monitoring systems today QoS and QoE are really part of the same continuum, both capable of delivering objective, empirical data without the artificial and spurious distraction of simulated ‘opinion’.

QoS and QoE data may come from different points in the delivery chain (QoE data arriving in large part from the point of consumption), and be represented in different ways to the monitoring staff, but it’s the close correlation between QoS and QoE data within the same integrated environment that allows operators to reach a higher plane in the pursuit of quality.

If your system can correlate and compare data on the factors that affect quality throughout the whole delivery chain, from ingest at the head end to viewing on the smartphone, then you have a
completely trustworthy objective set of data on which to assess – in real time – both quality of service and of experience.

And beyond that, if you want to measure subjective opinion, there’s always market research.

Bridge Technologies Reinforces Commitment to Interoperability at IBC 2017
At IBC, Bridge Technologies (Stand 1.F68) is focusing on how the company is responding to the industry requirement for maximum interoperability with ST2110 signal vendors and at the same time contributing to the AIMS, VSF, SMPTE, AES interop showcase with new analytics gear.
Bridge Technologies VB440-V Virtual Probe Wins IABM Award for Test, Quality Control & Monitoring
Bridge Technologies today announced that its innovative VB440-V virtual probe was chosen by the judges as the winner of the 2016 IABM Design & Innovation Awards in the “Test, Quality Control & Monitoring” category.
Bridge Technologies Remote Data Wall “Highly Commended” by CSI Judges
Bridge Technologies today announced that its innovative Remote Data Wall was ‘Highly Commended’ by the judges in the ‘Best monitoring or network management solution’ category of the CSI Awards at a presentation at IBC on 9 September.
Bridge Technologies Addresses New Classes Of User With Uniquely Portable Probe for Monitoring and Managing IP Networks
At IBC 2016, Bridge Technologies launched NOMAD, a unique, innovative and affordable tool for anyone tasked with managing, supporting and optimising IP networks and hybrid networks with RF signals.
Remote Data Wall (RDW) Receives IBC 2016 Innovation Award from Broadcast Beat
Remote Data Wall enables users with no special skills to create displays, extending over multiple screens in a videowall format, that deliver graphical representations of a broad range of data, significantly easing the monitoring, analysis and troubleshooting of media networks.

The OMEGA Program