Avoid blind spots that derail experiences
by Danielle Kost
Should companies let employees continue to work remotely after the COVID-19 pandemic ends? Evaluating the impact of remote working has involved a lot of guesswork for business leaders, but it doesn’t have to be.
Take the example of the Chinese travel agency Ctrip, now known as Trip.com. Its leaders conducted an experiment nearly a decade ago in which the company allowed a subset of randomly selected employees to work from home for nine months while the rest of the team stayed in the office. . It turned out that employees who worked from home became much more productive. This effort gave the company a critical data point and a unique head start in thinking about the problem.
CTrip is one of many companies that use experiences to guide their decisions, according to Michael Luca, professor at Harvard Business School.
“Executives need to understand when and how to integrate data and experiences into their organization,” says Luca, Lee J. Styslinger III associate professor of business administration. “They can’t just rely on data scientists. “
The experiments have ended up having a disproportionate influence within tech companies from Uber to Zillow, who test everything from new products and algorithms to site functionality to see what resonates with users online. Google and Amazon are part of a growing list of companies that perform tens of thousands of tests per year. A growing number of companies are applying these techniques to questions of marketing, strategy and even compensation.
However, the use of experiences remains uneven, even data-driven companies are still looking to make the most of them. After watching multiple tech companies make the same predictable set of strategic mistakes, Luca decided to examine how companies could more effectively leverage structured and random experiences to inform decision-making. Its goal: to enable C suite executives to avoid blind spots and errors that can arise when making decisions based on large-scale data and testing.
The project resulted in a new experiences course for second year MBA students and in his 2020 book, The Power of Experiments: Decision-Making in a Data-Driven World, co-authored with Max H. Bazerman, the professor Jesse Isidor Straus. business administration at HBS.
Five experimental mistakes to avoid
Thoughtful testing can give managers valuable insight, but there are potential pitfalls, Luca explains. They understand:
Do not link experiences to managerial decision-making. Some organizations tend to relegate testing to specialist teams without deep managerial involvement, creating silos that ultimately prevent experimentation from permeating company culture. Instead, organizations could learn from travel site Booking.com, whose leaders have forged a culture of rigorous testing that has guided the company to growth, says Luca.
The company has enabled 1,500 employees with varying backgrounds and roles to test in virtually every area of the business. Booking.com President Gillian Tans said the agility that comes with a culture of testing helps companies engage their employees and develop better products.
Set a goal that is too narrow. Managers can create major blind spots when they focus on too narrow goals. Consider the experience of Airbnb, which conducted experiments to assess booking strategies but ignored the potential for discrimination. After Luca and his colleagues’ research found evidence of widespread discrimination on the platform and proposed a way forward for the company, Airbnb made a series of changes and created a process for analyzing bias. If leaders had thought about how to factor unintended consequences into their outcome metrics – and perhaps hired a diverse user group to understand their experiences – they might have spotted this problem sooner, an oversight with significant reputational and legal costs.
Aim for too short a time horizon. Managers sometimes have to run experiments for weeks or months to fully understand their short- and long-term results, says Luca. It was the 24 Hour Fitness experience. A few years ago, the gym chain tapped the University of Pennsylvania’s Behavior Change for Good initiative to test new ways to get members to exercise. The researchers found that while all of their interventions resulted in an initial increase in activity, many of those gains faded over time. Monitoring longer-term measures enabled them to identify interventions that had lasted.
Too much focus on “what” instead of “why”. Companies often try to test a particular question or compare a short list of options in an attempt to find what works best with customers. However, too many of them stop when they have an answer, rather than asking more questions about the outcome. Alibaba, for example, set out a few years ago to test whether coupons entice shoppers to purchase items they had left in their digital carts. Executives found that these incentives didn’t boost sales, but rather than finding out what tactics could actually move the needle, the company moved on.
Fail to align experiences with broader organizational goals. Not understanding the end goal can lead to wrong decisions about when and how to run such tests. Experiments should answer clear questions that bring an organization closer to its goals. Does the company ultimately hope to show the value of a product to investors? Is he trying to decide which product to introduce first to make the most noise with customers? Would offering a bonus increase employee performance differently during a difficult part of the production cycle?
New opportunities for experimentation
While much of Luca’s work focuses on the tech industry, he sees valuable testing opportunities in a variety of business contexts. Here are some of the many questions that experiments can help answer:
How do employees appreciate flexible hours versus higher pay? As companies reconsider their post-pandemic work standards, many executives will weigh the pros and cons of letting employees retain the schedule freedom that comes with virtual work. But, downsizing comes with its own risks, so companies will likely want a full picture of the potential benefits of such a shift to employees and the organization.
A 2016 study by Alex Mas of Princeton and Amanda Pallais of Harvard University offers some interesting insight. They found that the average candidate for a job at a large call center was willing to take a big pay cut to get a predictable schedule. However, two-thirds of the applicants were unwilling to forgo a salary to better control their hours.
While the results of this experiment and CTrip’s experience can be invaluable, companies can also test policy proposals on a subset of employees, offering bonuses to those who volunteer to be on call, for example, to see if it’s worth adopting, says Luca.
How do higher wages affect productivity? Many companies pay above market wages on the premise that employees will work harder if they make more money. However, studies that have probed this hypothesis have yielded mixed results. One article found that an unconditional bonus before a project started increased productivity, while another study concluded that these benefits were short-lived.
A study by Luca, former HBS doctoral student Duncan Gilchrist, and Deepak Malhotra, Eli Goldston professor of business administration, found that two groups of contract workers, paid $ 3 or $ 4 an hour, delivered the same result in a data entry task. However, a third group paid $ 3 an hour plus an unexpected pay rise of $ 1 were 20% more productive.
An experiment (pdf) conducted by researchers at Harvard, the University of Chicago and the University of California at San Diego also found that the way employers design bonuses played an important role in their effectiveness. In the study, teachers who received bonuses at the start of the school year – money they had to repay if they did not meet achievement goals – significantly outperformed educators who received the bonuses. same bonuses at the end of the year.
Companies can conduct similar experiments to refine their compensation strategies. They can experiment with compensation at different sites and track the impact of employee productivity. They can change the way they communicate about pay (as in the teacher study) or test whether highlighting the reason for a bonus improves performance.
An essential skill for executives
As Luca and Bazerman note in The power of experiences, companies are still looking to create a culture of experimentation and to better understand when and how to leverage these efforts.
Too often, leaders rely on intuition, flawed assumptions, or their own experiences to make decisions without looking at the data, Luca explains. While data has grown in importance in many organizations, numbers can only take a business so far if senior executives don’t have the tools to harness them.
In his course and book, Luca aims to instill a scientific mindset in students so that they can confidently apply these approaches throughout their careers. After all, a thorough understanding of experimentation, both design and analysis, are key skills in today’s data-driven economy and a significant source of competitive advantage for businesses. companies.
“Business leaders need both leadership and good judgment to effectively leverage data and experiences to guide their biggest decisions,” says Luca.