I’ve worked for a while helping to test different types of small business training and consulting programs, with some of these efforts proving more successful than others (See this recent VoxDevLit update for a summary). But even when we find something that works, a big question is whether it can ever be delivered at a scale to meaningfully improve the growth and productivity of the massive number of microenterprises scattered throughout most developing countries. After all, Mexico has more than 4 million firms with 0 to 10 workers.
Using digital technology seems to offer one potential way of lowering costs and reaching more firms, and the COVID-19 pandemic accelerated the use of such technologies while making gathering for in-person training sessions not an option for a while. In a new working paper, Elwyn Davies, Peter Deffebach, Leonardo Iacovone and I report on the results of an experiment designed to test whether live training sessions by Zoom is now a viable and effective technology for teaching microentrepreneurs. In particular, we try to address three key questions:
· Is it actually feasible to train microenterprise owners by Zoom in a developing country setting?
· How much of a cost-saving does offering live classes by Zoom achieve over in-person classes?
· Is this remote training effective at improving business practices and performance?
I was both pleasantly surprised as well as disappointed, as you will see.
The Training Course and Recruiting a Sample
We partnered with the Mexican NGO Crea Comunidades de Emprendedores Sociales, which has been providing programs in Mexico since 2008 for women entrepreneurs in economically marginalized areas. They typically offer in-person training courses to women, funded by a range of government and private sector partners. With the pandemic making in-person classes not feasible, they launched a new program called Fortalece tu negocio (Strengthen your business). The program was advertised as a free online course where microenterprise owners could learn resilience, costing, prices, marketing, e-commerce, and making their business plan.
Recruitment for the program largely occurred through social media, and especially from paid advertising on Facebook, which is used by 84% of the Mexican adult population. As well as advertising in Mexico, we extended this to Guatemala. As you might expect, this approach has great reach, but low conversion rates: 3.3 million unique viewers saw the ads, 52,719 clicked through on the link, 10,700 pre-registered, and this resulted in 1,478 registering fully for the program and completing our baseline survey, costing $3.38 per person recruited. Our sample was boosted to 2,208 through a range of other recruiting efforts such as mailing lists, flyers, and telephone calls.
This approach shows one of the advantages of an online program – our sample includes businesses from across all 32 Mexican states, as well as 8% coming from Guatemala – so they are much more geographically diverse than in-person training tends to be. The businesses are all women-owned, are small in size, with only 59% having any employees, and have average monthly profits of around $100. The most common sectors are baked and prepared food, beauty, handicrafts, clothes, and accessories.
How feasible is Zoom training?
The sample was randomized into a control group of 695 businesses, and two treatment groups of 1513 businesses, where we tested two different ways of determining the topics of the training (top-down, selected by CREA, and bottom-up, selected by the participants). In practice both treatments ended up selecting similar content.
Training took place 2-3 times a week, typically in the evenings, for nine 2-hour sessions conducted over Zoom, to groups of around 20 participants at a time. Although they run small businesses, the types of women who sign up for Zoom training are on average younger and more educated than the average Mexican female entrepreneur, with an average age of 40, and 48% having some post-secondary education.
We were worried whether there would be lots of connection issues, and Zoom fatigue, and I was pleasantly surprised by the technology working pretty well overall. Only 3 out of 75 monitored sessions had to be rescheduled due to electricity cuts or hurricanes, and for the most part women were able to connect and watch on their phones without many difficulties. 80% of participants typed something in the chat. Take-up of training was pretty good overall: 80.7 percent of those assigned to treatment attended at least one session, with a mean of 5.5 sessions attended and 61.4 percent completing the course. This is in line with the average take-up rate for in-person training classes of 65 percent. So training by Zoom seems feasible.
Does it offer big cost-savings over in-person?
We had hoped that it would be considerably cheaper to offer training remotely. However, in practice the cost savings were limited, although a much broader geographic range could be covered. The estimated cost per participant trained by Zoom was $50 in Mexico, compared to $62 for in-person classes. The largest costs are personnel costs, especially the trainer, who is giving the live sessions. While these costs could be lower online if the online trainers could teach more women at the same time, in practice the difficulties in recruiting large numbers of women who all wanted to start and attend at the same time meant classes were similar in size to in-person. The costs of Zoom license fees, data plans, paid Facebook advertising and other recruitment costs were slightly smaller than the venue and transport costs for in-person training, but these account for only around one-fifth of total costs.
Is remote training effective at improving business practices and performance?
Since entrepreneurs were all recruited online from across the country, we had to do follow-up surveys by phone and online. We did follow-up surveys at 2 months and 6 months, with response rates of 72% and 73% (slightly higher for treatment than control, but results are robust to different adjustments for this).
We find:
· At 2-months, there is a small, but statistically significant, improvement in business knowledge, and a significant improvement in business practices. We measure 22 business practices, and the control group is doing 50.2 percent of these, with treatment leading to a 5.4 percentage point improvement. This is inline with short in-person trainings, and the biggest changes occur in planning and accounting practices, with no significant improvement in marketing practices.
· There is some evidence that this is accompanied by a short-term improvement in sales. Monthly sales increase by $240 in levels, a 24% increase, that is statistically significant, or 11.2% in logs, which is not statistically significant, and monthly profits by a not statistically significant 10%.
· However, none of these impacts persist at 6 months. The estimated impact on business practices has fallen to 0.8 percentage points, and the estimated impacts on sales and profits are negative in sign and not statistically different from zero.
We examine why these impacts do not persist, and document that there is considerable churn in business practices, something we had not seen studied in previous literature. Even in the control group, many firms are starting and stopping practices between rounds. We think people who applied to the training program were looking to make some changes in their business, and the recovery of the economy from the pandemic may have also spurred experimentation. Training seems to speed up some adoption, and then the control group catches up a bit. But also other practices seem to be dropped by the treatment group, with our qualitative surveys finding owners acknowledging that they had the discipline to implement new things while training was taking place, but then lost their discipline and reverted back to their old ways.
We also think it was hard for firm owners to tell whether the practices they were implementing were helping. They did not improve marketing or personal initiative practices, which are the areas which might best generate new customers and prolonged sales impacts – and then given the amount of volatility, even in our sample of over 2,000 firms, an impact of 10-13% increases in sales and profits was not statistically significant. So firm owners may give up practices if they can’t see detectable improvements from them.
For our methods-focused readers we try some modern methods to explore heterogeneity and sensitivity/robustness. We use the generic machine learning approach of Chernozhukov et al. (2020) to test whether some subgroup still benefited from training, and find no predictable heterogeneity in treatment effect. We also see whether our short-term results are being driven by a few observations by using the approximate maximum influence perturbation approach of Broderick et al. (2023). We find our business practice impacts are quite robust to selectively removing data, whereas our sales impact would change sign by dropping only 1.7% of the sample, reflecting the skewness of sales as an outcome.
Remote fieldwork and improving online delivery
We conclude with some suggestions for improving such training going forward, based on both our own observations of the training (one of the benefits to researchers of such an intervention is “remote fieldwork” can take place by observing some of the sessions ourselves), as well as from qualitative interviews. These include revamping some elements of the training, especially the marketing, to increase the number of specific actionable steps; better attempts to facilitate networking groups among training participants using Whatsapp or similar; and having the most dynamic and effective trainers teach larger classes to reduce costs and ensure the most effective trainers reach more students.
Join the Conversation