The latest issue of the World Bank Economic Review contains a paper by Alejandro Estefan, Martina Improta, Romina Ordoñez, and Paul Winters on digital training for franchise store owners in Guatemala. As well as being a nice addition to the recent literature on digital delivery of business training, the context and set-up is one with several desirable features that I think make it of particular interest for the IFC, and for researchers looking for ways to avoid having to rely on lots of surveys.
Context
The experiment comes out of a collaboration agreement between IDB Invest, the private-sector arm of the IADB, and Corporación Multi Inversiones, a multinational corporation based in Guatemala. The experiment works with Casas de Pollo Rey or “Houses of the King of Chicken”, one of the retail chains of the food sector branch of this multinational, which specializes in cooked and uncooked chicken and pork products. They operate a franchise business model throughout Guatemala, in which local entrepreneurs own and operate small-scale franchise stores. The majority of these franchises are owned and run by women, 88 percent have one or fewer full-time employees, and monthly sales have a median of US$1587, and they mostly serve local customers who shop frequently for chicken at their stores.
The Experiment and Intervention of Short Digital Training Videos
The experiment works with 498 of these franchise store owners, some of whom owned multiple stores, for a total of 539 stores. A stratified random assignment allocated these to a control group of 247 owners, and a treatment group of 251 owners.
The treatment group gets a digital training program that aims to help them upgrade their business skills. It lasts 7 weeks, and consists of 28 short video capsules of 1-7 minutes long each, a workbook with additional exercises and templates, and 3 one-to-one video meetings of 30 minutes each with a professional business consultant. Training includes topics like branding, operations, marketing, equipment maintenance, hygiene and food safety, financial and inventory management, customer satisfaction, digital payment options, etc. The corporation rewarded participants for each training module completed with digital money that could be used to purchase more chicken and pork inventories.
The sample was recruited in September 2021, Training took place October-December 2021, and a follow-up survey was collected six months after training in May-June 2022, and admin data is used to track sales through to December 2022.
This set-up has several very attractive features that make it a good context for doing an experiment.
Many units that can be independently allocated to treatment or control: experiments with a single firm face the challenge of avoiding spillovers and having enough units to experiment on. This has led to experiments in really large factories doing randomization at the production line level, or labor experiments in which some workers get different compensation or working conditions than others. There is then always the concern of spillovers or interactions between treated and control units. Here the franchises are physically separated from one another, and do not appear to interact much between themselves.
A homogenous industry and production function: one of the big challenges I’ve faced when working with government business training programs is that they typically target firms that differ a lot in size and industry. This presents two problems. The first is in designing the training content – it is difficult for training to provide sector-specific skills and for consultants to give sector-specific advice when every firm is working in a different industry. In contrast, here training can be tailored explicitly to the needs of these franchise owners. Second, lots of heterogeneity increases the variance of outcomes, making it more difficult to detect treatment impacts and to understand how the production function is getting changed. Here everyone is in the same industry, and while there is still a lot of heterogeneity and skewness in monthly sales (which range from $174 to $10,281), there is less noise coming from industry-specific shocks hitting different firms.
Administrative data on both take-up and sales: The digital training mobile app required user authentication, and then provides a log file that enables tracking of which training modules each franchise owner started, how long they spent watching them, and other measures of training take-up on a daily frequency. Even better, a key outcome of the training, store sales, can be obtained from administrative records from the franchise – the authors get monthly store-level information on the total value of sales, as well as sales quantities and sales broken down into three broad product categories. As well as tracking sales for many months before training, these records also have details on firm exits (9.6% attrited in the 9 months of the experiment, with about one-third permanently closing, and two-thirds shifting to a different (non-franchise) business).
The potential for a socially beneficial intervention that also adds value for the multinational: there seems to be a perception among some people I talk to who work with private sector clients that doing experiments would be an imposition on them, and potentially jeopardize the investment relationship. This is an illustration of how experiments can be used to test interventions that have the value to both help poorer people (the franchise owners) as well as deliver overall value to the multinational.
What did they find?
Take-up: like a lot of digital training efforts, many people start but do not finish the training. 99% completed the first video capsule, while only 50% completed the last one. 50.4% “graduated” which is defined as completing all three personalized consulting meetings and watching at least 70 percent of the video capsules in the mobile app. I see this as relatively high, and likely reflects that combination of these being short and able to be done at any time, being tailored to the firm, and being sent by the franchise owner, as well as coming with rewards for completion. While 70% have wi-fi at home, take-up is much less for those without home internet access, as well as for those with less than high school education – showing the limits of digital delivery for some groups. There is complementarity with the one-on-one consulting, as owners are more likely to watch a capsule on the same date as a consultant meeting.
Impact on sales: the authors find sales increase by 6 to 12.7 percent, or around $150-300 per month using the administrative data in the first year post training. Even with this monthly admin data and a homogenous sample, these impacts are typically only significant at the 10% level, and are not significant if one puts in zeros for the exited firms. In contrast, if the authors had instead relied on just their baseline and six-month follow-up surveys, they would get a small ($14) impact with a massive standard error ($222). So having the admin data helps a lot here, but even then measuring impacts is tough.
Is this profitable? Unfortunately while the admin data has sales (and I assume would have inventory purchases), it does not cover all expenses faced by the firms, nor have data on profits. Self-reported monthly profits in the survey increase $102 (s.e. $40). Using survey questions to try to understand how the training may have boosted sales and profits, the authors find the training seems to have led to more low-cost marketing (such as calling clients by name, building client contact lists, making flyers, taking orders by Whatsapp), as well as improved cost and inventory oversight. Under the assumption that these impacts last a year, the authors calculate that benefits are 2.3 times the costs of the training.
Last notes
I think this set-up of working in value chains or with large franchise owners, and using admin data to track key outcomes offers the potential to lower the costs of experimentation, and help better understand the constraints and solutions for the types of SMEs that are still relatively rare in developing countries. Another example of this type of work comes from several experiments run by Hugh Wu and his co-authors with a multinational spa chain in China. Of course one big concern with focusing on one particular franchise is generalizability, and so seeing a lot more of these studies with a richer array of types of franchises would be helpful in seeing how much we can learn from these cases.
A final thought from discussing this with the authors and from some of my own studies where there is the opportunity to use admin data to continuing tracking some basic outcomes is that we need to develop new innovative forms of publication that incentivize authors to provide longer-term updates on a paper without having to necessarily make it a whole new paper. Especially in cases like this where the admin data only contains a few key outcomes, but where it would be very interesting to continue to see impacts for several years afterwards. I’d love to see a 2-3 page paper format where authors can add on an updated figure and table with another year or two of data, and any discussion of changing economic circumstances and perhaps on whether the program is continuing to get used. Anyone know of other disciplines which have good models of doing this?
Join the Conversation