A large global organization has everyone working remotely due to the pandemic, and wants to know how to best onboard summer interns and ensure they have a good experience in an organization they never set foot in. No, while we could be talking about the World Bank, this is about a study in another anonymized “large global organization”, which faced this issue in the summer of 2020. As the pandemic has accelerated the introduction of remote work in many organizations, the question of how to onboard new workers in a virtual environment is one that is likely to persist, and so a new working paper by Iavor Bojinov, Prithwiraj Choudhury, and Jacqueline Lane is likely to be of lasting interest.
The setting and experimental intervention
The setting is a global organization that has a flagship summer internship program that hires undergraduates and MBA students, and uses this as a pipeline for employment offers, with as many as 75% of the interns accepting offers to join the company at the end of their studies. So think of this as interning for a multinational consulting company. In the summer of 2020, the organization held its summer internship program as a shortened five-week long virtual internship, and was concerned that the virtual experience would result in interns missing opportunities to socialize and gain rapport with others in their intern cohort, and to receive mentoring from senior staff.
The researchers worked with them to conduct an experiment which aimed to test whether giving interns “virtual water cooler” opportunities for synchronous and informal discussions would help their performance and experience. They took the entire North American cohort of 1,370 interns in 16 program cities and randomized them into the following groups:
1. Intern-only virtual water cooler (N=218): interns were given a scheduled 30 minute weekly slot to attend via Zoom with three to four other interns in the program, allowing for live and informal discussions with other members of their cohort.
2. Intern-senior manager virtual water cooler (N=219): interns were given a scheduled 30 minute weekly slot to attend via Zoom with a senior manager and three to four other interns, again designed for live and informal discussions. The senior manager was not the direct supervisor of any of the interns in the watercooler session, and thus was not responsible for measuring their performance or deciding whether to offer them full-time employment at the end.
3. Active control group 1 – asynchronous Q&A discussion forum (N=223): interns in this group were given the opportunity each week to type and submit a question to senior management, with some of these randomly selected to be answered by a senior manager on a discussion forum. So this interaction with senior managers was more arms-length, formal, and not live.
4. Active control group 2- intern group project (N=192): interns were randomly assigned to work 30 minutes each week on an intern group challenge with three to four other interns in their division. This gave them a formal work-oriented setting with which to interact with other interns.
5. Passive control group (N=518): they received no intervention, and only the standard opportunities for networking available to all interns (such as roundtables and speaker series).
Another nice feature is that the authors also randomized the dosage of the virtual water cooler treatment by randomizing how many weeks (out of 4) that interns in these groups met.
Results
The authors consider three main outcomes:
· Likelihood of receiving a fulltime job offer at the end of the internship: Overall a remarkable 91.5% of interns received a fulltime job offer (and 85% accepted). Getting assigned to the intern-senior manager virtual water cooler (treatment 2) increases the likelihood of getting this job offer by 4.7 and 7.3 percentage points (p.p.) compared to the asynchronous Q&A (active control 1) and to the intern group project (active control 2) (but only a statistically insignificant 2 p.p. compared to the passive control). The intern-only treatment does not have significant impacts, with estimates of 2.1 p.p. compared to the asynchronous Q&A, 4.6 p.p. compared to the intern group project, and -0.4 p.p. compared to the passive control. However, you could not reject equality of the two virtual water cooler treatments.
· Job performance: the direct supervisor (who didn’t participate in any treatments) gave the intern a final performance rating on a scale of 1=outstanding to 3 = needs improvement during the last week. The point estimates suggest positive improvements in rated job performance of both treatments, but they are only significant when comparing the intern-senior manager water cooler to the intern group project group.
· Interns self-reported end of program attitudes towards the remote intern experience: overall interns in the water cooler treatments reported greater satisfaction along several measures, but again not many of the comparisons are statistically significant.
These are the average effects. But then Figure 2 from the paper breaks down the effects by two important dimensions. The first is dosage– there are bigger job offer effects for those who received more informal meetings with senior managers. The second aspect is heterogeneity according to the demographic match between the intern and the senior manager. Impacts on receiving a job offer and on performance ratings were much stronger when interns were matched with a senior manager who is of the same gender and same ethnicity – indeed the impacts seem to be largely statistically insignificant if you are matched to someone outside your gender and ethnicity.
Limitations and policy implications
This is a really interesting and relevant experiment in today’s new work environment, but I had a couple of thoughts and concerns.
1. What are the main mechanisms? Unfortunately what actually goes on during these informal discussions is unobserved, and the authors do not ask about it in their surveys. They therefore rely on the heterogeneity analysis to suggest that there could be a role model effect, and perhaps some information sharing and advice from managers who are like you. The survey finds those in closer demographic matches report greater ease in asking others for help and in getting adequate opportunities for mentoring. But the authors acknowledge they cannot rule out that the senior managers provide endorsements or other more direct help in getting job offers. The intervention seems pretty light touch (only at most 4 30-minute group discussions) so it would be great to learn more of whether there are follow-up activities that occur beyond these group meetings.
2. Multiple testing and strength of the results: with five treatment groups, and then the splits by demographic match and by dosage, there are a lot of pairwise comparisons going on. Currently there are no adjustments made for multiple hypothesis testing, and it is unclear, for example, how we should interpret a significant result compared to one of the active control groups versus a smaller and insignificant result compared to the passive control group. Likewise, the lack of significance of the intern-only virtual water coolers leads the authors to focus a lot of the discussion on the intern-senior manager group, yet for most outcomes we cannot reject equality of the two treatments.
3. Is this reinforcing existing biases? The paper made me think of the recent paper by Zoë Cullen and Ricardo Perez-Truglia on the “old boys club”, in which male employers who smoke end up taking smoking breaks with their male managers are more likely to get promoted, with this explaining about a third of the gender gap in promotions in that firm. So if senior managers are predominately white males, these virtual water coolers may serve to reinforce current gender disparities in the organization. One answer would be to conduct special efforts to link women and minorities to senior managers like them who can serve as role models and mentors, but if there are not many such managers, this ends up putting extra burden on them.
This is where having more information on the mechanisms is especially important. If senior managers offer more help or more endorsements of people like them, then efforts to overcome this bias are needed. But if the mechanism is instead a role model effect, where interns either feel more comfortable asking for advice, or are more inspired to listen to this advice, from people “like them”, then this suggests the need for both broadening the diversity of mentors, as well as offering more information of what “like them” means. If all I observe about a manager is their appearance on Zoom, I might find their gender and ethnicity the most salient aspects and not see role models elsewhere. But someone “like me” could also mean someone who also grew up in a small town, or someone who is a big fan of rugby, or someone who runs marathons, or who watches the same tv shows, etc. Thinking about ways to create connections virtually on these other dimensions may then help in increasing the effectiveness of mentoring.
As we face another summer of largely virtual work, this paper hopefully gives us a lot to think about in terms of finding ways to reach out to new people joining the organization and to ensure they have a chance for the type of informal and live discussions that take place when we are all working in person.
Join the Conversation