Men at Work: Shhh!
This page in:
“Who can it be now?”
I turn my head around from my seat at the seminar table to see who it is this time that has interrupted the seminar speaker for the Nth time before she even got through her introductory slides: it was a man, of course.
A lot of people at econ seminars get annoyed at questions that would have been answered naturally had the audience just been patient enough to wait for, sometimes literally, another slide; the back and forth that sometimes ensues between a questioner and the speaker; and, of course, the inevitable consequence of the speaker rushing through their results because too much time has been sent on answering questions.
[People from other disciplines are often gob smacked when they attend an econ seminar, where the culture of interaction often runs adversarial. I once had a public health colleague attend a seminar I gave at Berkeley, which was completely routine. After the seminar, she came up and asked me why the audience had been so rude to me: I explained to her that I did not consider them rude (still don’t), and that a healthy exchange was part and parcel of a good econ seminar, even if sometimes disruptive. Ah, youth…]
To deal with the timing and volume of questions for the recruitment seminars this year, David McKenzie and I recently suggested to our recruitment committee that we should perhaps have a 15-minute moratorium on questions – anything other than the most urgent clarification ones. The idea is not new – I am told that BREAD has started doing this at their conferences a while back – but our recruitment committee was happy to adopt the suggestion and we have now done this for 5-6 seminars. The response, from what I can tell, has been overwhelmingly positive: I have had people come by my office to say that the number of unnecessary early questions were clearly down. My sense is that it would be reasonable to consider extending this rule beyond the recruitment season for our regular seminar series.
[Interestingly, the 15-minute moratorium seems to throw some speakers off, who, naturally, expect to be interrupted. I have notices speakers stumble, search for a transition, after a slide – clearly expecting a question at that point, based on previous experience. We have also had a speaker stop and say, upon reaching slide 5 or something, that she had never reached that slide without being interrupted before. Relax, speakers: the floor is yours. Enjoy…]
Anyway, at the first job market seminar after we adopted the 15-minute rule, I was naturally watching a bit more closely – mainly to assess compliance. But, I noticed something else instead: the first 8 questions after the moratorium came from men. It should not be news that men ask more questions at seminars, but the skew seemed extreme. So, I started keeping stats, which looked like this:
MMmMMmMMFMMMMMM
Each letter represents a question by either a male or a female participant. Small letters are for follow-up questions – almost always by the same person.
Dividing the 90-minutes seminar into three periods, I summarized the stats after that seminar for colleagues in the recruitment committee as follows:
- There was a total of 40 questions, meaning more than one interruption every other minute within the 75-minute period after the 15-minute moratorium.
- Of these 35 came from men and 5 from women
- My rough estimate of men/women in the room is about 1 to 2 (16 women and 32 men).
- That’s about 1.1 questions per man and 0.3 per woman, but my guess is that most men (about two thirds) asked no questions. Same with women…
Turns out, of course, that this phenomenon has been observed in the wild before. A colleague sent this article from the Economist after I shared the stats, highlighting the following sentences from it:
Pretty similar to our setting re: the male skew. And, again of course, this skew is not limited to econ seminars. Just this past Friday, The Letters Editors of the New York Times announced that women account for only a quarter to third of letter-to-the-editor submissions and that they want to change that (see the bottom of the page)."Men, however, were over 2.5 times more likely to pose questions to the speakers—an action that may be viewed (rightly or wrongly) as a sign of greater competence.
This male skew in question-asking was observable, however, only in those seminars in which a man asked the first question. When a woman did so, the gender split in question-asking was, on average, proportional to that of the audience.”
Intrigued by the fact that (a) there had seemingly been so many questions, but perhaps worse (b) men were more than three times more likely to ask questions, I decided to collect more data from the same seminar series. After all, this was only one seminar – was it really always like this? Here are the stats from the next two seminars, during which I collected the same data:
Seminar 2:
12:39-13:00: MMMFfMFFMmmMMFMMMM (5 F 13 M)
13:00-13:30: MmFfMmmMmMMmFMFMMFMmMmM (5 F 18 M)
13:30-14:00: MmMMMMFFMFMFMmMmMMmmFf (6F 16 M)
16 questions by women 47 by men, for a total of 63 questions. Seems less skewed, but the gender distribution was also more balanced that day, making the odds more or less as skewed as Seminar 1. A colleague noted that the likelihood of men asking follow-up questions is even more skewed – consistent with the hypothesis that men are more likely to enjoy the sound of their own voices. Notice the significantly higher number of questions and follow-ups…
Seminar 3:
12:45-13:00 – MmMmMmmMmMMMMMFMm (16 M 1 F)
13:00-13:30 - MMMMmFMmMMMmMM (13 M 1 F)
13:30-14:00 – MMMFfMmMm (7 M 2 F)
36 questions by men and only 4 by women in a room where the male/female ratio was about 2/1.
All in all, very consistent across the board. And, once you see this, it is impossible to unsee. So, what, if anything, should we be doing about this? The climate, I think, is right for culture change right now: I work in a research department, which, after years of status quo, has recently made a conscious effort to diversify its hiring, has a Lecture Series which features all women speakers, and in an institution that has been making strides to reach gender balance in its ranks, including senior management positions.
I think that the answers (or, even better, questions) need to come from women, but I have one idea for men: hush.
If every man at makes a resolution before coming to a seminar to pause and ask themselves whether they need to ask that question before raising their hand, there will be more room for others, including the speaker, to speak. Of course, people reading this at the World Bank and other workplaces will immediately identify with the phenomenon at meetings, gatherings, etc.
A few other observations, as a result of lots of colleagues stopping by my office to share their thoughts about my fun seminar stats:
- There is definitely a speaker fixed effect: Some speakers are much better at handling questions and moving on than others. But, it is not fair to expect the speakers, especially graduate students and junior colleagues, to confidently handle a large room. Chairpersons can either empower or protect the speaker as appropriate. Meaning that we may choose chairpersons not randomly, but purposefully from colleagues who can strike that delicate balance of allowing some healthy dialogue while keeping things moving and encouraging a more diverse set of voices.
-
What is the optimal number of questions at a seminar? It is true that the seminar with 60+ questions was much unrulier than the one with 40+. However, conditional on the number, some seminars seem to go swimmingly (we had one speaker finish 7 minutes early) while others are derailed (see the speaker FE point above). When I first counted the number and reported it, I thought that an interruption, on average, of every two minutes cannot possibly be optimal.
- But, first, averages can be misleading in this context (many questions are bunched, followed by a few minutes of the speaker progressing through their slide deck).
- Second, as a colleague suggested, think of the following thought experiment: suppose that you are having a long coffee with the job market candidate and she is telling you about their paper. You are so interested that you two devote 90 minutes for her to tell you everything about it. In that conversation over coffee, it is perfectly possible that you interject something every couple of minutes in the pleasant back and forth. While a seminar with a room full of different people is obviously different, it is not clear that the total number of questions per se is the problem.
If the issue is the distribution of who is asking questions, and the optimal number of total questions is hard to know, could you just set an allocation of questions/comments per person? You could set the total allocation above what you think is optimal as many people ask zero questions, but this would limit those who are abusing their share and encourage those less likely to speak up. It would also hopefully make those who are ask questions that will be answered later on hesitate.
As an economist, we all know quotas are distortionary in general. But, having said that, the market for questions is clearly male oligopolistic. How about enforcing question quotas? Say 30 questions total, 15 male and 15 female. When quota is reached for each , no more questions.
Thanks. Since this is becoming a theme and, at face value, seems like an intuitive solution, let me give my brief thoughts.
As you know economists like to argue everything ad nauseaum, meaning that such a proposal would immediately draw lots of counterproposals. The one I can think of off the top of my head is that equity is not equality and we're presumably after the first and not the second. If the talk is on trade policy and the trade folks want more questions allocated to them than me, that seems fair. Maybe, some gentleperson can "yield the remainder of their time" to some other gentleperson, causing some sort of market to trade question rights, you see where this is going.
I am sympathetic to such a concept that the chair is trying to enforce some sort of equity rule, while being reasonable (or without being draconian) about it. But, that puts a lot of pressure on the chair. Tough balancing acts...
B.
Remaining anonymous given the political sensitivity of this issue, but I honestly don't see the problem. The point of a seminar is to improve a paper and to convince an audience - that is, to advance science. We all know how to read, so a seminar is not just about one way information flow.
As noted, a private conversation about a paper with a friend is much more two sided, with many challenging questions. I have never been to a seminar which came close to covering the issues the crowd was actually skeptical about, and everyone has seen the "talk seems to go well but in the elevator back, turns out everyone hated the paper because X, Y, Z.". As a *speaker*, I want as much feedback as possible - I already know what it is the paper!
Of course rudeness is not acceptable. But, if anything, I quibble with the audience members who, out of a desire for social niceties, do not actually contribute to improving the paper or diving deeper into points related to their area of expertise. We are trying to do science, not be therapists.
Thanks: it seems that you don't see a problem with men being more than three times as likely to engage at a seminar than women. Perhaps, it's just revealed preference...
Like I suggested, self-reflection before commenting...
In grad school at development brown bags, we had the following rule of thumb: you are allowed to ask a number of questions equal to your year in the program (I think we made the rule after I asked too many questions as an overly excited second year). I found this useful because it made the opportunity cost of question asking salient to ME, as the question asker. I had to ask myself: "Is this idea really good enough to spend one of my chits on it?"
It seems that the men at your seminar series have a rule of thumb of asking one question per seminar, while the women ask at every other seminar. Perhaps the men should think of giving themselves one chit/question per two seminars.
Hi Jessica,
Thanks - variants of this are coming up with some regularity, which seems fair and natural for a bunch of economists (to come up with a price). However, I would like to clarify one thing: conditional on asking a question, the mean number of questions is roughly 3 (or more) for men and 1 for women.
In grad school we tried a variant on this during econ lectures, having observed 'airtime' as commons problem; a key difference being that in the event of a person resource (or group's pooled resources) having been exhausted, a question could still be posed but someone else in the room would have to 'pay' for it to be answered. This is a slightly different setting where the purpose is to make sure the whole class is getting something from the lecture and questions for the sake of one individual taking up airtime may be a misallocation of resources. But worth considering what are the multiple goals to be achieved through the seminar and factoring them all into any proposed solution system.
Clearly male showoff behavior is not compatible with most plausible goals for any such seminar - is there a way to minimize, e.g. having the chair or the group give approval/disapproval as to whether a question needs to be answered?
Berk: There is some evidence that if the first person called on to ask a question is a woman, many more women subsequently ask questions. If the first person is a man, then fewer women ask questions (https://www.economist.com/science-and-technology/2017/12/07/women-ask-f…)
So perhaps this is one more thing the chairs of seminars can do: Call on a woman first.
I was at a political science seminar recently, where every participant had read (and was expected to have read) the paper beforehand. The speaker was given 10 minutes (no interruptions) to make a brief statement, maybe regarding what they hoped to get out of the seminar that day, or what they thought some of the limitations and next steps would be.
People then put up their hands if they had comments. Fifteen hands shot up at once, and a list of names was recorded, which was followed in sequence, with a few additional names added to the list as the seminar went on. It was a remarkably civilized experience, and proved that it is possible to have a large amount of audience participation and back-and-forth discussion, without derailing the speaker or yielding the entire seminar to one or two intransigent audience members.
It might be hard to do this for a job talk, where there is a certain set of ritualistic expectations, but it seems like for regular academic seminars, this is a model that might be worth exploring.
19 years ago, I started attending WB seminars when I was at IFPRI. There were great presenters with great papers, and I was really excited to get to go to those. Then I stopped attending entirely because of the ridiculous constant questions, which, as Berk points out, were almost entirely questions that were clearly going to be answered in the next few slides. Constructive criticism was nearly entirely absent.
Since then, I've given 3 or 4 seminars at the Bank myself, and I've always imposed a 15 minute question moratorium, and I believe that that's the only reason I generally got very helpful constructive criticism.
I think there are two separate issues; one is that questions by anyone in the first 15 minutes are generally unhelpful, uninteresting, and not constructive -- for the speaker. A side effect is that it's also really annoying to the seminary participants who actually want to listen to the speaker speak.
I see this as separate from the gender issue, though potentially inter-related. It would be interesting to see if the 15 minute moratorium also leads to more gender balance in the questions. It might be if the moratorium also reduces the preening questions. But, you also noted that if a woman asked the first question then this also reduced the gender disparity, so a chair could make sure that if there are multiple hands up for the first question, she tries to choose a woman.
Cheers, Nancy
To Nancy and Shanta,
Thanks for this. I am skeptical of the promise of the "women ask the first question" approach because I think that pattern may be due to selection effects, meaning that the exogenous manipulation of the condition (call on a female first) may not cause a change in the composition of subsequent questions.
Having said that, if there are hands up, at least one of which belongs to a woman, it would not be hard for the chairperson to do this. I see female mystery clients in the horizon...
Hypothesis: male researchers ask more questions because: a) they need to show off; b) they are more socially isolated and need interactions (not joking); c) women have a distaste for preening.
Intervention for a) - take a few minutes at the beginning of each seminar to let audience members share any relevant work they've previously been engaged in. Perhaps start seminars 5 minutes early for any who want to do this.
Intervention for b) and c) - ask seminar attendees to pre-submit one- or two- sentence questions to the speaker, so that he/she can choose to address them during the talk, if he/she wants do.
Both interventions could have the knock-on effect of encouraging a more personalized engagement between the speaker and the audience.
For those of you who might be interested - this is a code of conduct that is being discussed within an econ department to tackle some of the issues Berk mentioned in the post. Hope it can be useful to others!
DRAFT Code of Conduct
Seminars play a vital role in the flow of ideas and in the reputation of speakers and departments. We strive for our seminars to promote a lively exchange of ideas, and we want our department to be a place that presenters look forward to visiting. This Code of Conduct is a guide to help achieve our goals for our seminars. We think of this Code of Conduct as an extension of the AEA Code of Conduct, which guides behavior in the profession as a whole. (https://www.aeaweb.org/about-aea/code-of-conduct)
We also note that it is important to think about how seminars relate to diversity in economics. Research shows that women and people of color often have very different experiences in seminars than white men, and often receive more questions, tend to present later, etc.
Quoting from the AEA code of conduct, economists’ individual and collective responsibilities “include supporting participation and advancement in the economics profession by individuals from all backgrounds, including particularly those that have been historically underrepresented.”
Seminars are the first and main contact many students, faculty, and visitors have with other members of the department. We want our seminars to facilitate productive and collegial conversations among diverse researchers.
Goals – What are seminars for?
Seminars play several key roles in advancing new knowledge and techniques within our discipline. In our department, we use seminars to achieve the following goals:
For the speaker:
- To hone their research by soliciting constructive feedback from the audience
- To demonstrate their quality as a researcher and advertise themselves as potential collaborators or future hires
- To share new ideas and methods within and across economics departments
- To practice presentation skills
- Giving students practice handling aggressive feedback is not a goal of our seminars – we believe other avenues that are more suitable for practicing that skill
For the audience:
- To learn new ideas, methods, or content
- To offer constructive feedback that will help the presenter improve their research, or help other audience members deepen their understanding
- To advertise their quality as potential future collaborators or advisors
- To advertise the department as a potential future employer, or to highlight an audience member’s potential in the presenter’s department
For both:
- To advance economics research in general and contribute to global knowledge
- Maintain an intellectually generous culture, assuming both competency and good intentions of other seminar participants
Norms – How can we achieve these goals?
In order to achieve these goals, we affirm a set of standards and best practices for seminar behavior.
1. Tone should always be courteous and respectful of all seminar participants.
- A polite tone enables productive discussions of ideas since the presenter will be more comfortable thinking on their feet and being open about their project’s strengths and weaknesses.
- A respectful tone will improve our department’s reputation and ability to recruit excellent speakers, graduate students, postdocs, and new faculty.
2. Attribution of arguments, questions, and ideas is important.
- Speakers and audience members should strive to credit the person who originally raised an issue or asked a question when the same idea is brought up later.
3. Questions are a valuable way to help clarify, point out a relevant shortcoming, or offer an insight. However, not all questions work towards these objectives. Different seminars and contexts have different goals, and therefore different norms, but we believe these norms apply to all questions in all of our seminars.
- Questions that should be asked in seminar:
- Questions that will be useful for other members of the audience
- Questions that are not already going to be answered in a couple of slides
- Questions that clarify the speaker’s point
- Questions that identify an issue in the research, and ideally provide a solution
- Questions that provoke more thought and help frame the research in a broader literature
- Questions that should be asked offline:
- Questions that stray from the scope of the research project
- Questions that were already asked by someone else in the audience
- Questions that were likely answered before you arrived at the seminar (if you are unable to make the beginning)
- Questions that remain unanswered after a brief back-and-forth. If this happens, thank the speaker and follow up later
- Questions that, for whatever reason, would disrupt or derail a talk, including “Is this economics?” or “Is this project worthwhile?”
- Questions that might be asked offline, but probably shouldn’t be asked at all:
- Questions designed to demonstrate your own expertise
- Questions mainly to test the speaker’s intellect or ability to think on their feet, rather than the caliber of their research, which historically excluded groups are disproportionately burdened by
4. Background conversations or gestures should be minimal, and very quiet.
- Sometimes asking your neighbor a question can help you decide whether to ask the speaker, or provide you with a quick clarification. But background conversations, even at a whisper, should not continue for a long time or recur throughout the talk, nor should they be audible to people around you.
5. Protected time at the beginning and end of the seminar can help reduce the number of unnecessary clarifying questions by giving the presenter the chance to fully motivate their topic.
- Seminar organizers should choose appropriate norms for their seminars.
- First 5–10 minutes: preserving the first 5–10 minutes as a question-free time allows the presenter to provide the full context and motivation for their project.
- Last 5 minutes: preserving the last 5 minutes as a question-free time allows the presenter to wrap up cohesively.
- The presenter should have the option to open up this time to questions.
- Invited speakers should be informed of these standards.
Feedback – How are we doing?
Providing and receiving feedback is an important part of putting these goals and norms into practice. We invite all seminar attendees to offer feedback using this anonymous Google form [link omitted]. You’re invited to submit feedback (positive or negative) about the seminar overall, or about particular moments that occurred.
We expect the department chair to periodically review the forms, and to discuss this Code of Conduct and norms of behavior with the department in the semesterly town halls and with those who receive individual feedback through the forms.
If seminar participants need additional avenues to express concerns, [specifics omitted] may be helpful.
This Code of Conduct applies to the department's students, faculty, postdoctoral fellows, and affiliates when they participate in department and non-department seminars alike, on or off campus. Similarly, people outside the department are expected to follow this code when participating in department seminars.
This code of conduct was should be revisited and reshared with the department every year.