A couple of years ago I wrote a post about several practical issues in using Lee bounds to deal with attrition. One of my pieces of advice has updated based on comments I received on that post, and then another practical issue has arisen in a different project I’m currently working on, so thought I’d share both of these additional practical issues here.
Practical Issue 1: Dealing with ties in the outcome or binary outcomes when trimming
In my previous post I had noted that sometimes you are in a situation where there are multiple observations with the value that you need to trim, but you only need to trim a fraction of them. I had mentioned setting a seed and randomly selecting a subset to trim, but Jon Roth in a comment gave the better advice of reweighting the data to deal with these issues. I thought I’d illustrate with a couple of examples from some analysis in progress on one project.
Example 1: Binary outcome to be trimmed
In this setting, the binary outcome Y takes value 1 or 0 (e.g. firm operating or not operating). I have 2.2 percentage points higher attrition in my treatment group than my control group, and my control group has a sample size of 392 observations. So I need to trim 0.022*392 = 8.6, which rounds to 9 observations from my control group.
In my control group, for those responding, I have 109 observations with value 1, and 119 with value 0.
For my Lee upper bound, we assume all the extra observations that attrited in the treatment group had the high value of 1, so want to downweight the control group values of 1 so that instead of representing 109 observations, they represent 100 observations. To do this, I create a weight like this in Stata:
gen weight_u=1
replace weight_u=100/109 if treat==0 & y==1
I then get my Lee upper bound estimate by running my treatment regression, using this weight:
reg y treat [aweight=weight_u], r
Likewise, for the Lee lower bound, we assume all the extra observations that attrited in the treatment group has the low value (0), so we want to downweight these values in the control group so they represent 9 fewer observations. To do this I create a weight for the lower bound:
gen weight_l=1
replace weight_l=110/119 if treat==0 & y==0
reg y treat [aweight=weight_l], r
Example 2: multi-valued outcome with ties
In this second example, the outcome y is a variable that is the average of several subcomponents that are all coded on a 1-5 scale. E.g. it could be a measure of how digitally ready the firm is, where the firm is scored from 1 to 5 on several different metrics. In this case, there is also some item non-response on top of the overall attrition, and differential attrition is 3.0 p.p., which results in 392*0.03= trimming 12 observations from the control group.
For the Lee lower bound, I again want to trim the 12 lowest values in the control group. However, when I look at the distribution of y in the control group, I find that I have 4 observations with value 1, 3 with value 1.16, 1 with value 1.33, and 6 with value 1.5. So here I need to do a mixture of trimming and reweighting. I start trimming, getting rid of the 8 observations with values below 1.5. But then I have 6 observations with value 1.5, and should only be trimming 4 of them. We deal with this by giving these observations weight 2/6. So I create a weight like this:
gen weight_l=1
replace weight_l=0 if y<1.5 & treat==0
replace weight_l=2/6 if y==1.5 & treat==0
and then I can estimate the Lee upper bound using the weighted regression as above. I can likewise check the distribution of values of y in the control group at the top of the distribution and see what mix of trimming and weighting are needed there.
Note 1: in my example here, attrition is higher in the treatment group than the control, which is why I am trimming or weighting the control. It is more common to have higher attrition in your control group, in which case the trimming or reweighting will happen for your treatment group instead of control group.
Note 2: note also that I am trimming/reweighting a different number of outcomes for example 2 than example 1, because of item non-response. You don’t just want to look at what overall differential attrition is, but check what it is for each outcome you are estimating.
Practical Issue 2: Dealing with Multiple Treatments
A second issue that I haven’t had to think about before is what to do when you have multiple treatments and attrition. To make this concrete, suppose I have a control group, with response rate 60%, and two treatment groups (T1 and T2), with response rates 82% and 85%. Then how should I trim the data to conduct Lee bounds? Here I’ll talk about trimming, but if there are ties, you can handle the ties as above.
My answer on this is that different trimming makes sense for different sets of comparisons. If I want to get bounds on my treatment effects of T1 and T2 when compared to the control, then in this case, I trim 22 p.p. of observations from T1 and 25p.p. from T2, trimming these from either the top or the bottom of the distribution depending on which bound I want to get. Then you can just run your standard treatment regression on this trimmed sample, which trims both T1 and T2.
This works when both treatments have higher attrition rates than the control. But if we are in a situation where e.g. T1 has lower attrition than the control, but T2 has higher attrition, the above solution won’t work. You would want to trim T1 and keep the control untouched for the T1 vs control comparison, but trim the control and keep T2 untouched for the T2 vs control comparison. So here I would run this as two separate regressions – one to get bounds on the T1 effect, throwing out T2, and one to get bounds on the T1 effect, throwing out T1.
However, note in my example that while I had high differential attrition relative to the control, I had pretty low differential attrition between the two treatments. If my main interest is in comparing the relative effect of T1 to that of T2, I don’t want to throw away a lot of data by trimming 22 p.p. of observations from T1 and 25p.p. of observations from T2 – but instead use more of the data and only trim the difference of 3 p.p. from T2. I could then either just leave the control group as is as a reference group, or else drop it and just directly estimate the differential treatment effect of T1 vs T2.
Join the Conversation