Can an index ever be a good measure of social inclusion?

|

This page in:

I really don’t like indices, particularly those that claim to measure what are termed “social issues”. And they seem to be everywhere. Ok, the Human Development Index did a lot to push countries to do more on health and education, and its rankings serve to pit countries in good competition with each other.  Single measures are also intuitive and easy for monitoring purposes.

Just to stop my initial train of thought here, I have two problems with indices that measure “well-being”: first, they are often weighted and the weights assigned to individual components expose the subjectivity of their creators.  If you think primary education is more important than reproductive health, and you assign weights that way, that’s what your index will pick up. 


Second, in a bid to make them comparable across countries, their creators make indices awfully generic - almost reductionist.  There’s no room for context specificity.  Even the HDI reduces “human development” to life expectancy at birth, schooling and gross national income per capita.  Both the problems I have with indices become really grave when you are trying to measure something as context-specific and as steeped in how people feel as is “social inclusion”.  So I’m always inclined to take my friend Jeff Hammer’s advice - “use indicators, not indices”!

​But what do I do when people ask me for a summary measure of social inclusion? Well, first I tell them I don’t like them.  If I’m pushed, I say look for a dashboard where you can construct your own measure based on the weights that you assign.  This reflects both the context you are operating in, and your framing of social inclusion or well-being.  The OECD’s Better Life Index does a really good job, but it’s limited to data-rich OECD countries.  The Multi-Dimensional Poverty Measure is another example of building a summary measure based on your context.

I have been mulling over the Social Progress Index, since the beta version came out in 2013. This “new” measure does two things that I like – first, it takes into account subjective views of citizens  (it’s not just how well your government or researchers think you are doing; it’s how well you think you’re doing).  Second, it does not include an economic measure (but you can easily plot the index against economic measures. The documents related to the Social Progress Index do this and show that countries that perform well on an aggregate economic measure don’t always do as well on social measures).  So, I just exposed my own subjectivity!

The Social Progress Index is a simple average of three dimensions of “social progress” – basic human needs, well-being and opportunity.  And each dimension has several indicators.  Of course, you may quibble with the terms, each of which is a contested concept in its own right. For example, I use this index to the advantage of our report, “Inclusion Matters: The Foundation for Shared Prosperity” because it is philosophically so aligned; I would say it is actually a good measure of “social inclusion”.

Social Indicators

There would be many who wouldn’t like the Social Progress Index.  It may be too touchy-feely for some. Others may ask: “what are you really measuring”?  Still others may say: “so what? What policy do I put in place?” Still others may say: “perception surveys that the Index uses are problematic – you don’t know what they are catching”. 

Agreed! But the detailed methodology report  (2MB .pdf) just seems to say – “here’s what we thought is important, here’s what we did and here’s how you can use it”.  It is well grounded in the history and philosophy of measuring well-being, which gives it a certain depth and takes us through the intellectual journey of the authors.

So, in all, the Social Progress Index helped me to move away from my (summary?) dislike of summary measures. But I wonder what you think. Do you like it or dislike it? Do you think it has too many indicators? Or would you add some others?

Meanwhile, join us for our next webinar on Social Inclusion on January 7– it’s on measurement! 

J. Cok Vrooman
January 03, 2015

If theorized and micro: yes
As long as an index and its underlying dimensions have a firm theoretical basis, and the indicators are valid and reliable, it may be quite informative. The index provides an overall assessment of the theoretical construct, while the scores on the sub dimensions and individual indicators allows one to understand group differences and changes over time. However, a good social exclusion index should be based on microdata and citizen's involvement; and it may not be a very good idea to assume linear relationships (as in a factor analytic approach). Cf. http://link.springer.com/article/10.1007%2Fs11205-012-0138-1

Bojan Radej
January 06, 2015

Dear Maitreyi,
Thank you so much for bringing forward this topic about measuring complex matters with simple concepts and single measures, when their content is incompatible and thus cannot be summarised. I am excited to study your methodology more in detail. There is increasing discomfort between methodologist of social research about construction of summary measures on qualitative social phenomena, like happiness, quality of life, human development or social inclusion… Aggregation problem has been recognised as important methodological issues by Scriven, Munda, Virtanen and several other eminent evaluators. May I add also my belief that resolution of this question will be decisive in setting direction for future development of methodology of social research in “anti-postmodern fashion”, which is reconstructing holistic aspiration of modernity together with enriching postmodern capacity to deal with social diversity.
Concerning weighting of input indices that set up composed index: not only this exposes index to subjectivity (which is nothing wrong) but also to relativism - which is very much wrong if one is concerned with measuring qualitative social issues (Sustainability, gender equality, cohesion, human development). Qualitative issues are: (i) equally important (equal weights) and (ii) few in numbers – the same qualities are not equally important everywhere but everywhere the most important issues are only few. This suggests that inquiry about methodology for construction compound indices of social qualities is not microscopic issue (weighting range of specific indices) not macroscopic (HDI as aggregate of three representative components) but mesoscopic.
The methodological dilemma is not therefore between a summary measure or a dashboard; they are both inappropriate for the purpose (measuring social inclusion) from opposite reasons; micro reasoning is relativist in relation to qualitative concerns, macro reasoning is blind for context and diversity.
The question therefore arises, how meso methodology is different from micro/macro methodology? Let me illustrate on the same case which you also apply – the HDI, in relation to your mentioning need for ‘multi-dimensional’ measure? What do you mean by ‘multi-dimensional’ measure? Is this only hierarchical structure so it allows for aggregation of detailed results from micro to macro or it includes also horizontal dimension, which denies possibility of full aggregation due to incommensurable values, contents? For instance, HDI is a vertical structure (indicator level – HDI index level) which on its way of construction kills its constitutive horizontal dimension (three domains of HDI) by their aggregation from component indices to a summary index. HDI methodology is developed precisely due to incommensurability of money with social qualities (health, education), but it nevertheless allows for aggregation of three components into summary measure of HD!?
The problem with SPI is similar. In present form it degrades the qualitative into quantitative challenge instead of leading it to new qualitative level of consideration. Its methodology needs to be derived consistently in relation to the main challenge, to measure one aspect of social quality. Quality (SP) consists of qualities, not of measurable elements, so the components of a given qualitative phenomenon cannot be treated as commensurable by definition and so they cannot be aggregated into summary measure of SPI.
The methodological challenge is not irresolvable, only conventional methodology is too narrow to deal with qualitative concerns. Summary conclusion is achievable on meso level – not as singular index but as a diagram (Venn’s), in graphical presentation that enables presenting diverse results of heterogeneous phenomenon (social inclusion), instead of aggregate of quantities. Please see two case studies for elaboration of methodological approach.
Resolving aggregation problem: http://www.sdeval.si/attachments/article/105/sde_DZ_1-14_Apples%20and%2…
Measuring Social inclusion (cohesion): http://www.sdeval.si/domov/39-objave/objave/490-divided-we-stand
Background philosophical and theoretical text is also available on request.
Kind regards,
Bojan Radej, Slovenian Evaluation Society

Maitreyi
January 08, 2015

Dear J. Cok Vrooman
Thank you for your comment and for sharing your paper. I agree with you that theoretical grounding is very important, but so is data availability. In the paper that you have shared (which I look forward to reading), it appears that you collected primary data to test your theoretical construct and then came up with a studied and methodologically rigorous single measure. That is not always possible and the issue of context specificity becomes really salient. Thanks very much for sharing your thoughts.
Maitreyi

Maitreyi
January 08, 2015

Dear Bojan Radej
Thank you for your thoughts on measuring social inclusion. We do not espouse any particular methodology, but make a plea for framing the right questions and adapting data and methods to context. This does become problematic in data scarce environments and if we want to make comparisons across contexts.
Thanks also for sharing your interesting papers. Both you and J. Cok Vrooman have highlighted the importance of theorizing and of data, as do we in our report, “Inclusion Matters: The Foundation for Shared Prosperity”.
Best, Maitreyi

Bojan Radej
January 12, 2015

dear colleague,
Following your response I am not sure any more that I properly understand your call for comments. My view is that SPI methodology is inconsistent from the same reasons as HDI. It does not matter if you lack data when your model is inconsistent in its core.
When you receive a comment supported with arguments it is hardly appropriate to respond so lightly with 'hmm, this is interesting', without any argument. I am dissapointed if this is the World Bank's level of discussion.
Best, Bojan