Sunday, December 11, 2011

A Study of Eight Cities

A Times of India-IMRB survey on the Quality of Life in eight major cities in India has been published today (ToI, 11 Dec, pp. 16-17) . The cities are Ahmedabad, Pune, Delhi, Mumbai, Bangalore, Hyderabad, Chennai and Kolkata.  Here is the link:

ToI - IMRB Survey on Quality of Life in 8 Indian Cities, 2011

There are some aspects of this survey that I found problematic. I'm not talking at this stage about the findings, because those are entirely dependent on the aspect of the survey with which I did have a problem, which is the methodology.

1. Sampling:

a. Layman sample: IMRB polled "roughly 150 respondents" from socio-economic categories A and B, in a 65:35 ratio in each of the eight cities. The respondents were "from age 18 onwards", and "the two genders were roughly equally represented".

b. Expert sample: ToI asked the same questions that had been asked to the layman sample, to 30 people in each of these cities "who track living conditions in them".

1a. About the layman sample: According to the Census of India, any town with a population of above a lakh of people (>1,00,000) is called a city. As per the provisional totals of the Census on India 2011, here are the populations of the municipal corporations of the cities covered by the ToI-IMRB survey:

  • Ahmedabad: (approx) 55 lakhs
  • Pune: (") 31 lakhs
  • Delhi: (") 1.1 crore
  • Mumbai: (") 1.25 crore
  • Bangalore: (") 84 lakhs
  • Hyderabad: ('') 68 lakhs
  • Chennai: (") 46 lakhs
  • Kolkata: (") 44 lakhs
It is to be noted that these are the provisionals of only that population covered by the city municipal corporations. At least two cities in the list above are in fact much larger than the area covered by the municipality. 

The point of sampling is representation, and in the case of our quite heterogeneous cities, a fairly complex challenge. Divisions have to be marked, parameters of representativeness drawn up. This requires knowledge of, and some access to diverse quarters of the city, and is certainly not accomplished by administering a questionnaire to 150 people, of any description. A net total of 150 people in an Indian city does not possibly represent even the number of people taking a bus from a single city bus-stop during morning rush hour during a single day. 

Now, the kind of people. SEC A and B, in a proportion of 65:35. This means the questionnaire was administered to about 97 individuals who were SEC A men or women, and 38 individuals who were SEC B men or women, in each city. SEC categories for demographic stratification are widely used in commercial research, including that for retail and media. Here's what they mean:

Since IMRB focuses on commercial sector research, it may be safe to assume that this is the manner in which SEC has been used, in the ToI-IMRB survey. This means that the data on which the survey is based has been collected from a very small number of the most elite people in eight quite heterogenous, more-than-quite-large, Indian cities. Only in the most imaginative sense can this exercise be called sampling.

1b. Expert Sample: This component, carried out by ToI, asked 30 people who track living conditions in a city to fill the same questionnaire.

But who are these people? What makes them experts? Why should I, as the reader of a survey, be asked to accept that these are credible experts in the sub-areas of urban development and governance, in whatever fashion? Are they, for example, associated with real estate or computers? the law? politics? are they senior, or junior bureaucrats? traders? social work? Different people watch the changing city in different ways, and it is the responsibility of the methodology of a survey to take into account these differences.

Sampling strategies will hugely influence what you find from research, and this effect is stronger with an expert sample, because of the significant role of expert opinion. Anyhow, not to even display a list of criteria for the expert sample, which is quite large relative to the layman sample (1:5), is just shoddy research.

2. Data Analysis

The data gathered from the sample discussed above was then subjected to some analysis. The methodology mentions factor analysis, regression analysis and average scores.

2.1 Regression analysis: Regression analysis demands, at the very least, a dependent variable and an independent variable. It is intended to point out a definite direction in causality: hypothetically, if a variable (say, social infrastructure) is low, then another, connected to it (say, environment) must also be affected in a certain way. It is to know such interconnections between variables that regression or correlation analysis is performed.
An understanding of the manner in which, say, 'peace of mind' is linked to 'leisure facilities', or 'social infrastructure' would be a very interesting result of this survey. But since this information is not available, I am not sure what this putative regression analysis was intended to perform.

2.2 Factor analysis: I would have really liked more detail on the factor analysis. I would, for example, have liked to know how important the perceived quality of schooling, or college, is to the rank achieved by a city. They do present some information that shows the relative importance of their seven major parameters to overall ranking. But this is a problematic procedure and these scores can hardly be added up to score overall rankings across cities.

2.3. Average Scores: This, I think, is the one procedure that has been accurately followed in the survey. The averages may be for 150+30=180, very elite people per city, but they are correct, in that.

This survey is conducted by the Times of India and the Indian Market Research Bureau, both vast organizations with proportionate resources. We have, in this country dense, diverse, vibrant cities.

Is this the best that can be done by way of mapping them?


Beq said...

What could it want to accomplish? It's a set up and it basically says nothing. Another way to help tom tom the "vibrant growth of Gujarat?"

olidhar said...

and my goodness, the category toppers! for instance, delhi (alongside mumbai) is the best city for commuting! i am sure they arrived at this conclusion taking into account not only the extensive public transport networks in the city, but also the safety of commuters in those networks.

Devalina said...

@Beq: On the other hand, it could just be bad, unthinking research. There's enough of it around that i am willing to believe that this is just another example.

@olidhar: I am sure that all the important factors you mention were taken into account. And more :)

RBC said...

Arrrgh! Nothing is more venial than a statistician with an agenda. Actually I'm less willing to believe it was an 'accident' that they didn't publish their criteria. A lot of foreign investors look at quality of life surveys to locate their offices because their whiny foreign executives want first world facilities for their families. Ahmedabad obviously wants the lion's share of these dollars. Imagine their shock when they find out no Jack Daniels.

Devalina said...

@RBC: Well, they'll get the 'sharaabi' certificate, about Rs 8,000 :) , and they can always drink at the five star hotels (Rs 500-800 a drink, i hear). Straight in somebody's exchequer
I also read in the newspaper, btw, that Modi is considering death to all who sell alcohol. I really should preserve that item of info

shonedeep said...

You could now do a study on the methodology behind TV viewership in India (TRP). You will have a field day with the asinine assumptions etc