Quality, reliability of NSS data at stake

THE QUALITY and reliability of the National Sample Survey (NSS) data have been under constant debate within the NSSO and outside. Though there have been significant improvement in the surveys, concerns about their reliability and the relative merits of NSS and other sources remain. These concerns - evident in the current debate on trends in poverty incidence - arise from (a) differences between the NSS estimates and those obtained from other sources, namely, data compiled by government agencies and surveys conducted by other organisations; (b) the quality of field work and the accuracy of information provided by respondents; and (c) the effect of changes in scope sampling and questionnaire design and field procedures.

That data obtained by different agencies differ is not surprising. They are to be expected because the scope, source, content and methodology used by them are seldom identical. Much effort has gone into critical examination of the differences in estimates from different sources and reconciling them. This is particularly noticeable in respect of private consumption expenditures from which poverty estimates are derived. These exercises are valuable but only up to a point. A satisfactory reconciliation is not possible because - as the poverty debate illustrates - it is difficult to adjust fully for differences in scope and concepts; because estimates from different sources are all subject to unknown margins of error; and typically, for most sources, comparability over time is affected by changing design and concepts.

No matter how well surveys are designed, the quality and veracity of information depends much on how questions are framed; how informants understand the questions; and whether they are willing and able to give accurate information. These sources of error that affect all systems of getting data are difficult to eliminate and measure. While sample surveys have the merit of being economical and are not subject to biasses of the kind affecting data collected by executive agencies, they are not free from errors arising from deficiencies in the conduct of fieldwork and inability or unwillingness of informants to give correct information.

The NSS field investigators are given special training and detailed instructions in every survey before fieldwork begins. Nevertheless, the scale of the surveys requires a large number of investigators dispersed widely all over the country. Though there are well defined, explicit procedures for supervision and checking of their work, it is difficult to ensure that they are enforced uniformly and strictly everywhere. That the field work depends wholly on a permanent cadre of government staff with limited opportunities for career advancement and that too unrelated to performance compounds the difficulty.

There is also evidence of systematic differences in the quality of fieldwork between the Central and State level organisations. Though both are supposed to adopt identical procedures estimates from State samples are known to differ significantly and systematically from those obtained from Central samples. While the reasons for these differences have not been properly investigated and corrected, there is good reason to believe that State bureaus are not as strict in supervision and checking as the Central organisation of NSS. In fact, most State bureaus publish only a fraction of the planned tabulations and even these are not easily accessible. Most of the detailed discussions based on NSS data rely on estimates from the Central sample.

Criticism over survey methodology

A legitimate criticism of the NSS is that it has not paid enough attention and resources to refining and improving survey methodology in the light of experience. In the initial stages, there was a great deal of emphasis on this aspect thanks to Prof. Mahalonobis and the fact that ISI was intimately involved in NSSO. Over the years, several improvements have been made but methodological research on questionnaire design, relative merits of single and multi purpose enquiries; the characteristics of respondents and their ability to give needed data; reference periods; field work procedures and related aspects has been neither sustained nor adequate. This requires close collaboration and interaction with academic institutions such as ISI which has greatly weakened during the last three decades.

A more general criticism - which applies not only to the NSS but to the statistical system as a whole - is that information is collected without adequate attention to the purposes they are meant to serve and the cost involved.; that only a fraction of the information collected is utilised; and that it takes an inordinately long time to publish them.

There is considerable force in the criticism about conducting too many census type enquiries and detailed/ disaggregated surveys whose data are not utilised, nor perhaps necessary. In some cases user agencies seek information at a level of detail and disaggregation (there are instances where they want data at the block or even village level) unmindful of the costs and in the event do not put them to any use.

For example, it is not clear what purpose observed by conducting quinquennial economic censuses and censuses of agricultural holdings and NCERT's education surveys at enormous cost in terms of money and diversion of staff from their regular duties. The economic census is meant to give a complete list of establishments engaged in non-agricultural activity to serve as a frame for sample surveys. Summary tabulations based on the census are published, but the detailed lists of establishments, which in principle should be available by village and town, are rarely maintained in a compiled form. They are not always available when needed; nor are they accessible to the public.

If the purpose is to provide a sampling frame, it is adequate to do a census of commercial and industrial establishments once in two decades. Provided the lists are properly maintained, the frame can be updated in select localities before selecting the ultimate sampling units. In the case of other censuses - including land holdings, minor irrigation, livestock and education - mobilising the necessary personnel and getting them to do a proper job of enumeration has proved difficult. Quality suffers because of lack of supervision - sometimes no supervision at all! Being based on village revenue records whose veracity in reflecting the actual current position is open to serious question.

Delay in processing data

As for delays in processing and publication, one reason is the inadequacy of resources allocated for this purpose relative to what is spent on collection. This is especially the case at the State level where even the basic tabulation programme approved by the NSS governing council is not completed. Very little of it is available in accessible publications.

This would not be a matter of concern if the agencies or researchers concerned are enabled to do more detailed tabulations and analysis depending on specific issues they want to examine. But this requires the preservation of the primary data and making them accessible to uses. Before computerisation, the information from individual sample units of NSS used to be stored in punch cards: there is a huge godown in Calcutta where they are kept but have become unusable over the years.

Computerisation has overcome this problem but only to some extent. Since the early 1990s, the NSS has been preserving primary data in tapes. Accessibility to outside users, especially indigenous researchers, has also improved considerably because the CSO and the NSS supply the data on CD-ROMS for a reasonable charge. But in the case of agricultural, livestock and establishment censuses the primary data are not maintained in a properly compiled and retrievable form. It is reported that the agricultural and minor irrigation census data have been computerised but many people, including officials and non- officials, who have tried to get hold of these data sets, have drawn a blank. What then is the use of such data?

Most, if not all, of the above mentioned concerns have been articulated and debated by several committees, seminars and conferences organised by professional associations and some by Government. To cite a few: the Indian Econometric Society conducted five seminars on various aspects of data base of the Indian economy during the 1970s and the 1980s. The papers and proceedings of all of them are published. During the 1990s, questions relating to data were again comprehensively reviewed in a seminar organised under the auspices of the ICSSR and another by the CSO. These discussions focussed on gaps and deficiencies in respect of practically every important aspect of the economy and came up with concrete suggestions to remedy them. That these have not been acted upon is, in large measure, a reflection of apathy among administrators and policy makers.

Ill informed scepticism

There is among them an ill informed scepticism, if not hostility, to data and empirical analysis. Policy makers and high level officials have shown little interest in ensuring that they get reliable and timely data. Certainly they have not exerted much to create institutional mechanisms for independent checking and verification of information they get from within their own departments.

As already noted, there are in-built incentives for biassed reporting within the Government, by the Government to the public and also by households and enterprises. Control over information and the ability to manipulate it are often - all too often - used by Government, as indeed by private interests to influence public opinion and policy decisions in particular directions.

There are instances of officials who insist on reporting data collected by following proper procedures have been punished because the figures were not in the State's interest. And when independent sources give estimates which are inconvenient to one or other interest, the tendency is to dismiss such data rather than commission a serious examination of the reasons for the difference statistical affairs.

More generally, the tradition of basing policy discussions and decisions on sound data-based analysis and of making the data and the analysis available in the public domain remains weak. Secretiveness and selective and motivated release of information being the norm, it is not surprising that there is so little interest in measures to ensure the integrity of the data system. Academia too has contributed to this situation by not exploiting the potential of published data, examining them critically and using them for rigorous analysis. Inadequacy of data and difficulty of access tend to be overused alibis.

A. Vaidyanathan

(The author is professor emeritus at the Madras Institute of Development Studies.)

Send this article to Friends by E-Mail

Section  : Business
Previous : Domestic insurance policies
Next     : Management of personal finance amid scams