THE MEASUREMENT OF QUALITY OF CARE IN PUBLIC SECTOR PSYCHIATRIC SERVICES BASED ON CONSUMER EXPECTATIONS

In this study the expectations o f consumers of public sector psychiatric care in South Africa were identified, and formulated in the form of 13 standards, each with a set o f criteria. During this phase input from the literature was incorporated, and expectations were validated with different groups o f consumers, so that rural/urban, ethnicity and regional differences were taken into account. Based on the comprehensive set o f standards and criteria, four instruments were developed to measure attainment o f these standards. These included a questionnaire to consumers and one to the Director o f Mental Health. It also included two schedules to be filled in by observers during site visits to hospital units and clinics. The observer teams included community members and consumers. The content validity o f the instruments was established by setting out the items measuring each criterium, and validating that with a group of experts. The instruments were then tested in one province. The inter-rater reliability o f the site visit schedules was calculated as 0,94, and the coding of the Director questionnaire by different coders was also tested. The average performance on all criteria was calculated, using items from all four data collection instruments. In the process items were revised, coding instructions developed, and criteria adjusted

Based on the comprehensive set o f standards and criteria, four instruments were developed to measure attainment o f these standards.These included a questionnaire to consumers and one to the Director o f Mental Health.It also included two schedules to be filled in by observers during site visits to hospital units and clinics.The observer teams included community members and consumers.The content validity o f the instruments was established by setting out the items measuring each criterium, and validating that with a group of experts.
The instruments were then tested in one province.The inter-rater reliability o f the site visit schedules was calculated as 0,94, and the coding of the Director questionnaire by different coders was also tested.The average performance on all criteria was calculated, using items from all four data collection instruments.In the process items were revised, coding instructions developed, and criteria adjusted BACKGROUND In a summary by Freeman (1992) the incidence of mental illness in the South African population varies between 12% and 39% in communities in South Africa.With a population of 40,7 million (Sunday Times, 1994), this gives a minimum figure of 4,9 million consumers or potential consumers of psychiatric care.
In the developed countries, strong consumer movements have developed since the 1970s.This probably has been a direct result of the de-institutionalization movement, which pushed psychiatric patients into the community without adequate support.In the Netherlands the "Clientenbond" was developed, in the UK the MIND consumer networks and in the USA the National Alliance for the Mentally 111 (NAMI).Similar to the other consumer groups NAMI, composed primarily of families of seriously mentally ill people, is dedicated to advocacy, education, mutual support and resource development.It has become extremely powerful at all levels of government, and has lead to significandy increased participation of consumers in planning of mental health services.
One of their activities over the last decade has been a regular rating of state programmes for the seriously mentally ill (Torrey, Wolfe and Flynn, 1988).They rated the services based on questionnaires filled in by state directors of mental health and vocational rehabihtation, and questionnaires filled in by members of two different consumer groups.The inpatient services are rated according to 28 standards in four categories (quality of staff, quantity of staff, quality of treatment, and environment), and the outpatient services according to 33 standards in five categories (quality of staff, quantity of staff, quality of treatment, coordination, outreach).They also rate rehabilitation services and housing.In all cases minimum and ideal standards are given.When all four components have been rated on a scale of 0 to 5, the total score for the state is calculated and the state is then put on a scale to compare it with other states.
They give as their reasons for the regular rating of services the following; 1.The provision of psychiatric care is big business, and consumes much of the tax payers' money.It therefore is appropriate to evaluate how well the services are succeeding in giving quality care.
2. The provision of psychiatric health care is an important function of the state and local government; a fimction which is often neglected.It therefore is part of the evaluation of a government to evaluate their record in terms of psychiatric care."One reason to do a state survey of services for the seriously mentally ill, then, is to politicize the issue in the best meaning of the word" (Torrey, Wolfe and Flynn, 1988, p.l).
3. Improvements in state services for the seriously mentally ill traditionally have followed public horror stories of disastrous care.Since public disclosure is the impetus 4.
for change, regular disclosures are essential to improve care.This is consonant with the trend towards greater public disclosure and transparency in terms of the quality of health care.
5. Competition and comparison are part of the competitive world we live in and cannot be avoided in public services.
All these reasons are applicable to South Africa.Psychiatric care is an expensive service which is funded mainly by tax payers.It is an important function of provincial governments in the new South Africa, and there is no better time to initiate a regular consumer rating system.The public has very little awareness of the plight of the mentally ill and their families, and should be informed and involved.A greater focus on the evaluation of quality is also true of South Africa, where an Interim Accreditation Board recently has been formed.And finally, one can hope that provinces will compete as enthusiastically in the field of psychiatric services as they do on the sports fields!With the change of government philosophy, democratization has become an important item on the agenda.In this regard self-advocacy is an important concept, which refers to psychiatric patients speaking out for themselves.Cooper and Hersov (in Brandon, 1991, p. 127) say self-advocacy "includes independence, freedom of choice, self-expression, and group awareness...So self-advocacy is not only about learning sets of skills, but also changing relationships and attitudes".Coupled with a parmership between professionals who advocate on behalf of clients, and citizens who advocate on behalf of the less powerful, this is an important element in ensuring a more democratic health system.

PROBLEM STATEMENT
The views of consumers about the psychiatric services in this country has not been explored systematically.The reasons why their views have not been given more prominence, are many: * They traditionally have been a hidden population, unorganized, and isolated from society by their very condition.
* They were seen by many, especially professionals, as not able to speak for themselves.
* Their extreme and long-term dependence on the health system has made it difficult for them to be assertive about their rights and needs.
* The very serious stigma attached to persons with psychiatric conditions and their families keeps many articulate potential spokes persons quiet.
* There was no clear route for them to make an input into service evaluation and planning.
If the psychiatric services in this country are to improve, it is essential for consumers to take an active part (WHO, 1978).However, for them to do this, there needs to be a recognized avenue for such inputs, and they need to be empowered to take part in the process.
This research proposed that the avenue for such empowerment and involvement is the regular rating of psychiatric services by consumers.This will provide a valuable tool for the consumer movement to organize themselves, and for the health service managers to involve them constructively.

AIMS AND OBJECTIVES
The aim of this research was to develop and test a method for consumers of psychiatric services to evaluate their care systematically.The research was also aimed at empowering the consumers by organizing them, by skills development through their involvement in the project, and by the information they obtained through the research.
The objectives were: 1. To identify and describe the expectations of care consumers have of the three components, in the form of standards of care.
2. To develop a measuring instrument to assess the satisfaction of consumers with the care they receive.
3. To develop a measuring instrument or instruments to assess compliance of services with the standards.

DEFINITION OF TERMS
A consumer is a person who has had or currently has a mental illness and has used the public sector psychiatric services, as well as the family members of such a person.Only persons who are not actively psychotic will be included in the research.
A standard is a statement describing an acceptable level of performance.

LITERATURE SURVEY
Although some professionals profess to value the evaluation of their programmes by consumers (Butterworth and Skidmore, 1981), there are surprisingly few studies of this kind reported in the literature.In an exhaustive overview of all forms of evaluation by clients, from novels and films to research, Brandon (1991) quotes a few smdies which give mainly evaluations of inpatient life in anecdotal form, while evaluation by patients in the community gives attention mainly to their quality of life, rather than the services.Studies about satisfaction with a specific treatment or programme are more common, for instance satisfaction with psychotherapy (Conte, Plutchik, Buckley, Spence and Karasu, 1989;Goyne and Ladoux, 1973) or with a specific programme or service (Shields, Morrison and Hart, 1988;Norman and Parker, 1990;Long, Blackwell andMidgley, 1991 andSheppard, 1992).Lebow (1983) pointed out that these studies contribute little to the development of standard measures of a body of knowledge about programmes in general.
It should be noted that the current study is not merely a consumer satisfaction study.
Consumers were asked to look more widely than the direct service when standards were formulated, and thus, evaluation included aspects not experienced directly by consumers.
Saraceno, Frattura and Bertolote (in WHO, 1993) propose that a systematic approach to the assessment of quality of psychiatric services includes the following set of indicators; The six starred indicators sets are included in this study.
Patient satisfaction studies: Lebow (1983) describes two conceptualizations of consumer satisfaction: Narrow deflnition: All inquiries into the felt adequacy of mental health treatment itself and of the surrounding milieu; cost, availability, accessibihty of care and the reaction to supporting services.Satisfaction with process and outcome is included.
Broad definition: Inquiries which add measures that correlate highly to the above measures and which therefore are in some substantial sense, indices of satisfaction.Such indices include complaints or praise for treatment, suggestions for improvement, multi dimensional descriptions of the treatment milieu.
In a review article on consumer satisfaction with mental health treatment, Lebow (1982) identifies many methodological problems with this kind of research at that time.Reliability of instruments are seldom addressed, and the validity of the studies can be questioned on many fronts.He made many suggestions to improve patient satisfaction research.

The importance o f consumer evaluation
Except for the reasons given by Torrey et al for regular consumer ratings given under the "background" heading, Mester and Gonen (1993) contend that waste can be prevented by consumer participation in evaluation.They also contend that the standards of care can be improved by such participation, and give one example of each of these benefits from the Israeli services.
The WHO (1993) points out that an understanding of the actual performance of a service in a given context cannot be understood merely by looking at quantitative data, but is only possible by involving consumers in qualitative analysis.They point out the limitations of outcome studies, and resource utilization reviews, and state that consumer participation often challenges the assumptions underlying these studies.Consumer involve ment in evaluation also leads to improved responsiveness of services.
The value of such evaluations are not universally accepted.Lebow (1982) lists the criticism against consumer satisfaction research as including questioning the validity of such studies, the fact that it is too positive to be of use, the argument that consumers cannot judge treatment adequately, and lastly the belief that only outcomes are important, and that satisfaction is not really important.

South African consumer evaluations
In South Africa consumers of psychiatric services have had no input in the planning and evaluation of services.They have not been organized, and the health system has not been organized in a democratic way.Both of these conditions have changed recently.
In 1992 a national organization was formed in which health workers and consumers came together to address the needs of the mentally ill and their families and the National Alliance for Mental Health was formed.Growth of this organization has been slow, but branches now exist in Durban, Bloemfontein, Pretoria and Cape Town.
In their recent evaluation of the mental health services in the Orange Free State, Freeman, Tennyson and William (1994) included interviews with consumers.These interviews included aspects such as whether the clinic helped them and whether they waited in queues.
Both of these developments show a move in the right direction.

Standards of care:
According to Gillies (1989), a standard is a descriptive statement of desired level of performance against which the quality of structure, process and outcomes can be judged.
Gillies differentiates between the following types of standards: * Normative standard: which describe practices considered good or ideal.
* Empirical standard: which describe standards of actual performance in a large number of care settings.
In practice, normative standards will be higher than empirical standards, and it is more appropriate for professional and consumer groups to use this kind of standard.
The setting of standards is seen as one of the first steps in quality improvement However, in the usual quality improvement process, standards for care are set by professionals.The American Nurses Association's (ANA) Council on Psychiatric and Mental Health Nursing has published the standards of care they expect from psychiatric/mental health nurses (1994).In this document, each of the eight standards is in the form of a nursing function, which is then elaborated by listing the "measurement criteria".
In a very recent stody in the UK, consumers were involv^ in the auditing of mental health services (Balogh and Bond, 1995).The Newcastle Audit System was developed by involving consumers in generating material in small informal groups, to represent the five audit categories.Consumers were also included in the multi-disciplinary audit teams.The research team considered the benefits from including consumers as follows: * clients' views were represented at all points in the audit cycle; * the views of clients in the care setting were collected by client representatives rather than hospital staff; * clients' expertise could be used; * the experience of collaboration was beneficial for all audit team members" (Balogh and Bond, 1995, p.27).
However, the authors point out that only experienced client advocates can undertake such a task, their participation has to be funded, and they cannot collect all types of data.
In South Africa, a whole range of standards have been recently developed by the Pilot South African Accreditation Programme for Health Services (Hendry,1994).In the case of Rehabilitation Standards, draft standards were based on Australian standards, and then circulated for comments amongst a large group of individuals and organizations, which included consumer groups.A similar format to that of the ANA was followed (two standards, with a list of criteria for each).
The standards set for a whole service, such as that developed by the Joint Commission on Accreditation of Health care Organizations, includes aspects such as: structure, functions, policies, procedures, and methods.

Instruments fo r consumer evaluation of services:
Larsen, Attkisson, Hargreaves and Nguyen (1979) produced a Client Satisfaction Questionnaire (CSQ) almost 20 years ago to measure consumer satisfaction.This eight-item four-point rating scale had a coefficient alpha reliability of 0,93, and it also seemed to be valid in that low scores correlated with early drop-outs and numerous missed appointments.No studies could be found in later literature which used this instrument.
The 1990 NAMI instruments consists of two extensive questionnaires (Torrey, 1994).A 29 page questioimaire sent to the chief official in each state of the USA, deals with 11 categories of information.A 22 page questionnaire to consumers or consumer groups, covering a similar range of topics, but getting the consumer perspective.
More recently, a national survey of the Australian mental health services for the seriously mentally ill was done by Andrews, Hoult and Leggat (1994).They used three questionnaires: the state directors questionnaire.
the area/regional directors questionnaire, and the consumer questionnaire.The questionnaire to the regional directors is much more detailed, and is 46 pages long.The consumer questionnaire is similar to that of the regional director but is slightly shorter (24 pages).
In 1993 the European Council of the Worid Federation for Mental Health made available a questionnaire which was developed by five people, to be used for consumer evaluation (Crepel, Glanville, van Horn, Jensen and Vendesborg, 1993).The User Evaluation of Mental Health services Questionnaire covers ten different areas in 15 pages.It seems to be aimed more at the individual consumer than either the USA or Australian questionnaires.
It would seem that some researchers advocate a more qualitative approach to consumer satisfaction research (WHO, 1993).Everett and Boydd (1994) found ethnography and triangulation methodology superior to interviewing of staff and consumers.

RESEARCH DESIGN
The research consisted of three distinct stages: Stage one : The development of an initial set of standards and criteria, using both focused groups with consumers, and the literature, and validating these with other groups of consumers.
Stage two : Developing instruments to measure adherence to these standards, and obtaining content validity for these instruments.
Stage three : Testing these instruments to establish inter-rater reliability and correspondence between items pertaining to the same standard or criterium.Investigating differences between samples to establish a convenient sampling method for future studies.

STAGE O N E : DEVELOPING STANDARDS
In this stage the expectation of care was workshopped with groups of consumers.Although two open meetings for consumen were scheduled and advertised, very few people attended these meetings.It was decided then to use outpatient psychiatric clinics for the workshops.This method was successful, and six groups of consumers participated in this way.The groups included of 66 patients and 11 family members or significant others.The three clinics serves all sectors of the population.
The participants were assisted to express their own expectations of care.These were organized around the three service areas; outpatient clinics, acute units, and long term units.After having listed all these on poster sheets, they were presented with the standards developed from the literature, and they were asked whether they agreed with these.This typically elicited more discussion, which also was written down.
The input of the participants were then collated and integrated with the standards from the literature.The standard list then had 13 standards and 127 indicators.
Validation: The standards were then validated with comparison groups of consumers, using both similar and contrasting groups in terms of urban/rural and cultural background as criteria.Groups validating standards were asked to answer the following questions with regards to each standard: * Is it clear?* Is it important?They will also be asked whether the full set of standards include all their concerns, or whether they would like to add standards.Three groups were used for validation: one in rural KwaZulu, which consisted of 30 patients and four family members attending a psychiatric clinic.The second was a group of 12 patients and eight family members, who attended a community meeting in Eersterus, Pretoria.The third was a consumer support group in Cape Town.
Much of the input from all three groups were in support of the standards, although two consumers groups stated that they did not understand what happened at management level, and were not very concerned about these standards.One of these groups stated that professionals should comment on these.The third group supported these, and it was decided to keep them in the final list.A few criteria were added by these groups, but the standards were accepted as they were.See Addendum A: list of standards.

STAGE TW O: INSTRUMENT DEVELOPMENT
Measuring instruments were then developed by the researchers to test each standard and criterium.This included the following; * A questionnaire to consumers covering those aspects of the standards which consumers can evaluate based on their experience, e.g.approachability of staff and accessibility of services.
* A questioimaire to service providers, in the shape of the Director of Mental Health of the province, about information which can best be 'obtained from service providers, e.g.statistics and financial data.
* Two site visit schedules which stipulate aspects of care which observers would evaluate during a visit to either a clinic or a hospital unit, e.g.programmes available.

Consumer questionnaire:
This questionnaire, aimed at both patients and their family members has a total of 68 items and starts with a demographic section of 10 items.
The rest of the instrument consists of seven broad sections: This instrument was based on the Group Health Association of America (GHAA) Consumer Satisfaction Survey instrument.This instrument and its manual was published first in 1988, and a revised edition in 1990 (Davies and Ware, 1991).It was developed as an easy-to-use instrument to compare the satisfaction of consumers enroled in different health benefit options.The scale gave a internal consistency reliability based on Cronbach's Alpha of 0,80 to 0,97 for different subscales.The validity arguments centred around content validity, predictive validity (scores predicted the recommendation of the service to others) and construct validity (distinct ratings on different subscales).
This instrument was adapted quite extensively with sections about health insurance left out, and a section on Incidents added.In the other sections nine items were left out, and 9 others added.Wording in many items had to be changed slightly, to make provision for South African service conditions, e.g."Convenience of location of the doctor's office" was changed to " Convenience of location of the psychiatric clinic/outpatients".
For testing in the province, the English version was translated into another official language by two independent translators.It was then given to the third person, who selected the best version of each item for the final translated questionnaire.

Questionnaire for Provincial Directors of Mental Health
This questioimaire was broadly based on the Questionnaire for Area/Regional Directors of Mental Health, which was developed for the study Rating Australian Psychiatric services.
This 15 page questionnaire was shortened to six pages by cutting out items which addressed detail not available in the current system, such as the number of seriously mentally ill patients as opposed to all others.Furthermore, sections dealing with special need patients, such as the homeless and "non-English speaking background" were omitted, since they seemed inappropriate.
Unfortunately, no reliability or validity data are available yet about this instrument The questionnaire is divided into the following sections: Each criterium was scored out of 10 and these scores were combined to produce the total score for that category, also out of 10.The total score was the geometric mean (GMy=n yl y2 y3...y) of all the criteria, in order not to mask low scores, and not allow high scores to have too great an influence.
In order to determine the individual scores for each criterium, several "levels of achievement" were defined for each one.The best score was always 10 out of 10, but the poorest score could be anything from 0,01 to 8 out of 10.These scores were dependent on the importance of the criterium in overall quality as defined in the standard.The items depended on direct observation at primary health care sites.
In the instruments used in this study, criteria were presented in categories for face validity of the instrument, but these categories were not .

INNOVATIONS AND RESEARCH 1 1 CONTINUITY OF CARE 7
.
MANAGEMENT 3 2 used to analyze data.Rather, the items were related back to the relevant criteria and standards in the "STANDARDS FOR PSYCHIATRIC CARE" document The structure of the site visit schedules are summarized in Table 1.While most of the items are dependent on direct observation, some data comes out of chart reviews, and some questions are asked of patients and staff. Validation: These instruments were then validated by getting an expert group together, consisting of members of the multi-professional team.The group consisted of the following persons: * Two psychiatric nurses working in different community psychiatric services.
* A clinical psychologist working in a community mental health project.
* A psychiatric nurse working in an inpatient setting, who is also the family member of a mentally ill person.
This group were asked to assess the following: * Is every criterium adequately evaluated by the items in the data collection instruments which refers to it?* Are the instruments and the instructions for completing them, clear?
They were also asked to give weights to the site visit items.This team changed three items and deleted four.
Although changes to standards and criteria by the expert team were not part of the plan, it was felt that this would be accepted.The team could see how these statements interacted with the measurement, and their changes increased the reliabihty of the measurement.Furthermore, there has been some criticism that professionals were not given adequate opportunity for input into this study, and this was seen as an opportunity to rectify that.
The team deleted two items from the instruments, and added eight.They then thought that the instruments covered the standards and criteria effectively.not strictly a provincial hospital, one of the Lifecare sanatoria was included in the sample, since it forms an essential part of the public psychiatric service in the province.A list of random numbers was then used to select a 10% sample of these, that is, two acute and five long term units.Table 2 sets out the distribution of the units selected.All the units were included in the sample, except for the acute unit in the general hospital, whose staff refused to participate.Their objections will be discussed under "Ethical aspects".

CLINICS:
The list of clinics in a document prepared by a provincial working group for the integration of services was used as the sample frame.In this list the clinics were divided into the following groups; hospital outpatient departments (51), fixed clinics (323) and mobile clinics ( 238).This gives a total of 612 clinic points.
It was decided to use a stratified random sample with 1 hospital OPD, 6 clinics and 3 mobile sites selected using a table of random numbers.This is quite a small sample (about 2%), but due to the random distribution, data-collection is too time-consuming to use a bigger sample.There were three main problems with sample realization: * All the mobiles in the random sample were very far from the capital, and when the researchers made plans to visit them, it became clear that one could travel 5 to 6 hours to attend a mobile clinic only to find that it was not going out due to the transport not being available, or the roads being too bad during the rainy season.Table3 sets out the clinic sample plan.It was decided to replace the mobile clinics with fixed clinics, and send the mobile clinics consumer questionnaires only.
Many of the fixed clinics saw psychiatric patients only one or two days per week.This made scheduling visits very difficult.
The field worker could not drive, and this meant that a driver had to be organized for each visit.She also had UNIS A examinations during the data collection time, which further complicated scheduling.Clinics closer to the researchers therefore were included for visits.

CONSUMERS:
The sampling plan for the consumer sample was changed quite a number of times.Initially it was envisaged to use a different set of clinics to access consumers from the clinics used for the clinic sample.The idea was to provide greater coverage of a very large province with a heterogeneous service.However, it was soon evident that this would be too time-consuming, and the clinics selected for the consumer sample were also used for the clinic sample.In each clinic as many of the consumers present on the day of the visit were to be included in the sample.
A list was made of all long term ( 45) and all acute (20) psychiatric units in the province.Although  It was envisaged also that a "hospital sample" would be obtained by obtaining the addresses of recently discharged patients from hospital records.The procedure around doing this in an ethical way, however, proved too onerous, and it was decided to accept the community sample only.
Voluntary sample: It seemed important to establish whether consumers who volunteer to fill in questionnaires differ significantly from those who are approached at clinics.A "voluntary" group of consumers was therefore also included in the study.The questionnaires were distributed to all clinics that can be reached, and consumers were also be alerted to the study through the radio and newspapers.The realization of this group is also reflected in Table 3.

DATA COLLECTION PROCEDURE
Data collection stretched over four months.
The audit team consisted of at least a health professional and either a consumer or another community member.Community members included in the audit team over this period included a business man, a retired nurse, and a physically disabled teacher.
The audit team each filled in their own instrument, but then they discussed the ratings afterwards, and came to a consensus score.Differences could often be attributed to one person seeing something that another missed.

ETfflCAL ASPECTS
Permission for the service survey had to be sought and obtained from health service authorities in the province.
Consumers who participate in the workshops were protected by anonymity in the sense that no names were used in research reports.However, since they participated in person in workshops and in advocacy, anonymity will not be complete.Participation was therefore to be voluntary in both of these activities.
Consumers who completed the questionnaires were more of a problem: Since the researchers would be present in clinics, in a sense it required the services to break the client confidentiality by givingthe researchers access to the identities of the patients served.However, the researchers and/or staff members asked the consumers' permission to interview them, and were never refused, except when the consumers had to hurry off for public transport.
For the site visits, all team members were asked to sign a declaration to keep all personal information totally confidential.However, only the professional did the record reviews.However, obtaining permission for especially the hospital unit visits was problematic.
Researchers were required to apply for ethical permission through the ethical committees of two of the four hospitals involved in the sample.
Since each has its own set of forms, and committees sit irregularly, this took much time.Furthermore, hospital authorities and the staff involved in the units showed much anxiety about being evaluated, especially by "outsiders".The letter from the psychiatrist from the unit which finally refused entry illustrates these fears: "The assessment of the unit is by its content an audit of the psychiatric facility and seen as having the potential for extensive adverse publicity." This fear of adverse publicity seemed to be the result of recent and consistent poor publicity about psychiatric services, and administrators saw this study as just one more attempt to discredit them.
Another aspect that was debated hotly, was the access to client records which the study demanded.During each of the site visits five client records were audited, to establish for instance whether a full history was done, and a full diagnosis given.Staff felt that this contravened the principle of privacy, although the permission of patients were obtained.It was felt that patients, especially in acute units, were not capable of giving informed consent.
This is true of course, but it should be remembered that the same patients may give permission legally for their admission to a psychiatric hospital.There therefore is a precedent for accepting the permission of a patient, even if his/her mental state is not up to the level of perfect understanding.The argument has been that admission with consent is beneficial for the patient, since it makes certification urmecessary.In the same way, one can argue that this study is to the benefit of consumers primarily, and therefore the patients' permission can be accepted.
The extreme defensiveness of some of the staff raises the question of whether it really is the patients they are defending.The extent of their power to block access to data is disturbing.
In terms of the whole research process, consumers will continue to be given maximum control over the process and the data.

DJTER-RATER RELLVBILITY
The clinic and hospital observation schedules were seen as a single instrument in establishing interrater reliability.The Spearman Rand correlation was calculated and a very good correlation of 0,9372 was established.
The Questioruiaire for the Director had to be coded, since simple yes/no answers were not possible.A coding system was developed.Since only one such a questionnaire was part of this stage of the research, inter-rater reliability could not be calculated.Instead, two raters used the coding system independently, and coded eleven responses and four differently.The coding system was then elaborated to improve clarity on those items.
Coding guidelines were worked out for the two site visit schedules, and in analyzing the data, it was found that a number of items did not work well.
The most common problem was that an item would refer to more than one criteria, so that it was impossible to distinguish between achievement on different criteria.This made repeated coding necessary.To remedy this problem, four items were improved in the final instruments.
In one case it was found that three items actually referred to the same criterium, although it was separated into three criteria in the standards list.
In this case the criteria were integrated into one.
It would seem that the problems in the service were identified well using the set of instruments.It identifies and quantifies limitations and strengths, and could be used to target interventions.

DIFFERENCE BETWEEN VOLUNTARY AND CLINIC SAMPLE OF CONSUMERS
There were 107 consumers in the clinic sample, and 38 in the voluntary sample, giving a total of 145.The Chi-square was calculated for each item, and since there were too many empty cells in some cases, some categories were collapsed to make calculations possible.Finally it was found that significant differences between the clinic and voluntary samples occurred in 16 items out of the total of 55 items.This represents almost 30% of the items, and it would therefore seems necessary to keep using both types of samples, since they seem to tap into different groups.
The level at which this province achieved the standards was 58%.There were two categories in which performance was below 20%, and these were funding and research and development Management was below 50%, while service provision and the general category, which included staff attitudes, achieved percentages higher than 50%.There was large variation between the quality of different hospital units and different clinics.

DISCUSSION:
With regard to the methodology for the measurement of quality of psychiatric care in the public sector, the standards and instruments developed in this study seem to be satisfactory in the following ways: 1.According to Gillies' categories of standards (1989) normative standards have been set in this study.However, as baseline data is gathered from all provinces, empirical standards will be possible.
2. According to Saraceno, Frattura and Berlolote (in WHO, 1993) ten types of indicators constitute a complete assessment of a service.In this study seven of these were used.Context indicators, patient outcome and impact indicators were not included.This does mean that the evaluation methodology is comprehensive enough to escape much of the criticism against such studies quoted by Lebow (1982).
3. The standards have been validated by different consumer groups, and the content validity of the instruments measuring these standards would seem to be good.The inter-rater reliability of the observation schedules are also excellent at 0,94.The validity of the measurementis also enhanced in that the score for some criteria were derived from different data sources.The same is true of almost all standards.
4. The data collection process is neither too cumbersome, nor too expensive for a province to do this exercise every few years.
Travel costs about R4 OCX), Freepost about R400, and paying consumers a very limited amount for their involvement about R500.This means that total cost per province is about R5 000.In terms of time, each clinic takes a morning, but three hospital units can be done in a day.If one keeps the clinic sample and the hospital units sample to ten each per province, the total time necessary is therefore about two weeks maximum.
5. The involvement of consimiers and community members did not create any problems.All observers dealt easily with the observation schedules, and gave useful input into the process.They were well accepted in the facilities visited, and expressed an interest in taking the process forward.
6.The results produced are sufficiently shaded and detailed to enable consumers to lobby around specific deficiencies, and to allow health service providers to initiate corrective measures.The importance of this approach is supported by the results which show consumer participation in the psychiatric services to be one of the lowest achievements in the province described here.
One of the problems which will continue to exist is the difficult and time consuming process of obtainhig permission for the research.Although Head Offices in all provinces were relatively quick to give permission, obtaining permission from the authority under which each clinic resorts, and from each hospital involved, proved to be a difficult and time consuming exercise.Two of the hospitals in this study required that their own ethical committee approve the protocol.This increased the difficulty, since the protocol for the total smdy was not detailed enough to answer their questions about this specific phase.This process may become easier during the second round of evaluations, since the non-sensational approach followed in the reporting might allay fears in health service authorities.They might also have used the first report sufficiently to welcome the need for a follow-up.One accepts that an institution must have the authority to evaluate whether the study can be accommodated within its activities, but it is difficult to see how sending a protocol to numerous ethical committees is a economical way of using the human resources of the health service and the research sectors.
A second problem has been the dearth of some statistical data on certain topics.Although this does not preclude the study from being concluded, six criteria could not be measured in this province due to data not being known to the informant.It would be important for the establishment of provincial and national baselines that these omissions be addressed.For instance, one would need to establish nation ratio's for psychiatric beds:population, and for forensic beds:population.Further differentiation might also be necessary, such as the type of beds, e.g. in a district or regional hospital vs provincial psychiatric hospitals.
The consumer questionnaire has been prepared in two languages only for this province.It would be necessary to make it available in all eleven languages for country wide application.

RECOMMENDATIONS:
It is recommended that this technology now be used in all eight other provinces so that national norms can be established.Without such national norms, the interpretation of the data remains difficult.For instance, the question whether the 58% average achieved by this province is good, bad or mediocre cannot be answered without national data being available.
It is further recommended that a plan for the empowerment of consumers to use the results from this province and the others which will be done in fumre, be enhanced by a planned strategy.It would, for instance, be most useful if a national workshop could be held, during which the different provincial reports could be workshopped with consumers, and during which they could also get training in lobbying.
It would be very usefiil for research bodies such as Health Systems trust, and the Medical Research Council to address the question of permission for national studies from individual health service providers.This needs to be a strategy which will be sufficiently inclusive and decreases the barriers to research.