Development and preliminary psychometric properties of the Transition Competence Battery for Deaf Adolescents and Young Adults.

Development and preliminary psychometric properties of the Transition Competence Battery for Deaf Adolescents and Young Adults. Developing vocational and independent living skills is critical tothe ultimate work and day-to-day success of people with hearingimpairments. It is crucial that reliable and valid assessment data begathered to guide and structure focused and effective instructionalprograms in these areas (DeStefano, 1987; Frey, 1984; Marut & Innes,1986; Shiels, 1986; Sligar, 1983). Unfortunately, few instruments aredesigned specifically to assess the transition skills of adolescents andyoung adults with hearing impairments (Reiman & Bullis, 1987).Typical practice is to administer traditional psychometric psy��cho��met��rics?n. (used with a sing. verb)The branch of psychology that deals with the design, administration, and interpretation of quantitative tests for the measurement of psychological variables such as intelligence, aptitude, and tests (e.g.,IQ tests) or functional measures designed for other populations (e.g.,vocational skill tests developed for adolescents with learningdisabilities). People who administer such measures (interpreters orclinicians) possess varying levels of sign language competence (Levine,1974; Stewart, 1986). Based on the lack of other assessmentalternatives, this approach is logical, but the veracity of datagathered in this manner--and subsequent intervention decisions based onthese results--are questionable for two primary reasons. First, deafness is a condition defined by its unique expressive andreceptive communication modalities that differ significantly from thoseof our English-based hearing society. Many deaf people This is an incomplete list of notable deaf people. Important historical figures in deaf history and cultureThe idea that a person who was deaf could achieve a notable or distinguished status was not common until the latter half of the 18th century, when Abb�� Charles-Michel de use a bona fidelanguage (American Sign Language or ASL ASL - Algebraic Specification Language ) that has no structuralrelationship to English; that relies on visual rather than auditoryencoding and decoding; and that has a rule-governed phonology phonology,study of the sound systems of languages. It is distinguished from phonetics, which is the study of the production, perception, and physical properties of speech sounds; phonology attempts to account for how they are combined, organized, and convey meaning , syntax,and morphology (Reiman & Bullis, 1989). Educators conducting anyassessment of deaf people's transition skills should consider thisfundamental communicative difference. Can the deaf person Noun 1. deaf person - a person with a severe auditory impairmentindividual, mortal, person, somebody, someone, soul - a human being; "there was too much for one person to do" use aninterpreter in an effective manner in a job interview? Does theindividual know his or her legal rights when interacting with a policeofficer? Can the person formulate a strategy to communicate effectivelywith coworkers? Questions such as these are highly relevant tosuccessful work and living experiences. A review of published researchon measurement procedures with this population, however, reveals thattests designed for other populations do not address these crucial skillsin any systematic way, nor have investigations delineated de��lin��e��ate?tr.v. de��lin��e��at��ed, de��lin��e��at��ing, de��lin��e��ates1. To draw or trace the outline of; sketch out.2. To represent pictorially; depict.3. the particularskills and content necessary for deaf people to succeed in work andliving endeavors in the community (Bullis & Reiman, 1989; Reiman& Bullis, 1987). Second, any time the administration procedures of a standardizedassessment tool are altered, the validity of the resulting assessmentdata must be questioned. For example, consider a measure of functionalskill knowledge that was devised for use with and standardized on agroup of adolescents other than persons who are deaf (e.g., learningdisabled). If that tool is administered using sign communication inplace of verbal instructions, such substitution violates thestandardization procedures of the measure and technically invalidatesthe tool (American Psychological Association, 1985; Gerweck &Ysseldyke, 1979; Yoshida & Friedman, 1986). Consequently, theresulting data are suspect because of the absence of psychometricstandards of the measure for that type of application. There is, then, a pressing need to develop language-appropriate,content-relevant, and psychometrically sound measures of transitionskills for deaf persons. The purpose of this article is to describe thedevelopment and initial standardization data of such an instrument, theTransition Competence Battery for Deaf Adolescents and Young Adults(TCB See trusted computing base. 1. (jargon) TCB - Trouble Came Back.2. (security) TCB - (Orange Book) Trusted Computing Base.3. (operating system) TCB - Task Control Block. ) (Reiman & Bullis, 1990). PRELIMINARY DEVELOPMENT PROCEDURES Three fundamental assumptions guided the development of this testbattery. First, it was, and is, our belief that one of the majorstumbling blocks in conducting research or understanding investigationsof this group, is that all too often the deaf population is regarded ashomogeneous. Quite the contrary, the deaf population is highlyheterogeneous, encompassing people with varying levels of auditorycapabilities, linguistic skills, cognitive abilities, social skills, andemotional development. Consequently, the construction of an assessmentbattery, which is content relevant for a particular subgroup, requiresthe clear delineation of the segment of the population for which theinstrument is to be used. Based on results of previous research(Bullis, 1985), the population for whom this instrument can beconstructed can be described in the following way. Note that thisdescription parallels the "low functioning" term that is oftenused to characterize a group of deaf people who do not attend 4-yearcolleges or succeed in work or living endeavors in the community, andfor whom few services and little research are available (Bowe, 1988). Members of the subject population are devoid of a seriouslycomplicating secondary disability, but some may present mild secondaryconditions (e.g., wear glasses, heart murmur Heart murmurSound during the heartbeat caused by a heart valve that does not close properly.Mentioned in: Mitral Valve Prolapseheart murmurSee Murmur. , etc.). Further, membersof the subject population possess limited English reading skills (e.g.,read at approximately the 3rd-grade level). The subject population doesnot include persons who go on to a 4-year college or university, but mayinclude persons who attend community college or vocational/technicaltraining centers. More than likely, members of the sample are peoplewho either drop out of high school, seek employment immediately afterleaving high school, or go on to some type of rehabilitation rehabilitation:see physical therapy. orcommunity-based training program. Finally, members of the subjectpopulation have little experience or training in employment andindependent living skills. Second, although multiple measures and perspectives should beemployed in the assessment of any deaf person, this process should beginwith an examination of the individual's knowledge of requisitetransition skills to work and live successfully in the community.Knowledge of how to behave is a necessary foundation of behavior(Bandura ban`dur´an. 1. A traditional Ukrainian stringed musical instrument shaped like a lute, having many strings. , 1977), and studies suggest that knowledge of functional skillsis correlated to actual skill performance for people with mild cognitiveimpairments (Bullis & Foss, 1986; Landman, Irvin, & Halpern,1980). Third, if assessment is to be connected to transition instructionor training, it is critical that the measurement tools represent thecontent of the particular domain of concern. That is, measures offunctional skills should be composed of items that adequately sample theknowledge and skills necessary for the deaf person to succeed in his orher transition from the school to the community. It follows that theparameters of the transition domain for the target group of deaf personsmust be clear. Despite some controversy (Clark & Knowlton, 1988;Rusch & Menchetti, 1988), we believe that transition is bestrepresented by two broad domains of community-based out-comes:employment and independent living skills (Bullis, Bull, Johnson,Johnson, & Kittrell, 1990). Given these assumptions, we adopted a domain sampling model of testconstruction (Nunnally, 1978). In this approach, we determined thecontent definitions of the employment and independent living domains forthe target population. We then used these boundaries as a blueprint fortest construction and generated test items within these boundaries. Asthe final part of the development process, we developed and tested aprototype form of the measure. The following sections describe each ofthese steps. Specification of the Test Battery's Content Blueprint There have been extensive writings on employment and independentliving for deaf persons, but relatively little research exists on thespecific skills comprising these areas. Comprehensive reviews ofliterature in this field since 1975, on both transition and assessment(Bull, Bullis, & Sendelbaugh, 1987; Bullis, Bull, Sendelbaugh, &Freeburg, 1987; Reiman & Bullis, 1987), reveal only a generaldescription of the particular problems, concerns, and issues facing deafpeople in transitions from school to community, or in communityadjustment. Accordingly, it was necessary to first identify theparticular skills necessary for a deaf person to successfully work andlive in the community. We conducted a two-step procedure to establishsuch a skill taxonomy (Bullis & Reiman, 1989). First, we held a workshop with 18 professionals from the Northwestin the field of deafness to identify critical work and independentliving skills for the target population. The workshop employed theNominal Group Technique The nominal group technique is a decision-making method for use among groups of many sizes, who want to make their decision quickly, as by a vote, but want everyone's opinions taken into account (as opposed to traditional voting, where only the largest group is considered). (NGT NGT NightNGT National Grid Transco (UK gas transporter)NGT Nominal Group TechniqueNGT Not Greater ThanNGT Next Generation TechnologyNGT Next Generation Telecom (China)NGT NASA Ground Terminal ) (Delbecq, van de Ven, & Gustafson,1975), a structured group interaction method, to answer two questions. 1. What are the five most important employment-related skills fora member of the target population? 2. What are the five most important independent living-relatedskills for a member of the target population? We grouped the lists of skills generated by workshop participantsinto three employment-related subdomains (job-seeking skills, workadjustment skills, and job-related social/interpersonal skills) andthree subdomains related to independent living (money management, healthand home, and community awareness). Second, these skill areas were evaluated in a national survey ofpractitioners and leading "experts" in the field of deafness,a technique designed to ensure broad geographical and social validity ofthe skills (Kazdin, 1977). Respondents rated each skill on two 4-pointLikert scales in terms of importance (its importance to the ultimateemployment or independent living success of members of the targetpopulation) and on presence (the percentage of persons in the targetpopulation possessing the competency). A total of 307 deaf and hearing service providers (representingresidential and mainstream schools, community colleges, andrehabilitation programs) completed the survey. We conducted analyseswith this data set to identify the most critical transitioncompetencies, from the perspective of professional opinion, for anindividual in the target population to possess in order to succeedworking and living independently. To complement and modify the contentdefinition drawn from both the NGT and the national survey, we used theliterature reviews to clarify, expand, and condense con��dense?v. con��densed, con��dens��ing, con��dens��esv.tr.1. To reduce the volume or compass of.2. To make more concise; abridge or shorten.3. Physicsa. this listing. Onthe basis of this empirical and conceptual examination, we generated aworking content blueprint for the TCB. Table 1 shows the six subdomainsand their associated content areas for the employment domain; Table 2shows the subdomains and content areas for the independent livingdomain. These six subdomains of transition for this part of the deafpopulation eventually became the six subtests of the TCB. The contentareas provided the framework used to structure the item-generationactivities. Generation of Test Items We assembled a second group of 20 professional service providers(hearing and deaf) for training on test item construction. We decidedthat persons with direct experience with the target population in workor independent living programs, and who possessed a working knowledge ofsign communication (ASL, Pidgin pidgin(pĭj`ən), a lingua franca that is not the mother tongue of anyone using it and that has a simplified grammar and a restricted, often polyglot vocabulary. Sign English, and Manually CodedEnglish Manually Coded English (MCE) is a general term used to describe a variety of visual communication methods expressed through the hands which attempt to represent the English language. ) would be able to write content-relevant and understandable testquestions. The goal of the workshop was to teach the requisite skillsfor writing test questions in each of the previously identified contentareas. Participants were paid expenses and a fee for attending; afterleaving, they were paid a fee upon satisfactory completion of an itemwriting task. That is, each of the participants agreed to write aspecified number of test items in each of the sub-domains and theirrespective content areas. [TABULAR DATA OMITTED] In the workshop, we placed particular emphasis on defining thosecategories in which TCB items would be written--the knowledge,comprehension, and application categories (Bloom, 1956)--in the form of3-option, multiple-choice questions. The format was chosen to minimizecorrect answers due to guessing (Nunnally, 1978) and because 3-optionmultiple-choice questions have been demonstrated to be valid for usewith adolescents and young adults with mild cognitive disabilities(Bullis & Foss, 1986; Landman et al., 1980). Workshop participantsdiscussed measurement studies relative to the deaf target group, basedon an earlier review of empirical studies Empirical studies in social sciences are when the research ends are based on evidence and not just theory. This is done to comply with the scientific method that asserts the objective discovery of knowledge based on verifiable facts of evidence. (Reiman & Bullis, 1987);then they listened to a presentation detailing rules for construction ofmultiple-choice items (Gronlund, 1977). The presenters identified andcautioned participants about the use of certain linguistic structures(i.e., conditionals, minimal information pronouns, comparatives,negatives, etc.) of potential difficulty for the target population.Then the presenters introduced and reviewed lists of words commensuratewith a 3rd-grade reading level (the TCB target level). Finally, theworkshop included highly structured practice time for item writing, andall participants wrote and critiqued practice sets of questions.[TABULAR DATA OMITTED] Six weeks after the workshop, participants submitted more than 900test items that we subsequently edited for content, duplication, andadherence to item-construction criteria. Editing involved eliminatingor significantly modifying inappropriate or inaccurate items anddistractors. At the conclusion of the editing process, we ended up withslightly more than 200 items that were distributed across the sixcontent sub-domains and that addressed each of the content areas.Figure 1 presents two examples from the item pool. Pilot Test of a Written and Signed (Video) Format To truly measure the subject population's knowledge of TCBcontent--and not merely reflect their English language English language,member of the West Germanic group of the Germanic subfamily of the Indo-European family of languages (see Germanic languages). Spoken by about 470 million people throughout the world, English is the official language of about 45 nations. capabilities--acombined written and signed administration approach seemed appropriate.Because few guidelines exist on which to base such a tool, we carefullyexamined the veracity of this administration approach. As a place tostart, we decided that a small-group (n=6 to 8) procedure that employedvideotaped, signed directions, coupled with a simply worded andillustrated test booklet, would be most expedient and cost effective.The idea was that a group of deaf persons could be shown the questionsand their respective responses on the monitor and read the question inthe test manual. On an individual basis, they would mark the correctanswer on a separate answer sheet. After a prespecified length of time,the entire group would then be administered the next question. From the edited item pool, we selected a subset of 30 test items.These items were representative of multiple presentation styles (e.g.,positioning and size of signer and character generation) and varyinglevels of reading complexity. (Note: Some items required students toread actual bus schedules or recipes because these are skills requiredin the "real world." In these instances, the reading level ofthe items was not limited to the 3rd-grade level.) We randomlypositioned correct answers and distractors for the 30 items, and wedeveloped written materials, including a test booklet and separateanswer sheet. We produced a structured process-evaluation form tosystematically query subjects regarding their understanding andsubjective experience of both the written and signed pilot instrument. Next, we developed a videotaped version of the 30 test items inAmerican Sign Language (ASL). A certified interpreter signed thequestion stem and the three possible responses. For many items, thesalient information contained in the stem or the responses wasreproduced using character generation, which appeared simultaneouslywith, and just to the left of, the signer. We administered the pilot test to 36 hearing-impaired subjectslocated in three sites: main-stream high school juniors and seniors, n= 8; residential high school seniors, n = 16; and community college deafprogram students, n = 12. The item difficulty (i.e., the percentage ofthe subjects who answered an item correctly) for the pilot test itemsranged from 13.9% to 97.2%, with an average item difficulty of 64.49%.The internal consistency In statistics and research, internal consistency is a measure based on the correlations between different items on the same test (or the same subscale on a larger test). It measures whether several items that propose to measure the same general construct produce similar scores. reliability index for the entire measure was.70. Because the items comprising the pilot test were drawn across thesix content areas of the TCB, the internal consistency reliability indexis lower than to be expected in a test of homogeneous content. Following each administration of the pilot test in the varioussites, we conducted a process evaluation. Subjects responded, through afacilitated group discussion format, to questions relating to relating torelate prep → concernantrelating torelate prep → bez��glich +gen, mit Bezug auf +accboth thecontent and administration procedures of the instrument. Overall, thesecomments were positive regarding the relevance of the test content andthe level at which it was presented, but two very important points wereapparent across groups. First, the number of subjects reportingdifficulty in understanding ASL raised serious questions regarding theviability of this language as the choice for the TCB. Second, subjectsreported annoyance with the videotape's moving too slowly. Boththe length of the countdown and the response time between items weredistressing for some students who, having answered the question, wereready to move to the next question without a prolonged waiting period.A clearly visible result of this situation was restlessness andpotentially disruptive behavior on the part of the early finishers. Wecarefully considered each of these issues in developing the next versionof the TCB. Development of the TCB From the results of the pilot test, we decided that the videotaped,signed version of the TCB should be presented in Pidgin Signed English(PSE PSE1. pale soft exudative pork.2. portosystemic encephalopathy. ) (sign communication using primarily English word order with ASLsigns and ASL grammatical features). The reported difficulties with theslow-moving videotape necessitated a reexamination re��ex��am��inealso re-ex��am��ine ?tr.v. re��ex��am��ined, re��ex��am��in��ing, re��ex��am��ines1. To examine again or anew; review.2. Law To question (a witness) again after cross-examination. of the timingpatterns within and between test items. At issue was the need to allot al��lot?tr.v. al��lot��ted, al��lot��ting, al��lots1. To parcel out; distribute or apportion: allotting land to homesteaders; allot blame.2. increments of response time that would be reasonable given theheterogenous (spelling) heterogenous - It's spelled heterogeneous. cognitive, linguistic, and intellectual abilities of eachgroup of six to eight subjects being tested. Based on staffobservations during the pilot testing of the actual time subjects usedto respond to each item, we adjusted the timing patterns of thevideotaped form downward and gauged the patterns to respond to themiddle on a continuum of subjects' demonstrated needs. Finally, wemaintained the group-administered, 3-option multiple-choice format forthe test battery. All subtests were produced in a professional recording studio usinga certified sign language interpreter and extensive use of charactergeneration on the monitor to highlight key aspects of the test questionsand their responses. The items in each subtest were randomly assigned aposition, and each item's responses (the correct response and thetwo distractors) were randomly assigned to be either "a,""b," or "c." The six subtests are listed below withthe amount of time required to administer each and their respectivenumber of items. * Subtest 1: Job-Seeking Skills for Employment (54 min:28 s; 38items) * Subtest 2: Work Adjustment Skills for Employment (40:45;32) * Subtest 3: Job-Related Social/Interpersonal Skills forEmployment (34:38;27) * Subtest 4: Money Management Skills for Independent Living(36:30;23) * Subtest 5: Health and Home Skills for Independent Living(42:39;33) * Subtest 6: Community Awareness Skills for Independent Living(37:34;28) STANDARDIZATION OF THE TCB The standardization of the TCB was complicated by a practicalissue. To administer the entire test battery required 2 to 3 days oftesting for 2 to 3 hr per day. Such a time commitment for schoolpersonnel, coupled with diversion of student time from classes,represented a very demanding commitment. Consequently, some sites agreedto administer only the employment tests or only the independent livingtests. Thus the sample size on which analyses were conducted variedsomewhat across subtests. Between 181 and 230 subjects, representing 14different sites across the United States United States,officially United States of America, republic (2005 est. pop. 295,734,000), 3,539,227 sq mi (9,166,598 sq km), North America. The United States is the world's third largest country in population and the fourth largest country in area. , participated in thestandardization of the test battery. The majority of the deaf personswho took the subtests were male (56% to 58%), were from residentialschools (53% to 74%), were deafened prelingually (before age 3) (79% to84%), and were between 18 and 19 years of age at the time of testing(means 18.69 to 19.07). Data were analyzed through the reliability anditem analysis programs of the Statistical Package for the SocialSciences (statistics, tool) Statistical Package for the Social Sciences - (SPSS) The flagship program of SPSS, Inc., written in the late 1960s.["SPSS X User's Guide", SPSS, Inc. 1986]. (SPSS A statistical package from SPSS, Inc., Chicago (www.spss.com) that runs on PCs, most mainframes and minis and is used extensively in marketing research. It provides over 50 statistical processes, including regression analysis, correlation and analysis of variance. Inc., 1989). The rest of this section provides anoverview of the preliminary psychometric characteristics of the TCB.[TABULAR DATA OMITTED] Item Statistics Two different kinds of item statistics were computed for eachsubtest: item difficulty, or the percentage of subjects who answeredeach item correctly; and point-biserial or item-total correlations thatindicate the relationship of each subtest item to the total subtest. Asa statistical guideline for retention in the TCB subtests, items were topossess a point-biserial correlation of at least .2 and a moderate levelof difficulty (between .4 and .8). Also, we decided that to beretained, each item should be conceptually appropriate. Thus, bothempirical and logical criteria were used in making the final contentdecisions for the subtests (Nunnally, 1978). By applying these standards, we deleted a total of 18 items fromthe subtests, as follows: from Subtest 1, 5 items; Subtest 2, 1;Subtest 3, 1; Subtest 4, 3; Subtest 5, 4; and Subtest 6, 4. Thesedecisions also reduced the amount of time required to administer eachsubtest to the following levels. * Subtest 1: Job-Seeking Skills for Employment (48 min: 13 s; 33items) * Subtest 2: Work Adjustment Skills for Employment (39:48; 31) * Subtest 3: Job-Related Social/Interpersonal Skills forEmployment (33:35; 26) * Subtest 4: Money Management Skills for Independent Living(32:27; 20) * Subtest 5: Health and Home Skills for Independent Living (38:34;29) * Subtest 6: Community Awareness Skills for Independent Living(33:12; 24) The third column of Table 3 presents the average item difficulty ofeach of the subtests, and the fourth column presents the averageitem-total, or point biserial Bi`se´ri`ala. 1. In two rows or series. , correlations for each subtest. Theaverage p values are from .507 to .725, and the average point-biserialcorrelations ranged from .247 to .404, values consistent with thepreviously set psychometric criteria. Subtest Characteristics The average score of each subtest and its standard deviation andthe average percentage of the questions answered correctly for eachsubtest are shown in columns 3 and 4 of Table 3. The standardizationsubjects tended to score lower on Subtests 4 and 5, at least in relationto the other four subtests. There are two possible explanations for thegroup's lower performance on these two subtests. First, thesesubtests focus on issues related to living and functioning independentlyin the community. It is well known that deaf adolescents are affordedcareer and vocational preparation in school settings (Ouellette &Dwyer, 1985), but independent living skill training is a less developedinstructional option (Ouellette & Loyd, 1980). Second, in reviewingthe subtest items, it is clear that some problems required the studentto read and understand a practical math or personal care skill (e.g.,calculate the savings when there is 15% off an item's cost, read aprescription and indicate how often the medication should be taken) andrequired a higher degree of English reading competence. Although we didnot--and do not want--to assess deaf persons' reading levels, wedid want to measure their functional academic survival skills, whichrequired that items reflect actual math and reading tasks they wouldface in community settings. [TABULAR DATA OMITTED] Reliability Two types of reliability indexes were computed for each of the TCBsubtests. Coefficient alpha internal consistency reliability indexesare presented in the first column of Table 4. This reliabilitycoefficient is regarded as a measure of the test's contenthomogeneity HomogeneityThe degree to which items are similar. , or the way the test items interrelate in��ter��re��late?tr. & intr.v. in��ter��re��lat��ed, in��ter��re��lat��ing, in��ter��re��latesTo place in or come into mutual relationship.in to one another. Fiveof the six subtests possess indexes above .75, a level generallyregarded as acceptable for group tests of this type (Salvia salvia:see sage. salviaAny of about 700 species of herbaceous and woody plants that make up the genus Salvia, in the mint family. Some members (e.g., sage) are important as sources of flavouring. &Ysseldyke, 1988). Subtest 4, on money management, exhibits a lowerinternal consistency reliability than the others, a result that is mostprobably due to its shorter length (20 items) and the lack ofvariability of its score distribution (Nunnally, 1978); that is, themajority of the persons taking the subtest tended to score low, thusskewing the distribution of scores. We also conducted a study of each subtest's test-retestreliability. Specifically, deaf students representing programs who werewilling to participate in this study, took either the three employmentsubtests or the three independent living subtests at one time. The samesubjects took the same subtests again 2 to 4 weeks later, and the twoscores were then correlated. Sixteen deaf persons representingmainstream and community college programs (9 males and 7 females, 13 ofwhom were deafened prelingually, with an average age of 20.387) wereinvolved in the test-retest study of the employment subtests.Twenty-eight deaf students from residential school programs (15 malesand 13 females, 25 of whom were deafened prelingually, with an averageage of 17.283) participated in the test-retest of the independent livingsubtests. The second column of Table 4 provides the test-retestreliability indexes calculated for each subtest. Again, Subtest 4exhibited the lowest reliability index, probably for the reasons listedpreviously. Validity In contrast to reliability, validity is a test property that mustbe established over time through repeated studies and in various ways(Messick, 1989; Nunnally, 1978). In this project, we were able toaddress only the content and construct validity construct validity,n the degree to which an experimentally-determined definition matches the theoretical definition. of the TCB. [TABULARDATA OMITTED] Nunnally (1978) stated that the content validity content validity,n the degree to which an experiment or measurement actually reflects the variable it has been designed to measure. of a measure isdemonstrated best through the procedures followed in its development.That is, the steps followed in identifying and sampling dictate in largepart whether or not the test adequately samples content from the domainof concern. Given the extensive procedures we followed to develop acontent matrix across the employment and independent living domains andto generate content-relevant test items, it can be judged that thecontent of the TCB and its six subtests is valid. The construct validity of assessment instruments is a very complexpsychometric property. Recently, experts in the field of measurement(Kerlinger, 1986; Messick, 1989; Nunnally, 1978) have taken the positionthat construct validity is the most important type of validity for atest to possess. Essentially, construct validity is demonstrated by theway the assessment instrument or instruments correlate to a theoreticalmodel of the relevant construct that is being measured. Table 5 showsthe intercorrelations of the TCB subtests and their correlations withpertinent demographic variables for the group of subjects (n = 158) thattook the entire test battery. Statistically significant, but weakcorrelations are exhibited among gender, type of school program(mainstream vs. residential), age, and subtest performance. A pattern oflow, negative correlations is apparent describing the relationshipbetween pre- and postlingual hearing loss postlingual hearing lossPostlingual deafness Audiology Hearing loss that follows the onset of speech; it is less severe but more stable and more common than prelingual hearing loss; PHL affects 10% of the general population by age 60, 50% by age 80; most PHL is (hearing loss before age 3 ascompared to hearing loss after age 3) and subtest performance. Theseindexes are not strong, but do suggest that successful performance onthe TCB is predicated somewhat on early exposure, and assumed greaterproficiency, in English. These results support the notion that certaindemographic variables were not pertinent to test performance, but thatexperience with English was related to test performance--at least tosome degree. A second method to provide initial evidence of the TCB'sconstruct validity used a social comparison approach (Bellack &Hersen, 1988; Bolton, 1987; Kerlinger, 1986; Nunnally, 1978; Wiggins,1973). In this technique, the researcher theorizes that a subject groupfor whom the particular measure was not constructed will perform in avery different way on the test than a subject group for whom the measurewas designed. Results confirming the hypothesis provides evidencesupporting the measure's construct validity. To establish such a criterion group for our purposes, we contactedthe deaf student organization at Western Oregon This article is about the region of Western Oregon. For the University, see Western Oregon University. Western Oregon is a geographical term that is generally taken to apply to the portion of the state of Oregon that is west of the Cascade Range. State College to recruit13 deaf undergraduate students to participate in this investigation. Wehypothesized that this group would score differently, that is, higher,on each of the six TCB subtests than would members of the targetpopulation because of abundant enculturation enculturationthe process by which a person adapts to and assimilates the culture in which he lives.See also: SocietyNoun 1. enculturation experiences, higher Englishlanguage performance, and more experience in work and independent livingendeavors. [TABULAR DATA OMITTED] To compare the college group's performance on the TCB withthat of the target population, two groups of residential subjects andtwo groups of mainstream subjects (each consisting of 13 subjects) wererandomly selected from the standardization pool of subjects. Twodifferent sets of planned orthogonal contrasts (Klockars & Sax (Simple API for XML) A programming interface (API) for accessing the contents of an XML document. SAX does not provide a random access lookup to the document's contents. It scans the document sequentially and presents each item to the application only one time. ,1986) were conducted between the college group's scores on each ofthe subtests and the scores of the other groups. In this type ofanalysis, the comparisons among groups are planned in advance by theresearcher and conducted according to according toprep.1. As stated or indicated by; on the authority of: according to historians.2. In keeping with: according to instructions.3. weightings established to examinepertinent, meaningful differences among groups. To achieveorthogonality (independence) of the comparison, the weightings must sumto 0 (Keppel, 1982). The two sets of contrasts used in this study areshown in Table 6. For each of these sets of comparisons, the effectsize, or magnitude, of the resulting statistic (Cohen cohenor kohen(Hebrew: “priest”) Jewish priest descended from Zadok (a descendant of Aaron), priest at the First Temple of Jerusalem. The biblical priesthood was hereditary and male. , 1988) wascomputed. The effect size was calculated as the mean of one group minusthe mean of the other group divided by the mean of the standarddeviations for both groups. Contrast Set 1 examined the difference in performance between thecollege students and the average subtest scores of the two residentialgroups and the average scores of the two mainstream groups. Because wewere interested only in differences that would favor the college group,we adopted a directional hypothesis. Specifically, the hypothesis wasthat the college group's average test score on each subtest wouldnot be greater than that of the average of either the residential ormainstreamed groups. To control for Type I errors, a correction for thealpha level used in each set of comparisons was employed (Keppel, 1982).Specifically, when a number or "family" of comparisonspertaining to a certain question are made, the alpha level adopted forthe set of comparisons is called the "familywise" error rate.The alpha level is then divided by the number of comparisons toestablish the per-comparison alpha level. For this first set ofcontrasts a familywise error rate In statistics, familywise error rate (FWER) is the probability of making one or more false discoveries, or type I errors among all the hypotheses when performing multiple pairwise tests[1][2]. of .05 was chosen, so theper-comparison alpha level for the two contrasts was then set at .025(.05/2). These results are shown in Table 7. Contrast Set 2 examined the difference between the collegegroup's scores on each of the subtests as compared to each of thefour randomly constructed groups. Again, we were concerned only withscore differences that would favor the college group, so again adirectional hypothesis was tested. That is, the college students'average performance on each subtest would not be greater than thatachieved by that of each of the other group of subjects. A familywiseerror rate, or alpha level, of .05 was chosen. Because four plannedcomparisons were conducted for each subtest, the per-comparison alphalevel was apportioned ap��por��tion?tr.v. ap��por��tioned, ap��por��tion��ing, ap��por��tionsTo divide and assign according to a plan; allot: "The tendency persists to apportion blame as suits the circumstances"at .0125 (.05/4). These results are shown inTable 8. For both contrast sets, highly statistically significantdifferences were found favoring the college students' performanceon each of the subtests. Also, calculation of the effect size yieldedresults that would be considered as "large" differences (i.e.,effect size greater than .80 between pairs of groups on a particularmeasure, [Cohen 1988]). Taken together, these results providepreliminary evidence on the construct validity of the TCB. But becausethe sample size for this study was relatively small, caution should beexercised in regarding these results as definitive. [TABULAR DATAOMITTED] DISCUSSION The TCB is the first test battery of its type developedspecifically for and standardized on a deaf population. The developmentof the battery addressed novel logistical issues, but the instrumentappears content relevant for the target population; and overall the TCBdemonstrates acceptable initial psychometric properties. At the same time, there are several issues that demand furtherinvestigation. First, is the 3-option multiple-choice approach correct?Is the group administration method appropriate? The choice of theseprocedures was made based on the sparse data on assessment practiceswith deaf persons and our own best guesses from studies with othersubject populations. Therefore, research should be conducted to verifythese choices. Second, we have found that the videotape medium is, attimes, cumbersome and slow, and that development of alternativeadministration media (e.g., videodisc) should be investigated. Third,Subtest 4, on money management, exhibited lower performance indexes andreliability coefficients than what is required for a group screeningmeasure. Clearly, more work is necessary to revise this subtest.Finally, what is the relationship of skill knowledge to actualperformance in real-life settings? We assume that knowledge is thebuilding block of behavior, but at this time there are no studies thatdescribe the interrelationship in��ter��re��late?tr. & intr.v. in��ter��re��lat��ed, in��ter��re��lat��ing, in��ter��re��latesTo place in or come into mutual relationship.in of knowledge and actual behavior for thissegment of the deaf population. [TABULAR DATA OMITTED] To conclude, this initial examination of the TCB is encouragingregarding its use and continued development. Such research willundoubtedly lead to its revision and strengthening and will increase ourunderstanding of the assessment process for adolescents and young adultswho are deaf. Moreover, we hope that investigations of this type willimprove service delivery efforts, contributing to the ultimate work andliving success of people who are deaf. REFERENCES American Psychological Association. (1985). Standards foreducational and psychological testing psychological testingUse of tests to measure skill, knowledge, intelligence, capacities, or aptitudes and to make predictions about performance. Best known is the IQ test; other tests include achievement tests—designed to evaluate a student's grade or performance . Washington, DC: Author. Bandura, A. (1977). Social learning theory. Englewood Cliffs, NJ:Prentice-Hall. Bellack, A., & Hersen, M. (1988). Behavioral assessment. NewYork New York, state, United StatesNew York,Middle Atlantic state of the United States. It is bordered by Vermont, Massachusetts, Connecticut, and the Atlantic Ocean (E), New Jersey and Pennsylvania (S), Lakes Erie and Ontario and the Canadian province of : Pergamon Press. Bloom, B. S. (1956). Taxonomy of educational objectives The Taxonomy of Educational Objectives, often called Bloom's Taxonomy, is a classification of the different objectives and skills that educators set for students (learning objectives). , handbookI: Cognitive domain cognitive domain,n area of study that deals with the processes and measurable results of study, as well as the practical ability to apply intelligence. . New York: McKay. Bolton, B. (Ed.). (1987). Handbook of measurement and evaluationin rehabilitation. Baltimore: Paul H. Brookes. Bowe, F. (1988). Toward equality: Education of the deaf.Washington, DC: U.S. Government Printing Office. Bullis, M. (1985). A dilemma: Who and what to teach in careereducation programs? In M. Bullis & D. Watson (Eds.), Careereducation for hearing impaired students: A review (pp. 55-75). LittleRock, AR: Research and Training Center on Deafness. Bullis, M., Bull, B., Johnson, B., Johnson, P., & Kittrell, G.(1990). School-to-community transition experiences of hearing impairedadolescents and young adults in the Northwest. Monmouth, OR: TeachingResearch Division. Bullis, M., Bull, B., Sendelbaugh, J., & Freeburg, J. (1987).The school to community transition of adolescents and young adults withdeafness. Washington, DC: The Catholic University, NationalRehabilitation Information Center. Bullis, M., & Foss, G. (1986). Assessing theemployment-related interpersonal competence of mildly mentally retarded Noun 1. mentally retarded - people collectively who are mentally retarded; "he started a school for the retarded"developmentally challenged, retarded workers. American Journal of Mental Deficiency mental deficiencyn.See mental retardation. , 91, 433-450. Bullis, M., & Reiman, J. (1989). Survey of professionalopinion on critical transition skills for deaf adolescents and youngadults. Rehabilitation Counseling rehabilitation counseling,n counseling started in the United States in 1920 to assist individuals disabled by industrial accidents; originally included physical, psychologic, and occupational training; expanded over the next 70 years and laid the Bulletin, 32, 231-242. Clark, G., & Knowlton, H. E. (1988). A closer look attransition issues for the 1990s: A response to Rusch and Menchetti.Exceptional Children, 54, 365-368. Cohen, J. (1988). Statistical power analysis for the behavioralsciences behavioral sciences,n.pl those sciences devoted to the study of human and animal behavior. (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. Delbecq, A., van de Ven, A., & Gustafson, D. (1975). Grouptechniques for program planning. Glenview, IL: Scott, Foresman. DeStefano, L. (1987). The use of standardized assessment insupported employment. In L. DeStefano & F. Rusch (Eds.), Supportedemployment in Illinois: Assessment methodology and research issues (pp.55-98). Champaign, IL: Transition Institute. Frey, W.(1984). Functional assessment in the '80s: Aconceptual enigma, a technical challenge. In A. Halpern & M. Fuhrer f��h��reralso fueh��rer ?n.A leader, especially one exercising the powers of a tyrant.[German, from Middle High German v��erer, from v��eren, to lead, from Old High German (Eds.), Functional assessment in rehabilitation (pp. 11-43). Baltimore:Paul H. Brookes. Gerweck, S., & Ysseldyke, J. (1979). Limitations of currentpsychological practices for the intellectual assessment of the hearingimpaired: A response to the Levine study. Volta Review, 77 243-248. Gronlund, N. E. (1977). Constructing achievement tests (p. 25).Englewood Cliffs, NJ: Prentice-Hall. Kazdin, A. (1977). Assessing the clinical or applied importance ofbehavior change Behavior change refers to any transformation or modification of human behavior. Such changes can occur intentionally, through behavior modification, without intention, or change rapidly in situations of mental illness. through social validation. Behavior Modification behavior modificationn.1. The use of basic learning techniques, such as conditioning, biofeedback, reinforcement, or aversion therapy, to teach simple skills or alter undesirable behavior.2. See behavior therapy. , 4,427-452. Keppel, G. (1982). Design and analysis: A researcher'shandbook (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall. Kerlinger, F. N. (1986). Foundations of behavioral research (3rded.). New York: Holt, Rinehart and Winston. Klockars, A., & Sax, G. (1986). Multiple comparisons. BeverlyHills Beverly Hills,city (1990 pop. 31,971), Los Angeles co., S Calif., completely surrounded by the city of Los Angeles; inc. 1914. The largely residential city is home to many motion-picture and television personalities. , CA: Sage. Landman, J., Irvin, L., & Halpern, A. (1980). Measuringlife-skills of adolescents. Measurement in Evaluation and Guidance, 13,95-106. Levine, E. (1974). Psychological tests Psychological TestsDefinitionPsychological tests are written, visual, or verbal evaluations administered to assess the cognitive and emotional functioning of children and adults. and practices with thedeaf: A survey of the state of the art. Volta Review, 76, 298-319. Marut, P., & Innes, C. (1986). The delivery of vocationalevaluation and adjustment services to deaf people. In D. Watson, G.Anderson, & M. Taff-Watson (Eds.), Integrating human resources The fancy word for "people." The human resources department within an organization, years ago known as the "personnel department," manages the administrative aspects of the employees. ,technology, and systems in deafness (pp. 135-144). Silver Spring, MD:American Deafness and Rehabilitation Association. Messick, S. (1989). Validity. In R. Linn linn?n. Scots1. A waterfall.2. A steep ravine.[Scottish Gaelic linne, pool, waterfall.] (Ed.), Educationalmeasurement (3rd ed., pp. 13-104). New York: Macmillan. Nunnally, J. (1978). Psychometric methods (2nd ed.). New York:McGraw-Hill. Ouellette, S., & Dwyer, C. (1985). A current profile of careereducation programs. In M. Bullis & D. Watson (Eds.), Careereducation of hearing impaired students: A review (pp. 27-54). LittleRock, AR: Research and Training Center on Deafness. Ouellette, S., & Loyd, G. (1980). Independent living skillsfor severely handicapped deaf people. Silver Spring, MD: AmericanDeafness and Rehabilitation Association. Reiman, J., & Bullis, M. (1987). Research on measurementprocedures for persons with hearing impairments: An annotatedbibliography An annotated bibliography is a bibliography that gives a summary of the research that has been done. It is still an alphabetical list of research sources. In addition to bibliographic data, an annotated bibliography provides a brief summary or annotation. . Monmouth, OR: Teaching Research Division. Reiman, J., & Bullis, M. (1989). Integrating students withdeafness into mainstream public education. In R. Gaylord-Ross (Ed.),Integration strategies for students with handicaps (pp. 105-128).Baltimore: Paul H. Brookes. Reiman, J., & Bullis, M. (1990). The transition competence attery for deaf adolescents and young adults. Monmouth, OR: TeachingResearch Division. Rusch, F., & Menchetti, B. (1988). Transition in the 1990s: Areply to Knowlton and Clark. Exceptional Children, 54, 363-364. Salvia, J., & Ysseldyke, J. (1988). Assessment in special andremedial education (4th ed.). Boston: Houghton Mifflin Houghton Mifflin Company is a leading educational publisher in the United States. The company's headquarters is located in Boston's Back Bay. It publishes textbooks, instructional technology materials, assessments, reference works, and fiction and non-fiction for both young readers . Shiels, J. (1986). Vocational assessment. In L. Stewart (Ed.),Clinical rehabilitation assessment and hearing impairment hearing impairmentn.A reduction or defect in the ability to perceive sound. (pp. 95-110).Washington, DC: National Association of the Deaf. Sligar, S. (1983). Commercial vocational evaluation systems anddeaf persons. In D. Watson, G. Anderson, P. Marut, S. Ouellette, &N. Ford (Eds.), Vocational evaluation of hearing impaired persons:Research and practice (pp. 35-56). Little Rock, AR: RehabilitationResearch and Training Center in Deafness and Hearing Impairment. SPSS Inc. (1989). SPSS-X user's guide (3rd ed.) Chicago:Author. Stewart, L. (1986). Clinical rehabilitation assessment and hearingimpairment: A guide to quality assurance: Washington, DC: NationalAssociation of the Deaf. Wiggins, J. (1973). Personality and prediction: Principles andpersonality assessment. Reading, MA: Addison-Wesley. Yoshida, R., & Friedman, D. (1986). Standards for educationaland psychological testing: More than a symbolic exercise. In R.Bennett & C. Maher (Eds.), Emerging perspectives on assessment ofexceptional children (pp. 187-193). New York: Haworth Press.