Wonderful! Thanks Steve. I appreciate your input. I'm self-taught with SAS and only a couple years in; so, I never really know if my roadblocks are due to lack of facility with the program or a mismatch between what I want to do and the data I have. My mind is now at rest that I can move on. Thanks, Mark ... View more

Hi Steve - Per your suggestion, I fit a model on just the paired observations and looked for whether there was a variance component due to observer, using the following code: proc glimmix data=WORK.Obsv method=laplace; where Pair='Y'; class Item Obsv_ID; model Value = Item / dist=multinomial link=cumlogit s cl ; random intercept / sub=Obsv_ID; run; and got: Covariance Parameter Estimates Cov Parm Subject Estimate Standard Error Intercept Obsv_ID 0.4977 0.2063 With a 1.96 critical value, this is significant at <.05. Do I interpret this as having a variance component due to observer? This might sound a little weaselly, but if so, it's not particularly huge. When I fit a proc mixed model with VCs estimated for School|Grade|Observer|Teacher|Item, the Observer main effect only accounts for 6% of the total variance and the calculated inter-rater reliability is pretty good (Generalizability coefficient of Eρ2=.80 and Index of dependability of Φ=.73). Please share your thoughts. If it helps in providing your response, I have a broader concern that I keep stumbling on: It strikes me that failure to account for the pseudo-replication would lead the program to "think" that my sample is larger than it really is, at the peril of my statistical inference. This might be what you meant by "reconcile to analyze separately by 'observational status'," but I could randomly select one of the paired observations along with the single observations and run it on those 122 observations, then conduct a sensitivity analysis by swapping out the sample with the other paired observations (sort of like a split-sample cross-validation). A final option I've considered is estimate factor scores in Mplus, using their new type=crossclassified analysis (cluster=Obsvr_ID Tchr_ID). This would generate a single fscore by teacher, which I could then plug into a proc mixed model and only have to deal with the variance due to School and Teacher(School). I appreciate any guidance you have to offer. Gratefully yours, Mark ... View more

Hi Steve - Thanks for the reply. I'm afraid I'm cursing already. Nevertheless, I appreciate your honest assessment. I'll pursue this track and see where it gets me. Gratefully yours, Mark ... View more

Hi Steve - I believe that I have a similar quandary. Could you please help me generalize your suggestion above to my context? In brief: This comes from a cluster randomized design with school as the unit of assignment (16 Treat; 15 Control). My data consist of teacher observations using a multi-item instrument; each item having a 5 ordered categorical response scale. One hundred twenty-two teachers were observed: 75 by a single a observer; 47 by a pair of observers (observing concurrently); 169 observations total. There were 18 observers, who conducted on average 9 observations each (SD=4, Range = 1-16). I wish to analyze for treatment effects and account for the cross-classification and nesting of the data. The below syntax is what I have so far. I'm following a G-theory framework for modeling the facets of measurement, where all facets are random except Item. Perhaps your suggestion above provides a work-around for my context--but I am coming up short making the one-to-one comparison. It occurred to me that I could create a variable where single observations were coded 1 and paired observations were randomly coded 1/2 or 2/1, which could serve as an index for time; though that seemed flawed given the paired data are contemporaneous. I appreciate your input. proc glimmix data=WORK.Obsv_v7; class Treat Schl_ID Obsv_ID Tchr_ID Item; model Scale = Treat Item / dist=multinomial link=cumlogit s; random intercept /sub=Schl_ID; random intercept /sub=Obsv_ID; random intercept /sub=Schl_ID*Obsv_ID; random intercept /sub=Tchr_ID(Schl_ID); random intercept /sub=Tchr_ID*Obsv_ID(Schl_ID); random intercept /sub=Schl_ID*Item; random intercept /sub=Obsv_ID*Item; random intercept /sub=Schl_ID*Obsv_ID*Item; run; Furthermore, I would be grateful for any additional guidance that may occur to you. For example, given that Item is fixed, perhaps the third and last three variance components should not be modeled; that is, I could see just modeling the main effects of School, Observer, and Teacher(School) [var comps 1, 2, and 4] as conceptually adequate, but for modeling purposes perhaps the interactions are necessary. Also, I'm think method=laplace and type=chol; do you think otherwise? Lastly, you mentioned being mindful of how the data are sorted; would you agree that the proper order of sorting is School>Observer>Teacher>Item? My apologies for piling on so many questions at once. Thank you in advance. Best regards, Mark ... View more