05-06-2010 08:00 AM
05-06-2010 08:27 AM
05-06-2010 12:08 PM
05-07-2010 07:16 AM
10-10-2013 11:21 AM
Hi Steve - I believe that I have a similar quandary. Could you please help me generalize your suggestion above to my context? In brief: This comes from a cluster randomized design with school as the unit of assignment (16 Treat; 15 Control). My data consist of teacher observations using a multi-item instrument; each item having a 5 ordered categorical response scale. One hundred twenty-two teachers were observed: 75 by a single a observer; 47 by a pair of observers (observing concurrently); 169 observations total. There were 18 observers, who conducted on average 9 observations each (SD=4, Range = 1-16). I wish to analyze for treatment effects and account for the cross-classification and nesting of the data. The below syntax is what I have so far. I'm following a G-theory framework for modeling the facets of measurement, where all facets are random except Item. Perhaps your suggestion above provides a work-around for my context--but I am coming up short making the one-to-one comparison. It occurred to me that I could create a variable where single observations were coded 1 and paired observations were randomly coded 1/2 or 2/1, which could serve as an index for time; though that seemed flawed given the paired data are contemporaneous. I appreciate your input.
proc glimmix data=WORK.Obsv_v7;
class Treat Schl_ID Obsv_ID Tchr_ID Item;
model Scale = Treat Item / dist=multinomial link=cumlogit s;
random intercept /sub=Schl_ID;
random intercept /sub=Obsv_ID;
random intercept /sub=Schl_ID*Obsv_ID;
random intercept /sub=Tchr_ID(Schl_ID);
random intercept /sub=Tchr_ID*Obsv_ID(Schl_ID);
random intercept /sub=Schl_ID*Item;
random intercept /sub=Obsv_ID*Item;
random intercept /sub=Schl_ID*Obsv_ID*Item;
Furthermore, I would be grateful for any additional guidance that may occur to you. For example, given that Item is fixed, perhaps the third and last three variance components should not be modeled; that is, I could see just modeling the main effects of School, Observer, and Teacher(School) [var comps 1, 2, and 4] as conceptually adequate, but for modeling purposes perhaps the interactions are necessary. Also, I'm think method=laplace and type=chol; do you think otherwise? Lastly, you mentioned being mindful of how the data are sorted; would you agree that the proper order of sorting is School>Observer>Teacher>Item?
My apologies for piling on so many questions at once. Thank you in advance.
10-11-2013 10:14 AM
I'll borrow a page from , and say that your model is probably overspecified for the amount of data that you have. With 169 observations in 31 schools with 122 teachers, it is going to be very difficult to get decent estimates for all of the variance components specified. I would (maybe) consider dropping the obsv_id terms, and letting them fall to error. The correlated observations/pseudo-replication of teachers due to concurrent observations makes this a tough problem. What about an exploratory analysis of just the 47 teachers observed concurrently, and looking at whether there is a variance component due to observer. If yes, then curse for about an hour, and reconcile to analyze separately by "observational status". If no, then do something like analyze maximum score across raters on an item (or minimum score, depending on research objectives). Start with a simplified model, with maybe just school and school*item as variance components, and add in others, so long as the estimates and algorithm remain stable. And I would definitely try exploring method=laplace for optimizing. I wouldn't go anywhere near type=chol as there really isn't any good reason to getting the covariances between subjects as yet. Stick with just getting the variances to get started.
10-11-2013 10:21 AM
Hi Steve - Thanks for the reply. I'm afraid I'm cursing already. Nevertheless, I appreciate your honest assessment. I'll pursue this track and see where it gets me. Gratefully yours, Mark
10-17-2013 02:28 PM
Hi Steve - Per your suggestion, I fit a model on just the paired observations and looked for whether there was a variance component due to observer, using the following code:
proc glimmix data=WORK.Obsv method=laplace;
class Item Obsv_ID;
model Value = Item / dist=multinomial link=cumlogit s cl ;
random intercept / sub=Obsv_ID;
Covariance Parameter Estimates
Cov Parm Subject Estimate Standard Error
Intercept Obsv_ID 0.4977 0.2063
With a 1.96 critical value, this is significant at <.05. Do I interpret this as having a variance component due to observer? This might sound a little weaselly, but if so, it's not particularly huge. When I fit a proc mixed model with VCs estimated for School|Grade|Observer|Teacher|Item, the Observer main effect only accounts for 6% of the total variance and the calculated inter-rater reliability is pretty good (Generalizability coefficient of Eρ2=.80 and Index of dependability of Φ=.73).
Please share your thoughts.
If it helps in providing your response, I have a broader concern that I keep stumbling on: It strikes me that failure to account for the pseudo-replication would lead the program to "think" that my sample is larger than it really is, at the peril of my statistical inference. This might be what you meant by "reconcile to analyze separately by 'observational status'," but I could randomly select one of the paired observations along with the single observations and run it on those 122 observations, then conduct a sensitivity analysis by swapping out the sample with the other paired observations (sort of like a split-sample cross-validation).
A final option I've considered is estimate factor scores in Mplus, using their new type=crossclassified analysis (cluster=Obsvr_ID Tchr_ID). This would generate a single fscore by teacher, which I could then plug into a proc mixed model and only have to deal with the variance due to School and Teacher(School).
I appreciate any guidance you have to offer.
10-21-2013 08:01 AM
I think factor scores are a very good idea under this design.
As far as the sensitivity analysis, I think it addresses the issue, but I would try a moderate number of random samples (say 5 or 6), rather than "all of A" vs "all of B", for the observer effect might not be "separable." Still, it is a good start, and should address the pertinent tests.
10-21-2013 08:41 AM
Wonderful! Thanks Steve. I appreciate your input. I'm self-taught with SAS and only a couple years in; so, I never really know if my roadblocks are due to lack of facility with the program or a mismatch between what I want to do and the data I have. My mind is now at rest that I can move on. Thanks, Mark