thafu Tracker

Re: RE: Splitting large data by variables

thafu — Mon, 26 May 2014 18:50:26 GMT

To all contributors, namely; PaigeMiller, PGStats, Jaap Karman, stat@sas, RW9, data_null_; and LinusH; Thank you very much.

I have gained sufficient insight on the issues and I will continue on from here. Should I need further aid, I shall be in touch.

Sincerely

Thafu

Re: RE: Splitting large data by variables

thafu — Wed, 21 May 2014 17:00:42 GMT

Dear SAS Friends

Sure, as some of you have mentioned, my statistical protocol is not precise as of now. This project intention
is to produce an accurate report on a sub-Saharan Africa country health situation, focusing on
the “socioeconomic determinants of health inequality and inequity”. The work to be based on the
previously applied protocol in two studies; one in Europe (link-doi: 10.1056/NEJMsa0707519) and another one in North Africa (link-doi:10.1186/1475-9276-10-23).

Preliminary data understanding revealed that evaluating all the 4K variables will definitely be a hard job. As noted in
my earlier communication, i was most interested in splicing the data arbitrary
into small sets, followed by grouping (where i suggested use of principalcomponent analysis).

My goal sharing was to get hint on simple and efficient ways of summarizing
or grouping these variables with limited error risks. I am following with keen interest
this discussion and hope to refine my methodology for the work. Therefore, you suggestions towards this end would
be highly appreciated.

Thafu

Re: RE: Splitting large data by variables

thafu — Mon, 19 May 2014 17:42:25 GMT

OK, I get the concerns projected.

I am at the initial stage of this project, currently trying to understand the entire data before deciding which variables to retain for the subsequent analysis.

Actually, the enormity(biggness) of the data and the inability to be evaluated (read) in its current form, in the STATA version in my possession, are what prompted my request for arbiterary subdivision.

I should probably have asked if there are any better ways to handle such enormous data, may be by automatically grouping related variables without having to go through variables visually and then physically coding for the selection from the ~4K variables.

Re: RE: Splitting large data by variables

thafu — Mon, 19 May 2014 11:33:34 GMT

Hello data_null_,

Thanks for the codes. However, for clarity, i have one request, could you please add brief descriptors to your codes to enable me follow through

Regards

Fred

Re: RE: Splitting large data by variables

thafu — Mon, 19 May 2014 11:14:39 GMT

Thank you all for your generosity,

To respond to your questions; this is a government data of household surveillance. The study objective is to "investigate the socio-economic déterminants of health inequality and inequity". The first step was to arbitrary break this large dataset into smaller ones, then asses the important variables for use. The selected varaibles could then be consolidate into afew manageble number via the principle component analysis. Finally, i am to merge the consolidated dataset and perform the core analysis of the study.

I also wished to split this dataset to enable alternative statistical manipulation in STATA 13 Platform, which i have more compétence, but the Platform has limited amount of data it can handle.

Warm regards

RE: Splitting large data by variables

thafu — Sun, 18 May 2014 17:05:35 GMT

Hello friends,

I need some help in data management, I have a large dataset of 17375 observations and 3997 variables. I wish to split this date into three sets of 17375 observations 1333 variables, while retaining all the observations and the unique identification code for future re-merging.

I wish to get help in developing this SAS code for doing the splitting

Thanks in advance, I would appreciate your assistance

Fred