rss.livelink.posts-in-node

Visual Analytics - Scatter Plot blank with large datasets

sc5 — Mon, 29 Apr 2024 00:21:56 GMT

Hi,

I'm trying to get scatter plots of some numeric columns in my dataset with ~1million records, but it shows up empty. I've tried multiple combinations of columns and it's always just blank, like this:

I'm guessing it's an issue with the no. of data points, because when I charted a test dataset i ~ = j with 1000 records it works.

But when I expand the same dataset to a million records it's blank.

Does anyone know how I can fix this and plot scatter plots with large datasets?

Thank you!

Re: logistic regression adjusting for covariate

Manije72 — Mon, 29 Apr 2024 00:05:38 GMT

Thank you for your response.

Can you help me where I can find P-value trend for logistic regression in SAS results? For example, in this table https://www.nature.com/articles/s41598-024-52946-z/tables/4, how is the P-value trend calculated?

Re: Create new date variable based on specific condition

tan-wongv — Mon, 29 Apr 2024 00:04:08 GMT

This code works for those who have UI = 0, new_date is equal to last_visit

However, it did not work for those who have UI = 1, such as DMRN 31. I would like new_date for DMRN31 to be as 4MAY2021 4MAY2021 4MAY2021

DMRN	last_visit	UI	recorded_time	new_date
31	26AUG2021	0	06APR2018	26AUG2021
31	26AUG2021	0	16JAN2020	26AUG2021
31	26AUG2021	1	4MAY2021	4MAY2021
33	24MAY2022	0	02MAR2020	24MAY2022
33	24MAY2022	0	24MAY2022	24MAY2022
35	01DEC2014	0	25MAR2013	01DEC2014

Re: Keep only value and date that occurred before or at last visit date

tan-wongv — Sun, 28 Apr 2024 23:38:50 GMT

Thank you so much for your guide!! I found the problem as you said. I fixed it and now your code works

Re: Find Standard deviation from 2, or more, columns/variables with aim of SQL

Tom — Sun, 28 Apr 2024 23:32:07 GMT

You asked this question already and you have been told you need to convert your variables into a single variable.

select south as new_var from have
union all
select north as new_var from have

Then you can calculate mean, std, var, and other statistics on this new variable.

Re: Keep only value and date that occurred before or at last visit date

Tom — Sun, 28 Apr 2024 23:07:55 GMT

@tan-wongv wrote:

Thank you so much for your response! I tried this code but the outcome just showed the observation where recorded_time is less than last_vist. It did not include the recorded_time that is equal to last_visit.

That could only happen if the date values are not integers. The DATE format will ignore fractions of a day so the values would print as being equal even though they would compare as not equal.

Try looking at the values for one of the cases where the values look to you like they are equal but did not get selected by the WHERE (or IF) test. Display the values with out the format or with some other format such as BEST32. or HEX16.

How did you enter those values into your SAS dataset?

Re: Keep only value and date that occurred before or at last visit date

tan-wongv — Sun, 28 Apr 2024 23:03:05 GMT

Thank you so much for your response! I tried this code but the outcome just showed the observation where recorded_time is less than last_vist. It did not include the recorded_time that is equal to last_visit.

Re: logistic regression adjusting for covariate

StatDave — Sun, 28 Apr 2024 20:52:38 GMT

If you have a continuous measure of fruit consumption, then reducing it to a 3 level variable just throws away information. So I would use your FRUIT variable as a predictor, not FRUITCODE. Then, I would use your first PROC LOGISTIC step, particularly if either GENDER or RACE has more than two levels. You should put a variable in the CLASS statement any time it is a categorical variable and has more than two levels. Then the CLASS statement will internally create a set of dummy-coded variables to represent it in the model. As I mentioned in your previous question, the test of FRUIT will then test the linear trend of fruit consumption on the log odds of diabetes. If that's not what you mean by "P-trend" then you need to clarify exactly what you mean by it. But be aware that it is possible that consumption is not linearly associated with with diabetes, and if so, your model does not accommodate that because it assumes a linear association. There are ways, such as splines, to allow the model to be more flexible if you have a reason to suspect nonlinearity.

Re: logistic regression adjusting for covariate

PaigeMiller — Sun, 28 Apr 2024 19:52:43 GMT

You asked the exact same question in 2023 and it was answered.

https://communities.sas.com/t5/Statistical-Procedures/The-code-for-P-trend-for-logistic-regression/td-p/857243

Re: Create new date variable based on specific condition

Tom — Sun, 28 Apr 2024 19:26:38 GMT

You seem to have already written the code.

Did you try running it? What happened?

data want;
  set have;
  if UI = 1 then new_date = recorded_time;
  else if UI = 0 then new_date = last_visit;
  format new_date date9.;
run;

logistic regression adjusting for covariate

Manije72 — Sun, 28 Apr 2024 19:09:09 GMT

I want to explore the relationship between fruit consumption and diabetes. I've categorized fruit intake into tertiles, while diabetes status is binary (yes/no). I want to adjust the findings to age (continuous), gender (categorical), race (categorical), and physical activity (categorical) in the analysis. Is the logistic regression code that I want to use correct?

proc logistic data=SS plots(only)=(effect oddsratio);

class Gender(ref='1') race(ref='1') physical_activity(ref='1')/param=ref ;

model diabetes(event='1')= fruitcode age Gender race physical_activity/Clodds=wald;

title " Association of fruit with diabetes (adjust for age, gender, race, physical activity) ";

run;

Fruitcode: I used fruit tertile in this code.

For finding the P-trend, I used this code:

proc logistic data=SS plots(only)=(effect oddsratio);

model diab(event='1')= fruit age Gender race physical_activity/Clodds=wald;;

run;

Fruit is continuous

Re: Keep only value and date that occurred before or at last visit date

Tom — Sun, 28 Apr 2024 18:33:00 GMT

Seems simple enough. First let's convert your existing (HAVE) data listing and expected (EXPECT) data listings into datasets. Then you just need to select the observations where recorded_time is less than or equal to last_visit.

data have;
  input DMRN $ last_visit :date. UI recorded_time :date.;
  format last_visit recorded_time date9.;
cards;
31 26AUG2021 0 06APR2018
31 26AUG2021 0 16JAN2020
31 26AUG2021 1 4MAY2021
31 26AUG2021 1 26MAY2022
33 24MAY2022 0 02MAR2020
33 24MAY2022 0 24MAY2022
35 01DEC2014 0 25MAR2013
35 01DEC2014 1 05JAN2015
;

data expect;
  input DMRN $ last_visit :date. UI recorded_time :date.;
  format last_visit recorded_time date9.;
cards;
31 26AUG2021 0 06APR2018
31 26AUG2021 0 16JAN2020
31 26AUG2021 1 4MAY2021
33 24MAY2022 0 02MAR2020
33 24MAY2022 0 24MAY2022
35 01DEC2014 0 25MAR2013
;

data want ;
  set have;
  where recorded_time <= last_visit;
run;

proc compare data=want compare=expect;
run;

Create new date variable based on specific condition

tan-wongv — Sun, 28 Apr 2024 18:16:01 GMT

I have a data set contains DMRN, last visit date, the presence of UI, and recorded date of UI.

DMRN	last_visit	UI	recorded_time
31	26AUG2021	0	06APR2018
31	26AUG2021	0	16JAN2020
31	26AUG2021	1	4MAY2021
33	24MAY2022	0	02MAR2020
33	24MAY2022	0	24MAY2022
35	01DEC2014	0	25MAR2013

I would like to create new_date for each DMRN based on these conditions:
if UI = 1 then new_date = recorded_time
else if UI = 0 then new_date = last_visit
The outcome would be like as follow:

DMRN	last_visit	UI	recorded_time	new_date
31	26AUG2021	0	06APR2018	4MAY2021
31	26AUG2021	0	16JAN2020	4MAY2021
31	26AUG2021	1	4MAY2021	4MAY2021
33	24MAY2022	0	02MAR2020	24MAY2022
33	24MAY2022	0	24MAY2022	24MAY2022
35	01DEC2014	0	25MAR2013	01DEC2014

Could you please help me with the code? Thank you very much

Keep only value and date that occurred before or at last visit date

tan-wongv — Sun, 28 Apr 2024 18:01:15 GMT

I have a dataset contains DMRN, last visit date, the presence of UI, and recorded date of UI

DMRN	last_visit	UI	recorded_time
31	26AUG2021	0	06APR2018
31	26AUG2021	0	16JAN2020
31	26AUG2021	1	4MAY2021
31	26AUG2021	1	26MAY2022
33	24MAY2022	0	02MAR2020
33	24MAY2022	0	24MAY2022
35	01DEC2014	0	25MAR2013
35	01DEC2014	1	05JAN2015

I would like to keep only UI value and recorded_time that occurred before or equal to last_visit date for each DMRN. Could you please help me with the code?The outcome would look like this:

DMRN	last_visit	UI	recorded_time
31	26AUG2021	0	06APR2018
31	26AUG2021	0	16JAN2020
31	26AUG2021	1	4MAY2021
33	24MAY2022	0	02MAR2020
33	24MAY2022	0	24MAY2022
35	01DEC2014	0	25MAR2013

Proc logistic effect plot using GTL

rs72 — Sun, 28 Apr 2024 17:52:08 GMT

hello ,
I am trying to use graph template use by Proc Logistics / effectplot , so that I can modify the graph template for some changes
like Y axis label . But when I use this below code

proc template;
source Stat.Logistic.Graphics.EffectCont/ file="&path/reg.txt";
run;

I get log message as-

"link Stat.Logistic.Graphics.EffectCont to Common.Zreg.Graphics.EffectCont;"

can someone tell me what does this above log message means ?

Making header border lines invisible/white in selective columns with PROC REPORT

smackerz1988 — Sun, 28 Apr 2024 15:17:46 GMT

Hello,

I'm creating an output that requires this header layout

I'm using this code

proc report data=final out=view missing nowindows split="*" contents=" ";
   column pag ("MedDRA System Organ Class /* Preferred Term" (" " aesoc))    
    ("trt (N=&trt_count.)" ("arm1*(N=&arm1_count.)" ('Subjects with TEAE*n (%)' arm1))
    ("arm2*(N=&arm2_count.)" ('Subjects with TEAE*n (%)' arm2)))
    ord;
   define pag / order  order = internal noprint ;
   define aesoc  / display " " left flow style(header)=[just=l] STYLE(column)=[&col_style asis=on cellwidth=12.00cm];
   define ord / order order =  internal noprint;   
   define arm2 / " " display   center flow STYLE(column)=[&col_style cellwidth=7.00cm];
   define arm1 / " " display center   flow STYLE(column)=[&col_style cellwidth=7.00cm];
   

    compute after pag / style ={&line_small};
        line "";
    endcomp;

    break after pag / page;

compute before _page_ / left style={&tit_style};
   line "Table 1.4.3 Severe treatment-emergent adverse events by System Organ Class and Preferred Term with incidence of >=5% in at least one arm - Separately by TPC regimen - Safety Analysis Set";   
endcomp;

	compute before ord / style ={&line_small};
		line "";		
	endcomp;

 compute after _page_ / left style={&foot_style};
   line "Notes: NE: Not evaluable; SOC: System Organ Class; PT: Preferred Term; TEAE: Treatment-emergent adverse event. Adverse events were coded using the MedDRA dictionary, Version 26.1 and graded according to NCI CTCAE v5.0.";   
endcomp;
run;

However this is my ouput

How am I able to make this border line for this column either not visibly or removed?

find the count of variable by group

CathyVI — Sun, 28 Apr 2024 14:54:45 GMT

Hello all,

I have initially posted this inappropriately, am sorry. I am creating a better representation of the issue so I can get help on this.

I have a dataset with category (teachers, students) and a list of forms (form2, event46, case32) which they are expected to fill within 60days after the school resume. Event46 has a group variable (grp=A, B, C). That is someone can be in any of the group. Among the teachers and students, some did not fill the form (0), some filled the form within 30 days (1), and some filled the form within 60 days (2).

The 0,1,2 represents the days that the form was filled below:

No form filled=0

Form filled within 0-30days=1

Form filled within 31-60days=2

This is what I want

Table 1:

1. If a teacher or student only filled form2 within 30days and did not fill event46 or case32 they should be in the form2 only column. Same apply for other forms.

If they filled the form after 30 days, they will be in the 31-60 days count by category.

2. If anyone filled more than one(1) forms either within 30 days or within 60 days or both, they should be in the multiple column.

Table 2:

3. Finally, I want a total of those who did not fill any form in both category(teacher or student).

I have created a sample output and a demo dataset. Thanks so much for your help.

data have;
input category $ form2 case32 event46 grp $;
datalines;
Teacher 1 0 0 .
Student 1 0 0 .
Teacher 0 1 0 .
Teacher 1 0 1 A
Teacher 0 0 1 C 
Teacher 0 0 1 B
Teacher 1 0 0 .
Student 0 1 1 C
Student 0 0 1 C
Student 0 0 1 C
Student 1 0 0 .
Student 0 1 0 .
Student 0 2 0 .
Teacher 1 1 1 B
Teacher 0 0 2 A 
Teacher 0 0 2 B
Teacher 2 0 0 .
Student 0 1 1 C
Teacher 0 0 0 .
Student 0 0 0 .
Student 0 0 1 B 
Teacher 0 0 1 B 
Teacher 0 2 0 .
Teacher 2 2 0 .
Student 0 1 0 .
Teacher 2 0 1 A
Student 0 0 1 A 
Teacher 0 0 0 .
Student 2 0 0 .
Teacher 0 0 2 C
Student 0 0 2 C
Student 0 0 2 B
Student 0 0 2 A
;
run;

The expected output should look like this:

	Form2 only	Case32 only	Event46 A only	Event46 B only	Event46 C Only	Multiple	Total
Teacher fill form within 0-30 days
Teacher fill form within 31-60 days
Student fill form within 0-30days
Student fill form within 31-60days
Total

	Total
Teacher who did not complete any form within 0-60 days
Student who did not complete any form within 0-60 days
Total

Re: Need Advice on Handling High-Dimensional Data in Data Science Project

FreelanceReinh — Sun, 28 Apr 2024 13:44:35 GMT

Also, have you already tried some of the suggestions from the Reddit thread you copied your post from, e.g., to "aggregate the 100+ categories into a smaller set of categories"?

Re: Need Advice on Handling High-Dimensional Data in Data Science Project

PaigeMiller — Sun, 28 Apr 2024 12:44:58 GMT

Can you describe the project you are working on? Can you tell us what the desired outcome(s) is of the analysis of this data? When you say you "need advice on handling", well there are a gazillion ways to "handle" such data and unless we know what desired outcome(s) are, we can't really give you good advice.

Need Advice on Handling High-Dimensional Data in Data Science Project

tress — Sun, 28 Apr 2024 11:57:46 GMT

Hey everyone,

I’m relatively new to data science and currently working on a project that involves a dataset with over 60 columns. Many of these columns are categorical, with more than 100 unique values each.

My issue arises when I try to apply one-hot encoding to these categorical columns. It seems like I’m running into the curse of dimensionality problem, and I’m not quite sure how to proceed from here.

I’d really appreciate some advice or guidance on how to effectively handle high-dimensional data in this context. Are there alternative encoding techniques I should consider? Or perhaps there are preprocessing steps I’m overlooking?

Any insights or tips would be immensely helpful.

Thanks in advance!