About gyambqt

gyambqt · ‎04-02-2015

Hello Xia, I have tried to add you on the wechat. Is this your wechat ID:xia-keshan

gyambqt · ‎04-02-2015

Hi Kurt, Could you explain briefly why hash table only take a single I/O operation. data on disk----> memory (single operation?) Thanks

gyambqt · ‎04-01-2015

Hi Linus, do you mean if the source data is not sorted and you want to do data step merge. So the hard disk for data records has to be read several time from top to bottom to ensure every single records is read and matched from A with the every records in B. So it caused a lot of I/O operation. Like the following example: assume the following dataset is not sorted. Data A has 10k records Data B has 10K records SAS buffer has capacity 1k records if we match A with B, so initially SAS buffer take 1k records from A and B, and do the matching at PDV (transfer record from buffer to PDV 1 by 1). If all the records in the SAS buffer for B has been used for matching, then buffer will remove previously selected records for B and take new portion of records from B to match with A. this process will be repeated until all the records from B are selected to match with first record in A. then after the first record for A is matched with all the records in B and it will move to the second record in A. This process will be repeated until all the records in A are selected for matching. So it resulted a lot of read input and output process. Compare to sorted datasets. not all the records will be selected by SAS buffer to match with every single record in A because it is sorted. So it resulted in less read input and output process. Am I correct?

gyambqt · ‎04-01-2015

Hi Kurt, http://web.utk.edu/sas/OnlineTutor/1.2/en/60477/m81/m81_2.htm http://web.utk.edu/sas/OnlineTutor/1.2/en/60477/m81/m81_3.htm If you read the above links(very short), it said : SAS copies the data from the input data set to a buffer in memory one observation at a time is loaded into the program data vector A buffer can be treated as a container in memory that is big enough for only one page of data. A page is the unit of data transfer between the storage device and memory The amount of data that can be transferred to one buffer in a single I/O operation is referred to as page size. Page size is analogous to buffer size for SAS data sets. I think the buffer here means the SAS buffer rather than OS buffer(can we ignore OS here). The size of buffer can be adjusted in SAS, the bigger it is , the more data it can be taken each time so consequently less I/O. My Question is: if buffer size is small, for example it only can take 10k records each time, but you have 50k records want to load to hash table. It still takes 5 I/O operations to transfer the 50k records from hard disk to hash(memory). I think it is little bit different to your remark that only one I/O operation required to load the 50k records data from hard disk to hash. My understanding of I/O is when data records from hard disk are read by SAS to SAS buffer. If SAS buffer size is small then it will require more I/O operation to finish reading complete dataset.. Do I have any problem to understand that?

gyambqt · ‎04-01-2015

THx jaap.

gyambqt · ‎04-01-2015

I have read the powerpoint about PDV, the buffer is still existed but according to kurt remarkthere is no good way to control how it read the data records on hard disk. I have read other people's remark, for 1 to 1 matching, there is no differencn between data step merge and hash table match in the aspect of i/o. since dataset are accessed sequentially using data step merge. Shouldn't it cause more I/o than hash table match?

gyambqt · ‎04-01-2015

Thanks for your reply. I was confused by the following link indeed: http://web.utk.edu/sas/OnlineTutor/1.2/en/60477/m81/m81_2.htm It said SAS copies the data from the input data set to a buffer in memory one observation at a time is loaded into the program data vector each observation is written to an output buffer when processing is complete the contents of the output buffer are written to the disk when the buffer is full

gyambqt · ‎04-01-2015

Hi Jaap, Do you mean in the data merge the data records for both datasets are loaded to PDV directly for matching? i thought the record would be loaded to buffer first then from buffer load to pdv for matching.

gyambqt · ‎04-01-2015

Hi KurtBremser, I am trying to understand you with the following example, please let me know if I am wrong. Assume we have two SAS datasets, A and B (they are not sorted based on the by variables) We want to match A and B, but every record of A can match multiple records in B so it is One to Many match. If I use data step merge statement, then many records from A will be selected and loaded to a buffer (memory) and many records from B will be selected and loaded a buffer (memory). If records in the buffer from A is NOT found in the buffer for records in B, then the buffer for B will be empty and another set of records will be selected from B into the buffer to match the records for A (in the buffer). So this step will be repeated until all the matching records are found, so it can cause a lot of I/O (read data from disk). But Hash can solve this problem because it holds B so there is only one I/O to match with records from A( the number of I/O will be equal to number of records from A). Am I correct? If I am wrong please let me know why. Maybe my understanding of buffer and I/O is wrong...

gyambqt · ‎04-01-2015

Hello Experts, Based on page 9 of the following link, it said data step merge is"Not memory intensive because only the current observation from each input dataset is loaded into memory". http://www.lexjansen.com/nesug/nesug07/bb/bb16.pdf I was bit confused with this statement as I thought the data sets should be firstly loaded (can be many records) to a buffer(memory) before it loads to PDV 1 by 1. So if the buffsize is large enough, all the SAS datasets can be loaded there. So I think data step merge sometimes can be quite memory intensive as well. Please let me know why I am wrong. Thank you

gyambqt · ‎04-01-2015

Hello experts? I have seen people say Hash table can improve I/O process and do not understand why. Assume you have two tables want to merge each table contain 50k records, Table A and Table B . Also Input buffer can only take 10k records each time. if you use data step merge then 50k+50k records will be loaded to input buffer for 10 times ((50k+50k)/10k=10 times) 50k+50k on hard disk---->Input buffer----->PDV---->etc. if you use hash to hold table A, then it takes 5 times(50k/10k=5times) for data records on table A to load to input buffer then to HASH table. it will also take 5 times (50k/10k=5times) for data records in table B to be loaded to input buffer. so the total is 5+5=10 times. there is no difference.... 50k records in table A on hard disk---->input buffer---->PDV----->HASH 50k records in table B on hard disk-----> input buffer-----PDV--->etc. Please explain. Thanks so much!!!!!

gyambqt · ‎03-31-2015

Hi KurtBremser, I was bit confused why Hash can improve I/O process? assume you have two tables want to merge each table contain 50k records. Table A and Table B and input buffer can only take 10k records each time. if you use data step merge then 50k+50k records will be loaded to input buffer 10 times ((50k+50k)/10k=10 times) 50k+50k on hard disk---->Input buffer----->PDV---->etc. if you use hash to hold table A, then it takes 5 times(50k/10k=5times) for data records on table A to load to input buffer then to HASH table. it will also take 5 times (50k/10k=5times) for data records in table B to be loaded to input buffer. so the total is 5+5=10 times. there is no difference.... 50k records in table A on hard disk---->input buffer---->PDV----->HASH 50k records in table B on hard disk-----> input buffer-----PDV--->etc. Please explain.

gyambqt · ‎03-30-2015

Hi Tom, There are thousands record in SAS table each contains different email address. I want to use hash iterator to read each email record in the table and execute the email macro to send out the email to their corresponding users. Thanks

gyambqt · ‎03-30-2015

Hi Ballardw, where should I put %superq(error)?

gyambqt · ‎03-30-2015

Hello experts, For the code below, if "error" variable in the following work.merge dataset loaded into hash table has values that contain many special characters like [,./;'etc]. How do I make the following code works? (please just focus on error variable which is highlighted in blue, that is where I get the error in the log) %macro Email(email=,suite=,reference=,error=,time=,abortcode=,); FILENAME Mailbox EMAIL "&email" Subject= ""; DATA _NULL_; FILE Mailbox; PUT "Hello"; PUT "Error message: &error"; PUT "Run time: &time"; PUT "Abortcode: &abortcode"; RUN; %mend; data _null_; length email $200 suite $200 reference $200 errormsg $200 time $200 abortcode $200; if _n_=1 then do; declare hash myhash(dataset: 'work.merge',ordered:'a'); /* sort order is ascending */ declare hiter iter('myhash'); myhash.definekey('suite'); /* define a key */ myhash.definedata('Email','Suite','Reference','errormsg','time','abortcode'); /* getting the columns */ myhash.definedone(); /* finish the definitions */ call missing(email, suite,reference,errormsg,time,abortcode); /* avoid uninitialized variable notes (miss in the dataset not in hash)*/ end; /* Iterate through the hash object and output data values */ rc = iter.first(); do while (rc = 0); call execute('%Email(email='||Email||',suite='||Suite||',reference='||Reference||',error='||errormsg||',time='||time||',abortcode='||abortcode||')' ); rc = iter.next(); end; run;

Online Status	Offline
Date Last Visited	‎02-26-2024 02:16 AM

How to write a SAS program to find relationship between nodes

How to conduct impact analysis of a metadata table using SAS code

Re: Missing date column load data from Oracle to SAS

Missing date column load data from Oracle to SAS

Transfer text files from remote SFTP sever to local linux server

How to write a SAS code to extract SAS DI metadata objects?

How to identify log files differences between two log files?

Re: SAS bulkload issue

Re: SAS bulkload issue

Re: SAS bulkload issue

Re: why you can't use a macro funcation like %scan inside of a data st...

Re: How do I get the first part of the string using %scan

Re: How do I write a SAS program to re-construct the table?

Re: SAS bulkload issue

stored process questions (Stream vs Package outputs result)

Re: Advantage and disadvantage of hash table?

Re: Advantage and disadvantage of hash table?

Re: Can hash improve I/O process?

Re: Advantage and disadvantage of hash table?

Re: Advantage and disadvantage of hash table?

Re: Advantage and disadvantage of hash table?

Re: Advantage and disadvantage of hash table?

Re: Advantage and disadvantage of hash table?

Re: Advantage and disadvantage of hash table?

Buffer question

Can hash improve I/O process?

Re: Advantage and disadvantage of hash table?

Re: SAS Code issues

Re: SAS Code issues

SAS Code issues