Thank you Reeza, Just a quick question, I presume you used regular expressions with Python, which can also be used within SAS. Is that the main approach you use? ie, what is the advantage of going with Python vs. SAS with regular expressions? I did come across natural language processing toolkit in Python, not sure how good that is... I am not that familiar with Python, but familiar with other programming languages, and R. If Python provides a strong tool to analyse natural language that is not provided through other programming environment then it is worthwhile knowing that and learning the language. Keen to hear your feedback.     ... View more

Thank you @RW9 for taking the time to answer this question. I fully agree with you on every single point you mentioned and especially regarding the importance capturing data in a structured form to begin with. However, and as you know too well, the world is not that structured nor that easy to deal with. Being a physician, a researcher and a dedicated coder/programmer, I have been dealing with the full spectrum of the problem. I have worked with pharmaceutical sponsored studies, the data capture forms are tedious and no one outside these well sponsored studies will ever use these detailed forms on a daily basis. I tried designing webforms, for use on desktops and smart phones. I tried splitting data collection so that the patients will fill their part and the physicians will do theirs. I even used digitally readable form (where the responder would check a textbox and the form would then be converted to structured data after scanning the answer sheet and putting it through an OCR software. The problem: missing data (huge problem) as people ignore fileds or dont have time to fill in the forms. Forcing them to complete a filed is not the answer either. The more complex and structured form you create to collect more data, the less answers you get. Combining different modalities makes some aspect of the problem easier but can quickly build up to a very complex system relative to the task needed to be accomplished.  Employing clerks/data analysts to go through captured data is very slow, very expensive and prone t errors.  The other problem is, as you know, that there are numerous programs and solutions used out there for capturing information. Each department and each agency has its own systems. IT units in different companies adapt different solutions and systems. However, at the end of the day, all the data is stored in a database, be it Oracle, or SQL or something else, and it is there that work should be done. You would imagine that, living in a time when Googles AI beats the world nr 1 at "Go" that we would have the tools to do more complex tasks than just working with structured data (just a metaphore, I understand the difference in the problem description here) What I am doing now is a combination of bits and pieces to solve the problem. Matching the text against a table with the name of the surgeons would tells me who did the procedure, patient name is extracted by getting the string between (Mr/Mrs/Miss/Master) and a date regular expression (representing date of birth) and so forth Then splitting the text into sentences. Then applying  keywords (like "Stomach)  to extract the sentence containing information about the stomach. In that sentence, I would look for expected results (like Ulcer). An example of the problem at this level is the following sentences return a positive result for ulcer: (there was an ulcer) ( there was no ulcer) (no identifiable ulcer was seen) (examination reveled healing ulcer) (After through inspection, an ulcer couldnt be seen) etc will all be picked by reg expression as containing the key-term "ulcer" and the code is not that smart to understand negation, and it is virtually impossible to include every possible way of negating the presence of an ulcer in a simple code format. Maybe text miner could pick this up but note sure if it could be coded and embedded at part of a SAS code.  This is just one example of a problem that you might run into with reg expression (and the list could get very long) Yet, you would think (maybe with machine learning?) that a set of general rules could be programmed into the code. Like: IF sentence X contains the term "ulcer" AND there is no negation THEN Ulcer_variable=1 ELSE Ulcer_Variable=0   Maybe wishful thinking, but as data volume, computing power and the sophistication of coding (and human laziness) increase, asking humans to input data using structured forms is not the way forward. We need tools to understand and analyse natural human language     As I said, maybe wishful thinking...  at least today   Thanks  AM ... View more