I tried running the code below, to retrieve 1000 tweets by changing the count value to 1000 and changing the _n_ value of the do loop. In the output, I only see 47 tweets. I was wondering what “rc” indicates in this code. When I comment  /*tweets.callBooleanMethod("hasNext",rc);*/,and run the program, I get 1000 observations. However, after the 47th observation, all the remaining observations until the 1000th one is simply the 47th line repeated. /*----------------------------------------------------------------------------------------------------------------------------------------- *-Usage Parameters */     %let api_key      =     %let api_secret   =     %let search_query = %23AMAZONFIRE+OR+%23AMAZONSMARTPHONE; /*----------------------------------------------------------------------------------------------------------------------------------------- *-Collect Ivy Jar */     filename ivy "%sysfunc(pathname(work,l))/ivy.jar";     proc http        method = "get"        url    = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"        out    = ivy;     run;      /*----------------------------------------------------------------------------------------------------------------------------------------- *-Compile GROOVY/JAVA Classes */     filename cp  temp;     proc groovy classpath=cp;     *add the ivy jar just downloaded and include in classpath;     *we need this to use @Grab notation below for additional dependencies;     add classpath=ivy;     *the following line requires modification based on OS and SAS Version, below accurate for 9.4M1 on Linux 64;     *this was necessary because JsonSlurper was not available initially;     *alternatively, you could collect this with @Grab, should you groovy-all.jar not contain this module;     * @Grab(group='org.codehaus.groovy', module='groovy-json', version='2.1.3');     add classpath="C:\Users\Resh\Downloads\groovy-all-2.3.4.jar";     submit parseonly;     @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')     @Grab(group='commons-codec', module='commons-codec', version='1.2')     import groovy.json.JsonSlurper     import org.apache.commons.codec.binary.Base64     import groovyx.net.http.HTTPBuilder     import static groovyx.net.http.ContentType.JSON     import static groovyx.net.http.Method.GET     import static groovyx.net.http.Method.POST     class TwitterApi {         def api_key         def api_secret         def search(query) {             def bearer_token = getBearerToken()             def tweets = []             def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=100")             api.request(GET,JSON) { req ->                 headers."Authorization" = "Bearer " + bearer_token                 response.failure = { resp, json ->                     println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"                     resp.headers.each { println "${it.name} : ${it.value}" }                     ret = reader.getText()                     println ret                 }                 response.success = { resp, json ->                     json.statuses.each {                         tweets << [                                 id: it.id,                                 text         : it.text,                                 truncated    : it.truncated,                                 createdAt    : it.created_at.minus("+0000"),                                 retweet      : it.retweet_count                                                         ]                     }                 }             }             return tweets         }         private getBearerToken(){             def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))             def api = new HTTPBuilder("https://api.twitter.com")             def access_token             api.request(POST,JSON) { req ->                 uri.path = "/oauth2/token"                 headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"                 headers."Authorization" = "Basic "+encoded_basic                 body = "grant_type=client_credentials"                 response.success = { resp, json ->                     access_token = json.access_token                 }                 response.failure = { resp, json ->                     println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"                     resp.headers.each { println "${it.name} : ${it.value}" }                     ret = reader.getText()                     println ret                 }             }             return access_token         }     }     endsubmit;     submit parseonly;     import java.util.ArrayList;     import java.util.Iterator;     import java.util.LinkedHashMap;     public class TwitterSAS {         public String api_key = "";         public String api_secret = "";         public String search_query = "";         public void main() {             TwitterApi api = new TwitterApi();             api.setApi_key(api_key);             api.setApi_secret(api_secret);             tweets = api.search(search_query);             iter = tweets.iterator();         }         public boolean hasNext() {             return iter.hasNext();         }         public void getNext() {             tweet = ((LinkedHashMap) (iter.next()));         }         public String getString(String key) {             return tweet.get(key).toString();         }         protected ArrayList tweets;         protected Iterator iter;         protected LinkedHashMap tweet;     }     endsubmit;     quit; /*----------------------------------------------------------------------------------------------------------------------------------------- *-Add our PROC GROOVY output to CLASSPATH */     options set=classpath "%sysfunc(pathname(cp,f))"; /*----------------------------------------------------------------------------------------------------------------------------------------- *-Collect Twitter Search Results into SAS DATA SET */     data twitter;        length id text truncated createdAt retweet  $ 140;        dcl javaobj tweets("TwitterSAS");        tweets.setStringField("api_key"      , "&api_key."      );        tweets.setStringField("api_secret"   , "&api_secret."   );        tweets.setStringField("search_query" , "&search_query." );        tweets.callVoidMethod("main");        tweets.callBooleanMethod("hasNext",rc);        do _n_=1 to 100 while(rc);           tweets.callVoidMethod("getNext");           tweets.callStringMethod("getString","id",id);           tweets.callStringMethod("getString","text",text);           tweets.callStringMethod("getString","truncated",truncated);           tweets.callStringMethod("getString","createdAt",createdAt);                     tweets.callStringMethod("getString","retweet",retweet);                     output;          tweets.callBooleanMethod("hasNext",rc);        end; run; ... View more

Since it's a jar file, do we need to install a copy of JAVA Runtime Environment to be able to successfully reference it and eventually run the code or is that not required. Currently I don't have any software that can open the jar file ... View more

Below is the proc groovy statement: proc groovy classpath=cp;             add classpath=ivy;   add classpath="M:\nandanm\groovy-2.3.4.zip";                  submit parseonly; @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7') @Grab(group='commons-codec', module='commons-codec', version='1.2')     import groovy.json.JsonSlurper     import org.apache.commons.codec.binary.Base64     import groovyx.net.http.HTTPBuilder     import static groovyx.net.http.ContentType.JSON     import static groovyx.net.http.Method.GET     import static groovyx.net.http.Method.POST     class TwitterApi {         def api_key         def api_secret         def search(query) {             def bearer_token = getBearerToken()             def tweets = []             def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=10") api.request(GET,JSON) { req -> headers."Authorization" = "Bearer " + bearer_token response.success = { resp, json -> json.statuses.each { tweets << [                                 id: it.id,                                 text         : it.text,                                 truncated    : it.truncated,                                 createdAt    : it.created_at.minus("+0000"),                                 userId       : it.user.id,                                 userName     : it.user.name,                                 userScreenName: it.user.screen_name,                                 location     : it.user.location, description  : it.user.description,                                 url          : it.user.url,                                 userFollowers: it.user.followers_count, userFriends  : it.user.friends_count,                                 userListed   : it.user.listed_count,                                 retweet      : it.retweet_count,                                 favorite     : it.favorite_count ]                     }                 }             }             return tweets         }         private getBearerToken(){             def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))             def api = new HTTPBuilder("https://api.twitter.com")             def access_token api.request(POST,JSON) { req -> uri.path = "/oauth2/token" headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8" headers."Authorization" = "Basic "+encoded_basic                 body = "grant_type=client_credentials" response.success = { resp, json -> access_token = json.access_token                 }             }             return access_token         }     }     endsubmit;     submit parseonly;     import java.util.ArrayList;     import java.util.Iterator;     import java.util.LinkedHashMap;     public class TwitterSAS {         public String api_key = "";         public String api_secret = "";         public String search_query = "";           public void main() {             TwitterApi api = new TwitterApi(); api.setApi_key(api_key); api.setApi_secret(api_secret);             tweets = api.search(search_query);             iter = tweets.iterator();         }         public boolean hasNext() {             return iter.hasNext();         }         public void getNext() {             tweet = ((LinkedHashMap) (iter.next()));         }         public String getString(String key) {             return tweet.get(key).toString();         }         protected ArrayList tweets;         protected Iterator iter;         protected LinkedHashMap tweet;     }     endsubmit;     quit; The code gives the following error. Below is the Entire log: 1 The SAS System                                21:46 Sunday, July 13, 2014 1 ;*';*";*/;quit;run; 2          OPTIONS PAGENO=MIN; 3          %LET _CLIENTTASKLABEL='Program'; 4          %LET _CLIENTPROJECTPATH='M:\nandanm\sasGF2014.egp'; 5          %LET _CLIENTPROJECTNAME='sasGF2014.egp'; 6          %LET _SASPROGRAMFILE=; 7          8          ODS _ALL_ CLOSE; 9          OPTIONS DEV=ACTIVEX; 10         GOPTIONS XPIXELS=0 YPIXELS=0; 11         FILENAME EGSR TEMP; 12         ODS tagsets.sasreport13(ID=EGSR) FILE=EGSR STYLE=HtmlBlue 12       ! STYLESHEET=(URL="file:///C:/Program%20Files/SASHome/SASEnterpriseGuide/5.1/Styles/HtmlBlue.css") NOGTITLE NOGFOOTNOTE 12       ! GPATH=&sasworklocation ENCODING=UTF8 options(rolap="on"); NOTE: Writing TAGSETS.SASREPORT13(EGSR) Body file: EGSR 13         14         GOPTIONS ACCESSIBLE; 15         proc 15       !      groovy classpath=cp; NOTE: The ADD CLASSPATH command completed. 16                   16       !  add classpath=ivy; NOTE: The ADD CLASSPATH command completed. 17           add classpath="M:\nandanm\groovy-2.3.4.zip"; NOTE: The ADD CLASSPATH command completed. 18                   19         20             submit parseonly; 21         22         23 @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7') 24 @Grab(group='commons-codec', module='commons-codec', version='1.2') 25         26         27             import groovy.json.JsonSlurper 28             import org.apache.commons.codec.binary.Base64 29             import groovyx.net.http.HTTPBuilder 30             import static groovyx.net.http.ContentType.JSON 31             import static groovyx.net.http.Method.GET 32             import static groovyx.net.http.Method.POST 33         34         35             class TwitterApi { 36                 def api_key 37                 def api_secret 38         39         40                 def search(query) { 41 def bearer_token = getBearerToken() 42 def tweets = [] 43         44         45                     def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=10") 46         47         48 api.request(GET,JSON) { req -> 49 headers."Authorization" = "Bearer " + bearer_token 50         2 The SAS System                                21:46 Sunday, July 13, 2014 51         52 response.success = { resp, json -> 53                             json.statuses.each { 54                                 tweets << [ 55                                         id: it.id, 56 text         : it.text, 57                                         truncated    : it.truncated, 58 createdAt    : it.created_at.minus("+0000"), 59 userId       : it.user.id, 60 userName     : it.user.name, 61 userScreenName: it.user.screen_name, 62 location     : it.user.location, 63 description  : it.user.description, 64                                         url          : it.user.url, 65 userFollowers: it.user.followers_count, 66 userFriends  : it.user.friends_count, 67                                         userListed   : it.user.listed_count, 68 retweet      : it.retweet_count, 69 favorite     : it.favorite_count 70                                 ] 71                             } 72 } 73 } 74 return tweets 75                 } 76         77         78 private getBearerToken(){ 79 def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes)) 80 def api = new HTTPBuilder("https://api.twitter.com") 81 def access_token 82 api.request(POST,JSON) { req -> 83 uri.path = "/oauth2/token" 84 headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8" 85 headers."Authorization" = "Basic "+encoded_basic 86 body = "grant_type=client_credentials" 87         88         89 response.success = { resp, json -> 90                             access_token = json.access_token 91 } 92                     } 93 return access_token 94                 } 95             } 96         97         98 endsubmit; ERROR: The SUBMIT command failed. org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: script14053113013111409775131.groovy: 3: unable to resolve class groovy.json.JsonSlurper @ line 3, column 5. @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')        ^ 1 error      at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:296)      at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:860) 3 The SAS System                                21:46 Sunday, July 13, 2014      at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:521)      at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497)      at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474)      at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:292)      at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)      at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:207)      at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:217) 99         100        101            submit parseonly; 102        103        104            import java.util.ArrayList; 105            import java.util.Iterator; 106            import java.util.LinkedHashMap; 107        108        109            public class TwitterSAS { 110 public String api_key = ""; 111 public String api_secret = ""; 112 public String search_query = ""; 113        114 public void main() { 115 TwitterApi api = new TwitterApi(); 116 api.setApi_key(api_key); 117 api.setApi_secret(api_secret); 118 tweets = api.search(search_query); 119 iter = tweets.iterator(); 120                } 121        122        123 public boolean hasNext() { 124                    return iter.hasNext(); 125                } 126        127        128 public void getNext() { 129 tweet = ((LinkedHashMap) (iter.next())); 130                } 131        132        133                public String getString(String key) { 134 return tweet.get(key).toString(); 135                } 136        137        138 protected ArrayList tweets; 139 protected Iterator iter; 140 protected LinkedHashMap tweet; 141            } 142        143        144 endsubmit; ERROR: The SUBMIT command failed. org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: script14053113013901958481143.groovy: 14: unable to resolve class TwitterApi  @ line 14, column 24. TwitterApi api = new TwitterApi(); ^ 4 The SAS System                                21:46 Sunday, July 13, 2014 script14053113013901958481143.groovy: 14: unable to resolve class TwitterApi  @ line 14, column 30. TwitterApi api = new TwitterApi();                                 ^ 2 errors      at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:296)      at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:860)      at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:521)      at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497)      at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474)      at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:292)      at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)      at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:207)      at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:217) 145        146        147            quit; NOTE: The SAS System stopped processing this step because of errors. NOTE: PROCEDURE GROOVY used (Total process time):       real time           0.10 seconds       cpu time            0.00 seconds       148        149        GOPTIONS NOACCESSIBLE; 150        %LET _CLIENTTASKLABEL=; 151        %LET _CLIENTPROJECTPATH=; 152        %LET _CLIENTPROJECTNAME=; 153        %LET _SASPROGRAMFILE=; 154        155 ;*';*";*/;quit;run; 156        ODS _ALL_ CLOSE; 157        158        159        QUIT; RUN; 160        ... View more

Here is the entire log with the warning message that I was taking about. 1 ;*';*";*/;quit;run; 2          OPTIONS PAGENO=MIN; 3          %LET _CLIENTTASKLABEL='Program'; 4          %LET _CLIENTPROJECTPATH='M:\nandanm\sasGF2014.egp'; 5          %LET _CLIENTPROJECTNAME='sasGF2014.egp'; 6          %LET _SASPROGRAMFILE=; 7          8          ODS _ALL_ CLOSE; 9          OPTIONS DEV=ACTIVEX; 10         GOPTIONS XPIXELS=0 YPIXELS=0; 11         FILENAME EGSR TEMP; 12         ODS tagsets.sasreport13(ID=EGSR) FILE=EGSR STYLE=HtmlBlue 12       ! STYLESHEET=(URL="file:///C:/Program%20Files/SASHome/SASEnterpriseGuide/5.1/Styles/HtmlBlue.css") NOGTITLE NOGFOOTNOTE 12       ! GPATH=&sasworklocation ENCODING=UTF8 options(rolap="on"); NOTE: Writing TAGSETS.SASREPORT13(EGSR) Body file: EGSR 13         14         GOPTIONS ACCESSIBLE; 15         16             submit parseonly; 17         18         19             @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7') 20 @Grab(group='commons-codec', module='commons-codec', version='1.2') 21         22         23             import groovy.json.JsonSlurper 24             import org.apache.commons.codec.binary.Base64 25             import groovyx.net.http.HTTPBuilder 26             import static groovyx.net.http.ContentType.JSON 27             import static groovyx.net.http.Method.GET 28             import static groovyx.net.http.Method.POST 29         30         31             class TwitterApi { 32                 def api_key 33                 def api_secret 34         35         36                 def search(query) { WARNING: Apparent symbolic reference COUNT not resolved. 37 def bearer_token = getBearerToken() 38 def tweets = [] 39         40         41 def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=10") 42         43         44 api.request(GET,JSON) { req -> 45 headers."Authorization" = "Bearer " + bearer_token 46         47         48 response.success = { resp, json -> 49                             json.statuses.each { 50                                 tweets << [ 51                                         id: it.id, 52 text         : it.text, 53                                         truncated    : it.truncated, 54 createdAt    : it.created_at.minus("+0000"), 2 The SAS System                                21:46 Sunday, July 13, 2014 55 userId       : it.user.id, 56 userName     : it.user.name, 57 userScreenName: it.user.screen_name, 58                                         location     : it.user.location, 59 description  : it.user.description, 60 url          : it.user.url, 61 userFollowers: it.user.followers_count, 62 userFriends  : it.user.friends_count, 63 userListed   : it.user.listed_count, 64                                         retweet      : it.retweet_count, 65 favorite     : it.favorite_count 66                                 ] 67                             } 68 } 69                     } 70 return tweets 71                 } 72         73         74 private getBearerToken(){ 75 def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes)) 76                     def api = new HTTPBuilder("https://api.twitter.com") 77 def access_token 78 api.request(POST,JSON) { req -> 79 uri.path = "/oauth2/token" 80 headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8" 81 headers."Authorization" = "Basic "+encoded_basic 82 body = "grant_type=client_credentials" 83         84         85 response.success = { resp, json -> 86                             access_token = json.access_token 87 } 88 } 89 return access_token 90                 } 91             } 92         93         94 endsubmit; 95         96         97             submit parseonly; 98         99         100            import java.util.ArrayList; 101            import java.util.Iterator; 102            import java.util.LinkedHashMap; 103        104        105            public class TwitterSAS { 106 public String api_key = ""; 107 public String api_secret = ""; 108 public String search_query = ""; 109        110 public void main() { 111                    TwitterApi api = new TwitterApi(); 112 api.setApi_key(api_key); 3 The SAS System                                21:46 Sunday, July 13, 2014 113 api.setApi_secret(api_secret); 114 tweets = api.search(search_query); 115 iter = tweets.iterator(); 116                } 117        118        119 public boolean hasNext() { 120 return iter.hasNext(); 121                } 122        123        124 public void getNext() { 125 tweet = ((LinkedHashMap) (iter.next())); 126                } 127        128        129                public String getString(String key) { 130 return tweet.get(key).toString(); 131                } 132        133        134 protected ArrayList tweets; 135 protected Iterator iter; 136                protected LinkedHashMap tweet; 137            } 138        139        140 endsubmit; 141        142        143            quit; 144        145        GOPTIONS NOACCESSIBLE; 146        %LET _CLIENTTASKLABEL=; 147        %LET _CLIENTPROJECTPATH=; 148        %LET _CLIENTPROJECTNAME=; 149        %LET _SASPROGRAMFILE=; 150        151 ;*';*";*/;quit;run; 152        ODS _ALL_ CLOSE; 153        154        155        QUIT; RUN; 156        ... View more

Now that I checked and put in the full path, the code ran but it gave a warning saying: Apparent symbolic reference COUNT not resolved. Would that pose a problem going forward? ... View more

Here's the problem. When I run "Compile Groovy/Java Classes" - this part of the code both the submit commands fail to run and heres the error that pops up. let me know if I need to provide further information. ERROR: The SUBMIT command failed. org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: script14053071187391409775131.groovy: 3: unable to resolve class groovy.json.JsonSlurper @ line 3, column 5. @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7') ^ ERROR: The SUBMIT command failed. org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: 4 The SAS System 21:46 Sunday, July 13, 2014 script14053071188481958481143.groovy: 14: unable to resolve class TwitterApi @ line 14, column 24. TwitterApi api = new TwitterApi(); ^ script14053071188481958481143.groovy: 14: unable to resolve class TwitterApi @ line 14, column 30. TwitterApi api = new TwitterApi(); ^ Thanks ... View more

Thanks for your help. It did help me get past the first hurdle but I still can't run the submit commands successfully. ... View more

Thanks for your response fried egg. You are correct  when you say that I'm using a virtual machine. The specific part of the code that's giving me an issue is - Compile GROOVY/JAVA Classes. When I run this part of the code, I get the following errors: ERROR: Failed to add sasjar groovy_2.3.4 (I downloaded this version so I changed it from 2.3.1 in your code to 2.3.4 in mine). And I get this error twice: The SUBMIT command failed. Can you help me fix this? Thanks ... View more

Hey Guys, I have my API Key and API secret. But I need to know a stepwise way to write the code to fetch tweets from twitter. I understand that we need to initialize the key and secret but if anyone could articulate one piece of code that would get the job done,I would appreciate it. I am using a Macintosh by the way. Thanks ... View more