Hi Guys,
I am in the process of investigating how I can make best use of Proc Groovy. I found the below code in one of the SAS blogs, but I just can't seem to get it working. Basically anything in uppercase are modifications that I have made to the existing code to ensure that it works in my environment.
The following is the error I get, which may be due to the fact that I have attempted to obtain a bearer key too frequently as per https://api.twitter.com/oauth2/token. So I am looking for a way to invalidate the current one.
Any help would be greatly appreciated.
ERROR: java.io.IOException: Server returned HTTP response code: 403 for URL:
%LET CONSUMER_KEY=&TWITKEY.;
%LET CONSUME_SECRET=&TWITSEC.;
%LET JSON_TWEET_FILE=E:\RESPONSECONTENT1.TXT;
%LET CSV_TWEET_FILE=E:\RESPONSECONTENT2.TXT;
%LET TWEET_QUERY=%23SASGF13+OR+%23SASUSERS+OR+%23SASGF14;
DATA _NULL_;
NJSON_TWEET_FILE=TRANWRD("&JSON_TWEET_FILE.",'\','/');
NCSV_TWEET_FILE=TRANWRD("&CSV_TWEET_FILE.",'\','/');
CALL SYMPUT('NJSON_TWEET_FILE',TRIM(NJSON_TWEET_FILE));
CALL SYMPUT('NCSV_TWEET_FILE',TRIM(NCSV_TWEET_FILE));
RUN;
/* Create a temporary file name used for the XMLMap */
%macro tempFile( fname );
%if %superq(SYSSCPL) eq %str(z/OS) or %superq(SYSSCPL) eq %str(OS/390) %then
%let recfm=recfm=vb;
%else
%let recfm=;
filename &fname TEMP lrecl=2048 &recfm;
%mend;
/* create temp files for the content and header input streams */
%tempFile(in);
%tempFile(hdrin);
/* keep the response permanently */
filename out "&JSON_TWEET_FILE.";
/* post request content is the grant_type */
data _null_;
file in;
put "grant_type=client_credentials&";
run;
/* request the bearer token by providing consumer key and secret */
data _null_;
file hdrin;
consumerKey = urlencode("&CONSUMER_KEY.");
consumerSecret = urlencode("&CONSUME_SECRET.");
encodedAccessToken = put( compress(consumerKey || ":" || consumerSecret),$base64x32767.);
put "Authorization: Basic " encodedAccessToken;
run;
proc http method="post"
in=in out=out
headerin=hdrin
url="https://api.twitter.com/oauth2/token"
ct="application/x-www-form-urlencoded;charset=UTF-8";
run;
/* Store bearer token in macro variable 'BEARER_TOKEN' */
proc groovy CLASSPATH="D:\SASHOME\SASFOUNDATION\9.3\GROOVY\EMBEDDABLE\GROOVY-ALL-2.3.3";
submit "&JSON_TWEET_FILE.";
import groovy.json.JsonSlurper
def input = new File(args[0]).text
def result = new JsonSlurper().parseText(input)
println "Recieved bearer token: ${result.access_token}"
exports.putAt('BEARER_TOKEN',result.access_token)
endsubmit;
quit;
Here is an alternate program that does not utilize PROC GROOVY at all to perform the authentication step.
%let API_KEY=<YOUR API KEY>;
%let API_SECRET=<YOUR API SECRET>;
filename head temp;
filename body temp;
filename resp temp;
data _null_;
file head lrecl=2048;
auth = put( catx(':' , symget('API_KEY') , symget('API_SECRET')) , $base64x32767.);
put 'Authorization: Basic ' auth;
run;
data _null_;
file body;
put 'grant_type=client_credentials&';
run;
proc http
url="https://api.twitter.com/oauth2/token?"
method="post"
ct="application/x-www-form-urlencoded;charset=UTF-8"
in=body
headerin=head
out=resp;
run;
data _null_;
infile resp dlm='"';
input @'access_token' + 3 bearer_token : $1024.;
call symputx('bearer_token' , bearer_token);
run;
According to the Twitter API, the oAuth2 method will return a 403 if you attempt it too frequently.
Thanks FriedEgg. I was able to locate that, but can't find a way of rectifying the situation.
I have tried the following without success.
proc http method="post"
url="https://api.twitter.com/oauth2/invalidate_token";
run;
Do you have any ideas?
First I would actually validate what the error code is that you are receiving from your post:
Before the PROC GROOVY and after the PROC HTTP add the following to your SAS session, or simply open the file through any other means you have in which to view it...
data _null_;
infile out;
input @;
put _infile_;
run;
You may also want to modify the PROC HTTP call to retireve the outbound header from the server, to help debug, with the option HEADEROUT=<fileref>, then do the same as above to view the contents of the file.
Hi FriedEgg,
There is nothing in the out file and the following in the HeadOut
HTTP/1.1 403 Forbidden
content-type:application/json; charset=utf-8
x-frame-options:DENY
x-mid:9d8aa088047514ddd72fa2ca092775963d2ed955
last-modified:Sun, 06 Jul 2014 07:41:27 GMT
status:403 Forbidden
date:Sun, 06 Jul 2014 07:41:27 GMT
x-ua-compatible:IE=edge,chrome=1
x-transaction:d9024281258f1139
vary:Accept-Encoding
pragma:no-cache
cache-control:no-cache, no-store, must-revalidate, pre-check=0, post-check=0
x-xss-protection:1; mode=block
x-content-type-options:nosniff
expires:Tue, 31 Mar 1981 05:00:00 GMT
x-runtime:0.00647
set-cookie:guest_id=v1%3A140463248796066399; Domain=.twitter.com; Path=/; Expires=Tue, 05-Jul-2016
07:41:27 UTC
set-cookie:_twitter_sess=BAh7CToHaWQiJWViYTU1YTAxMWI5Mzc1NWYxNDRkMGM1ZjgzN2UxZGEyOg9j%250AcmVhdGVkX
2F0bCsIHNieCkcBIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVy%250AOjpGbGFzaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsAOgx
jc3JmX2lkIiViZDkz%250AZDQ1YTVmYzQ0MWI0M2Q1YWQ3MTU4N2ZkMTRl
content-length:105
server:tfe
strict-transport-security:max-age=631138519
What does this tell us?
I am having the same problem , can i ask how you generated the consumer key and secret? I created an app that gave me
API key
API secret
so not sure if thats actually a consumer key and secret if it is then we have some confusion
this is what happens in my case from tghe SAS log
29 proc http method="post"
30 in=in out=out
31 headerin=hdrin
32 url="https://api.twitter.com/oauth2/token"
33 ct="application/x-www-form-urlencoded;charset=UTF-8";
34 run;
ERROR: java.io.IOException: Server returned HTTP response code: 403 for URL:
https://api.twitter.com/oauth2/token
NOTE: PROCEDURE HTTP used (Total process time):
real time 1.40 seconds
cpu time 0.00 seconds
NOTE: The SAS System stopped processing this step because of errors.
Let's simplify the program and only utilize PROC GROOVY:
%let API_KEY=<YOUR API KEY>;
%let API_SECRET=<YOUR API SECRET>;
/* Since I like to use @Grab in groovy, but SAS does not include the Ivy prerequisite, let's get Ivy */
filename ivy "%sysfunc(pathname(work,l))/ivy.jar";
proc http
method = "get"
url = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"
out = ivy;
run;
filename cp temp; *A temporary file for proc groovy output that is currently not necessary;
proc groovy classpath=cp;
add classpath=ivy; *Include the Ivy jar we downloaded;
add sasjar="groovy_2.1.3" version="2.1.3.0_SAS_20130517000930"; *You will need to adjust this line to fit your installation, this is what I have from 9.4 on 64-bit Linux. We are collecting groovy.json.JsonSlurper here, although we could get it with @Grab instead.;
submit "&API_KEY" "&API_SECRET";
@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')
@Grab(group='commons-codec', module='commons-codec', version='1.2')
import groovy.json.JsonSlurper
import org.apache.commons.codec.binary.Base64
import groovyx.net.http.HTTPBuilder
import static groovyx.net.http.ContentType.*
import static groovyx.net.http.Method.*
def getBearerToken(api_key,api_secret) {
def auth = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))
def http = new HTTPBuilder("https://api.twitter.com")
http.request(POST,JSON) { req ->
uri.path = "/oauth2/token"
headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
headers."Authorization" = "Basic "+auth
body="grant_type=client_credentials"
response.success = { resp, json ->
bearer_token = json.access_token
}
response.failure = { resp, json ->
println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
resp.headers.each { println "${it.name} : ${it.value}" }
ret = reader.getText()
println ret
}
}
return bearer_token
}
exports.bearer_token = getBearerToken(args[0].toString() , args[1].toString())
endsubmit;
quit;
%put &BEARER_TOKEN;
Here is an alternate program that does not utilize PROC GROOVY at all to perform the authentication step.
%let API_KEY=<YOUR API KEY>;
%let API_SECRET=<YOUR API SECRET>;
filename head temp;
filename body temp;
filename resp temp;
data _null_;
file head lrecl=2048;
auth = put( catx(':' , symget('API_KEY') , symget('API_SECRET')) , $base64x32767.);
put 'Authorization: Basic ' auth;
run;
data _null_;
file body;
put 'grant_type=client_credentials&';
run;
proc http
url="https://api.twitter.com/oauth2/token?"
method="post"
ct="application/x-www-form-urlencoded;charset=UTF-8"
in=body
headerin=head
out=resp;
run;
data _null_;
infile resp dlm='"';
input @'access_token' + 3 bearer_token : $1024.;
call symputx('bearer_token' , bearer_token);
run;
Thanks FriedEgg. I really appreciate how much work you have put into this.
This was a fairly entertaining endeavor. As such, I finished up a full PROC GROOVY implementation for the Twitter Search API. With this we can dump the results directly into a data set. There is still a lot that could be done to improve this, such as additional get<TYPE> methods, other than just String, and actually implementing some exception handling... Maybe another time.
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Usage Parameters
*/
%let api_key = <YOUR TWITTER API_KEY>;
%let api_secret = <YOUR TWITTER API_SECRET>;
%let search_query = %23SASGF13+OR+%23SASGF14+OR+%23SASGF15;
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Collect Ivy Jar
*/
filename ivy "%sysfunc(pathname(work,l))/ivy.jar";
proc http
method = "get"
url = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"
out = ivy;
run;
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Compile GROOVY/JAVA Classes
*/
filename cp temp;
proc groovy classpath=cp;
*add the ivy jar just downloaded and include in classpath;
*we need this to use @Grab notation below for additional dependencies;
add classpath=ivy;
*the following line requires modification based on OS and SAS Version, below accurate for 9.4M1 on Linux 64;
*this was necessary because JsonSlurper was not available initially;
*alternatively, you could collect this with @Grab, should you groovy-all.jar not contain this module;
* @Grab(group='org.codehaus.groovy', module='groovy-json', version='2.1.3');
add sasjar="groovy_2.1.3" version="2.1.3.0_SAS_20130517000930";
submit parseonly;
@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')
@Grab(group='commons-codec', module='commons-codec', version='1.2')
import groovy.json.JsonSlurper
import org.apache.commons.codec.binary.Base64
import groovyx.net.http.HTTPBuilder
import static groovyx.net.http.ContentType.JSON
import static groovyx.net.http.Method.GET
import static groovyx.net.http.Method.POST
class TwitterApi {
def api_key
def api_secret
def search(query) {
def bearer_token = getBearerToken()
def tweets = []
def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=10")
api.request(GET,JSON) { req ->
headers."Authorization" = "Bearer " + bearer_token
response.failure = { resp, json ->
println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
resp.headers.each { println "${it.name} : ${it.value}" }
ret = reader.getText()
println ret
}
response.success = { resp, json ->
json.statuses.each {
tweets << [
id: it.id,
text : it.text,
truncated : it.truncated,
createdAt : it.created_at.minus("+0000"),
userId : it.user.id,
userName : it.user.name,
userScreenName: it.user.screen_name,
location : it.user.location,
description : it.user.description,
url : it.user.url,
userFollowers: it.user.followers_count,
userFriends : it.user.friends_count,
userListed : it.user.listed_count,
retweet : it.retweet_count,
favorite : it.favorite_count
]
}
}
}
return tweets
}
private getBearerToken(){
def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))
def api = new HTTPBuilder("https://api.twitter.com")
def access_token
api.request(POST,JSON) { req ->
uri.path = "/oauth2/token"
headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
headers."Authorization" = "Basic "+encoded_basic
body = "grant_type=client_credentials"
response.success = { resp, json ->
access_token = json.access_token
}
response.failure = { resp, json ->
println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
resp.headers.each { println "${it.name} : ${it.value}" }
ret = reader.getText()
println ret
}
}
return access_token
}
}
endsubmit;
submit parseonly;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.LinkedHashMap;
public class TwitterSAS {
public String api_key = "";
public String api_secret = "";
public String search_query = "";
public void main() {
TwitterApi api = new TwitterApi();
api.setApi_key(api_key);
api.setApi_secret(api_secret);
tweets = api.search(search_query);
iter = tweets.iterator();
}
public boolean hasNext() {
return iter.hasNext();
}
public void getNext() {
tweet = ((LinkedHashMap) (iter.next()));
}
public String getString(String key) {
return tweet.get(key).toString();
}
protected ArrayList tweets;
protected Iterator iter;
protected LinkedHashMap tweet;
}
endsubmit;
quit;
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Add our PROC GROOVY output to CLASSPATH
*/
options set=classpath "%sysfunc(pathname(cp,f))";
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Collect Twitter Search Results into SAS DATA SET
*/
data twitter;
length id text truncated createdAt userId userName userScreenName location description url userFollowers userFriends userListed retweet favorite $ 140;
dcl javaobj tweets("TwitterSAS");
tweets.setStringField("api_key" , "&api_key." );
tweets.setStringField("api_secret" , "&api_secret." );
tweets.setStringField("search_query" , "&search_query." );
tweets.callVoidMethod("main");
tweets.callBooleanMethod("hasNext",rc);
do _n_=1 to 10 while(rc);
tweets.callVoidMethod("getNext");
tweets.callStringMethod("getString","id",id);
tweets.callStringMethod("getString","text",text);
tweets.callStringMethod("getString","truncated",truncated);
tweets.callStringMethod("getString","createdAt",createdAt);
tweets.callStringMethod("getString","userId",userId);
tweets.callStringMethod("getString","userName",userName);
tweets.callStringMethod("getString","userScreenName",userScreenName);
tweets.callStringMethod("getString","location",location);
tweets.callStringMethod("getString","description",description);
tweets.callStringMethod("getString","url",url);
tweets.callStringMethod("getString","userFollowers",userFollowers);
tweets.callStringMethod("getString","userFriends",userFriends);
tweets.callStringMethod("getString","userListed",userListed);
tweets.callStringMethod("getString","retweet",retweet);
tweets.callStringMethod("getString","favorite",favorite);
output;
tweets.callBooleanMethod("hasNext",rc);
end;
run;
Hi,
I am getting following error, can you help me on this.
Thanks in Advance.
Error message:
2184 data twitter;
2185 length id text truncated createdAt userId userName userScreenName location description
2185! url userFollowers userFriends userListed retweet favorite $ 140;
2186
2187
2188 dcl javaobj tweets("TwitterSAS");
2189
2190
2191 tweets.setStringField("api_key" , "&api_key." );
2192 tweets.setStringField("api_secret" , "&api_secret." );
2193 tweets.setStringField("search_query" , "&search_query." );
2194
2195
2196 tweets.callVoidMethod("main");
2197
2198
2199 tweets.callBooleanMethod("hasNext",rc);
2200 do _n_=1 to 100 while(rc);
2201 tweets.callVoidMethod("getNext");
2202 tweets.callStringMethod("getString","id",id);
2203 tweets.callStringMethod("getString","text",text);
2204 tweets.callStringMethod("getString","truncated",truncated);
2205 tweets.callStringMethod("getString","createdAt",createdAt);
2206 tweets.callStringMethod("getString","userId",userId);
2207 tweets.callStringMethod("getString","userName",userName);
2208 tweets.callStringMethod("getString","userScreenName",userScreenName);
2209 tweets.callStringMethod("getString","location",location);
2210 tweets.callStringMethod("getString","description",description);
2211 tweets.callStringMethod("getString","url",url);
2212 tweets.callStringMethod("getString","userFollowers",userFollowers);
2213 tweets.callStringMethod("getString","userFriends",userFriends);
2214 tweets.callStringMethod("getString","userListed",userListed);
2215 tweets.callStringMethod("getString","retweet",retweet);
2216 tweets.callStringMethod("getString","favorite",favorite);
2217 output;
2218 tweets.callBooleanMethod("hasNext",rc);
2219 end;
2220 run;
ERROR: Transcoding failure at line 2203 column 11.
ERROR: DATA STEP Component Object failure. Aborted during the EXECUTION phase.
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.TWITTER may be incomplete. When this step was stopped there were 0
observations and 16 variables.
NOTE: DATA statement used (Total process time):
real time 1.31 seconds
cpu time 0.01 seconds
/*Here is my java picklist*/
2221 proc javainfo picklist 'base/groovy.txt'; run;
Picklist URLs:
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/activation_1.1.1.0_SAS_201
21211183202/activation.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/groovy_1.7.1.0_SAS_2012121
1183404/groovy-all.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jansi_1.2.0.0_SAS_20121211
183401/jansi.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jaxb_ri_2.1.12.0_SAS_20121
211183328/jaxb-api.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jaxb_ri_2.1.12.0_SAS_20121
211183328/jaxb-impl.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jaxb_ri_2.1.12.0_SAS_20121
211183328/jaxb1-impl.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jaxb_ri_2.1.12.0_SAS_20121
211183328/jaxb-xjc.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/stax_api_1.0.1.a_SAS_20121
211183307/jsr173_api.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/stax_parser_4.0.8.0_SAS_20
121211183332/stax2-api.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/stax_parser_4.0.8.0_SAS_20
121211183332/wstx-asl.jar
file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/xstream_1.3.1.0_SAS_201212
11183400/xstream.jar
Total URLs: 11
/*SAS code:*/
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Usage Parameters
*/
%let api_key = <API_KEY>;
%let api_secret = <API_SECRET>;
/* %let search_query = %23SASGF13+OR+%23SASGF14+OR+%23SASGF15;*/
/* %let search_query = %23SASJOBS;*/
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Collect Ivy Jar
*/
/* filename ivy "%sysfunc(pathname(work,l))/ivy.jar";*/
/* */
/* */
/* proc http*/
/* method = "get"*/
/*/* url = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"*/*/
/* url = "C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/groovy_1.7.1.0_SAS_20121211183404/groovy-all.jar"*/
/* out = ivy;*/
/* run;*/
;
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Compile GROOVY/JAVA Classes
*/
filename cp temp;
proc groovy classpath=cp;
*add the ivy jar just downloaded and include in classpath;
*we need this to use @Grab notation below for additional dependencies;
/* add classpath=ivy;*/
add classpath="C:\groovy-2.4.0-rc-2\embeddable\groovy-all-2.4.0-rc-2.jar";
*the following line requires modification based on OS and SAS Version, below accurate for 9.4M1 on Linux 64;
*this was necessary because JsonSlurper was not available initially;
*alternatively, you could collect this with @Grab, should you groovy-all.jar not contain this module;
* @Grab(group='org.codehaus.groovy', module='groovy-json', version='2.1.3');
/* add sasjar="groovy_2.1.3" version="2.1.3.0_SAS_20130517000930";*/
/* add sasjar="groovy_2.4.0" version="2.4.0.0_SAS_20121211183404";*/
submit parseonly;
@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')
@Grab(group='commons-codec', module='commons-codec', version='1.2')
import groovy.json.JsonSlurper
import org.apache.commons.codec.binary.Base64
import groovyx.net.http.HTTPBuilder
import static groovyx.net.http.ContentType.JSON
import static groovyx.net.http.Method.GET
import static groovyx.net.http.Method.POST
class TwitterApi {
def api_key
def api_secret
def search(query) {
def bearer_token = getBearerToken()
def tweets = []
def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=100")
api.request(GET,JSON) { req ->
headers."Authorization" = "Bearer " + bearer_token
response.failure = { resp, json ->
println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
resp.headers.each { println "${it.name} : ${it.value}" }
ret = reader.getText()
println ret
}
response.success = { resp, json ->
json.statuses.each {
tweets << [
id: it.id,
text : it.text,
truncated : it.truncated,
createdAt : it.created_at.minus("+0000"),
userId : it.user.id,
userName : it.user.name,
userScreenName: it.user.screen_name,
location : it.user.location,
description : it.user.description,
url : it.user.url,
userFollowers: it.user.followers_count,
userFriends : it.user.friends_count,
userListed : it.user.listed_count,
retweet : it.retweet_count,
favorite : it.favorite_count
]
}
}
}
return tweets
}
private getBearerToken(){
def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))
def api = new HTTPBuilder("https://api.twitter.com")
def access_token
api.request(POST,JSON) { req ->
uri.path = "/oauth2/token"
headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
headers."Authorization" = "Basic "+encoded_basic
body = "grant_type=client_credentials"
response.success = { resp, json ->
access_token = json.access_token
}
response.failure = { resp, json ->
println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
resp.headers.each { println "${it.name} : ${it.value}" }
ret = reader.getText()
println ret
}
}
return access_token
}
}
endsubmit;
submit parseonly;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.LinkedHashMap;
public class TwitterSAS {
public String api_key = "";
public String api_secret = "";
public String search_query = "";
public void main() {
TwitterApi api = new TwitterApi();
api.setApi_key(api_key);
api.setApi_secret(api_secret);
tweets = api.search(search_query);
iter = tweets.iterator();
}
public boolean hasNext() {
return iter.hasNext();
}
public void getNext() {
tweet = ((LinkedHashMap) (iter.next()));
}
public String getString(String key) {
return tweet.get(key).toString();
}
protected ArrayList tweets;
protected Iterator iter;
protected LinkedHashMap tweet;
}
endsubmit;
quit;
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Add our PROC GROOVY output to CLASSPATH
*/
options set=classpath "%sysfunc(pathname(cp,f))";
%let search_query = %23SAS;
/*-----------------------------------------------------------------------------------------------------------------------------------------
*-Collect Twitter Search Results into SAS DATA SET
*/
data twitter;
length id text truncated createdAt userId userName userScreenName location description url userFollowers userFriends userListed retweet favorite $ 140;
dcl javaobj tweets("TwitterSAS");
tweets.setStringField("api_key" , "&api_key." );
tweets.setStringField("api_secret" , "&api_secret." );
tweets.setStringField("search_query" , "&search_query." );
tweets.callVoidMethod("main");
tweets.callBooleanMethod("hasNext",rc);
do _n_=1 to 100 while(rc);
tweets.callVoidMethod("getNext");
tweets.callStringMethod("getString","id",id);
tweets.callStringMethod("getString","text",text);
tweets.callStringMethod("getString","truncated",truncated);
tweets.callStringMethod("getString","createdAt",createdAt);
tweets.callStringMethod("getString","userId",userId);
tweets.callStringMethod("getString","userName",userName);
tweets.callStringMethod("getString","userScreenName",userScreenName);
tweets.callStringMethod("getString","location",location);
tweets.callStringMethod("getString","description",description);
tweets.callStringMethod("getString","url",url);
tweets.callStringMethod("getString","userFollowers",userFollowers);
tweets.callStringMethod("getString","userFriends",userFriends);
tweets.callStringMethod("getString","userListed",userListed);
tweets.callStringMethod("getString","retweet",retweet);
tweets.callStringMethod("getString","favorite",favorite);
output;
tweets.callBooleanMethod("hasNext",rc);
end;
run;
Below is log summary.
Please advise what options need to be changed.
2224 proc options group=languagecontrol; run;
SAS (r) Proprietary Software Release 9.4 TS1M2
Group=LANGUAGECONTROL
DATESTYLE=MDY Specifies the sequence of month, day, and year when ANYDTDTE, ANYDTDTM, or
ANYDTTME informat data is ambiguous.
DFLANG=ENGLISH Specifies the language for international date informats and formats.
DS2ACCEL=NONE Provides support for DS2 code pass-through acceleration.
DSACCEL=NONE Provides support for code pass-through acceleration.
EXTENDOBSCOUNTER=YES
Specifies whether to extend the maximum number of observations in a new SAS data
file.
LOCALEDATA=SASLOCALE
Specifies the location of the locale database.
NOLOGLANGCHG Disables changing the language of the SAS output when the LOCALE= option is
changed.
NOLOGLANGENG Write SAS log messages based on the values of the LOGLANGCHG, LSWLANG=, and
LOCALE= options when SAS started.
LSWLANG=LOCALE Specifies the language for SAS log and ODS messages when the LOCALE= option is set
after SAS starts.
MAPEBCDICTOASCII= Specifies the transcoding table that is used to convert characters from ASCII to
EBCDIC and EBCDIC to ASCII.
NONLDECSEPARATOR Disables formatting of numeric output using the decimal separator for the locale.
NOODSLANGCHG Disables changing the language of the SAS message text in ODS output when the
LOCALE option is set after start up.
PAPERSIZE=LETTER Specifies the paper size to use for printing.
RSASIOTRANSERROR Displays a transcoding error when illegal values are read from a remote
application.
TIMEZONE= Specifies a time zone.
TRANTAB=(lat1lat1,lat1lat1,wlt1_ucs,wlt1_lcs,wlt1_ccl,,,)
Specifies the translation table catalog entries.
URLENCODING=SESSION
Specifies whether the argument to the URLENCODE function and to the URLDECODE
function is interpreted using the SAS session encoding or UTF-8 encoding.
NODBCS Disables double-byte character sets.
DBCSLANG=NONE Specifies a double-byte character set language.
DBCSTYPE=NONE Specifies the encoding method to use for a double-byte character set.
ENCODING=WLATIN1 Specifies the default character-set encoding for the SAS session.
LOCALE=EN_US Specifies a set of attributes in a SAS session that reflect the language, local
conventions, and culture for a geographical region.
NONLSCOMPATMODE Encodes data using the SAS session encoding.
NOTE: PROCEDURE OPTIONS used (Total process time):
real time 0.04 seconds
cpu time 0.00 seconds
The options you need to change are only effective on SAS invocation, so one more this to check is: how are you interacting with SAS? Are you using EG or another of the GUI tools? If so, are you using a local install or connecting to a remote server. In the case of the remote server, you are out of luck (unless you can alter these options for your entire site). If you are using SAS directly, then you can just choose to open it with a UTF-8 session through either a start menu link in Windows or the sas_u8 script in /SASHome/SASFoundation/9.X/bin in Linux. Specifically the options that are most likely causing your problems are NODBCS and ENCODING. If you are not able to modify these options, there are other things we can do as well. The most simple of which would be to change your query:
%let search_query=%23SAS&lang=en;
This will ask twitter to filter tweets, as best it can, to those identified as being written in the English language. This will not prevent problems like this completely, however. The next step would be to modify the groovy code to translate these characters or to create a trantab table in SAS to add the necessary compatibility.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 16. Read more here about why you should contribute and what is in it for you!
Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.
Find more tutorials on the SAS Users YouTube channel.
Select SAS Training centers are offering in-person courses. View upcoming courses for: