Proc Groovy\HTTP Headaches

[Scott_Mitchell]

Hi Guys,

I am in the process of investigating how I can make best use of Proc Groovy.  I found the below code in one of the SAS blogs, but I just can't seem to get it working.  Basically anything in uppercase are modifications that I have made to the existing code to ensure that it works in my environment.

The following is the error I get, which may be due to the fact that I have attempted to obtain a bearer key too frequently as per https://api.twitter.com/oauth2/token.  So I am looking for a way to invalidate the current one.

Any help would be greatly appreciated.

ERROR: java.io.IOException: Server returned HTTP response code: 403 for URL:

%LET CONSUMER_KEY=&TWITKEY.;

%LET CONSUME_SECRET=&TWITSEC.;

%LET JSON_TWEET_FILE=E:\RESPONSECONTENT1.TXT;

%LET CSV_TWEET_FILE=E:\RESPONSECONTENT2.TXT;

%LET TWEET_QUERY=%23SASGF13+OR+%23SASUSERS+OR+%23SASGF14;

DATA _NULL_;

NJSON_TWEET_FILE=TRANWRD("&JSON_TWEET_FILE.",'\','/');

NCSV_TWEET_FILE=TRANWRD("&CSV_TWEET_FILE.",'\','/');

CALL SYMPUT('NJSON_TWEET_FILE',TRIM(NJSON_TWEET_FILE));

CALL SYMPUT('NCSV_TWEET_FILE',TRIM(NCSV_TWEET_FILE));

RUN;

/* Create a temporary file name used for the XMLMap */

%macro tempFile( fname );

  %if %superq(SYSSCPL) eq %str(z/OS) or %superq(SYSSCPL) eq %str(OS/390) %then

    %let recfm=recfm=vb;

  %else

    %let recfm=;

  filename &fname TEMP lrecl=2048 &recfm;

%mend;

/* create temp files for the content and header input streams */

%tempFile(in);

%tempFile(hdrin);

/* keep the response permanently */

filename out "&JSON_TWEET_FILE.";

/* post request content is the grant_type */

data _null_;

   file in;

   put "grant_type=client_credentials&";

run;

/* request the bearer token by providing consumer key and secret */

data _null_;

   file hdrin;

   consumerKey = urlencode("&CONSUMER_KEY.");

   consumerSecret = urlencode("&CONSUME_SECRET.");

   encodedAccessToken  = put( compress(consumerKey || ":" || consumerSecret),$base64x32767.);

   put "Authorization: Basic " encodedAccessToken;

run;

proc http method="post"

    in=in out=out

    headerin=hdrin

    url="https://api.twitter.com/oauth2/token"

    ct="application/x-www-form-urlencoded;charset=UTF-8";

run;

/* Store bearer token in macro variable 'BEARER_TOKEN' */

proc groovy CLASSPATH="D:\SASHOME\SASFOUNDATION\9.3\GROOVY\EMBEDDABLE\GROOVY-ALL-2.3.3";

    submit "&JSON_TWEET_FILE.";

      import groovy.json.JsonSlurper

      def input = new File(args[0]).text

      def result = new JsonSlurper().parseText(input)

      println "Recieved bearer token: ${result.access_token}"

      exports.putAt('BEARER_TOKEN',result.access_token)

    endsubmit;

quit;

[FriedEgg]

According to the Twitter API, the oAuth2 method will return a 403 if you attempt it too frequently.

https://dev.twitter.com/docs/api/1.1/post/oauth2/token

[Scott_Mitchell]

Thanks FriedEgg.  I was able to locate that, but can't find a way of rectifying the situation.

I have tried the following without success.

proc http method="post"

    url="https://api.twitter.com/oauth2/invalidate_token";

run;

Do you have any ideas?

[FriedEgg]

First I would actually validate what the error code is that you are receiving from your post:

Before the PROC GROOVY and after the PROC HTTP add the following to your SAS session, or simply open the file through any other means you have in which to view it...

data _null_;

infile out;

input @;

put _infile_;

run;

You may also want to modify the PROC HTTP call to retireve the outbound header from the server, to help debug, with the option HEADEROUT=<fileref>, then do the same as above to view the contents of the file.

[Scott_Mitchell]

Hi FriedEgg,

There is nothing in the out file and the following in the HeadOut

HTTP/1.1 403 Forbidden

content-type:application/json; charset=utf-8

x-frame-options:DENY

x-mid:9d8aa088047514ddd72fa2ca092775963d2ed955

last-modified:Sun, 06 Jul 2014 07:41:27 GMT

status:403 Forbidden

date:Sun, 06 Jul 2014 07:41:27 GMT

x-ua-compatible:IE=edge,chrome=1

x-transaction:d9024281258f1139

vary:Accept-Encoding

pragma:no-cache

cache-control:no-cache, no-store, must-revalidate, pre-check=0, post-check=0

x-xss-protection:1; mode=block

x-content-type-options:nosniff

expires:Tue, 31 Mar 1981 05:00:00 GMT

x-runtime:0.00647

set-cookie:guest_id=v1%3A140463248796066399; Domain=.twitter.com; Path=/; Expires=Tue, 05-Jul-2016

07:41:27 UTC

set-cookie:_twitter_sess=BAh7CToHaWQiJWViYTU1YTAxMWI5Mzc1NWYxNDRkMGM1ZjgzN2UxZGEyOg9j%250AcmVhdGVkX

2F0bCsIHNieCkcBIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVy%250AOjpGbGFzaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsAOgx

jc3JmX2lkIiViZDkz%250AZDQ1YTVmYzQ0MWI0M2Q1YWQ3MTU4N2ZkMTRl

content-length:105

server:tfe

strict-transport-security:max-age=631138519

What does this tell us?

[robm]

I am having the same problem , can i ask how you generated the consumer key and secret? I created an app that gave me

API key

API secret

so not sure if thats actually a consumer key and secret if it is then we have some confusion

[robm]

this is what happens in my case from tghe SAS log

29   proc http method="post"

30       in=in out=out

31       headerin=hdrin

32       url="https://api.twitter.com/oauth2/token"

33       ct="application/x-www-form-urlencoded;charset=UTF-8";

34    run;

ERROR: java.io.IOException: Server returned HTTP response code: 403 for URL:

https://api.twitter.com/oauth2/token

NOTE: PROCEDURE HTTP used (Total process time):

      real time           1.40 seconds

      cpu time            0.00 seconds

NOTE: The SAS System stopped processing this step because of errors.

[FriedEgg]

Let's simplify the program and only utilize PROC GROOVY:

%let API_KEY=<YOUR API KEY>;

%let API_SECRET=<YOUR API SECRET>;

/* Since I like to use @Grab in groovy, but SAS does not include the Ivy prerequisite, let's get Ivy */

filename ivy "%sysfunc(pathname(work,l))/ivy.jar";

proc http

   method = "get"

   url    = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"

   out    = ivy;

run;

filename cp  temp; *A temporary file for proc groovy output that is currently not necessary;

proc groovy classpath=cp;

add classpath=ivy; *Include the Ivy jar we downloaded;

add sasjar="groovy_2.1.3" version="2.1.3.0_SAS_20130517000930"; *You will need to adjust this line to fit your installation, this is what I have from 9.4 on 64-bit Linux.  We are collecting groovy.json.JsonSlurper here, although we could get it with @Grab instead.;

submit "&API_KEY" "&API_SECRET";

@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')

@Grab(group='commons-codec', module='commons-codec', version='1.2')

import groovy.json.JsonSlurper

import org.apache.commons.codec.binary.Base64

import groovyx.net.http.HTTPBuilder

import static groovyx.net.http.ContentType.*

import static groovyx.net.http.Method.*

def getBearerToken(api_key,api_secret) {

  def auth = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

  def http = new HTTPBuilder("https://api.twitter.com")

  http.request(POST,JSON) { req ->

    uri.path = "/oauth2/token"

    headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

    headers."Authorization" = "Basic "+auth

    body="grant_type=client_credentials"

    response.success = { resp, json ->

       bearer_token = json.access_token

    }

    response.failure = { resp, json ->

       println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

       resp.headers.each { println "${it.name} : ${it.value}" }

       ret = reader.getText()

       println ret

    }

  }

return bearer_token

}

exports.bearer_token = getBearerToken(args[0].toString() , args[1].toString())

endsubmit;

quit;

%put &BEARER_TOKEN;

[FriedEgg · Accepted solution]

Here is an alternate program that does not utilize PROC GROOVY at all to perform the authentication step.

%let API_KEY=<YOUR API KEY>;

%let API_SECRET=<YOUR API SECRET>;

filename head temp;

filename body temp;

filename resp temp;

data _null_;

   file head lrecl=2048;

   auth = put( catx(':' , symget('API_KEY') , symget('API_SECRET')) , $base64x32767.);

   put 'Authorization: Basic ' auth;

run;

data _null_;

   file body;

   put 'grant_type=client_credentials&';

run;

proc http

   url="https://api.twitter.com/oauth2/token?"

   method="post"

   ct="application/x-www-form-urlencoded;charset=UTF-8"

   in=body

   headerin=head

   out=resp;

run;

data _null_;

   infile resp dlm='"';

   input @'access_token' + 3 bearer_token : $1024.;

   call symputx('bearer_token' , bearer_token);

run;

[Scott_Mitchell]

Thanks FriedEgg.  I really appreciate how much work you have put into this.

[FriedEgg]

This was a fairly entertaining endeavor.  As such, I finished up a full PROC GROOVY implementation for the Twitter Search API.  With this we can dump the results directly into a data set.  There is still a lot that could be done to improve this, such as additional get<TYPE> methods, other than just String, and actually implementing some exception handling...  Maybe another time.

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Usage Parameters

*/

    %let api_key      = <YOUR TWITTER API_KEY>;

    %let api_secret   = <YOUR TWITTER API_SECRET>;

    %let search_query = %23SASGF13+OR+%23SASGF14+OR+%23SASGF15;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Collect Ivy Jar

*/

    filename ivy "%sysfunc(pathname(work,l))/ivy.jar";

    proc http

       method = "get"

       url    = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"

       out    = ivy;

    run;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Compile GROOVY/JAVA Classes

*/

    filename cp  temp;

    proc groovy classpath=cp;

    *add the ivy jar just downloaded and include in classpath;

    *we need this to use @Grab notation below for additional dependencies;

    add classpath=ivy;

    *the following line requires modification based on OS and SAS Version, below accurate for 9.4M1 on Linux 64;

    *this was necessary because JsonSlurper was not available initially;

    *alternatively, you could collect this with @Grab, should you groovy-all.jar not contain this module;

    * @Grab(group='org.codehaus.groovy', module='groovy-json', version='2.1.3');

    add sasjar="groovy_2.1.3" version="2.1.3.0_SAS_20130517000930";

    submit parseonly;

    @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')

    @Grab(group='commons-codec', module='commons-codec', version='1.2')

    import groovy.json.JsonSlurper

    import org.apache.commons.codec.binary.Base64

    import groovyx.net.http.HTTPBuilder

    import static groovyx.net.http.ContentType.JSON

    import static groovyx.net.http.Method.GET

    import static groovyx.net.http.Method.POST

    class TwitterApi {

        def api_key

        def api_secret

        def search(query) {

            def bearer_token = getBearerToken()

            def tweets = []

            def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=10")

            api.request(GET,JSON) { req ->

                headers."Authorization" = "Bearer " + bearer_token

                response.failure = { resp, json ->

                    println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

                    resp.headers.each { println "${it.name} : ${it.value}" }

                    ret = reader.getText()

                    println ret

                }

                response.success = { resp, json ->

                    json.statuses.each {

                        tweets << [

                                id: it.id,

                                text         : it.text,

                                truncated    : it.truncated,

                                createdAt    : it.created_at.minus("+0000"),

                                userId       : it.user.id,

                                userName     : it.user.name,

                                userScreenName: it.user.screen_name,

                                location     : it.user.location,

                                description  : it.user.description,

                                url          : it.user.url,

                                userFollowers: it.user.followers_count,

                                userFriends  : it.user.friends_count,

                                userListed   : it.user.listed_count,

                                retweet      : it.retweet_count,

                                favorite     : it.favorite_count

                        ]

                    }

                }

            }

            return tweets

        }

        private getBearerToken(){

            def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

            def api = new HTTPBuilder("https://api.twitter.com")

            def access_token

            api.request(POST,JSON) { req ->

                uri.path = "/oauth2/token"

                headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

                headers."Authorization" = "Basic "+encoded_basic

                body = "grant_type=client_credentials"

                response.success = { resp, json ->

                    access_token = json.access_token

                }

                response.failure = { resp, json ->

                    println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

                    resp.headers.each { println "${it.name} : ${it.value}" }

                    ret = reader.getText()

                    println ret

                }

            }

            return access_token

        }

    }

    endsubmit;

    submit parseonly;

    import java.util.ArrayList;

    import java.util.Iterator;

    import java.util.LinkedHashMap;

    public class TwitterSAS {

        public String api_key = "";

        public String api_secret = "";

        public String search_query = "";

        public void main() {

            TwitterApi api = new TwitterApi();

            api.setApi_key(api_key);

            api.setApi_secret(api_secret);

            tweets = api.search(search_query);

            iter = tweets.iterator();

        }

        public boolean hasNext() {

            return iter.hasNext();

        }

        public void getNext() {

            tweet = ((LinkedHashMap) (iter.next()));

        }

        public String getString(String key) {

            return tweet.get(key).toString();

        }

        protected ArrayList tweets;

        protected Iterator iter;

        protected LinkedHashMap tweet;

    }

    endsubmit;

    quit;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Add our PROC GROOVY output to CLASSPATH

*/

    options set=classpath "%sysfunc(pathname(cp,f))";

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Collect Twitter Search Results into SAS DATA SET

*/

    data twitter;

       length id text truncated createdAt userId userName userScreenName location description url userFollowers userFriends userListed retweet favorite $ 140;

       dcl javaobj tweets("TwitterSAS");

       tweets.setStringField("api_key"      , "&api_key."      );

       tweets.setStringField("api_secret"   , "&api_secret."   );

       tweets.setStringField("search_query" , "&search_query." );

       tweets.callVoidMethod("main");

       tweets.callBooleanMethod("hasNext",rc);

       do _n_=1 to 10 while(rc);

          tweets.callVoidMethod("getNext");

          tweets.callStringMethod("getString","id",id);

          tweets.callStringMethod("getString","text",text);

          tweets.callStringMethod("getString","truncated",truncated);

          tweets.callStringMethod("getString","createdAt",createdAt);

          tweets.callStringMethod("getString","userId",userId);

          tweets.callStringMethod("getString","userName",userName);

          tweets.callStringMethod("getString","userScreenName",userScreenName);

          tweets.callStringMethod("getString","location",location);

          tweets.callStringMethod("getString","description",description);

          tweets.callStringMethod("getString","url",url);

          tweets.callStringMethod("getString","userFollowers",userFollowers);

          tweets.callStringMethod("getString","userFriends",userFriends);

          tweets.callStringMethod("getString","userListed",userListed);

          tweets.callStringMethod("getString","retweet",retweet);

          tweets.callStringMethod("getString","favorite",favorite);

          output;

          tweets.callBooleanMethod("hasNext",rc);

       end;

    run;

[saidi]

Hi,

I am getting following error, can you help me on this.

Thanks in Advance.

Error message:

2184      data twitter;

2185         length id text truncated createdAt userId userName userScreenName location description

2185! url userFollowers userFriends userListed retweet favorite $ 140;

2186

2187

2188         dcl javaobj tweets("TwitterSAS");

2189

2190

2191 tweets.setStringField("api_key"      , "&api_key."      );

2192 tweets.setStringField("api_secret"   , "&api_secret."   );

2193 tweets.setStringField("search_query" , "&search_query." );

2194

2195

2196 tweets.callVoidMethod("main");

2197

2198

2199 tweets.callBooleanMethod("hasNext",rc);

2200         do _n_=1 to 100 while(rc);

2201 tweets.callVoidMethod("getNext");

2202 tweets.callStringMethod("getString","id",id);

2203 tweets.callStringMethod("getString","text",text);

2204 tweets.callStringMethod("getString","truncated",truncated);

2205 tweets.callStringMethod("getString","createdAt",createdAt);

2206 tweets.callStringMethod("getString","userId",userId);

2207            tweets.callStringMethod("getString","userName",userName);

2208 tweets.callStringMethod("getString","userScreenName",userScreenName);

2209 tweets.callStringMethod("getString","location",location);

2210            tweets.callStringMethod("getString","description",description);

2211 tweets.callStringMethod("getString","url",url);

2212 tweets.callStringMethod("getString","userFollowers",userFollowers);

2213 tweets.callStringMethod("getString","userFriends",userFriends);

2214 tweets.callStringMethod("getString","userListed",userListed);

2215 tweets.callStringMethod("getString","retweet",retweet);

2216 tweets.callStringMethod("getString","favorite",favorite);

2217            output;

2218 tweets.callBooleanMethod("hasNext",rc);

2219         end;

2220      run;

ERROR: Transcoding failure at line 2203 column 11.

ERROR: DATA STEP Component Object failure. Aborted during the EXECUTION phase.

NOTE: The SAS System stopped processing this step because of errors.

WARNING: The data set WORK.TWITTER may be incomplete. When this step was stopped there were 0

         observations and 16 variables.

NOTE: DATA statement used (Total process time):

      real time           1.31 seconds

      cpu time            0.01 seconds

/*Here is my java picklist*/

2221  proc javainfo picklist 'base/groovy.txt'; run;

Picklist URLs:

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/activation_1.1.1.0_SAS_201

21211183202/activation.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/groovy_1.7.1.0_SAS_2012121

1183404/groovy-all.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jansi_1.2.0.0_SAS_20121211

183401/jansi.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jaxb_ri_2.1.12.0_SAS_20121

211183328/jaxb-api.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jaxb_ri_2.1.12.0_SAS_20121

211183328/jaxb-impl.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jaxb_ri_2.1.12.0_SAS_20121

211183328/jaxb1-impl.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/jaxb_ri_2.1.12.0_SAS_20121

211183328/jaxb-xjc.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/stax_api_1.0.1.a_SAS_20121

211183307/jsr173_api.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/stax_parser_4.0.8.0_SAS_20

121211183332/stax2-api.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/stax_parser_4.0.8.0_SAS_20

121211183332/wstx-asl.jar

file:/C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/xstream_1.3.1.0_SAS_201212

11183400/xstream.jar

Total URLs: 11

/*SAS code:*/

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Usage Parameters

*/

    %let api_key      = <API_KEY>;

    %let api_secret   = <API_SECRET>;

/*    %let search_query = %23SASGF13+OR+%23SASGF14+OR+%23SASGF15;*/

/*    %let search_query = %23SASJOBS;*/

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Collect Ivy Jar

*/

/*    filename ivy "%sysfunc(pathname(work,l))/ivy.jar";*/

/* */

/* */

/*    proc http*/

/*       method = "get"*/

/*/*       url = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"*/*/

/* url    = "C:/Program%20Files/SASHome2/SASVersionedJarRepository/eclipse/plugins/groovy_1.7.1.0_SAS_20121211183404/groovy-all.jar"*/

/*       out = ivy;*/

/*    run;*/

;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Compile GROOVY/JAVA Classes

*/

    filename cp  temp;

    proc groovy classpath=cp;

    *add the ivy jar just downloaded and include in classpath;

    *we need this to use @Grab notation below for additional dependencies;

/*    add classpath=ivy;*/

      add classpath="C:\groovy-2.4.0-rc-2\embeddable\groovy-all-2.4.0-rc-2.jar";

    *the following line requires modification based on OS and SAS Version, below accurate for 9.4M1 on Linux 64;

    *this was necessary because JsonSlurper was not available initially;

    *alternatively, you could collect this with @Grab, should you groovy-all.jar not contain this module;

    * @Grab(group='org.codehaus.groovy', module='groovy-json', version='2.1.3');

/*    add sasjar="groovy_2.1.3" version="2.1.3.0_SAS_20130517000930";*/

/*      add sasjar="groovy_2.4.0" version="2.4.0.0_SAS_20121211183404";*/

    submit parseonly;

    @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')

    @Grab(group='commons-codec', module='commons-codec', version='1.2')

    import groovy.json.JsonSlurper

    import org.apache.commons.codec.binary.Base64

    import groovyx.net.http.HTTPBuilder

    import static groovyx.net.http.ContentType.JSON

    import static groovyx.net.http.Method.GET

    import static groovyx.net.http.Method.POST

    class TwitterApi {

        def api_key

        def api_secret

        def search(query) {

            def bearer_token = getBearerToken()

            def tweets = []

            def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=100")

            api.request(GET,JSON) { req ->

                headers."Authorization" = "Bearer " + bearer_token

                response.failure = { resp, json ->

                    println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

                    resp.headers.each { println "${it.name} : ${it.value}" }

                    ret = reader.getText()

                    println ret

                }

                response.success = { resp, json ->

                    json.statuses.each {

                        tweets << [

                                id: it.id,

                                text         : it.text,

                                truncated    : it.truncated,

                                createdAt    : it.created_at.minus("+0000"),

                                userId       : it.user.id,

                                userName     : it.user.name,

                                userScreenName: it.user.screen_name,

                                location     : it.user.location,

description  : it.user.description,

                                url          : it.user.url,

                                userFollowers: it.user.followers_count,

userFriends  : it.user.friends_count,

                                userListed   : it.user.listed_count,

                                retweet      : it.retweet_count,

                                favorite     : it.favorite_count

                        ]

                    }

                }

            }

            return tweets

        }

        private getBearerToken(){

            def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

            def api = new HTTPBuilder("https://api.twitter.com")

            def access_token

            api.request(POST,JSON) { req ->

                uri.path = "/oauth2/token"

                headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

                headers."Authorization" = "Basic "+encoded_basic

                body = "grant_type=client_credentials"

                response.success = { resp, json ->

                    access_token = json.access_token

                }

                response.failure = { resp, json ->

                    println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

                    resp.headers.each { println "${it.name} : ${it.value}" }

                    ret = reader.getText()

                    println ret

                }

            }

            return access_token

        }

    }

    endsubmit;

    submit parseonly;

    import java.util.ArrayList;

    import java.util.Iterator;

    import java.util.LinkedHashMap;

    public class TwitterSAS {

        public String api_key = "";

        public String api_secret = "";

        public String search_query = "";

        public void main() {

            TwitterApi api = new TwitterApi();

            api.setApi_key(api_key);

            api.setApi_secret(api_secret);

            tweets = api.search(search_query);

            iter = tweets.iterator();

        }

        public boolean hasNext() {

            return iter.hasNext();

        }

        public void getNext() {

            tweet = ((LinkedHashMap) (iter.next()));

        }

        public String getString(String key) {

            return tweet.get(key).toString();

        }

        protected ArrayList tweets;

        protected Iterator iter;

        protected LinkedHashMap tweet;

    }

    endsubmit;

    quit;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Add our PROC GROOVY output to CLASSPATH

*/

    options set=classpath "%sysfunc(pathname(cp,f))";

     %let search_query = %23SAS;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Collect Twitter Search Results into SAS DATA SET

*/

    data twitter;

       length id text truncated createdAt userId userName userScreenName location description url userFollowers userFriends userListed retweet favorite $ 140;

       dcl javaobj tweets("TwitterSAS");

       tweets.setStringField("api_key"      , "&api_key."      );

       tweets.setStringField("api_secret"   , "&api_secret."   );

       tweets.setStringField("search_query" , "&search_query." );

       tweets.callVoidMethod("main");

       tweets.callBooleanMethod("hasNext",rc);

       do _n_=1 to 100 while(rc);

          tweets.callVoidMethod("getNext");

          tweets.callStringMethod("getString","id",id);

          tweets.callStringMethod("getString","text",text);

          tweets.callStringMethod("getString","truncated",truncated);

          tweets.callStringMethod("getString","createdAt",createdAt);

          tweets.callStringMethod("getString","userId",userId);

          tweets.callStringMethod("getString","userName",userName);

          tweets.callStringMethod("getString","userScreenName",userScreenName);

          tweets.callStringMethod("getString","location",location);

          tweets.callStringMethod("getString","description",description);

          tweets.callStringMethod("getString","url",url);

          tweets.callStringMethod("getString","userFollowers",userFollowers);

          tweets.callStringMethod("getString","userFriends",userFriends);

          tweets.callStringMethod("getString","userListed",userListed);

          tweets.callStringMethod("getString","retweet",retweet);

          tweets.callStringMethod("getString","favorite",favorite);

          output;

          tweets.callBooleanMethod("hasNext",rc);

       end;

    run;

[FriedEgg]

,

This issue is most likely a result of your SAS session encoding.

Run the following to check your SAS options:

proc options group=languagecontrol; run;

[saidi]

Below is log summary.

Please advise what options need to be changed.

2224  proc options group=languagecontrol; run;

    SAS (r) Proprietary Software Release 9.4  TS1M2

Group=LANGUAGECONTROL

DATESTYLE=MDY     Specifies the sequence of month, day, and year when ANYDTDTE, ANYDTDTM, or

                   ANYDTTME informat data is ambiguous.

DFLANG=ENGLISH    Specifies the language for international date informats and formats.

DS2ACCEL=NONE     Provides support for DS2 code pass-through acceleration.

DSACCEL=NONE      Provides support for code pass-through acceleration.

EXTENDOBSCOUNTER=YES

                   Specifies whether to extend the maximum number of observations in a new SAS data

                   file.

LOCALEDATA=SASLOCALE

                   Specifies the location of the locale database.

NOLOGLANGCHG      Disables changing the language of the SAS output when the LOCALE= option is

                   changed.

NOLOGLANGENG      Write SAS log messages based on the values of the LOGLANGCHG, LSWLANG=, and

                   LOCALE= options when SAS started.

LSWLANG=LOCALE    Specifies the language for SAS log and ODS messages when the LOCALE= option is set

                   after SAS starts.

MAPEBCDICTOASCII= Specifies the transcoding table that is used to convert characters from ASCII to

                   EBCDIC and EBCDIC to ASCII.

NONLDECSEPARATOR  Disables formatting of numeric output using the decimal separator for the locale.

NOODSLANGCHG      Disables changing the language of the SAS message text in ODS output when the

                   LOCALE option is set after start up.

PAPERSIZE=LETTER  Specifies the paper size to use for printing.

RSASIOTRANSERROR  Displays a transcoding error when illegal values are read from a remote

                   application.

TIMEZONE=         Specifies a time zone.

TRANTAB=(lat1lat1,lat1lat1,wlt1_ucs,wlt1_lcs,wlt1_ccl,,,)

                   Specifies the translation table catalog entries.

URLENCODING=SESSION

                   Specifies whether the argument to the URLENCODE function and to the URLDECODE

                   function is interpreted using the SAS session encoding or UTF-8 encoding.

NODBCS            Disables double-byte character sets.

DBCSLANG=NONE     Specifies a double-byte character set language.

DBCSTYPE=NONE     Specifies the encoding method to use for a double-byte character set.

ENCODING=WLATIN1  Specifies the default character-set encoding for the SAS session.

LOCALE=EN_US      Specifies a set of attributes in a SAS session that reflect the language, local

                   conventions, and culture for a geographical region.

NONLSCOMPATMODE   Encodes data using the SAS session encoding.

NOTE: PROCEDURE OPTIONS used (Total process time):

      real time           0.04 seconds

      cpu time            0.00 seconds

[FriedEgg]

The options you need to change are only effective on SAS invocation, so one more this to check is: how are you interacting with SAS?  Are you using EG or another of the GUI tools?  If so, are you using a local install or connecting to a remote server.  In the case of the remote server, you are out of luck (unless you can alter these options for your entire site).  If you are using SAS directly, then you can just choose to open it with a UTF-8 session through either a start menu link in Windows or the sas_u8 script in /SASHome/SASFoundation/9.X/bin in Linux.  Specifically the options that are most likely causing your problems are NODBCS and ENCODING.  If you are not able to modify these options, there are other things we can do as well.  The most simple of which would be to change your query:

%let search_query=%23SAS&lang=en;

This will ask twitter to filter tweets, as best it can, to those identified as being written in the English language.  This will not prevent problems like this completely, however.  The next step would be to modify the groovy code to translate these characters or to create a trantab table in SAS to add the necessary compatibility.