topic Re: Proc Groovy\HTTP Headaches in SAS Procedures

Proc Groovy\HTTP Headaches

Scott_Mitchell — Sun, 29 Jun 2014 07:37:21 GMT

Hi Guys,

I am in the process of investigating how I can make best use of Proc Groovy. I found the below code in one of the SAS blogs, but I just can't seem to get it working. Basically anything in uppercase are modifications that I have made to the existing code to ensure that it works in my environment.

The following is the error I get, which may be due to the fact that I have attempted to obtain a bearer key too frequently as per https://api.twitter.com/oauth2/token. So I am looking for a way to invalidate the current one.

Any help would be greatly appreciated.

ERROR: java.io.IOException: Server returned HTTP response code: 403 for URL:

%LET CONSUMER_KEY=&TWITKEY.;

%LET CONSUME_SECRET=&TWITSEC.;

%LET JSON_TWEET_FILE=E:\RESPONSECONTENT1.TXT;

%LET CSV_TWEET_FILE=E:\RESPONSECONTENT2.TXT;

%LET TWEET_QUERY=%23SASGF13+OR+%23SASUSERS+OR+%23SASGF14;

DATA _NULL_;

NJSON_TWEET_FILE=TRANWRD("&JSON_TWEET_FILE.",'\','/');

NCSV_TWEET_FILE=TRANWRD("&CSV_TWEET_FILE.",'\','/');

CALL SYMPUT('NJSON_TWEET_FILE',TRIM(NJSON_TWEET_FILE));

CALL SYMPUT('NCSV_TWEET_FILE',TRIM(NCSV_TWEET_FILE));

RUN;

/* Create a temporary file name used for the XMLMap */

%macro tempFile( fname );

%if %superq(SYSSCPL) eq %str(z/OS) or %superq(SYSSCPL) eq %str(OS/390) %then

%let recfm=recfm=vb;

%else

%let recfm=;

filename &fname TEMP lrecl=2048 &recfm;

%mend;

/* create temp files for the content and header input streams */

%tempFile(in);

%tempFile(hdrin);

/* keep the response permanently */

filename out "&JSON_TWEET_FILE.";

/* post request content is the grant_type */

data _null_;

file in;

put "grant_type=client_credentials&";

run;

/* request the bearer token by providing consumer key and secret */

data _null_;

file hdrin;

consumerKey = urlencode("&CONSUMER_KEY.");

consumerSecret = urlencode("&CONSUME_SECRET.");

encodedAccessToken = put( compress(consumerKey || ":" || consumerSecret),$base64x32767.);

put "Authorization: Basic " encodedAccessToken;

run;

proc http method="post"

in=in out=out

headerin=hdrin

url="https://api.twitter.com/oauth2/token"

ct="application/x-www-form-urlencoded;charset=UTF-8";

run;

/* Store bearer token in macro variable 'BEARER_TOKEN' */

proc groovy CLASSPATH="D:\SASHOME\SASFOUNDATION\9.3\GROOVY\EMBEDDABLE\GROOVY-ALL-2.3.3";

submit "&JSON_TWEET_FILE.";

import groovy.json.JsonSlurper

def input = new File(args[0]).text

def result = new JsonSlurper().parseText(input)

println "Recieved bearer token: ${result.access_token}"

exports.putAt('BEARER_TOKEN',result.access_token)

endsubmit;

quit;

Re: Proc Groovy\HTTP Headaches

FriedEgg — Tue, 01 Jul 2014 16:44:31 GMT

According to the Twitter API, the oAuth2 method will return a 403 if you attempt it too frequently.

https://dev.twitter.com/docs/api/1.1/post/oauth2/token

Re: Proc Groovy\HTTP Headaches

Scott_Mitchell — Wed, 02 Jul 2014 02:03:07 GMT

Thanks FriedEgg. I was able to locate that, but can't find a way of rectifying the situation.

I have tried the following without success.

proc http method="post"

url="https://api.twitter.com/oauth2/invalidate_token";

run;

Do you have any ideas?

Re: Proc Groovy\HTTP Headaches

FriedEgg — Wed, 02 Jul 2014 02:28:22 GMT

First I would actually validate what the error code is that you are receiving from your post:

Before the PROC GROOVY and after the PROC HTTP add the following to your SAS session, or simply open the file through any other means you have in which to view it...

data _null_;

infile out;

input @;

put _infile_;

run;

You may also want to modify the PROC HTTP call to retireve the outbound header from the server, to help debug, with the option HEADEROUT=<fileref>, then do the same as above to view the contents of the file.

Re: Proc Groovy\HTTP Headaches

Scott_Mitchell — Sun, 06 Jul 2014 07:47:04 GMT

Hi FriedEgg,

There is nothing in the out file and the following in the HeadOut

HTTP/1.1 403 Forbidden

content-type:application/json; charset=utf-8

x-frame-options:DENY

x-mid:9d8aa088047514ddd72fa2ca092775963d2ed955

last-modified:Sun, 06 Jul 2014 07:41:27 GMT

status:403 Forbidden

date:Sun, 06 Jul 2014 07:41:27 GMT

x-ua-compatible:IE=edge,chrome=1

x-transaction:d9024281258f1139

vary:Accept-Encoding

pragma:no-cache

cache-control:no-cache, no-store, must-revalidate, pre-check=0, post-check=0

x-xss-protection:1; mode=block

x-content-type-options:nosniff

expires:Tue, 31 Mar 1981 05:00:00 GMT

x-runtime:0.00647

set-cookie:guest_id=v1%3A140463248796066399; Domain=.twitter.com; Path=/; Expires=Tue, 05-Jul-2016

07:41:27 UTC

set-cookie:_twitter_sess=BAh7CToHaWQiJWViYTU1YTAxMWI5Mzc1NWYxNDRkMGM1ZjgzN2UxZGEyOg9j%250AcmVhdGVkX

2F0bCsIHNieCkcBIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVy%250AOjpGbGFzaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsAOgx

jc3JmX2lkIiViZDkz%250AZDQ1YTVmYzQ0MWI0M2Q1YWQ3MTU4N2ZkMTRl

content-length:105

server:tfe

strict-transport-security:max-age=631138519

What does this tell us?

Re: Proc Groovy\HTTP Headaches

robm — Mon, 07 Jul 2014 14:19:38 GMT

I am having the same problem , can i ask how you generated the consumer key and secret? I created an app that gave me

API key

API secret

so not sure if thats actually a consumer key and secret if it is then we have some confusion

Re: Proc Groovy\HTTP Headaches

robm — Mon, 07 Jul 2014 14:20:53 GMT

this is what happens in my case from tghe SAS log

29 proc http method="post"

30 in=in out=out

31 headerin=hdrin

32 url="https://api.twitter.com/oauth2/token"

33 ct="application/x-www-form-urlencoded;charset=UTF-8";

34 run;

ERROR: java.io.IOException: Server returned HTTP response code: 403 for URL:

https://api.twitter.com/oauth2/token

NOTE: PROCEDURE HTTP used (Total process time):

real time 1.40 seconds

cpu time 0.00 seconds

NOTE: The SAS System stopped processing this step because of errors.

Re: Proc Groovy\HTTP Headaches

FriedEgg — Mon, 07 Jul 2014 23:19:57 GMT

Let's simplify the program and only utilize PROC GROOVY:

%let API_KEY=<YOUR API KEY>;

%let API_SECRET=<YOUR API SECRET>;

/* Since I like to use @Grab in groovy, but SAS does not include the Ivy prerequisite, let's get Ivy */

filename ivy "%sysfunc(pathname(work,l))/ivy.jar";

proc http

method = "get"

url = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"

out = ivy;

run;

filename cp temp; *A temporary file for proc groovy output that is currently not necessary;

proc groovy classpath=cp;

add classpath=ivy; *Include the Ivy jar we downloaded;

add sasjar="groovy_2.1.3" version="2.1.3.0_SAS_20130517000930"; *You will need to adjust this line to fit your installation, this is what I have from 9.4 on 64-bit Linux. We are collecting groovy.json.JsonSlurper here, although we could get it with @Grab instead.;

submit "&API_KEY" "&API_SECRET";

@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')

@Grab(group='commons-codec', module='commons-codec', version='1.2')

import groovy.json.JsonSlurper

import org.apache.commons.codec.binary.Base64

import groovyx.net.http.HTTPBuilder

import static groovyx.net.http.ContentType.*

import static groovyx.net.http.Method.*

def getBearerToken(api_key,api_secret) {

def auth = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

def http = new HTTPBuilder("https://api.twitter.com")

http.request(POST,JSON) { req ->

uri.path = "/oauth2/token"

headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

headers."Authorization" = "Basic "+auth

body="grant_type=client_credentials"

response.success = { resp, json ->

bearer_token = json.access_token

}

response.failure = { resp, json ->

println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

resp.headers.each { println "${it.name} : ${it.value}" }

ret = reader.getText()

println ret

}

return bearer_token

}

exports.bearer_token = getBearerToken(args[0].toString() , args[1].toString())

endsubmit;

quit;

%put &BEARER_TOKEN;

Re: Proc Groovy\HTTP Headaches

FriedEgg — Tue, 08 Jul 2014 01:22:09 GMT

Here is an alternate program that does not utilize PROC GROOVY at all to perform the authentication step.

%let API_KEY=<YOUR API KEY>;

%let API_SECRET=<YOUR API SECRET>;

filename head temp;

filename body temp;

filename resp temp;

data _null_;

file head lrecl=2048;

auth = put( catx(':' , symget('API_KEY') , symget('API_SECRET')) , $base64x32767.);

put 'Authorization: Basic ' auth;

run;

data _null_;

file body;

put 'grant_type=client_credentials&';

run;

proc http

url="https://api.twitter.com/oauth2/token?"

method="post"

ct="application/x-www-form-urlencoded;charset=UTF-8"

in=body

headerin=head

out=resp;

run;

data _null_;

infile resp dlm='"';

input @'access_token' + 3 bearer_token : $1024.;

call symputx('bearer_token' , bearer_token);

run;

Re: Proc Groovy\HTTP Headaches

Scott_Mitchell — Thu, 10 Jul 2014 05:26:35 GMT

Thanks FriedEgg. I really appreciate how much work you have put into this.

Re: Proc Groovy\HTTP Headaches

FriedEgg — Fri, 11 Jul 2014 17:31:15 GMT

This was a fairly entertaining endeavor. As such, I finished up a full PROC GROOVY implementation for the Twitter Search API. With this we can dump the results directly into a data set. There is still a lot that could be done to improve this, such as additional get<TYPE> methods, other than just String, and actually implementing some exception handling... Maybe another time.

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Usage Parameters

%let api_key = <YOUR TWITTER API_KEY>;

%let api_secret = <YOUR TWITTER API_SECRET>;

%let search_query = %23SASGF13+OR+%23SASGF14+OR+%23SASGF15;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Collect Ivy Jar

filename ivy "%sysfunc(pathname(work,l))/ivy.jar";

proc http

method = "get"

url = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"

out = ivy;

run;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Compile GROOVY/JAVA Classes

filename cp temp;

proc groovy classpath=cp;

*add the ivy jar just downloaded and include in classpath;

*we need this to use @Grab notation below for additional dependencies;

add classpath=ivy;

*the following line requires modification based on OS and SAS Version, below accurate for 9.4M1 on Linux 64;

*this was necessary because JsonSlurper was not available initially;

*alternatively, you could collect this with @Grab, should you groovy-all.jar not contain this module;

* @Grab(group='org.codehaus.groovy', module='groovy-json', version='2.1.3');

add sasjar="groovy_2.1.3" version="2.1.3.0_SAS_20130517000930";

submit parseonly;

@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')

@Grab(group='commons-codec', module='commons-codec', version='1.2')

import groovy.json.JsonSlurper

import org.apache.commons.codec.binary.Base64

import groovyx.net.http.HTTPBuilder

import static groovyx.net.http.ContentType.JSON

import static groovyx.net.http.Method.GET

import static groovyx.net.http.Method.POST

class TwitterApi {

def api_key

def api_secret

def search(query) {

def bearer_token = getBearerToken()

def tweets = []

def api = new HTTPBuilder("https://api.twitter.com/1.1/search/tweets.json?q=${query}&count=10")

api.request(GET,JSON) { req ->

headers."Authorization" = "Bearer " + bearer_token

response.failure = { resp, json ->

println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

resp.headers.each { println "${it.name} : ${it.value}" }

ret = reader.getText()

println ret

}

response.success = { resp, json ->

json.statuses.each {

tweets << [

id: it.id,

text : it.text,

truncated : it.truncated,

createdAt : it.created_at.minus("+0000"),

userId : it.user.id,

userName : it.user.name,

userScreenName: it.user.screen_name,

location : it.user.location,

description : it.user.description,

url : it.user.url,

userFollowers: it.user.followers_count,

userFriends : it.user.friends_count,

userListed : it.user.listed_count,

retweet : it.retweet_count,

favorite : it.favorite_count

]

}

return tweets

}

private getBearerToken(){

def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

def api = new HTTPBuilder("https://api.twitter.com")

def access_token

api.request(POST,JSON) { req ->

uri.path = "/oauth2/token"

headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

headers."Authorization" = "Basic "+encoded_basic

body = "grant_type=client_credentials"

response.success = { resp, json ->

access_token = json.access_token

}

response.failure = { resp, json ->

println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

resp.headers.each { println "${it.name} : ${it.value}" }

ret = reader.getText()

println ret

}

return access_token

}

endsubmit;

submit parseonly;

import java.util.ArrayList;

import java.util.Iterator;

import java.util.LinkedHashMap;

public class TwitterSAS {

public String api_key = "";

public String api_secret = "";

public String search_query = "";

public void main() {

TwitterApi api = new TwitterApi();

api.setApi_key(api_key);

api.setApi_secret(api_secret);

tweets = api.search(search_query);

iter = tweets.iterator();

}

public boolean hasNext() {

return iter.hasNext();

}

public void getNext() {

tweet = ((LinkedHashMap) (iter.next()));

}

public String getString(String key) {

return tweet.get(key).toString();

}

protected ArrayList tweets;

protected Iterator iter;

protected LinkedHashMap tweet;

}

endsubmit;

quit;

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Add our PROC GROOVY output to CLASSPATH

options set=classpath "%sysfunc(pathname(cp,f))";

/*-----------------------------------------------------------------------------------------------------------------------------------------

*-Collect Twitter Search Results into SAS DATA SET

data twitter;

length id text truncated createdAt userId userName userScreenName location description url userFollowers userFriends userListed retweet favorite $ 140;

dcl javaobj tweets("TwitterSAS");

tweets.setStringField("api_key" , "&api_key." );

tweets.setStringField("api_secret" , "&api_secret." );

tweets.setStringField("search_query" , "&search_query." );

tweets.callVoidMethod("main");

tweets.callBooleanMethod("hasNext",rc);

do _n_=1 to 10 while(rc);

tweets.callVoidMethod("getNext");

tweets.callStringMethod("getString","id",id);

tweets.callStringMethod("getString","text",text);

tweets.callStringMethod("getString","truncated",truncated);

tweets.callStringMethod("getString","createdAt",createdAt);

tweets.callStringMethod("getString","userId",userId);

tweets.callStringMethod("getString","userName",userName);

tweets.callStringMethod("getString","userScreenName",userScreenName);

tweets.callStringMethod("getString","location",location);

tweets.callStringMethod("getString","description",description);

tweets.callStringMethod("getString","url",url);

tweets.callStringMethod("getString","userFollowers",userFollowers);

tweets.callStringMethod("getString","userFriends",userFriends);

tweets.callStringMethod("getString","userListed",userListed);

tweets.callStringMethod("getString","retweet",retweet);

tweets.callStringMethod("getString","favorite",favorite);

output;

tweets.callBooleanMethod("hasNext",rc);

end;

run;

Re: Proc Groovy\HTTP Headaches

NandanM — Sat, 12 Jul 2014 18:19:14 GMT

Hey Guys,

I have my API Key and API secret. But I need to know a stepwise way to write the code to fetch tweets from twitter. I understand that we need to initialize the key and secret but if anyone could articulate one piece of code that would get the job done,I would appreciate it. I am using a Macintosh by the way.

Thanks

Re: Proc Groovy\HTTP Headaches

FriedEgg — Sun, 13 Jul 2014 02:22:18 GMT

NandanM,

The last piece of code which I posted here will collect tweets into a data set. The comments provide a very high level overview of the few required steps. Since you state you are using a Mac, I assume you are running SAS inside a Virtual Machine of some kind, since the last version of SAS I am aware of to run natively was 6.2 or so for PowerPC... If you are having a specific issue attempting to utilize this code I will do what I can to aide you through it.

Regards,

FriedEgg

Re: Proc Groovy\HTTP Headaches

NandanM — Sun, 13 Jul 2014 03:33:03 GMT

Thanks for your response fried egg. You are correct when you say that I'm using a virtual machine. The specific part of the code that's giving me an issue is - Compile GROOVY/JAVA Classes. When I run this part of the code, I get the following errors: ERROR: Failed to add sasjar groovy_2.3.4 (I downloaded this version so I changed it from 2.3.1 in your code to 2.3.4 in mine). And I get this error twice: The SUBMIT command failed. Can you help me fix this?

Thanks

Re: Proc Groovy\HTTP Headaches

FriedEgg — Sun, 13 Jul 2014 04:20:26 GMT

Okay,

That is a area I would expect a person not familiar with PROC GROOVY to normally encounter their first issue here. The statement I am using is:

add sasjar="<build module"> version="<build version>";

This is used to reference a jar file included in you SAS installation, not something you would manually download, as you state you have done. To find what bundle and version you have included in your installation, which may already be sufficient run the following:

proc javainfo picklist 'base/groovy.txt'; run;

If the version is 1.7 then you are going to need an updated version. In some cases, SAS has already included an additional version in the SASVersionedJarRepository, which you can search through and find the appropriate version and bundle info in the META_INF.

Since you have already downloaded the 2.3.4 jar file we can just utilize that instead, but you need to use:

add classpath="/path/to/groovy-all.2.3.4.jar";

instead of add sasjar ...;

This should get you past this first hurdle.

Re: Proc Groovy\HTTP Headaches

NandanM — Sun, 13 Jul 2014 04:51:40 GMT

Thanks for your help. It did help me get past the first hurdle but I still can't run the submit commands successfully.

Re: Proc Groovy\HTTP Headaches

FriedEgg — Sun, 13 Jul 2014 05:49:46 GMT

I can't help you further if you give me nothing to work with... What is the next issue you are having?

Re: Proc Groovy\HTTP Headaches

robm — Sun, 13 Jul 2014 18:53:14 GMT

hmmm I get this error now that I am not referecing the jar directly

17 add sasjar="groovy-all-2.2.1";

java.lang.IllegalArgumentException: Failed to find sasjar groovy-all-2.2.1 0.0.0.

CLASSPATH=C:\Program Files\Java\jre7\classes

C:\Program Files\Java\jre7\classes>dir
Volume in drive C has no label.
Volume Serial Number is 889C-285D

Directory of C:\Program Files\Java\jre7\classes

07/13/2014 08:51 AM    <DIR>          .
07/13/2014 08:51 AM    <DIR>          ..
07/13/2014 08:39 AM         6,558,498 groovy-all-2.2.1.jar
               1 File(s)      6,558,498 bytes
               2 Dir(s) 23,812,456,448 bytes free

C:\Program Files\Java\jre7\classes>

Re: Proc Groovy\HTTP Headaches

FriedEgg — Mon, 14 Jul 2014 00:54:14 GMT

@robm,

You should refer to my previous response to NandanM, about ADD SASJAR vs. ADD CLASSPATH for a more in-depth answer.

Replace your add sasjar statement with add classpath="C:\Program Files\Java\jre7\classes\groovy-all-2.2.1.jar"; or whatever the correct full path is.

Regards,

FriedEgg

Re: Proc Groovy\HTTP Headaches

robm — Mon, 14 Jul 2014 01:46:16 GMT

ok thanks that worked , however I keep getting this 403 error ....

310 endsubmit;

Unexpected error: 403 : Forbidden

cache-control : no-cache, no-store, must-revalidate, pre-check=0, post-check=0

content-length : 105

content-type : application/json; charset=utf-8

date : Mon, 14 Jul 2014 01:40:13 GMT

expires : Tue, 31 Mar 1981 05:00:00 GMT

last-modified : Mon, 14 Jul 2014 01:40:13 GMT

pragma : no-cache

server : tfe

set-cookie :

_twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZ

WR7ADoMY3NyZl9pZCIlZDNkMDJhMjc0YjAzNGIzOWJm%250AODI0MzY2YWZiNzQ0MjE6B2lkIiUxMjFlNThiNTZhZWNmOTdj

OTU0ZTFkY2My%250ANmY4Yzk0NjoPY3JlYXRlZF9hdGwrCAH9hjJHAQ%253D%253D--863dd82fd57c3cef15985bbb3aa3a

ee5bdda6b7e; domain=.twitter.com; path=/; secure; HttpOnly

set-cookie : guest_id=v1%3A140530201317976673; Domain=.twitter.com; Path=/; Expires=Wed,

13-Jul-2016 01:40:13 UTC

status : 403 Forbidden