Proc Groovy\HTTP Headaches

[saidi]

Hi,

I am using PC SAS on server, I have the control to modify options on server, can I please know what option do I need to change.

I tried the following query which didn’t work.

%let search_query=%23SAS&lang=en;

Thank you for your time.

[FriedEgg]

You want to initialize the SAS session on the server with the options "-dbcs -encoding UTF-8."  Or you can use the shortcut that should have been created as a part of the installation process.  Or you can append the config file for UTF-8 support that should have been created as part of your installation to your session.  This file should be located at something like: "C:\Program File\SASHome\SASFoundation\9.4\nls\u8\sasv9.cfg"

[NandanM]

Hey Guys,

I have my API Key and API secret. But I need to know a stepwise way to write the code to fetch tweets from twitter. I understand that we need to initialize the key and secret but if anyone could articulate one piece of code that would get the job done,I would appreciate it. I am using a Macintosh by the way.

Thanks

[FriedEgg]

NandanM,

The last piece of code which I posted here will collect tweets into a data set.  The comments provide a very high level overview of the few required steps.  Since you state you are using a Mac, I assume you are running SAS inside a Virtual Machine of some kind, since the last version of SAS I am aware of to run natively was 6.2 or so for PowerPC...  If you are having a specific issue attempting to utilize this code I will do what I can to aide you through it.

Regards,

FriedEgg

[NandanM]

Thanks for your response fried egg. You are correct  when you say that I'm using a virtual machine. The specific part of the code that's giving me an issue is - Compile GROOVY/JAVA Classes. When I run this part of the code, I get the following errors: ERROR: Failed to add sasjar groovy_2.3.4 (I downloaded this version so I changed it from 2.3.1 in your code to 2.3.4 in mine). And I get this error twice: The SUBMIT command failed. Can you help me fix this?

Thanks

[FriedEgg]

Okay,

That is a area I would expect a person not familiar with PROC GROOVY to normally encounter their first issue here.   The statement I am using is:

add sasjar="<build module"> version="<build version>";

This is used to reference a jar file included in you SAS installation, not something you would manually download, as you state you have done.  To find what bundle and version you have included in your installation, which may already be sufficient run the following:

proc javainfo picklist 'base/groovy.txt'; run;

If the version is 1.7 then you are going to need an updated version.  In some cases, SAS has already included an additional version in the SASVersionedJarRepository, which you can search through and find the appropriate version and bundle info in the META_INF.

Since you have already downloaded the 2.3.4 jar file we can just utilize that instead, but you need to use:

add classpath="/path/to/groovy-all.2.3.4.jar";

instead of add sasjar ...;

This should get you past this first hurdle.

[NandanM]

Thanks for your help. It did help me get past the first hurdle but I still can't run the submit commands successfully.

[FriedEgg]

I can't help you further if you give me nothing to work with...  What is the next issue you are having?

[robm]

hmmm I get this error now that I am not referecing the jar directly

17   add sasjar="groovy-all-2.2.1";

java.lang.IllegalArgumentException: Failed to find sasjar groovy-all-2.2.1 0.0.0.

CLASSPATH=C:\Program Files\Java\jre7\classes

C:\Program Files\Java\jre7\classes>dir
Volume in drive C has no label.
Volume Serial Number is 889C-285D

Directory of C:\Program Files\Java\jre7\classes

07/13/2014  08:51 AM    <DIR>          .
07/13/2014  08:51 AM    <DIR>          ..
07/13/2014  08:39 AM         6,558,498 groovy-all-2.2.1.jar
               1 File(s)      6,558,498 bytes
               2 Dir(s)  23,812,456,448 bytes free

C:\Program Files\Java\jre7\classes>

[FriedEgg]

@robm,

You should refer to my previous response to NandanM, about ADD SASJAR vs. ADD CLASSPATH for a more in-depth answer. 

Replace your add sasjar statement with add classpath="C:\Program Files\Java\jre7\classes\groovy-all-2.2.1.jar"; or whatever the correct full path is.

Regards,

FriedEgg

[robm]

ok thanks that worked , however I keep getting this 403 error ....

310  endsubmit;

Unexpected error: 403 : Forbidden

cache-control : no-cache, no-store, must-revalidate, pre-check=0, post-check=0

content-length : 105

content-type : application/json; charset=utf-8

date : Mon, 14 Jul 2014 01:40:13 GMT

expires : Tue, 31 Mar 1981 05:00:00 GMT

last-modified : Mon, 14 Jul 2014 01:40:13 GMT

pragma : no-cache

server : tfe

set-cookie :

_twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZ

WR7ADoMY3NyZl9pZCIlZDNkMDJhMjc0YjAzNGIzOWJm%250AODI0MzY2YWZiNzQ0MjE6B2lkIiUxMjFlNThiNTZhZWNmOTdj

OTU0ZTFkY2My%250ANmY4Yzk0NjoPY3JlYXRlZF9hdGwrCAH9hjJHAQ%253D%253D--863dd82fd57c3cef15985bbb3aa3a

ee5bdda6b7e; domain=.twitter.com; path=/; secure; HttpOnly

set-cookie : guest_id=v1%3A140530201317976673; Domain=.twitter.com; Path=/; Expires=Wed,

13-Jul-2016 01:40:13 UTC

status : 403 Forbidden

[FriedEgg]

@robm,

Please replace the getBearerToken method with the following so we can get additional information in the log (the new portion is highlighted in blue):

        private getBearerToken(){

            def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

            def api = new HTTPBuilder("https://api.twitter.com")

            def access_token

            api.request(POST,JSON) { req ->

                uri.path = "/oauth2/token"

                headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

                headers."Authorization" = "Basic "+encoded_basic

                body = "grant_type=client_credentials"

                response.success = { resp, json ->

                    access_token = json.access_token

                }

                response.failure = { resp, json ->

                    println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"

                    resp.headers.each { println "${it.name} : ${it.value}" }

                    ret = reader.getText()

                    println ret

                }

            }

            return access_token

        }

[robm]

Whoops posted that last one too fast thought I was usuing your new code the get bearer token

replaced private getBearerToken() with private getBearerToken(api_key,api_secret)

now below is the full output (edited out the api_ token and secret)

782

783  %let API_KEY='<API_KEY>' ;

784  %let API_SECRET='<API_SECRET>' ;

785

786  /* Since I like to use @Grab in groovy, but SAS does not include the Ivy prerequisite,

786! let's get Ivy */

787  filename ivy "%sysfunc(pathname(work,l))/ivy.jar";

788  proc http

789     method = "get"

790     url    =

790! "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"

791     out    = ivy;

792  run;

NOTE: PROCEDURE HTTP used (Total process time):

      real time           2.23 seconds

      cpu time            0.24 seconds

793

794  filename cp  temp; *A temporary file for proc groovy output that is currently not necessary

794! ;

795  proc

795!      groovy classpath=cp;

NOTE: The ADD CLASSPATH command completed.

796

797  add classpath=ivy;

NOTE: The ADD CLASSPATH command completed.

797!                    *Include the Ivy jar we downloaded;

798  add classpath="C:\Program Files\Java\jre6\classes\groovy-all-2.2.1.jar";

NOTE: The ADD CLASSPATH command completed.

799  /*add sasjar="groovy-all-2.2.1"; version="2.2.1.0_SAS_20130517000930"; You will need to

799! adjust this line to fit your installation, this is what I have from 9.4 on 64-bit Linux.

799! We are collecting groovy.json.JsonSlurper here, although we could get it with @Grab

799! instead.;*/

800

801  submit "&API_KEY" "&API_SECRET";

802

803  @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder',

803! version='0.7')

804  @Grab(group='commons-codec', module='commons-codec', version='1.2')

805

806  import groovy.json.JsonSlurper

807  import org.apache.commons.codec.binary.Base64

808  import groovyx.net.http.HTTPBuilder

809  import static groovyx.net.http.ContentType.*

810  import static groovyx.net.http.Method.*

811

812

813

814

815

816  /*

817  def getBearerToken(api_key,api_secret) {

818    def auth = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

819

820    def http = new HTTPBuilder("https://api.twitter.com")

821

822    http.request(POST,JSON) { req ->

823      uri.path = "/oauth2/token"

824      headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

825      headers."Authorization" = "Basic "+auth

826

827      body="grant_type=client_credentials"

828

829      response.success = { resp, json ->

830         bearer_token = json.access_token

831      }

832

833      response.failure = { resp, json ->

834         println "Unexpected error: ${resp.statusLine.statusCode} :

834! ${resp.statusLine.reasonPhrase}"

835         resp.headers.each { println "${it.name} : ${it.value}" }

836         ret = reader.getText()

837         println ret

838      }

839    }

840  return bearer_token

841  }

842  */

843

844  /*------------------------------------------debug-----*/

845

846          private getBearerToken(api_key,api_secret){

847              def encoded_basic = new

847! String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

848              def api = new HTTPBuilder("https://api.twitter.com")

849              def access_token

850              api.request(POST,JSON) { req ->

851                  uri.path = "/oauth2/token"

852                  headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

853                  headers."Authorization" = "Basic "+encoded_basic

854                  body = "grant_type=client_credentials"

855

856

857                  response.success = { resp, json ->

858                      access_token = json.access_token

859                  }

860

861                  response.failure = { resp, json ->

862                      println "Unexpected error: ${resp.statusLine.statusCode} :

862! ${resp.statusLine.reasonPhrase}"

863                      resp.headers.each { println "${it.name} : ${it.value}" }

864                      ret = reader.getText()

865                      println ret

866                  }

867              }

868              return access_token

869          }

870

871

872  /*------------------------------------------debug----*/

873

874

875

876  exports.bearer_token = getBearerToken(args[0].toString() , args[1].toString())

877

878  endsubmit;

Unexpected error: 403 : Forbidden

cache-control : no-cache, no-store, must-revalidate, pre-check=0, post-check=0

content-length : 105

content-type : application/json; charset=utf-8

date : Mon, 14 Jul 2014 14:43:43 GMT

expires : Tue, 31 Mar 1981 05:00:00 GMT

last-modified : Mon, 14 Jul 2014 14:43:43 GMT

pragma : no-cache

server : tfe

set-cookie :

_twitter_sess=<TWITTER_SESSION>

c15bb60c498; domain=.twitter.com; path=/; secure; HttpOnly

set-cookie : guest_id=v1%3A140534902318015243; Domain=.twitter.com; Path=/; Expires=Wed,

13-Jul-2016 14:43:43 UTC

status : 403 Forbidden

strict-transport-security : max-age=631138519

vary : Accept-Encoding

x-content-type-options : nosniff

x-frame-options : DENY

x-mid : b47ae7b5172e14e30aaec9feea96ffa59d85add5

x-runtime : 0.00624

x-transaction : 4f8c2174eeb116db

x-ua-compatible : IE=edge,chrome=1

x-xss-protection : 1; mode=block

groovy.lang.MissingPropertyException: No such property: reader for class:

groovyx.net.http.HTTPBuilder$RequestConfigDelegate

Possible solutions: headers

     at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:49)

     at

org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:49

)

     at

org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSi

te.java:241)

     at Script10$_getBearerToken_closure1_closure3.doCall(Script10.groovy:63)

     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

     at java.lang.reflect.Method.invoke(Unknown Source)

     at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)

     at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)

     at

org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)

     at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)

     at groovy.lang.Closure.call(Closure.java:276)

     at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:494)

     at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1070)

     at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1044)

     at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:506)

     at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:425)

     at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:374)

     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

     at java.lang.reflect.Method.invoke(Unknown Source)

     at

org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.inv

oke(PojoMetaMethodSite.java:229)

     at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)

     at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)

     at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)

     at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)

     at Script10.getBearerToken(Script10.groovy:49)

     at Script10.this$4$getBearerToken(Script10.groovy)

     at Script10$this$4$getBearerToken.callCurrent(Unknown Source)

     at

org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)

     at

org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:143)

     at

org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:155)

     at Script10.run(Script10.groovy:75)

ERROR: The SUBMIT command failed.

879

880  quit;

NOTE: The SAS System stopped processing this step because of errors.

NOTE: PROCEDURE GROOVY used (Total process time):

      real time           1.06 seconds

      cpu time            0.17 seconds

WARNING: Apparent symbolic reference BEARER_TOKEN not resolved.

881

882

883  %put &BEARER_TOKEN;

&BEARER_TOKEN

[FriedEgg]

@robm,

Please refer to the following link for a full implementation, you are just attempting to run the first portion of what I had posted, to just collect a bearer token, not to actually collect search results...

I believe you are still missing some dependents as the issue appears to possibly be related to parsing the JSON error response in the body of the 403 forbiden you are getting.  Let's see what the error is by switching to a TEXT content type for the response, just to see what we get there.

*don't put quotes on these...;

%let API_KEY=<API_KEY>;

%let API_SECRET=<API_SECRET>;

filename ivy "%sysfunc(pathname(work,l))/ivy.jar";

proc http

   method = "get"

   url    = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"

   out    = ivy;

run;

filename cp  temp;

proc groovy classpath=cp;

add classpath=ivy;

add classpath="C:\Program Files\Java\jre6\classes\groovy-all-2.2.1.jar";

submit "&API_KEY" "&API_SECRET";

@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')

@Grab(group='commons-codec', module='commons-codec', version='1.2')

import groovy.json.JsonSlurper

import org.apache.commons.codec.binary.Base64

import groovyx.net.http.HTTPBuilder

import static groovyx.net.http.ContentType.*

import static groovyx.net.http.Method.*

def getBearerToken(api_key,api_secret) {

   def auth = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))

   def http = new HTTPBuilder("https://api.twitter.com")

   def bearer_token

   http.request(POST,TEXT) { req ->

      uri.path = "/oauth2/token"

      headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"

      headers."Authorization" = "Basic "+auth

      body="grant_type=client_credentials"

      response.success = { resp, text ->

         bearer_token = text

      }

      response.failure = { resp, text ->

         ret = reader.getText()

         println ret

      }

   }

   return bearer_token

}

exports.bearer_token = getBearerToken(args[0].toString() , args[1].toString())

endsubmit;

quit;

%put &BEARER_TOKEN;

[robm]

Aaaagh dammit ! it was the '' I put around the api_key and secret ..thanks for all the help fried it seems to be working now I eventried your to dataset code and it worked