Hi,
I am using PC SAS on server, I have the control to modify options on server, can I please know what option do I need to change.
I tried the following query which didn’t work.
%let search_query=%23SAS&lang=en;
Thank you for your time.
You want to initialize the SAS session on the server with the options "-dbcs -encoding UTF-8." Or you can use the shortcut that should have been created as a part of the installation process. Or you can append the config file for UTF-8 support that should have been created as part of your installation to your session. This file should be located at something like: "C:\Program File\SASHome\SASFoundation\9.4\nls\u8\sasv9.cfg"
Hey Guys,
I have my API Key and API secret. But I need to know a stepwise way to write the code to fetch tweets from twitter. I understand that we need to initialize the key and secret but if anyone could articulate one piece of code that would get the job done,I would appreciate it. I am using a Macintosh by the way.
Thanks
The last piece of code which I posted here will collect tweets into a data set. The comments provide a very high level overview of the few required steps. Since you state you are using a Mac, I assume you are running SAS inside a Virtual Machine of some kind, since the last version of SAS I am aware of to run natively was 6.2 or so for PowerPC... If you are having a specific issue attempting to utilize this code I will do what I can to aide you through it.
Regards,
FriedEgg
Thanks for your response fried egg. You are correct when you say that I'm using a virtual machine. The specific part of the code that's giving me an issue is - Compile GROOVY/JAVA Classes. When I run this part of the code, I get the following errors: ERROR: Failed to add sasjar groovy_2.3.4 (I downloaded this version so I changed it from 2.3.1 in your code to 2.3.4 in mine). And I get this error twice: The SUBMIT command failed. Can you help me fix this?
Thanks
Okay,
That is a area I would expect a person not familiar with PROC GROOVY to normally encounter their first issue here. The statement I am using is:
add sasjar="<build module"> version="<build version>";
This is used to reference a jar file included in you SAS installation, not something you would manually download, as you state you have done. To find what bundle and version you have included in your installation, which may already be sufficient run the following:
proc javainfo picklist 'base/groovy.txt'; run;
If the version is 1.7 then you are going to need an updated version. In some cases, SAS has already included an additional version in the SASVersionedJarRepository, which you can search through and find the appropriate version and bundle info in the META_INF.
Since you have already downloaded the 2.3.4 jar file we can just utilize that instead, but you need to use:
add classpath="/path/to/groovy-all.2.3.4.jar";
instead of add sasjar ...;
This should get you past this first hurdle.
Thanks for your help. It did help me get past the first hurdle but I still can't run the submit commands successfully.
I can't help you further if you give me nothing to work with... What is the next issue you are having?
hmmm I get this error now that I am not referecing the jar directly
17 add sasjar="groovy-all-2.2.1";
java.lang.IllegalArgumentException: Failed to find sasjar groovy-all-2.2.1 0.0.0.
CLASSPATH=C:\Program Files\Java\jre7\classes
C:\Program Files\Java\jre7\classes>dir
Volume in drive C has no label.
Volume Serial Number is 889C-285D
Directory of C:\Program Files\Java\jre7\classes
07/13/2014 08:51 AM <DIR> .
07/13/2014 08:51 AM <DIR> ..
07/13/2014 08:39 AM 6,558,498 groovy-all-2.2.1.jar
1 File(s) 6,558,498 bytes
2 Dir(s) 23,812,456,448 bytes free
C:\Program Files\Java\jre7\classes>
ok thanks that worked , however I keep getting this 403 error ....
310 endsubmit;
Unexpected error: 403 : Forbidden
cache-control : no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-length : 105
content-type : application/json; charset=utf-8
date : Mon, 14 Jul 2014 01:40:13 GMT
expires : Tue, 31 Mar 1981 05:00:00 GMT
last-modified : Mon, 14 Jul 2014 01:40:13 GMT
pragma : no-cache
server : tfe
set-cookie :
_twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZ
WR7ADoMY3NyZl9pZCIlZDNkMDJhMjc0YjAzNGIzOWJm%250AODI0MzY2YWZiNzQ0MjE6B2lkIiUxMjFlNThiNTZhZWNmOTdj
OTU0ZTFkY2My%250ANmY4Yzk0NjoPY3JlYXRlZF9hdGwrCAH9hjJHAQ%253D%253D--863dd82fd57c3cef15985bbb3aa3a
ee5bdda6b7e; domain=.twitter.com; path=/; secure; HttpOnly
set-cookie : guest_id=v1%3A140530201317976673; Domain=.twitter.com; Path=/; Expires=Wed,
13-Jul-2016 01:40:13 UTC
status : 403 Forbidden
@robm,
Please replace the getBearerToken method with the following so we can get additional information in the log (the new portion is highlighted in blue):
private getBearerToken(){
def encoded_basic = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))
def api = new HTTPBuilder("https://api.twitter.com")
def access_token
api.request(POST,JSON) { req ->
uri.path = "/oauth2/token"
headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
headers."Authorization" = "Basic "+encoded_basic
body = "grant_type=client_credentials"
response.success = { resp, json ->
access_token = json.access_token
}
response.failure = { resp, json ->
println "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
resp.headers.each { println "${it.name} : ${it.value}" }
ret = reader.getText()
println ret
}
}
return access_token
}
Whoops posted that last one too fast thought I was usuing your new code the get bearer token
replaced private getBearerToken() with private getBearerToken(api_key,api_secret)
now below is the full output (edited out the api_ token and secret)
782
783 %let API_KEY='<API_KEY>' ;
784 %let API_SECRET='<API_SECRET>' ;
785
786 /* Since I like to use @Grab in groovy, but SAS does not include the Ivy prerequisite,
786! let's get Ivy */
787 filename ivy "%sysfunc(pathname(work,l))/ivy.jar";
788 proc http
789 method = "get"
790 url =
790! "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"
791 out = ivy;
792 run;
NOTE: PROCEDURE HTTP used (Total process time):
real time 2.23 seconds
cpu time 0.24 seconds
793
794 filename cp temp; *A temporary file for proc groovy output that is currently not necessary
794! ;
795 proc
795! groovy classpath=cp;
NOTE: The ADD CLASSPATH command completed.
796
797 add classpath=ivy;
NOTE: The ADD CLASSPATH command completed.
797! *Include the Ivy jar we downloaded;
798 add classpath="C:\Program Files\Java\jre6\classes\groovy-all-2.2.1.jar";
NOTE: The ADD CLASSPATH command completed.
799 /*add sasjar="groovy-all-2.2.1"; version="2.2.1.0_SAS_20130517000930"; You will need to
799! adjust this line to fit your installation, this is what I have from 9.4 on 64-bit Linux.
799! We are collecting groovy.json.JsonSlurper here, although we could get it with @Grab
799! instead.;*/
800
801 submit "&API_KEY" "&API_SECRET";
802
803 @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder',
803! version='0.7')
804 @Grab(group='commons-codec', module='commons-codec', version='1.2')
805
806 import groovy.json.JsonSlurper
807 import org.apache.commons.codec.binary.Base64
808 import groovyx.net.http.HTTPBuilder
809 import static groovyx.net.http.ContentType.*
810 import static groovyx.net.http.Method.*
811
812
813
814
815
816 /*
817 def getBearerToken(api_key,api_secret) {
818 def auth = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))
819
820 def http = new HTTPBuilder("https://api.twitter.com")
821
822 http.request(POST,JSON) { req ->
823 uri.path = "/oauth2/token"
824 headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
825 headers."Authorization" = "Basic "+auth
826
827 body="grant_type=client_credentials"
828
829 response.success = { resp, json ->
830 bearer_token = json.access_token
831 }
832
833 response.failure = { resp, json ->
834 println "Unexpected error: ${resp.statusLine.statusCode} :
834! ${resp.statusLine.reasonPhrase}"
835 resp.headers.each { println "${it.name} : ${it.value}" }
836 ret = reader.getText()
837 println ret
838 }
839 }
840 return bearer_token
841 }
842 */
843
844 /*------------------------------------------debug-----*/
845
846 private getBearerToken(api_key,api_secret){
847 def encoded_basic = new
847! String(Base64.encodeBase64((api_key+":"+api_secret).bytes))
848 def api = new HTTPBuilder("https://api.twitter.com")
849 def access_token
850 api.request(POST,JSON) { req ->
851 uri.path = "/oauth2/token"
852 headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
853 headers."Authorization" = "Basic "+encoded_basic
854 body = "grant_type=client_credentials"
855
856
857 response.success = { resp, json ->
858 access_token = json.access_token
859 }
860
861 response.failure = { resp, json ->
862 println "Unexpected error: ${resp.statusLine.statusCode} :
862! ${resp.statusLine.reasonPhrase}"
863 resp.headers.each { println "${it.name} : ${it.value}" }
864 ret = reader.getText()
865 println ret
866 }
867 }
868 return access_token
869 }
870
871
872 /*------------------------------------------debug----*/
873
874
875
876 exports.bearer_token = getBearerToken(args[0].toString() , args[1].toString())
877
878 endsubmit;
Unexpected error: 403 : Forbidden
cache-control : no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-length : 105
content-type : application/json; charset=utf-8
date : Mon, 14 Jul 2014 14:43:43 GMT
expires : Tue, 31 Mar 1981 05:00:00 GMT
last-modified : Mon, 14 Jul 2014 14:43:43 GMT
pragma : no-cache
server : tfe
set-cookie :
_twitter_sess=<TWITTER_SESSION>
c15bb60c498; domain=.twitter.com; path=/; secure; HttpOnly
set-cookie : guest_id=v1%3A140534902318015243; Domain=.twitter.com; Path=/; Expires=Wed,
13-Jul-2016 14:43:43 UTC
status : 403 Forbidden
strict-transport-security : max-age=631138519
vary : Accept-Encoding
x-content-type-options : nosniff
x-frame-options : DENY
x-mid : b47ae7b5172e14e30aaec9feea96ffa59d85add5
x-runtime : 0.00624
x-transaction : 4f8c2174eeb116db
x-ua-compatible : IE=edge,chrome=1
x-xss-protection : 1; mode=block
groovy.lang.MissingPropertyException: No such property: reader for class:
groovyx.net.http.HTTPBuilder$RequestConfigDelegate
Possible solutions: headers
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:49)
at
org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:49
)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSi
te.java:241)
at Script10$_getBearerToken_closure1_closure3.doCall(Script10.groovy:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:276)
at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:494)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1070)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1044)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:506)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:425)
at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:374)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.inv
oke(PojoMetaMethodSite.java:229)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at Script10.getBearerToken(Script10.groovy:49)
at Script10.this$4$getBearerToken(Script10.groovy)
at Script10$this$4$getBearerToken.callCurrent(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:143)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:155)
at Script10.run(Script10.groovy:75)
ERROR: The SUBMIT command failed.
879
880 quit;
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE GROOVY used (Total process time):
real time 1.06 seconds
cpu time 0.17 seconds
WARNING: Apparent symbolic reference BEARER_TOKEN not resolved.
881
882
883 %put &BEARER_TOKEN;
&BEARER_TOKEN
@robm,
Please refer to the following link for a full implementation, you are just attempting to run the first portion of what I had posted, to just collect a bearer token, not to actually collect search results...
I believe you are still missing some dependents as the issue appears to possibly be related to parsing the JSON error response in the body of the 403 forbiden you are getting. Let's see what the error is by switching to a TEXT content type for the response, just to see what we get there.
*don't put quotes on these...;
%let API_KEY=<API_KEY>;
%let API_SECRET=<API_SECRET>;
filename ivy "%sysfunc(pathname(work,l))/ivy.jar";
proc http
method = "get"
url = "http://central.maven.org/maven2/org/apache/ivy/ivy/2.3.0-rc1/ivy-2.3.0-rc1.jar"
out = ivy;
run;
filename cp temp;
proc groovy classpath=cp;
add classpath=ivy;
add classpath="C:\Program Files\Java\jre6\classes\groovy-all-2.2.1.jar";
submit "&API_KEY" "&API_SECRET";
@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')
@Grab(group='commons-codec', module='commons-codec', version='1.2')
import groovy.json.JsonSlurper
import org.apache.commons.codec.binary.Base64
import groovyx.net.http.HTTPBuilder
import static groovyx.net.http.ContentType.*
import static groovyx.net.http.Method.*
def getBearerToken(api_key,api_secret) {
def auth = new String(Base64.encodeBase64((api_key+":"+api_secret).bytes))
def http = new HTTPBuilder("https://api.twitter.com")
def bearer_token
http.request(POST,TEXT) { req ->
uri.path = "/oauth2/token"
headers."Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
headers."Authorization" = "Basic "+auth
body="grant_type=client_credentials"
response.success = { resp, text ->
bearer_token = text
}
response.failure = { resp, text ->
ret = reader.getText()
println ret
}
}
return bearer_token
}
exports.bearer_token = getBearerToken(args[0].toString() , args[1].toString())
endsubmit;
quit;
%put &BEARER_TOKEN;
Aaaagh dammit ! it was the '' I put around the api_key and secret ..thanks for all the help fried it seems to be working now I eventried your to dataset code and it worked
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 16. Read more here about why you should contribute and what is in it for you!
Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.
Find more tutorials on the SAS Users YouTube channel.
Select SAS Training centers are offering in-person courses. View upcoming courses for: