BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
MarkPeskir
Obsidian | Level 7

** This is all public data. If this code gets to working, this script can become a resource for anyone who needs hourly weather details! **

 

My goal: create a process to automatically pull all NCEI Land-based station weather data files for 2017, for NC/SC stations.

 

Version SAS 9.4M3, SAS Enterprise Guide, Windows local machine.

 

List of stations: ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-history.txt

Hourly weather data folder: https://www.ncei.noaa.gov/data/global-hourly/access/2017/

 

This pulls all stations in the US. It works fine.

 

 

filename testurl ftp "isd-history.txt" cd="/pub/data/noaa/" host="ftp.ncdc.noaa.gov" user="anonymous";

data AUTORES.WEATHER_STATIONS;
infile testurl  FIRSTOBS=24; 
input USAF $ 1-6
	WBAN $ 8-12
	STATION_NAME $ 14-43
	CTRY $ 44-45
	ST $ 49-50
	CALL $ 52-55
	LAT 58-64
	LON 66-73
	ELEV 75-81
	BEGIN 83-90
	END 92-99
	;
	IF CTRY EQ "US";
run;

 

 

This creates a series of macro variables, "URL1" to "URL###", where ### is the number of stations to pull data for.

Variable values are the direct URL to the file to download.

This works as expected. 

 

 

data _NULL_;
set AUTORES.WEATHER_STATIONS end=eof;
retain i;
if _n_ = 1 then i = 1;
if (ST EQ "NC" OR ST EQ "SC") AND END GT 20170000;
thisurl = cats("""https://www.ncei.noaa.gov/data/global-hourly/access/2017/", USAF, WBAN, ".csv""");
urlnum = cats("URL", i);
call symput(urlnum, thisurl); 
	strI = i;
	call symput("urlcount", left(put(strI, 8.)));
	i = i + 1;
run;

 

 

Here's where it goes south: the macro loops as expected, but always only pulls the data for the LAST URL created (in this specific case, URL172). 

 

%MACRO GET_WEATHER_STATION_READINGS;
%DO i = 1 %TO &urlcount; 
data WORK.NEW_WEATHER;
filename WFile url &&URL&urlcount debug; 

infile WFile DLM=',' DSD MISSOVER firstobs=2;

input 
	STATION : $20.
	DATE : E8601DT20.
	SOURCE : $8.
	LATITUDE : 10.6
	LONGITUDE : 10.6
	ELEVATION : 10.6
	NAME : $30.
	REPORT_TYPE : $10.
	CALL_SIGN : $10.
	QUALITY_CONTROL : $10.
	WND : $20.
	CIG : $20.
	VIS : $20.
	TMP : $20.
	DEW : $20.
	SLP : $20.
	AA1 : $20.
	AA2 : $20.
	AY1 : $20.
	AY2 : $20.
	GA1 : $20.
	GA2 : $20.
	GA3 : $20.
	GE1 : $20.
	GF1 : $20.
	MW1 : $20.
	REM : $20.
	EQD : $20.
	;
format DATE DATETIME20.;
run;

proc append base=AUTORES.WEATHER_HOURLY data=WORK.NEW_WEATHER;
run;

sleep(5, 1); 

%end;
%MEND GET_WEATHER_STATION_READINGS;


%GET_WEATHER_STATION_READINGS;

As you may guess, i need it to pull URL1 - URL172, each one once.

It's a macro value issue, I suspect. I've got some basic understanding of macros, but I'm not sure how to change this into what I need.

(If the FILENAME command would accept a variable for a URL path, I doubt I'd need macros at all!!)

 

I've been fighting with this for three days, and solved a lot of other errors along the way. I humbly ask you please do not propose solutions without testing your code.  I do expect the final result file AUTORES.WEATHER_HOURLY to be a couple GB in size.

 

Thank you for your time.

Mark

1 ACCEPTED SOLUTION

Accepted Solutions
PGStats
Opal | Level 21

Shouldn't it be

 

filename WFile url &&URL&i debug;

PG

View solution in original post

2 REPLIES 2
PGStats
Opal | Level 21

Shouldn't it be

 

filename WFile url &&URL&i debug;

PG
MarkPeskir
Obsidian | Level 7
Ha, well, yes. Yes it should be.
So many other technical hurdles I jumped to reach this point, only to get tripped up on loop handling.
Thanks for your observant eye.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 2 replies
  • 1033 views
  • 0 likes
  • 2 in conversation