Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

SAS Web Crawler - Piping output to custom storage?

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 12
Accepted Solution

SAS Web Crawler - Piping output to custom storage?

Hi,

I'm new to Information Retrieval Studio and SAS Crawler, but I was wondering if it was possible to pipe the output of the crawler to a storage place of my choosing - for example, and elasticsearch index?

Cheers,

Seb


Accepted Solutions
Solution
‎11-07-2014 10:01 AM
Occasional Contributor
Posts: 12

Re: SAS Web Crawler - Piping output to custom storage?

Posted in reply to GrokAndRoll

Ok I've since found that custom postprocessors can be written in Python and executed as steps in the Pipeline Server. Therefore you could write something in Python which took the document and stored it in, say elasticsearch.

Thanks,

Seb

View solution in original post


All Replies
Solution
‎11-07-2014 10:01 AM
Occasional Contributor
Posts: 12

Re: SAS Web Crawler - Piping output to custom storage?

Posted in reply to GrokAndRoll

Ok I've since found that custom postprocessors can be written in Python and executed as steps in the Pipeline Server. Therefore you could write something in Python which took the document and stored it in, say elasticsearch.

Thanks,

Seb

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 1 reply
  • 340 views
  • 0 likes
  • 1 in conversation