topic SAS Web Crawler - Piping output to custom storage? in SAS Data Science https://communities.sas.com/t5/SAS-Data-Science/SAS-Web-Crawler-Piping-output-to-custom-storage/m-p/160912#M1710 <HTML><HEAD></HEAD><BODY><P>Hi,</P><P></P><P>I'm new to Information Retrieval Studio and SAS Crawler, but I was wondering if it was possible to pipe the output of the crawler to a storage place of my choosing - for example, and elasticsearch index?</P><P></P><P>Cheers,</P><P>Seb</P></BODY></HTML> Fri, 07 Nov 2014 12:01:55 GMT GrokAndRoll 2014-11-07T12:01:55Z SAS Web Crawler - Piping output to custom storage? https://communities.sas.com/t5/SAS-Data-Science/SAS-Web-Crawler-Piping-output-to-custom-storage/m-p/160912#M1710 <HTML><HEAD></HEAD><BODY><P>Hi,</P><P></P><P>I'm new to Information Retrieval Studio and SAS Crawler, but I was wondering if it was possible to pipe the output of the crawler to a storage place of my choosing - for example, and elasticsearch index?</P><P></P><P>Cheers,</P><P>Seb</P></BODY></HTML> Fri, 07 Nov 2014 12:01:55 GMT https://communities.sas.com/t5/SAS-Data-Science/SAS-Web-Crawler-Piping-output-to-custom-storage/m-p/160912#M1710 GrokAndRoll 2014-11-07T12:01:55Z Re: SAS Web Crawler - Piping output to custom storage? https://communities.sas.com/t5/SAS-Data-Science/SAS-Web-Crawler-Piping-output-to-custom-storage/m-p/160913#M1711 <HTML><HEAD></HEAD><BODY><P>Ok I've since found that custom postprocessors can be written in Python and executed as steps in the Pipeline Server. Therefore you could write something in Python which took the document and stored it in, say elasticsearch.</P><P></P><P>Thanks,</P><P>Seb</P></BODY></HTML> Fri, 07 Nov 2014 15:01:06 GMT https://communities.sas.com/t5/SAS-Data-Science/SAS-Web-Crawler-Piping-output-to-custom-storage/m-p/160913#M1711 GrokAndRoll 2014-11-07T15:01:06Z