topic how to remove emoji from an xml file using a unix command in SAS Enterprise Guide

how to remove emoji from an xml file using a unix command

alepage — Wed, 03 Jul 2024 14:12:03 GMT

Hello,

I am download a file using encoding = "UFT-8" and I have found that I have emoji into the xlm file.

I need to remove only the emoji from the xml file without changing the file structure.

How do we do that?

Please note that accented characters need to be kept into the xml file due to French language.

ex;

What I have:

<QID2_TEXT>La dame a été très patience et gentille et que ça fait longtemps que je suis assurée avec vous.
Si c’était possible, je me demandais s’il n’y aurais pas possibilité de diminuer le coût de l’assurance? 🙂</QID2_TEXT>

What I want:

Re: how to remove emoji from an xml file using a unix command

ChrisNZ — Thu, 04 Jul 2024 05:55:23 GMT

You could remove all unwanted characters using Perl.

Example of the Perl syntax to process a string:

echo ' Cœur coût de l’assurance? 🙂</QID2_TEXT>' | perl -C -pe 's/[^[:alnum:][:space:][:punct:]]+//g'

where you only keep alphanumeric characters, spaces and punctuation symbols results in:

Cœur coût de l’assurance? </QID2_TEXT>

1. Can't you have your SAS session using UTF-8? Your organisation should move to UTF8 to avoid this kind of headaches.

2. Note that the alnum posix expression is locale-specific

Re: how to remove emoji from an xml file using a unix command

alepage — Thu, 04 Jul 2024 10:01:31 GMT

I am loosing the apostrophe. How to keep / allow apostrophe

Cur coût de lassurance? </QID2_TEXT>

Re: how to remove emoji from an xml file using a unix command

ChrisNZ — Thu, 04 Jul 2024 10:47:10 GMT

That's strange. That's not the case for me.
The apostrophe is part of [:punct:] so should be conserved.
https://www.regular-expressions.info/posixbrackets.html
[:punct:]
Punctuation (and symbols).
!"#$%&'()*+,-./\:;<=>?@[]^_`{|}~

Re: how to remove emoji from an xml file using a unix command

alepage — Thu, 04 Jul 2024 11:34:19 GMT

How to apply your perl script to the xml file

ex:

perl -C -pe 's/[^èàûéîôÇÉÇÈ"@-_<>[:ascii:][:alnum:][:space:][:punct:]]+//g' /finsys.../VirageSurvey_2.xml

Re: how to remove emoji from an xml file using a unix command

ChrisNZ — Thu, 04 Jul 2024 12:01:59 GMT

The answer is easy to find if you'd just search.

perl -pe 's/[..]//g' < file.xml > file2.xml

Re: how to remove emoji from an xml file using a unix command

alepage — Thu, 04 Jul 2024 14:15:25 GMT

It works well thank you. But I still have the issue with the apostrophe that are missing. Do you know a work around to keep apostrophe since punct does not keep it.

Re: how to remove emoji from an xml file using a unix command

ChrisNZ — Thu, 04 Jul 2024 23:37:21 GMT

If you have more characters to conserve, just add them to the list.

Different characters can be used for apostrophes, beyond the single quote that punct preserves, like

 ' ’ ’