There are more than one way to skin a cat.
As mentioned above, over-protect the specific library with ACT(s) that prevent(s) the service account in metadata which does the export to access the library reference. Questionable.
Or, export selectively some metadata object types and exlude library ref (-types / -exclude etc.). Easier but still questionable.
Or, maybe better still if you can, change the folder path storing the Library reference to another branch apart from the root folder you want to export.