In order to use a library, users need to be able to physically access the directory (which is the physical representation of a SAS library) and the files in that library.
So, on the operating system level, the users in question need read and execute permissions on that directory, either per username, group or "others", or through properly managed access control lists. Mind that my explanation is for UNIX, not Windows toyboxes.
Once this is verified, define the libraries in SAS MC; in the Authorization tab, make sure that either the users or a group to which they belong have the necessary permissions on the metadata and data. Also make sure that the SAS folder in which the library appears in the Folders structure can be "seen" by the users in question. If you want to restrict access, the first groups that need to have their settings set to "Denied" are PUBLIC and SASUSERS.
It may be that some combinations can't be done if group permissions are too permissive, so denying a single user may fail because of other more permissive settings. You may have to resort to some tricks in the operating system for this, as OS permissions trump everything else.
If you positively need to restrict access for security reasons, blocking users on the OS level is mandatory, as they can always execute a simple LIBNAME statement themselves when they have physical access.