Assume the following scenario: “You wish to break inheritance on a document contained within a SharePoint document set and grant access to it to a specific user”. In other words, the user you are trying to grant access to the document to does not have access at the site level, at the library level, nor at the Document Set level, but is granted explicit contribute rights at the document level. So in this case, if the user tries to access the document directly by its URL he should have access to view and edit the document.
However, if you have the SharePoint server Publishing Infrastructure site collection feature activated, you may run into a case where the user receives an access denied error, even though he’s been granted implicit access to the document. This is because by default, the moment you activate the Publishing Infrastructure feature, SharePoint also activates a second site collection feature named Limited-access user permission lockdown mode.
This feature serves a specific purpose: Prevents anonymous users or users having Limited Access from accessing a folder on your site. This makes sense since the Publishing Infrastructure feature is used whenever you are creating a public internet site using SharePoint. Basically, whenever this feature is activated, users will get an access denied the moment they hit a folder on which they have Limited Access in the URL. Let me clarify, assume the path to your document within the document set is the following:
The user has limited access to every part of the URL that is underlined above (site, the document library and the document set). The moment SharePoint tries to resolve a part of the URL to which the user has Limited Access, it will return an access denied error.
The solution to this is to simply disable the Limited-access user permission lockdown mode feature at the site collection level. You however need to understand the possible consequences of turning this off if your SharePoint site allows anonymous access, you potentially open the dorr for them to access application pages for your lists and document libraries, which you probably don’t want to have happen.
For more information on the Limited-access user permission lockdown mode, please refer to the following Office Support article (scroll down to the last section).