In the matrix you can enter notes on both the individual stakeholder and for the overall matrix. You can manage the access to the overall matrix notes field via a permission set by following this guide.
The permission set ARPEDIO Matrix User gives read and edit access to the overall matrix notes field. If you want to change this access for some users, then you should clone this permission set. In Object Settings, choose the object Arpedio StakeHolderMap and change the field permission for Note.