You can hide it by default in a document but the user can always turn it back on again if they want to, as there's no way to disable the right-click menus.
The boolean /TB tag which controls it is inside the annotation's /3DA activation dictionary. Set it to false and the toolbar is hidden.