Is it possible that admin can only submit the answer?
Yes, it can be possible by disabling answer form.
To disable the answer form for the frontend, set the following configuration to No
Stores > Configuration > Magediary > Product Questions > Answer > Show an Answer Form : No