I am comfortable with using an adze in this way, and I have been using one for 20 years now, on and off. I would suggest that if you are new to using this tool that you always work between the legs so that any follow through or miss strokes will not be an issue.
As with any hand tool we are using it with highly controlled motions, not widely throwing it through the air in the hope that it will cut in the right place. Every stroke counts, well that is my aim, even the best of us occasionally have glancing blows.
The adze is a great tool and works by cutting having the pivot point which is the end of the handle in a fixed and stable position. My hand usually holds the handle firmly against my crotch or thigh. Once you have found and anchored into this point then everything falls into place. If the hand holding the end of the handle is floating then it can be very difficult to create a good finish.