It sounds to me like you've got cavitation. Basically, the blades are turning so fast that the water around the blades boils. Boiling can happen at normal pressure and high temperature, like on a stove, or at normal temperature and low pressure, as in your case.
The blades boil the water, then the prop is surrounded by vapor instead of water and there isn't anything to push.
Here's a hobby thread on cavitation, where the OP complains,
I was driving my boat on saturday and I noticed sometimes my motor would spin very fast and the boat would go no where as if it were not spinning the prop.
and another poster replies, in part,
Cavitation is caused by the prop spinning too fast... You get something simular if the prop is 'too far' out of the water at some point. Lots of 'splashing', not much water moving.
(Emphasis added). That post goes on to say that it's not exactly cavitation, but it 100% is cavitation.
You need to slow the prop down. You basically have three options to slow the prop down:
- Write smaller values to the ESC to request a lower speed (easiest), or
- Use a gearbox to change motor speed to prop speed (most efficient), or
- Get a larger and/or more aggressive prop such that the torque required to turn the prop naturally lowers the motor speed (least efficient).
The problem with option 1 is that there is some "maximum throttle" at which you'll start cavitating, and then you're limited to the power and speed output of the motor from 0% to whatever that maximum throttle is. Even a low kV BLDC motor like you have still naturally spins very fast. You may have controller issues running a BLDC motor under relatively high torque at such a low speed.
Option 2 is the most expensive because it requires you to purchase a gearbox, but that's the way it's done on commercial and military ships - run the (steam or gas) turbines at high speed, where they operate most efficiently, then gear that down to a low speed, where the prop operates most efficiently.
Option 3 is probably not going to get you very far, because you'll probably find it hard to get a suitable prop installed in the correct location.
Of course, there's an option 4 also: replace the motor. You would want to go for the lowest speed motor you could find; something like a 100rpm motor would probably be perfect.