A fully charged LiPo battery cell should has a voltage of 4.2 V. Assuming that you have three cells in series, that would give 12.6 V. When you apply a big load, this voltage of course, will decrease instantly but if you measured 11.98 V in open-circuit that means the batteries were not fully charged. I guess your batteries were not charged at all.
Here is the discharge characteristic of the Samsung IRC16850, taken from this site.

A discharge current of 7 A will drain your battery flat in 20 minutes.
Furthermore a discharged battery's voltage is not the same when the load is still connected and when it is in open circuit. After disconnecting the load the battery voltage will rise, this will be the OCV (open circuit voltage).

This is a figure of a LiPo battery pack, 12 cells in series. When the load is disconnected the voltage has risen from 2.7 V to ~3.2 V in a short period of time (PB 6 voltage). This happens with your batteries too but if you reconnect the motor the voltage will drop back.
And finally, if you measure 5-6 V on your battery pack that means one cell's voltage is around 2 V and the cells are probably damaged now. The Samsung ICR18650's discharge cut-off voltage 2.75V, which means that you should not use it when the voltage is lower. Actually you should not use the cells but recharge when the voltage drops to 3 V.
If a cell is heating up it means that its internal resistance has risen. The dissipated heat will be:
$$ P = R_{internal} \times I_{discharge}^2 $$
this means that a part of the stored energy is wasted on heating the cell itself (which is very DANGEROUS as LiPo cells can easily catch fire). The effective battery capacity is lower in this case.
Without load your motor should drain 800 mA and the batteries should last 2.5 hours with that. I suggest to make some tests without any load attached to you motor, or just some light load. These batteries would be capable of supporting that, just make sure that they are fully charged. But probably you will need new cells too.