For a 5-DOF manipulator you will have 15 DH parameters. Three sets of joint angles will yield 9 equations, so that will not be enough to determine the parameters. And this is assuming you know the end-effector position, whereas if you are simply trying to use the constraint that all three positions are equal then you will have even fewer equations to work with.
You should be able to estimate the $\alpha$ parameters simply by inspecting the robot -- hopefully they are all 0 deg or $\pm$90 deg.
If you are measuring the end-effector position then you can simply set up as many test points as are necessary to get all of the parameters. This can be done strategically to simplify the process. Start with the outermost link, rotate that joint only, and look at how the end-effector position changes, then you can determine that link's parameters. Then move to the next link and repeat.
Alternatively, you can use the forward kinematics in a differential form as would be used to determine end-effector velocity, $\dot{p}$, in terms of joint angle velocities, $\dot{\theta}$, based on the Jacobian, $J$.
$\dot{p} = J \dot{\theta}$
Consider this relationship in terms of differences in end-effector position related to differences in joint angles:
$\Delta p = J \Delta \theta$
The above relationship can then be used to solve for the DH parameters given a series of measured $\Delta p$ and $\Delta \theta$ values using a non-linear least-squares solution.
