Your statement about the need to reimplement the same feature in order to test it could be applied for all software which needs to be unit-tested and it is not specific only to your problem. A unit test does not implement redundant functionality, in this case an error cannot be attributed to a faulty implementation in the tested code, as it can just as easily be a faulty implementation of the test code.
Unit tests for transformations usually work with data where you can devise the correct output for a given input by other means that the currently implemented software which needs testing. Ways to do this are:
- Fringe cases like zero transformations where the output is the same as the input
- Cases where the answer is easily predictable, like 90 (180, 270, 360, -90, -180, -360) degrees rotations without translations or simple translations without rotations
- Back-and forth transformation chains: define a complex chain and define the exact opposite transformations and at the end you have to end up in the starting point
- Compound transformation chains: e.g. two consecutive rotations around the same axis should be equal to one rotation around the same axis with the compounded magnitude, or more complex compound operations can be also defined.
- Compare against mock data originating from other systems: create a dataset in Matlab, Octave, a CAD Software package or any other trusted source which includes the origin the transformation and the transformed entity. Save these to a file, read them in for the test and compare the results. (you could do also connect to such software and execute the transinformation, but that would introduce unnecessary dependencies in your tests)
- Carefully hand-craft a "master" or "golden" dataset for validation, same as above, but the file is created manually.
Probably a combination of the above will lead to good results, but these test should be treated with an appropriate amount of scepticism.