You are basically describing an object 3D-reconstruction device. Though usually they use a turntable and fixed camera, but the principle is the exact same. See how this dataset were created. I also know for sure that people sometimes instead use a robotic arm rather than a turntable, so your idea is definitely feasible. The field you want to look into is called multi-view stereo.
I am just going to assume you can find a working multi-view stereo algorithm. So you need to find a way to estimate the positions between the cameras. A Mono-VIO algorithm would work, but is probably not the best solution. One problem is that many Mono-VIO algorithms assume they are always moving to a new area. Basically it forgets past data, because it is irrelevant to the current position. However, in your case the camera is moving in a limited area(the span of the table). So you would you actually want your algorithm to store all the data related to the table, and just use that for pose estimation. They also tend to drift over time, which is again related to not using past data. If you are familiar with SLAM/VIO terms, you want your algorithm to always be doing localization, rather than odometry.
If you do decide to still go the Mono-VIO route then I would recommend ORB-SLAM3. It actually does do localization, so it should hopefully mitigate the drifting problem. However, it would still probably do a large amount of irrelevant work.
As I said before this is actually a pretty common sort of device to build. So you do have other potential options.
Use the robot arm positioning system. If you have a high quality robot arm with good encoders, then it will know how far it moves better than any Mono-VIO system can estimate. You need to calibrate your camera/robot arm together using Hand-eye calibration software, and now when your robot moves you also know how far the camera moves.
Use fiduciary markers like aruco/April tags. Basically put a couple of markers around the table. Your camera can then estimate its position relative to those markers.
Build custom VIO/SLAM system that does more relocalization, and is more customized to your specific problem.
To answer your questions:
Is it possible to get depth information with metric scaling about the
objects lying on my desk from the received data of the Mono-VIO
Can someone tell me if this works or if the drift is too high or the accuracy is too low.
Could work with the right Mono-VIO algorithm. A specialized version ,however, would be better.
Overall I would recommend options 1 or 2. Option 3 or an off the shelf Mono-VIO algorithm should only be attempted if you have to.