How would you go about building a robot that can use a computer? Type on the keyboard, move & click mouse? I am talking about physically manipulating the hardware inputs, and the robot would be able to see the screen. Not connected to anything. It's purely autonomous. My hope is that this will replace human QA testers.
Your question contains a bunch of questions which i would try to reveal in the first place:
Q: What kind of drive would i need to solve the task of typing keyboards, clicking a mouse etc.?
A: Basically this is answered by your requirements of motion which are force, velocity, range to name the most important. Every type of drive has its own privileges according these requirements, but you totally have an eye on costs and your control as well. For these kind of tasks i would suggest a pneumatic drive for example. But it could be a combination of different types as well, e.g. wrist - electrical drive, finger - pneumatics and so on.
Q: What kind of control would i need to control these motions?
A: For this you have to consider several aspects such as complexity, your chosen drive, energy consumption (mobile systems!!) and costs as well. Hardware would not be a problem since embedded systems nowadays are quite powerful and performant. But in your case your control would contain closed loops since you need a feedback on force to not destroy your touchpad and keyboard. Which is in general quite complex and difficult to solve in realtime. If we are speaking of recognition by images etc., then hallelujah..
Q: Which tools do i need?
A: First of all a your well trained brain :) But in general you would start to desing a simulation for your drives and control to be sure everything COULD work out fine. Then you would need tools for design and layout. These tools are quite expensive as you may have guessed already..
Q: Where to start?
A: Think in modules. Start with the very basic part and easily spoken integrate all parts to subsystems, all subsystems to a whole system. But don't forget to run tests / simulations on every level! Every change afterwards will exploder your time (and cost) consumption exponential!