How would you go about building a robot that can use a computer? Type on the keyboard, move & click mouse? I am talking about physically manipulating the hardware inputs, and the robot would be able to see the screen. Not connected to anything. It's purely autonomous. My hope is that this will replace human QA testers.

  • $\begingroup$ It would defeat the purpose of Quality Testing. You would not catch bugs in software or otherwise. Which is the entire purpose of quality testing. $\endgroup$
    – Naresh
    Jun 11 '13 at 7:31
  • $\begingroup$ not if you have a good AI. I don't see why not? QA testers are fat and lazy anyhow, and perform mundane repetitive tests that only require the usage of a small part of the brain that even pigeons use to distinguish specific colors. $\endgroup$ Jun 11 '13 at 7:32
  • $\begingroup$ We'd first need to solve the visual recognition task for the same. Which by itself is pretty difficult. It would be far easier in practice to simply write a script to do the boring work. $\endgroup$
    – Naresh
    Jun 11 '13 at 8:13
  • $\begingroup$ no my goal is to physically build a robot arm that can manipulate any device, and start testing it out, with the eventual long term goal of replacing QA testers, developers, designers, etc. $\endgroup$ Jun 11 '13 at 8:35
  • $\begingroup$ This doesn't answer your question exactly, but you can always have a software that captures the screen and controls the mouse and keyboard inputs. You don't need to build anything physical for that. $\endgroup$
    – Shahbaz
    Jun 11 '13 at 8:43

Your question contains a bunch of questions which i would try to reveal in the first place:

Q: What kind of drive would i need to solve the task of typing keyboards, clicking a mouse etc.?

A: Basically this is answered by your requirements of motion which are force, velocity, range to name the most important. Every type of drive has its own privileges according these requirements, but you totally have an eye on costs and your control as well. For these kind of tasks i would suggest a pneumatic drive for example. But it could be a combination of different types as well, e.g. wrist - electrical drive, finger - pneumatics and so on.

Q: What kind of control would i need to control these motions?

A: For this you have to consider several aspects such as complexity, your chosen drive, energy consumption (mobile systems!!) and costs as well. Hardware would not be a problem since embedded systems nowadays are quite powerful and performant. But in your case your control would contain closed loops since you need a feedback on force to not destroy your touchpad and keyboard. Which is in general quite complex and difficult to solve in realtime. If we are speaking of recognition by images etc., then hallelujah..

Q: Which tools do i need?

A: First of all a your well trained brain :) But in general you would start to desing a simulation for your drives and control to be sure everything COULD work out fine. Then you would need tools for design and layout. These tools are quite expensive as you may have guessed already..

Q: Where to start?

A: Think in modules. Start with the very basic part and easily spoken integrate all parts to subsystems, all subsystems to a whole system. But don't forget to run tests / simulations on every level! Every change afterwards will exploder your time (and cost) consumption exponential!

  • $\begingroup$ this is a good start but too vaguely worded maybe some starting reference links would help. I know this is a radically new way of thinking about the problem but I'm confident that in the next 20 years, QA testers will be completely replaced by robots (innovative destruction my Econ prof called it). $\endgroup$ Jun 11 '13 at 18:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.