# Cloth (non-rigid body) manipulation with computer vision? [closed]

I am currently working on a prototype device to automate cloth ironing task. So basically the mechanical design involves 4 manipulators and a separate manipulator arm with a hot pad (iron) to manipulate a given cloth to perform the ironing task. I understand its a very complicated task and I have come up with the following steps to potentially go about solving this problem. I would appreciate any kind input on the plausibility of this idea (mechanical and software challenges).

For my first EB (Engineering build), I'm solely focusing on solving this problem for a T-shirt, probably extend this solution to other cloth types with modifications.

1. From a pile of clothes, choose a certain cloth for ironing using background subtraction and colour based image segmentation (already programmed this part and its working!)

2. Classify or recognise the chosen cloth into one of any known classes using CNN (Convoluted neural networks). I personally think this part is quite achievable given that I have enough training data to train my network.

3. This is where the problem begins!. After classifying the given cloth, find coordinates of points of interest to manipulate the cloth in a certain 'predetermined' mechanical movement to facilitate the manipulation of cloth. What I mean by points of interest here is, for example, in the case of a T-shirt (see image below); the algorithm should localise the locations the following parts 1) Midpoint of T-shirt hem (see image for details) 2) Corner point of T-shirt hem (see image for details) 2) Start of yoke 3) End of the yoke

I believe (through my mental visualisation) that these are the key points to place the T-shirt flat on the surface (a known STATE). When the T-shirt is flat on the surface, it could be stretched just enough to make the cloth wrinkle free to facilitate ironing through some kind of force sensing mechanism (Haven't really though about it, but I presume its not huge challenge).

1. When the cloth is in a known STATE, proceed to ironing using path planning algorithms.

I do understand that this is a very complex problem, just want to get some inputs to see if I am on the right path. I am really not sure if STEP 4 is mechanically achievable since non-rigid body manipulation is a huge problem for robots. I am quite confident on the software aspect of the project, I believe a large enough data set and well tested algorithm will help me identify the afore-mentioned points of interest in a test cloth. But I would like hear the mechanical challenges and plausibility of this project from design engineers standpoint. Hope I'm clear about my query.

Any input is appreciated. Thank you so much and cheers :)

• Welcome to Robotics, VinokanthVelu. Unfortunately, questions like this fall into a group of questions that are "open-ended design questions." Question about how to accomplish or achieve something don't have a clear, concise answer because there are so many factors to consider. Are all the shirts you are going to work with uniform in nature? Are they all T-shirts, as you've depicted, or do you need to work with buttoned shirts as well? Do any have pockets? What material are they made of? The list goes on. – Chuck Jan 28 '17 at 16:19
• Generally, when you get to a high-level flow chart, ironing a shirt really is just find the shirt, ID the shirt, lay it flat, then iron it. The problem you're going to face is that the shirts are in a pile. What if it's inside-out? Finding the seams may or may not be terribly difficult, if you have a good enough algorithm with enough test data. That is really what all your "points of interest" are - side seam, hem seam, collar seam, etc. For manipulating the fabric, you need to pull hard enough to get the wrinkles out, but not so hard you tear or stretch it. Look up compliant control. – Chuck Jan 28 '17 at 16:19
• So, like I mention above, your question, as it stands, is currently off-topic. If you'd like to chat about your project, please join us in Robotics Chat. If you could narrow this down from, "Am I on the right track?" to something that is specific and answerable, then please edit your question to reflect that. As I said, generally speaking, yes you find the shirt, lay it flat, and iron it. When you have specific problems trying to implement that logic, those are the questions suitable for the Q&A format. – Chuck Jan 28 '17 at 16:22
• @Chuck Hello bro, thank you so much for your kind and insightful comment. As you advised, I have modified the question with sufficient details. Thank you :) – Vino Jan 29 '17 at 8:30