Choosing the correct prop/motor combination can be challenging. Even worse, the vast majority of RC motors will have vague or non-existant specifications. This makes it very hard to do "real" math. Instead you will have to rely on prop thrust ratings or use a known combination that other people have tested.
Propellers:
Manueverability of a quad is dependent on how quickly you can change the thrust (how fast you can change the RPM of the props).
A smaller prop is easier to speed up and slow down whereas a large prop takes a very long time to change speeds. Beyond a certain size you essentially are unable to fly. This is why you see hexacoptors and octocopters that use more, smaller props instead of giant quadrotors.
On the flipside, propeller efficiency is related to the area of a prop (or radius squared) so a small increase in prop diameter will make the quad vastly more efficient.
The faster you want to go, the more aggressive a pitch you want. Since quadrotors are generally hovering this means you want the lowest pitch available. If you want to go somewhere fast a higher pitch might be apropriate.
Motors:
Generally you just choose a motor that matches the given prop. Having a motor that is too large will be heavy, and a motor that is too small will perform poorly or burn up.
A smaller prop requires a higher RPM motor because they must spin faster to generate equivalent lift.
If you put everything together, an efficient quad will have properly sized, low RPM motors with very large props.
An acrobatic quad will want smaller, more aggressive props, faster motors, and you might even overvolt the motors for faster response.
While you will be hard pressed to find torque curves for Rc motors, you will have thrust and current ratings for a given prop which makes it fairly straightforward to determine how much you can lift and approximately how long you can fly once you have chosen a combinatoin.
As a general rule you want a 2:1 thrust/weight ratio for a standard quad and 3:1 or higher for something acrobatic. 1.5:1 has been known to fly.