For quadcopter designs, you generally want to have around a 2:1 thrust-to-weight ratio. That is, at 100% throttle, you want your combined propeller thrust to be capable of lifting two times the weight of the craft.
You then need to determine the amount of power the motors need to generate that thrust (usually given on the datasheet). When you have that amount of power, you can easily calculate the amount of time your battery will last.
This page on Quadcopter Performance gives a good example:
Looking at the data from the motor performance chart, it looks like
the absolute maximum weight the motors can support is 560 each, so
2240 grams. This is when the motors are working at 100%, using 95.2
watts, which means about 8.6 amps at 11.1 volts.
Looking at the data from the flight weight vs power and battery life
graph, it seems that the helicopter should not exceed 61 ounces (1700
grams). At that weight, 61 watts of power is used, which is 5.5 amps
at 11.1 volts.
The graph also shows that one ounce of flight weight would need one
watt of power, and the two are linearly correlated. Assuming the craft
is 60 ounces, 60 watts of power is needed.
This quote is a bit misleading though. The original author is missing the words "per motor" in their conclusion. 230W/4 motors = 57.5 Watts. That's where they got the "60 ounces needs 60 Watts (per motor)" conclusion. However the general method of plotting performance is beneficial to answering the question.
If you increase the maximum thrust-to-weight ratio, you can keep your throttle lower, use less power, and stay in the air longer. So you can either increase thrust or decrease weight to achieve longer flight times.
To more concisely answer your question, the point of diminishing returns comes when the additional weight added by the battery brings your thrust-to-weight ratio below 1.5:1 on the low end.
Something else to consider is the energy density of your battery technology. How much power per unit weight the battery is capable of. You're probably using Lithium-ion batteries which are probably the best available from a price to performance perspective. But if you're really trying to get longer flight times without increasing weight, you can consider some of the more esoteric (and much more expensive) technologies out there.
And it's probably worth mentioning that even within the Lithium-ion category, not all batteries are created equal.