How is trajectory planning normally carried out in off-road/unstructured environments? In particular I'm interested in how stereo vision can be used to plot trajectories over obstacles - for example, how does one generate ground planes when the environment is extremely rough and the robot is sitting at an angle?
In general, you want to build a 3D map of the environment, or more likely an approximation of a 3D map. Typically such maps are grid-based across the horizontal plane, and each cell contains some additional information like height (see '2D occupancy maps', 'digital elevation models' and '2.5D grid maps' for more information). They often focus on geometric attributes of the environment, but may also include things like appearance (from which you may be able to infer other properties). These types of maps can be created from stereo camera data.
Then in order to be able to path plan you need some way of quantifying how good/bad a given path is. I suggest you look for research around 'traversibility' for various methods of evaluating this for some given terrain but as a basic idea, the 'cost' of traversing (or moving into) some area may be calculated based on its height, gradient, roughness, surface material, difference vs neighbouring areas, some properties in relation to specific vehicle attributes etc. Once you have a map containing this information, you can apply common path planning techniques e.g. A* on a grid based map. In rougher or more unstructured environments, there may be more unknown information e.g. the vehicle can't see over a ridge, or behind an obstacle, or its field of view might not include ground that contains holes ('negative obstacle'), so vehicle movement will need to take these into account - both in terms of being able to create a useful and accurate map by observing enough of the environment, and also by avoiding potentially dangerous areas.
for example, how does one generate ground planes when the environment is extremely rough and the robot is sitting at an angle?
To address this specifically, you don't. You can instead use a different approach like the one described above which does not involve ground planes. Though in my (perhaps limited) experience, a lot of rough and unstructured environments featured in robotics research usually aren't really that rough, and often still have significant and relatively flat (compared to the scale of the vehicle) ground planes.
Short answer: At first, a motion capture recording of the robot is created. Secondly, the recording is converted into a task model.
Long answer: A human operator is the natural source for providing high quality control signals. A well trained operator is able to let the robot solve difficult situations. The only problem is, that the actions of a human operator are not formalized in software code, so that a planner can't use the existing heuristics. To solve the issue, a motion capture device is used which records a set of features including the actions of the human operator. This recording is converted into a prediction model which can be used by a motion solver autonomously.