If you assume the IMU lies on a fixed incline plane with pitch $\theta$, and you define yaw $\psi$ as the rotation between the IMU and the downhill direction, then the answer is
$$\psi = \text{atan2}(-a_y, a_x)$$
where $a_x$ and $a_y$ are the X and Y accelerometer readings respectively. Note that this is independent of the inclined plane pitch $\theta$ (which is not equal to the pitch as measured by the IMU; the IMU may be facing perpendicularly to the descent direction).
There are many definitions of yaw but the one we are using here is with respect to the downhill direction of the inclined plane. If that inclined plane were to rotate around the $z$ axis, the yaw as defined using e.g. Euler ZYX angles would change, but our yaw would stay the same. I suspect this is the disconnect between the question and the other answers. To be clear, this computation for $\psi$ is only the "local" yaw with respect to the incline. Getting the angle with respect to a fixed north or world $x$-axis given just an accelerometer reading and no other information is not possible. This is because gravity points down, and rotations around that axis (the down axis) will not cause a change in the accelerometer reading.
To derive this expression for $\psi$, note that the orientation of the IMU, $R$, in this inclined plane scenario only really has one degree of freedom: the yaw.
The orientation can be described by first pitching downwards by an angle of $\theta$, then rotating around the new body $z$-axis by $\psi$.
We can compute an expression for this orientation given $\theta$ and $\psi$ by multiplying an elementary rotation around the $y$ axis and an elementary rotation about the $z$ axis. See https://en.wikipedia.org/wiki/Rotation_matrix for their definitions. Since the rotation around $z$ is intrinsic (w.r.t. the new $z$-axis), we post multiply $R_y(\theta)$ with $R_z(\psi)$.
$$
R = R_y(\theta)R_z(\psi) = \begin{bmatrix} \cos\theta & 0 & \sin\theta \\ 0 & 1 & 0 \\ -\sin\theta & 0 & \cos\theta \end{bmatrix} \begin{bmatrix} \cos\psi & -\sin\psi & 0 \\ \sin\psi & \cos\psi & 0 \\ 0 & 0 & 1\end{bmatrix} = \begin{bmatrix} \cos\theta \cos\psi & - \cos\theta \sin\psi & \sin\theta \\ \sin\psi & \cos\psi & 0 \\ -\sin\theta \cos\psi & \sin\theta \sin\psi & \cos\theta \end{bmatrix}
$$
Now what does the accelerometer measure? It measures the gravity vector in the body frame*. The gravity vector in the world frame is $\begin{pmatrix} 0 & 0 & g \end{pmatrix}^\top$, so in the body frame, it is
$$ R^\top \begin{pmatrix} 0 \\ 0 \\ g \end{pmatrix} = \begin{pmatrix} -g\sin\theta \cos\psi \\ g\sin\theta \sin\psi \\ g\cos\theta \end{pmatrix} = \begin{pmatrix} a_x \\ a_y \\ a_z \end{pmatrix} $$
Now note that $$ -\frac{a_y}{a_x} = \tan \psi $$
and we have a simple way to compute $\psi$.
Note that when both $a_x$ and $a_y$ are zero, i.e. $\theta = 0$, we are not able to get any information about $\psi$. This corresponds to the intuition that on flat ground, the accelerometer cannot tell you about yaw.
*I am assuming there is no linear acceleration, which is a very common assumption. If the linear acceleration is known, it can be subtracted. For example, if this is a mobile robot, linear acceleration is typically close to zero or it can be estimated using the control inputs.