I want to determine Jacobian determinant of spherical wrist structure, but my Jacobian is 6x3, so it is not square. How can I get it?
If $J(q)$ isn't square, you're looking for some alternate metric to take its place. You can take the determinant of $J(q)J^T(q)$. per this tutorial or any number of other pseudo-inverse methods (from Introduction to Inverse Kinematics with Jacobian Transpose, Pseudoinverse and Damped Least Squares methods, by Samuel Buss).
As highlighted by @SteveO and @PetchPuttichai in the comments, it would appear that the $J(q)J^T(q)$ only when $J(q)$ has full row rank. In any other case, use $J^T(q)J(q)$.