As @jsotola guessed in the comments, it is possible you have magnetic interference between your motors and hall effect encoders. This exact thing happened to me on a very compact electo-mechanical design for a robot hand seen here. If I remember correctly, we wanted to keep our magnetic encoders so we moved the motors farther away. But you might be able to switch to an optical encoder.
The other possibility is to switch to a full quadrature encoder. Most encoders already do this, but some very cheap encoders (like on the Roomba Create) are only half-quadrature. With full quadrature, you should be able to determine the direction of travel from the signals, so any vibration will only make it oscillate between 2 values, not increase indefinitely. With half-quadrature, it is only a square-wave and you have to assume a direction of travel, so noise might make the values increase as you said.