I'm designing a PMSM controller with the motor has an optical encoder, so I can get position information from the motor. But how to measure the velocity of the motor. Now, my design is to calculate the difference, then add a low pass filter to estimate velocity. The approach is not a good way when the motor runs at a low speed. I have read some research to improve velocity estimation, like Luenberger Observer, Extended State Observer, or Kalman filter. But I don't know what should I choose since I have almost no theoretical basis.
You could employ all the techniques you mentioned that are model-based though, meaning you need to provide a good model of your system and then tune the parameters.
Alternatively, there are other methodologies needing less tuning. Among all these, one of the best technique to use in this situation is definitely the Savitzky-Golay filtering, which performs least-squares fitting over a given time window.
In a nutshell, you have to fit in the least-squares sense the position samples contained within a temporal moving window using a line. The coefficient of this line represents the velocity.
In slow velocity error will be more since you are PWM mode. Try to use one Schmidt trigger circuit followed by a counter t measure speed. No need to use even filter.