My question is: what is the difference between DC motor with encoder and DC without encoder? As long as I can control the speed of DC motor using PWM, for example on the Arduino, what is the fundamental difference?
As @szczepan mentions, the difference is one of feedback.
There are many ways to get feedback regarding the motion of a dc motor. People implementing a control system will often put a mark on the motor's shaft, or attach a "flag" of masking tape, so that they can visually see the turning of the shaft. This helps to ensure the direction of motion is correct (otherwise the dc polarity needs to be reversed). It also helps for observing motion at slow speeds - but is otherwise not appropriate for an automated system.
If you want to automate the observations of the motor's rotation, you must implement some type of sensor that provides appropriate information to the control computer. There are many ways to do this. For example, you can monitor the current being consumed by the motor's windings, and use the motor constant $k_\tau$, to infer the torque generated by the motor. This can be related to acceleration of the shaft by using an appropriate dynamic model of the system. This method, however, is not very accurate and is prone to modelling errors and signal noise. This method is similar to you monitoring the PWM output and inferring motion dynamics - neither is robust to changing dynamics of the system. Another approach is to glue a magnet to the shaft, and monitor it with a Hall effect sensor. This will provide a single pulse to the computer for each rotation of the shaft. This is frequently a solution for high-temperature, or dirty, environments (such as in automotive applications). However, often you need finer granularity of the motion. That is where encoders come in.
There are two basic types of encoders: incremental and absolute. They can be further characterized as quadrature, or non-quadrature encoders. A non-quadrature incremental encoder provides a single pulse to the controller for every incremental motion of the motor shaft. As the previous answer makes clear, this position feedback can be interpreted to infer velocity, acceleration, and possibly jerk (although three derivatives of a sensed value are "spikey" in most applications). This type of encoder, however, only provides information when the position changes, and it does not provide any information about the direction of motion. A quadrature encoder provides two pulses, out of phase, that can be used to detect direction also.
An absolute encoder can provide not only the same information as the incremental encoder does, but it also has many more bits of information from which you can know the angular position of the shaft, in addition to detecting the incremental changes in position.
You can make a very simple encoder by using a disc with slots cut into it. Place a photodiode on one side of the disc, and a photodetector on the opposite side. You will get one pulse each time a slot passes between the sensor elements. As you can see, the accuracy of motion detection is determined by the number of slots in the disc. Encoders are available with many different numbers of pulses per revolution.
I suggest reading a book such as this one if you want to know more about motion metrology:
Fundamental difference is closed loop feedback. Without encoder you just know how much speed you sent to engine, but have no information about speed that is on it - lower battery voltage will decrease your speed. Encoder gives you ability to measure engine
speed position, from which you can calculate speed, and regulate those parameters, for example with PID regulator. Usage example can be micromouse robots, where odometry is crucial in moving through maze.