Return to Answer

tl;dr
You made a bunch of noob errors because you suck at electronics right now.

Full answer
Conceptually you had everything set up correctly, but you made a lot of basic mistakes that caused the signal to go fubar. Here is the setup you should use to record the signal from one of the encoder outputs:

If you set it up exactly this way, without any errors, you should get a clean signal if your motor/encoder is not broken. Once it's up and running, seeing the signal is simple. It's as easy as checking the square wave coming off of any common function generator: fiddle with your oscilloscope until it looks good (or just hit autoset and any good oscilloscope will do it for you, or read the excellent post by Chuck for more help on that).

Your problems seemed to stem from an accumulation of rust when it comes to dealing with real, wire and breadboard, electronics projects:

1. You forgot that the top/bottom of the breadboard power rails are typically separated, assuming continuity between them. Always keep a multimeter handy and do simple continuity tests before doing more esoteric troubleshooting. When stuff's busted, never assume continuity. Test for continuity.
2. The fact that the green wire from the motor is ground confused you, and you swapped your connections to the green and blue jacks (encoder Gnd and encoder Vcc) on the motor, so this trashed your signal. Checking that your wires are connected correctly is pretty much the electronics equivalent of "Is your computer plugged in?". Just like you should always make sure your computer is plugged in and turned on, in the world of circuits, always make sure your wires are attached in the correct spots (especially if you are prone to get disoriented when using non-black ground wires).

Conceptually everything was set up correctly, but a number of basic mistakes affected the signal. Here is the set-up which should be used to record the signal from one of the encoder outputs:

A set up like this should result in a a clean signal if your motor/encoder is not broken. Once it's up and running, seeing the signal is simple. It's as easy as checking the square wave coming off of any common function generator: fiddle with your oscilloscope until it looks good (or just hit autoset and any good oscilloscope will do it for you, or read the excellent answer by Chuck for more help on that).

The underlying problems seem to stem from an accumulation of rust when it comes to dealing with real, wire and breadboard, electronics projects:

1. The top/bottom breadboard power rails are typically separated, so you cannot assume continuity between them. Always keep a multimeter handy and do simple continuity tests before doing deeper troubleshooting. When stuff's busted, never assume continuity. Test for continuity.
2. Be careful not to confuse motor ground with encoder ground, motor ground is likely to be electrically very noisy,m which will corrupt your encoder signal.

tl;dr
You made a bunch of noob errors because you suck at electronics.

Full answer
Conceptually you had everything set up correctly, but you made a lot of basic mistakes that caused the signal to go fubar. Here is the setup you should use to record the signal from one of the encoder outputs:

If you set it up exactly this way, without any errors, you should get a clean signal if your motor/encoder is not broken. Once it's up and running, seeing the signal is simple. It's as easy as checking the square wave coming off of any common function generator: fiddle with your oscilloscope until it looks good (or just hit autoset and any good oscilloscope will do it for you, or read the excellent post by Chuck for more help on that).

Your problems seemed to stem from an accumulation of rust when it comes to dealing with real, wire and breadboard, electronics projects:

1. You forgot that the top/bottom of the breadboard power rails are typically separated, assuming continuity between them. Always keep a multimeter handy and do simple continuity tests before doing more esoteric troubleshooting. When stuff's busted, never assume continuity. Test for continuity.
2. The fact that the green wire from the motor is ground confused you, and you swapped your connections to the green and blue jacks (encoder Gnd and encoder Vcc) on the motor, so this trashed your signal. Checking that your wires are connected correctly is pretty much the electronics equivalent of "Is your computer plugged in?". Just like you should always make sure your computer is plugged in and turned on, in the world of circuits, always make sure your wires are attached in the correct spots (especially if you are prone to get disoriented when using non-black ground wires).

Basically, I'm embarrassed and would prefer to have deleted the question than write this answer. But screw it, here it is. I'm rusty and I blew it.

tl;dr
You made a bunch of noob errors because you suck at electronics right now.

Full answer
Conceptually you had everything set up correctly, but you made a lot of basic mistakes that caused the signal to go fubar. Here is the setup you should use to record the signal from one of the encoder outputs:

If you set it up exactly this way, without any errors, you should get a clean signal if your motor/encoder is not broken. Once it's up and running, seeing the signal is simple. It's as easy as checking the square wave coming off of any common function generator: fiddle with your oscilloscope until it looks good (or just hit autoset and any good oscilloscope will do it for you, or read the excellent post by Chuck for more help on that).

Your problems seemed to stem from an accumulation of rust when it comes to dealing with real, wire and breadboard, electronics projects:

1. You forgot that the top/bottom of the breadboard power rails are typically separated, assuming continuity between them. Always keep a multimeter handy and do simple continuity tests before doing more esoteric troubleshooting. When stuff's busted, never assume continuity. Test for continuity.
2. The fact that the green wire from the motor is ground confused you, and you swapped your connections to the green and blue jacks (encoder Gnd and encoder Vcc) on the motor, so this trashed your signal. Checking that your wires are connected correctly is pretty much the electronics equivalent of "Is your computer plugged in?". Just like you should always make sure your computer is plugged in and turned on, in the world of circuits, always make sure your wires are attached in the correct spots (especially if you are prone to get disoriented when using non-black ground wires).

Conceptually you had everything set up correctly, but you made a lot of basic mistakes that caused the signal to go fubar. Here is the setup you should use to record the signal from one of the encoder outputs:

If you set it up exactly this way, without any errors, you should get a clean signal if your motor/encoder is not broken. Once it's up and running, seeing the signal is simple. It's as easy as checking the square wave coming off of any common function generator: fiddle with your oscilloscope until it looks good (or just hit autoset and any good oscilloscope will do it for you, or read the excellent post by Chuck for more help on that).

Your problems seemed to stem from an accumulation of rust in fiddling with real electronics:

1. You forgot that the top/bottom of the breadboard power rails are typically separated, and expected them to be connected to one another. Always keep a multimeter handy and do simple continuity tests before doing more esoteric troubleshooting. When stuff's busted, never assume continuity. Test for continuity.
2. The fact that the green wire from the motor is ground confused you, and you swapped your connections to the green and blue jacks (encoder Gnd and encoder Vcc) on the motor, so this trashed your signal. Checking that your wires are connected correctly is pretty much the electronics equivalent of "Is your computer plugged in?". Always make sure your computer is plugged in and turned on. For circuits, always make sure your wires are attached, and attached where they are supposed to be attached (especially if you are prone to get confused when using non-black ground wires).

tl;dr
You made a bunch of noob errors because you suck at electronics.

Full answer
Conceptually you had everything set up correctly, but you made a lot of basic mistakes that caused the signal to go fubar. Here is the setup you should use to record the signal from one of the encoder outputs:

If you set it up exactly this way, without any errors, you should get a clean signal if your motor/encoder is not broken. Once it's up and running, seeing the signal is simple. It's as easy as checking the square wave coming off of any common function generator: fiddle with your oscilloscope until it looks good (or just hit autoset and any good oscilloscope will do it for you, or read the excellent post by Chuck for more help on that).

Your problems seemed to stem from an accumulation of rust when it comes to dealing with real, wire and breadboard, electronics projects:

1. You forgot that the top/bottom of the breadboard power rails are typically separated, assuming continuity between them. Always keep a multimeter handy and do simple continuity tests before doing more esoteric troubleshooting. When stuff's busted, never assume continuity. Test for continuity.
2. The fact that the green wire from the motor is ground confused you, and you swapped your connections to the green and blue jacks (encoder Gnd and encoder Vcc) on the motor, so this trashed your signal. Checking that your wires are connected correctly is pretty much the electronics equivalent of "Is your computer plugged in?". Just like you should always make sure your computer is plugged in and turned on, in the world of circuits, always make sure your wires are attached in the correct spots (especially if you are prone to get disoriented when using non-black ground wires).

Basically, I'm embarrassed and would prefer to have deleted the question than write this answer. But screw it, here it is. I'm rusty and I blew it.