Yes, wear your retainers or else they'll shift pretty quickly.
After I finished my treatment, I only wore my retainer trays while I sleep. Terrible, I know. It's been over a year, and my teeth still shift like crazy. If I forget to wear my retainer for a night, you can feel a lot of pressure in your gums. You can imagine how bad it was when I forgot to bring the tray with me on a 1 week calgary vacation.
I was never able to conquer the lisp while wearing my tray