I checked the oil level and it's actually topped off already. I'm pretty sure the oil sensor is able to sense the oil since I wasn't really going too fast and the light also came on when I was reversing into my garage at low speed, so the oil moving around in my oil pan shouldn't be the problem.
However I just read on a couple other forums and it seems like this is a common problem for the E46. The problem is no one really knows the real cause of this.