The clip below discusses why it is so disorienting to go into one of those rooms that are just like normal rooms, except that they are tilted. Even being aware of the source of the effect does not prevent people from feeling a strong sense of discomfort, even nausea.
The problem is that we cannot just take in external sense data. We also impose upon it our prior knowledge and expectations of how things should look and behave and when there is a mismatch between the two, this causes problems.
Over time, we may be able to reconcile the two. There was a famous experiment done over fifty years ago (Thomas Kuhn refers to it in his book The Structure of Scientific Revolutions) in which people were outfitted with glasses that made everything look upside down. Of course, they could barely function at the beginning. But over time, as they got used to the new reality and created a new set of expectations, they became able to act almost normally.