Don't feel claustrophobic desu.
I also removed the mesh renderer and just added a mesh collider on the collider mesh, but my avatar was doing small jumps at all times. Using just the normal mesh worked well thou, so I probably run with that.
Now to make prefabs that I can decorate my map with!