>>371500757
please use primitive colliders if you can help it. 128 tris isn't too bad but mesh colliders (especially non-convex ones) have... issues.
consider using a mesh collider for the walls only, not for the floor.