The lenses work both ways, that's why you don't want to get sunlight on them. 

Think about it, the lenses focus the light emitted from the displays, down to a point which enters your eye, they do the same in reverse, focusing the suns light into a point on the display, which can cause heat damage.