>>352972471
Oh god I found a bug report for the issue and it isn't fixed until a 2021 version. Material swaps with element 2 and 4 are bugged.

https://issuetracker.unity3d.com/issues/material-is-applied-to-two-slots-when-applying-material-to-a-single-slot-while-recording-animation