They are usually built so that the swaying is damped by the building. For very large ones there’s a pendulum or a similar device that absorbs the swimming. It’s the opposite of how you play with a swing (the playground toy) and try to go faster – instead you try to stop yourself from swinging as much as possible (without skidding your feet on the ground, because buildings don’t have that option.)
The shape us also made so that the resonance frequency (the frequency at which the swaying is most impactful) is different from typical swaying in a building. Resonance is also well understood by the swing-example, it’s the rhythm your parents/friends/siblings get into when shoving you to go faster.