Small buildings are more affected, or shaken, by high frequency waves (short and frequent).
For example, a small boat sailing in the ocean will not be greatly affected by a large swell. On the other hand several small waves in quick succession can overturn, or capsize, the boat.
In much the same way, a small building experiences more shaking by high frequency earthquake waves.
Tall High Rises:
Large structures or high rise buildings are more affected by long period, or slow shaking.
For instance, an ocean liner will experience little disturbance by short waves in quick succession. However, a large swell will significantly affect the ship.
Similarly, a skyscraper will sustain greater shaking by long period earthquake waves, than by the shorter waves.