Each round of Knock Your Blocks Off takes place in two phases, the first with players building up a structure and the second with players trying to knock the crown off an opponent's building.

In the first phase – the Building Phase – players roll their six colored blocks simultaneously, then race to build one of the five approved structures: a six-block tall tower, a six-block long wall, a 2x3 fort, a 3x2 wall or steps. The blocks are part white, part colored, and on the face visible to the blocks' owner, the colors of adjacent blocks must match. Once a structure is finished, a player caps the structure with his crown block. The first player to complete his building grabs the demolition die from the table and (if he built a wall, fort or steps) receives a victory token.

In the second phase – the Destruction Phase – the player with the die chooses a neighboring player, then rolls the die to see whether he attacks like an Ogre (by tossing the die underhanded at that opponent's building), like a Dragon (by dropping the die straight down on that building), or with a boulder (by placing the die on the table, then flicking it at that building). If the attack dislodges that building's crown, the attacker wins a victory token; if not, the defender wins a victory token. Each type of building provides its owner a special power: Forts are immune to boulders, Towers earn an extra victory token when defended, and so on. The attacked player then attacks someone else until everyone has attacked and been attacked once, then a new round begins.

As soon as someone claims his eighth victory token, he wins the game. For a more difficult game, or to balance play between adults and children, the minimum distance of an attack can be increased from one foot to two feet.