The Cleckheaton Lawson Tweed used in this pattern calls for a size 5mm hook, but I found the 4mm to be perfectly suitable. If using another yarn, find one that requires a 5mm hook and use a 4mm hook.

Medium - 4mm
Large - 4.5mm
XLarge - 5mm

Instead of using a 3 or 3.5 for a smaller hat, try using a thinner ply wool, perhaps one that requires a 4mm hook (such as 8ply).