The Castle, Rebuilding History