Because, if there aren't, he doesn't speak for his own party, let alone the House of Representatives, and shouldn't be the Speaker of the House.
Maybe someone should call Jim Ryun, Jerry Moran and Todd Tiahrt to see what they think of that sort of reckless talk.
This isn't just partisan politics. The Speaker is 3rd in line for the presidency. That means that he would be making speeches like this, and making decisions about these matters, if a crisis struck the nation and killed the President and Vice-President. For instance, if they were in San Francisco when an earthquake hit. We wouldn't want him to take the oath of office and start musing about the prudence of rebuilding California's greatest city.
This is a time when we have to think about issues of succession, and whether we don't want someone a little more responsible to be #3. Maybe someone who wouldn't rather be shopping for antique cars or glad-handing contributors than voting on a plan to provide relief to hurricane stricken Gulf.