Lots of optimistic views on the future hinge on us getting dramatically more power through superhuman AI. But power that can make things exponentially better can also make things exponentially worse.
Anyone who sees superhuman AI helping solve aging has to also acknowledge that it could also be used to create the deadliest bioweapon ever. Power to do great things is also power to do terrible things.
Also more power makes even accidental mistakes more damaging. Hitting a bump at 10mph is no big deal, but hitting a bump at 100mph is a disaster.