Google: Don't Worry Too Much About Page Speed

I see so many SEOs always cite the page speed of a site when it comes to why a site may not rank that well. Truth is, from what I understand, the only way page speed impacts a site's rankings in Google is if the site is unbearably slow...