Hi there Matt,
There are several threads here and on Linked in about this already.
I will quote one of the key things Google said that people have glossed over or omitted:
"For now it's only a very lightweight signal - affecting fewer than 1% of global queries, and carrying less weight than other signals such as high-quality content - while we give webmasters time to switch to HTTPS," Google's Zineb Ait Bahajji and Gary Illyes said in the blog post.
"But over time, we may decide to strengthen it, because we'd like to encourage all website owners to switch from HTTP to HTTPS to keep everyone safe on the web."
So right now - Its not a factor that will ruin your ranking.
Note how the quote has words like "MAY" decide etc.