I was talking to a tech on twitter yesterday and he said to post here. He tested my equipment, and I've done all the normal boring steps like running speed tests, resetting the modem, my router/firewall, connecting directly ETC.
The problems are pretty sporadic, sometimes on youtube I can do HD for a little while during the day, and other times I'm lucky if I get 360p. Twitch will sputter and buffer as well. Speed tests appear fine, ping times to various servers are usually fine, though a ping test to my web server hosted in SFO are a little high at >100ms. Ideally, SFO to Edmonton should be 16ms, so something around 40-60ms would be acceptable to account for longer distances due major POPs and backhaul networks not being in ideal locations so 100+ ms isn't really great. A friend of mine that lives maybe a 10-15min drive away seems to be having similar issues, but worse!
Basically it started to happen a bit after I upgraded to Wide Open Internet 150 and got a new Hitron modem.
Not entirely sure what could be causing it. I started writing this post because it started acting up again, but now as I get to the end here, things have improved again, but I'm sure will get bad again.