Leaving Twitch
In February of 2021 I resigned from Twitch after 6 years of employment and joining Discord tomorrow after writing this post. It’s scary leaving any company where you spent 1/6th of your life inside of. The question has come up enough times now about why I’m leaving and to explain that I first have to talk about how it all started.
To understand why I joined Twitch I need to start with how I met Trance. Cristian Tamas, also known as “Trance” on the internet was one of the founders of “The GD Studio”, which back in 2012 was one of the few influential broadcasters within the Starcraft 2 esport scene. I got introduced to Trance through my work at DreamHack and helped him and James “2GD” Harding with some of the shows up until 2014. Trance was one of the first employees of Twitch in Europe with the intent of acting as a tech/BD arm in Europe, as esport was heavily European at the time. In 2014, the studio was awarded the contract from Valve Software to produce a “pre-show” to their DOTA2 tournament with $10 million USD in prize pool that would stretch over 2 weeks. The GD Studio’s idea here was to produce a 24/7 show, flying in all the talent to their house up in Tyresö, Stockholm.
Trance reached out to me to help assemble the production behind the show, as there were tons of parts the studio needed to make this concept work. Although the budget was significant for the studio, they had somehow still managed to bite more than they could chew. Apart from shuttling 20-30 people from Stockholm to Lidingö at any hours of the day they had to cook, commentate games, entertain the people and somehow also produce a show. On top of this they also had 10 newborn kittens behind the sofa. In reality I have no idea how they pulled it off but the technical architecture of the show worked and I became good friends with Trance as a result of this event.
I ended up leaving my job at DreamHack in September to pursue more freelancing gigs similar to the one above. Kept talking to Trance a lot and helping him out with streaming configurations and whatnot. In November of 2014 I got a call from a man named Cyrus Hall in the middle of the night. A recruiter had reached out a while earlier and asked if I was interested in working for Twitch through recommendation from Trance and I said sure, knowing that nothing would come of it. Cyrus had scheduled to talk to me for about an hour but we ended up spending closer to 2-3 hours on the phone, talking about networks, video and my general thinking about where esport broadcasting was headed. It’s hard to remember today when esport broadcasting often rivals or beats traditional broadcasting what scrappy operation it used to be and I felt that my role in all this was to help the esport orgs with the broadcast knowledge I had. At the same time I stood on the other side of the fence, communicating from the esport orgs to the streaming platforms about the requirements that a “broadcast” has in comparison to UGC (user generated content) gaming stream.
In February of 2015 I accepted the offer to join Twitch to help them figure out how to make esport broadcasters happy from a technical standpoint. As both esport and Twitch grew at a rapid pace, the scope of my work expanded quicker than I was able to personally tackle, which had me build up a team with people that could help me build the relationships and architect the tooling. There were a lot of onsite events and meetings at this period in life, mostly to build a relation with the organisations on the other end and understand their problems.
One needs to mention all the great colleagues I met at Twitch. Since Twitch encompassed my sober year and I had a lot of time on my hands, I took up playing DOTA2 with coworkers from Twitch while visiting San Francisco. Staying in the office until 04.00 in the morning while bashing out game after game, dreading the meeting starting in less than 5 hours or the flight to some random location. Had the absolute pleasure to meet one of the best directors (now VP) who served partly as my boss but more as my mentor and great friend. If it wasn’t for this individual I wouldn’t be at Twitch past 2016 when he flew out to Sweden. Me and Trance took him to the Twitch afterparty in Jönköping and while I don’t have a great picture of him, I saved this gem taken at 5AM with Trance.
How the sausage is made
Everyone who has met me knows that I never shy away from solving time sensitive problems. It’s a skill that has come from my years in working with live shows & touring gigs and broadcast. One gets used to the high pressure and develops systematic ways of approaching situations like that after a while. That’s the same reason why first responders can act swift but stay calm, they’ve practiced it so many times that it’s not foreign. The same goes for working with live shows, the tone is often tense and agressive but people are only interested in resolving issues as fast as possible.
Of all the weird things that happened over my time at Twitch, the best “how the sausage is made” moment I experienced was during E3 2018. During one keynote that was broadcasted on Twitch with a large number of viewers, the feed started exhibiting visual artifacts. Since the viewership on Twitch was significant, people started getting really worked up about this issue, as viewers were spamming the chat about it. I was in the production studio for another reason and got asked by an engineer if I had any idea what could be going on. In broadcast you don’t get to redo. There is no “let’s fix that one for next time” when you have a show that runs once a year with massive viewership.
When approaching broadcast problems I’ve found that it helps to always approach it “left to right”. As in you start with verifying the absolute source and then work your way “to the right” as in down the chain to figure out where the issue is introduced. Baseband video is funny in that way that you can essentially tap the feed at any given point and view it without having to do complex network routing or subscribing to a livestream, just route the matrix to an SDI output and watch the source. This goes against the common debug strategy that people often take, working right to left from where the problem is exhibited towards where the problem isn’t visible any longer. My problem with this approach is while it seems good on paper it fails to take all the branching patterns a signal can take which makes it hard to easily trace backwards. After stepping through the signals it was clear that the issue happened before entering the SDI matrix. Tracing the cable backwards I found that the actual input source was looped through a scan converter (broadcast term for video scaler). The actual fiber feed into the building didn’t have these artifacts and they used the scan converter to split the signal into primary/backup. In this case, both were affected.
To solve this I realized that we needed to replace the scan converter that drove the entire feed for the studio (single point of failure), mid-show, without affecting the video. On top of this we had about 3 minutes to solve it. I realized that the content that was shown at the time was game trailers for up and coming games, all ending on a black slate for about a second. If we could time the switchover to occur exactly at that second, no one would notice.
Doing this however required three physical cables to actually be switched in less than a second. The fiber input needed to be switched to a new scan converter and the lines going out from the old scan converter had to be swapped into the new scan converter. Here I also realized that the feed we got had about 1.2 seconds of latency, meaning that if I dialed into the production bridge of the company who’s feed we were tapping, heard their countdown, we would be able to monkey patch it and notify our production exactly in time.
Here I am, sitting on the floor while waiting to switch the SDI cable. Once we felt that it was time, I notified our production that it was going to happen in less than 30 seconds and waited for the cue on my other headphones. The switch went flawless, the new scan converter locked onto the signal and no one watching saw the switch happen. Of course this was dumb and the show should never had split the signal in that way but this is sadly the way it goes a lot of the times in broadcast. It’s a balance between budget and time and often the right decisions gets made but on the wrong risk profile.
Moving on
Twitch realized that we had built one of the worlds best platforms for live video on the internet and went on to build Amazon IVS which I was lucky to be part of. This was a big undertaking and has taken a lot of my focus for the past years, working in a variety of different functions but primarily managing other people and eventually managers. IVS almost speaks for itself, it’s the battle tested video system that drives Twitch packaged up as an AWS service for customers to use in new and unexpected ways. As I had spent the years prior to this meeting customers I helped the person leading the product on a number of customer meetings for IVS.
So why leave Twitch? Isn’t life great? Exciting work on the horizon? Sure, that’s the problem.
I joined Twitch to be part of the transformation from “linear TV” to online media and the already transformation happened. Society went from “old media” with news anchors and linear programming to Netflix, YouTube and Twitch. Streamers, influencers and individual content producers have become part of our daily life and have started overshadowing the traditional media machine. I’m sure there are tons of exciting new developments happening but the large ship has already sailed. It might not seem like it but consider where you get the majority of your entertainment today, chances are it’s OTT services or UGC services mixed and matched after your interests.
COVID was probably an eye-opener for many people but what this worldwide event has highlighted is a glimpse of how our social fabric would look if it moved onto the internet. Zoom has become a behemoth of video conferencing and the online media services have seen tremendous growth over 2020. What I think has passed under the radar for many people though is how the concept of hanging out with your friends has started to move onto the internet. Someone like me grew up using software like Ventrilo and Mumble to connect to VOIP chat rooms with friends all the way back in 2005 but that experience never really went mainstream.
Over the winter break I realized that Discord had finally done just that, taken the experience I had when in Ventrilo when I was 15 and built a product that enables people to actually drop into a “space” and hang out with whomever happens to be there for the moment. It’s all transient which tends to eliminate. I myself communicate with my friends back home in Sweden using Discord on the daily so I personally believe that they are onto something that no other service currently manages to fill. For that reason I decided to pursue an opportunity at Discord, ending my tenure at Twitch.
In the words of Primal Scream:
“I’m movin’ on up now.
Getting out of the darkness.”
