I love having conversations with people who are behind multiple games and can

kind of see from behind the scenes. My journey in the gaming industry actually started 17 years ago at Game

Forge, a leading free-to-play publisher and having all these problems and issues launching games and operating large

games. Today, I’m joined by the CTO of Nitrada, who’s spent 17 years building the

backbone of online games. Nachado is the world’s leading game server hoster behind Game Fabric and Steel Shield,

which powers some of the biggest multiplayer launches in games history. He’s helped studios like Last Epoch

survive 150,000 player surges, fight off DOS attacks, and keep games alive when

success hits faster than expected. One very, very important lesson, and I only

can repeat myself multiple times, is do performance and load tests globally in

every infrastructure. People tend to forget to test these kind of services.

You need to able to scale everything needed in order to operate your game. What we talk about in today’s playbook

is why most studios plan for launch but not for scale. How infrastructure works in the games industry and the steps to

take for a successful multiplayer launch. I truly believe that playing together with people is an essential success

factor for games nowadays. Looking at past 20, 30 years of gaming. I mean, this is it. The development which made

gaming big from someone who spent 17 years building the backbone of online games. This episode’s guest, Felix Estler.

Felix, welcome to the show. Hey, nice to be here. Thank you very much. Thank you. Thank you. Really excited. I

love having conversations with people who are behind multiple games and can

kind of see from behind the scenes. So really excited about this conversation cuz we’re going to get into the big

games, right? Hundreds of thousands of players. So for people who don’t know, a quick intro to what you do and also

Nitrado and Game Fabric. That would be great. Sure. Yeah. So I’m Felix, uh currently

CTO uh at at Nitrado since 5 years. Um my journey in the gaming industry

actually started um 17 years ago uh at Game Forge u a leading free-to-play

publisher. I started as a uh an engineer uh and and worked my my way up up the

ladder and uh to be honest I I had such great experiences during these 17 years

with working with so much studios and uh with with games and launching games and

having all these problems and issues launching games and operating large games. So this is this was quite a

journey until today. Um and by joining Nitrado actually a uh a fundamental

thing um which which uh was a part of my

previous career at Game Forge uh is to help studios uh launch their games and

operate it on large scale right and worldwide and in global infrastructure. And this this became basically my

passion also over the over the last years. And uh with joining this became this became reality. So I was in the

position to okay we we work with really large studios like uh studio wildard Bohemia and later on uh 11th hour games

and and and CCP nowadays. So this became something where I was able to peel all

my experiences uh into what we have today and also work with all the great

people at Nitrado to build something truly um uh unique in the in in the game

server hosting industry, right? Um and we call it game fabric and and game

fabric is a an orchestration platform for uh for studios and publishers to uh

operate their game servers on global scale, right? And best best thing at game at game fabric is that that you

actually uh you are in control of everything, right? We we we

developed game fabric um or we started to develop game fabric three and a half years ago. And when we started that

journey, we we we sat down and thought, okay, we don’t want to actually build another hosting platform, right? So we

we we want to build something which brings true value to game developers and

publishers and therefore we we centralized all our thought around the

game developer and that was always our center thought when we developed the

features the platform and and uh this basically uh is is is a fundamental part

of our thinking today. Let’s get into it then. So I’ll be kind of role playing. I’m not a game

developer, right? I’m sure there’s a lot of people listening who want to understand what could go wrong, what has

gone wrong, and maybe things that they could avoid or kind of also learn more about how this stuff works. Like I was

in my prep for this podcast, I found out that DOS is apparently very expensive. I thought it was just a server problem.

Yeah, like little stuff like this. So yeah, I’m excited to dig into it. So for people at home, we’re going to

dig into what happens when a game kind of blows up unexpectedly and the things

that happen on the back end. So this will be useful for anyone who’s maybe making an indie game. You are probably

preparing for success. We should also know what happens when that happens. Also for the people who are in studios

now, what happens when you add a zero to the player count or like add 10 zeros to the player count. There are things that

happen and sometimes they find out probably a bit too late or once they’re in the thick of it. So the idea is to

kind of paint that picture. So let’s get started. All right. So you we were chatting before the podcast

like one of the biggest examples of a game kind of blowing up and the things that happened and you were there for that. So I’d love to take us to the

moment of like last epoch like when that went live and the numbers playing out like can you set the scene for us there

and I would love to dig into kind of maybe there’s some lessons we can learn from that. Yeah sure. Uh and then this is obviously

a very great example of uh things blow up um in a positive way. Um but in order

to to explain what we what we did there together with the studio is we we need to take some steps back and um have a

have a review of actually the road to launch right because how how we work how we work is is not not the way we we

provide you the platform the access and the API keys and whatnot and and then you you go by yourself and you host a

game and then you have fun launching a game. This is not the way we work with studios and publishers or devs. um we

actually um work very closely with our partners. So this is also a a fundamental core piece of our strategy.

So we are not the offhands platform provider. We are the really uh supporting partner you have at hand

launching and make your game successful, right? And um actually we we started to

work with 11th hour games uh way before the launch and we we sat together and say okay this game will be a huge

success and we we all were u very convinced that this will be a success. So we we prepared for success and uh

there were several parts of uh of the preparation and um one part is obviously

the basic stuff right we we need to we need to uh estimate okay how much base

capacity we want to have and when when we speak about when I speak about basic capacity I mean you have these different

layers or levels of capacity you have a bare metal capacity which is cheap and operating which does not cost a lot

where traffic is cheap where the operating uh the the platform is cheap as as well. Um but then you have um the

uh scale options with hyperscalers and this is always not that cheap like like

bare metal right with bare metal you have the full core uh the full core performance and in in cloud you always

need to you need to uh look for for um the performance you you get provided by

by the hyperscalers and the individual core performance and also noisy neighbors and all these things. So you

need to plan ahead, right? And with when working with studios, we always keep in mind that um we want to be as

costefficient as possible, right? But not to sacrifice quality. So um we said

ask you a question here. Yeah, sure. Yeah. So for people at home like when they’re thinking about developing their game and

are they even thinking about this stuff like how soon in the process does this happen? Like you mentioned bare metal, you mentioned hyperscalers. From the

people I speak to, I just hear, “Oh, yeah, we’ll plug into they don’t actually think about it as

something very conscious, but from my understanding, when does the game developer start or should start

thinking about kind of this problem? It should be at the very start. Should be once they know if the game’s going to be a success, like we paint a picture of

like what should I do here with this information?” Yeah. So, in my opinion, you should uh plan this as early as possible. So a a

decision for a platform needs to be an early decision because only when you when you are aware okay that’s that’s

the capabilities of the platform you can also plan ahead okay this is the possibilities I have this will be the

cost if my game is successful so this is something you need to really have your uh your thoughts in the beginning

because otherwise you will end up being on a platform which is ridiculously

expensive when when you launch the game and scale uh and then switching platform

forms during launch times or whatever. This will this will be anointed, right?

So, so our our strategy is definitely we want to be part uh of the development process of the game as early as possible

because we see us as a as I said as a partner and a consultant, right? So we

want to build this thing together and and that’s why we want to or we aim to being part of as early as possible and I

would recommend everybody who is developing a game with with uh the plans to be a global a global hit make your

decisions for a platform early. Um talk to talk to people who have experience in

in scaling infrastructure and and servers. Um so this is a fundamental decision you will not change easily

after you you have launched. Great. So in the last epoch example like is any lessons learned there from like

the days things started to blow up? Definitely. So lessons learned or one very very important uh lesson and I only

can repeat myself multiple times is do perform performance and load tests

globally in every infrastructure. Right? So uh when we we were previous

launch of last epoch we we did massive scale tests around the world with different providers with different

hosting environments. So we were actually simulating thousands of players and game servers in order to be sure

okay platform is is stable hosting providers deliver what they what they what they need to deliver um and the

game server actually itself is scalable and the backends as well. So this is also a very important part. Um what you

really need to test it’s not only hosting game servers worldwide and then scale them to 100 200,000 or whatever.

It’s also test your backends right it’s not it’s not only game servers which you need to

scale. Um there are multiple uh supporting services uh around game

server right server list matchmaking service uh loginins lobbies or whatever

and people tend uh to forget to test these kind of services.

What goes wrong if that does Yeah. I mean um uh if if you can if you imagine um you

you you invested a lot of time in in testing this the uh global scale of your game server. So you’re confident, okay,

I can my game servers can handle two, three, 400,000 players uh in parallel. My my infrastructure is ready. Um but

you never tested the uh let’s let’s say the server list where everybody comes together and selects a match or a

server. Uh this service was not tested. This service is not scaled, right? And you are not sure how many requests is that

service able to handle. Um so so this is something we we see regularly that these

kind of services are forgotten because everybody only focuses on infrastructure scaling game servers whatever but

actually the the central piece of of your game infrastructure where everybody connects first or login or searches for

for matches this is sometimes forgotten and this is a key lessons uh key key

lesson learned uh of the past 20 years. Beautiful. So, in this example where the

game’s blowing up, I’m wondering how granular we can get here, like how

important, let me phrase this differently. So, when you’re I’m guessing speaking to all

these studios who are looking to kind of look for a new partner, like do they have this mistaken belief about how

these things work? like I wonder if we can break some beliefs today where they think it works certain way but actually

when they actually get involved they realize oh that’s actually very different than what they expected and I want to make people understand at home

like hey don’t wait till the last minute to find out that this is the case like is there any big mistakes that you hear

over and over again oh um I mean what I just mentioned is something which people tend to forget

over and over again but maybe and one example is and this is also something people tend tend to forget is let’s

imagine your game is very successful. You scale globally with hundred thousands of game servers running in

parallel um and all of these game servers uh create logs. So what happens

in the session and they’re writing logs and you actually uh you have a central

piece of uh software which which collects all the logs and suddenly the things blows up because you wasn’t this

central component as well was not prepared for scale. Right? So uh and this is only one example. You can you

can uh extend that that example to any kind of infrastructure component. You need to operate uh processes or or

services on the infrastructure. You need to not only be able to scale game servers, you need to a be able to scale

everything needed in order to operate your game. And this is logging systems, monitoring systems, uh metric collecting

systems, right? You need to be aware of okay this if things go way beyond what I

have planned with then we need to actually scale everything and then it also comes down to okay

uh if you have a monitoring system uh which collects all the logs and and stores all the data you need to be very

careful when it comes to costs right so this can easily also blow up your bill if if this is not really well designed

or or scaled or built up with a with a a provider which is which is capable of

doing that. It’s taking me to maybe the existential question like why should I bother with a

multiplayer game at this point? Cuz it sounds very complicated. Like why? It’s

interesting because a lot of the games I see at least on YouTube are ones which are at least multiplayer co-op I guess

or the ones that take at least for me mind share as in the games I see spoken about. There’s a multiplayer element.

But yeah, this sounds like it’s a big cost involved. I’m wondering like should people be making multiplayer games? Like

what’s your thoughts on like how many do you feel like there’s a shortage of multiplayer games? I’m wondering like what’s your point of view on this? Like

should people be making more multiplayer games? Definitely. It’s not only because uh

this is our business, but uh it’s also my my true belief, right? So coming from the old uh MMO world like World of

Warcraft and all these uh different kind of games and now being in a world where uh fast-paced session based games are uh

the the only thing people play at the moment. Um so I’m tr I truly believe

that playing together with people is an essential success factor for games nowadays. Um, and it’s not only because

you you have that competitive uh parts, right? When we when we look at

Battlefield 6, which which got released uh some weeks ago, I mean, everybody’s playing that at the moment. Even uh

myself is uh playing that uh nearly every evening. And what I really like

about such games is okay, you you you can actually um play with your friends, you can you can skill up with your

friends together, right? you have your you have your squad and you can have your missions you can have your sessions which you join. I think this is and yeah

looking at the past 20 30 years of gaming I mean this is a this is the development uh which which made gaming

big right and um when when I look at games like um uh arc for example or

Minecraft where uh userenerated content is a very very big thing. Uh I mean this

is my true belief that user users can create their own content, their own

custom worlds. I think this is truly a uh accelerator for multiplayer games in

the future because imagine you build your map, you build your your custom style thing within a game and you can

show it to your friends. Your friends can interact with it. Your friends can play together with with with you on your

own island in your own structures. I think this is this is the essential part of of of multiplayer and why it’s so su

successful right yeah I’d agree for sure like it’s funny I watch a lot of Runescape so I used to

play a lot of time on Runescape like over a year play time of my life something crazy and it’s funny the

content I’m watching is like they people make their own game modes in the game and there’s like a whole new reason to play and then there’s people involved

they make their own game shows in the game cuz it’s like not that much of a sandbox element but you can just place rules and then play

it in a different way And I feel like because you can play with other people and there’s competition involved, it

makes it a decade 20 year plus game like for Minecraft or Runescape, the ability

to play with someone else means there’s some competitive nature. And unless it’s a game I feel like if it’s a single

player game, the only way it lasts is if it becomes an awesome speedrunning game. That’s what I feel anyway. Otherwise,

you pay it once and then it kind of leaves the zeitgeist. Otherwise, if you have a multiplayer game, it gives you the ability to keep playing and playing.

It’s like, all right, I’ve kind of completed the game, but I can just add a new rule now, a whole new way to play, right? And we go again.

Yeah. And this is what keeps games alive, right? It’s it’s true. Uh obviously, uh the the developer or

publisher still needs to publish content from time to time because people love that. But people who who create their

own content, they keep the game alive. And this is what we see with all the the uh successful franchises out there. they

all have this this kind of of basic model with within their games. So the thing I never really understood

was the cost of all of this. So there’s been some evolution with server techch I’m guessing over the last 10 years like

you have your new product. You mentioned some terms before and I want to for maybe someone who’s new in the industry

or just love to hear your take on this. So I would love to break down kind of

okay how do I as a studio pay money to solve this problem. So this is my

rudimentary understanding. I’d love for you to either correct it and fill in some gaps. So hyperscalers are like AWS

or the other version of that. I think Google probably have an version where I can just pay and rent by the hour like

server cost and I can plug in and then I’ll have a bill. Cool. And bare metal as the name implies is a

physical server that I’m either renting or buying. I don’t know if it’s on the studio floor and I’m just like all right

now I own this piece of servers and I’m paying a fixed cost and it will do as much as it can. The hyperscaler I can do

basically pay as you go phone contract but then the other one is like I’m paying for that amount. So other than

that, okay, I’d love to understand is that true? Is this how it works? And

also what’s the cost involved? Because if I’m paying 60 bucks for a game, but it’s got multiplayer for 5 years, am I

paying for that user? Like I might even lose money on the game, maybe. Is that why like people close service down? I’m just very curious about the business of

service. Could you break that down for me? Yeah, sure. Uh I mean, uh uh 10 15 years

ago, um hypers scales were were not that big in gaming. Um and everybody was

renting out machines and do did everything by their own, right? They had their own engineering teams. they they

invested a lot of time and efforts to build their own uh automation scripting

orchestration systems and and with with the uh

evolution of of platforms and technologies and and also hyperscalers um this this change right and uh I

mentioned in the beginning when it comes to to cost planning or costs in general

uh it’s very important for us to educate people, hey, let’s have a

realistic look on on the outlook of your game, right? How how how many players you you expect, how many servers you

expect, and then let’s have a very cheap in in brackets, but a very cheap uh bare

metal base capacity, right? And and we help you or general if you if you speak

to bare metal providers like us um and and others, we we always strive for

okay, what is actually your requirement? what do you really need? So what we aim

is we we don’t want to sell you the biggest box we have in our inventory. We want to have a tailored uh uh machine

which fits your requirements, right? And uh this is how we work together with with studios. We we plan the base

capacity around the world. So we have let’s say we can we can host 10,000 CCUs

uh on our base capacity. Uh but then when you launch a uh a new DLC or an

update, then we will we will plan the the the the launch weekend together and

then you will have the the ability to scale into a hyperscaler of your choice, right? And uh but only for that weekend.

And the the the charming thing uh about our platform is you as a studio you you

you can choose or you can decide by yourself how how cost aware or risk

aware you want to be. So you can say okay you have these different pools of of capacity and uh what we actually do

is we we put a transparent layer above the the capacity. So you don’t need to

worry about okay I have that machine and this machine and and and a different config there or whatever. So we put a

transparent layer above that. So you need don’t need to worry about these things. You should always worry about

your game making your game successful building a good game. You don’t need to worry about the hardware stuff. That’s

our part. We we we take care of that and um that’s why we cover that and you

only get the bare metal capacity and the bare metal performance uh within one bucket. And then you have the ability to

scale out to cloud, right? Uh to to extend your your fleet. Um but always

always with the uh with the uh thought in mind, okay, when this when the when

the spike is over, you easily can scale back to bare metal and be as cost efficient as possible, right? That’s

that’s that’s how we approach that. Got it. So has bare metal changed over

the last couple years? Like what’s the difference? Is there a difference between your bare metal and someone

else’s bare metal? Should I be thinking about that? Like what does game fabric do? Is it doing anything different

compared to doing it themselves? Yeah. So um I would not say game fabric

but when we look at infrastructure or uh hardware uh per se we we have as I said

different layers. So we have our own uh collocations worldwide. We have 10 collocations we operate by ourselves. Uh

and these collocations are optimized for gaming, right? So we have 20 years experience of hosting very large titles,

very huge infrastructure for game servers. And these 20 years and the experience we we gained during this

years, this flew into the design of our collocations. And when it’s it’s not

only the the uh the server architecture we use uh or CPUs and uh the the amount

of of memory in the in the servers, it’s it’s also the network design we build up

in our collocations. Because when you think about games, uh it’s not only the servers, the compute box, it’s also the

network which needs to be as fast and latency optimized as possible. So you

want to serve everybody around the world. Everybody should be able to play your game without any lags or whatever.

So you need to optimize your network. You need to optimize your uh your internet service provider. you say you need to optimize. Should the studio be

thinking about this or should they just be looking at providers that are gaming specific? Cuz I didn’t realize there’s

they could just rent servers from companies that aren’t gaming specific. It would still work. But what I’m

hearing is servers by a company who only do gaming. There’s extra efficiencies

there. Yeah. So, uh, exactly what I want want to say is of course you can rent a

server somewhere and this will work. This is this is not a question. But when you when you work with us um with the u

game tailored network we operate, we always make sure to have uh

gaming optimized network setups and uh um capacities especially um in order to

and you you mentioned that in the beginning DOS attacks. This is a a huge problem in the gaming industry and yeah

let’s dig into it. Yeah. Yeah. Not not only every prov not every provider is prepared to handle DOS attacks. So you

can you explain a do attack because I Yeah, for sure. I think I have 10% of an understanding. I mean, imagine it like um

you have uh you have your phone and uh

obviously someone can call you and can pick up and you can talk. Exactly. So imagine uh 10,000 people want to want to

reach you uh at the same time. So uh not everybody will get through because only

one one one person is able to connect to you directly and you can pick up that

one person but everybody else need to wait or is is frustrated and stops

calling you right. Um so this is easily transferable to to Adidas attack when you look think about a game server. A

game server is also a piece what you want to reach with your client. And um

when a bad actor uh tries to uh to harm you, they actually they call your game

server a thousand times from all across the world, right? And your game

virtually simulate that or they is this Yeah. So um you you can you can

basically use basic network techniques uh like um I’m sure you know a ping

right ping when you ping something and you can overload a game server or a

network component uh easily by sending huge amount of requests no matter which

protocol or whatever that doesn’t need to be the game specific things I’m sorry just understand

they can go directly to the server or do they have to like log into the game and basically flood the game. They don’t

even need to do that. No. When you think about think about a Counter-Strike server, so you go into

the the Counter Strike lobby, you see the the IP of the server in the server list. Uh and then you have the the IP

address of the of the machine the server runs on and then you easily can attack this IP address with

That’s all I need, the IP address. Yeah, that’s all you need. I feel like they should have fixed that when they were inventing servers.

Yeah. And I mean there are sol there obviously there are smart people out there including us which which were

faced with with such a problem right and one one really uh funny is maybe the

wrong word but challenging story is um so as you know we host ARC uh survival

and we were the or we are still the the official hoster of Arc Survival and uh

seven to eight years ago ARC had had a uh a huge problem with DOS attacks

because people um people used DOS attacks against uh servers of ARC as a

competitive advantage, right? So if you if you do an a server um then this

server gets unresponsive and the game is not available anymore. So people started to use DOS attacks in order to uh uh

make the server stop working or the machine basically the server runs on. And uh with that behavior they gained

advantages in the game and then we we we sat down and said okay there needs to be

a solution and obviously also 8 to 10 years ago there were solutions out there. So there are uh big appliances

which you can leverage you can put them in the data center and they they can filter traffic for you. So they they see

these attacks like a castle. It’s like you got to put troops outside and they got Yeah. Exactly. Exactly. So you you can

have these appliances within your network and then they monitor actually the the network traffic and they see bad

behavior and but the the the the challenging thing with these appliances

uh is always they are quite reactive. So they monitor traffic, they see bad

behavior and then they react. But in game server uh environments and this is

also something which is which is very important to remind yourself as a game

developer is it doesn’t need to be a huge amount of traffic which brings down

your server or your game game uh process right it also could be a malformed

packet where your game server is actually actually receiving from the attacker and then thinks I don’t know

how to deal with that packet and I decide to die, right? And then there’s only one mis malformed packet the game

server receives and this already can lead to a problem for the game server because nobody thought about okay how do

we handle uh bad packages or or whatever. So there are different kinds

of attacks when we think about DOS attacks. There are voluminatric attacks. So you have that big cannon and you

point it towards one IP address and then this IP address or server is gone. But there are also very tailored and focused

attacks uh where you don’t have that big cannon. So you only have maybe you also found a a small exploit in the game

server which you can trigger from the outside with a mobile phone packet. And this was an a challenge we uh we we saw

eight years ago. And um we also looked at these appliances which were in the market out there and they all were

reactive. They all were very um generic. And for for games there was no solution

there. No not a fitting solution, right? Not a really tailored solution. Even if it’s reactive, what I’m thinking

is if there’s a competitive game happening and you send a bunch of packets, that might be enough. Like it’s lagged. It’s already lagged even if it

comes back later. Exactly. Like the the bad damage was happen like I see it all the time. Won’t name the

game but like the final part of the game. Great. Every single time there’s a final

event there’s a DOS. It’s like what are you going to do? I’m like how can well I see like a very big studio have

this issue like is it solvable? So I’m very curious like what’s what’s the current tech? Yeah it’s it’s for sure solvable. Um so

um back then we we sat down and we decided okay we need there there’s nothing really tailored for games out

there. So let’s see if we can build something by our own. Um so we started to build what we today call Steel

Shield. So Steel Shield is a uh patented uh anti-do solution um tailored to

multiplayer games. And the the the key part of steel shield is of course it’s

able to uh defend you against these voluminitric attacks, right? It can

detect it, it can block it, it can rate limit traffic, whatever. Uh it’s capable

of doing that. But the very important part and the centerpiece of game of steel shield is

um a protection mode which we call uh proof of identity. And with proof of

identity, we uh implemented a uh solution where the game client itself

first needs to authenticate against Steel Shield and say, “I’m a valid

client. I’m a valid player. Am I am I allowed to play?” And then Steel Shield

says, “Yes, of course, you are a valid player. I know your user ID. I know uh your you are a valid client.

Go ahead and play on the game server.” But in the same time, steel shield monitors every packet which wants to

reach the game server and if there is a packet with a nonauthoritized

uh identity um then steel shield will block that packet. So in in theory we we

put a um a a guard between the client

and the game server which checks every connection and only if it’s doesn’t that solve DOS like when you say

everything doesn’t that just solve it it it just solves the DOS problem. We solved it for uh Arc Survival Evolve. We

solved it for Arc Survival Scand. How long has been out for? That seems like a big news. No eight years uh but patented for two

years. you know and um this is actually something true it’s unique in the in the

industry and the approach is unique and uh we had tremendous success with various of studios I can’t mention all

of them now but a lot of studios uh use that technology and protect their games with that and with with steel shield you

actually can get rid of DOS attacks what I’m thinking is like shouldn’t every big game just use steel

shield in this example like this like is it is there like that’s what my question is as the dev here is like great I don’t

want do I use steel shield is there is there an unexpected cost I should be thinking about here like is there a

situation where you don’t use steel shield like I’m just trying to understand is if there’s any catch in my in my opinion there is no uh no

world where you shouldn’t use an anti-dos solution like like steel shield right especially when it’s uh when it’s

easy to integrate and um with steel shield and game fabric combined it’s

very easy to integrate with steel shield because it comes out of the If you if you uh work with us and and use game

fabric, you can also easily use Steel Shield. But um if you have a a multiplayer game

which is competitive, which where where people uh uh are dependent on the the

the online status of the game server, in my opinion, you need to think about Adidas problem right from the beginning.

If you if you if you are in the middle of a success story and suddenly you you face a lot of DOS attacks, this will

become a huge challenge uh to to implement in in a running live game, right? This so this is also something

where uh we also consult our customers right from the beginning when we are

part of these development phases where we say okay let let’s talk about DOS. What happens if your game server gets

DOS? Is it a problem? uh and if yes, let’s work on a solution here because we have one.

So what’s the difference? Because we’re obviously preparing for users being online. So like you

mentioned before and there’s DLC, there’s a bunch more users. So for example, I remember watching Fortnite

streams when I was a bit younger. The day a Fortnite drop happened, everyone was queuing in to get in. So I’m

guessing they budgeted for a certain amount of servers and now there’s a queue. But there’s also a situation where I remember playing MMOs. so many people on

a server things are lagging or so many people in one space things are lagging like you can’t prevent that or can you

like when it comes to designing your game like the lag that you experience like is lag all server related is that

different like I’m just trying to think like if I’m a studio trying to prevent lag is it only the servers I should be

looking at no of course not um so the the the examples you mentioned is Fortnite uh

with the Q system this is a very smart way of uh of preventing lag or preventing full servers, right? Because

what what happens in the background if they put up a queue, they scale their server infrastructure in the background. So they have

autoscalers in place with hyperscalers and and different platform. That takes time. So that’s why they Exactly. That takes a bit of time,

right? So systems monitor the the the user demand the the CCUs and if certain

thresholds kick in, then the platform goes ahead and scales up new game server instances. So the queue can be reduced.

But the queuing system actually is a smart way of of telling people, hey, let’s wait a second. We need to scale

and once everything is set up and ready for you, you can go in. So this is a very smart way of of delaying that that

the or or setting expectations actually, right? Because back in the days where no

when nobody used queuing systems, everybody was hammer the connect button and and was not and nobody knew okay why

is it not working? Oh my god, everybody was frustrated. queuing system put expectations on you right now. That’s

that’s that that’s a smart way when it when it comes to different architectures like MMOs with huge servers and then

suddenly a lot of players on the on the server and then it becomes laggy can

have tons of uh reasons, right? It could be the server software itself uh which

which is not able to handle so many many players in parallel. uh it could be the

underlying physical machine where CPU is maxed out or memory is maxed out wherever or network is a bottleneck

right because suddenly everybody’s connecting then the network card or the the the connection of the game server to

the internet is is uh is saturated so there are multiple multiple reasons for

lags um and this all of these scenarios they need to uh optimally uh need to be

tested before you release a game. So you need to be you need to know your limits. This this is always a very important

thing. Um or you then need to have the capability and the tooling and the visibility to quickly see okay here’s

the bottleneck it’s CPU or oh it’s network up there and switch switch port is saturated right so you need to have

the visibility across the whole environment. I have a question for you as a gamer

here. So when I play games, I just Googled now, Battlefield reverted from

128 players to 64 players. And my understanding is they didn’t do that because of server reason. It was just

because the gameplay felt better at 64. And it got me thinking, okay, like on

Runescape, it was one I have the most experience with, like a server capped at 2,000. And I think other MMOs I’ve heard

like a,000 2,000. I’m just curious because you’re in the thick of it. Is there a theoretical limit of the amount

of players you can have on a server? Like if I’m designing a multiplayer game, should I have a player count in

mind where like yeah, maybe make games with 50 to 100 max or has the tech changed where there isn’t really a

limit? I’m just curious. What’s the limit for like the amount of players you can have in one place right now?

I I wouldn’t say there’s a there is a a very good question because there’s no

real limit, right? So because when you design your game, you can you can say, “Okay, I want to have a world with uh

10,000 players uh playing together in that world.” But how you uh divide that

world is up to your architecture, right? You for the player, it could it could uh

uh um be visible as one world, but in the background, it’s actually 50,000 servers, right? But the player doesn’t

doesn’t really see that it’s 50,000 servers because your architecture of the game and the engine is just covering

that, right? And you think you are uh in a world of with 10,000 other players. Uh

but this is actually uh comes down to your game design. If you want to have that, if you, as you said, right, if if

if you want to have a a matchbased game where it doesn’t make sense to have 10,000 people uh shooting at each other,

you want to have very fast-paced small small matches, then it’s your decision for your game design. But in the end, it

it really comes down to that. I in my opinion, there’s there’s no real limit. uh as soon as you have uh uh the right

architecture for your for your game or your game servers uh in order to have or to to to to build your your vision.

Sure. What’s is there the biggest game that you’ve guys worked on like in terms of amount of players in one place at a

time? Very curious. I think that that’s definitely last epoch. Um so this is up until now um the

uh biggest biggest environment we we spin up within hours. Uh this was this was really a intense in intense week. Um

but yeah obviously very successful and and and very cool to see that everything worked out and the the studio uh had

that tremendous success with the game up until today. Um, so yeah, I think that

that’s this was one of the of the major major big launches.

Is there anything we feel like we haven’t covered from a game fabric or server situation where like a game

dealer should know that this exists that maybe you keep find yourself repeating? Um

yeah maybe I mean I mentioned in the very beginning that we put a lot of uh

efforts uh into game fabric to make it a developer focused platform. Yeah. What do you mean by that then like

people care about that? Yeah. Um because what what what I saw a

lot in the in the last couple of of years are just generic platforms where of course you also can can host your

workload there. Um but when we thought about game fabric we we we invested quite a lot in in in tooling for you as

a game dev. You can integrate um multiple tools in your game development environment on your development machine

um in order to um spin up game servers right right out of your um IDE for

example. So this is something which we which we integrated quite quite early in the process. Um we we thought about okay

we would we work often with studios or publishers with different teams

different testing uh uh units and QA departments and and whatever and then

dev teams who need to operate the game servers on our platform. So we adopted a

environment uh concept very early in development because the thing we see is

okay you have these huge publishers and they want to bring in their QA team to test a specific version of the game on

our platform and this QA team also needs to be able to change settings on the game servers, restart the game servers

and change match parameters and stuff. Uh and this is often not possible in in

in other Yeah. So it it it’s basically a environment concept where you have okay

you have Q QA environment which w with a specific version of the game or the game server and the QA team can manage that

individually and they don’t need developers or engineers to do that and we also can make sure okay if you have

access to game fabric and to that specific environment you don’t have access to prod right so this this was a

very early adoption of uh of a developer focused um product. Um, and then of

course, uh, the t testing capabilities, testing and debug capab capabilities is something especially in very early

phases of a game developer, you you want to have and I mentioned that earlier, you you want to have the visibility of

what actually happens. If I deployed that game server with that specific version in uh, Miami uh, in the data

center, I want to see what actually is happening there. And this was something which we implemented right from the

beginning. And out of the box you get metrics, you get not only the basic metrics like CPU and memory. This is

something everybody provides but we also integrate deeply within your engine, right? So we provide you uh um also

metrics which you can define by yourself. So for example, you you have your dinosaurs running around in your

game server and you want to see okay, how many dinosaurs are running around in in a specific area of my world or map.

So you can expose that metric and we can we can we display it automatically for you. This is one part and the other part

is uh tracing of your game server uh behavior. So when you when you launch a

game server and it it it breaks or it uh uh yeah it uh

it it it malfunctions somewhere and you you want to see okay what actually happens why this happens. So we provide

you with debugging capabilities within our platform and this is also this is also come comes out of the box. So you

don’t need to implement anything. So we provide you crash dumps and very very deep insights of why your process

crashed, right? And you can analyze that at games as a game developer and that makes it really easy for you to iterate

fast, right? And this this is basically some some parts of our uh philosophy of having a uh developer focused platform.

Yeah, you mentioned so many things there like if I’m shopping around there’s so many things to keep in mind apart from just cost,

right? like the amount of hidden cost I’m hearing preventing a DOS attack, maybe

preventing all this back and forth with the Q8, people needing to test, capturing stuff because

if it’s more intuitive to use, probably more testing will be done, right? Like it’s like a mental thing as well. So

yeah, there’s a lot of things that you don’t really think about and I’m guessing developers aren’t thinking about either because they’re probably

focusing on the game, right? So I think it’s they should they should focus on the game. We we we should be

it makes sense to have this inhouse in terms of like a human doing this because I know there’s a few companies who do

that right there engineer right like when does that make sense? Yeah, I mean

I I was uh I was also one of of such engineers who who who who said okay let

let’s build this in-house because we know what we need right um but uh so back in the days but um the more I

worked with different studios and publishers and games and infrastructure and technology for games I become aware

okay publishers or game developers they should not worry about the underlying

technology they should really focus on building a good game and not worry about CPUs or databases or or matchmaking

services or whatever. They they should consume services which are operated by

people who actually are experienced in doing that. And um I mean obviously

there are huge companies out there. They have huge tech teams. They have their own platforms built up, right? That this

is totally fine if you want to do that. Um but as especially as a new studio, a

small studio, the the cost constraints of building up engineering teams and having uh experts on board, let’s say

for Kubernetes or uh cloud infrastructures, uh this these people are expensive.

These people are hard to find. Um so you should not worry about such things. You probably don’t need them 40 hours a

week either. Yeah, in the beginning, yes. But, uh, when you launch and you’re successful,

you even need more of them and then it becomes a even bigger challenge. Yeah. For me, it reminds me of like if

I’m a company, I’m like, “Ah, I don’t need to pay for Jira or project management software. Let’s make our

own.” It’s like, “Okay, but you’re now a project management software developing company.” Exactly. uh where yes over the long term

you might save on the thing but like you say at some point something needs to change and then you need to either hire

more or there isn’t enough time so I feel like it’s yeah it definitely opened my eyes on

kind of how this works yeah as I said I mean um this is a journey and uh obviously there are

companies who who who said to themselves in the beginning we will do this by by ourselves um but we had a lot of

examples um where we where we saw this and then we came in very late in the

process and we helped them to scale right because the engineering teams they were not able to do this anymore because

the was just a overly complex architecture uh designed by themselves

and it was not based on on on standards or default technology know what they don’t know right like

you’ve guys have done it over and over again it’s just one of those where there’s some efficiencies where

they just they can’t know because the I’m guessing the space evolves or what have you. Yeah, exactly. And that that’s why why I

also said already lessons learned is talk to people who are experienced in

building up global multiplayer games. They will tell you is it doesn’t cost anything to you.

Yeah, exactly. You you can you can ping me on LinkedIn. I I’m happy to talk about all these things. But talk to

people. They will everybody will tell you don’t build it by yourself. Yeah, that’s that’s for sure.

Cool. I’d love to get into Felix before we close out because your journey is quite interesting, right? You started as

a motor engineer and now your CTO kind of we’re in a situation I’d like to

spend a little bit of the end of the podcast to kind of give advice on kind of how people can grow as a human and as

a career. So I’m wondering what was the one thing you would tell kind of Felix maybe 5, 10, 15 years ago that you wish

they knew then when it comes to like growing in their career? Is there any lessons you learned the hard way?

Definitely and uh you you you already mentioned it definitely don’t be a tools company

focus on what’s really important. So in my early days I was as as you said I was

like we can do that by ourselves. DIY baby let’s do it. Yeah exactly but this is maybe one of

the biggest learnings is focus on what’s important not only on the product side or feature side. So it it’s really focus

on your core strength and that is that is uh you can also transfer that to

to to me personally right so because I I I

experience that I’m I love to work with people I love to work with talent people and individuals who inspire me right and

um this is how I became into or how I came into leadership u 13 years ago now

and Um I love to develop people. I love to build up teams, structures and all of

these things. And uh through all that process, it it it became clear that

focus is no matter where focus is the essential thing. Focus on okay your team

needs to focus on on a specific task. You can’t just overload them or whatever. You need to have a focused

team. You need to have a focused company, a focused product. And uh I think yeah f focus is one of my uh my my

main mantras I tell myself every day um because I hate nothing more than

being uh being the person who go to my my engineering teams and say oh we need these 50 things this week uh please make

it somehow happen uh and this this will not work right and this is something you need to really experience in my opinion

and you need to learn that obviously I can tell you that now. Um, and you can

be smart and and and and take that uh advice, but in my opinion, you need to

really experience that focus is going through that right now can be a key success factor.

Yeah, I’m going through that right now, Felix. So it’s you’re going to laugh at me maybe but business started in April

last year and just maybe yeah last month I’m actually using proper project

management software and before it was like ah we have less than 10 clients we can just check each

client’s board for example and what’s happening there but now we have people and it’s not just me and it’s not all in

here so I’m like all right this needs to go somewhere and then I’m realizing there’s so many things we can improve so we started doing a dashboard for are

writers for example and what I’m realizing now there’s so many things I want to improve and I’m the person who’s

like I’ll do it every day and what started to happen because I’ve got the visualization of what can be improved a

month ago I was actually quite a chill guy I was like ah business is growing this is going well but now that I can see all the stuff that can be done I’m

like I want to do everything and then I don’t do everything or I don’t do the important stuff this is maybe a too big of a question

but what do I do man how do I prioritize I feel like I’ve done so many different

prioritization techniques and I’m like, “Ah, yeah, but just want to do it now.” Or this thing’s more shiny. Let me work

on this thing. Like I’m wondering if you’ve got any tips that’s helped you when it comes to I just now that I can see all the stuff

I can do, I feel like I’m behind when I know I wasn’t behind. I don’t know. It’s a weird feeling.

Yeah. You I’m in a sim I’m in a similar position, right? I have a lot of ideas what we could do, what we could

implement as features in our platforms uh or for our customers. But obviously it always comes down again on okay is

that really in in line with this and this is a very important part is it in line with the

strategy right so you you you have a strategy for yourself or for your company and we obviously also have a

strategy on on what we want to achieve or where we want to be and I always ask myself okay is this in line with

strategy or not and if not okay why I

should maybe then proceed with the idea. Uh nevertheless, right? Right. There there are some ideas

where you then could just pivot away from from the surgery because it’s a maybe a new adventure or whatever. Um

but in the end by on me it always comes down okay let’s focus let’s sit down

with with my experts and this is always something I I I do I have my experts very close to me.

So I talk to them, I pitch my ideas. You say experts, are these internal or do you mean? Yeah. No, we have a huge team. So we are

two 200 people worldwide. We have a huge engineering team. Um and we have uh close or there’s a there’s a leadership

team within within each tactical question, Felix. When you say you bring together the experts, so I’m having a

situation now where I have the event side of the team because we’re doing more mixes and conferences next year and

I have the content side of the team. And if I come up with an idea, I want people to feel ownership of the idea. But then

also I’m realizing I don’t have that much time because I’ve got today is podcast day, tomorrow is retro day, and

then I’ve got to write my content. And like the time it takes to come together and agree something. I’ve noticed that

it started to take a lot more time than I expected. So how do you make sure that happens maybe quickly? I don’t know if

you’ve got any ways to do that. Yeah. So when I was in a similar situation like you where uh I have these

these teams very close to me and I was in the daily operations all day long and

and filling with all the different piece of parts in my opinion it’s essential that you

build up um maybe leads is is a big word but if you you need to have key people

who are driving these individual uh disciplines right when it comes to content or events you need to have someone and I’m sure you have senior

people in your in your in your teams where you trust and for me it always

comes down to trust. So I have I have experts for all the different fields we need to work on and I have people who I

trust and if I have an idea or if I want to have something to get done I go to

these people and say guys please think about a solution for this this and that and then I trust these people to work

out the solution by themselves. So I not I don’t need to care about what they what they actually do if the result is

right because I trust them to operate in in uh in in in in guard rails where I

definitely believe in they have the same values the same uh um ways of doing

things than I have because I trust him a lot. You set the strategy and I’m guessing they know the strategy. So they go into

the same goal. But like I’ve had this situation where you want to I want to give ownership

but then you get the result and it’s like oh if only I was involved from the start I might have maybe helped you have

this resource or this resource and I find like I’m very hard trying to find that balance. Right now I’m a bit too

all micromanagyish where I’m like maybe try this, maybe do this but then again I don’t want to be in there doing it but I

feel like oh this would help so much. Then the other side I want to give full ownership because it’s probably very

annoying for someone to come and say hey try this you know. Yeah. This is a tough one because it

obviously it sounds like it’s a very new team and you you just growing phase of right. So

yeah. So it’s it’s really it really comes down to to get closer to each other to align your I mean you you need

to align your your thoughts. You need to be open to your people. You need to be uh uh you provide them your vision so

they can actually have a chance to uh to do the things you would do. But I I

definitely understand. I I’m sometimes micromanaging by myself, right? This is

this is this is changing from time to time, right? If you have someone uh who who you think, okay, I I now need to be

close to that person because I I I really want to get this going and I I exactly know how this is how this needs

to be. So sometimes I’m I’m the same way. But I I think it’s very important

to have a balance, right? You need to make a step back, reflect. Okay, what I can me as a manager, how what what I can

do better or what I can provide additionally to my to my seniors or

experts in order that they can make their job better next time so I don’t have to micromanage them in the future

because this is this is a killer for all of us I think the way it is quite helpful.

You’re presenting new information that may help them reach the goal. So it’s like the framing of it, not like

hey why didn’t you do it this way? It’s like, “Oh, by the way, here is information that is useful.”

Yeah. And also don’t expect them to just be able to go and find that information or know that information. I think I’ve done a little bit of that where it’s like,

“Oh, why didn’t you try this?” Well, I should be like, “Okay, why should they try this?” Like, “How would

they have even known?” You’re like, “Yeah, fair enough.” Yeah. I mean, ask them, “How would you solve that specific situation?” And then work

work your way through these basic basic questions and and help them develop themselves, right? And only if you if

you provide them freedom and if you allow them to make mistakes and come back to you and present their solution

and you say ah maybe that’s not that what I thought about maybe do it different only if you if you open up yourself and

have and and create an environment where people have no fear going to you and present something which is maybe not

optimal or ideal. Yeah. I only then then this is an environment where people can grow. uh if if they

would fear you go to you and present you something and they always know okay he will freak out again I I I don’t want to

do that I think creating a creating an environment of trust and uh allowing

people to also present maybe not yeah not not that good solutions uh I think

this is part of the journey and people will grow that’s that’s my experience

beautiful great I will leave with one question what game are you playing right now very curious

battlefield feel six all day every day. I’m so scared to play those games for you. Every second day I would say

I feel so scared to play those games cuz I feel like I’ll get addicted. I’m on Clash Royale right now and

I made a little joke post the other day where I have to install a thing where it’s not this

card, but basically I have to unlock my phone with a card now so I get to play my Clash Royale. That’s how addictive I get. Oh wow. Okay.

But also I’m one of those where I’m like I need to have a screen. Not healthy, but I’m getting better for now.

Awesome. Can relate. Lovely. Felix, thank you so much. How can people get in touch? What’s the best way?

LinkedIn. Hit me up on LinkedIn. Uh, happy to chat about everything game

related or infrastructure or techreated. Just shoot me a message. Beautiful. And yeah, that’s not a small

offer as well. Like, you’re a very busy man. So, if anyone’s listening to this podcast and you’re honestly genuinely curious, take him up on the offer. Get

get in touch on LinkedIn the description. Oh, the name of won’t be find him on LinkedIn also stuff in the

descriptions. If this is interesting as well, you can check out Game Fabric. I’ll put all that in the description.

And if you’ve listened so far, maybe follow, share it with a friend. Lovely. Felix, thank you so much. Have a great

day. Thank you very much. Goodbye, everyone. You too. Sew it.