Before I am "Canceled" - Server Meshing - Fact or Fiction

[ColdDog]

Traffic mgt, security, and observability. Looking at the possibilities CIG is having meshing problems is beyond me. For one, we are paying premium for AWS and for CIG to blame meshing is beyond me, with their failed architecture. CIG is small potatoes compared to many more of Amazons customers.

So, why have we been discussing “meshing for the last 2-3 years” … I forget how many years. Every time I play and spend two hours mining I get a disconnect. Every time someone brings up the Server/network meshing bullshit. We all know it is easy to throw the problem over the fence for non-technical customers. I’m calling out CIG as bullshit… either they have a horrible infrastructure, or they are lying to us flat out.

I do not know how fucked up the CIG PU is but it sounds pretty fucked up if they have not been able to fix the meshing issue yet. Please do not tell me another 5 years for meshing, because I am at my last thread.



That is my thesis… so, lets look at this. Please see my links below.

• Amazon is not a lightweight… they have tech for everything, from meshing to databases plus much more.
• Service meshing is a click away.
• Microservices vs meshing… microservice is old school and things have changed.
• Cloud services by definition – instances grow and contract by demand from spinning up new servers to memory to storage.
• Seems like a relational vs no-sql database issue. Elastic is the standard for no-sql and I do not understand why their issues are so bad. If they are using Microsoft sql sever, they are throwing money away every day.
• Ask yourself what the banking industry uses…
• We have already spent 5 years correcting the problems from 10 years ago, game in Alpha (10 fricken years) … If this were a real business, it would have failed 7 years ago. I am more convinced that CIG is riding the gravy train, and every year that passes CIG falls into the vaporware category.
• Every year that passes that this game is in Alpha, the legal argument gets stronger for future legal action.
• Making an effort is not the same as producing.

Conclusion, as I said in my thesis, CIG does not seem to be seriously addressing the meshing problem because if they were, it would not be a 3 year problem, but a 3 month problem.

What's a service mesh?

A service mesh is an infrastructure layer built into an app which documents how services interact making it easier to optimize communication and avoid downtime.

www.redhat.com

… way to control how different parts of an application share data with one another.

Each part of an app, called a “service,” relies on other services to give users what they want.

This new service will communicate with a database of product tags to make recommendations, but it also needs to communicate with the same inventory database that the product page needed—it’s a lot of reusable, moving parts.

But what if some services get overloaded with requests, like the retailer’s inventory database? This is where a service mesh comes in—it routes requests from one service to the next, optimizing how all the moving parts work together.

Amazon meshing

https://docs.aws.amazon.com/app-mesh/latest/userguide/meshes.html

Matter of clicks

A service mesh is a logical boundary for network traffic between the services that reside within it. After you create your service mesh, you can create virtual services, virtual nodes, virtual routers, and routes to distribute traffic between the applications in your mesh.

https://docs.aws.amazon.com/app-mesh/latest/userguide/what-is-app-mesh.html

Elastic

https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-service-mesh-istio.html

Istio

https://istio.io/latest/docs/concepts/what-is-istio/

Istio makes it easy to create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, with few or no code changes in service code. You add Istio support to services by deploying a special sidecar proxy throughout your environment that intercepts all network communication between microservices, then configure and manage Istio using its control plane functionality



Amazon Database

Databases

https://aws.amazon.com/rds/

 

[Gotin]

Go apply to work at CIG then, and speed it up for the rest of us.

 

[Michael]

¯\_(ツ)_/¯

And since i can't argue pro or con on this

 

[Grimbli]

¯\_(ツ)_/¯

And since i can't argue pro or con on this

I don't know what to say either so I'm all for shitposting!


ColdDog's first day at CIG or something...idk, I'm not a professional.

 

[Lorddarthvik]

I think there is a very simple explanation to this delay. You said it yourself: "their failed architecture ". It's their architecture, and it's a fucking mess. It has been since the start.
If you want the long version, I'm pretty sure you can find some extensive documentation on how CryEngines netcode sucks ass, and is entirely unsuitable for what CiG is doing. Thus they had to rewrite the whole thing, and they keep adding stuff, so they need to fix it again and again.

- they have their own jank code, which has been pointed out time and time again by critics, is a huge issue going forward. SC is not built to any industry standard, it does does not follow standard practices, and thus it is not compatible with off the shelf meshing systems. Hence the constant refactoring and rewriting of the game's systems.
This game isn't WoW from 15 years years ago, not everything is just a number in an SQL database. It should be, but it isn't. That's basically what the magical mystical oh-so-special i-cache will be. A glorified unified database of all things in the verse. This should allow for stable and "easy" meshing.

- they might not have figured out how it all should work in practice exactly. They don't know who should get meshed with whom, when, how far away, friends list and the reputation and PvP system should be considered as well... it's complicated. They might know what they want to base meshing off of, but they might not have the neccesery data to draw from in the game yet. Especially when you have ppl like CR wanting things that are unnecessarily complicated. And when they finally figure out the exact method, they need to code it. And that's when they face "their failed architecture".

I believe that they have had server meshing up and running since they first talked about it. But up and running, as in for 5 seconds before it crashed everything, lit the server room on fire, and caused the nearest power station to melt down. Cos their code is messy.


Note that I'm not saying this to excuse CiG for it's lack of foresight. On the contrary.
But that's "agile in practice" in a nutshell. Make up shit code that wouldn't hold up in a middleschools computer science class, as fast as possible, fuck what happens one tier up when it needs to be integrated. Then spend ages cobbling everything together, and ending up rewriting the whole thing 4-5 times, and patching it endlessly.

 

[BUTUZ]

I mean no other real time first person game has managed to implement anything like what GIC are aiming for, it's blue sky thinking. I mean christ when I played elite dangerous I could only invite 3 other people into my party. And we could go shoot another party with 4 peeps in it. 4x4 woop.

It's frankly amazing that we can play 50 together in a planetary scale gorgeous sound and graphics game as it is!

Meshing is gonna be the hardest thing they do and take the longest to get right.

 

[Grimbli]

I mean no other real time first person game has managed to implement anything like what GIC are aiming for

This was my understanding. Sure, server meshing type stuff has been done before. And you could point to it and say, "See?!". But with the crap CIG is trying to do it appears to be brand new territory.

Hell, Guild Wars 2 had server meshing years ago, but they also culled people off the screen, had psuedo-instances, and loading screens. Not to mention their game world was stupidly small compared to a single planet and it was all on a 2d playing field.

I'm not sure there's any game in existence that has the server meshing CIG is looking to accomplish. It's like pointing to the Death Star and saying we've had space stations before, how hard can this be?

 

[Sayora]

The type of Dynamic meshing that CIG is aiming for is, to put it bluntly crazy. Servers that will dynamically resize themselves based on how many players in an area, spool new servers as needed, move players in and out of servers seamlessly. It's a technological minefield, and that's not even considering CryEngine which is not the most MMO friendly of engines to start with.

I am fairly confident we will see a static mesh by the end of the year or early next, they can't add new systems or even many new POI's till that happens. But since this tech isn't needed by Squadron 42, it's not a top priority (it's still important, but CIG is pushing SQ42 hard)

 

[Richard Bong]

Traffic mgt, security, and observability. Looking at the possibilities CIG is having meshing problems is beyond me. For one, we are paying premium for AWS and for CIG to blame meshing is beyond me, with their failed architecture. CIG is small potatoes compared to many more of Amazons customers.

So, why have we been discussing “meshing for the last 2-3 years” … I forget how many years. Every time I play and spend two hours mining I get a disconnect. Every time someone brings up the Server/network meshing bullshit. We all know it is easy to throw the problem over the fence for non-technical customers. I’m calling out CIG as bullshit… either they have a horrible infrastructure, or they are lying to us flat out.

I do not know how fucked up the CIG PU is but it sounds pretty fucked up if they have not been able to fix the meshing issue yet. Please do not tell me another 5 years for meshing, because I am at my last thread.



That is my thesis… so, lets look at this. Please see my links below.

• Amazon is not a lightweight… they have tech for everything, from meshing to databases plus much more.
• Service meshing is a click away.
• Microservices vs meshing… microservice is old school and things have changed.
• Cloud services by definition – instances grow and contract by demand from spinning up new servers to memory to storage.
• Seems like a relational vs no-sql database issue. Elastic is the standard for no-sql and I do not understand why their issues are so bad. If they are using Microsoft sql sever, they are throwing money away every day.
• Ask yourself what the banking industry uses…
• We have already spent 5 years correcting the problems from 10 years ago, game in Alpha (10 fricken years) … If this were a real business, it would have failed 7 years ago. I am more convinced that CIG is riding the gravy train, and every year that passes CIG falls into the vaporware category.
• Every year that passes that this game is in Alpha, the legal argument gets stronger for future legal action.
• Making an effort is not the same as producing.

Conclusion, as I said in my thesis, CIG does not seem to be seriously addressing the meshing problem because if they were, it would not be a 3 year problem, but a 3 month problem.

What's a service mesh?

A service mesh is an infrastructure layer built into an app which documents how services interact making it easier to optimize communication and avoid downtime.

www.redhat.com

… way to control how different parts of an application share data with one another.

Each part of an app, called a “service,” relies on other services to give users what they want.

This new service will communicate with a database of product tags to make recommendations, but it also needs to communicate with the same inventory database that the product page needed—it’s a lot of reusable, moving parts.

But what if some services get overloaded with requests, like the retailer’s inventory database? This is where a service mesh comes in—it routes requests from one service to the next, optimizing how all the moving parts work together.

Amazon meshing

https://docs.aws.amazon.com/app-mesh/latest/userguide/meshes.html

Matter of clicks

A service mesh is a logical boundary for network traffic between the services that reside within it. After you create your service mesh, you can create virtual services, virtual nodes, virtual routers, and routes to distribute traffic between the applications in your mesh.

https://docs.aws.amazon.com/app-mesh/latest/userguide/what-is-app-mesh.html

Elastic

https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-service-mesh-istio.html

Istio

https://istio.io/latest/docs/concepts/what-is-istio/

Istio makes it easy to create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, with few or no code changes in service code. You add Istio support to services by deploying a special sidecar proxy throughout your environment that intercepts all network communication between microservices, then configure and manage Istio using its control plane functionality



Amazon Database

View attachment 20384


Databases

https://aws.amazon.com/rds/

The issue is that, while the servers are capable of doing the job, the software net code isn't. The game software needs to understand it can "push the button" and when it should "push the button."

 

[Richard Bong]

The type of Dynamic meshing that CIG is aiming for is, to put it bluntly crazy. Servers that will dynamically resize themselves based on how many players in an area, spool new servers as needed, move players in and out of servers seamlessly. It's a technological minefield, and that's not even considering CryEngine which is not the most MMO friendly of engines to start with.

I am fairly confident we will see a static mesh by the end of the year or early next, they can't add new systems or even many new POI's till that happens. But since this tech isn't needed by Squadron 42, it's not a top priority (it's still important, but CIG is pushing SQ42 hard)

Actually AWS servers are designed to do that. Cryengine, and the spinoff Lumberyard isn't.

 

[Bambooza]

Traffic mgt, security, and observability. Looking at the possibilities CIG is having meshing problems is beyond me. For one, we are paying premium for AWS and for CIG to blame meshing is beyond me, with their failed architecture. CIG is small potatoes compared to many more of Amazons customers.

So, why have we been discussing “meshing for the last 2-3 years” … I forget how many years. Every time I play and spend two hours mining I get a disconnect. Every time someone brings up the Server/network meshing bullshit. We all know it is easy to throw the problem over the fence for non-technical customers. I’m calling out CIG as bullshit… either they have a horrible infrastructure, or they are lying to us flat out.

I do not know how fucked up the CIG PU is but it sounds pretty fucked up if they have not been able to fix the meshing issue yet. Please do not tell me another 5 years for meshing, because I am at my last thread.



That is my thesis… so, lets look at this. Please see my links below.

• Amazon is not a lightweight… they have tech for everything, from meshing to databases plus much more.
• Service meshing is a click away.
• Microservices vs meshing… microservice is old school and things have changed.
• Cloud services by definition – instances grow and contract by demand from spinning up new servers to memory to storage.
• Seems like a relational vs no-sql database issue. Elastic is the standard for no-sql and I do not understand why their issues are so bad. If they are using Microsoft sql sever, they are throwing money away every day.
• Ask yourself what the banking industry uses…
• We have already spent 5 years correcting the problems from 10 years ago, game in Alpha (10 fricken years) … If this were a real business, it would have failed 7 years ago. I am more convinced that CIG is riding the gravy train, and every year that passes CIG falls into the vaporware category.
• Every year that passes that this game is in Alpha, the legal argument gets stronger for future legal action.
• Making an effort is not the same as producing.

Conclusion, as I said in my thesis, CIG does not seem to be seriously addressing the meshing problem because if they were, it would not be a 3 year problem, but a 3 month problem.

What's a service mesh?

A service mesh is an infrastructure layer built into an app which documents how services interact making it easier to optimize communication and avoid downtime.

www.redhat.com

… way to control how different parts of an application share data with one another.

Each part of an app, called a “service,” relies on other services to give users what they want.

This new service will communicate with a database of product tags to make recommendations, but it also needs to communicate with the same inventory database that the product page needed—it’s a lot of reusable, moving parts.

But what if some services get overloaded with requests, like the retailer’s inventory database? This is where a service mesh comes in—it routes requests from one service to the next, optimizing how all the moving parts work together.

Amazon meshing

https://docs.aws.amazon.com/app-mesh/latest/userguide/meshes.html

Matter of clicks

A service mesh is a logical boundary for network traffic between the services that reside within it. After you create your service mesh, you can create virtual services, virtual nodes, virtual routers, and routes to distribute traffic between the applications in your mesh.

https://docs.aws.amazon.com/app-mesh/latest/userguide/what-is-app-mesh.html

Elastic

https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-service-mesh-istio.html

Istio

https://istio.io/latest/docs/concepts/what-is-istio/

Istio makes it easy to create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, with few or no code changes in service code. You add Istio support to services by deploying a special sidecar proxy throughout your environment that intercepts all network communication between microservices, then configure and manage Istio using its control plane functionality



Amazon Database

View attachment 20384


Databases

https://aws.amazon.com/rds/

How far down the rabbit hole do you wish to go?

I mean no other real time first person game has managed to implement anything like what GIC are aiming for, it's blue sky thinking. I mean christ when I played elite dangerous I could only invite 3 other people into my party. And we could go shoot another party with 4 peeps in it. 4x4 woop.

It's frankly amazing that we can play 50 together in a planetary scale gorgeous sound and graphics game as it is!

Meshing is gonna be the hardest thing they do and take the longest to get right.

Honestly this is pretty accurate. We are starting to see multiplayer FTP games using ballistic mechanics (projectiles) over hitscan for things beyond sniper ammo and rockets (Fornite still uses hitscan for most weapons) The advantage is time to target is calculated by the server instead of insta hit that is hitscan. The downside is each bullet has to be tracked for ttl and that increases the work load.

AWS servers farms are massive but you have to remember that there is limits to the amount of resources a single server can have. https://aws.amazon.com/ec2/instance-types/ and so each server is still constraint by this as well as network bandwidth. A lot of it is that the engine has had minimal optimization passes and code trimming on top of supporting a huge game field especially if all 50 players go in different directions its still entirely possible the server ends up loading up most of the games assets. So yes AWS has lots of servers but the game server is still restricted to running on one physical host.

CIG is using the wrong game engine for what this project has become. While it can be debated I have no problem with them picking CryEngine for the initial concept it was at the time a graphic pretty engine that did ok networking of small fps maps (small as in asset count not map size). With the massive success of the project the scope grew well beyond what the engine was capable of supporting even on AWS servers and at a minimal the backend server code should have been scrapped in favor of creating a ground up solution using modern techniques. Still not an easy task but doable. But I understand it as I am still working on a legacy application that should have been eol 5 years ago and has significant performance problems when scaled up. Its hard to see the pitfalls looking forward especially when the scope has not even come close to having boundaries applied.

I also wonder just how many developers they have working on networking, server code and mesh or if these developers are getting pulled all over the place and not truly dedicated to the task.

Spoiler: Rabbit hole

Now your examples are not as applicable to the situation CIG is faced with their game server meshing. While they share the same term and have some similarities there is still a lot of differences in the execution.

Elastic and Istio are used to track the utilization of docker containers availability and expand as needed. The best way to visualize this is to think of straws. The amount of water that can flow through a single straw is fixed and the length of time it takes water to flow with out increasing the pressure is fixed. Adding more straws does not increase the amount of time water flows through the straw just how much water can flow. AWS allows CIG to do this already by spawning new servers as the player count increases. It does not allow the individual servers to handle more players just the total players able to play the game by spinning up new servers and creating their own independent instance of the game. Most of this technology came out of e commerce where the time it takes for a individual user to process their order is not as important as being able to handle thousands of concurrent order processing. But you can see how this technology sort of sucks towards fps games if you have to wait seconds to see if your shot was successful or not even if the server mesh could handle thousands of shots at the same time. The other thing this doesn't handle very well is handing off client connections between servers in the mesh. Typically you code the interaction to be as stateless as possible so each request is handled as a unique one off transaction with most of the data needed for the transaction sent as part of the client request.

So the examples provided for industry server meshing is used primarily for horizonal scaling of micro services, adding more straws to handle more concurrent connections but not decreasing the amount of time it takes to process each request. Often times you will find that there is bottlenecks in these implementations when they start to try and share or store the state of the transactions and depending on how rigid the transaction has to be and often times a principle of eventual consistency is applied. https://solace.com/blog/eventual-consistency-in-microservices/

AWS is starting to do some nice things with their Amazon Elastic Container Services (https://aws.amazon.com/app-mesh/faqs/) especially when it comes to traffic routing.