Server meshing discussion

[Cugino83]

From time to time I lurk into spectrum to see if, by any chance, there is some interesting discussion that is not another ranting tread abount thing being missing of bugs already well known... well, It seams I've found one.

Meshing The Servers - Star Citizen Spectrum

MESHING THE SERVERS In the Unity 3D engine we use a similar process to load/unload game objects in and out of the scene depending on what the player is looking at (Occlusion Culling) as well as...

robertsspaceindustries.com

This is an interesting techincal discussion about server meching carryed ouut by some guys that are clearly coompetent since they also work on game development and have to deal with the same kind of problem CIG is trying to solve with the server meshing solution.

I think this worth a reading since I found it very complementary and more detailed of the short explanation we have from ISC and SCL where the whole problem is condensed in few minute of video.

 

[Michael]

More on the topic:
https://robertsspaceindustries.com/spectrum/community/SC/forum/50259/thread/how-is-server-meshing-possible
and
https://robertsspaceindustries.com/spectrum/community/SC/forum/50259/thread/predict-needs-for-server-meshing-splitting
and
https://robertsspaceindustries.com/spectrum/community/SC/forum/50259/thread/server-meshing-traveling-between-meshes/3372399

 

[CosmicTrader]

Thanks for the info.

 

[Mich Angel]

Good Stuff!

 

[Esctasy]

I heard that they will have to implement this before they release the Pyro system, is that true? Because Pyro is supposed to be launched in early 2022.

 

[Michael]

I heard that they will have to implement this before they release the Pyro system, is that true? Because Pyro is supposed to be launched in early 2022.

Well yes and no.
Server Meshing (at least static) is needed to connect Star Systems and keep performance or even make it possible.

But they also hinted that they somehow could fake it. (Travel would be a loading screen with connection to another server)

 

[Richard Bong]

I heard that they will have to implement this before they release the Pyro system, is that true? Because Pyro is supposed to be launched in early 2022.

Have you ever been in a crowded area of the Stanton system and have low frame rates, and people (and even NPC) rubberband? Server meshing should fix that.

Need to properly fully crew a cap ship? We need a bigger server cap. Server Meshing should allow that.

Making the game meet Chris Roberts' stated vision requires server meshing.

As our fearless leader likes to call these kind of things, like icache, server meshing is a "Jesus patch."

 

[Vavrik]

I used to use Unity a lot, but not for developing games. For me it was scientific and industrial simulations. We still had the same problems though. It's very hard for me to describe in language that a common user can understand - and I haven't done anything with Unity for 2 1/2 years now. My god it's been a long time.

But they also hinted that they somehow could fake it. (Travel would be a loading screen with connection to another server)

Yep, but the transfer of a user to another region in a game is almost instantaneous. What they're simulating is the distance between star systems, and the reason why is it's kind of odd to just kind of "go next door" and be in a different star system when we know they should be light years away. Its also extremely expensive and wasteful to make more empty space than you need. So why not invent some kind of FTL, call it a wormhole... and make it kind of an interesting place to be.

BTW - if someone wants to get started with game development or just to learn about it a bit, I recommend the community edition of Unity. C# is what it uses as a scripting language, but you can also extend the game by writing your own classes. You can use other languages other than C#, even C++ to do that. It's free to use, and anything you develop can be sold, shared or whatever - as long as you don't make over 100K/year, something like that.

 

[Bambooza]

Have you ever been in a crowded area of the Stanton system and have low frame rates, and people (and even NPC) rubberband? Server meshing should fix that.

It has the potential to fix part of it but given CIG is going after a server authoritative game world model some of the rubber-banding will still exist due to network packet loss and client side prediction algorithms (attempts to smooth animation between server world state updates as they are currently set at 30 updates a second). Your char update rubber-banding is often caused by network packet loss as your client attempts to update the server on input commands and not update location cords. I have to say its an interesting choice most fps games either allow local client motion to be client side authoritative (Escape from Tarkov) thus the client updates the server on its current location and the server relays that information to other clients or shared authoritative in that the client uploads its current location and how it got there and the server validates if its with in possibility (Battlefield games post 4). The first allows for warping in that the client side hack allows the player to jump to any location while the later allows for speed running and some limited ghosting in that packet manipulation/delay/destroy can cause server updates to be sporadic and given server limits are typical slightly above what should be possible with in a normal action due to packet latency, server latency and when client update packets can fall on server ticks to prevent most legitimate actions from being rejected. So this reduces most of the possible movement cheats from player actions (does not prevent wall hacking and aim bots) it does put a significant load on the server and creates more room for latency introduced animation glitches like rubber banding.

From time to time I lurk into spectrum to see if, by any chance, there is some interesting discussion that is not another ranting tread abount thing being missing of bugs already well known... well, It seams I've found one.

Meshing The Servers - Star Citizen Spectrum

MESHING THE SERVERS In the Unity 3D engine we use a similar process to load/unload game objects in and out of the scene depending on what the player is looking at (Occlusion Culling) as well as...

robertsspaceindustries.com

This is an interesting techincal discussion about server meching carryed ouut by some guys that are clearly coompetent since they also work on game development and have to deal with the same kind of problem CIG is trying to solve with the server meshing solution.

I think this worth a reading since I found it very complementary and more detailed of the short explanation we have from ISC and SCL where the whole problem is condensed in few minute of video.

It is intresting but I would also caution reading anything in this thread to be taken with a large grain of salt as there is a far amount of speculation and others coming off as some sort of programing authority while either not articulating their ideas well or talking about things they have no clue about.
Take for example

https://robertsspaceindustries.com/spectrum/community/SC/forum/3/thread/meshing-the-servers/3744825"
Not really.

Server Meshing implies that the entire "game world" is split up into numerous "zone servers" (each a physically individual server) that deal with a certain region/area. This is generally what most MMOs do, as running an entire game world on one, lone server is highly inefficient and very CPU/memory intensive. Considering players can free travel, we have to assume that CIG will split the entire universe into equal-size "zones" that are handled by each of their one "zone servers". So let's say for the current "game world" we have in SC right now, there is ~64 zone servers.

While they are mostly correct in Server Meshing is spliting up the game world into zones with each zone being handled by a server they are incorrect in that the zones are equal-size as they have said the zones are going to be auto configured as players move around to keep server utilization with in a given margin. (last update I read from Clive said he still doesn't know how they are going to handle server zones based upon what region (rl) players are in given AWS has sever farms all over the world it would be possible to place the zone on a server located near the area the players in its zone physically are thus reducing network latency thus if you have 20 players from Australia all engaged in a fleet battle there is no reason to have them playing on a server located in Seattle)

Another thing that seems to be a common source of confusion is client side Object container streaming and Server side Object container streaming as being some sort of network traffic or server mesh technology when in fact in its simplest form is attempting to achieve a reduction of what items are currently residing in a computers RAM as there is no reason to retain in memory objects that are currently not in the game world. (this also creates the fun bugs of game objects not being rendered like pilots seats/doors/landing gear).

 

[Richard Bong]

I used to use Unity a lot, but not for developing games. For me it was scientific and industrial simulations. We still had the same problems though. It's very hard for me to describe in language that a common user can understand - and I haven't done anything with Unity for 2 1/2 years now. My god it's been a long time.


Yep, but the transfer of a user to another region in a game is almost instantaneous. What they're simulating is the distance between star systems, and the reason why is it's kind of odd to just kind of "go next door" and be in a different star system when we know they should be light years away. Its also extremely expensive and wasteful to make more empty space than you need. So why not invent some kind of FTL, call it a wormhole... and make it kind of an interesting place to be.

BTW - if someone wants to get started with game development or just to learn about it a bit, I recommend the community edition of Unity. C# is what it uses as a scripting language, but you can also extend the game by writing your own classes. You can use other languages other than C#, even C++ to do that. It's free to use, and anything you develop can be sold, shared or whatever - as long as you don't make over 100K/year, something like that.

Both Unity and Unreal would have been much better choices than Cryengine/Lumberyard. Their netcode is significantly better and offer a better base for world building.

 

[Bambooza]

Both Unity and Unreal would have been much better choices than Cryengine/Lumberyard. Their netcode is significantly better and offer a better base for world building.

I would agree, but we have to remember when this game was first envisioned and described to us as part of the Kickstarter event Cryengine was a great match as Unity was still far from the product it is today and Unreal was rather costly to license. As for the netcode it was envisioned not as this massive multiplayer it has become but as a smaller more co op freelancer experience. A lot of the core game engine elements they are struggling with is simply due to the vision changing due to the huge success of the game. In fact it is the curse of Kickstarter that a very successful campaign can cause the founders to try and find a way to improve on their original concept to utilize the increase revenue and it often causes the whole project to fail. With CIG it really caused them to spin their wheels for the first few years and it really wasn't until the release of 3.0 that the scope of the game really got described and the true understanding of what was required realized.

 

[Vavrik]

I would agree, but we have to remember when this game was first envisioned and described to us as part of the Kickstarter event Cryengine was a great match as Unity was still far from the product it is today and Unreal was rather costly to license. As for the netcode it was envisioned not as this massive multiplayer it has become but as a smaller more co op freelancer experience. A lot of the core game engine elements they are struggling with is simply due to the vision changing due to the huge success of the game. In fact it is the curse of Kickstarter that a very successful campaign can cause the founders to try and find a way to improve on their original concept to utilize the increase revenue and it often causes the whole project to fail. With CIG it really caused them to spin their wheels for the first few years and it really wasn't until the release of 3.0 that the scope of the game really got described and the true understanding of what was required realized.

I don't think CIG evaluated Unity at the time. I think they evaluated what they were familiar with, and that was CryEngine vs the cost of developing their own engine. KSP was first developed in Unity starting in 2011 (and is still) so it was clearly capable even in 2012/2013. Multi-user has been a bit problematic in Unity, but there have been ways of doing that for a long time. CIG has not overcome that hurdle yet, while some competitive games have.
THIS will be the acid test to see if Star Citizen is worthy. 50 players in a universe is not going to make this an interesting game.

 

[Michael]

I don't think CIG evaluated Unity at the time. I think they evaluated what they were familiar with, and that was CryEngine vs the cost of developing their own engine. KSP was first developed in Unity starting in 2011 (and is still) so it was clearly capable even in 2012/2013. Multi-user has been a bit problematic in Unity, but there have been ways of doing that for a long time. CIG has not overcome that hurdle yet, while some competitive games have.
THIS will be the acid test to see if Star Citizen is worthy. 50 players in a universe is not going to make this an interesting game.

i remember reading about backers which have been concerned about cryengine back then. But i'm pretty sure no one ever mentioned unity.

 

[Richard Bong]

I would agree, but we have to remember when this game was first envisioned and described to us as part of the Kickstarter event Cryengine was a great match as Unity was still far from the product it is today and Unreal was rather costly to license. As for the netcode it was envisioned not as this massive multiplayer it has become but as a smaller more co op freelancer experience. A lot of the core game engine elements they are struggling with is simply due to the vision changing due to the huge success of the game. In fact it is the curse of Kickstarter that a very successful campaign can cause the founders to try and find a way to improve on their original concept to utilize the increase revenue and it often causes the whole project to fail. With CIG it really caused them to spin their wheels for the first few years and it really wasn't until the release of 3.0 that the scope of the game really got described and the true understanding of what was required realized.

I was running QA for a 3D graphics company then. We were looking at physics, and getting our content into game engines back then.

In 2012 Unity was free for development and had much better netcode than CryEngine. (No change there.) Cryengine was not cheap then, in fact they wouldn't even give you the price in 2012, unless you asked for it and proved you were a large studio. (I couldn't get a copy for testing purposes.) Unreal was more expensive than Unity but was on par with Cryengine. Actual world building was, and still is, easier in Unity.

Cryengine was great for an FPS, but is not designed to do MMO. (No change there.)

That was also when Nvidia introduced real time cloth and hair simulation, and both Unreal and Unity adopted it.

 

[Bambooza]

I was running QA for a 3D graphics company then. We were looking at physics, and getting our content into game engines back then.

In 2012 Unity was free for development and had much better netcode than CryEngine. (No change there.) Cryengine was not cheap then, in fact they wouldn't even give you the price in 2012, unless you asked for it and proved you were a large studio. (I couldn't get a copy for testing purposes.) Unreal was more expensive than Unity but was on par with Cryengine. Actual world building was, and still is, easier in Unity.

Cryengine was great for an FPS, but is not designed to do MMO. (No change there.)

That was also when Nvidia introduced real time cloth and hair simulation, and both Unreal and Unity adopted it.

While Unity did have a free version from 2009 it was only for small development studios that made less then 10k gross revenue a year (I believe) and was greatly limited in what features it offered. Prior to 2009 it had a Indie version that was priced at 199. The only other offering as you mentioned at the time was Source engine the others required a sizeable contract commitment. Unreal became free March of 2015 (the year prior it was a monthly subscription of 19 a month) and Amazon released Lumberyard In May 2015 and shortly afterwards did Unity finally opened up its full engine for free to small development studios and hobbyist.

I would say even to this day Unity, Unreal and Cryengine (lumberyard) are not very functional as any sort of mmo out of the box. All of them are focused on the single player/ small multiplayer experience that loads a level (all of the assets used in the level are loaded into memory) prior to the start of the game. While Unity did allow for textures, audio and video to be loaded at runtime its mesh saving/ loading was a bit clunky and performance intensive. I believe all three now offer better real time mesh serialization but its a feature I have not had a need to spend any real time with.

I would also say that Unity has always been a bit behind the other engines when it comes to graphic fidelity and keeping up with graphic card optimizations. And back in 2012 CryEngine 3 which was released in Dec 2011 was graphically stunning.

We have the luxury now to look back and say the CryEngine perhaps wasn't the best engine to utilize for the current game CIG is making but back in 2012 it was a great fit for the game scope pitched.