Extensive explanation about what dynamic server meshing does

[Blind Owl]

I strongly advise against wearing all mesh while wandering outside on MicroTech, especially around NB. I believe that doing so would result in something similar to a flash freezing effect.

I'm Irish-Canadian; the cold will have no effect.

You can read into that any way you'd like. ;)

 

[Vavrik]

I'm Irish-Canadian; the cold will have no effect.

You can read into that any way you'd like. ;)

Well, if you've ever been to Antarctica...you know, while looking for beer... even as a Canadian of Irish descent that has an effect. But it won't shrink anything, it just runs and hides if you pull it out. Your balls end up somewhere near your stomach and your member will be hiding between your lungs screaming at you not to do that again. I say so as a fellow Canadian of Irish descent.

 

[Blind Owl]

Well, if you've ever been to Antarctica...you know, while looking for beer... even as a Canadian of Irish descent that has an effect. But it won't shrink anything, it just runs and hides if you pull it out. Your balls end up somewhere near your stomach and your member will be hiding between your lungs screaming at you not to do that again. I say so as a fellow Canadian of Irish descent.

Antarctica, no.
Arctic, yes. On a winter exercise. With outdoor bathrooms. When fluids are freezing as they hit the ground, it makes you wonder, "Why? Why am I here? Who did I piss off?" Lol.

 

[unobtanium]

Hello, I am the creator of the Unofficial Road to Dynamic Server Meshing presentation and just found out about this forum thread :)

After checking it out I'd be careful of the information presented as a lot of it is false or misleading.

I would like to clear up any misunderstandings in case my explanations were not precise enough and improve them in case that I worded some explanations too simply to the point where they could be understood in multiple ways and mislead more than they help. Thats the last thing I want happen when creating this presentation.

Like streaming entities in an out reduces the amount of entities that have to be computed by the client cpu. Culling and adding objects depending on what's in the client's sphere visual sphere does not decrease the cpu requirements in fact it generally increases the CPU utilization as it has to maintain the object trees in memory. All client-side object container streaming does is reduce the client's ram memory requirements as not all objects for the game world need to be loaded at game launch and can be added and removed at runtime. Its why games have load screens and even now most games do not utilize this method as it's demanding on the CPU and hard disk which consoles struggle with. You will also find that players who play SC and are not utilizing SSD drives will also be negatively impacted as they move around the verse.

The idea that all clients need to keep track of all objects on each game tick is also false without client container streaming. While servers do need to keep track of all objects on each tick clients are only feed entity changes within their sphere of influence. This is mostly due to bandwidth limitations and not due to cpu/memory limitations as updating object list location changes per tick is simple. This is why CIG was able to implement client-side object container streaming without needing to change the server code. All they did was change the client-side game engine to not load all objects until the client gets an update on an object id it currently does not have in memory.

First of all, i think it is important that we differentiate between the process of rendering on the GPU and the process of entity computation on the CPU. Because for both of them there exist the technique of "culling". For rendering, it means that some objects are not rendered if they are behind other objects (e.g. behind a wall). That saves on geometry that has to be rendered and thus rendering time that could be used for other geometry. For entity computation, culling allows objects to be loaded and unloaded based on object size and distance to the player. I think at this point many modern open world games already support this as well. (A few weeks ago I already added a few more introduction slides on this topic and the difference between CPU and GPU).

However, in a multiplayer setting, this becomes more difficult because usually the server expects that the clients stays in sync with the server (but I guess that also depends on the implementation, like does the client simulate the game world alongside the server and when the send server state is not same does the client considered it to be the ground truth and overwrite its own state. Or does the client not simulate the game world itself and just update its entities with the data provided by the server). So the server also has to know which objects are loaded and being computed on the client so that the server does not disconnect the client accidentally. Because of this, CSOCS is not a client-side only implementation. In fact, the server is the one that tells the clients which objects they need to load and unload as well as start and stop computing. The clients job is to do the loading and unloading process and start and stop the computing (or start and stop receiving updates send over the network) when the server tells it to. So CSOCS did not just change how things work on the client side but also on the server side. The server had to be okay for clients to not simulate the entire game world like the server does. So it needed to be aware of which entities are being loaded on which client and had to manage that. Clients do not do that on their own, they get all their info from the server.

Therefore, streaming entities on the client should reduce the amount of entities to only the relevant ones and thus also reduce the load on the client CPU as well. CSOCS has nothing to do with rendering culling on the GPU, only with entity culling on the CPU. CSOCS is much more than just reducing the clients memory usage. CSOCS also reduces networking tick rate of entities based on distance to the player to also reduce the load on the CPU. Dual Universe and their implementation does it similarly.

But (again) I guess some aspects depend on if the client does simulate the game world like the server does with occasional rollbacks (what we usually perceive as rubberbanding) if server and client start to go out of sync or if the client merely updates its entities based on the server state and does not simulate the entities itself as well. I think it is the former because it usually results in a better experience for the players (and also reduced network bandwidth because only the actions have to be send, not necessarily every single state inbetween. Its basically the client-side prediction technique). But maybe we are not there yet and the upcoming Actor Networking Rework would the first step toward that. I am indeed not entirely sure about that, so I posted a Ask the Devs post on Spectrum to have a dev clarify that.

Client-side object container streaming is also loading objects that are currently not visible to the client but are still nearby this prevents the situation of objects that peek around a corner from not showing up for a few seconds due to needing to be loaded before rendered. I do not remember if they are using a sphere or a teardrop for object load/culling.

That is true! I probably should elaborate on this in my presentation and highlight better that its not about direct visibility at all (but it is difficult to explain it without going into too much detail :S). Indeed all of the nearby entities are being loaded into memory in case they quickly become visible to the player and need to behave as expected. But I believe the engine still differentiates between loaded entities and computing entities and therefore not all entities that are loaded into memory also have to be computed (or receiving updates from the server).
I believe the server uses a Radix Tree and Geohashcodes for the Object Containers, so it can traverse the tree and quickly figure out which entities need to be loaded or unloaded. I would love to know more about how that algorithm works and how it factors in the size of objects as well :D

 

[maynard]

Hello, I am the creator of the Unofficial Road to Dynamic Server Meshing presentation ...

thanks for taking the time and effort to put that together

it's too much for me to take in all at once ( I am a TESTie after all )

I will read it in pieces over time

 

[Mich Angel]

And..

CHEERS!

 

[NaffNaffBobFace]

Hello, I am the creator of the Unofficial Road to Dynamic Server Meshing presentation and just found out about this forum thread :)




I would like to clear up any misunderstandings in case my explanations were not precise enough and improve them in case that I worded some explanations too simply to the point where they could be understood in multiple ways and mislead more than they help. Thats the last thing I want happen when creating this presentation.




First of all, i think it is important that we differentiate between the process of rendering on the GPU and the process of entity computation on the CPU. Because for both of them there exist the technique of "culling". For rendering, it means that some objects are not rendered if they are behind other objects (e.g. behind a wall). That saves on geometry that has to be rendered and thus rendering time that could be used for other geometry. For entity computation, culling allows objects to be loaded and unloaded based on object size and distance to the player. I think at this point many modern open world games already support this as well. (A few weeks ago I already added a few more introduction slides on this topic and the difference between CPU and GPU).

However, in a multiplayer setting, this becomes more difficult because usually the server expects that the clients stays in sync with the server (but I guess that also depends on the implementation, like does the client simulate the game world alongside the server and when the send server state is not same does the client considered it to be the ground truth and overwrite its own state. Or does the client not simulate the game world itself and just update its entities with the data provided by the server). So the server also has to know which objects are loaded and being computed on the client so that the server does not disconnect the client accidentally. Because of this, CSOCS is not a client-side only implementation. In fact, the server is the one that tells the clients which objects they need to load and unload as well as start and stop computing. The clients job is to do the loading and unloading process and start and stop the computing (or start and stop receiving updates send over the network) when the server tells it to. So CSOCS did not just change how things work on the client side but also on the server side. The server had to be okay for clients to not simulate the entire game world like the server does. So it needed to be aware of which entities are being loaded on which client and had to manage that. Clients do not do that on their own, they get all their info from the server.

Therefore, streaming entities on the client should reduce the amount of entities to only the relevant ones and thus also reduce the load on the client CPU as well. CSOCS has nothing to do with rendering culling on the GPU, only with entity culling on the CPU. CSOCS is much more than just reducing the clients memory usage. CSOCS also reduces networking tick rate of entities based on distance to the player to also reduce the load on the CPU. Dual Universe and their implementation does it similarly.

But (again) I guess some aspects depend on if the client does simulate the game world like the server does with occasional rollbacks (what we usually perceive as rubberbanding) if server and client start to go out of sync or if the client merely updates its entities based on the server state and does not simulate the entities itself as well. I think it is the former because it usually results in a better experience for the players (and also reduced network bandwidth because only the actions have to be send, not necessarily every single state inbetween. Its basically the client-side prediction technique). But maybe we are not there yet and the upcoming Actor Networking Rework would the first step toward that. I am indeed not entirely sure about that, so I posted a Ask the Devs post on Spectrum to have a dev clarify that.




That is true! I probably should elaborate on this in my presentation and highlight better that its not about direct visibility at all (but it is difficult to explain it without going into too much detail :S). Indeed all of the nearby entities are being loaded into memory in case they quickly become visible to the player and need to behave as expected. But I believe the engine still differentiates between loaded entities and computing entities and therefore not all entities that are loaded into memory also have to be computed (or receiving updates from the server).
I believe the server uses a Radix Tree and Geohashcodes for the Object Containers, so it can traverse the tree and quickly figure out which entities need to be loaded or unloaded. I would love to know more about how that algorithm works and how it factors in the size of objects as well :D

Welcome to TEST, thanks for taking the time to reach out to us once you found we'd been talking about this.

It was a TL;DR for me even condensed and with pictures so no further questions from myself :D

 

[Bambooza]

Hello, I am the creator of the Unofficial Road to Dynamic Server Meshing presentation and just found out about this forum thread :)

Hello and welcome, here have a beer.

First of all, i think it is important that we differentiate between the process of rendering on the GPU and the process of entity computation on the CPU. Because for both of them there exist the technique of "culling". For rendering, it means that some objects are not rendered if they are behind other objects (e.g. behind a wall). That saves on geometry that has to be rendered and thus rendering time that could be used for other geometry. For entity computation, culling allows objects to be loaded and unloaded based on object size and distance to the player. I think at this point many modern open world games already support this as well. (A few weeks ago I already added a few more introduction slides on this topic and the difference between CPU and GPU).

I mostly agree. I know of a lot of open worlds that do not do object streaming but instead either rely on simpler geometric shapes or hide level loading behind transitional points. No Man's Sky is a prime example of both with the level loading happening when you warp and simpler geometric shapes to cut back on the memory requirements.

However, in a multiplayer setting, this becomes more difficult because usually the server expects that the clients stays in sync with the server (but I guess that also depends on the implementation, like does the client simulate the game world alongside the server and when the send server state is not same does the client considered it to be the ground truth and overwrite its own state. Or does the client not simulate the game world itself and just update its entities with the data provided by the server).

All clients run the game world simulation (why there is wall hacks) due to latency issues and how it would impact the graphic updates and control input. As to if its server side authoritative or not depends on what happens when the game world state is out of sync. A lot of console games are client authoritative as it cuts back on the server costs needed to support the game (allows for the wonderful cheats of speed hacks, warping and insta kills). Star Citizen is server side authoritative.

So the server also has to know which objects are loaded and being computed on the client so that the server does not disconnect the client accidentally. Because of this, CSOCS is not a client-side only implementation. In fact, the server is the one that tells the clients which objects they need to load and unload as well as start and stop computing. The clients job is to do the loading and unloading process and start and stop the computing (or start and stop receiving updates send over the network) when the server tells it to. So CSOCS did not just change how things work on the client side but also on the server side. The server had to be okay for clients to not simulate the entire game world like the server does. So it needed to be aware of which entities are being loaded on which client and had to manage that. Clients do not do that on their own, they get all their info from the server.

Therefore, streaming entities on the client should reduce the amount of entities to only the relevant ones and thus also reduce the load on the client CPU as well. CSOCS has nothing to do with rendering culling on the GPU, only with entity culling on the CPU. CSOCS is much more than just reducing the clients memory usage. CSOCS also reduces networking tick rate of entities based on distance to the player to also reduce the load on the CPU. Dual Universe and their implementation does it similarly.

But (again) I guess some aspects depend on if the client does simulate the game world like the server does with occasional rollbacks (what we usually perceive as rubberbanding) if server and client start to go out of sync or if the client merely updates its entities based on the server state and does not simulate the entities itself as well. I think it is the former because it usually results in a better experience for the players. But maybe we are not there yet and the upcoming Actor Networking Rework would the first step toward that. I am indeed not entirely sure about that, so I posted a Ask the Devs post on Spectrum to have a dev clarify that.

CSOCS is fully client side and while it is not rendering culling its only purpose is in maintain what game objects are maintained in memory for both the game-world and graphic pipeline access (its mostly the render maps that take up the most memory, but its easier to maintain the object container then to try and only load/dump the graphic maps). The server is not telling each client which object to load or unload it is up to the client side object container to manage the objects it is maintain in memory. Fixed placement objects like space stations are not part of the server/client communication as their place in the world does not change nor all of the objects attached to it. While CSOCS will load and unload the station object container depending on the current player location it is not being instructed to do so by the server. What the server is doing is sending the current location cords of all moveable objects that have had a delta change since last update and until recently it was the full player universe. Now the server knows how to only provide updates on moveable objects with in each players sphere thus reducing the amount of bandwidth required (Network Bind Culling introduced in 3.1).

CSOCS has nothing to do with network game update rate (tick rate) its only task is memory management of game objects. Client update of moveable objects and their frequency is controlled by Network Bind Culling. The current game world update rate from the server is set at 30hz and its still unknown if there is a varying update of far away objects or if there will be. Currently all moveable objects with in your clients known sphere is being updated at 30hz.

Rubber-banding of the client is caused by the client updates being rejected by the server in a server authorization model, typically caused by drop packets, slow network updates or a server resource issue causing it to run slower. Rubber-banding/Warping of moveable objects can be caused by the client dropping update packets from the server which causes issues with the client side prediction algorithms. The other cause is game-world out of sync with other clients in a client authorization model but is not relevant to Star Citizen.

That is true! I probably should elaborate on this in my presentation and highlight better that its not about direct visibility at all (but it is difficult to explain it without going into too much detail :S). Indeed all of the nearby entities are being loaded into memory in case they quickly become visible to the player and need to behave as expected. But I believe the engine still differentiates between loaded entities and computing entities and therefore not all entities that are loaded into memory also have to be computed (or receiving updates from the server).
I believe the server uses a Radix Tree and Geohashcodes for the Object Containers, so it can traverse the tree and quickly figure out which entities need to be loaded or unloaded. I would love to know more about how that algorithm works and how it factors in the size of objects as well :D

CSOCS makes no distinction between different parts of the game object simply if the game object is close enough to the players location to be loaded into memory it is up to the different pipelines of the game engine to determine if, when and how its used. I agree it would be interesting to know the details of the algorithm used and its impact on hardware requirements and game play.

 

[August]

 

[unobtanium]

Hello and welcome, here have a beer.

Thanks for the warm welcomes, everyone!



Let me attempt to respond to some of your paragraphs with ones of my own :'D

I mostly agree. I know of a lot of open worlds that do not do object streaming but instead either rely on simpler geometric shapes or hide level loading behind transitional points. No Man's Sky is a prime example of both with the level loading happening when you warp and simpler geometric shapes to cut back on the memory requirements.

True, Mass Effect elevator loading screens come to mind as well :D
Yep, LODs are usually heavily used in open world games. The further away objects are, the less geometry is needed and thus more geometry can be used for the closer objects again.

All clients run the game world simulation (why there is wall hacks) due to latency issues and how it would impact the graphic updates and control input. As to if its server side authoritative or not depends on what happens when the game world state is out of sync. A lot of console games are client authoritative as it cuts back on the server costs needed to support the game (allows for the wonderful cheats of speed hacks, warping and insta kills). Star Citizen is server side authoritative.

I am not actually sure if its server side authoritative already for all component types. Indeed, the Actor Networking Rework Upstream Parts main purpose was to make player and NPC movement server side authoritative because back then it was still client side authoritative. So I wonder if all the other parts are already server side authoritive as well or if they only will follow after the Actor one is rolled out.

It does not effect ship or weapon networking, they're different enough that they use different systems (though they do all have some common elements/strategies).

CSOCS has nothing to do with network game update rate (tick rate) its only task is memory management of game objects. Client update of moveable objects and their frequency is controlled by Network Bind Culling. The current game world update rate from the server is set at 30hz and its still unknown if there is a varying update of far away objects or if there will be. Currently all moveable objects with in your clients known sphere is being updated at 30hz.

Hm, I am fairly certain there already are varying updates of components based on distance of entities to the player. From the JumpPoint article (not sure if I can share the link publically since its subscriber content :S):

The Entity-Component-Update-Scheduler is a system designed to control the update of entity components, based on how they are spatially placed relative to the player. By doing this, we can skip updates of components which are too far away (another benefit of the component design, we can do the update policy on a per component basis).

Also we have the Serialized Variables API which allows the developers to set specific policies to components so that some only send updates when the state changes. But some states most likely will be ineded sending state the entire time, no matter if they change or not (most likely those who change a lot like players).

CSOCS is fully client side and while it is not rendering culling its only purpose is in maintain what game objects are maintained in memory for both the game-world and graphic pipeline access (its mostly the render maps that take up the most memory, but its easier to maintain the object container then to try and only load/dump the graphic maps).

Once again, I am fairly certain, that CSOCS is not a client-side-only feature. It most definitely was about improving the performance on the client, just like SSOCS was ment to improve the performance on the server. Still, both features required massive changes to the server code to have it do new stuff. So on the code-side of things, the server code most likely had the most changes to it.

While that system [Entity-Component-Update-Scheduler] helped with server performance, clients still had to pay an unnecessary cost; we didn’t yet have a system in place to decide which client requires which information. On top of this, each component had to run its update when it received new network information, resulting in a measurable performance cost.
[...], we decided to build a system that would only send network updates to clients if that client is in proximity of the updated Entity-Aggregate.

That last functionality is what Serialized Variable Culling does. This should confirm that the server keeps track of which clients require which information/entity state changes.

The server is not telling each client which object to load or unload it is up to the client side object container to manage the objects it is maintain in memory. Fixed placement objects like space stations are not part of the server/client communication as their place in the world does not change nor all of the objects attached to it. While CSOCS will load and unload the station object container depending on the current player location it is not being instructed to do so by the server. What the server is doing is sending the current location cords of all moveable objects that have had a delta change since last update and until recently it was the full player universe.

Hm, why would they differentiate between fixed-placed and dynamic-placed entities? Especially since they still plan to have planets orbit the sun and thus space stations would have to change their position relative to the planet Object Container they are in. With almost everything being physicalized soon and everything having damage states which will be persisted in the iCache databases, it most definitely will have to be required for the server to send data of all entities to the client, no matter if they are stationary or moving. Thus, wouldnt it have been much simpler to already implement a feature that gives the server full reign of which Object Containers and entities the client should load or unload? Why would they make a more complex distinction when they know that they wont need it anymore later? I mean, maybe they did that as a workaround at first, but even then I am not sure why they wouldnt just do it the required way right away. It doesnt seem like a lot of additional work to me. Why implement two different systems when you can just implement just one that handles everything?

The server has the database of all of the entities which were unloaded when no players were around. How would the client know which objects to load when they are stored in the database on the server? Of course, the server just tells the client which Object Containers to load (its not actually sending the Object Containers itself, we download those and their resources with the patches), but the server will tell the client which states the entities of those Object Containers have to sync them up.

Now the server knows how to only provide updates on moveable objects with in each players sphere thus reducing the amount of bandwidth required (Network Bind Culling introduced in 3.1).

In the JumpPoint article on OCS the devs wrote that Network Bind Culling was introduced in 3.3. I think it was just the term the developers used for CSOCS. I think, the term CSOCS came a little bit later, probably to better distinguish Client Side from Server Side OCS.

Network-Bind-Culling was released to the public in Alpha 3.3

For many clients, there was a very large performance improvement when we released Network-Bind-Culling to the public with Alpha 3.3.

However, the Serialized Variable Culling feature (which kind of is a big part of CSOCS) released in Alpha 3.1 (that features functionality might be what they refered to as Network Bind Culling while still in development and then they renamed both which definitely would be confusing if thats what they did):

We wanted to ship this feature [Serialized Variables Culling] with Alpha 3.0, but in defiance of the heroic effort of the network team, we had to let this feature slip into Alpha 3.1.

Rubber-banding of the client is caused by the client updates being rejected by the server in a server authorization model, typically caused by drop packets, slow network updates or a server resource issue causing it to run slower. Rubber-banding/Warping of moveable objects can be caused by the client dropping update packets from the server which causes issues with the client side prediction algorithms. The other cause is game-world out of sync with other clients in a client authorization model but is not relevant to Star Citizen.

Yep. But pretty sure in a client authoritative model (not peer-to-peer) its mainly caused by other players lagging. Thats how it worked before the Upstream Actor Networking Rework released.

 

[Bambooza]

I am not actually sure if its server side authoritative already for all component types. Indeed, the Actor Networking Rework Upstream Parts main purpose was to make player and NPC movement server side authoritative because back then it was still client side authoritative. So I wonder if all the other parts are already server side authoritive as well or if they only will follow after the Actor one is rolled out.

NPC's by their very existence are subroutines run on the servers which is why when the server performance is taxed the NPC's behaviour goes wonky as they are not given enough cpu cycles to perform their logic lookup trees and act accordingly.

I had forgotten that Gordon said of Feb 2019 client movement was client authoritative but I believe that has been changed a few versions ago (3.5).

Hm, I am fairly certain there already are varying updates of components based on distance of entities to the player. From the JumpPoint article (not sure if I can share the link publically since its subscriber content :S):

Also we have the Serialized Variables API which allows the developers to set specific policies to components so that some only send updates when the state changes. But some states most likely will be ineded sending state the entire time, no matter if they change or not (most likely those who change a lot like players).

No clue if it's in the game yet or not I do know it was at one point on the road map but it seems to have disappeared. I could be its slowly just being implemented with other features or it could be that it's still on the todo list. Either way, it will help with the network requirements in that not all things need 30 updates a second sent to every client

Once again, I am fairly certain, that CSOCS is not a client-side-only feature. It most definitely was about improving the performance on the client, just like SSOCS was ment to improve the performance on the server. Still, both features required massive changes to the server code to have it do new stuff. So on the code-side of things, the server code most likely had the most changes to it.

Yes, CSOCS and SSOCS are all about performance improvements by managing what objects are or will be needed shortly. CSOCS is the client-side of the performance improvement there were some sever side updates because the client is no longer keeping in sync the whole game world and needs more information when unserializing objects (mostly about the current state of said objects).

Object Container Streaming

Object Container Streaming (OCS) is the umbrella term for all the technology in Star Engine that makes a vast seamless universe possible, by which Star Citizen...

starcitizen.tools

And from CIG_ChadMcKinney:

[To improve performance] by far the most important thing is to get object container streaming working. Object container streaming will allow us to only have a subset of entities on the client and server that are the minimum required for gameplay, which should dramatically reduce update time as well as a fair amount of memory usage. It will also make other game code behavior such as zone queries faster by making these queries operate on a smaller set of potential entities.

That last functionality is what Serialized Variable Culling does. This should confirm that the server keeps track of which clients require which information/entity state changes.

Hm, why would they differentiate between fixed-placed and dynamic-placed entities? Especially since they still plan to have planets orbit the sun and thus space stations would have to change their position relative to the planet Object Container they are in. With almost everything being physicalized soon and everything having damage states which will be persisted in the iCache databases, it most definitely will have to be required for the server to send data of all entities to the client, no matter if they are stationary or moving. Thus, wouldnt it have been much simpler to already implement a feature that gives the server full reign of which Object Containers and entities the client should load or unload? Why would they make a more complex distinction when they know that they wont need it anymore later? I mean, maybe they did that as a workaround at first, but even then I am not sure why they wouldnt just do it the required way right away. It doesnt seem like a lot of additional work to me. Why implement two different systems when you can just implement just one that handles everything?

the reason why you have a differentiation between fixed place and dynamic placed entities is because of how much network bandwidth you have available. Let's take your example of a space station. It is correct the parent object to the station might be dynamic as it orbits the planet but all of the objects within the station do not have their coordinates relative to their absolute position within the system. Their location is fixed relative to the position of the parent and since they cannot be destroyed or modified in any way they are fixed and no longer need any updates by the server or client.

Let's take the below picture as an example. While the planet spins the hab location on the planet is fixed and immutable (cannot be changed) doesn't matter what we do to it or how many test pilots crash into it. Within the hab is thousands of objects from the grow beds to the microscope lights and screens. Each of these objects has a fixed place within their parent object but their current location or condition does not need to be updated by the server. In fact, if you look at any of the animations on the screens even though are typically not synched by the server and if you had two players in the same place and looked at the same screen they most likely will see the animation script playing at different points along its loop.

In fact, this gets to the fundamental differences between the client-side and the server-side. The client-side when approaching a moving station gets the current stations' absolute position from the server and any dynamic parent objects within that location so it can load them into memory. The server not only needs to load the same objects into memory but it needs to run update statements on all of the dynamic objects since they were last serialized like moving the station to the correct location updating all of the npc locations within the station based upon their schedules.

The server has the database of all of the entities which were unloaded when no players were around. How would the client know which objects to load when they are stored in the database on the server? Of course, the server just tells the client which Object Containers to load (its not actually sending the Object Containers itself, we download those and their resources with the patches), but the server will tell the client which states the entities of those Object Containers have to sync them up.


In the JumpPoint article on OCS the devs wrote that Network Bind Culling was introduced in 3.3. I think it was just the term the developers used for CSOCS. I think, the term CSOCS came a little bit later, probably to better distinguish Client Side from Server Side OCS.


However, the Serialized Variable Culling feature (which kind of is a big part of CSOCS) released in Alpha 3.1 (that features functionality might be what they refered to as Network Bind Culling while still in development and then they renamed both which definitely would be confusing if thats what they did):

The only thing the server has to update the client on is the placement of dynamic objects within the client's sphere. The client already has most object relative positions like all of the objects within an 890Jump just not the parent current location and vector.

Bind Culling purpose is to limit the number of network updates each client receives. If I am drinking water on Port Olisar and you are on Hurston engaging with the seedy underbelly. You do not need location updates and script start and stop updates about what I am doing as I do not need to know the NPC locations, bullet location updates of what you are doing. Prior to Bind Culling ever dynamic object update was shared with every player on the server. Now your client only gets updates on the objects within its sphere, which greatly reduces the amount of network bandwidth needed while slightly increasing the CPU cycles needed on the server.

Serialized Variable Culling is a big feature of CSOCS and SSOCS as its what allows them to load and unload objects and their child objects. Just like the hab example above let's say we leave the hab and fly back into space. While the Planet object will still reside in memory the Hab which is a child of the planet is no longer needed and all of the children objects within the hab are also no longer needed so they are culled. And because you where the last player in the area the server is also able to serialize any state changes and placement of any dynamic objects left in the hab like the penguin and save those off to the database.

So they are two different things that act on different parts of the game engine. Bind Network Culling reduces the updates sent to each player to what the player needs to know about based upon their sphere. Serialized Variable Culling is used in the unloading of objects in memory that are not being used at that moment in time.

 

[Blind Owl]

Holy fuck. I was going to read all this, but realized I don't know how to. So I just liked everything and now I'm leaving to drink beer.

 

[Bambooza]

Holy fuck. I was going to read all this, but realized I don't know how to. So I just liked everything and now I'm leaving to drink beer.

Its ok have another beer. But if you want to try and understand or have any questions do feel free to ask.

 

[Gotin]

gah, corona.....why'd it have to be corona....that poor, poor woman must be employed in some hipster bar :(

 

[Gotin]

Soooo......does any of this explain why the aquariums on the 890 have no fish?


......that really bothers me. How can you have vodka aquariums if there are no fish in them?

 

[Bambooza]

Soooo......does any of this explain why the aquariums on the 890 have no fish?


......that really bothers me. How can you have vodka aquariums if there are no fish in them?

Because someone keeps trying to replace the water with beer thinking the fish need to drink too.

 

[Gotin]

Because someone keeps trying to replace the water with beer thinking the fish need to drink too.

Look....if I'm going to pay that much for a ship it had better come with some genetically engineered beer fish damit.

 

[Bambooza]

Now I want some beer-battered fish and chips.

 

[Blind Owl]

Its ok have another beer. But if you want to try and understand or have any questions do feel free to ask.

Jokes aside, I'm quite interested to sit and read through this when time allows. And when I do, I most certainly shall reach out with any questions I do have.

Cheers brother.

 

[unobtanium]

NPC's by their very existence are subroutines run on the servers which is why when the server performance is taxed the NPC's behaviour goes wonky as they are not given enough cpu cycles to perform their logic lookup trees and act accordingly.

True, the NPC logic is being run on the server side. Hm, I do wonder how much information is being send to the client. Theoretically, the server wouldnt have to constantly send the position of each NPC to the client, it would just have to send the initial actions of the NPC. Like for example if the NPC decides to go from A to B, then the server can just tell the client that and the client can execute that action on its own. If new action is send from the server then the client will adapt the behavior of the NPC accordingly. But both NPCs and players are actors they will run on the same logic, I am not sure if thats how it will work. It might indeed just send the position and other data of each NPC on each tick. Anyways, I digress...

I had forgotten that Gordon said of Feb 2019 client movement was client authoritative but I believe that has been changed a few versions ago (3.5).

Yeah, it definitely was already changed to server authoritative with the Upstream part of the Actor Networking Rework. On the official roadmap is says it was released with Alpha 3.6. I just checked again in my presentation, it seems I made a typo and wrongly wrote Alpha 3.5. Fixed.

No clue if it's in the game yet or not I do know it was at one point on the road map but it seems to have disappeared. I could be its slowly just being implemented with other features or it could be that it's still on the todo list. Either way, it will help with the network requirements in that not all things need 30 updates a second sent to every client

I am very certain that Serialized Variable Culling is already in the game. They said in the JumpPoint article it was released with Alpha 3.1. Serialized Variables was most likely released with Object Containers in 2017 already. But I am sure they kept adding functionality afterwards as well.

Yes, CSOCS and SSOCS are all about performance improvements by managing what objects are or will be needed shortly. CSOCS is the client-side of the performance improvement there were some sever side updates because the client is no longer keeping in sync the whole game world and needs more information when unserializing objects (mostly about the current state of said objects).

I still believe that there were more server side code changes than client side code changes for CSOCS. With Object Containers they were already able to stream Object Containers from drive into memory on the client. They mostly had to put most of the logic in place for the server to send the correct data and the client just merely had to receive it and handle it. Therefore, most of the ground work for the client for CSOCS was already done with Object Containers and the main work for CSOCS was happening on the server code.

Btw, we got a response from Chad McKinney (kinda at least, its still not very definitive in my eyes):

Spectrum - v7.3.0

RSI’s Spectrum is our integrated community and player interaction service, including chat, forums, game integration, and Player Organization facilities. Player Organizations (“Org”...

robertsspaceindustries.com

It does sound like that thanks to the Entity Component Update Scheduler certain entity components can reduce their tick rate depending on set policies, like distance to players, visiblility and more. And both client and server will share the same tick rate.
However, he wasnt very clear yet how much of the game is determined by the server and how much the client is simulating alongside the server.

the reason why you have a differentiation between fixed place and dynamic placed entities is because of how much network bandwidth you have available. Let's take your example of a space station. It is correct the parent object to the station might be dynamic as it orbits the planet but all of the objects within the station do not have their coordinates relative to their absolute position within the system. Their location is fixed relative to the position of the parent and since they cannot be destroyed or modified in any way they are fixed and no longer need any updates by the server or client.

Let's take the below picture as an example. While the planet spins the hab location on the planet is fixed and immutable (cannot be changed) doesn't matter what we do to it or how many test pilots crash into it. Within the hab is thousands of objects from the grow beds to the microscope lights and screens. Each of these objects has a fixed place within their parent object but their current location or condition does not need to be updated by the server. In fact, if you look at any of the animations on the screens even though are typically not synched by the server and if you had two players in the same place and looked at the same screen they most likely will see the animation script playing at different points along its loop.

In fact, this gets to the fundamental differences between the client-side and the server-side. The client-side when approaching a moving station gets the current stations' absolute position from the server and any dynamic parent objects within that location so it can load them into memory. The server not only needs to load the same objects into memory but it needs to run update statements on all of the dynamic objects since they were last serialized like moving the station to the correct location updating all of the npc locations within the station based upon their schedules.

The only thing the server has to update the client on is the placement of dynamic objects within the client's sphere. The client already has most object relative positions like all of the objects within an 890Jump just not the parent current location and vector.

I agree that static objects do not have to be syncronized. Afterall, they dont change (any relevant) state, so there wouldnt need to be any communicate for anything.

However, we know that all assests (like each table) are wrapped in their own Object Container, no matter if it is static or dynamic. For base building and housing in the future, we need almost everything to be dynamic objects (e.g. for push and pull functionality). We know that each Object Container or entity has its own id, probably 64 bits in size. Therefore, they could just send the unique id of the Object Container plus the absolute/relative position and rotation to the client for each one. Thats shouldnt be a lot of data. 64bits*7= 448 bits per Object Container. Compared to the actual data being networked of the more complex entities, that should be nothing. I think it would be feasible to load each and every Object Container over the network.

I actually did a test to see how much network bandwidth the game actually requires and played the game solo for 90 minutes. Spawned at new babbage, ran around for a bit, then spawned a ship, flew to a few moons, did some deliveries and returned to tessler spaceport. Only at the initial loading screen the download jumped above 100kb/sec, at New Babbage and the Spaceport it stayed around 60-80kb/sec and on moons it was around 40kb/sec. I would like to see a similar test with some combat with multiple players and ships and see how that impacts the bandwidth.

I created another Spectrum post and asked about it. Someone commented that people were able to run the PU in offline mode and indeed large Object Containers like planets, spacestations and cities are being loaded, some other entities are not or very limited. So that probably confirms that you were indeed correct. A lot of object containers are not being streamed in from the server. I still suspect that that might change in the far future, like when planets start orbiting and pretty much every object and item has states.

Is the current PU on the client loaded from a level file or all streamed from the server? - Star Citizen Spectrum

So I recently had a discussion with someone and we were unsure if the server only needs to tell the client to load dynamic objects (entities) and not static objects (for example a spacestation), or...

robertsspaceindustries.com

Bind Culling purpose is to limit the number of network updates each client receives. If I am drinking water on Port Olisar and you are on Hurston engaging with the seedy underbelly. You do not need location updates and script start and stop updates about what I am doing as I do not need to know the NPC locations, bullet location updates of what you are doing. Prior to Bind Culling ever dynamic object update was shared with every player on the server. Now your client only gets updates on the objects within its sphere, which greatly reduces the amount of network bandwidth needed while slightly increasing the CPU cycles needed on the server.

Serialized Variable Culling is a big feature of CSOCS and SSOCS as its what allows them to load and unload objects and their child objects. Just like the hab example above let's say we leave the hab and fly back into space. While the Planet object will still reside in memory the Hab which is a child of the planet is no longer needed and all of the children objects within the hab are also no longer needed so they are culled. And because you where the last player in the area the server is also able to serialize any state changes and placement of any dynamic objects left in the hab like the penguin and save those off to the database.

So they are two different things that act on different parts of the game engine. Bind Network Culling reduces the updates sent to each player to what the player needs to know about based upon their sphere. Serialized Variable Culling is used in the unloading of objects in memory that are not being used at that moment in time.

Yep.
I am slightly confused, is that not how I explained the technologies in the presentation? Were the explanations not precise enough or are it easy to be misunderstood? Do some of the explanations need changes? Just making sure in case I misrepresented something :D