Extensive explanation about what dynamic server meshing does

[Bambooza]

True, the NPC logic is being run on the server side. Hm, I do wonder how much information is being send to the client. Theoretically, the server wouldnt have to constantly send the position of each NPC to the client, it would just have to send the initial actions of the NPC. Like for example if the NPC decides to go from A to B, then the server can just tell the client that and the client can execute that action on its own. If new action is send from the server then the client will adapt the behavior of the NPC accordingly. But both NPCs and players are actors they will run on the same logic, I am not sure if thats how it will work. It might indeed just send the position and other data of each NPC on each tick. Anyways, I digress...

Just like every other dynamic object that has a delta change since the last tick update. (30 times a second) .

I am very certain that Serialized Variable Culling is already in the game. They said in the JumpPoint article it was released with Alpha 3.1. Serialized Variables was most likely released with Object Containers in 2017 already. But I am sure they kept adding functionality afterwards as well.

Serialized Variable Culling is part of the object container stream. It's the mechanism for moving the object out of memory into the database. The serialization of an object is simply taking all of the member objects of a class and storing the information into a byte stream so that it can be easily transferred. In this case, it has two parts. One is the serialization and the culling is iterating over the object container tree to remove objects no longer needed.

I still believe that there were more server side code changes than client side code changes for CSOCS. With Object Containers they were already able to stream Object Containers from drive into memory on the client. They mostly had to put most of the logic in place for the server to send the correct data and the client just merely had to receive it and handle it. Therefore, most of the ground work for the client for CSOCS was already done with Object Containers and the main work for CSOCS was happening on the server code.

Lumberyard did not originally have a mechanism for streaming objects in and out of memory. It like Unity loads all of the objects for a level at level start into memory. Without talking to the dev's involved or reviewing the code changes I do not have the specifics but given my understanding of game engines, I would wager the client-side object container stream was purely client while the server-side was purely server and given the server is keeper of the game state would require a lot more work as it needs a way to update objects that are reloaded. The client-side can simply rely on the server updates to reposition objects it had removed from memory.

Btw, we got a response from Chad McKinney (kinda at least, its still not very definitive in my eyes):
[URL [unfurl="true"]https://robertsspaceindustries.com/spectrum/community/SC/forum/50259/thread/client-side-ocs-reduced-tick-rates-and-entity-stat/3372301/URL]
It does sound like that thanks to the Entity Component Update Scheduler certain entity components can reduce their tick rate depending on set policies, like distance to players, visiblility and more. And both client and server will share the same tick rate.
However, he wasnt very clear yet how much of the game is determined by the server and how much the client is simulating alongside the server.

The variable tick rate that Chad is talking about is the dynamic object updates to the client. The client's update to the server is limited to the inputs the user issues and that needs to be as often as possible. Any mouse movement, keyboard input, joystick action, face tracking, etc. The server updates the client with all of the game world changes that are relevant to the client (network culling limits, prior it was all objects on that server). The second part of Cad's message is mostly about how the client displays the game world and how they blend animations and object updates to try and make it appear as smooth as possible especially with any sort of network issues between server and client.

I agree that static objects do not have to be syncronized. Afterall, they dont change (any relevant) state, so there wouldnt need to be any communicate for anything.

However, we know that all assests (like each table) are wrapped in their own Object Container, no matter if it is static or dynamic. For base building and housing in the future, we need almost everything to be dynamic objects (e.g. for push and pull functionality). We know that each Object Container or entity has its own id, probably 64 bits in size. Therefore, they could just send the unique id of the Object Container plus the absolute/relative position and rotation to the client for each one. Thats shouldnt be a lot of data. 64bits*7= 448 bits per Object Container. Compared to the actual data being networked of the more complex entities, that should be nothing. I think it would be feasible to load each and every Object Container over the network.

The easiest way to think about it is if a player can move it then its a dynamic object. Pico is a dynamic object. You can pick up the plushie and move it around, take it with you and place it in the co-pilot seat of your ship.

In fact, Unity Manual has a pretty well-written article for Network GameObjects

Unity - Manual: Networked GameObjects

docs.unity3d.com

Star Citizen gets a little more complex due to the fact that the ship itself is a dynamic object but the objects within the ship for the most part are static. Dynamic objects typically also include any object that can be player customized like the ship seats even if they don't move relative to the ship.

I actually did a test to see how much network bandwidth the game actually requires and played the game solo for 90 minutes. Spawned at new babbage, ran around for a bit, then spawned a ship, flew to a few moons, did some deliveries and returned to tessler spaceport. Only at the initial loading screen the download jumped above 100kb/sec, at New Babbage and the Spaceport it stayed around 60-80kb/sec and on moons it was around 40kb/sec. I would like to see a similar test with some combat with multiple players and ships and see how that impacts the bandwidth.

View attachment 18671

I created another Spectrum post and asked about it. Someone commented that people were able to run the PU in offline mode and indeed large Object Containers like planets, spacestations and cities are being loaded, some other entities are not or very limited. So that probably confirms that you were indeed correct. A lot of object containers are not being streamed in from the server. I still suspect that that might change in the far future, like when planets start orbiting and pretty much every object and item has states.

Is the current PU on the client loaded from a level file or all streamed from the server? - Star Citizen Spectrum

So I recently had a discussion with someone and we were unsure if the server only needs to tell the client to load dynamic objects (entities) and not static objects (for example a spacestation), or...

robertsspaceindustries.com

Remember the server only gives the current state of the object not what the object looks like. So any static objects the client would have no issue in loading. Any of the dynamic objects state the client would rely on the server to tell it where the object is and without the server, it would never load in the object.

Yep.
I am slightly confused, is that not how I explained the technologies in the presentation? Were the explanations not precise enough or are it easy to be misunderstood? Do some of the explanations need changes? Just making sure in case I misrepresented something :D

I'd have to go back over the original post again.

 

[unobtanium]

Just like every other dynamic object that has a delta change since the last tick update. (30 times a second) .

Yes.

Serialized Variable Culling is part of the object container stream. It's the mechanism for moving the object out of memory into the database. The serialization of an object is simply taking all of the member objects of a class and storing the information into a byte stream so that it can be easily transferred. In this case, it has two parts. One is the serialization and the culling is iterating over the object container tree to remove objects no longer needed.

Serialized Variables: Introduce network update policies to variables, e.g. variables been constantly networked on each tick or only when they change to reduce and optimize network bandwidth.
Entity Component Update Scheduler: Provides information of how far away an object is and if it is visible to the player. It also already supported the functionality to be able to skip network updates of entities that are far away from players. This most likely is the game update loop of the engine.
Serialized Variable Culling: The Entity Component Update Scheduler stopped networking entities to the clients that were too far away from the players, but all clients still were send network updates of all entities around ALL players on that server, no matter how far apart the players or entities were from each other. Serialized Variable Culling changed that by the server only sending each client the updates of entities which were near that specific player.
Network Bind Culling or CSOCS: The previous features were only about limiting network updates, entities were still loaded on the client. This was changed with Network Bind Culling or what we now know as CSOCS. This feature added the functionality for the server to tell the client when to load and unload them on each client.

The quotes are from the JumpPoint article about OCS again:

Network-Bind-Culling is how we reference to Object-Container Streaming on the client.

Utilizing all the preparation above, we could develop Network-BindCulling. In that system, instead of skipping network updates while having all entities present on each client (as we did for Serialized Variable Culling), we will, driven by the server, load and unload entities on the client.

At the same time, we only had the update culling of the Entity-Component-Scheduler. While that system helped with server performance, clients still had to pay an unnecessary cost; we didn’t yet have a system in place to decide which client requires which information.

These features all sound very server side on the implementation side to me, because it is the server that starts and stops sending network updates of entities, as well as tells the client when to load and unload entities from/to memory.

Therefore, I believe what you described was the functionality of the Entity Component Update Scheduler plus the Serialized Variables API, not the Serialized Variable Culling feature or CSOCS.

Lumberyard did not originally have a mechanism for streaming objects in and out of memory. It like Unity loads all of the objects for a level at level start into memory. Without talking to the dev's involved or reviewing the code changes I do not have the specifics but given my understanding of game engines, I would wager the client-side object container stream was purely client while the server-side was purely server and given the server is keeper of the game state would require a lot more work as it needs a way to update objects that are reloaded. The client-side can simply rely on the server updates to reposition objects it had removed from memory.

I dont think I claimed that Lumberyard originally had a streaming functionality. We know that CryEngine didnt have it. Everything was loaded on the loading screen. The functionality of streaming was only introduced with Object Containers and then utilized once CSOCS and SSOCS was implemented.
Yes, CSOCS allowed streaming of objects on the client and SSOCS allowed streaming of objects on the server.

The variable tick rate that Chad is talking about is the dynamic object updates to the client. The client's update to the server is limited to the inputs the user issues and that needs to be as often as possible. Any mouse movement, keyboard input, joystick action, face tracking, etc. The server updates the client with all of the game world changes that are relevant to the client (network culling limits, prior it was all objects on that server). The second part of Cad's message is mostly about how the client displays the game world and how they blend animations and object updates to try and make it appear as smooth as possible especially with any sort of network issues between server and client.

Hm, you are correct. I might be misundestanding the developers at times when they just say "update". I am confused if they mean the update in the game loop or the network update send to the client. I now think it might be the latter, but I still wish they would be more precise to not introduce potential ambiguity in their explanations. Or maybe I am just not familiar with their terminologies.

The easiest way to think about it is if a player can move it then its a dynamic object. Pico is a dynamic object. You can pick up the plushie and move it around, take it with you and place it in the co-pilot seat of your ship.

In fact, Unity Manual has a pretty well-written article for Network GameObjects

Unity - Manual: Networked GameObjects

docs.unity3d.com

Star Citizen gets a little more complex due to the fact that the ship itself is a dynamic object but the objects within the ship for the most part are static. Dynamic objects typically also include any object that can be player customized like the ship seats even if they don't move relative to the ship.

Yep, whenever an object can be interacted with or has an behavior it is dynamic. If it just exists and nothing else then it is static.
Not quite sure how they handle large objects like rotating planets and spaceships. I guess it would have to do something with their zone system and relative coordinates to minimize coordinate updates of entities inside other entities to a minimum.

Remember the server only gives the current state of the object not what the object looks like. So any static objects the client would have no issue in loading. Any of the dynamic objects state the client would rely on the server to tell it where the object is and without the server, it would never load in the object.

Yeah, I am aware that it doesnt actually network anything related to graphics. The rendering of frames is something entirely separate to the computation of entity states. Except the xyz positions, rotations and animation states which will be used for the creation of the visuals.

I'd have to go back over the original post again.

I mean that was your concern from your first post, right? That it wasnt descriptive enough to the point where it could be misunderstood or being entirely incorrect.
I can understand if thats too much work now, months later. I was just curious in case you did have some feedback for me. I think reading your comment and some of the dev texts, I think I might have understood some parts in some place incorrectly, like the different between skpping network updates or skipping game loop updates. I thought updates are also skipped on the server for further away entities but that might not actually be the case after re-reading it. So I made some adjustments to the presentation already. So I already have to thank you for the feedback that you provided! :)