Educational: icache explained video

[Montoya]

Do you stay up and night wondering what icache is and how much you should care?

 

[ColdDog]

You were reading while the rest of us were multitasking.

Appreciate the context - makes your videos worth listening to.

iCache a freaking Database... good grief, not like databases are new. I'd assume it stores the location, attributes, how long it remains/available, etc. Why do we always act like this is a new tech - for those not in the computer industry? Again, this is all bs, because this has been done before with countless other games... work on the freaking Universe for goodness sake and stop pretending you are introducing new tech - Fallout 4 was good at this just to name one (single player example) - Elder Scrolls, Wow, Lord of the Rings, etc. Get over it CIG and get on with it. CIG the longer you take the harder we're going to push and the more we are going demand... this is a function of how much bs the fan base willing to take. Good job with icache... you get a gold star... move on.

 

[NaffNaffBobFace]

Thank you, Glorious Leader!

How long do you think bodies persist? Asking for a friend with a full freezer...

 

[CosmicTrader]

Good explaination.

 

[Michael]

Chad Mckinney recently on icache

iCache, global persistence, and physical inventory are several very large features being worked on from multiple angles by multiple teams. It is one of our highest priorities as a company and is actively being worked on. At this point I think icache/global persistence tier 1 will land before the end of the year but I don’t want to specify exact dates, one thing to mention is that long term persistence did take some resources away from icache work, but we decided that it was worth it since the quality of life improvement to players would be such a benefit. Physical inventory heavily relies on icache and global persistence, and our entire system was designed around it (unlike the legacy system which was built entirely around itemports on players and ships and so concepts such as loose items and physical vs. legal ownership were unnatural to get working) . For that reason physical inventory will naturally come after that work is completed, though we are trying to find ways to get work started on it in parallel where possible.”

What is the Difference between pCache, iCache & Global Persistence?

“pCache (persistence cache) is the legacy system, using a monolithic cache for queries to the persistent item DB. As you can imagine, this does not scale well, and worse, when it goes down the game becomes essentially unplayable. For everyone! The new icache (item cache) uses a proliferated fleet of services to optimize querying in a scalable fashion, while using best practices for fault tolerance and recovery so for example all nodes have their data replicated in the network so even if one goes down, only partial data response is briefly lost and quickly a new one is rebuilt in it’s place, so automatic regeneration. The new icache is also being built with our gameplay systems in mind and understanding of how we need to query data from game servers. At this point we have a much better idea of how the game works than when the legacy p cache was created, so we can design our item data and query schema to make this new system efficient and minimal, which will also help with stability and scalability.

Global persistence is the term some of you may have noticed me bringing up in some of my comments and that’s because it is a related but different feature set that also needs to be implemented. Just because you have a database and cool querying system for that data doesn’t mean the game somehow automatically uses it in all the game systems, nor does it make servers/shards have persistent state, it just gives you a tool to use to enable that. In order to get server history persisted for everything you need to do a lot of work in server code to persist out and load back in said state, this is global persistence. I bring it up all the time because it touches so much, for example physical inventory. Inventories are just containers of items, but they are physicalized because they themselves are persistent entities with state. This means you can have a backpack with weapons or other items in it and have a way to stow/unstow from that inventory, but also that inventory can be instanced in the world and left in a vehicle or dropped off on the top of a mountain on a moon for another player to pick up. So physical inventory relies on global persistence, which relies on icache.

But there are other features that rely on global persistence as well, for example now we can explore gameplay that can permanently or on long term scales modify locations in the game, for example space station damage. Port Olisar could suffer an attack on a shard and it would be damaged for some time until repairs could take place. This would persist even if the server/shard went down as it is no longer just in memory state. There’s a ton more, such as full server recovery after a server crash. People often complain about losing cargo after a server crash, as we can’t get players back into their ships right now after a server crash unlike a client crash where we lose the state of the server. With global persistence this will be recorded in the shard record it was associated with, so quickly we spin up another instance, it recovers the shard record, then we can use matchmaking to get players back into the shard they just lost connection to, and boom, they’re back in action right where they left off.”

What About The Mission System & Reputation, are these handled by those services?

“The mission system is undergoing it’s own large refactor for server meshing and one of the big questions is of course how you deal with state. It won’t be with icache, but it could use a similar system with the same kinds of features but just not literally the item database.”

Also a great reason why you usally make the tech first and then create the shiny graphics

 

[Cugino83]

:rise hand: Prof can we take a brake... I need to grab another beer!
Nice quick explanation of iCache, and I like the idea of generated mission based on actuall assets that are created in the game by player action (such as ship wrecks).

I'm looking forward to your video on Morphologis... whould be interesting to have a video of both of you discussing some new ship.... (HINT: MSR on PTU EVO rigt now....)

 

[Bambooza]

You were reading while the rest of us were multitasking.

Appreciate the context - makes your videos worth listening to.

iCache a freaking Database... good grief, not like databases are new. I'd assume it stores the location, attributes, how long it remains/available, etc. Why do we always act like this is a new tech - for those not in the computer industry? Again, this is all bs, because this has been done before with countless other games... work on the freaking Universe for goodness sake and stop pretending you are introducing new tech - Fallout 4 was good at this just to name one (single player example) - Elder Scrolls, Wow, Lord of the Rings, etc. Get over it CIG and get on with it. CIG the longer you take the harder we're going to push and the more we are going demand... this is a function of how much bs the fan base willing to take. Good job with icache... you get a gold star... move on.

Ah welcome to the world of software development, everything is cyclical. In my job we are working on moving one of our larger applications away from the database to in memory cache, writing a journal file for ease of scaling. 10 years ago we moved it from cache flat files to a database because of issues with loss of data, concurrency and reporting.

As for CIG, while a lot of what they are attempting to do has been done before its still not an easy process to introduce it into a game engine that does not have native support. The other issue is the amount of data that needs to be stored and quickly retrieved. Not just the number of objects but also the amount of information that describes that object in the game world and how long it should exist in the game world (after all you need ways to prune the database) who is the parent container object and it all needs to be done at run time.

All of these delays are caused by the massive scope creep that continues to expand the game world of SC. In someways its to be expected due to the massive success of the crowed funded project (many have crashed and burned under a highly successful kick starter causing a massive blow up of the project ambitions as the team feels obligated to expand the project with the new found large budget, instead of maintaining the original goals and just pocketing the money) but it was also to be expected given Chris has never been one to draw lines in the sand and stick to them.

I would agree with you @ColdDog and would be raising a pitchfork in protest if it wasn't for the PU. But with the PU we get to see the progress in real time as well as the reality that the amount of time spent already is far exceeding most triple A titles. Yes it sucks waiting for all of the game features to come online to truly dig into the aspirations we have for this game, but at the same time if we look back at what was originally promised, I would say the restarts and do overs that have costed this project many years are worth it given what we already have and what we can expect to have in a few more years. Go ahead re-watch the kick start video https://www.kickstarter.com/projects/cig/star-citizen and tell me we don't already have something head and shoulders above what was originally promised. I liked the original concept and would have had fun playing it for maybe a hundred hours but it really wasn't anything special. What we have now I have already spent more then a hundred hours in and each patch brings new things to see and do and I am for one happy with the progress they have been making especially since 3.0.

 

[ColdDog]

Yes it sucks waiting for all of the game features to come online to truly dig into the aspirations we have for this game, but at the same time if we look back at what was originally promised, I would say the restarts and do overs that have costed this project many years are worth it given what we already have and what we can expect to have in a few more years.

I live in the world of non-relational databases with petabytes of data. Performance is always an issue, but what the hell. I do not know what the schema looks like or even really care but this type of stuff should not be server side and should be local for your personal inventory with synchronization between the local and remote hosts. They seem to be pushing the assets to the limit already, not to mention our SSDs. Hire a MIT whiz kid and get it done... backers have provided the funding. While you're at it get the MIT whiz kid to build us an economy too.

 

[Sky Captain]

At this point in development, I'm looking for CIG to be focused more on how many shots from a Javelin that an Idris can stand up to before it goes boom!

 

[Bambooza]

I live in the world of non-relational databases with petabytes of data. Performance is always an issue, but what the hell. I do not know what the schema looks like or even really care but this type of stuff should not be server side and should be local for your personal inventory with synchronization between the local and remote hosts. They seem to be pushing the assets to the limit already, not to mention our SSDs. Hire a MIT whiz kid and get it done... backers have provided the funding. While you're at it get the MIT whiz kid to build us an economy too.

The issue with non server side is then its free to be modified by us .

 

[ColdDog]

The issue with non server side is then its free to be modified by us .

Not really, it can be secured. In the old days we use to have config data in .ini or text files. Today this data can be obfuscated and secured. It can run in memory as a table (I could assume this could be called a "cache"), and synchronizing local assets. I am not a game programmer, but I do see instances of this in the technology I use.

 

[Bambooza]

Not really, it can be secured. In the old days we use to have config data in .ini or text files. Today this data can be obfuscated and secured. It can run in memory as a table (I could assume this could be called a "cache"), and synchronizing local assets. I am not a game programmer, but I do see instances of this in the technology I use.

It doesn't take long for obfuscated and secured data to be compromised if its stored locally. Think of all of the trainer tools created for single player games that stored game state in a binary file. First rule of any online programing is to not trust the client. And since we are not trusting the client that means we have to validate all of the client data and that requires storing the information which requires a database or cluster cache.