Database Synchronization Between Clients

All projects that somehow make usage of code.
Post Reply
aspidites
Posts:101
Joined:Mon Apr 11, 2011 22:39
Database Synchronization Between Clients

Post by aspidites » Mon May 23, 2011 04:22

On the blog, snowdrop asked how OMG would handle syncing of databases. For example, what would happen if a player adds cards to a database and wants to connect to another who obviously doesn't have his changes.

I see two use-cases:

Developers:
I plan to solve this problem much like any version control system would. When connecting to a game, a three-way merge would run on each player's database with the host of the game being prompted with ways to resolve merges. If new cards are encountered, he may choose which version of the card to keep, or to cherry pick certain parts of it . If a new card is discovered, he may either keep it or discard it. Once the merge was complete, the new database woud be made available to both players.

Users:
A simpler approach would be taken here. When hosting a game, a specific version of a database would be used. When finding games to connect to, you would have to use the 'hosted' database. There could also be a config option that allows the host to give his opponents the ability to use newer cards.

In either case, some kind of checksum could be used to idenitify which version of a database decks were built against.

Feel free to discuss alternative (read: better) solutions than what I have presented here. Keep in mind, however, that this part of overmind isn't actually implemented, so this whole conversation could very well be premature.
User avatar
IsharaComix
Posts:7
Joined:Tue Apr 19, 2011 00:13
Location:Raleigh, NC, USA
Contact:

Re: Database Synchronization Between Clients

Post by IsharaComix » Sun Jul 31, 2011 06:16

I got to talk with Aspidites a bit back, and I had an idea for card database synchronization. I'm posting it as a reply here to keep things as on-topic as I can.

Proposal: Dealing with Database Inconsistencies.

When a client connects to a server and presents its deck, it has to say: "My deck is built from <foo> card database using card numbers 7,8,10,10,11...". At this point, the server has to answer up to four questions.

"Is the client using the same database as me?"
"If not, is the client's deck a subset of my database?"
"Does the client have cards in his deck that I don't know about?"
"Do I accept these cards that I don't know about into the game anyway?"

As we know, each card is a "plugin" as far as Overmind is concerned. Furthermore, these cards are organized in collections called "card databases", which are also "plugins". One thing about these plugins is that they can be downloaded and stored locally, which allows the user to build a deck offline and then present it to the server in order to play a game.

Each card should be represented by a unique hash fingerprint that takes everything that makes the card unique into consideration: name, description, stats, etc. The card database should also have a hash, which could simply be a chain hash of the cards. When a client connects, it would send a message along the lines of "I'm using database 12900AF923 with cards 7,8,10,10,11..." with 7,8,etc being the indices within that database.

The first thing the server does is validate the database. If the client is using the same database as the server, then there's no issue. Just use the cards. However, if the client is using a different database, then the server determines which cards are being used. The client then reads off the hashes of each card in the deck. As long as all of these hashes match cards saved on the server, then everything is still good.

The problem occurs when the client has cards that the server doesn't know about. If the server doesn't know about them, it's possible that our client's opponent also doesn't know about them (consider the server as a third party in this case). The most prudent course of action is to reject the user "I'm sorry, you're using cards unrecognized by this server. You need to build another deck." After all, can we trust this user to send the card data to the server, who would then send it to his opponent? (configurable?)

On a related note, what if the database is a newer version of an old database? What if the card is a newer version of another card? One thing that would be cool would be to keep a version history in the plugin, which is just a collection of the old hashes the card used to have: "it looks like your card is a newer version of our old card. While you're here, we'll treat your card as if it were the old card."

Let me know what you think. It's late and I'm not thinking clearly, so if anything isn't clear, then let me know. One limitation is that the history idea works well for "versions" of a card, but not so great for "forks" (since we know quite well from FOSS that ideas don't evolve linearly!).

(also: shouldn't Overmind have its own subforum in the Code section?)
User avatar
snowdrop
developer
Posts:798
Joined:Mon Feb 01, 2010 15:25
Location:Sweden
Contact:

Re: Database Synchronization Between Clients

Post by snowdrop » Sun Jul 31, 2011 16:34

What I like with LackeyCCG is how it handles parts of this:

1. Installation of new games is super easy - you just point the program to a "patch" (which is just a prepared bunch of text (db) files with info and the card images) URL, which in return gives it a textfile with webserver paths and whatnot, telling Lackey to download it all.

2. When a game is installed you can update it by redoing the same process or pressing "update" button. It then connects and compares hashes. If there is a change the new textfile/card image will be downloaded and replaces the old.

I don't know how it resolves conflicts etc between two-three players though and I currently don't have any bright ideas about how to do it.

I believe the above procedures though to be more than sufficient and also solve a number of issues. It's still super easy for any game developer to setup a "central repo" for the game - all you have to have access to is a webserver somewhere. It's also easy for players to manually update their db to reflect the developers latest expansion releases and revisions.

I see the above as being open and easy to handle. It would of course still be wise to allow players to load patches locally from their own harddisk as well, just as Lackey does.

What happens when you go for a multiplayer game (yeah, there is no other than that and solitary vs yourself ; ) is that it could then just check with the main repo and update accoridnlgy if its needed. It would, when connecting to the other player after that, check what repo the other player uses, so people can easily see if they use same central db to fetch their favourite game from. Lastly, when a player loads a deck, and you are in the multiplayer game and connected to each other and all, <i>then</i> it checks the hashes of all cards in every players deck. (This is to reduce time and work - why ever check hashes for cards that wont be used that game??) .

If I have a card called "Urza murza" in my deck my opponents client will check that its version of "Urza murza" is the same as mine. If it's the same, nothing happens. If not, a conflict must somehow be resolved: Who's version would be used? I think it would be cool to pop up a window at both players end and asking them how to resolve it, and maybe also notify them who has the latest central version from the repo or not, and ask which one should be used: P1:s, P2:s or the central one in repo. This scenario should however seldom happen if both players clients are auto-checking repo every time they are fired up and auto-fetching latest version of the db.

I'm sure there might be better ways, as I didn't quite follow every strain of thought in your posts above. In the end though I think

(I moved this to the code section. I will probably remove the subforums as thy will seldom be used. If there are many activ threads around code that relate to the different projects I will re-instate them)
User avatar
IsharaComix
Posts:7
Joined:Tue Apr 19, 2011 00:13
Location:Raleigh, NC, USA
Contact:

Re: Database Synchronization Between Clients

Post by IsharaComix » Sun Jul 31, 2011 22:56

snowdrop wrote:What I like with LackeyCCG is how it handles parts of this:

1. Installation of new games is super easy - you just point the program to a "patch" (which is just a prepared bunch of text (db) files with info and the card images) URL, which in return gives it a textfile with webserver paths and whatnot, telling Lackey to download it all.

2. When a game is installed you can update it by redoing the same process or pressing "update" button. It then connects and compares hashes. If there is a change the new textfile/card image will be downloaded and replaces the old.
I didn't think about this until just now, but it would be cool if Cardscape could dynamically generate these "plugin urls" (wtactics.org/cardscape/lackey_plugin or /overmind_plugin). A dynamic php link would be able to produce the plugin, allowing players to download and cache the most recently updated version of the database (so it'd still be useful for developers, since Cardscape is a collaborative card-editing system). Conflict resolution is simply based on the authoritative link URL and whatnot. If you have a card in your deck that's not in the database, you'll get rejected by the server.
aspidites
Posts:101
Joined:Mon Apr 11, 2011 22:39

Re: Database Synchronization Between Clients

Post by aspidites » Mon Aug 08, 2011 02:21

I'm far too lazy/tired to quote specific portions of your replies, so I'll generalize instead.

I don't think Cardscape and Overmind should be deliberately linked together. Rather than automatically generating 'plugin URLs', I think any linking of external software should be done explicitly. In this case, I think it would be most feasible for cardscape to have some kind of API that can be coded to. For example, it makes sense to me that cardscape would be able to generate a valid json representation of the database. From that, overmind would be able to communicate with cardscape using its public API to create database usable in OMG or similar clients.

As for the idea of a prompt that allowed for interactive resolving of conflicts, that is what i envisioned. Actually, I planned to take it a step further and have a configurable option which allowed a user to either a) automatically reject conflicts, b) automatically accept conflicts, or c) prompt when a conflict is enountered (kind of how you can configure your web browser to behave when it encounters web page with unsecured data on it).

As for data integrity checks, I had originally imagined something like this:
1. when the server is started, a specific version of a database is selected
2. when clients connect, the version of the database used to build the deck is checked against the version used to run the server
3. if versions match, it is assumed that all cards are compatible. if the server's is newer, depending on a user's preferences, he is asked if he wants to update the database. I hadn't quite figured out the best thing to do in case the server's version is older. For now, I imagine the program would error out, and the user would be asked to have server admin update the database. If one of the players has access to the server, i suppose they could be prompted to update the database.

The way I saw it, when a user interacts with his deck, hand, field, etc, card data is fetched from the server directly if not located locally. If there is a local copy, since database versions are checked, the card contents should always be up to date.

I imagine there could also be a way for users to set up a server to allow any version of any card to be used (eg, for playtesting new cards), and in such instances, cards are fetched between clients, rather than from the server.

I'll be doing some coding tomorrow, and in the process, I'll also have to do some design work. Ishara, I don't think it will be as complicated as you have presented, but you have definitely highlighted some factors to take into consideration. Snowdrop, I don't think it will be as simple as what you have presented, but you make one valid point: to the user, it should be this simple. Thanks.
Post Reply