This tweet got me thinking, what Verison Control System (VCS) friendly file formats exist, or can you think of?

Here's a pasting of an example of the format Juan from Godot came up with:

[gd_scene load_steps=3 format=2]

[ext_resource path="res://icon1.png" type="Texture" id=1]
[ext_resource path="res://icon3.png" type="Texture" id=3]

[node name="Node2D" type="Node2D"]

[node name="icon1" type="Sprite" parent="."]
position = Vector2( 138, 127 )
texture = ExtResource( 1 )

[node name="icon3" type="Sprite" parent="."]
position = Vector2( 782, 109)
texture = ExtResource( 1 )

Json isn't the most VCS friendly. YAML has the potential to be, albeit much more complicated, and slower to parse. The above format reminds me of an ini format with grouping, where the group defines the asset, and the key value pairs (kvp) are the asset parameters. It's a nice touch. And ini's are really easy and fast to parse. I like it. Can this be improved? What else can you think of?

3 hours ago, leetNightshade said:

Json isn't the most VCS friendly. YAML has the potential to be, albeit much more complicated, and slower to parse. 

If you supply your own 3-way merge tool to your VCS to use when merging Json files (or any hierarchical format!) then you can always automatically end up with a valid merge result ?

57 minutes ago, Hodgman said:

If you supply your own 3-way merge tool to your VCS to use when merging Json files (or any hierarchical format!) then you can always automatically end up with a valid merge result ?

This idea is more for artists and designers, to make things easier on them. And making it harder for them to break something. I don't expect them to have a 3-way merge tool on their computer.

1 hour ago, leetNightshade said:

This idea is more for artists and designers, to make things easier on them. And making it harder for them to break something. I don't expect them to have a 3-way merge tool on their computer.

Yeah that's the use case I was aiming for, too  
They do have a merge tool on their computer if they have VCS software installed - it will pop up whenever there's a merge conflict.

SVN / Git, P4, etc can all be configured to use custom merge tools instead of the crappy default ones. Artists/designers/etc still need to install the VCS software and the game engine / project before they can work, so it's not too much harder for "the engine" to include it's own diff-tools and scripts to plug them into the VCS software. Some engines like Unity/Unreal/etc already try to integrate with your VCS software anyway -- having config screens in the engine-editor where you select which VCS software you're using, etc... The engine can use that info to attach it's custom diffing tools for it's own file formats. If the engine includes custom diff/merge tools, you can make it automatically resolve file conflicts in a smart way, without requiring a human to hand-select text file changes at all, which completely solves the problem.

As for that original godot example text though, I can still see some merge issues occurring... e.g. say you start with this file:

[ext_resource path="res://icon1.png" type="Texture" id=1]
[node name="icon1" type="Sprite" parent="."]
texture = ExtResource( 1 )

And then two different people simultaneously want to add another icon of their own, and they increment that "id" field when adding their new resource:

Change A. Append this:
+ [ext_resource path="res://icon_foo.png" type="Texture" id=2]
+ [node name="icon_foo" type="Sprite" parent="."]
+ texture = ExtResource( 2 )

Change B. Append this:
+ [ext_resource path="res://icon_bar.png" type="Texture" id=2]
+ [node name="icon_bar" type="Sprite" parent="."]
+ texture = ExtResource( 2 )

When resolving that conflict, you can just choose "Mine, then theirs" to add both blocks to the end of the file, but there's still a problem because both changes have added an external resource with ID #2, and now it's ambiguous which filename each of the node's is referencing  

To avoid this issue, don't use simple ID numbers in these files -- use filenames as ID's / GUIDs / etc...

Or another example, someone might change that original file as below (i.e. they've added a single line on the end -- the position attribute).

[ext_resource path="res://icon1.png" type="Texture" id=1]
[node name="icon1" type="Sprite" parent="."]
texture = ExtResource( 1 )
+ position = Vector2( 138, 127 )

While simultaneously another user might change it to add a new node:

[ext_resource path="res://icon1.png" type="Texture" id=1]
[node name="icon1" type="Sprite" parent="."]
texture = ExtResource( 1 )
+ [node name="icon2" type="Sprite" parent="."]
+ texture = ExtResource( 1 )

When resolving that conflict, depending on the order that you apply the two simultaneous additions, you could end up with either of these results:

Theirs then mine:
[ext_resource path="res://icon1.png" type="Texture" id=1]
[node name="icon1" type="Sprite" parent="."]
texture = ExtResource( 1 )
position = Vector2( 138, 127 )
[node name="icon2" type="Sprite" parent="."]
texture = ExtResource( 1 )

Or Mine then theirs:
[ext_resource path="res://icon1.png" type="Texture" id=1]
[node name="icon1" type="Sprite" parent="."]
texture = ExtResource( 1 )
[node name="icon2" type="Sprite" parent="."]
texture = ExtResource( 1 )
position = Vector2( 138, 127 )

To avoid this issue, you could make it so that there are no optional properties. In this example, the problem arose because a node didn't originally have a position attribute at all. If every node always had every property present, this kind of conflict would be much less likely.

Also, in all these examples, the VCS software will just throw it's hands up in the air, complaining that there's a conflict, and will launch an interactive tool for the user to resolve the conflict (picking from the above choices by hand). So artists/designers are still require to read and hand-edit these file formats, which is a bummer  
e.g. Above, the first one is probably correct, but you require the artist doing the merge to look at both options in their merge / "resolve conflict" tool and pick the right one.

There is admittedly much less room for them to screw up than with a picky format like JSON, so it is better in that respect! But it doesn't completely solve the problem.

Perhaps the biggest part of the solution is to let all agree who works on what. That avoids getting changes in the same area. I read at least once in a VCS book that "merge tool is not a replacement for communication".

What would also work is the single writer like with RCS, only one person has write access at any time. May not work very well though with people having write access and leaving without making the file available to others.

Those engines e.g. their editors aren't made for real collaboration. I think this is a general issue they have to solve in the future as more teams work globaly on games. I think of something like a project server similar to what Google Docs offer

7 hours ago, Shaarigan said:

Those engines e.g. their editors aren't made for real collaboration. I think this is a general issue they have to solve in the future as more teams work globaly on games. I think of something like a project server similar to what Google Docs offer 

I wish I remembered more about the implementation, though I could ask some of the engineers with more time on the project: Red 5 Studios' assets were backed by a database, instead of modifying files on the VCS, you made changes in the data editor and when saved/submitted would be pushed to the database everyone is reading and modifying. Instead of modifying files you're modifying database fields, basically. There's still the potential for conflict, and I forgot how they handled that.

12 hours ago, Alberth said:

Perhaps the biggest part of the solution is to let all agree who works on what.

At a AAA studio where designers, programmers, and scripters are modifying the same files in some cases, that's not always possible. In that case though, modifying certain files should be mostly relegated to one department. But in the effort to save time with less beurocracy, conflicts can arise, especially on deadlines.

 

Quote

What would also work is the single writer like with RCS, only one person has write access at any time. May not work very well though with people having write access and leaving without making the file available to others.

I do like that idea, but file formats would have to be designed or people adding stuff to break things up so there's not too much stuff in one file. Otherwise with a database implementation, being able to lock fields would be handy. The locks could be time expired, or use a tool to detect activity and expire after inactivity, for small changes.

On 4/12/2019 at 8:01 PM, leetNightshade said:

Red 5 Studios' assets were backed by a database, instead of modifying files on the VCS

Yes, we are following a similar approach with our software. Anything is stored on the project server's database and updates are commited frequently to it and spread to all users silently while also maintaining a local copy of the project. We have a conflict solution by showing to everyone that the object is about to be edited by someone else with the editor's name. If changes are made on the same time, the first one who has finished editing commits his/her changes to the server while the other one will get a warning about the conflict and may "preview" the changes from the server, then decide which version to keep