19 hours ago, Alberth said:

The problem, is that constructing a large string is quite deadly for performance, so that's what you should try to avoid. I don't know C#, so I used Java, which is close enough I hope to get the idea across:

            byte[] bytes = {1, 2, 35, 8, 127};

            // Open a text file for writing
            BufferedWriter handle = new BufferedWriter(new FileWriter("output.txt"));

            // Convert each byte to a string, and write the string
            for (byte b : bytes) {
                String s = Byte.toString(b);
                handle.write(s);
                handle.write(' '); // Add a space after each number.
            }
            handle.write('\n'); // Add a newline at the end.
            handle.close(); // Close the file.

The code constructs a string for each value, and writes that string to a text file. This code makes many small strings, which is not a major problem.

 

And the file looks like (as you'd expect)

1 2 35 8 127

 

I hope you can convert this to something C#-ish if you like the idea.

I would suggest doing this as well. I've worked on smaller projects and I had to export arrays to text files, and I just did it like this. It's very simple, fast, and easy to read. It's also easy to load back in without problems.

Take this as a learning experience, we can always learn new and better ways to do something.  

6 hours ago, Lactose said:

If we take away the hostility/rudeness from your list of points, then I would agree with them a lot more -- even more so if some explanations were added in order to help people understand and learn.

Ah, sorry.  I didn't realize this was in the beginner's section.  I actually have to deal with code like this at work, with (ostensibly) professionals who should know better, and this post just struck a nerve.

Let me clarify my points for beginners:

• Using the + or += operators in C# creates a brand new string EVERY time you add something.  This has to copy all of the characters from both of the input strings to the new string.  If you do this in a loop once per character, the amount of memory read/written increases exponentially.  This can drastically slow down even for a few thousand characters in the final string.  A StringBuilder is optimized to allow appending lots of things because it uses separate arrays when it needs to expand, minimizing the number of character copying it needs to make.
• Casting from an int to a char treats the integer as a character code.  For example, if your integer is 32, you get a space character.  If it's 65, you get an 'A'.  In the code posted above there's no way I could see that being intended.

Things that I would use if I had to use text files:

• Write the file out using StreamWriter.  This lets you do it without building the whole string in memory at once.
• Based on the format example you gave in the first post, I would read each line in from a file using StreamReader.ReadLine.  Similar to StreamWriter, you don't need to load the entire file into memory as a single string.  You can read one line at a time, split it up, then use int.TryParse and float.TryParse on each section.

I typically use binary files instead.  Even though some people like text because it's human-readable, binary is more compact and simpler to read and write with code.  If you get familiar with hexeditors, you can examine files even if they're binary.  This will come in handy since there will be some point where you'll need to use someone else's binary format.

• With binary, use BinaryReader and BinaryWriter.  These have functions to read/write C#'s primary data types directly (ints, floats, bytes, chars, strings, etc).  Strings written by a BinaryWriter have an additional length prefix (in binary) and are therefore NOT readable using a StreamReader.
• When you use BinaryReader/Writer, your reading code typically looks nearly identical to your writing code:  If you had 'Write(int) Write(byte)' you'd have ReadInt32() then ReadByte().
• If you need to write lists or arrays, typically you want to write out the length of the list/array first, then foreach through the list and write each element.  When you read, you read the length, create the list/array of that size, then write something like: for (int i=0; i<length; ++i) { list.Add(binaryReader.Read____(); }

Like everyone else has said, you probably want to use a binary format. . Like @Nypyren said using + and += on strings is horribly inefficient.  If you're dead set on using strings, you have to get rid of the + and += operators and use StringBuilder instead.

- Eck

24 minutes ago, Nypyren said:

I typically use binary files instead.  Even though some people like text because it's human-readable, binary is more compact and simpler to read and write with code. 

As a sort of compromise, there's also the option of using human-readable data in the beginning, and the swapping over to binary later on once things have finalized a bit. Meaning that, even though you might start with fixing up the string writing now, you can always go back and change things to binary as the project progresses.

5 minutes ago, Lactose said:

As a sort of compromise, there's also the option of using human-readable data in the beginning, and the swapping over to binary later on once things have finalized a bit. Meaning that, even though you might start with fixing up the string writing now, you can always go back and change things to binary as the project progresses.

Surprised it took two pages for someone to mention this. As its a technique I'd say it is semi-common, you can even use preprocessor directives to automate the process somewhat, though then you deal with the scenario of having data files in two different formats. You'd probably only want to use binary for release realistically.

Of course like all things, flexibility tends to cost time, obviously just writing one method would be less work in the end, but if text files will let you debug things easier I would say they can't really hurt, and there is nothing wrong with using text files they might even be just what you want if you'd prefer the user can edit files easily.

I got the performance down to an acceptable level (it now takes less than a second to load my stress test game, which is 13 mb large). Thank you all for your input. I did two things to get it there; first I used a stream writer and wrote the file bit by bit instead of constructing a huge string and secondly I stopped doing my conversions bit by bit and instead did array wise converting (which I didn't know you could do). I certainly learned alot, which was purpose of writing this engine.

@Nypyren thank you for your lengthily explanation. A small comment though is that the behavior you get by converting an int32 to a char is exactly the behavior I want to get. Maybe I was not clear on this but all my data first passes an intermediary string stage before getting converted to unicode code, and then it get serialized by the processing of putting char delimiter infront of each strings. If you look at the attached file in my first post you will understand the method, which is perhaps not the best one, but one that works really well for my purpose.

I find all of this very encouraging as it's kind of a lifelong dream to write my own 3d engine and make an adventure game with it. In the span of 1.5 months I now have a game editor (complete with a scene view and pretty much every feature I need) and a 3d engine that supports pretty much almost everything I need to make that adventure game. I will probably close the project in a short while when I'm done making the game itself and write an article on it. (I think that people, even newbe, don't realize how quite accessible it is to make a game from scratch if you are willing to do things the simple way instead of the proper way) Next project I do will probably be on a proper fully featured game engine, like Unity3D.