Search
Twitter Feed
Navigation
« Major update to Spartan Athletics approved | Main | Unity and accessing common MonoBehaviour manager classes »
Thursday
Mar312011

Using protobuf-net serialization in Unity iPhone

If performance is an issue for your serialization in a Unity iPhone project, this post shows you how to use the protobuf-net library.

Some Background

As part of a new game project I'm working on, I had to store level information in a file and read it in for each new level. I had created a simple level editor as a windows WPF C# project and then I would serialize my level model class onto the file system then include it in my Unity project to be loaded in during runtime.

I initially used simple XML serialization, but as the levels increased in complexity it was taking longer and longer (well, 100s of miliseconds, but every bit counts ;-) to deserialize. So, off to the web to find the fastest and easiest serialization. The best candidate was Marc Gravell's protobuf-net, which provides 'Fast, portable, binary serialization for .Net' using Google's protocol buffers technology. And he's not kidding on the 'Fast' part either, see the performance stats here.

So I grabbed the library, chuck it into Unity and start testing. It worked in Unity Windows, worked in Unity Mac but when I deployed it to my iPhone I hit this:

ExecutionEngineException: Attempting to JIT compile method

Now, I'm no expert on the inner working of .Net and it's relationship with iOS running Mono, but according to this thread on the Unity forums, the culprit is the JIT (Just in Time) compilation of classes that have not been seen by the system before. Mono on iOS is an AOT (Ahead of Time) only system, so that's why it craps out. I'm sure smarter people than me could provide a better explanation, but that will do for now.

So, I emailed Marc Gravell for help because he mentioned in this thread that version 2 of protobuf-net would have a 'pre-compile to dll' option, meaning that the serializer/deserializer classes can be pre-made in a dll instead of on the fly (and JIT'ed). I think that's how it works.

Anyway, he sent me an alpha version of a Unity iPhone friendly protobuf-net and that's the one that works. Woohoo.

I also have to say that this is all info that Marc sent me so all credit goes to him here. I wouldn't know a protocol buffer if it came up and bit me in the arse to be honest.

The Solution

First up, download the alpha version of the protobuf-net libraries. The link Marc sent me is this one but go check out the site to see if there is a later version.

This contains two libraries, the 'Light Framework' and 'Full Framework' versions of protobuf-net. The basic procedure is this:

  • Create a library dll (assembly) of the model classes you want to serialize/deserialize. That is, just create a new 'Class Library' Visual Studio project and have it contain only your model. (I don't use MonoDevelop but I'm sure it's a similar process). You will have to reference the 'Light Framework' dll in this project in order to use the  [ProtoContract]/[ProtoMember] attributes, described in the Getting Started guide. Build this project to produce your MyModel.dll.
namespace ProtoTest
{
    // Simple model classes, with some inheritence and generics thrown in. 
    [ProtoContract]
    public class MyModel
    {
        [ProtoMember(1)]
        public int int1 { get; set; }
        [ProtoMember(2)]
        public TestEnum enum1 { get; set; }

        public List intList { get; set; }
        [ProtoMember(3)]
        public List floatList { get; set; }
        [ProtoMember(4)]
        public List stringList { get; set; }

        [ProtoMember(5)]
        public List anotherClassList { get; set; }
    }

    [ProtoContract, ProtoInclude(10, typeof(DerivedClass))]
    public class AnotherClass
    {
        [ProtoMember(1)]
        public string string1 { get; set; }
    }

    [ProtoContract]
    public class DerivedClass : AnotherClass
    {
        [ProtoMember(1)]
        public float float1 { get; set; }
    }

    public enum TestEnum
    {
        run,
        walk,
        skip
    }
}

    Note: Make sure your model project is set to .Net 2.0 in the project properties, otherwise Unity will throw up the following error:

    Unhandled Exception: System.TypeLoadException: Could not load type 'System.Runtime.Versioning.TargetFrameworkAttribute' from assembly 'MyModel'
  • Next you need to create the serilization/deserialization classes. So create a new 'Console Application' Visual Studio project. Now, for this one you need to reference the Full Framework protobuf-net library as well as obviously your newly created MyModel.dll assembly. 
  • Now you need the code to create the libraries.
var model = TypeModel.Create();

model.Add(typeof(AnotherClass), true);
// Note: you don't need to add DerivedClass here, in fact it craps out if you do. 
model.Add(typeof(TestEnum), true); 
model.Add(typeof(MyModel), true);
model.Compile("MySerializer", "MySerializer.dll");
  • This will output 'MySerializer.dll' 
  • Now we have our serialization library that we can use in our Unity project. So now you have to add three assemblies to your Unity iPhone project:
    • MyModel.dll
    • MySerializer.dll
    • Protobuf-net.dll (Light Framework)

And we're good to go.

To serialize the files in my external application, I used the following code. I haven't played around with writing files to the iOS file system so I won't post that code, but I'm sure it's similar once you get the paths correct.

MyModel myNewModel = new MyModel();

MySerializer mySerializer = new MySerializer();

using(var file = File.Create("TestFile001.bytes"))
{
    mySerializer.Serialize(file, myNewModel);
}

In my case I was creating my game level files in an external application, so having the libraries external was actually more convenient. Once I had run my level editor app and created the binary serialized output files, I figured the easiest way to load them in Unity was via TextAsset class. TextAsset can be used to load files from the Resources folder just like any other resource, and despite the name, it is also fine for binary files. 

Note: From the Unity docs on TextAsset

If you're using the text asset to contain binary data, you should make sure the file has the .bytes extension. For any other of the extensions the TextImporter will try to strip nonascii characters if it is unable to parse the file as an utf8 string.

So inside our Unity project scripts, to read in the binary file we just use this.

TextAsset textFile = Resources.Load("TestFile001") as TextAsset;

MySerializer mySerializer = new MySerializer();

MyModel readInMyModel;

using (System.IO.Stream s = new System.IO.MemoryStream(textFile.bytes))
{
    readInMyModel = mySerializer.Deserialize(s, null, typeof(MyModel)) as MyModel;
}

I had a look via Reflector and the second parameter to Deserialize() there is used in case your type variable is null, so I assume you can use either one.

And there you have it. A bit more work than just using a library directly, but if performance is an issue then it is well worth the effort. I haven't done proper metrics yet, but from a quick look it seems at least an order of magnitude faster than the XmlSerializer I was using before.

 

Edit: In response to the comment below about not being able to use Vector3.

    [ProtoContract]
    public class MyVector3
    {
        [ProtoMember(1)]
        public float x { get; set; }

        [ProtoMember(2)]
        public float y { get; set; }

        [ProtoMember(3)]
        public float z { get; set; }

        public MyVector3()
        {
            this.x = 0.0f;
            this.y = 0.0f;
            this.z = 0.0f;
        }

        public MyVector3(float x, float y, float z)
        {
            this.x = x;
            this.y = y;
            this.z = z;
        }

        public static implicit operator Vector3(MyVector3 v) 
        {
            return new Vector3(v.x, v.y, v.z);
        }

        public static implicit operator MyVector3(Vector3 v)
        {
            return new MyVector3(v.x, v.y, v.z);
        }
    }

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (6)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    [...]Welcome to Friction Point Studios - Friction Point Blog - Using protobuf-net serialization in Unity iPhone[...]
  • Response
    Response: ashleymerriman.com
    Welcome to Friction Point Studios - Friction Point Blog - Using protobuf-net serialization in Unity iPhone
  • Response
    Response: jachty motorowe
    Welcome to Friction Point Studios - Friction Point Blog - Using protobuf-net serialization in Unity iPhone
  • Response
    Welcome to Friction Point Studios - Friction Point Blog - Using protobuf-net serialization in Unity iPhone
  • Response
    Response: Survey Bypasser
    Welcome to Friction Point Studios - Friction Point Blog - Using protobuf-net serialization in Unity iPhone
  • Response
    Welcome to Friction Point Studios - Friction Point Blog - Using protobuf-net serialization in Unity iPhone

Reader Comments (18)

Sam, thanks this is super helpful for my Unity mobile project.

April 4, 2011 | Unregistered CommenterAlex Rice

No worries.

FYI: Just added a note to the post about making sure your model project is a .Net 2.0 project, in case anyone is getting weird 'System.TypeLoadException' errors.

April 4, 2011 | Registered CommenterSam Cox

Sam, I was able to replicate your experiment above. Yay! Have you found any way to keep both the .cs source and the .dll for your protobuf Model classes in the Unity Assets folder? But I found Unity gets confused unless there is only one present: either the .cs file, or the .dll. I like having all my source code in my Assets folder, but that may not work in this case.

April 26, 2011 | Unregistered CommenterAlex Rice

* using Monodevelop 2.6b2 to build the .dlls

April 26, 2011 | Unregistered CommenterAlex Rice

Hi Alex, glad you got it working.

I'm not sure about having both the source and dll in space visible to Unity. It will complain because you'll effectively have two classes with the same name in the same namespace etc. Not sure if Unity has any [PLEASE IGNORE THIS CLASS] preprocessor tags or anything.

I stuck my library project in the folder below my Unity project, together with the library generator project. Not sure what MonoDevelop is like but I use Visual Studio and make the library project a dependency of the generator project so when I do a build it builds one then the other. Then I use a post-build script to spit the output dlls into the Unity Assets folder so it picks them up automagically.

April 26, 2011 | Registered CommenterSam Cox

v2 beta is out :)
http://code.google.com/p/protobuf-net/

May 26, 2011 | Unregistered CommenterAlex Rice

Hi Sam,

By any chance, have you used protobuf-net with non-primitive data types, such as Vector3s? I am adding the basic ProtoMember but I am getting the exception "InvalidOperationException: No serializer defined for type: UnityEngine.Vector3"

June 30, 2011 | Unregistered CommenterJustin

Hi Justin,

Yeah, I ran into that problem too. Since you don't have access to the Vector3 source you can't mark it with the [ProtoContract] tags so you can't use it with protobuf-net.

BUT, I found an acceptable workaround. I just created an equivalent class, MyVector3 and marked it with ProtoContract then added some implicit converters so it can be used interchangeably with the normal Vector3. Then, in whatever library class you are converting you just use MyVector3 instead. This seems to work pretty well.

I usually have different model classes for serialisation and general use (kinda a model and ViewModel). That is, the model class that I use within Unity for normal operation is then converted into a SerialisationModel class which is then serialised. In the above case, the model class would use Vector3 and the SerialisationModel class would use MyVector3. With implicit conversion you can assign a Vector3 to a MyVector3 and vice versa.

I'm not sure if I can add code nicely to comments so I've appended my MyVector3 implementation to the end of the post. Check it out.

I emailed Marc Gravell for advice about this approach but haven't heard a response yet, might have been lost in the ether. Perhaps there is some easier way...

June 30, 2011 | Registered CommenterSam Cox

Thanks Sam, I'll try it out.

I was also looking into this issue and it looks like Marc responded on StackOverflow about a similar issue -

http://stackoverflow.com/questions/5808402/how-to-add-a-class-to-protobuf-net

His solution was to use a RuntimeTypeModel object but I wasn't able to get anywhere with it.

June 30, 2011 | Unregistered CommenterJustin

Thanks a ton for this Sam. Just got this working in our project and it's blazing fast and compact.

February 4, 2012 | Unregistered CommenterLane Daughtry

I just tried to do this in MonoDevelop, it was all going well until I came to create MySerializer. At that point i get this consol error:

"Unhandled Exception: System.TypeLoadException: Could not load type 'ProtoBuf.Meta.RuntimeTypeModel' from assembly 'protobuf-net, Version=2.0.0.480, Culture=neutral, PublicKeyToken=257b51d87d2e4d67'."

Any idea what the issue could be?
Thanks for your help.

June 16, 2012 | Unregistered CommenterMatthew Atkins

@Matthew, just a guess here, but hope this helps. Your model source just needs to have at the top:
using ProtoBuf;

But your serializer needs to have:
using ProtoBuf;
using ProtoBuf.Meta;

namespace YourSerializer
{
class MainClass
{
public static void Main (string[] args)
{
var model = TypeModel.Create ();
model.Add(typeof(SomeType), true); // add any custom types or classes which your model has
model.Add (typeof(YourModel), true); // add your model class too
model.Compile ("YourSerializer", "YourSerializer.dll");
}
}
}

June 16, 2012 | Unregistered CommenterAlex Rice

The latest build of protobuf-net, v2 r561, thanks to Marc Gravell, has a precompile.exe included! Use it like this:

mono precompiler.exe Model.dll -o:Serializer.dll -t:MySerializer

now serializer.dll has a MySerializer class which knows how to serialize/deserialize Model's.

August 7, 2012 | Unregistered CommenterAlex Rice

Anyone aware of a step-by-step tutorial for getting a simple data set using Protobuf-net into Unity 3.5 and deploying to iOS and Android using Mac OS X (and MonoDevelop) as a development environment? I'd like to do some quick benchmarking using Protocol Buffers, but I haven't found all the steps in one place to get this running quickly.

October 3, 2012 | Unregistered CommenterBrent A. Thale

To answer my own question above: This is what I had to do to get Protocol buffers(on Mac OS X 10.7.3) using protobuf-net (r580) working on Unity for iOS in a data-driven manner (using generated C# code):

1 Lay out a .proto file describing the classes you want to serialize.
2 Run "protoc" on the proto file, generating an intermediary binary file.
3 Run "protogen" on the binary file from step 2, producing a C# source file containing your classes.
4 Use mono to build the C# source file from step 3 into a DLL.
5 Run "precompile.exe" on the DLL from step 4, producing a serialization DLL, necessary because Unity for iOS doesn't support just-in-time compiling used by the non-Unity protobuf-net.
6 Write an offline tool to take your designers' data and use the serialization DLL to write out a binary Protocol Buffer file.
7 Add the DLL from step 4 (TestProto.dll below), the DLL from step 5 (Serializer.dll below) and the protobuf-net DLL from the protobuf-net "CoreOnly" folder (unclear on which version for Android) to Unity under Assets/Plugins.
8 Load the binary Protocol Buffer file from step 6 in Unity, use the serialization DLL to produce an object with the designer's expected values.

Issues: In step 2 if you're running on Mac OS X, you'll need to build "protoc", which is a Google-authored tool independent of protobuf-net available on the Google Protocol Buffers site. I downloaded protobuf-2.4.1, and compiled the source for Mac OS X to get a protoc that works for me.

Note that if you just install Unity 3.x you may not have the Mono command line tools for Mac which you kind of need for this. Go to the Mono site and download them so "mono" is available on your console.

For some reason, the examples on the protobuf-net site showing how to use protogen didn't work for me. He shows protogen working with one step but I have to use two or it fails, so I had to do this:
protoc -I. ProtocolBufferTest.proto -oProtocolBufferTest.bin
mono protogen.exe -i:ProtocolBufferTest.bin -o:ProtocolBufferTest.cs


Precompile command line:
// precompile is a managed .NET executable that takes a class library and produces a managed DLL that knows how to serialize deserialize the class.
// this is necessary because just-in-time compiling is not available on Mono for iOS, so we generate the code here.
// TestProto.dll is the C# DLL from step 4 above containing your classes.
// Serializer.dll is output from precompile.exe containing a class called "MySerializer" that knows how to serialize/deserialize the classes in TestProto.dll.
mono precompile.exe TestProto.dll -o:Serializer.dll -t:MySerializer

October 25, 2012 | Unregistered CommenterBrent A. Thale

4 Use mono to build the C# source file from step 3 into a DLL.
how to do no.4 use command, because i want to use on sript build all my .proto files.

July 12, 2013 | Unregistered CommenterMike

can you give me a script like python do this
1 Lay out a .proto file describing the classes you want to serialize.
2 Run "protoc" on the proto file, generating an intermediary binary file.
3 Run "protogen" on the binary file from step 2, producing a C# source file containing your classes.
4 Use mono to build the C# source file from step 3 into a DLL.
5 Run "precompile.exe" on the DLL from step 4, producing a serialization DLL, necessary because Unity for iOS doesn't support just-in-time compiling used by the non-Unity protobuf-net.
6 Write an offline tool to take your designers' data and use the serialization DLL to write out a binary Protocol Buffer file.
7 Add the DLL from step 4 (TestProto.dll below), the DLL from step 5 (Serializer.dll below) and the protobuf-net DLL from the protobuf-net "CoreOnly" folder (unclear on which version for Android) to Unity under Assets/Plugins.
8 Load the binary Protocol Buffer file from step 6 in Unity, use the serialization DLL to produce an object with the designer's expected values.

July 23, 2013 | Unregistered CommenterMike

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>