Questions :-

• To sell a C++ game library/framework/engine, does it usually have to be open-sourced? 
• For a C++ template class (real template), it is quite hard not to be open-sourced, correct?
• If it is open-sourced, how to protect the copyright?
• How to enforce people to just use it, but not copy?
• ... I think it is practically very hard, because people can see the source code, and "rewrite" it.
• Does a shop like Unity's asset store make thing safer?  If so, how?

Arrogant assumption :-

• The game library/framework/engine is high-quality, well-documented, and it would be marvellously sell well.
• Besides some free libraries like Bullet and Ogre, it is developed all by myself.
• The library/framework/engine is not bound to any legal obligation (e.g. the most restriction is MIT).

I have these questions after reading a news  :-

• Muhagames develop a game named Thea Awakening (C#).
• The company use its own Honey Hex Framework (C#) as a library to develop the game.
• The company sell Honey Hex Framework via Unity's asset store to boost its financial position.

There are also others old examples e.g. Unity itself or Unreal, which is now open-source ... How do they protect it?

Actually, I don't plan to sell any library. I am just curious.

• To sell a C++ game library/framework/engine, does it usually have to be open-sourced? No.
• For a C++ template class (real template), it is quite hard not to be open-sourced, correct? Yes, in the sense of "the code is visible". 'Open source' usually implies the use of certain licenses, and using those is entirely your choice (unless you're using someone else's code too...)
• If it is open-sourced, how to protect the copyright? Keep an eye out for infringement, then send lawyers.
• How to enforce people to just use it, but not copy? "Open source" usually implies that copying is okay, since most open source licenses say so. But if you're selling a library with full source code but don't want it copied, it's just down to the lawyers again.
• ... I think it is practically very hard, because people can see the source code, and "rewrite" it. That's not a question.
• Does a shop like Unity's asset store make thing safer?  If so, how? No. You can sell libraries there without including the source, but you could do that anyway.

Kylo's responses are on point, but want to add a couple notes:

1. If you release your library as open source, you aren't necessarily required to release any other source code BUT the library itself depending on the you choose. LGPL is usually a good bet if you want to avoid having to release the code as well. 

2. The objective of enforcement under a copyleft/open source license is to prohibit people from releasing the code without attribution or without releasing the library or whatever code is shared as open source. So if you're going to enforce your open source license, you need to make sure they're actually "infringing" under the license itself, not under a traditional notion of copyright infringement. 

3. Under any EULA you can specify what people can and cannot do with code that's made available. As with any license breach, if they fail to comply, that's infringement, because the license becomes invalid. So you can say in the license "licensee may use, modify, improve, update, or create derivative works of the Code (herein "Modifications"), but may not directly copy the same absent Modifications." Please note that this isn't something you'll see in an open source license-- it would be unique to your own EULA. 

4. Unity has its own Terms of Service, which does offer some limitations of liability, etc., but those are mostly to protect Unity, not you. You will still want to make sure you have your own EULA for anything you publish or sell if you're concerned about what people can and cannot do with your content, and how you can be held liable. 

Thank.  

As a follow-up question, assume a below situation :-

• Mr.B1,B2,B3,..., and Mr.B1000 signs contract to use a game engine from Mr.C.
• In the contract, Mr.C get 10% of revenue of every game that Mr.Bx developed using the game engine. 
• Mr.C send header files and .lib to every Mr.B via email.
• Some Bs successfully deployed the game and paid C, but some Bs abandoned and used other engines instead.
• However, B1 is different from other Bs.
• Using C's game engine, B1 has developed a game without any collaborator.
• (^ Thus evidence is very hard to find.)
• After finishing the game, Mr.B1 wanted to cheat. 
• Mr.B1 obfuscated C's header and .lib, and sell a game using the engine without paying C.
• Furthermore, Mr.B1 accidentally leaked C's obfuscated game engine in the internet. 
• A random Mr.D found the copy. 
• Mr.D knew that it is actually C's, but decided to use it.  It is free.

How can C detect such case?   It is practically impossible to detect, right? 

Reason :-

• C can't access to source of every game in the market, including B1's game and D's game.
• Suppose that C can, there are a lot of game sold every day!
• If C's engine is also open-sourced (not free; it is like some contract of Unity/Unreal; easier to obfuscate), possibility to detect is almost zero.  Correct?

Don't be paranoid. Assuming you see yourself as C, if you catch copyright and/or license infringement, you are welcome to unleash the lawyers (if it's worth the expense); if you don't, remember that your business is selling a game engine to the largest possible number of well-behaved B's, not wasting effort on quantitatively unimportant and/or imaginary pirates like B1 and D.

On 11/16/2017 at 4:58 PM, LorenzoGatti said:

... remember that your business is selling a game engine to the largest possible number of well-behaved B's, not wasting effort on quantitatively unimportant and/or imaginary pirates like B1 and D.

That totally sounds reasonable, thank.