Advertisement

Why we have a lot of 3d graphic APIs?

Started by September 10, 2018 01:00 PM
25 comments, last by swiftcoder 6 years, 4 months ago

There are dozens of 3D graphics APIs. For example, OpenGL, Direct3D, GNM, LibGCM, OpenGLES, Vulkan, Mantle, Metal.

And all of them are created for the same purpose: hardware accelerated rasterization. And all of them are using the same techniques (algorithms) for rasterization, there is no difference between them, except apis'. Why we humans reinventing the wheel? This makes cross platform game development very hard. Because you must learn all of them, and create abstraction layer on top of them. Even shading languages are different. You must create source-to-source transpiler for solving this problem.

Sorry for the my bad english. I hope I expressed myself. Thanks.

At the risk of stating the obvious, and not mentioning any names, but some would say that the primary raison d'etre of some of these is precisely TO make cross platform development hard, and vendor lock-in both for developers and players. :)

Advertisement

There are big differences between high level and low level APIs - big enough to justify multiple APIs in case of GL/DX11 vs. VK/DX12. Console APIs can have even more low level features.

Beyond that most APIs are tied to a unique platform. Apple does not want to support Windows, Windows does not want to support Linux or Sony, etc. So Khronos APIs are the only ones targeted at multi platform development, but MS / Sony may not want to support that either, for obvious reasons.

First of all, there are ownerships and royalties using the API. Remember the Google-Oracle lawsuit which is still going on regarding the API use. API may, or may not, be copyrightable, but I don't think too many people want to get risked of being sued by any of the tech giants.

Using royalties-free API would of course make you free from those problems, still there are legal stuffs that you have to work with....

Secondly, general use API like DirectX, OpenGL, and Vulkan are designed with supporting as many hardware as possible. That doesn't sound when you have a handful of hardware to support, ie. PlayStation 4 or Nintendo Switch. I think for the platform owner, creating their own API makes them able to control the design of the api so it works well with the hardware. In fact, I think many Console's API are just a very thin wrapper around the hardware/driver.

Also they can eliminate the API's compatibility certification altogether.  Well at least I don't think PS4 needs LibGCM's compatibility certificate....

Lastly, probably the politics. ... Especially in the desktop-mobile area. Windows have support OpenGL before MS create DirectX (correct me if I'm wrong), and looks like they tried hard to deprecate it (not success it seems). Mac OSX also has support for OpenGL before Apple creates Metal. AMD creates Mantle because they want to win over Nvidia in terms of performance. Although they did not success doing so, Mantle was used as a baseline of creating Vulkan. Vulkan is pushed by Khronos group, which basically a group of hardware manufacturer, to balance the power between the platform owner (MS, Apple, Google) and the hardware vendors. 

PS. Please don't be sorry about your English. Mine is much worse :D

http://9tawan.net/en/

Also, its more than just re-inventing the wheel as not hardware is created equal. As an API designer you want features in the API to map directly to the HW as much as possible ( efficiency ), but with numerous different HW configuration, this is no easy task. Which then spawn the next issue, in order to be as efficient as possible on platform/hw that doesn't have direct mapping for API features, 'hacks' are then introduced into the API which muddies the interface and introduce unnecessary complexity.
However, the biggest reason, imo and others have mentioned this above is politics...

Three reasons come to mind...

1)  An API that works for a desktop or laptop might be too heavy for a mobile device so a lite version is used.  Example would be OpenGL and OpenGLES.

2) An API eventually becomes so out of date that the time has come start over from scratch( sometimes recycling code if it makes sense ).  I assume that this might be the case for OpenGL and Vulkan, with OpenGL being kept on for legacy while Vulkan is for new projects going forward.

3) A company needs an API that they can trust and control, and not have the rug pulled out from under their feet to safe guard the service life for a product.  Microsoft has DirectX and I think Apple now has a new API called Metal.

Languages; C, Java. Platforms: Android, Oculus Go, ZX Spectrum, Megadrive.

Website: Mega-Gen Garage

Advertisement

I would say that is a healthy competition for the market. Everyone wants their own piece of pie, and people prefer the ones who can offer more utility, compatibility, ease of access. 

That is driving them to compete with each other and push toward more features, even if it seems that they use the same algorithms, the applicability is still varies depending on the platform, and one is better than the other in different fields. 

Various capacities for rendering allows to not depend on one monopolized setting, and it is healthy for the market, for the developers and overall, when one goes down, there are others to fill the gap. 

On 9/10/2018 at 11:19 PM, lawnjelly said:

At the risk of stating the obvious, and not mentioning any names, but some would say that the primary raison d'etre of some of these is precisely TO make cross platform development hard, and vendor lock-in both for developers and players. :)

Sorry to be blunt, but that's just conspiracy theory tin foil hat stuff.

On 9/10/2018 at 11:00 PM, elgun said:

OpenGL, Direct3D, GNM, LibGCM, OpenGLES, Vulkan, Mantle, Metal

Story time. Let's start at the start, with GL.

OpenGL is meant to be the "open"/"portable" API, but it's just terrible.

Seriously, who though that this was a good idea:


//Instead of:
void DoThing(int foo);
...
DoThing(42);

//We should have:
enum SlotType { PARAMETER_ONE }
namespace internal {
  int g_parameterOne;
}
void BindSlot( SlotType );
void SetSlotValue( int );
void DoThing();
...
BindSlot(PARAMETER_ONE);
SetSlotValue(42);
DoThing();

If anyone ever did this kind of thing in an API ever again, they'd be slapped for such insanity. GL gets a pass because it grew out of 1980's supercomputers... and no one knew how to make a graphics API in the 80's, or what the future market for GPU's was even going to look like (originally -- a completely separate cluster of computers running over a network was probably more what they were thinking of!). Then after that ugly start, they just kept tacking more and more stuff onto that legacy for decades, mutating it into a horrible mess of conflicting ideas. It's an indefensibly bad, failure of an API, which is why we have others now. As a side note, when trying to bring GL to mobiles, it wasn't portable enough, so we got OpenGL|ES as a spin-off.

In the early days of Windows, a stupid company organisation issue / politics / schism meant that one team who wanted OpenGL, wasn't allowed to have it, so they bought out a different graphics company and re-branded their tech as D3D, and we ended up with two APIs on windows. GL was overseen by a committee who were constantly required to make stupid decisions because that's how committees work, while MS was able to be a "benevolent dictator" and completely rewrite D3D from scratch whenever they felt like it. By the time we got OpenGL v2.0 (which, incidentally, didn't actually throw out any of the v1.0 stuff!), we also had D3D v9 (that's a lot of rewrites!).

Game consoles were traditionally used by developers who were used to porting all their OS-level systems to new APIs, so they weren't demanding to use something like OpenGL everywhere. Game consoles also tend to be under-powered (if not at launch, then definitely at the end of their shelf-life), so devs prefer slim, efficient APIs rather than bloated portable ones.

Sony's PS3 API, GCM, was a very slim driver for NVidia's G71 architecture. It was incredibly low-level compared to PC API's at the time (D3D9, OpenGL2), and allowed developers to extract amazing efficiency out of that GPU. For example -- you can play GTA5 on a PS3, but there's no way you can possibly play GTA5 on a GeForce 7900GT (same chip). Microsoft responded to this low-level console API by modifying D3D9 into D3D9.x, which was based on the PC API, but slimmed down and massively extended into an unholy demon that powers the Xbox360.

We had the API war for a while - D3D9 vs GL2, D3D10 vs GL3, D3D11 vs GL4...
D3D11 was extremely mature and for a while, Microsoft was happy to say that it was going to be the final version of D3D (with just minor extensions added over time -- 11.1, 11.2, etc)... During this time Microsoft had been acting as the Quality Assurance mechanism on D3D drivers (shipped by Intel, NVidia, AMD), making sure that their drivers all behaved properly and implemented the D3D spec correctly. Meanwhile, GL's comittee had no such equivalent in the real world, so driver quality from Intel/AMD/NVidia varied greatly, including which features actually worked, which features were fast, and what shader code was allowed. To put on a tin foil hat, you could say that we actually had the different hardware companies trying to use game developers as pawns to damage each other's reputations -- e.g. NVidia would accept invalid shader code that they knew would (correctly) crash an AMD GPU, to make people think that AMD was buggy, when actually it was their competition playing dirty. Anyway, at this point it was pretty much accepted that D3D had beaten GL in the "API war" -- GL was now much more rarely used than D3D in big games, due to complexity, reliability, portability across hardware, driver issues, etc... D3D11 was the king.

Then a new generation of consoles came out, and as is their tradition, Sony created a new API, GNM, that was incredibly low-level compared to PC API's at the time (D3D11, OpenGL4) and was basically a very slim driver for AMD's GCN architecture. Again, Microsoft responded by creating another unholy demon -- D3D11.x, which powered the XboxOne. Side note: Because GNM is so low-level, they also created a wrapper around it called GNMX, which made it a lot closer to a D3D11 level of abstraction, making it easier to port games from D3D11 to PS4. Sony has also provided examples of wrappers around GCM/GNM that implement parts of GL|ES to help people who are trying to port mobile games to PS3/PS4.

Game studios who were enjoying these low-level APIs started petitioning for such a thing to exist on PC's. Microsoft again claimed that D3D11 was good enough for everyone, and Khronos (GL's committee) said that OpenGL5 could fill this gap and everyone laughed. Several big game studios worked with AMD to create Mantle, which was a proof-of-concept, low-level, cross-platform, cross-vendor (it was actually designed to be something that NV/Intel could implement, not just AMD-only) API for PC, proving that such a thing could be a good idea and work well enough to displace D3D11.

Microsoft immediately got to work designing D3D12 (as an answer to Mantle on PC, and D3D12.x to replace D3D11.x).
Apple immediately got to work designing Metal as a replacement for OpenGL / OpenGL|ES on their platforms, because GL is shit.
Khronos were still fiddling with OpenGL5 ideas so AMD dumped the Mantle spec in their laps and said "make this", so they did, and they renamed it Vulkan, after letting Intel/NVidia/Qualcomm/etc tweak the design slightly.

Mantle was very quickly retired, because it had served it's purpose of forcing change. It's gone now.
Metal was the first low-level API to be released in the mainstream, followed by D3D12, and finally Vulkan a year later.

So we end up with this situation where there's no clear winner. Depending on what platforms you care about, you might be able to pick a single API and use it on all your platforms, if you're lucky... But there's no true cross platform solution, so it's up to you to write your low-level rendering code in a portable style, where you can write multiple back-ends for different APIs.


Platform|Vulkan| GL  | GLES | D3D | Metal | Custom |
PS3     | No   | No  | Emu* | No  | No    | Yes    |
PS4     | No   | No  | Emu* | No  | No    | Yes    |
Xb360   | No   | No  | No   | Yes*| No    | Yes    |
XbOne   | No   | No  | No   | Yes*| No    | Yes    |
WiiU/Wii| No   | No  | No   | No  | No    | Yes    |
Switch  | Yes  | Yes | Yes  | Emu | No    | Yes    | <- wow!
Windows | Yes  | Yes | Emu  | Yes | No    | No     |
Linux   | Yes  | Yes | Yes* | Emu | No    | No     |
Android | Yes  | No  | Yes  | Emu | No    | No     |
MacOS   | Emu  | Old | Emu  | Emu*| Yes   | No     |
iOS     | Emu  | No  | Old  | Emu*| Yes   | No     |

Custom = an API designed just for that platform. Old = Yes, but no longer updated/supported. Emu = some kind of emulated / translated support. * = important caveats / should almost be 'No' or 'Emu'.

On 9/10/2018 at 11:00 PM, elgun said:

Even shading languages are different. You must create source-to-source transpiler for solving this problem.

We used the exact same HLSL code on PS3 (GCM), Xb360 (D3D9), PS4 (GNM) and XbOne (D3D11).
Official support for using HLSL to write shaders for Vulkan exists now as a work-in-progress.

It's only OpenGL and Metal that require shader translation :) 

On 9/10/2018 at 11:00 PM, elgun said:

This makes cross platform game development very hard. Because you must learn all of them, and create abstraction layer on top of them.

Use someone else's abstraction :) 
I've worked in this role at a games company, and would estimate about two man-months to port a game to a new API, which is not that much money when it comes to the total development cost of a game.

Certainly it is not all politics by any stretch of the imagination, and I agree that GL in particular has been plagued by a lot of historical cruft, however what I personally find notable is that DirectX has not been an open standard.

It is a difficult problem, as the fundamental techniques used to create 3d graphics have changed so much over the years (and will no doubt continue to change). There seems to be both a need to have a stable longterm high level API where perhaps some performance / access to techniques could be sacrificed, and a need for a changing lower level API that can 'throw out the last version' and have the most suitable access to the latest hardware paradigm (or multiple high / low level APIs).

My personal preference would be for some stable higher level APIs that 'translate' down to the some more changing lower level APIs that are also accessible. And for the APIs to be as open as possible, i.e. free to use and not patent encumbered, and not governed by any one company, although I understand the perils of design by committee. Another thing I would like is the option for standardised feature sets, so it would be possible to target a certain minimum feature set and be sure they were supported by the hardware.

2 hours ago, lawnjelly said:

There seems to be both a need to have a stable longterm high level API where perhaps some performance / access to techniques could be sacrificed, and a need for a changing lower level API that can 'throw out the last version' and have the most suitable access to the latest hardware paradigm (or multiple high / low level APIs).

Game/graphics engines fill this void, like BGFX or Unity :) If I was starting a new mobile + other platforms project, Unity would be a no-brainer due to how portable it is!
We have our own graphics API which is coincidentally quite similar to BGFX -- still much lower level than a retained scene graph, but way higher level than an immediate state machine of D3D/GL/etc. My preference is to go just high enough to make the underlying APIs stateless / use state-objects -- we did this on GCM/GXM/D3D9 and coincidentally, this turned out to map really well to pipeline-state-objects once the new APIs came out. We only had to make a few very minor design changes when Mantle came out, and we were ready to support VK/D12 back-ends :D

2 hours ago, lawnjelly said:

My personal preference would be for some stable higher level APIs that 'translate' down to the some more changing lower level APIs that are also accessible

OpenGL should really transition into this! There's no reason that OpenGL shouldn't be an open source library (instead of a driver) that's maintained by Khronos, and is implemented on top of Vulkan, Metal, GNM, etc... There's already VKD3D (D3D12 implementation that maps to Vulkan) and dxvk (D3D10/11 implementation that maps to Vulkan) which are being funded by industry titans as a way to make porting games to Linux easier ("just keep using D3D!"). It will be interesting to see if MS lets these projects slide, or if they'll challenge them in some way.

This topic is closed to new replies.

Advertisement