I must support non-uniform scaling and skewing, my main intention is use this engine as basis of modeling software and then I want to build a game engine on top of it later as second (or parallel) purpose.

To do that I need 4x4 matrix (or least 3x3) as local transform (if exists). But I can provide an option/configuration to switch to quaternion then non-uniform scaling and skewing will not be supported in this configuration, like this:

gk_opt_set(GK_OPT_TRANSF_ALWAYS_USE_QUAT, true);

Probably there will be one or a few indirect (pointer) function call cost in every frame. After finished implementing all features, I will try to eliminate all costs. In the end the user will decide what should be supported or not

16 hours ago, galop1n said:

i even did not mention that you do not need full precision floats for a quaternion if you really want to push it ( i pack the TBN of my meshes in a single HALF4 + sign for example, other even encode it in 32bits total ).

Cool! I never used 16 bits float so added it to my and cglm's TODOs (https://github.com/recp/cglm/issues/15)

Does your engine is accessible on internet or in-house? 

You can have a glimps of the latest engine i worked by buying Black Ops 3 haha

May i ask why it has to be a modeling tool ? Because their is a lot of cons not just relying on existing solution like maya/max/modo…

2 hours ago, galop1n said:

You can have a glimps of the latest engine i worked by buying Black Ops 3 haha

I'll buy it if it can run on my macbook pro (or wine or virtual windows). 

2 hours ago, galop1n said:

May i ask why it has to be a modeling tool ? Because their is a lot of cons not just relying on existing solution like maya/max/modo…

I think I can't answer this as I feel, as I programmer I like drawings including 3D, but I'm not painter or 3D modeler. I want to build 3D modeling, rendering and animation software/tool plus game engine, I want to see what I can build. There is no realistic business plan right now (maybe never will be), maybe it is just hobby. Maybe I will fail who knows, but I'll continue what I started.

AssetKit (http://github.com/recp/assetkit) is born because I needed to an importer, then math lib, then renderer, data structures, ui... While I'm trying to achieve my goal, I think I'm also contributing to open source by providing new libraries (most of them is C but C++ helpers/wrappers can be written later). After finished shadow mapping and OIT Transparency I will back to AssetKit then I will implement new features e.g. PBR materials or animations... then I will implement the new features to my renderer which is called  Graphics Kernel or Graphics Kit... This is my current loop  

17 hours ago, recp said:

I want to build 3D modeling, rendering and animation software/tool plus game engine, I want to see what I can build. There is no realistic business plan right now (maybe never will be), maybe it is just hobby. Maybe I will fail who knows, but I'll continue what I started.

Well yes, but... 20 years ago, maybe 12 years ago, this was affordable. But today, things are different. I mean, yes, almost any programmers with some 3D skills (understanding space coordinates, trigonometry, vectors and matrices) can start an engine, a modeler or a simple ray tracer. But where things start to become less easy, are the things that start to become interesting. But you'll have to spend time until arriving to these points.

If you have that time, then go and try all what you want. But if you don't, just surfacing these will not give you enough information about the real today's questions of each. Maybe I'm wrong there, but this is really what I think.

17 hours ago, recp said:

AssetKit (http://github.com/recp/assetkit) is born because I needed to an importer, then math lib, then renderer, data structures, ui... While I'm trying to achieve my goal, I think I'm also contributing to open source by providing new libraries

There is assimp which is widely used. For 3ds only, there is lib3ds. Both these libraries are fine for 'simple' use. But both of them (the latter might be dead thought) would need more efforts for supporting better the formats they are supposed to support (ie assimp fbx is really shitty, both assimp and lib3ds can shit when using non-conventional 3ds files - ones without nodes). Thus making professional softwares to rely on non open/free libraries. Thus making people like you and me to become frustrated because the model/scene we wanted to use looks like an awful haystack (since we use these free libraries)

17 hours ago, recp said:

I will back to AssetKit then I will implement new features e.g. PBR materials

Looks like something lacking in the open source world. I would like to follow this.

56 minutes ago, _Silence_ said:

Well yes, but... 20 years ago, maybe 12 years ago, this was affordable. But today, things are different. I mean, yes, almost any programmers with some 3D skills (understanding space coordinates, trigonometry, vectors and matrices) can start an engine, a modeler or a simple ray tracer. But where things start to become less easy, are the things that start to become interesting. But you'll have to spend time until arriving to these points.

If you have that time, then go and try all what you want. But if you don't, just surfacing these will not give you enough information about the real today's questions of each. Maybe I'm wrong there, but this is really what I think.

Thank you for sharing your thoughts, I want to arrive (or pass) those points. I'm learning lot of things while working on these stuff and it makes me happy to spend / spare my time on these projects and sub libraries

1 hour ago, _Silence_ said:

There is assimp which is widely used. For 3ds only, there is lib3ds. Both these libraries are fine for 'simple' use. But both of them (the latter might be dead thought) would need more efforts for supporting better the formats they are supposed to support (ie assimp fbx is really shitty, both assimp and lib3ds can shit when using non-conventional 3ds files - ones without nodes). Thus making professional softwares to rely on non open/free libraries. Thus making people like you and me to become frustrated because the model/scene we wanted to use looks like an awful haystack (since we use these free libraries)

AssetKit will full support COLLADA 1.4, 1.5+ and glTF. It is not finished yet but it has nice features already; single interface for COLLADA 1.4, 1.5 and glTF, hierarchical fantastic memory allocator, small binary size, you can convert document's coord-sys to any new coord-sys e.g. convert LH -> RH or even custom one while importing (or later), it can load sub / external COLLADA files and its files (relative to document or any URI including HTTP), and caches (until freed), fixes external doc's coord-sys for opened one... Also it supports technique_hint e.g. by settings ak_setPlatform("PS3") will affect how to load / select instance effects, computing normals, bbox while loading (optional)...... User can enable/disable some features: https://github.com/recp/assetkit/blob/master/include/ak-options.h I like to provide options. It will support some other formats e.g. .obj as extra library

I'm happy with results I hope it will be better by time

I updated the way to creating matrices, I will update it for CSM, it seems work with aliasing problems, I hope I will fix that with CSM/PSSM

void
gkTransformsForLight(kScene  *scene,
                     GkLight *light,
                     mat4    *viewProj,
                     int      splitCount) {
  mat4      view, proj;
  GkCamera *cam;

  cam = scene->camera;

  switch (light->type) {
    case GK_LIGHT_TYPE_DIRECTIONAL: {
      vec4   *corner, v;
      vec3    box[2], target;
      int32_t i;

      memset(box, 0, sizeof(box));

      glm_vec_add(cam->frustum.center, light->dir, target);
      glm_lookat(cam->frustum.center, target, GLM_YUP, view);

      corner = cam->frustum.corners;
      for (i = 0; i < 8; i++) {
        glm_mat4_mulv(view, corner[i], v);

        box[0][0] = glm_min(box[0][0], v[0]);
        box[0][1] = glm_min(box[0][1], v[1]);
        box[0][2] = glm_min(box[0][2], v[2]);

        box[1][0] = glm_max(box[1][0], v[0]);
        box[1][1] = glm_max(box[1][1], v[1]);
        box[1][2] = glm_max(box[1][2], v[2]);
      }

      glm_ortho(box[0][0], box[1][0],
                box[0][1], box[1][1],
                box[0][2], box[1][2],
                proj);
      break;
    }
    case GK_LIGHT_TYPE_POINT:
    case GK_LIGHT_TYPE_SPOT:

    default:
      break;
  }

  glm_mat4_mul(proj, view, viewProj[0]);
}

I always used Y_UP here because I thought that since all models/primitives will be transformed with same matrix even it maybe be upside down or rotated somehow, and since we don't show the results to user without depth visualizing and it is just depth testing, so it will not be problem I think, please correct me if I'm wrong, I don't want to do that in wrong way. Any feedback is welcome to fix the way for creating matrices if there is better way.

The next step is CSM/PSSM I will follow tutorials to implement it.

What if dot(GLM_YUP,lightdir) == 1/-1 ?

26 minutes ago, galop1n said:

What if dot(GLM_YUP,lightdir) == 1/-1 ?

Goog point! It was attractive at first, I must find a perpendicular one  Before splitting frustum this implementation must be correct

I found a way to find perpendicular vector easily, I like it: https://www.quora.com/How-do-I-find-a-vector-perpendicular-to-another-vector
second answer (Tom's). I must add this to cglm

if v1 = (x, y, z) one possible perpendicular is v2 = (y-z, z-x, x-y) and it makes dot(v1, v2) = 0

Updated version:
 

up[0] = dir[1] - dir[2];
up[1] = dir[2] - dir[0];
up[2] = dir[0] - dir[1];

glm_vec_normalize(up);

glm_vec_add(cam->frustum.center, dir, target);
glm_lookat(cam->frustum.center, target, up, view);

What do you think now?