suppose i have 4 solids like this:
and the solid have shear stress and tensile stress 12N/mm2 and 15N/mm2 respectively,
suppose i know the area of each surface,
how can i judge whether this solid is able to withstand 20N load?
what kind of formula should i use?

This is a simple question with a complicated answer. You need two things, a material model, and a structural mechanics model.

A structural mechanics model, like the Finite Element Method, will give you element stresses. This is not a difficult method to implement assuming the following, you aren't interested in fracture, you aren't interested in absolute accuracy. Violation of either of those assumptions makes this problem particularly difficult.

The material model can be relatively simple, assume an elastic material and define some failure stress. If the element stresses exceed the failure stress then you know the solid has failed. You could make this more complicated by using a plasticity model.

To fully explain these techniques is beyond the scope of a forum thread. I would suggest getting a good book on FEM, and studying one of the many open source codes available.

brucedjones is correct. You should probably look up a good material model. However, you can also do a quick and dirty answer since you know the stresses.

If you know the material (steel, wood, etc.) why not just compare the stresses to the material's respective yield stress? This would allow you to see if it would fail under a elastic/perfectly plastic representation. You could then avoid brucedjones suggestion of using a plasticity model. BTW, yield strength in shear is usually around half of the materials yield strength.

Also, I'm assuming some when you say "withstand." If you want to go a little more advanced without more work, you can go with the fracture strength of the material. This will allow the material to stretch due to excessive forces before failure. The fracture strength in shear will be around 3/4th that of the materials fracture strength.

If you want to see the shape change, unfortunately you'd have to employ a good constitutive equation, which there are a lot of good ones. However, you'd have to go into some finite element modeling.

Yup, definitely a finite element problem. Normally, the blanket answer is to use finite element programs or write your own (not the best option), but I guess it could depend on what you're trying to do. You might be able to get away with a simpler solution than implementing a general method. For example, if all your elements will be of a certain type (rod, plate, tet, etc.), then you can possibly reduce the amount of work you'll have to do. I don't know if that helps, but if you detail out more what you're trying to accomplish in the long run, maybe there's a better option available for you.