Does anyone here have experience with transfer learning?
I found Berak's work here:
https://berak.github.io/smallfry/transfer.html
and here:
https://answers.opencv.org/question/191359/ml-svm-k-nn-image-recognition-examples-in-c/#191374
However, I am trying to teach the network not image classification, but rather the quaternion multiplication operation. Does your favourite transfer learning method support input and output with larger range than just 0 through 1?
“transfer learning” and “cell activation range” have approximately nothing to do with each other.
The valid range for values in a neural network is based on its underlying implementation (e g, INT8 accelerators or FLOAT32 based cores) and the activation functions (e g, ReLU can do 0 .. whatever, but tanh can only go -1 .. 1)
Well, the traditional ANN accepts input components with larger than 1 for a value. Likewise for the output.
@taby “transfer learning” is a method to bootstrap some model from some other model. “ANN” is a neural network topology, and whether it accepts values greater than 1 depends on where you run it. For example, if you run it on the typical INT8 cores used by “edge device” accelerators (anything from IoT custom chips, to the NVIDIA DLA cores) then your network can literally not represent a value outside the 0 .. 1 or -1 .. 1 range.
You might ask whether “shopping carts” support “oranges.” You can usually put one into the other, and if one is too big (or the other too small, say, a doll sized shopping cart) then that might not work, but they aren't linked any tighter than that.
Instead of using transfer learning, we used a custom neural network in PyTorch. I trained it to do multiplication, and for large batches, it runs faster than the traditional multiplication. The code is at: https://github.com/sjhalayka/ann_quat_fractal
The image below shows a quaternion Julia set generated by the AI (left) and by traditional multiplication (right). Pretty close!
Something doesn't add up here. A quaternion multiplication is something like 16 multiply-add operations. You say your neural network is faster, but evaluating it involves something like 11,620 multiply-add operations. If it turns out that doing the calculation in Python is so abysmally slow, you can do the computation using NumPy, SciPy or even PyTorch. But using a neural network for this is insane.
