Hello! I'm creating a terrain/heightmap editor and am wondering what the typical "Smooth Terrain" tools algorithm would be. I'm currently using the Box Filter algorithm (take the surrounding heightmap pixels and average them together to get a final height). But this doesn't produce satisfying results, nor does it look like what Unity3d's terrain smoothing tool produces (Unity3d being an example of what I'm looking for). Does anyone have any ideas as to what Unity or other terrain editors use for there smoothing tool? Thank you!

Increase the size of the box? Sorry I don't know unity, but I'm pretty sure if you set each heightpoint to the height of the average in a large box instead of the average neighbour's height, then you should see a more dramatic effect?

Use gaussian blur (the heighmap is just a grey-colored greyscale image).

the heighmap is just a grey-colored image

:P

More seriously, Ashaman is right. What you can further take from his statement is that a heightmap can be treated like any single channel image, thus you can use whatever image filter you want on it (blurs, distortions, noise, low pass, high pass, etc).

Alright I'll go ahead and will try both increasing the box size and the new filter. Thanks guys! I'll make sure to get back to you as soon as the changes have been done.

Here the code I use to smooth a part of the terrain or the whole heightmap when I import from an image :

float CTerrain::Average( const Int32 i, const Int32 j ) const
{
  float avg = 0.0f;
  float num = 0.0f;
  for( Int32 m = i - 1; m <= i + 1; ++m )
  {
    for( Int32 n = j - 1; n <= j + 1; ++n )
    {
      if( InBounds( m, n ) )
      {
        avg += m_Heightmap[ ( m * m_InitInfo.Width ) + n ];
        ++num;
      }
    }
  }
  return avg / num;
}

Is it really better to use a gaussian filter ?