Like terrain that's sent from the server to the client and can be edited.So far I've tried making voxel and tile based terrain,but they both look ugly and squarish...what else is there?

You can use voxel terrain and use an algorithm to polygonize it in a non blocky way (marching cubes)

There are also libraries which can handle that for you i think.

Like are you talking minecraft style? Or just a useful tool for terrain creation?

If it's the latter, I'm a fan of world machine. It's flexible, versatile, and can export to just about any SDK you can think of. The program was created by one guy, and its interface is flowgraph based. You can get a trial version at http://www.world-machine.com/








Its got alot of cool options that save you some of the labors usually reserved for environment artists, here's an example of something I exported form world machine to Cryengine 3





Hope that helps! This was my first post, hello world!

Wurmonline is great example.

World machine is a great program. But i think OP asks for an player editable terrain. Since you have already tried voxels most possible next step is to try marching cubes like Waterlimon suggests.

Note: Shouldn't this post be on programming forums?