Let me start by saying I'm pretty new at 3d modeling. I've followed tutorials and read guides for almost a year now, and there are still lots of things I'm uncertain about. I focused primarily on character and prop design, but now I'm wanting to try my hand at making a world.

I started by making a very basic 2d map in my favorite paint program, just to have a general layout of the world. Then I went into max and began with a 2048x2048 meter plane. I designated the segments so that each square would occupy 8 square meters, and then turned the whole thing into an editable poly. After a long time spent extruding and assigning dummy-materials, I had a boxy representation of what the world would be like in 3d. This is where I start to run into a wall.

For reference:
http://img408.imageshack.us/img408/1293/boxworldtop.png
http://img560.imageshack.us/img560/227/boxworld34.png
http://imageshack.us/photo/my-images/39/boxworldcloser.png/

First I considered trying to use that mesh as the base for the world. The problem comes when I want to smooth everything out. I want all those horizontal stair-steps that you see when looking at it from top-down to be lines, or at least curved edges. Relaxing doesn't do what I want, because it results in a bunch of bowing-out patterns instead of diagonal lines. Also, the fact that you can only relax in three dimensions means that it'll change the heights of the tops and bottoms of the edges I'm trying to align. So the only solution I've found so far is to select the verts of each corner and drag them into place manually, but I have a problem with this for two reasons: One, it results in some funny-shaped 4-sided polygons adjacent to the corners I'm trying to move. Don't know if that's conventional in the world of level design, but it certainly doesn't look pleasant. And two, it feels like it's too much effort, as if I'm going about this entirely the wrong way.

I'd like to know if there are any ways to speed up this process, or if I should just change my approach. So far, I haven't worked with terrain, because I find it kind of cumbersome and limiting. However, I could see myself starting over with just a ground plane, and making each mountain and rock formation as its own static mesh. With that said, how would I go about making these mountains smooth if I started from scratch? I'm really only familiar with the plane-modelling method right now, which would guarantee the staistep corners. I could try splines, but I fear that might result in some wonky >4-sided polys. Some direction would be greatly appreciated.

Hi Tixus,

I liked your first screenshot a lot, that top view. Terrains are really hard to work with, so I empathize with your difficulty.
First let's understand what's the problem: extruding sections of faces will give you that leveled look, so it's worth keeping it in mind if you ever want to go for that on purpose.

But you want a smooth, realistic terrain. There's an easy way to achieve this. You need two new things:

• A grayscale heightmap of your terrain.
• A Displace Modifier.
  Open up Photoshop or your favorite painting program and based on that first screenshot you have, start painting your grayscale heightmap. Make sure to keep some pictures of real mountains and ridges open so you can take a look and reproduce some of the features you see on them - the way the erosion works, the way they chaotically form etc.
  Once you have your heightmap ready, go to 3DS Max, create a plane with lots of segments but no more than necessary, add a Displace Modifier and inside the modifier's options, load your heightmap bitmap. Adjust settings so it looks good. If it does, convert it to an Edit Poly and bingo, you have an editable terrain. You can use Edit Poly's push\pull tools to paint further deformation on your terrain if you need to do some touch-ups, but prefer to apply them directly to the heightmap (this is especially important if your game-engine supports loading heightmaps).
  
  If you are satisfied with your terrain's shape then it's time to color it. There are a few ways to go about this, but it's very game-engine specific. I'll let you finish the above first.
  Report back if you want. Bye!

Thanks Kryzon. The advice is good, and I've been thinking about using heightmaps. My main concern is that it'll get rid of the straight-upward jutting nature of the walls, and the flatness of the surrounding ground Since I'm going for something a little cartoony and not so realistic, I want to keep some of the flatness on the terrain you can travel on. But with that being said, I'll definitely look into heightmaps for the mountain areas you can't travel on. What I'm thinking of is a simple plane of all travel-able regions (So I can make it as flat or consistently inclined as I want), and then have the mountains be their own layer that poke through the ground.

Try several levels of the Bevel tool in Edit Poly to make the edges more smooth.

You can still have flat spaces or sharp levels with a heightmap if you don't use any bluring and just use a single color for those areas.

Made a heightmap I was pleased with, and used it for the world terrain. Might have skipped a few essential steps. At a distance, the world looks roughly like what I was hoping for, but up close, things are pretty ugly. Considering that each square is vastly larger than the player in size, I don't think I can get away with the zig-zagging and stretched polys that seem to be a prominent feature in terrain modeling.

Reference:
http://img809.imageshack.us/img809/6744/heightworld1.png
http://img6.imageshack.us/img6/8240/arrghh.png

Those zig-zagging polygons are the same problem I ran into earlier, only now I fear I have less control than when I extruded. I've considered two approaches:

1) Use splines to create static meshes of every mountain. Would give me the smoothness I want, but there are always going to be that N-sided polygon on the top, no matter how I drag in the edges and cut. But all game engines I know of use triangles anyway, so is it an issue if I end up with a bunch of quads and a few stray triangles?

2) Break the map up into a bunch of 512x512 regions (This'll require some simple reorganization of the layout). Then build separate heightmaps with a higher polygon resolution to try and get rid of the jagginess. But it comes with some big drawbacks--I don't particularly like how terrain tends to use inclined polygons to approximate straight surfaces, leading to some serious stretching of textures. Also, I don't know a thing about dynamic terrain loading in Unity or UE3, but I get the feeling I might have to start building my world with "loading regions" in mind.

I'm not entirely sure what how many polygons I should be budgeting for this kind of endeavor. In an open-world game, how many polygons typically make up each map? I assume they split the world up into smaller regions, on account of trying to model an entire world would be too intense on hardware.

I can understand why that happens; it's just the best way the quadriculated plane can represent those sharp shapes. I see no way to solve this.
I agree, you should try to manually model your terrain. It's going to take time but it'll look exactly like you want. After you have a flat shape, check the Subdivide Modifier.

1) Use splines to create static meshes of every mountain. Would give me the smoothness I want, but there are always going to be that N-sided polygon on the top, no matter how I drag in the edges and cut. But all game engines I know of use triangles anyway, so is it an issue if I end up with a bunch of quads and a few stray triangles?

Every quad or multiple-sided polygon is considered as triangles internally by 3DS Max.
Engines only work with triangles. In 3DS Max we have the illusion that there are other types of polygons with more sides than 3, but really it's just their edges that are hidden. If you want to see it as triangles you can convert your mesh to Editable Patch to make the triangulation visible. Then you can convert again to editable poly to keep the triangles visible or just undo and continue working as you were. It's not something you should worry about.

2) Break the map up into a bunch of 512x512 regions (This'll require some simple reorganization of the layout). Then build separate heightmaps with a higher polygon resolution to try and get rid of the jagginess. But it comes with some big drawbacks--I don't particularly like how terrain tends to use inclined polygons to approximate straight surfaces, leading to some serious stretching of textures. Also, I don't know a thing about dynamic terrain loading in Unity or UE3, but I get the feeling I might have to start building my world with "loading regions" in mind.

I'm not entirely sure what how many polygons I should be budgeting for this kind of endeavor. In an open-world game, how many polygons typically make up each map? I assume they split the world up into smaller regions, on account of trying to model an entire world would be too intense on hardware.
[/quote]
Polygon budget is something you can research into, I remember a list going around with the polygon budget for several games. Keeping your entire environment at 60k triangles is very safe.
You should only worry about breaking your level into smaller sections if your engine doesn't support any form of optimization (quadTree for terrains, Octree for game environments etc).
This is why it's best to keep your terrain always in heightmap form, because most game engines can load that heightmap, keep it in memory and use algorithms to represent that terrain in real-time with more detail in the foreground and less in the background, wherever the camera lies. The terrain is procedurally generated each frame.

Thanks for the info about triangles. I was always wondering about that. Now for my concern with the logistics of the world I'm trying to build:

The size of the world is roughly 2 square kilometers (2048x2048). While I agree that using heightmaps and terrain is the ideal way to build and store the world, there are some serious issues with the numbers I'm running into. If each pixel represents a polygonal space on the map, then just using a 2048x2048 bitmap would use more than 4 million quads. Each quad would occupy a square-meter of area, which is too large. Subdividing and increasing the plane segments would help this, but it would turn into more quads than my system can handle...and that's without converting to an editable poly.

After some reading (Of some very vague sources, I really couldn't find much), I'm willing to go up to between 300-500 thousand triangles for the world. What I could really use, but unfortunately aren't available, are some reference numbers to poly counts of environments in existing open-world games...but all of my searches just lead to character poly counts. I guess low poly scenes I could actually load and look at in max would also help me out, but again, I can't seem to find much of anything.

I recommend 60k ~ 200k. You just have to worry about polycount budget when you're building the level and getting close to blowing it. Worry about making the level first, and keeping it consistent with your art direction and concept - these have higher priorities than polycount.

A heightmap does not map 1 pixel to 1 quad. The heightmap is projected onto a mesh, so it can have any size you want. It would only map 1:1 with the mesh if you scale your mesh to have the same size in 3D units as the size in pixels of the heightmap (if the heightmap is 2048 pixels, you would get 1:1 ratio by having your mesh be 2048 world units wide, with each quad on it being 1 unit in size, representing one pixel each).
But you can have it any size you want: you just have to scale the texture down or up, fiddling with the Mapping Gizmo of the Displace Modifier as you please. Presto!   

A heightmap does not map 1 pixel to 1 quad. The heightmap is projected onto a mesh, so it can have any size you want. It would only map 1:1 with the mesh if you scale your mesh to have the same size in 3D units as the size in pixels of the heightmap (if the heightmap is 2048 pixels, you would get 1:1 ratio by having your mesh be 2048 world units wide, with each quad on it being 1 unit in size, representing one pixel each).
But you can have it any size you want: you just have to scale the texture down or up, fiddling with the Mapping Gizmo of the Displace Modifier as you please. Presto!

Yeah. Forgot to mention that my heightmap was in fact 2048. Going higher than that for better geometry would mean even more quads that I'm sure my computer wouldn't be able to handle. And with that, I think I'm going to build the mountains out of static meshes. No more talk of heightmaps. I'm going to be heading away for the holiday, so I'll probably have more questions in a week or so. Thanks for all the guidance so far.