As far as the nonexistent history books on modern game designs goes, the originator of "achievements" as a term and systematical categorization of gaming accomplishments began with the Xbox 360 and Xbox Live, the 360's online system/marketplace. A "gamer score" is displayed prominently for every Xbox Live player profile (see mine at the top of this page) and points are added to this score whenever an achievement is unlocked during the playthrough of any given game. Microsoft's system allots 1000 gamer points (G) to any retail game and 200 points to any game distributed through the Xbox Live Arcade Marketplace. A typical achievement, then, has anywhere from 5-50G attached to it and when it is unlocked by a player's in-game exploits, a box pops up saying that an achievement has been unlocked. And, in a majority of cases, this elicits a very positive player reaction. Achievements have become such a mainstay of the Xbox 360 that there sites that are dedicated to categorically listing every achievement for games. For some gamers, the concept of having a high gamer score is important enough to some gamers that games are played ">solely for the purpose of getting easy achievements.
Achievements serving as one of the foundations of Xbox Live created a very unique situation for what are, essentially, a loose system of decorations over in-game accomplishments that gamers were previously doing just to say that they were able to do them. Back in the days of Final Fantasy VII gamers toiled over nigh-impossible boss encounters like Ruby and Emerald weapon that were, in the context of the game, completely and totally optional. Not that I was one of those people, but I was. The Xbox Live achievement system, then, reinforces a metagame that gamers have participated in since video games were created: competition is fun and, in single-player games where there was no form of competition against other human opponents, people got creative about aspects of a game that could theoretically be used as competitive set pieces. Born from this desire for competition are encounters like Emerald and Ruby Weapon and, lest anyone forget, speed runs.
Such forms of competition and accomplishment are simply out of the picture for more casual gamers and there isn't a person in the world who doesn't enjoy the feeling of an accomplishment. As a result of the success that the Xbox Live achievement metagame, other developers have implemented their own in-game achievement systems. What's more is that other developers have learned something that Microsoft hasn't (or cannot): achievements can be more than a pop-up message box. The recently released Saints Row 2 rewards players with tangible in-game objects when he/she completes optional minigames; for instance, base jumping a certain height will yield a permanent invulnerability to falling. Unlimited ammo for Uzis and pistols are rewards for other diversions in the game. Team Fortress 2 gives players access to entire new weapons once a certain number of achievements (for the classes that have them) have been met; these achievements are also permanently stored in a player's account in a pseudo recreation of the Xbox Live achievement system.
The goal in both of the above scenarios is in line with the widely-understand concept of achievements: positive reinforcement for gamers that doesn't alter the base gameplay in a negative way. No developer wants players to run through their games a single time then ship the game back to Gamefly. Achievements have, in part, made owning a single-player-only game more enjoyable due to the increased replayability that some gamers find within achievements. Earth Defense Force 2017 rewards players who beat the game once for each difficulty -- but does that mean that players are forced to perform those tasks to fully enjoy the game? Of course not, but there is a widely-recognizable "badge" to those players who choose to do so.
The problem with achievements tend to lie in games which encourage negative behavior in multiplayer games. The first example that really stood out to me is one in Halo 3 where an achievement is unlocked for getting a triple kill with the sword. This isn't an inherently bad idea but, in practice, there are people running around in team games with swords trying to end up in an uncommon scenario where they find three enemy opponents in a row to kill; such behavior isn't really conducive to, say, a team game of capture the flag. There are Team Fortress 2 achievements which reward Medics who play unnecessarily offensively in games instead of focus on healing (which is what the class is typically needed the most for).
The Halo 3 and Team Fortress 2 examples are negligible in their importance but they do serve a useful point in that game designers should rely on achievements as a certain kind of psychological reward for players. Games are, first and foremost, a form of entertainment. Achievements are a form of positive reinforcement to gamers in spite of whatever challenge they may be facing as a result of the game design; they're not typically a tangible enough incentive to completely negate the frustration of a player who has died one hundred times, but they are a nice piece of candy to be distributed without compromising the integrity of a design. Of course, a game could always follow the Ninja Gaiden 2 example with its "Indomitable Spirit" achievement which is unlocked once a player has died and continued his/her game for the hundredth time.
With most games it's like: "Ok, I finished the game on the hardest difficulty using every playable character - Nothing left - Next game..."
If you had an achievement, say "Finish the game on the hardest difficulty, without ever wearing any Armour" - This would add additional value to the game.
Ok, yes, I can finish the game on the hardest difficulty with every playable character - But am I good enough to do it, without any Armour?
This adds an additional challenge, and additional gameplay tactics, and possible human interaction ("OMG! How the *&^% did you kill the Uber-Spike-Shooting Boss on Lvl 3, with no Armour???" - "Simple - Every time he shoots, just roll twice." - "Oooooh!") and also bragging rights ("HOLY *&^%! You killed every dude in FPS_Game_X, using only head-shots! WOW!")
It is also easier for developers to add achievements, as opposed to full expansions ("Ok, we've got a kill-counter - What about an achievement when the player gets 10,000 kills?")
To sum up, achievements add in many ways to the value of the game, and make the game last longer, and more fun to play.