Rarity, and thus value, is determined by a number of factors. So while a book that's been in your family for generations may have great sentimental value to you, unless that book is a Gutenberg Bible or Shakespeare's First Folio it's probably neither rare nor valuable. There are several variables to consider regarding a book's value, and each is important:
1. Condition, condition, condition. Always buy a book in the best condition you can possibly afford. A book is not valuable simply because it's old, and a very old book in poor condition is worth little or nothing. For modern editions, the condition of the dust jacket is easily as important as the condition of the book itself when determining value. In fact, the dust jacket can account for up to 90 percent of the value of the book. For example, a "clipped" dust jacket (one where the original price on the inside cover has been clipped off) can cut the value of a book by 75 percent or more.
Books are graded according to condition. Typical grades include As New, Fine, Very Good, Good, Fair, Poor, Ex-Library, and Book Club Edition. You will often see "Near Fine" as well, and it is important to note that Book Club and Ex-Library Editions have next to no value except as reading copies. The problem, especially when purchasing books on the Internet, is that what one person calls Fine may in fact only be Good. If you are unable to personally inspect a book before buying it, at least ask for photographs of the dust jacket, binding, and copyright page.
2. In most cases, only the first printing of a first edition is of interest to collectors. This is one reason it is important to see the copyright page; especially for books printed in the past 20 years, the edition is typically clearly marked. There will be a series of numbers near the bottom of the page, and if a "1" is not visible, then you probably don't have a first printing. For example, you would want to see "First Edition" and 1 2 3 4 5 6 7 8 9 10 or 2 4 6 8 10 9 7 5 3 1. This is not true for all publishers, however, and it is worth the time and money to become thoroughly acquainted with the different ways some publishers identify a true first printing.
When speaking of first editions and first printings, we are always referring to the hardcover edition. The only exception to this occurs when the book has no initial hardcover run and is released only in soft cover. This is rare for literary fiction, but does occur more frequently in the mystery and science fiction/fantasy genres. When only a paperback first edition exists, the rules regarding condition still apply.
3. Unless a later book was particularly notable( for example, winning a Pulitzer Prize), an author's first book will always be the most valuable. This is because a first book is usually released with a small first print run, making the book scarce from the outset, and more so if the author becomes popular later. J.K. Rowling is a perfect example of this: the first UK print run of "Harry Potter and the Philosopher's Stone" ("Sorcerer's Stone" in the US) was a tiny 500 copies, of which 300 went to libraries; a first printing now sells for tens of thousands of dollars. The final book in the series, "Harry Potter and the Deathly Hallows," had a first print run of 12 million, assuring that this book will never be collectible unless it has Rowling's elusive signature. A book must be either scarce or rare, or both, to generate enough interest to cause the value to increase.
4. Signed copies are, in most cases, worth more than unsigned copies, but the rules regarding condition and edition trump an autograph. In other words, while a signed first printing of "The Kite Runner" in Fine condition can be worth well over $500, a signed copy of a fifth printing in Good condition will be worth less than the original cover price. Also remember that signed copies of books by an author who is hot today may be over-inflated, and could easily drop in value as time goes on. If you want to collect signed editions, the best way is to go to signings by the author where you can have them signed for free. Some authors will sign and return copies sent to them, but this happens far less often today than in the past. Always check with the author before sending anything.
One last thing to consider is that unless you are planning to become a full-time book dealer, you should stick to collecting books by authors that interest you. This way, even if the value of a particular book doesn't increase (or worse, decreases), you will still have a book in your collection that you actually want, rather than something you bought simply as a commodity.