Profit margin. It varies, but a good seller (no matter what the item) will try to sell an item for a 60% profit. It is a good target that leaves room for negotiations, "friendly deals", etc. As a collector your goal is to break even. So buy below the catalog value that includes shipping.
Unless you like them, don't buy penny notes, the common notes, they will just waste money. Before you buy your first note, decide what you want to collect and buy as near to UNC you can. Most of the time, it is better to buy one 100 dollar banknote, than 100 "one dollar" notes. When you go to sell the 100 "one dollar" notes you may be hard pressed to find a dealer who will give you 50 bucks for the lot. And very few of these will increase in value. Where as the 100 dollar note you may come closer to selling it for 100.
It is the same rule for pretty much everything. Buy low, sell high.