If you notice odors while you are using the fireplace, then you almost certainly have a smoking problem. See the article “Smoking Fireplaces” for details on identifying and correcting fireplace smoking problems.
As a general rule, in any case of fireplace odors, a chimney cleaning is in order if you haven't had it done recently (and certainly, if it has been over a year and you use the fireplace). A buildup of soot in the chimney will increase odor problems significantly.
A special case – fireplaces that once served a wood stove
If your fireplace once had a wood stove connected to it, it is possible that there might be considerable deposits of creosote in the flue. Creosote, especially when wet, has a very powerful odor. And since it poses a serious fire hazard, even small amounts must be removed by a professional chimney sweep for fire safety. If your fireplace ever served a wood stove, have it checked by a chimney professional.
If you have had the chimney cleaned and you still have an odor problem, here are some possible causes and solutions.
Open fireplace damper
If the damper is open and you are not using the fireplace, close it, to minimize air movement from the chimney into the house.
A chimney cap helps prevent rainwater and snow from entering the flue. And since moisture increases odors, a drier chimney will tend to emit less odor. If you don't have one, ask your chimney professional about having one installed.
If it has been rainy or wet lately, a masonry chimney might be soaked with water. Especially if the chimney is built up the outside of the house, water penetration can be a problem. Water draws heat from the flue, reducing the natural draft that helps draw odors up through the chimney and out. Sometimes having a water-repellent treatment (available through your chimney professional) will help.
If the chimney is too short, you will not only be likely to experience smoking problems when using the fireplace, but you will also be more likely to experience odor problems. Chimneys that are too short tend to have inadequate draft to draw smoke and odors up and out. Details on correct chimney height are provided in another article. The solution is to add height.
A note on odor problems: Poor chimney draft, and the resulting problem of odors entering the home, can also be caused by depressurization of tightly-insulated homes. If the house is so tight that there is not enough air available to replace the air leaving the house up through the chimney, then the chimney draft can actually reverse, causing air to be drawn into the house from the chimney, and odors with it.