Call of Cthulhu, Dead Reign, Beyond the Supernatural, Dread, Nightbane, Vampire the Masquerade, Eclipse Phase, Paranoia. These are but a few of the many horror based rpg's that we can find at our local game shop. I have had the privilege of playing a few of these and know the fun that can be had in playing them. On that subject I would like to discuss the dynamic of a horror game what I feel works well and what doesn't.
First I have had the pleasure of gaming a CoC, Call of Cthulhu, with a great storyteller who literally made me lose sleep over the game. He set the mood by having us climb into a old Physics lab, turn out the lights, and play by a small desk lamp. This was so eerie that another player asked me to escort him home as he was too scared to travel alone. To create tension when we were separated he had us play one on one with him while the other PC's had to stand in a dimly lit hallway alone. Just thinking to the game makes me uneasy. Jonathan Schrack probably will still go down in the books as one of the best horror GM's I have ever had the pleasure of playing under.
Second I have had the unfortunate displeasure of being in another game where to despite the DM trying his hardest the players were responsible for the failure of the setting. He would bring creepy storytelling and a robust setting designed to make us paranoid, however it would not work as two of the players treated the game as more of a powerhouse high fantasy rather than what the setting is.
SO in conclusion I am of two theories. Firstly Good storytelling can set a mood for horror and make you squirm. Tension is key. Second if a player does not come into the game with the mentality that it is indeed horror then it will be hard to change them. A good example is someone who walks through a haunted maze laughing at the employees. So what can we do to get the ultimate scenario. I think it takes a good mix of storytelling, mood, and players who come into the game with a horror mentality.
"Thanks for not killing me. Oh wait!"- Daniel Henry
"I hate you"- Derek Russell
"Go look at your stupid Catgirls" - Chase Jones