First off, I want to straighten things out here. Btw, I am a pretty experienced psychonaut (not practicing anymore though) and i am extremely well read on the subject.
LSD is PHYSICALLY one of (if not THE) most researched and safest drug known to man.
Mushrooms (as long as you don't eat a poisonous one) and Cactus (mescaline) are physically pretty safe as well.
As someone stated earlier, things to stay away from would be:
Datura, and pharmaceuticals (dramamine, benedryl). YOU WILL NOT ENJOY IT!
Datura actually turns you schidzophrenic for about 40 hours, and could kill you. If you somehow make it out alive, You probably won't remember anything anyway. Dramamine and benedryl trips are probably the scariest things imaginable. What happens is you essentially hallucinate your worst fears. Imagine a ten foot spider in your room, or snakes crawling all over you. And it seems REAL. With most psychadelics, you have some idea you're tripping, not with these puppies. STAY AWAY! NO ONE EVER HAS A GOOD TIME!
Now back on to topic:
You should not be worried about the physical effects of psychadelics, because most are pretty harmless. It's the PSYCHOLOGICAL effect you need to be concerned about. If you've tripped or talked to someone who has, you know that tripping is quite literally off the freakin' wall. It's like nothing you've ever experienced before or can even imagine. Natural laws can be violated, your world can cease to exist. Things you have known to be true your whole life change. For example, you KNOW that your chair is not going to turn into a talking dog. You KNOW that the walls remain steady and things don't change color. You KNOW that the environment you're in is constant, that you're not suddenly going to find yourself in a desert, talking to God. For alot of people, seeing your world melt away (literally and figuratively), can be VERY disturbing.
For some people, trips can be amazing experiences. For some, it can lead to some serious psychological damage. If you want to trip, check out these sites:
shroomery.org
erowid.org
Read the trip reports so you can know what to expect.
I'm not condoning or looking down on tripping, but you should know what you're getting into, and be sure you wan't to get into it; because most trips last about 6 hours for mushrooms and 8 hours and beyond for everything else (some people have tripped for DAYS on cactus. LSD is usally 12 hours). So it's not something that you can just turn off when you decide you've had enough.
Also, the reason sleep deprivation makes you hallucinate is because generally after 72 hours of waking, the brain starts to produce DMT, which is the strongest naturally occuring psychadelic. So in that aspect, you are still using a drug, it's just one that your brain makes on its own in certain situations.
Interesting side note: Most humans get a big (big enough to produce hallucinations) DMT dose twice in their lives: at birth and at death. This helps to explain near-death victims who "go to heaven and talk to God." Really, they were just trippin' Quite literally lol.