Caves can be constructed from many things, rocks, flower pots, PVC pipe and jars are good choices. Many times people will collect various rocks for use in their aquariums. Be sure to treat and test any of these before using them in your aquarium.
Treating and testing rocks:
Wash and scrub rocks very well.
Pour some white vinegar on the rocks, if they bubble, they are not fit for aquarium use.
Fill a bucket with water that has 1-2 cups of household bleach added. Soak your rocks for 24 hours in this solution, drain well, rinse, fill the bucket up with clean water and soak rocks for an addition 48 hours, changing the water every 12 hours to ensure all the bleach is gone. Rinse well, they should now be ready for use.
Fill a large pot with fresh water, boil rocks for about 6-8 hours, rinse, allow rocks to cool before using them.
Aquarium silicone, this is available at your local fish store or hardware store. You must ensure that it is aquarium safe silicone!
Various rocks (stay away from rocks with sharp edges for safety)
Flower pots and pickle jars, cleaned very well.
Caves out of jars and flower pots
Silicone various rocks to the outside surface of the jar or pot for some interesting, natural looking caves. Cover the entire outer surface, even add some to the mouth of the flower pot or jar. These look neat when they blend in with your aquarium decor.
Silicone larger rocks into a rock formation. Your imagination is the limit. Avoid the use of sharp rocks as your fish may be injured.
You can make all of these projects as simple or as sophisticated as you'd like. This is actually something fun to do, and it's much cheaper than purchasing your decorations.