A classic Norman mott and bailey castle, built by Bishop Gundulph in approximately 1088 to defend the Medway Estuary from invaders. The keep is the tallest in England (113 feet, or 34.4m), and no longer has any floors, so visitors walk around specially constructed walkways around the inside of the walls.
As there is a 100 foot drop from the upper walkway to the bottom of the keep, this is not a trip for Granny or for anyone who doesn't like heights.
Even if you don't fancy the idea of going to the top of the castle, you can still wander around the gardens, shown in this photo. It's free to get in (there's a small charge to enter the castle), and there's a hot-dog/ice-cream shop.
Three of the castle's towers are square, and one is round. The round tower replaced the original square one, which was demolished by King John when he layed siege to the castle in 1215. His army tunnelled under the tower and set light to "the fat of forty pigs", which weakened the foundations and caused the original tower to collapse. It was replaced in 1226 with a round tower, as these are stronger than square ones.
In the moat there is a cemetery, which was where Charles Dickens apparently wanted to be buried. In this photo, taken from just outside the cathedral, you can see the gravestones. Apparently as soon as Dickens died his body was taken to Westminster Abbey for burial, despite his wishes. A grave was being prepared for him in Rochester when it was revealed that he'd been moved.
On a more contemporary (well, 1980's) note, the castle is featured on the front of the Clannad album "Legend":