The woman mentioned in the rhyme is Lady Katherine Banbury. She was the wife of Lord Jonathan Banbury. Among Lady Banbury's jewels were many beautiful rings and the bells mentioned were commonly used to trim the edges of a lady's velvet saddle cloth.
Lyrics for 'Ride a Cock Horse to Banbury Cross'
Ride a cock-horse*
To Banbury Cross,
To see a fine lady
Upon a white horse,
Rings on her fingers,
And bells on her toes,
She shall have music
Wherever she goes.
*A cock-horse is something used as a toy horse, like a rocking horse or a parent's knee.