The original core of the castle consists of a quadrangular trapezoidal fortress with corner and intermediate towers. It was built at the behest of Roger the Norman around 1131 upon pre-existing Byzantine dwellings. It was most probably erected to defend, but, above all, control the city of Bari.
In 1156, the castle was severely damaged by William the Wicked. Some time later, however, Frederick II of Swabia (1194-1250) had the castle’s surviving fortress walls and internal layout restored (circa 1233-1240). As Frederick II wanted to highlight the castle’s residential and representative appearance, he had the entrance hall and courtyard embellished with foliate capitals (bearing the signatures of Minerrus de Canusia and Melis de Stelliano), a portico erected, and the austere aspect of the towers softened with ornate windows and fretwork oculi. Later on, between the fourth and fifth decades of the 13th century, he had a decorative and symbolic portal built on the west side of the inner core (near the Tower of Minors).
Charles I of Anjou (1226-1285), on the other hand, commissioned master builders Pierre d'Agincourt and Jean de Toul to carry out major refurbishment on the castle that included the construction of a second castle entrance on the north side, overlooking the sea.
King Ferdinand of Aragon attributed the Duchy of Bari to the Sforza family, and donated the castle to them. During the first half of the 16th century, Isabel of Aragon (1470-1524) and her daughter Bona Sforza (1493-1557) had other features added to the castle; namely, the outer sloping rampart walls, and the imperial Renaissance staircase in the inner courtyard. They also had the fortress wall on the north side reinforced, incorporating the former portico (today, the "Swabian Hall").
The two duchesses were firm and highly educated, but, most importantly, they were well-liked by the citizens of Bari. They managed to transform the castle into a prestigious princely residence where scholars and influential men came together. Bona Sforza died in the castle in 1557 after a long absence during which she had followed her husband Sigismund I King of Poland, of the Jagiellon dynasty. Bona Sforza had strong emotional ties to Poland, and, for this reason, the Queen dedicated the chapel overlooking the eastern front of the courtyard, formerly dedicated to St. Francis, to St. Stanislaus. Bona Sforza’s daughter Anna dedicated a funeral monument to her that is located in the apse of St. Nicholas’s Basilica. In the 19th century, the castle was first used as a prison for minors (hence, re-named the "Tower of Minors") and, later, as barracks for the infantry and gendarmerie.