The construction of the prestigious architectural complex, residence of the noble Ferretti family, seat of the Museum since 1958, dates back to the mid-16th century. The external façade is attributed to Antonio da San Gallo the Younger, a civil and military architect active in Ancona in the 40s of 1500.

The decoration of the noble floor, commissioned by Count Angelo di Girolamo Ferretti, was carried out by the mannerist painter Pellegrino Tibaldi and assisted shortly after 1560; elaborate wooden ceilings were created, the walls of the main hall and various rooms connected to it were frescoed: the hall of emblems, the hall of myths, the hall of the labyrinth. The 3rd floor has frescoed ceilings with grotesques, fantastic landscapes, monsters, mythological figures and masks in the late sixteenth-century Roman style (Federico Zuccari, Luzio Luzi?).

In 1759 the original building was renovated, probably on a project by Luigi Vanvitelli, with the enlargement of the facade, the construction of the balcony, the central portal, the grand staircase and the hanging terrace, with portico and loggias above. This extension was enriched by statues and busts probably from the Ancona workshop of Gioacchino Varlè (1731-1806). The very refined grotesque vaults of some of the lounges on the 3rd floor also date back to the 18th century.

The building has undergone several restoration interventions: in the years 1928-1931 on behalf of the Ferretti family who entrusted the work to the Milanese architect Tornaghi; in the early 1950s, following the aerial bombardments; finally in the 70s and 80s, after the 1972 Ancona earthquake.