Standing on a steep crag, Garth Castle is a plain 14th century keep, square in plan with a flush parapet crowning two walls.
The entrance at ground level has a strong iron yett. The stair climbs within the thickness of the walls up to parapet level. The basement is vaulted and lit with slits. The hall on the first floor was also vaulted at one time, as was the storey above. The castle was built by Alexander Stewart, the Wolf of Badenoch, Lord of Badenoch and Earl of Buchan, 4th son of Robert II. He torched Elgin Cathedral and town, as well as Forres, after being excommunicated by the Bishop of Elgin for forcing marriage upon his wife, the Countess of Buchan, then deserting her. He died here in 1396, and is buried in Dunkeld Cathedral.
In 1502 Nigel Stewart of Garth attacked nearby Weem Castle, burned it, and took Sir Robert Menzes prisoner. Stewart put Menzes in the valued dungeon at Garth, threatening to have him killed unless he signed away some of his lands. Stewart was nearly executed for the crime, only the intervention of the Earl of Atholl saving him. Stewart was later suspected of murdering his wife, Mariota, and was imprisoned in Garth until his death in 1554.
The property then passed to the Stewarts of Drumcharry. The castle was abandoned about the middle of the 18th century, and became ruinous. It was partly restored in 1880, and again in the 1960's. It is occupied.