Past, present and future inhabitants have been and always will be fascinated by the adjacent Norman Motte and Bailey, for which the council kindly provided a sign post opposite number 3 Blue Row in the 1990s! Generations of children born and brought up in Blue Row must have played on the banks, without much regard for the long history of the site. It was probably built in the 11th or 12th century - see Professor Martyn Jope's conclusions below. Nothing can be seen of the castle itself now, apart from the mound, but when it was built, it would have been a stone and earth base with a timber framed construction above.
From Professor Martin Jope, who excavated the site in the 1950s:
"Swerford Castle Hill is a small but extremely good example of a Motte and Bailey Castle such as we know to have been erected in England in the late eleventh and early twelfth centuries, as a result of Norman influence and conquest."
Some other useful information on timber castles of this period is found on this web site - follow the links on it to look at information on timber castles and minor castles.
The article below was written by E.M Jope after his excavations in the 1950's - further information and more photographs are in the Ashmolean Archives. Little research appears to have been done since Professor Jope's excavations!
Castle Hill, Swerford By E.M.Jope,
Swerford Castle Hill is a small but extremely good example of a Motte and Bailey Castle such as we know to have been erected in England in the late eleventh and early twelfth centuries, as a result of Norman influence and conquest. Scarcely any evolution of design is traceable in these castle plans in the present state of knowledge, and Swerford, with its conical motte and kidney-shaped bailey (not separated by any ditch) is of the type set up at Ewias Harold ,and Richard's Castle, Herefordshire, just before the conquest. Brinklow, of the same type, is mentioned in the Pipe roll of 1135, and one at South Mimms, Middlesex, was possibly constructed by Geoffrey de Mandeville in 1142. Although King John erected many mottes in Ireland as late as 1210, the construction of this type of castle probably ceased in England soon after the middle of the twelfth century, and we are justified in the inference that Swerford was built within the century after the Norman Conquest.
The earliest documentary reference to Swerford is in the Domesday Book, when "Robert"  holds 5 hides in SURFORD; land to 8 ploughs. Now in the demesne 8 ploughs and 3 bondmen, 7 villaines, 6 (bondmen??) with 6 ploughs. There is a mill of (6??) and 12 acres of meadow and 12 acres of pasture; it is worth 100's.
After this there is no reference to Swerford throughout the whole of the 12th century, save that of Alexander, "decanus" of Swerford witnessed a land transaction in Great Rollright, the next parish, between 1170 & 1180. In 1200 to 1201 the Curia Regis Rolls record a dispute over the rightful holding of Swerford, and, although they do not slate their findings on this point, in 1258 we find Reginald Fitzpeter, whose family were in some way concerned in the 1200--1201 Curia Regis proceedings, now Lord of the Manor of Swerford, claiming the right to compel the Oseney Abbey tenants in Hooknorton to grind corn at his mill. He held Swerford presumably under John de Plessets (a favourite of Henry III), Lord of Hooknorton, as he held it under Hugh, John's son, in 1279. John's great grandson, Hugh Plessets III, held in 13O2 HOKE NARTON cum Hamletto SWERFORDIA. Throughout the thirteenth century Hooknorton was held of the King by Barony and Swerford as a manor under it.
In the earliest cartulary of Oseney Abbey, in handwriting of c. 1195, is a copy of Robert d'Oilli II's original charter of foundation given in 1129, By this the Canons held Hooknorton (presumably land therein, not the Barony), but no mention is made of Swerford. There is, however, another forged charter of handwriting c. 122O, presumably used for Royal confirmations, in which "Maneria de Swerfordia" has crept in beside the Hooknorton entry. Oseney certainly held the advowson of Swerford, as in an Oseney document of 1221 - 1228 the presentation of Radulphus Dimmock is recorded but it is doubtful if "Maneria de Swerfordia" can refer to anything more than the land free of feudal rights to the Lord of the Manor. However for our immediate purpose the essential facts yielded by these documents are the almost complete absence of record of Swerford in the twelfth century, and that in 1200 it is not clear who is the rightful holder of the manor. It therefore seems that even in 1200 its history in the preceding century was not quite clear,
Nowhere, even in the ample thirteenth century record, is there any mention whatever of the Castle, so we must look elsewhere for an indication of its date and purpose. As we have seen from its design it could have been constructed at any time during the century following the conquest. The most probable time within these limits is the Anarchy of the reign of Stephen, when the chroniclers relate how the countryside was the oppressed victim of plundering from the numerous illicit private castles.10 Oxfordshire lay between the influence of Stephen and that of the Empress Mathilda in Wessex, and from 1145 to 1153 was subjected to the tyranny of William Chesney, whose castle lay at Deddington, and whose territory extended to the borders of Swerford at Great Rollright. He had received this property from the d'Oilli family, and if in addition he had during his period of power seized Swerford, this might account for the confusion over the rightful holder in 1200.
The position of the Castle next to the church suggests a manorial rather than a purely military purpose, and that their origin is fairly closely related, The dedication of the church to St Mary makes it almost certainly a twelfth century foundation, possibly later rather than early, and as Oseney Abbey church was also dedicated to St Mary this may indicate foundation by the Abbey, It is unlikely that Robert d'Oilli or his successors would build a castle at Swerford while they held the Barony of Hooknorton, in which it was included, as this was only a very small part of their property and they lived in Oxford Castle. However, during Stephen's reign some measure of protection of the manor lands became necessary, and possibly Swerford was chosen as the site for the castle protecting the d'Oilli property in that region. After the accession of Henry II in 1154 the church of S. Mary may have been founded to serve the dwellings which had grown up under the protection of the castle. Henry II gave orders for the slighting of private castles in 1155, and possible slighting is noted below.
Finally we have the evidence of trial excavations conducted in June, 1938. A trial trench across the top of the mound revealed below the modern turf line one occupation layer of blackish earth about six inches thick. This lay upon the surface of the piled local limestone of which the mound is built. The majority of the pottery (including all the rims) was found in this occupation layer amongst a few stones, not specifically laid, towards the southern side of the mound. As will be seen below, these sherds may be dated to about the middle of the twelfth century, which confirms the conclusions reached above.
Swerford Castle may therefore provisionally he considered as a product of the unrest of Stephen's reign, though not necessarily 'adulterine' in the sense that it was used for plundering the neighbouring country. This evidence is as yet very slender, but, such as it is, quite definite, and it is to be hoped that in the near future a more extensive excavation may be conducted, especially as the results from a small trial trench proved so fruitful.
Swerford Castle is constructed upon the gentle southern slope of the River Swere valley, in a position of no particular strategic significance. (Plate 5, No. 1.) The bailey stretches towards the upward slope northwards from the motte, which is in the most defensible position possible here. (Figs 1 & 2) The original entrance appears to be the gap now existing between the motte and the rampart to the east, just where one would expect to find it upon such a site. (Plate 4, No.2.) The bailey is defended on all sides by a rampart with a steep outer face, and on the north & east this has a deep ditch beyond it. (Plate 4, No 1). On the north this is cut into the hillside, but on the east it is bounded by a second rampart following right round to the southeast and covering the entrance. The curious outworks on the south west are inexplicable at present unless they are the remains of siege work. (Plate 5 No 2) This bailey rampart is constructed of earth with a layer of piled stone six feet thick on the outer face; presumably a space was deliberately left to pile up the stone that came out last from the cutting of the ditch. The motte is made of piled lumps of limestone and seems to have been destroyed somewhat on the north; perhaps this is deliberate slighting. (Plate 6)
 This is certainly Robert d'Oilli who held the adjacent manor of Hooknorton in D.B., and much other land on Oxfordshire: he built Oxford Castle c1070 and lived there.
 This can scarcely be Alexander de Swerford, the generally accredited compiler of the Red Book of the Exchequer. He died in Oct 1246, and as he sat as Baron of the exchequer throughout the preceding trinity term can scarcely have been over 70 at his death. His date of birth is usually suggested as 1176-7, and thus he would have been a mere infant in 1180.
 Curia Regis Rolls I: 176,376,420,429
 Hundred Rolls II, 726
 Feudal Aids IV 160. He was a minor, a ward of the King.
 Bodley MSS Ox Hist Soc (1929)
 The Dimmock family were land owners in Hook Norton
 Author thanks those who accompanied him on the excavation.