The concept of regeneration is one of the most unique elements of the Doctor Who television series and one of the reasons why it’s the longest-running science fiction series in the world. The concept was first introduced due to the deteriorating health of William Hartnell, the First Doctor, in 1963. The show was a massive success, and BBC didn’t want to end it, so they came up with the idea that the Doctor’s species had discovered the process that allowed them to rejuvenate their bodies, making every cell in their body healthy and new, allowing them to “re-emerge” as new people.
Admittedly, this concept granted each successive actor and writer the freedom to tackle the character in their own way, and as long as the Doctor isn’t cruel or cowardly, things should be golden by the showrunners’ standards. However, for a concept that’s so crucial to the show, it seems that the Doctor’s ability to regenerate isn’t entirely explored in greater detail, the reasons for which The Timeless Child storyline later confirmed. The Doctor apparently doesn’t regenerate as the rest of the Time Lords do. In fact, we’ve seen other Time Lords regenerate, and while most of them do undergo certain personality changes, none are as extensive as the Doctor’s.
Fandom has argued for decades that this happens because the Doctor isn’t regenerating properly since he never got the hang of regeneration. But then again, it’s entirely possible that the Doctor just likes the random results. Ultimately, either and neither one could be true. The concept of regeneration, at least in the Doctor’s case, seems intentionally shrouded in mystery — and for a good reason, as it could lay the fertile soil for future retcons. Let’s remember that the First Doctor isn’t actually the first Doctor, just the incarnation we were introduced to back in 1963. Who knows how many lives and regenerations the Doctor underwent before the Division wiped his memory? He’s a Timeless Child, so the answer might forever remain unlimited.
Regardless, we know that the process of a Time Lord’s regeneration only seems random, but it can be controlled or at least influenced by outside factors. This was first shown in The Night of the Doctor, where a dying Eighth Doctor was presented with several potions that would determine his next incarnation’s personality traits. The Eighth ultimately chose to become a warrior, which triggered his regeneration into the Ninth Doctor — the War Doctor. Interestingly enough, regenerations are usually triggered by old age or fatal injury, but the Second Doctor was forcibly regenerated into the Third and sent to Earth. In addition, the Time Lady Romana, from Destiny of the Daleks, has shown the ability to trigger regenerations intentionally, leading the fandom to believe that some extent of control over the regeneration’s outcome does exist.
Another interesting theory suggests that every new incarnation brings in different traits to respond to the previous incarnation’s subconsciousness and emotional stress. This was a long-standing theory that was ultimately canonically proven when the Twelfth Doctor recognized his current face as that of the man the Tenth Doctor had previously saved. This opened up another interesting opportunity for double casting within the show. A perfect example that encapsulates the entire theory and confirms it is the fact that the Fifth Doctor chose the face of Commander Maxil for his Sixth incarnation. As a result, the Sixth Doctor was more likely to employ subterfuge to outwit his foes, considering that Maxil was a humorless and treacherous man.
In fact, the Third Doctor regenerated into a Bohemian Fourth Doctor after visiting Metebelis Three, and the Fourth Doctor died thinking of his former companions, regenerating into a kind-hearted and paternal Fifth Doctor, and then into the cunning Sixth Doctor. Something similar happened to the Tenth Doctor, who died because of his deep emotional attachment to his companions, resulting in a more distant and off-putting Eleventh Doctor.
Now, for the counter-question — how do other Time Lords in the series retain all of their personal traits, and the Doctor doesn’t? Unfortunately, this assumption simply isn’t correct. Our primary source of reference regarding regenerations and changes they induce has always been the Doctor, and his arch-nemesis, the Master. Of course, there are other Time Lords in the Who universe, but these two are among the rare ones whose regenerations we had the privilege to witness on a semi-consistent basis. The misconception regarding personality shifts happened because the changes underwent by the Doctor were very prominent, while the Master remained the same old villain to a lesser or more tinged degree in every incarnation.
However, even this is not entirely standard and immutable. The Master has also evolved through the series and New Who gave the character even more background, depth, and complexity. For example, with the introduction of Missy, the first female incarnation of the Master portrayed by the talented Michelle Gomez, we got to see a very different incarnation of the Master. Missy was more morally ambiguous than most of her predecessors and even underwent a unique character arc, in which she acted as the Doctor’s friend on the road to redemption. This ultimately proved that all Time Lords undergo certain personality shifts during their regenerations, mostly influenced by the intellectual or emotional stage of the “transitioning” incarnation.
So, to answer the question, the Doctor’s personality shifts are likely the normal biological effects and after-effects of the regenerative process, something akin but ultimately a unique evolutionary trigger found in the Time Lords but not wholly dissimilar to the maturation process of other species. But then again, the Doctor isn’t even necessarily native to his own “who-verse”. He’s a Timeless Child from a universe outside ours, who traversed space and time through a wormhole, wearing the face and shouldering countless incarnations that we’ll probably never be able to count.
