I, like many others, like to use AI to assess my financial investments from time to time. Recently, when I was doing exactly that, the response had a surprising ending. It basically said that I was investing in the same companies at the end of the day under different mutual fund names.
After reading that, I went down a rabbit hole and started figuring out how diversified I actually was, and to my surprise, the funds moved as if they held the same few companies.
This is a correlation heatmap I made for some well-known mutual funds, 44 to be exact. A shade of dark blue indicates that a fund moves similarly to the fund in comparison, while a shade of light blue means that they have a lower correlation and do not move in a trend together, while the lightest shade would indicate that the funds move in opposite directions.
Notice how a lot of the heatmap is relatively a shade of dark blue, and that is exactly why you may be under the illusion that you are diversified.
On the other hand, some funds like the Debt or International funds seem to have a lighter shade, and in my analysis I clubbed 3, 5, 7 and 10 equity funds to see how much diversification was provided with each addition of an equity fund, as it may be evident by now – not much.
When we simply add 1 debt fund to an existing equity portfolio, the effective number of bets we take, i.e. diversification, increases significantly – way more than adding an equity fund would result in.
This chart shows exactly that: our effective number of bets went from 1.32 to 1.39 when we added 1 equity fund to an existing 3-fund portfolio, a difference of 0.07, but simply adding one liquid debt fund took the effective number of bets to 1.97, a difference of 0.65, about a 9-fold increase in the difference of effective number of bets.
A correlation of 0 would provide a 29% reduction in risk according to the formula, while a typical pair in the data had a positive correlation of 0.89, moving almost identically and giving only a 3% reduction in risk as compared to holding the same fund instead of diversifying.
Without getting into the mathematics behind it, if we use Principal Component Analysis (PCA), we can find a pattern of sorts. This pattern is just a row of numbers, like a mood: when the numbers go up, the funds also tend to go up. The factor we found has a positive correlation of 0.95 with the Nifty 50, so we can assume for now that the factor closely follows the Nifty 50.
This one factor alone drives 85% of the funds' movement. If all 44 funds were completely independent, each factor would explain an equal share of the drive, i.e. 100/44 ≈ 2.3%.
Basically, most of the equity funds move in a highly similar fashion to the Nifty 50.
When the same analysis was done for longer durations – up to 128 months – the result was almost the same.
§01How I Built This
An interesting thing I noticed while collecting the data is how most of it was mainly available from 2013. It turns out SEBI had made it mandatory for every fund house to offer a separate direct plan for every mutual fund they offered, from 1 January 2013.
Every chart in this issue is built on real data, and the code is public.
Fund data: Daily NAV histories for 44 mutual funds across 13 categories, pulled from mfapi.in. I used only Direct plans, Growth option, and converted month-end NAVs into monthly returns.
Method: Correlations between every pair of funds, then Principal Component Analysis to find the shared movement (the "one factor"), then the effective number of bets, a measure of how evenly that movement is spread across factors. Portfolio charts use random combinations of funds: 600 per size for the funds-vs-bets chart and 1,500 for the debt-fund chart. Everything is in Python.
The window: The main results use the 22 months (Nov 2024 – Aug 2026) that all 44 funds have in common. I repeated the analysis on longer histories, up to 128 months, in the robustness chart.
What's excluded, on purpose:
- Holdings. I measured how funds move, not what they own, so I haven't measured overlap between portfolios.
- Short window. Longer histories gave slightly more bets (2.3–2.6 vs 2.04), so the headline "about 2" is if anything on the low side, and may slightly exaggerate the problem.
- Equal weighting. Every fund counts the same. Real portfolios have unequal weights, which changes the numbers.
- The liquid fund. Its correlations are noisy because its returns barely move, so treat the debt-fund result as a direction, not a precise figure.
- Data fixes. HDFC Liquid Fund's return for Aug 2015 was distorted by a face-value change, so it was dropped and that month was filled with the average of the months either side. This doesn't affect any result in this issue, and it's documented in the repo.
- The Nifty check. It uses the UTI Nifty 50 Index Fund (Direct plan) as a stand-in for the Nifty 50. That fund is one of the 44, but rebuilding the factor from the other 43 gives almost the same result (0.94).
Fund names appear as examples of data, not recommendations.
This newsletter is for educational purposes and is not investment advice. Past returns are not indicative of future results.
Code: github.com/shanzxt/n2-Diversification, covering the data pull, cleaning and the maths.
Try it yourself: the fund-picker tool on my website lets you enter your own funds and see your real bets.
Try it yourself
live · runs in your browserEducational, not investment advice. Fund names are examples from the dataset, not recommendations.
If you spot a bug in the maths, tell me. That's the whole point of shipping the code.