Hidden Knowledge helped estimate District GDP!


There is perhaps infinite knowledge within each of us.  Some, we are fully aware of and some we are not, the latter is sometimes also referred to as tacit knowledge.  Tacit knowledge comes from many sources – experience of the past, conversations with friends and family over the years or simply from osmosis where we sometimes unknowingly absorb information from the infinite day-to-day occurrences around us. I first read Michael Polanyi’s 1966 work on Tacit Knowledge long time back; he is also supposed to have coined the term, not to mention his famous quote – “We know more than we can tell”!

Tacit knowledge is typically unstructured, it resides in bits and pieces and may even be completely randomly structured, filed by our mind in a way that we will typically not even realize we have it.  Try out this experiment – ask your grandmother how to make a samosa.  Now follow her recipe very precisely, your samosa will most definitely taste horrible.  Because samosa making technology, as simple as it may seem, has a very high tacit element – how much to heat the oil, how moist should the flour be, how long to fry, how much to boil the potato, whether to cover the pan, and so forth.  When an accomplished cook makes the samosa, the right kind of knowledge is unconsciously accessed by our minds.  Now repeat this samosa making, and every time you redo it, it should get better.

The concept of learning by doing is the other side of the tacit knowledge.  If you or your team-members do not have the required knowledge, you will need to resort to learning by doing.  But learning by doing is extremely costly method of knowledge acquisition.  It can take many tries to get the samosa right, and it can take many tries to get your product right.  These tries will eat up time, money and energy from your whole team and also you.  And the start-up is no place to indulge in expensive and time consuming asset building.  In other words, once you have identified an area, having the tacit knowledge is a valuable asset – and so most start-ups’ products or service have some connect with the past avatar of the entrepreneur, or his key employees.  The paragraphs below replay that experience.

Estimating District GDP

Any good data exercise always needs some method of validation.  There are broadly three different forms of validations, that of the (a) process (b) of consistency and (c) from external sources.  Most decent analytics professionals validate their processes, in that mistakes or errors do not creep in due to badly structured algorithms (though it is surprising how many do not).  Some look into internal consistency of the data, by correlating or comparing various data estimates. But few validate using outside or 3rd party data.  The reason typically is that such data are not really available easily or cheaply enough.

When we estimated household incomes for each district of India we found that we could only validate the process and consistency, but not using external validation.  That’s when the hidden knowledge first showed up – having GDP estimates at the district level would be a great thing to have we thought.  Not only would the resultant validations be of very good quality, having access to GDP data could enable us to do many more things with the data.

To the uninitiated, the link between incomes and GDP is so obvious, that many people overlook it.  At the time we first came up with our estimates we realized that the incomes of every other entity were not adding up, and were typically much lower than the governments GDP estimates.  This included among the topmost think tanks, large global consultancies, market research databases – somehow no one it seemed had cared to check household income estimates against GDP estimates.  But to the economist in me, it so obvious that anything else was pointless!

This was deep domain knowledge working.  And I also knew that a couple of states had attempted to estimate district level GDP.  Meanwhile a team member fished out from the net a document on how to estimate district level GDP.  This was put together by a government entity about decade before, but they forgot about it!  It seems those people must have retired, and the system forgot this from its active memory.  Anyhow, we decided that there was no way we would be able to build a good steady product with stable estimates without a good GDP series underlying our data, and this would also ensure that our data are forever in sync with India’s National Income estimates.  This practice lies at the heart of all our data and continues till today.

But no such district GDP existed.  And so the maverick’s instinct took over and decided to estimate it in-house! The stated and practical reason – we need something to validate our household income estimates; the instinctive reason – it was a ‘not-done-yet’ challenge!  When we started to work on it, we realized that the government’s document on GDP was flawed in parts, and we needed to improve upon it.  But this was learning by doing in operation, and it took us far longer than it should have had we had someone from the governments statistical office advising us.  But this also allowed us to innovate.  We took a production function approach instead of an additive one, and found our estimates were looking very good.

How did we do it?  Everywhere two elements were occurring repeatedly – first was the maverick, do what is has not been done before and irrespective of difficulty.  The second was the economist’s instinctive knowledge, if you have a hold over GDP data and its components, you can do many different kinds of things with that data.  The hedge fund experience with large databases had also brought in its own tacit element.  And so despite one failure after another and many months over-budget, we continued doing what to my knowledge at-least no private sector company does anywhere else in the world – estimate GDP data!

We first came up with a series that contained GDP estimates for each district of India, and also for 11 sectors for each district – agriculture, manufacturing, mining, etc.  This data helped us have a ready insight into the economy of all districts of country.  Only we had such data, and this would later help us validate our household income estimates better than anyone else.  When we shared this with some of our clients we were quite surprised at the excitement it generated – we were doing this for in-house needs!  But soon we had a new stream of revenues.

More gratifying was to see the erstwhile Planning Commission, RBI and even the GOIs Ministry of Finance buying this completely unintentional product.  And that established a differentiator so powerful, a completely new space was created, and no one could break into our monopoly for more than a decade.  Not simply that, our brand simply skyrocketed in our space.

Yes one could claim great foresight and vision, but it was nothing of that sort!  It did not happen by accident either, so serendipity can also not be given credit! It was simply instincts and the DNA of an organization defined by tacit knowledge picked up over the years.  I had worked for a hedge fund and understood how to work with big data even before the term was coined.  I then had spent days and nights working at NCAER and understood the flaws and strengths of Indian data like few others.  And then there was the need for excitement for the maverick in me.  Things just came together, not by luck, nor by vision, given my background and instincts they would have under any circumstances.

The entrepreneurs’ exposure, whether through experience or background, can provide him with a deep critical edge that comes from the tacit dimension of knowledge, but the entrepreneur’s business has to be in a related space for him to benefit from it.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s