What is Big Data? It is Matrix Reloaded

What is Big Data?
Is it really about technology where large data sets are being analysed by computers? Not exactly. It is the culmination of all the Organizational and behavioral theories being approached by Maths.

See one of the objectives of marketing has been The Issue of Choice, more like an issue of someone being able to control your choice without you realizing it, leading to you parting with your cash. The marketeer wants to modify your behaviour. He wants you to like the brand, leading to a particular action which in some simple steps leads to profit. Traditionally this is known as AIDA model.

This requires a certain amount predictability on the marketeers side to gauge your response to stimulus as a collective . Which means the message A will Lead to action X and message B will lead to action Y. The variable here is CHOICE. The consumers inalienable right to choose.

Since 1920's this has led to a lot of research in psychology to find the inherent reasons behind choice.  An attempt to understand the Black box and reach answers like, emotional responses supercede logical responses. Most of the creative people are still trying to follow and decipher this Black Box called Mind. Some things which will be universally liked and responded to. And the learnings now go into advertisements, movies, speech writing, Politics, product design and all the creative processes, where one product has to suite many people.

In such a scenario, the goal is maximization of acceptance. This means that it is understood that not everyone will be influenced, but as long as a profitable majority of the population is affected by the creative product, its viable. Therefore a Hit movie, a moving speech, a good advertisement, an iPhone are products which were successful to get a favorable response to the message in larger percentage than others.

Some science has crept into this process by focus groups, with a small population set to put some data behind creativity. People realized that one person saying "yes" is choice, many people saying "yes", is a market.This also led to the development of customer demographics and targeting.

 As population grew and products became specialized, it became important to market it only to a certain set of people. With growing population, it became possible to cater to a small set of people and still achieve the economy of scale to break into profit.

Science and creativity really mixed with the coming of computers. It is virtually a mind, but that runs on materials science & mathematics. Computers are actually machines that mimic the mind (for now) and as the mimicry becomes complex and accurate, it sometimes becomes difficult to separate the original and the fake. You might say that Brain also is a computer but a biological one, and i wouldn't dispute your argument, but would just say that the process of learning is different.

So as i was saying, as science and maths entered the creative realm, the natural progressive question was, How many demographics actually exist in a population? Are we missing any? Is there a Gap? and someone came back with the answer x^n. If population is X and the number of dimensions being looked into is n, then there are X*X*X*....X(n times) demographics. But that is very vast. Also some of these demographics and psychographics might sit too close together and might not be useful at all. So what is a good answer. Then it dawned. The best estimate of demographics in a population is

Number of Demographics in Population = Population itself = each individual is one demographic

We called it Personalization. When a product or service is made specifically for you as per your own choice. A product that reflects your individual demographic and psychographic. A product as similar, or as Unique as you are in this world.

Something that was being done in services since ages. When you got a haircut, or ordered a food, it was as you wanted (theoretically). It is possible that two people might ask the same order, but theoretically you could just have 3 strands of hair removed and have the most unique haircut in the world. 

But that doesn't happen in real life, people's choices can often be grouped together. When you look at all the individual choices that people make in a barbershop, you can see that they will fall under a particular bell shaped curve. Some choices really rare, and others made more often and similar. It is a result of sharing this world and the experiences that come with it.

Till now, these individual choices could only be studied using grouping, which simplified things and works on maximization rather than accuracy. This meant that we could only gather data and analyze it in the broader sense on a population or group level. But as computing is becoming cheaper, keeping a track of these choices and making a truly personalized product is increasingly becoming possible. And this chunk of data, a set of all those choices you made, is BIG DATA.

There is an issue with Freedom to Choose. We think that we make unique choices. But thats not true. Because to choose, you can only choose freely, when you have infinite things to choose from. If you are choosing from a finite set, the number of choices have already been limited to a finite set. You are actually choosing freely from inside the finite set. Ever had to make a choice between the Devil and the Deep blue sea? Will that choice be called a free choice?

So if the set of choices you make in an instance is finite, we can always assign a probability of that choice happening. When you stack a set of choices in sequence, we can still calculate the probability. However, it becomes a little difficult to do so. But if you have cheap computing, its mighty easy. Because its too difficult for an ordinary mind to calculate, it gives us an illusion of freedom. But in actual its is pretty much pre-determined.

In my opinion, If you have a large enough data set for a system to learn from, and enough time and computing power, you can actually predict an entire person's or human race's future. (theoretically, and i know it may sound absurd, but its true)

In the above scene with architect, this is what the architect shows in his multiple screens. After every question, you see all these screens, with all the Neo's, with all the possible things that Neo can say, and then Neo says one thing.At no point, Neo can say anything, thats not already thought out or expected. There is never a blank screen.

This sequence of choices is what makes us feel that our life is unique. it gives us an illusionary sense of purpose and meaning. But if you really had a free choice, you would have kept trying to burn your hand again and never modify your choice based on previous outcome. You would have exercised your freedom. You would give up smoking the next cigarette once you realized that this choice was not good.

When we change choices based on previous experience, we call it learning.

When we keep on making bad choices, and not learn from experience, its addiction.

Ever seen a Mad man. He does something. And we can be pretty sure that it must be making sense to him, and we can be pretty sure that he is exercising his freedom to choose as well. Then whats wrong there? His set of actions are not following the rules of addiction or learning or predictability.

If BIG DATA keeps on expanding and becoming all pervasive, we might stop thinking at one point. What happens when a system becomes really good at predicting your choices. And what happens when it keeps on giving you a set to choose from, which is exactly the kind of set you wanted to choose from. After sometime, you will learn, that if the system is telling you it works.

In sometime, all you need to do is say, i need blue shirt. And the system will know what kind of shirt, what shade of blue, what size. All pretty much decided for you. And you will love the first 500 times its right, and then you will accept it as right.

Next, all you will do is ask for a blue shirt, and buy what the system predicts. But then, how will you try out something new. Something that is truly unique or different. 

How will you change your choices, when all that is being offered is a set of products based on your previous choices?

Economy is not growing but moving

The economy is moving, its shifting. Not moving as motion, but moving as in its being taken piecemeal and replicated in other parts of the world. We call it growth.

Think of All the things that make developed countries developed, as colorful blue, green, yellow, Lego blocks. And now imagine a country like India with brown dusty Lego blocks. Now take off a brown Lego block from India, and replace it with a colorful Lego block from US or Japan or any other developed country.

Its important because it means, every smallest piece of lego block will eventually follow others and move.

That is today's economy.
For the longest time, i used to think that the Indian entrepreneur was a specialist in copying Ideas from the west. But today i had an epiphany, and i realize that they are not being copied. I think this is what is happening, and will continue to happen.

Each human being has needs. These needs are fulfilled by products. Then a product (say mobiles) comes along which is a paradigm shift in all the products. It fulfills a need, which cannot taken care of by the current products in the economy, and then that product needs to be either imported, or either copied as is.

When you look at competing economies, they are like a net. Each product and service exists in the economy because it fulfills a need, a gap, in the system. All the products and services are therefore interdependent. You can't take away a hand without a body attached to it. Since it happens over time, we don't see it as an immediate cause an effect.

Lets expand the idea further using the same example of Mobiles  as above.

So you got computers. But then more Mobile companies follow, because if one company is making profits here, other companies need that too. Above all these companies can't let another company play unfair by simply expanding the geographical area in which they were playing. If company A and B were competing within US market, both of them were focused on customers and creating efficient processes. Now that company B adds India as a market, company A needs to join India too, to be not at a disadvantage. So end result, Company A & B are in in India now.

Now what has started is a Lego piece of US economy landed in India. And this piece needs other pieces to be efficient and be in natural environment, just like its DNA. Some of the needs are immediately fulfilled by local Lego pieces, but there's just too much difference.
The company C meanwhile which was catering to company B, now decides that it already has a ready customer in India, and it opens up an India office..and then so does D & E & F...all the way to the end of the alphabet.

The only way to stop this migration, is to copy the Lego piece as soon as the company B lands here. So that there is no place for company C to follow. More like a prosthetic arm, so that you don't need the original arm.
But even then, from a customers perspective, the need for mobile phone creates other needs which needs to be fulfilled. Higher data speeds, local repair shops, and various other things.

These are the same needs which rose in US, and then companies were formed to fill the gap. Similarly, companies here will rise with the same DNA, fulfilling the same needs that came up in US. If that doesn't happen, some one will come from US to fulfill the needs.

This means that there are two kinds of companies.
One that is copying the Lego bricks locally creating competition and others that are Moving the Lego bricks.

Two strategies. Competition or Expansion.
This will keep on happening till the world is homogeneous.

Once this is understood, its very easy to see source for new business ideas.
Either blindly copy the next step of evolution in the exact same sequence as it happened in US.
Or either compete with the latest  sequence  which has just landed in the local market and compete.

First is a little more risky, the second a little less risky. The choice depends on your risk appetite, means the funding available to you.

On Karma vs Free Will

One of the philosophical confusions that people face is
"If bad things are happening to me because of past bad karma i accumulated, then whoever did bad things to me doesn't seem to be exercising free will, and is bound by the law of karma itself"

In short my bad karma made another person do bad things, and therefore, there is no free will.

The argument looks viable, and the gentleman here  has very nicely (bazinga) attributed it to following

" If you do bad things, bad things will happen to you. This is a very simplistic way of understanding karma. This is what we tell children. But actually there are a lot of things along with karma influencing your decision...blah..blah..blah"

Essentially, if you can't solve a philosophical problem, then just go ahead and change the construct to suite your needs. Perhaps, the gentleman is learned and wants to stick to vedanta stream of thought and there is no good solution inside the thought stream.

I think this is the solution. But let me build some context by quoting ram charit manas, kishkindha kaand from here

धर्म हेतु अवतरेहु गोसाईं। मारेहु मोहि ब्याध की नाईं॥

मैं बैरी सुग्रीव पिआरा। अवगुन कवन नाथ मोहि मारा॥3॥

भावार्थ:-हे गोसाईं। आपने धर्म की रक्षा के लिए अवतार लिया है और मुझे व्याध की तरह (छिपकर) मारा? मैं बैरी और सुग्रीव प्यारा? हे नाथ! किस दोष से आपने मुझे मारा?॥3॥

English: Oh Lord, you took birth to protect Dharma and still you chose to ignore the laws of war and killed me, hidden like a hunter? What was my fault? How is it that i became your enemy and sugreev became your dear.

* अनुज बधू भगिनी सुत नारी। सुनु सठ कन्या सम ए चारी॥

इन्हहि कुदृष्टि बिलोकइ जोई। ताहि बधें कछु पाप न होई॥4॥

भावार्थ:-(श्री रामजी ने कहा-) हे मूर्ख! सुन, छोटे भाई की स्त्री, बहिन, पुत्र की स्त्री और कन्या- ये चारों समान हैं। इनको जो कोई बुरी दृष्टि से देखता है, उसे मारने में कुछ भी पाप नहीं होता॥4॥

English: (Lord says) Your younger brother's wife, your sister, your son's wife and your daughter are all equals. And killing anyone who casts an evil eye towards them, is not a sin.

The point being, an action that is bad for you, doesn't define the action. The Karma of the action is defined from the viewpoint of the "Karta" or the doer. So Something might be bad for you and be the result of your accumulated bad karma, but such a thing might actually be accumulating good karma for the other person, since punishing a sinner, is a good deed.

If someone steals a bag belonging to you, and you catch him. His act of stealing was bad. Your act of catching him was good as well as your act of catching him was bad for him.

Whether you catch the person or let go of him, is your free will. Whether you punish the sinner or not is your free will. The sinner chooses to commit a sin is his free will.

In essence there is no conflict between the law of karma and the free will. Free will is the act of choosing an action. Karma is the result of such an action.

The only time Karma will stop working out is when no one in the world is capable of making a good choice or executing a good action. Which means no one is accumulating good karma, the clock needs to be reset, and the Lord himself has to appear to kill off the sinners.

जब जब होहिं धर्म की हानि।

बाढ़हिं असुर महा अभिमानी।।

तब तब प्रभु धरि मनुज शरीरा।

हरहिं शोक मम सज्जन पीरा।।

Translation: Whenever Dharma is lost from the world, and the world is full of sinners, then, the Lord takes human form to take away the pain of the oppressed.

How does the clock for karma reset if the Lord kills you? Well here is what Ravana thinks when her sister tells him that his equals  Khar & Dushan have been killed by Lord Rama.

* सुर रंजन भंजन महि भारा। जौं भगवंत लीन्ह अवतारा॥

तौ मैं जाइ बैरु हठि करऊँ। प्रभु सर प्रान तजें भव तरऊँ॥2॥

भावार्थ:-देवताओं को आनंद देने वाले और पृथ्वी का भार हरण करने वाले भगवान ने ही यदि अवतार लिया है, तो मैं जाकर उनसे हठपूर्वक वैर करूँगा और प्रभु के बाण (के आघात) से प्राण छोड़कर भवसागर से तर जाऊँगा॥2॥

English: If it is indeed Lord who has taken a human from, then i will go and fight him. When i get killed, i will be out of this cycle of birth and rebirth and will attain heaven.

Tip: Shortest way to heaven is getting killed by the Lord himself.

On Books & E-Books Pricing

Just read a Gigaom Article on "new tips in books pricing"

Summary of article

Nothing new suggested here but still highlighting the points made just in case... and then taking forward from there.
The article discusses 2 /3  "new" strategies for e-books.

• Bundling
• Free - 
• Giving the first book for free in a series
• Giving an earlier title from an author free when new one is coming out. 

The article seems to be aimed at softening publishers and authors  by applying a stamp of approval on the strategy. Since most of the times publishers and authors resist the idea, and the holy promise in the article should help them bring around.

As the "strategy" has always been known to retailers. There's no surprise that Free Works.

Above all, all the article is focusing on is an electronic application of offline models. Internet and technology provide many more opportunities which most e-book sellers seem to be missing. Instead of going into that, i will quickly go into more relevant offline models which have not yet been ported online.

Thoughts

Free has always worked and always will. But every time you give out something for free, you raise customer expectations and mis-inform them. If you just made things free, how did you add value? Free was always an option.Why would you make your products compete with free?

There are only some scenario where price doesn't affect customer choices, or is not the prime consideration at-least.

• Value Vs. Perceived Value - Gym membership models.
• Occasion - Gifting, Airports, etc.
• Monopoly of content - Prescribed textbooks etc.

Loyalty based Models.

Rather it will make more sense  if we built more loyalty based models like Book clubs which allows you to get special prices. Most of profits can come from a GYM membership kind of models, where non-readers are taxed more than readers.

Gifting E-Books

Gifting is another model which tech/e-books haven't explored much. People always want to gift things to people. In this age of social media, where you may just know the e-mail address of a good online friend, gifting is tough.

Conclusion

Technology and Big Data seem to have removed the premium placed on editorial selections, but in reality, the editorial selection has become a hidden player. So must not be underestimated.

Data based electronic selection models are vicious circle which feeds itself, and keeps driving the selection quality down (includes shopping cart based models). There will always be more fiction readers than non-fiction readers. And since most models treat all books similar, fiction keeps on coming up as suggestions to a reader and it keeps growing.

With the same data, publishers realize they need to print more fiction than non-fiction, so more fiction gets published and so on..you know where it goes.

Social Responsibility

If you are an author or a publisher, its not just a job. You have a moral and social responsibility. That is the reason that society in many countries doesn't levy sales tax.

If you feel that there is a social responsibility in selling books, then just be more aware of the larger picture, and please consciously try to improve situations.

If its just a job, then screw society, and welcome profits, till you find you are selling glorified porn. 50 shades is a start. At some point in my life, i can see myself writing an article on how we can implement porn pricing models on books.

Start-ups are like Rock Bands, Orchestras like corporations

"Its very important to realize early in life, whether you are a rock band player or you thrive in an orchestra."

I was recently watching this movie called "Whiplash'. The story is about a guy in a music school who wants to be the best drummer he can be, but is facing a challenge from a demanding and vicious conductor/teacher who feels "good job" are the two most damaging words to an individuals progress.

Anyways, this got me thinking on the level of coordination required in an orchestra vs. a band.

I realized this.

Orchestras like Corporations

• Every person in the orchestra has to be supremely talented to be playing in it, more than most rock artists.
• Each person needs to be supremely coordinated with the other people in it.
• There is a structure, harmony and coordination, every one facing towards the conductor.
• The music largely still remains the compositions of the masters, replayed over and over again, with emphasis on interpretation/movement and excellence of performance than the music itself. 
• New music is created by the composer often the conductor, but follows the rules laid out by the previous masters, and needs to use the entire team.
• Its very hard to become a conductor/composer and takes years of hard work.
• Orchestras don't depend on people, and live beyond a person's life.
• Organized seating, everyone facing the conductor.
• You need to develop taste to appreciate what.
• You will not have personal glory, but a stable life.

Rock Bands like start-ups

• The number of people is very limited when the band is formed.
• The emphasis is always on a new kind of sound or music.
• As the band becomes popular, the artists become rich, and the staff increases multi-fold, but the people playing generally remain same.
• A lot of bands fail.
• When a band succeeds, its because they found a sound that people loved, and they remain a success till they follow people's taste.
• Band players may not be  very talented, or synchronized by rule, but should sound good together. That's the only metric.
• Very people dependent, if the people move out, some can be replaced, but definitely impacts success.
• Universal music, generally doesn't need a lot of education in taste to appreciate the music.
• You can have personal glory, but the chances of that happening are very tough.

The point

Its very important to realize early in life whether you are a rock band player or you thrive in orchestra.

Both are playing music, but have a very different approach to music.

Picture both the scenarios in your head, and find out which picture looks more appealing. Then build the strength required to succeed.

For a Corporation - Warm

Political - Social super intelligence + Political awareness + Idea hunting + Replicate

Neat - Quality over speed. Do it right every time. Neat

Pitch - People feeling work was done = Work was done.

For a Start-up

Execution - Idea + Execution + Common objectives based relationships

Messy - Speed over quality. One can always reiterate. messy.

Measure - Growth & Profitability not achieved = No work was done.

Man made things to measure a man's life

Came and read my own blog after 2 years of last posting. It was really interesting. A lot of things happened at once.

• Why did i stop writing?
• Oh..how true...wait..i wrote that.
• Umm...i still think that way
• I wish i had written more, so that i could have read more.

Its like the song "photograph" from movie Her. I have a snapshot of my mind right here on these pages! I wish i had more snapshots. Not because i am in love with myself. But because some of these thoughts are so familiar yet forgotten.

Forgotten in the humdrum of life.
Making sense of things.
Acquiring more.
Being on top
Running.
Forever
More
Fast.
Yes.
?
Why.
What for.
How much?
How much time?
Will i really be on TOP?
Sense of things that change?
Life? Snake eating its own tail?

Hhhmmmph....empty your lungs in one go. Take a shallow breath. let it out.Stop.Close your eyes. Open them. Look at everything around you. How does a baby look at these things. Imagine that. Look at things again. Your stuff. All around you. this is you. So much time spent acquiring it. hoarding it. saving it.caring for it. Take a deep breath. Look at it again. As a baby does.

Look at that bag in the corner..Imagine all the places that it has been.Imagine how it was manufactured. How it was designed.Go back, when the bag was just an idea in some one's head. Some one in this world. Is he alive? is he dead? he contributed to your life. Who? the bag designer? No.Not just Him. Everyone who ever touched everything that you have.

Do you see what i see? First feel the wonder. How much stuff is there in the world. And some of it is yours. and the entire world came together to make it happen.

And then feel the sadness. The world came together. but for what? These things? Things that sum up your life. Quantify it? things that are the measure of your life?

Man made things to measure a man's life.

Do you feel the sadness? Why did we not grow as people?