An “Alien” of Extraordinary Ability

Posted on by

I’ll pick up this post exactly from where I left if off on LinkedIn:

Sometime last year, a small little envelope arrived in my mailbox. It was thin, light, unassuming. Its physical appearance not at all representative of the freedom it was about to unleash in my life. 

The card inside the envelope goes by many acronyms: EB1A, GC, PR etc. (If any of these sound familiar to you, you might want to hear what I’ve got to say).

Since this news has traveled around a bit in my professional network, I have been getting many inquiries into how I managed this. I want this post to serve as the one stop shop for everything I want to say on the topic. For all the noisy discourse around legal immigration to the US, I invite you to focus on (and never forget) just 2 things:

The US immigration system is clogged and as such designed to block legal immigrants from India and China. Death by a thousand degrees is the general going theme. Feelings run high in all camps, the opposition, supporters and the casualties of this system. In this scenario, most legal immigrants are stuck between a rock and a hard place. And EB1A is increasingly becoming a way out. This is reflected in the fact that many borderline charlatans have popped up on the internet: “guides” who will “coach” you through the process, media publications who will boost your SEO, “influencers” who will teach you how to game social media. All for a fine fee ofcourse.

This is unnecessary. Firstly I would refer you to this excellent and realistic primer put together by my good friend Nikunj Kothari on what this visa category is all about. Next skip the charlatans, save your money and instead pay an amazing immigration lawyer (like Wells Wakefield).

Secondly, as the “influencers” keep deceiving the boundary of what qualifies for EB1A, so will the adjudicators become more random/unstable in the approval of this visa. This ever-increasing artificial randomness (like the one that already exists for H1B, both for approval and sadly even vetted renewals) is imposed on us by 2 groups of politicians. Someone sitting in a room detached and disconnected from the outcomes of their decisions locks a bunch of people in a quagmire of misaligned incentives. If any real progress is to be made for all the parties involved (opposition, supporter, casualty), it must be made by attacking that artificial randomness. Until then, even if some “influencers” slip “up” the cracks by whatever means (like making it their life’s mission to game the existing sorry-ass of a “system”) we are all mostly stuck in a lose-lose situation.

I personally do not want to be party to B.O.N.K-ing (Brevity Overshadowing Nuanced Knowledge) or RADSHaM-ing (Rapid Attention Demanding Simplified Hyperbolic Message)- (all fun acronyms concocted for me by chatGPT and amazing alternate names for social media :D). So, below is the real scoop on how I found myself with an EB1A GC.

A Rebuttal for the Extrovert Ideal

I am not going to waste words here on explaining how hard it is to qualify for an EB1A visa. A simple google search and the top few results should make that abundantly clear. What is not clear though is that to get there it is necessary to take non-obvious bets. When you take these bets, there is no guarantee that it will work out. By its very definition taking a risk means giving it all, sweat and tears without knowing whether its going to play out. 

My generation has something extra, a special salt shaker on the inevitable scars that one will accumulate while taking a risk. Its the constant blast of shameless promotion on social media. 

On any given day, at any point of time in history, it is hard enough to find out where the real value proposition lies. To create value and then to capture it. This is hard enough. Add to this the constant din of hyperbole from “influencers” who just can’t stop talking on social media and we suddenly find ourselves living in a communal FOMO induced hysteria. This situation is rooted not only in engagement greedy social media algorithms but also in the Extrovert Ideal : “the omnipresent belief that the ideal self is gregarious, alpha and comfortable in the spotlight”. I suppose every generation has had its salt shaker – print media, tv advertising etc. Social media takes all of that to a steroidal next level.

Still the R in “IRL” hasn’t changed. Creating value is still extremely hard and requires obsession, discipline, focus and risk taking. So if you do find yourself on that path, how do you maintain focus? Where should you spend your time? Building vs. distributing? Is distribution enough? Many “influencers” think so.

But then, what happens when you are focused on building? Chasing value. Balancing on that tightrope of focus while survival risk thanks to immigration and ADHD induced self promotion leaps up at you from every corner? When you find yourself out of energy and time for participating in this frenzy because you spent it all on surviving the tight rope.

For a long time nothing. Stone cold silence in the face of gut wrenching doubt. 

And then something changes:

From the moment my partner heard about the EB1A program, he knew we were a match made in heaven. We did a few profile evaluations with a few lawyers and one of them, Wells Wakefield decided to take up my case on a success fee basis, i.e. he would get paid only if my application was approved. He thought my profile had a great chance. I didn’t fully digest this enthusiasm or confidence at the time but the deal was too good to pass. The EB1A visa needs you to meet 3 criteria at the minimum to be considered for approval. Over the 3 months that we worked on our application, we discovered that I actually met 6 of them. It doesn’t stop there. In about a month’s time a bunch of spectacularly esteemed people had signed my endorsement letters for the USCIS. Going against somewhat conventional EB1A application wisdom, Wells submitted my application with premium processing and in 11 days it was approved. It doesn’t stop there. I had my green card in my mailbox in 5 months. Even my lawyer was amazed at the speed of execution and he later mused that this was one of the fastest green card processes he has ever seen in his career.

It has taken me quite a long time to digest this, to tell the background processor of immigration in my brain to shut down.

This is the good part. That small little part which one calls happiness. A “the dots connected” moment. What were the dots?

Waking up 3 Giants

Deep learning algorithms have changed the fields of computer vision and natural language processing. These algorithms have a massive appetite for data. And who has massive amounts of data represented on the internet? 

Anyone who has built ML models which interact with the real world daily, quickly runs into the limitations of the answer to that question. What about the under-represented majority (often incorrectly labeled minority) use cases which are not well represented on the internet? How to make ML models work for them? (This question is going to become even more relevant in the new era kickstarted by chatGPT).

This under-represented majority has another name. The Long Tail. The Long Tail is notoriously hard to model. Making ML models which are inherently built for the represented, also work for the under-represented requires a lot of expertise and creativity. 

I essentially found myself in this position early on in my career. A senior experienced engineer had tinkered around with a classical language modeling technique to do this. I was tasked with scaling this tinkering and seeing if it stood that test. The one of Global Scale that is. As I worked to scale this technique for languages and regions that fell in the long tail, integrating location and speech signals seamlessly in a privacy conscious way to improve the intelligibility of long tail words, it seemed to beat every test it was offered.

Now here’s the fun part. In some review by a powerful conservative government (1st giant), it was flagged and put under export control. (“Lets stop the entire thing”, the classic way conservative regulation deals with what it doesn’t understand). The whole platform could operate cross-border except for this little module whose bits and bytes would not be allowed to cross the optical fiber lines beyond the physical border of the country. Yes- the digital bits and bytes are being treated like physical goods here. 

In an internal report (2nd giant), it was found that this model was being used 35 times every second every day. Let that sink in. 35 times every second every day. This kind of “engagement” would give many “product” folks an orgasm. (Also, so much for the infrequent long tail). 

Kudos to my lawyer Wells Wakefield for understanding the full legal and economic impact of this work and adding that to my petition (for the 3rd Giant).

The Big Picture

If you survived this long into the post, I want you to click on the link of the paper, the core language modeling technique which provided a systematic way to deal with the long tail. I didn’t share this link simply to sound credible or to throw around jargon.

Among the motley crew of authors on that paper who have done phenomenal work in NLP, 2 names went on to have a disproportionate impact on the financial landscape of America. And then one of them went on to have an even more disproportionate impact on the political landscape of America. The king maker behind Donald Trump. There is an extremely good chance that his life’s work, both political and otherwise, in some way or another, has impacted you directly.

The simple, unassuming nature of that 1992 paper, belies the power and the genius of (the) Mr-Hyde-of-Ren-Tech. If his political beliefs had their way, a person like me would never have been able to carry on his intellectual legacy and bring it to the present. No one on my team who worked on making this idea live and breathe would have been able to do so if the immigration system was cast in the mold of those political beliefs.

This is the cruel irony. 

Gratitude

Against the backdrop of a constant one-step-forward-2-steps-back cadence of politicians and their immigration policies, the world actually moves forward because of builders.

It is the mark of true builders that all the esteemed people who provided me endorsement letters for my USCIS petition believe in the values of privacy over fame. As much as I dislike it, I refrain from citing them lest they be unduly flooded with such requests in the future. This group includes but is not limited to a phenomenal self-made entrepreneur and venture capitalist, a founding CEO of one of the largest and youngest media houses in India (its a she btw ;)), an ML researcher who had worked directly with Geoffrey Hinton way before deep learning was cool and another ML researcher who is changing the field of multimodal computing as we speak.

Again – What do you get for focusing on building value? The company of other value builders. This in and of itself is probably the best reward I have received for my work over the years.

Last but not the least, my best friend, partner and better half, Sahil. We have been together for 11 years now (married for 5 of them). Sahil was there with me every step of the way for every success (and every failure) that got me here. He has seen it all with me and been my rock through the mighty highs (and the inevitable lows that come with trying to scale “mighty” highs). Sahil managed all the moving parts and external dependencies of this process so seamlessly that we achieved this in record time while managing massive other life changes, both personal and professional. 

I don’t fit the stereotype of someone who should be this successful and independent. Neither in my country of birth, nor in my country of immigration. I receive that message from both those societies regularly enough. But the world moves forward not at the behest of naysayers but because of focusing on the few that make it worth the while. Waking up to Sahil is a daily reminder of that fact.

Epilogue

I hope that serious and deserving EB1A applicants find some solace and then some strategies in my story. All the best!!!

The Lifecycle of Open Source Projects on Github

Posted on by

Aaksha Meghawat, Chris Bogart, Carolyn P. Rose

In 2016, I started out with the goal of using natural language processing & discourse analysis to improve developer productivity on Github. Thanks to early success in our computational approaches, we were soon asking the question of whether we could predict if an open source project was on the path to failure or success. This was further motivated by the fact that a developer’s time is limited and therefore in Open Source communities, they need to make important decisions regarding which projects to depend upon as resources and which projects to contribute towards.

Our qualitative analysis to develop computational measures for conversational quality revealed that projects went through different phases of activity (Fig. 1). First, to help in characterization of the ‘activity’ phases, we collected several important activity parameters such as commits, pull request merges, pushes, issue comments, issues opened, issues closed, commit comments, pull request rejections and issues remaining open totaled per month of a project’s timeline on Github.

Fig. 1: The project starts with a lot of community involvement (evidenced by ‘merges’) but later on ‘pushes’ dominate the version releases 

Next, we needed a model that would be able to detect these phases of a project assigning a clear phase identifier for the months of activity of the projects. This identification would make the wide variety of projects comparable at different points in time. For example, this would help in comparing the coordination style of a new small-sized project in a phase with an older project’s earlier days when it had been a small-sized project and was in the same activity phase.

“One can think of this as a health checkup of the open source project. The activity indicators are like the vital stats of the project. This combined with its history when compared to life histories of other successful or unsuccessful open source projects can give some early indication of its potential “health” problems and survival rate”.

We hoped that enabling such comparisons would drive insight as to the quality of coordination that may have helped a project in surviving long enough to attract more resources and become a big project. A clear interpretation of these states in terms of the parameters collected would help in evaluating the soundness of the model’s states. Further, our understanding of how the phases progressed and what kind of coordination and conversation would result in a project reaching favorable or unfavorable stages in their life cycles would improve. For this reason, we decided to apply models that would infer discrete states for the project timelines.

The other important characteristic of the project timelines that we discovered in our preliminary qualitative analysis was that developers worked in bursts of activity. This clearly meant that the activity (and phase) in one month was not un-correlated with the previous month’s activity in the sequence of months that formed the timeline of a project. We needed a model that would consider the temporal dependence of the activities reflecting the fact that some activity in one time unit increases the probability of seeing more activity in the next time unit. At the minimum, we needed a model that would relax the i.i.d. assumption for its data points (time units of months in our case).

We conceptualized that the discrete state would be an indicator of the phase of the project’s lifecycle which would be expressed via the activity visible on the platform. The project’s current state would determine the state that it would occupy in the next time unit. Since the phase of a project was not visible and was not specified directly on Github, our statistical model would have to infer the hidden state from the visible activity snapshots and learn a probabilistic distribution over these activity variables to better fit and account for the variety of activity that happens on the platform.

The Hidden Markov Model (HMM) fit all these requirements neatly. The markov property of the model captured the intuition that the present state determined the future state of a project. The HMM is a statistical Markov model in which the system being modeled is assumed to be a Markov process with unobserved (i.e. hidden) states. Additionally, we chose to model the probabilistic relation between the discrete states and the activity on the platform via a multivariate Gaussian distribution. This way we could leverage the flexibility of a probabilistic fit on the observed activity to acquire a better model. At the same time we could take advantage of the interpretability of the Gaussian distribution by understanding every discrete state that the model inferred in terms of the mean value of the observed activity parameters.

Data Collection

We attempted to achieve the fine balance between having homogeneous samples in terms of the type of projects and them being diverse enough in terms of work coordination. For this reason we chose a single domain of development, Pypi projects and began with a large set of projects within this domain in Github. We began with a list of 48,668 github repositories mentioned in PyPi metadata records. From these we altered for projects having at least 10 stars or 3 contributors. We took stars and contributors as an approximation of the community’s interest in the project, leaving us with 16,682 projects.

There were discrepancies in many projects because of projects whose names had changed or for which data was missing or deleted from one data source or another. As we were looking at fine-grained interactions over short timespans, we wanted a particularly clean set of projects. We finally had 13,479 pypi projects in our set.

We trained a multivariate Gaussian hidden Markov Model (HMM) on these activity parameters. For the month level HMM, 12 discrete states provided the best fit on the activity time lines of a held out set of projects. We then included all the months that were labeled for discrete states of the Gaussian HMM which showed significant levels of activity. We focused our analysis on 6 states with significant transition probabilities (Figure 2). After filtering for the active months, only 4934 projects remained in our set as of the 13479 projects that we started with, only these many had active months.

Figure 2: Summary of the key HMM states capturing the lifecycle of an open source project (Github)

Interpretation

The transition probabilities of the HMM states confirmed most of our qualitative conclusions of the activity patterns of projects. Any project was most likely to start and stay in the dormant state. Furthermore a project’s probability to maintain its current state, whether dormant or active was higher than transitioning to a different state. The trend of staying in the dormant state created the issue of data sparsity. The trend of staying in the active state created the phenomenon of bursts. Both these intuitions are visible in the HMM model.

It may be argued that one could set a low threshold on the activity parameters to capture most activity. However, in our qualitative study we found that low activity which preceded or followed high active states was also crucial in determining the coordination quality during that burst. All low activity is not important however and can be ignored to avoid noise. A markov model can automatically take care of such subtleties.

Figure 3: The HMM model applied to a project’s activity indicators to summarize its different development phases.

Future Work

This HMM lifecycle framework has already been used as a filtering mechanism to focus on different phases of a project by other researchers. This helps them zero in on the phases of a project they are most interested in to observe or measure other metrics relevant to their problem statement.

Potential future work also includes gaining a better understanding of the kinds of events/factors that cause the project to transition from one state to another, eventually estimating their survival rate.

Github Conversations: Characterizing Framing & Measuring Openness

Posted on by

Aaksha Meghawat, Steven Moore, Qinlan Shen, Steven Dang, Carolyn P. Rose

Prologue

Who decides what is right? Who decides what is a ‘best practice’? The way open source developers, teams and software organizations deal with these questions is crucial to their success and the community as a whole.

The research I document in this post was carried out by me and a few of my amazing lab mates in 2016. With the rise of remote work and most work/design conversations (negotiations, decisions) moving to Slack, Zoom and other such online mechanisms, I have a new found appreciation for the insights we gleaned back in 2016.

General opinion seems to be that (other than likes, retweets) measures of the quality of a discussion such as openness, topic diversity, framing styles etc. cannot objectively be measured in any useful way. We challenged this assumption in our work and came across some interesting insights into developer productivity. Using our designs of conversation quality, we were able to improve the F1 score of predicting acceptance/rejection of a pull request by 66%. For 2 other key takeaways, you can jump directly to key takeaways.

The ultimate vision of this work was to translate our conversational quality measures into tools and interventions that would help developers and all kinds of teams have more productive conversations. Unbeknownst to us, many smart people had the same idea. Our ultimate dream would be make this a seamless real time conversation quality measure to help people have better conversations. The closest thing to this (out there) is Bridgewater Associates’ Dot Collector, (from 9:30 onwards in this video).

Introduction

Discourse in online work communities differs from that found in weblogs, online support groups, or news sites — the task-focused yet social nature of the community influences the nature of the conversation, as discussions are not solely focused on the ideas and issues at hand but also the social dynamics of the group (Richards, 2006). This raises the question of what are the dominant manifestations of this social influence on task discussions. Here we propose an analysis of Github pull-request conversation threads, looking at both how Openness and Framing are leveraged as well as how the two may interact to influence the success of a community pull-request contribution.

Looking at the data through the perspective of Openness, we wanted to see how accepting a project’s community was when discussing a pull request. We also looked at the data through a Framing lens to see how the topics discussed in a thread were indicative of its success. Combining these two lenses would hopefully provide insights into how a thread’s contributors made their ideas heard while also playing their part in the project’s community.

We analyzed pull request conversations from a subset of GitHub projects for Ruby. The data consists of 1862 projects that contain at least one pull request. From these projects, we picked out threads that have at least three different contributors. 

Research Questions

We define “success” of a conversation as acceptance/merge of a pull request. Our primary research questions then are whether the following have an impact on this “success” metric:

  • Openness: How do we define & detect openness computationally? (RQ1)
  • Framing Mechanisms: How do we define & detect different ways of framing, computationally? (RQ2)
  • Topic Diversity & Topic chains (Transactivity) (RQ3)

Data Description

We analyzed pull request conversations from a subset of GitHub projects for Ruby. The data consists of 1862 projects that contain at least one pull request. From these projects, we picked out threads that have at least three different contributors.

We picked out threads for qualitative analysis to get started on RQ1 & RQ2. We sampled threads from both highly active and less active communities. To do this, we stratified the communities according to total number of pull requests and sample two projects above and below the median number of pull requests. For each of these communities, we sampled 3 pull request threads. The threads in the selected communities were divided into three groups based on the number of comments in each thread (low, medium, and high number of comments).

Measuring Openness (RQ1)

Openness/Expansiveness in our work is defined as a statement which creates more dialogic space for another participant’s expression or opinion. This is often operationalized through a question or asking for explicit participation from other people. However it may also be demonstrated via expressing alternative views. We term a statement that has the opposite effect as Contractive. (Note that the usual positive connotation attached to ‘Openness’ should not be done so here. Vice versa for ‘Contractive’ statements. This is because too many expansive statements can also lead to confusion or slowness in decision making, resulting in failure of merging in pull requests, as we will show later in examples).

Following qualitative analysis, a list of expansive and contractive patterns was created from common patterns found. These initial patterns were input into the model where we then checked to see if they were appropriately tagging units as expansive or contractive. After a small sample was both qualitatively & quantitatively coded for expansive/contractive patterns with potentially interesting markers, we proceeded with affinity diagramming. During the affinity diagramming process, the markers that didn’t directly fit into expansive or contractive, such as using a particular acronym or ‘+1’, were grouped into various themes. After fitting them into their final groups, the context of the units for which these markers were used into was counted. This provided us with a ratio of many expansive, contractive, or neutral units these markers fell into. Markers that were most commonly found as either expansive or contractive by 75% or more were used as patterns for the respective classification. For example, the acronym ‘imo’ was used as a contractive pattern and the use of ‘?’ was used as an expansive pattern. 

Openness% Sample% Computational Tagging
Expansive20.216.36
Contractive3.782.67

Table 1: Percentage of units tagged as expansive or contractive (sample vs. computationally tagged dataset)

Characterizing Framing (RQ2)

We distilled 5 different framing types to be key for developer conversations on Github from our qualitative analysis:

CodeDefinitionExample
Establishing StatementIntroducing or supporting the factual/valid nature of a propositionIt’s very difficult to reproduce error because it depends on terminal width.
Alignment StatementStating agreement/disagreement on a specific discussion point including statements of appreciationGood idea @jferris. “+1”
Contrasting StatementDiscussing demonstrated or proposed attributes in contrast to alternatives
I would phase it out slowly, i.e. have has_plugin? recognize both current name and the gem name initially, then slowly deprecate it (with a warning possibly if the old name way is used).
Alignment RequestRequesting statement of stances on a specific discussion pointCould you please have a look there and comment?
Clarification RequestRequesting restatement/elaboration/clarification of a specific discussion pointsIf we leave this in: Is this something that needs to be generated during clearance install, if the users table exists?
Table 2: Framing Code Dictionary
Qualitative Analysis Methodology

Qualitative analysis began with reviewing 3 conversations annotating conversation fragments for two primary characteristics: framing related discourse moves and linguistic indicators of described framing moves. From this annotation library, an initial coding dictionary was formed by synthesizing the annotations into groups and forming operational definitions of each group. These codes were carried forward into the analysis of an additional 3 conversations in order to evaluate their robustness, ability to differentiate between statements, and coverage, ability to categorize all statements in the data. New framing moves were annotated and new synthesized definitions were formed from the revised groups. This analysis was repeated for the remaining 2 groups of 3 conversations until a final codebook was formed as shown below in Table 2.

Definitions of Framing Codes

The resulting code book discovered two broad classes of discourse moves related to Framing, Statements and Requests. Statements describe sentences that introduce information to the discourse while Requests are statements that solicit information from others to be introduced, where information can be factual or opinion-based in nature.

Establishing statements were the foundation of most discourse where they are describing all discourse moves that are introducing information with the purpose of establishing its factual nature. From a framing perspective these statements would be important to identify in the sense of determining which statements are introducing factual information and being able to identify which statements are debated and which are ignored and/or accepted.

Alignment statements are specifically discourse moves which introduce an individual’s agreement or disagreement with a particular fact or discussion point. These statements capture categories of discourse moves intended to introduce individual opinion into the conversation.

Contrasting statements are defined as discourse moves that introduce information intended for comparison or contrasting with previously stated or assumed information. These statements specifically contain multiple perspectives and relational information between each perspective.

Alignment Requests are defined as statements explicitly soliciting the opinions of others. Requests differ from statements in that they can also introduce information, but the primary goal of the discourse move is to introduce information with the intent of soliciting opinions of others regarding the stated information.

Clarification Requests are defined as statements explicitly soliciting additional information from others usually with respect to previously introduced or assumed information. These requests may also introduce information, but with the intent of supporting the specifics of the request for additional information.

Key Observations: Usually a contribution is elaborated early in the conversation and one or two specific points are discussed for the remainder of the thread. In the sample analyzed, two threads from the same community introduced comparable features, lazy loading, but each thread presented the value of the idea from different perspectives. The result is that one thread was merged and the other was rejected. On the other hand many of the rejected threads followed a pattern of short elaborated descriptions followed immediately by at least one contrasting statement which is then followed by alignment statements. This indicates that capturing patterns of these base categories of statements could be valuable in understanding the conversational dynamics.

Similar to Openness, we extracted patterns for our Framing categories. Then we tagged sentences in each post as belonging into one of the five framing classes. Using the sentence tokenizer in NLTK, we split each post into individual sentences. For each sentence, we then counted how many patterns fit the sentence for each of the framing categories. We took the class with the most number of pattern hits as the framing class of the sentence. Sentences that do not exhibit any of the patterns are assigned to the Establishing Statement class. For each post, we then maintained a percentage of sentences belonging to each category.

Transactive Chains (RQ3)

A major aspect of framing is the control of what topics are discussed in a thread. To operationalize this aspect of framing for our model, we introduced the concept of transactivity chains, a time-dependent chain of comments in a thread that are similar in what they discuss. The rationale behind these transactivity chains is that they represent different “flows” in the thread about what is being discussed — one transactivity chain represents the entire lifespan of a certain conversation subject. By examining the different transactivity chains across a thread, we can gain insight into what kind of conversational structures may be indicative of successful collaboration.

We define transactivity chains as a time-ordered chain of comments where posts that are connected have some semantic similarity. To measure the semantic similarity between posts, we run LDA (Blei et al. 2003) over our entire dataset, using 50 topics and treating individual comments as documents. This generates a topic model over our dataset and assigns each post a distribution over the generated topics. We then consider two posts to be semantically similar if they have a cosine distance of less than 0.5.

We provide a simple example to demonstrate how we represent the transactivity chains as features for our model. Consider Figure 1, where each numbered node represents a comment (time-ordered) in a thread and edges represent that two nodes are part of the same transactive chain. The thread represented by the figure consists of 6 posts that make up two transactivity chains.

Figure 1: Example transactive chain structure

We encode each of the transactivity threads as 3 features: the number of comments in the transactivity chain (#Comments), the number of comments the transactivity chain spans (ΔT), and the normalized comment distance of the final comment in the chain from the end of the thread (LastTimeSlot). This gives us the following set of features for each of transactivity chains in Figure 1:

ID#CommentsΔTimeLast Time Slot
1366/6
2335/6
Feature extraction for Transactive chain shown in Figure 1

To summarize these features at the thread level, we use the median, mean, min, and max of the transactivity chain features. We also use the median, mean, min, and max values for the percentage and total number of statements tagged as each of the openness and framing features among the transactivity chains of a thread as transactivity chain features.

Results & Key Takeaways

The surprising relationship between Openness & Transactivity

Graph 1: No. of Transactive Chains vs. Expansive Units

In Graph 1 we see that points with greater than ~50 expansive units are sparse in terms of demonstrating a trend. We intuitively expected that the more expansive conversation units there were in a conversation, the more topics (and no. of transactive chains) would be found in the thread. However as seen in Graph 2, the general tendency to the left of the red line is that an increase in expansiveness does not necessarily increase the number of transactive chains. On looking at some of the conversations our intuition was that increasing expansiveness created greater dialogic space that encouraged discussion about the current topic, infact encouraging continuity of a transactive chain and potentially greater consensus building.

Graph 2: Average number of topic chains in a conversation vs. number of Expansive comments.

This intuition is also reflected in Graph 3 where the length of the longest transactive chain increases with increasing presence of expansiveness.

Graph 3: Length of longest transactive chain vs. Expansiveness

Conversely, according to our expectation threads higher in Contrastiveness should have had a lower count of transactive chains, as the posts would tend to be monoglossic and dead end. However, we found that in the threads shown in Table 3, the thread’s commenters reference other users in the project who have not yet commented on the thread. This brings those users into the thread’s conversation where they then contribute to the discussion and generate new transactive chains. It also appears to add to the contrastiveness of the conversation, as it’s often the case that the users come into the conversation and immediately provide their opinion.

Example Thread# Transactive ChainsContrastive %age
https://github.com/jekyll/jekyll/pull/1682525
https://github.com/mitchellh/vagrant/pull/1592521.43
https://github.com/resque/resque/pull/772621.05
https://github.com/vcr/vcr/pull/404521.48
Table 3: Threads and their percentage of clarification requests and expansive units

Unconvering a best practice to increase developer productivity

An interesting finding we came across was the interaction between one of our framing categories, clarification requests, and our measure of openness. We found that in threads which had a high percentage of clarification requests and an equally high percentage of expansive units, developers were following a common pattern to make their voice heard. The developers always began by referencing a particular portion of the commit with a code block and then proceeded to ask a question about it. By following such a pattern, the commenters were able to have their question(s) and opinion immediately addressed.

This insight can be used directly to design a feature in github. For example, every time a developer has a question, they can be encouraged to refer to a particular block of code relevant to the question to encourage a more useful and productive discussion.

Prediction of Pull Request Acceptance (Primary Research Question)

The final experiment that we ran was to use our features to train a classifier to predict whether a pull request would be accepted or rejected. We train a logistic regression classifier and use 5-fold cross validation on our fully tagged dataset (which includes our openness tags, framing category tags, and transactivity chain features) to compare the performance of four different feature configurations:

  • Openness: % and total number of each openness category
  • Framing: % and total number of each framing category
  • Openness + Framing: combination of Openness and Framing features
  • Openness + Framing + TC: Openness, Framing, and Transactivity Chain Features (described in the Transactivity Chains section)

We compare these models to a baseline model, which predicts the majority class (Accepted).

ModelAccuracyPrecisionRecallF1
Baseline53.826.950.0034.98
Openness53.8752.4551.5047.41
Framing59.2459.2557.7256.67
Openness + Framing59.3159.1457.9757.24
Openness + Framing + TC59.5559.2458.4458.04
Table 4: Classification results for predicting the acceptance of a pull request

We improve the F1 score over baseline by 66% using our features of measuring openness, framing styles and topic diversity/transactive chains. This indicates that the features we have developed encapsulate meaningful quality measures of developer discussions and can be used to gauge the potential productivity of a group of open source developers.

Data Driven Case Study

As an example we examined the outlier thread (circled in red in Graph 1). We were able to neatly summarize the negotiation that happened in the conversation using our metrics.

Figure 4: The longest conversation in our data whose general flow was neatly summarized by our features.

Works Cited

  • Argamon, S., Whitelaw, C., Chase, P., Hota, S. R., Garg, N., & Levitan, S. (2007). Stylistic text classification using functional lexical features. Journal of the American Society for Information Science and Technology, 58(6), 802-822.
  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. the Journal of machine Learning research, 3, 993-1022.
  • Chatterjee, M. (2007). Textual engagements of a different kind? In Proceedings from Australian Systemic Functional Linguistics Association Congress.
  • Chong, Dennis, and James N. Druckman. “Framing theory.” Annu. Rev. Polit. Sci. 10 (2007): 103-126.
  • Cosentino, Valerio, Javier Luis Cánovas Izquierdo, and Jordi Cabot. “Three Metrics to Explore the Openness of GitHub projects.” arXiv preprint arXiv:1409.4253 (2014).
  • Howley, I., Mayfield, E., & Rosé, C. P. (2012). Linguistic analysis methods for studying small groups. The International Handbook of Collaborative Learning.
  • Gee, J. (2011). An Introduction to Discourse Analysis: Theory and Method, Routledge. 
  • Jo, Y., & Oh, A. H. (2011, February). Aspect and sentiment unification model for online review analysis. In Proceedings of the fourth ACM international conference on Web search and data mining (pp. 815-824). ACM.
  • Martin, J. & White, P. (2005). The Language of Evaluation: Appraisal in English, Palgrave
  • Nguyen, V. A., Boyd-Graber, J., Resnik, P., Cai, D. A., Midberry, J. E., & Wang, Y. (2014). Modeling topic control to detect influence in conversations using nonparametric topic models. Machine Learning, 95(3), 381-421.
  • Read, J., & Carroll, J. (2012). Annotating expressions of appraisal in English. Language Resources and Evaluation, 46(3), 421-447.
  • Richards, K. (2006). Language and Professional Identity: Aspects of Collaborative Interaction, Palgrave, Chapter 3
  • Taboada, M., & Grieve, J. (2004, March). Analyzing appraisal automatically. In Proceedings of AAAI Spring Symposium on Exploring Attitude and Affect in Text (pp. 158-161). AAAI Press.
  • Tsay, J., Dabbish, L., & Herbsleb, J. (2014). Let’s talk about it: evaluating contributions through discussion in GitHub. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 144-154). ACM.

This research work was supported by an NSF “BIGDATA” grant. Code, Data and Features can be provided on request.

Musings on a BITSian Life: Epilogue

Posted on by

This post serves an Epilogue for the series on ‘Musings on a BITSian Life‘.

There is a time and opportunity cost attached to everything. Everything. This is probably one of the most humbling realizations of adulthood. Its probably why people came up with the proverb: “Youth is Wasted on the Young”. (Not sure why more young people are not doing something to change this, well anyway).

The 20s can be an incredible time of one’s life. Youth has a definition in terms of age. I would like to add to that definition. Youth is when your time belongs to you, that beyond certain basic things, you are answerable/accountable to almost no one. Think of the vast possibilities and freedoms such a state of mind brings. It so happens that in the current setup of our lives, mostly 20s is the age around which people find themselves to be in this state naturally. (Imagine those who never get this chance and those who never realize that they are experiencing it, bah)!

Having some inkling of this, I set out in my 20s with the goal to have a reasonably good idea as to where I stood on the Risk-Endurance graph I describe in Future of Work: An Individual Perspective. Even the dating world has a version of this goal paraphrased in its own language. It goes something like this: “Before an I Love You, there is an I”. Armed with these aphorisms, my 20s were filled with experiments. Filled with questions and zealous attempts to answer them. The world tested me and I tested the world. It was tiring, painful but very, very revealing.

I will go out on a limb and say that close to graduation, take a blank piece of paper. Sketch out the axes of the 2 graphs (1,2) I have described in Future of Work: An Individual Perspective and put a few dots on them (for yourself and the people important to you). You may get a graduation degree at the end of your time at BITS but that little piece of paper will be your own (real) graduation certificate.

In my last few days on BITS campus, I found myself once again at odds with some sections of the administration on a technicality. Most people I knew had said their goodbyes, packed up bags and left. I was determined to find a way out of the technicality however (because the alternative was not acceptable to me). In the searing desert heat of Pilani, I visited my room in the small breaks from that rebellious attempt to empty it out in time for final summer closure. On one of those days, a junior from the Athletics team I had captained a year ago gave me a hand-painted poster with this little couplet from my all time favourite poem:

Lives of great (wo)men all remind us
We can make our lives sublime,
And, departing, leave behind us 
Footprints on the sands of time;

Footprints, that perhaps another, 
Sailing o’er life’s solemn main, 
A forlorn and shipwrecked brother, 
Seeing, shall take heart again.

My particular circumstances at the time had rendered me pretty lonely and depressed and I was losing sight of the journey. (My favorite poem coming back to me in such an incredible way felt like a sign from the universe). So, I would like to thank her for making me realize that I was leaving footprints in the sands of time and that someone was noticing. 

I have argued at length about the increasing role of specialization in the future of work. And I began this epilogue with how everything has a time and opportunity cost. Given the nature of the competitive field I am in, it would have been far more beneficial professionally for me if I had spent the time instead writing some research paper to get published in a fancy AI/ML conference. That background processor was always running as I churned out some of this. 

However, I wanted to honor the sign the universe sent me on a cruelly hot summer afternoon in Pilani in 2012 and leave honest footprints in the sands of time. Brutally honest ones at that. I hope that this helps some forlorn and shipwrecked soul somewhere, someday, take heart again! 

The Future of Work: An Individual Perspective

Posted on by

COVID-19 has forced us all to rethink many aspects of our lives. Job losses, stimulus checks, 0% interest rates and stock market turbulence dominate conversations when avoiding the more morbid topics of death and disease. Most large scale systems are struggling to deal with this pandemic in a coherent way and these struggles offer a rare lens into what we value as a society. As I work longer and longer hours, and fight with a variety of electronic screens to protect my cognitive real estate, I’ve been compelled to analyze the relationship between Work, Education and Value (Productivity).

“Never let a good crisis go waste.” Sometimes the music is not in the notes but in the spaces between them. Taking this pandemic as that silence between the notes, in a series of blog posts I examine deeply the relationship between the 3 pillars of the knowledge economy: Work, Education and Value.

In The Future of Work, but first a History, I briefly tour history to understand how common ideas about Work, Education & Value (Productivity) became ‘common’.

In Future of Work, but first Now and the Near Future I talk about why those ‘common’ ideas are not a reflection of reality any more (pandemic notwithstanding). I try to understand reality and where we are headed in the near future if nothing is done about fixing the gap between that reality and the common ideas/assumptions.

In Future of Work: A Vision, I first propose a hypothesis of the crux of the gap problem. And then, based on all of this, I present a vision for the future of work. 

In this piece I suggest a framework to prepare for that future at an individual level, (one I have used at every decision point of my career). I also intend this piece to serve as a final one for my series on ‘Musings on a BITSian Life’.

To recap from Future of Work: A Vision

“As the rate of change of relevant skills becomes higher, one will have to take bigger and more frequent bets with their time to develop new skills and specializations. This cannot happen if one is not passionate, disciplined, talented etc. but most importantly this cannot happen if one’s work is not a good representation of one’s self, one’s aspired identity. The resilience and endurance required to keep meeting waves of change cannot be developed if the actions required for this are not in sync with who you are and who you want to be.”

The Future of Work: A Vision

To have some clarity on that last bit, I propose that a person should have a reasonably good idea of where they stand on two graphs.

Risk Endurance Profile

In Asian households, there is a misguided refrain oft repeated motivating children to put in grueling efforts for grades/entrance exams to have a shot at an elite education. It goes something to the effect of “Study for 10th grade /national board exams, you can enjoy life later”. Then after a few years: “Study for 12th grade/SAT exams, you can enjoy life later”. Then after a few years: “Study engineering/medicine, you can enjoy life later”. Then after a few years: “Do an MBA, you can enjoy life later”. Then after a few years: “Get into investment banking or consulting, you can enjoy life later.” Then after a few years: “Get promoted, you can enjoy life later”. This is a version of the work treadmill I have described, for young people. This sentiment and its psychological impact on young people has been captured beautifully in the international hit movie: 3 Idiots.

As I have argued here that this advice has several problems with it but the most important one is that it is outdated and irrelevant. With the highly visible success of technology companies today via stock market, valuations, IPOs, and the buzzwords related to AI being hyped everywhere, newer parents might advise their children to study computer science because that seems to be the future. That would indeed update the advice, however it would still remain mistaken.

Just as an example, the competitive advantages of knowing generic computer science are also thinning away rapidly. It is already advisable to develop expertise in particular branches of computer science or machine learning or combination fields such as robotics, NLP. And that trend is only going to continue. 

The real point is that if you are on a path or a trajectory, moving forward on it will only require you to become more and more of what got you there. Not only will you have to do more of it but you will most likely have to get very good at it. If you have to change yourself too much to get somewhere, its a great indication that the whole destination should be re-evaluated. Because the dissonance between the journey and the self, if it exists, usually only gets worse. As career trajectories become more exponential, winner-takes-most in nature, this question of re-evaluation will become more important for more people.

Therefore when standing at a decision point i.e. choosing a journey and a destination, one must have some understanding of where one stands on this graph. What are the risks of that path and does one have a reserve of endurance to face their downsides? A lot of people know this somewhat intuitively and frame the question in different ways but those questions are essentially finding one’s place in this graph:

Risk Endurance Profile

There are two very important notes to keep in mind during this thought exercise: 1) This assessment can change over time. For example, you may find yourself in possession of more financial resources increasing your appetite for risk or endurance. Similarly your life’s circumstances may change due to external factors making it impossible to take high risks. So very crucially, 2) this assessment is not a judgement. Some egoistic “self-unaware” people would like to think that placing oneself in a low-risk-low-endurance category would be undesirable. However this assessment is not necessarily intrinsic to your character. For example most societies suffer from a systemic lack of imagination about women’s roles in the future of work, education and productivity. There are societies where nobody wants to invest in them (ensuring 0 endurance) and nobody wants to take a bet on them (ensuring 0 risk taking ability). This leads to social systems forcing a certain quadrant on a certain demographic of society. Therefore placing yourself on this quadrant is not so much a judgement but an honest look at where you stand and what may lie ahead. 

Aggressive-Passive-Independent-Conventional

The first graph was mostly about placing oneself on it. I also argue that it is not intrinsic to who you may be as a person. But the next graph may be more so.

No (wo)man is an island. I struggled for a very long time to crystallize the many ways in which you also need to understand and place the people who surround you, a mental model of sorts for people. But then I came across this exceptional essay by Paul Graham on ‘The Four Quadrants of Conformism’ and there was an aha moment. Here is my attempt at a pictorial representation of the quadrants he has laid out. Of course every word of that essay explaining the quadrant is worth a read.

Axes of Conformism

Why is this useful? Sometimes when you are too embedded inside a group of people like high-school friends, college friends or family, it is hard to see the bigger or the real picture. The usual tendency is to consider anyone with a different orientation inferior. Thinking for a few minutes about people surrounding you on these axes might bring reality closer home. 

For example in a previous essay, I have described how groups which don’t have a clear purpose tend to be exceptionally subjective. Groups that don’t have much clarity on why they are together look for artificial reasons to stick together, often fermenting and rotting in each other’s company. Instantaneously becoming more rigid, closer and suspicious of newer arrivals and further entrenching that process. If your bond with someone/something derives strength from stepping on someone else, ironically that entity you collectively despise becomes the most important defining aspect of the relationship. Only groups that have strong clarity on why they are together can also at the same time be agile enough to discover new people.

Which kind of group are you in? Which group of people do you have a problem with? Placing both on a graph can lead to better understanding and better preparation for the future.

I’ll end with two things (relevant mostly for young people). It is remarkable how much our institutions encourage pattern-following and rarely ever provide the tools for pattern discovery (let alone pattern-questioning, maybe the reason we are set up in a race with ourselves for self destruction, hello climate change!).

As I elaborate on this point in the context for college students in this essay, developing a good question and finding the right people to ask it are both incredibly important, yet very hard. But an inexperienced person is in the unique position to seek out many people, study many journeys and observe the outcomes. To separate the traveler from the journey and the destination. To carry out good great pattern discovery.

The worst* kind of young people are people who think an earlier time in history was a better time, who look back at history, convention and tradition and believe that some prior combination of these was better. It means that they have bought into the power systems which have caused the world as it stands today. (You could be a completely ignorantly blissful person and say “so what’s the problem with the world as it stands today”, but I highly doubt such a person would even be reading this right now).

The best kind of old people are people who are excited for the future and wish they could live a little longer because it demonstrates a vision for an improved possibility, likely one that their life’s work has contributed to. You must decide pretty early on which kind of young person and which kind of old person you want to be.

Epilogue

I must acknowledge Sahil Shah, my husband, who brought Paul Graham’s essay on conformism to my attention.

If you ever wonder, why I wrote all of this, I answer that here.

*P.S. Thanks to a new trend of hyped up lists of 30-under-30, 40-under-40 entities (people, startups, wannabe twitter celebrities etc.), there is a new contender for the worst category. I do not want to waste too many words on this professional version of attention seeking. Also, I am totally in the market for any comprehensive analysis which could answer questions like: How many startups which showed up in the 30-under-30 category were able to survive 5 years after that. Or if any managed to transition to the 40-under-40 category. Analogous questions for people. Or if the number of twitter followers or facebook fans is actually correlated to any meaningful real life metric such as funding, valuation, revenue etc. Enough words wasted already. Moving on.

The Future of Work: A Vision

Posted on by

COVID-19 has forced us all to rethink many aspects of our lives. Job losses, stimulus checks, 0% interest rates and stock market turbulence dominate conversations when avoiding the more morbid topics of death and disease. Most large scale systems are struggling to deal with this pandemic in a coherent way and these struggles offer a rare lens into what we value as a society. As I work longer and longer hours, and fight with a variety of electronic screens to protect my cognitive real estate, I’ve been compelled to analyze the relationship between Work, Education and Value (Productivity).

“Never let a good crisis go waste.” Sometimes the music is not in the notes but in the spaces between them. Taking this pandemic as that silence between the notes, in a series of blog posts I examine deeply the relationship between the 3 pillars of the knowledge economy: Work, Education and Value.

In The Future of Work, but first a History, I briefly tour history to understand how common ideas about Work, Education & Value (Productivity) became ‘common’.

In Future of Work, but first Now and the Near Future I talk about why those ‘common’ ideas are not a reflection of reality any more (pandemic notwithstanding). I try to understand reality and where we are headed in the near future if nothing is done about fixing the gap between that reality and the common ideas/assumptions.

In this post, I first propose a hypothesis of the crux of the gap problem. And then, based on all of this, I present a vision for the future of work. Most importantly for the reader, in Future of Work: An Individual Perspective I describe a decision framework at the individual level about how to prepare for that future of work and the changing dynamic between Work, Education and Value (Productivity).

The Real Implications of ‘Winner-Takes-Most’

I’ll draw upon the shift that is happening in the world of computer vision to demonstrate my point. Deep learning, the latest buzzword in the field today really established its place at the forefront of machine learning in 2010. Deep learning approaches were being developed by a group of researchers since the 1990s. A culmination of advances in processing power like GPUs and large labeled datasets like ImageNet created the perfect arena. One in which the deep learning approach finally demonstrated that it could significantly out do traditional computer vision approaches. Quite rapidly, almost overnight, the expertise of traditional computer vision researchers & practitioners became outdated. While the original scientists and researchers worked in relative obscurity to develop these ideas, it was their students and 2nd generation PhD students who find themselves in the most advantageous position today. External viewers of this phenomenon (such as journalists, new students etc.) are processing this disruption in 3 ways:

right-time-right-place

This attitude supposes that all or (at the very least) the 2nd wave of early students in the field were in the right place at the right time to capture the deep learning wave.

rapture

This attitude places the original researchers, the 2nd and 3rd wave of researchers all in the same category. That category being one of iconoclastic geniuses more or less.

this-is-the-usual (technological-disruption)

This attitude looks at history and normalizes the deep learning wave as yet another example of the long history of technological disruptions.

Each of these perspectives captures and yet misses some important part of the whole picture. 

right-time-right-place attitude acknowledges that several veins of technological/science research require similar kinds of ingenuity, creativity, and technical brilliance. It tries to reduce the perceived gap in the merit of less successful but similar efforts carried out with the same intention and integrity.

this-is-the-usual tries to bring some perspective to the scale of upheaval a technological disruption brings about by normalizing it with history. However, it trivializes the fact that there is hardly ever anything “usual” about birthing something new. 

Both right-time-right-place and this-is-the-usual (technological disruption) attitudes do not give adequate recognition to the amount of risk, courage of conviction and effort it takes to believe/test new ideas. Especially in the face of powerful convention.

The rapture, hero-worship attitude in contrast to the other two gives ample recognition to the risk and courage of conviction it takes to birth new ideas in the face of excruciating uncertainty and occasional hardline naysayers. Yet it is probably off the mark the most in terms of creating a gap between the many similarly courageous people and their efforts.

These prevalent attitudes are not merely my observation. They are pretty much directly reflected in the common wage structure today. In 2014, two researchers from University of Bangkok and HBS conducted a survey about “How Much (More) Should CEOs Make?”. I encourage the reader to stare for a little while at the 2 graphs on a blog post about this study.

Briefly, in the United States, people estimated that CEOs make about 30 times more than an “unskilled” worker. Again most people think that should number should be 7.

30 and 7. Thats one chasm.

Now here’s the real deal, that number in reality is 350.

7 (ideal) –> 30 (estimated) –> 350 (actual)

That is an even bigger chasm.

A Hypothesis

A common framework that everyone tries to fit the changing equation of Work, Value/Productivity into is one of capitalism and socialism. But these frameworks are no longer sufficient and out of touch with reality. The crux of the problem is this:

All attempts to solve society’s problems/create value can be categorized into 3 buckets. 

  1. Successful attempt – the one that actually worked.
  2. Unsuccessful attempts – the ones that lacked one or more factors preventing them from becoming Successful attempts.
  3. No attempt
  4. <Fraud>

Pure socialism is the right-time-right-place/this-is-all-luck attitude taken to its extreme. It assumes that the gap between (1), (2) and (3) is near zero. Crucially it almost makes no distinction between unsuccessful attempts and those who make “No attempt”. Any person who has tried to create something of value (whether successful or not) cannot be true to themselves and accept this value system at the same time (especially those who have suffered the downsides of being in category (2)).

Pure capitalism is the rapture attitude taken to its extreme. It assumes that there is a huge gap between (1) and (2) and that it is justified. Specifically, it assumes that the reward gap between a successful approach and other similar approaches should exist (in the name of incentive/competition etc). Because of this crucially, it also places (2) and (3) at the same level (of near zero utility).

So here is my hypothesis for the crux of the problem:
Current Value systems underestimate how much a Successful attempt (1) is actually a function of Unsuccessful attempts(2). We need several different perspectives, attempts and approaches for the many unsolved problems facing the world today. This underestimation is another reason why current Education, Work, Value systems are so comfortable ignoring the human potential of women and large swathes of other under-represented demographics. Moreover and equally importantly, any value distribution framework will have to find a way to first differentiate between (1) and (2) and (3) and subsequently maintain those differences at “reasonable” levels.

True, real innovators mostly appreciate how deeply and extensively (1) is a function of (2). Geoffrey Hinton, a pioneer of deep learning, the field and example I have chosen to anchor this discussion, recently said, “The future depends on some graduate student who is deeply suspicious of everything I have said.” (*There is greatness and then there is transcendence). 

Grounded in this hypothesis of value creation, I lay out a vision of the Future of Work.

A Vision (for Work & Value)

About Work.

The lifecycle of change and disruption is going to become shorter and the rate of change faster in all walks of life. Thanks to automation, technological disruption and shifting labor markets, the gains from specialization are going to be exponentially high, massive. Leading to increasing winner-takes-most scenarios. But only until that specialization is relevant and only until that winner-takes-most advantage is not commoditized or regulated away. To become a winner in this winner-takes-most scenario and to reestablish a specialization in a new area, one has to take bets and spend time in areas that may generate value in the future. Areas for which it would not at all be clear or certain that spending time in it is going to lead to those substantial exponential gains that will make that time bet worth it. This is a vision for ‘Work’ part of the equation.

Now about ‘Value’.

In this changing scenario, Maslow’s Hierarchy, a concept devised in 1943, holds the key for how the perception of ‘Value’ is going to change. 

Maslow’s Hierarchy of Needs

Lower levels of Maslow’s hierarchy are classified as ‘Deficiency’ needs. Two very important characteristics of those needs are that the longer they are denied, the more a person becomes motivated to fulfill them. However, once they are fulfilled to reasonable levels, people are more motivated to transition to higher levels of the pyramid instead of fulfilling those needs in greater amounts or to a higher quantity.

The picture changes completely when an individual transitions to fulfilling a ‘Growth’ need such as creativity, self-expression etc. These needs may become stronger as they are engaged. They do not exist because of the lack of something but from the desire of an individual to reach higher levels of self actualization.

Herein lies the key to that future of work. As the rate of change of relevant skills becomes higher, one will have to take bigger and more frequent bets with their time to develop new skills and specializations. This cannot happen if one is not passionate, disciplined, talented etc. but most importantly this cannot happen if one’s work is not a good representation of one’s self, one’s aspired identity. The resilience and endurance required to keep meeting waves of change cannot be developed if the actions required to do them are not in sync with who you are and who you want to be.

As a corollary, traditional rat races (manifested in entrance exams, school grades bell curves, promotion bell curves, traditional careers and their ladders) are not only dehumanizing etc. but that they are already and will rapidly become irrelevant. Talk to most super achievers in their fields, whether politics, technology, art, law etc. and you will find that their hours only increase as they transition up the pyramid, not decrease. External observers (like journalists, retired past generation) focus only at the value society provides top performers in exchange for these hours in terms of money and labels it a success. However, this situation can only realistically be called a success if those hours contribute to that person’s idea of who they want to be. For example, I do believe this is one of the reasons top actresses have such a hard time making peace with who they are when they are at the top, irrespective of the pay. Because they realize that suddenly the sex symbol image that was helpful on the way to the top is no longer in sync with their needs for respect, prestige and control over pay in a highly sexually misogynistic society.(*1)

A thumb rule to discern these kinds of ‘top performers’ which has anecdotally worked for me is: People who’s success is out of sync with who they want to be, go to great lengths looking for external validation. They regularly engage in ego-matches and show-off contests to bridge the gap. Anecdotally, the thumb rule also seems to have a good prediction rate for sustained future success of that entity (person, group, company, community etc.).

This was the vision for future of work as it would look like to an individual. At the system level, institutional level I do think that the survival of a society will depend on how much we can enable people to do (2) so that as a group we have a real shot at (1). The key may really be in enabling a lot of people to take risks and experiment. It is to provide more people both the incentive and the safety net to experiment with new approaches, take risks and improve existing or outdated systems. And reward them if through their risks, they develop something of true value to society that solves a real problem. (Universal Basic Income is one financial vehicle that is being touted as a possible way to achieve this).

As political systems have demonstrated again and again, they rarely wake up in time to deliver opportune solutions to people’s problems. They usually have to be jolted into such a state with ugly processes like war, revolutions, violent protests and painful sacrifices of their victims. Since governmental politics is inter-connected with education, the wheels of motion are slow there too. As COVID-19 is already demonstrating, most universities in the developed world are having a hard time coming up with any logical, coherent approach for their modus operandi for the crisis. Not having invested in modernizing education, most are struggling with upgrading to online methods and are quite unabashedly passing on the buck to students. Therefore it is unlikely that large scale systems will change in any meaningful way to handle the already changed or changing nature of work and productivity.

There are some people who have the “good fortune” even today of living in that future of work. The upsides seem to be

“offices in the rich world’s capitals are filled with clever people working collaboratively.…The pleasure lies partly in flow, in the process of losing oneself in a puzzle with a solution on which other people depend…shaping high-quality, bespoke products from beginning to end…design, fashion, smooth and improve, filing the rough edges and polishing the words, the numbers, the code or whatever is our chosen material. At the end of the day we can sit back and admire our work – the completed article, the sealed deal, the functioning app – in the way that artisans once did, and those earning a middling wage in the sprawling service-sector no longer do….Work is a wonderful refuge (from emotional troubles).” 

Ryan Avent: “Why we work so hard?”

It looks great so far but in the face of lacking systemic support to step out of it, the same things that provide these advantages are also the reasons that over time will become points of weaknesses.

“…it does not allow us much time with newborn children or family members who are ill;…Or to develop hobbies, side-interests…it makes failure or error a more difficult, humiliating experience. Social life ceases to be a refuge from the indignities of work. The sincerity of relationships becomes questionable…they befriend their clients because they spend too much time with them…Stepping off the treadmill does not just mean accepting a different vision of one’s prospects with a different salary trajectory. It means upending one’s life entirely: changing locations, tumbling out of the community, losing one’s identity. Spending our leisure time with other professional strivers buttresses the notion that hard work is part of the good life and that the sacrifices it entails are those that a decent person makes. This is what a class with a strong sense of identity does: it effortlessly recasts the group’s distinguishing vices as virtues.”

Ryan Avent: “Why we work so hard?”

As this vision of work becomes a reality for more and more people, the revolution needed to prepare for it will have to be borne by the individual in their own personal lives. In the face of lacking leadership, support from the system, how do you prepare for this eventuality? This is what I talk about in my next article: The Future of Work: An Individual’s Perspective.

Epilogue

I must acknowledge Mahaveer Meghawat, my father, as many of the ideas in this post were sharpened over many many discussions with him.

I must also talk a little about ‘Education’. Of the 3 pillars from Work, Education & Value, Education is the least ‘valued’ today but probably the most fundamental to how individuals will choose Work in the future and perceive its Value. One school of thought posits that education reform is ‘THE’ key to looking at a brighter future. My favorite author on education reform in the Indian context, Prof. Arvind Kudchadker outlines a bold yet practical vision in his book ‘Creating a New Technological Institute’. This position is obviously true and relatable for almost every knowledge worker.

This 2014 essay on “Productivity And The Education Delusion” by Techcrunch editor Danny Crichton captures beautifully the many dilemmas this school of thought faces. (Totally worth reading).

What does an extreme form of this vision for the Future of Work look like? I encourage you to watch Chef’s Table episodes on Grant Achatz and Jeong Kwan in their entirety for a glimpse of this. (The episodes are an absolute treat to watch as an added benefit).

(*1)- I would refer the interested reader to the movie: ‘My Week with Marilyn’ and her wiki page for a real world example of this.

The Future of Work, but first Now & the Near Future

Posted on by

COVID-19 has forced us all to rethink many aspects of our lives. Job losses, stimulus checks, 0% interest rates and stock market turbulence dominate conversations when avoiding the more morbid topics of death and disease. Most large scale systems are struggling to deal with this pandemic in a coherent way and these struggles offer a rare lens into what we value as a society. As I work longer and longer hours, and fight with a variety of electronic screens to protect my cognitive real estate, I’ve been compelled to analyze the relationship between Work, Education and Value (Productivity).

“Never let a good crisis go waste.” Sometimes the music is not in the notes but in the spaces between them. Taking this pandemic as that silence between the notes, in a series of blog posts I examine deeply the relationship between the 3 pillars of the knowledge economy: Work, Education and Value.

In The Future of Work, but first a History , I briefly tour history to understand how common ideas about Work, Education & Value (Productivity) became ‘common’.

In this post, I talk about why those ‘common’ ideas are not a reflection of reality any more (pandemic notwithstanding). I try to understand reality and where we are headed in the near future if nothing is done about fixing the gap between that reality and the common ideas/assumptions.

In Future of Work: A Vision, I first propose a hypothesis of the crux of the gap problem. And then, based on all of this, I present a vision for the future of work. Most importantly for the reader, in Future of Work: An Individual Perspective I describe a decision framework at the individual level about how to prepare for that future of work and the changing dynamic between Work, Education and Value (Productivity).

The Treadmill (Now)

Technology, decentralization of work, globalization, these forces mean that increasingly most jobs are becoming winner-take-all or winner-take-most competitions.

“Banks and law firms amass extraordinary financial returns, directors and partners within those firms make colossal salaries, and the route to those coveted positions lies through years of round-the-clock work. Securing a place near the top of the income spectrum in such a firm, and remaining in it is a matter of constant struggle and competition”.

Ryan Avent: “Why we work so hard?”

Constant competition, a requirement of long work hours for years on end has several other social outcomes. Firstly such competition encourages concentration i.e. such people cluster together. Therefore same relatively high earning people drive up the cost competing for other resources such as real estate, professional services, education etc.

Finally, this forms a feedback loop. The cost to achieve the same lifestyle keeps going up and so does the need to work ever longer hours for increased professional achievement to meet those costs. It is as if you are on a constant hedonic treadmill. This phenomenon has excruciating social outcomes for the individual, something that has formed the fodder for many TV series and movies. (E.g. Suits, Scrubs etc.).

(Notice that the so called 40 hour work week concept is practically irrelevant for knowledge economy workers today). Career trajectories are rapidly changing from linear to exponential for most parts of the economy. As mentioned earlier, while low skilled workers are forced to accept ever smaller pay rises to stay in work, high skilled workers need to work longer and longer to maintain their place in these exponential trajectories. Because that next jump will mean exponentially more for the individual than not fighting for it and staying in the same place.

The pyramid to the top has been a reality since the dawn of human society. However the point I’m trying to make is that the reward gaps in the pyramid are widening. The gap will gradually become unsustainable for those participating in it its lower rungs socially, psychologically and eventually financially (unless they score the exponential jumps up the pyramid). The 2nd point I want to make is that this pyramid is becoming longer, wider and more common in all kinds of industries. We may not realize but almost every knowledge economy job will start or already resembles the set up of competitive sports athletes.

Near Future

As articulated brilliantly in Armando Fox’s amazon book review of Ryan Avent’s book on the topic, if the current setup of the economy continues:

“Future employment opportunities will likely satisfy at most 2 of the following 3 conditions (employment trilemma):

  1. High Productivity & Wages
  2. Resistant to Automation
  3. Potential to absorb large amounts of Labor

Example of (1) ❌, (2) ✅, (3) ✅

To see the dynamic, consider the solar-panel industry. Increased productivity in manufacturing solar panels has caused them to drop in cost, creating a large market for solar panel installers, a job resistant to automation (meets criteria 2 and 3). But that same increased productivity means most of the cost of acquiring solar is the installation labor, limiting wage growth for installers (fails criterion 1).

As another example, consider healthcare. As technology increases the productivity of (or automates) other aspects of care delivery, healthcare jobs will concentrate in non-automatable services requiring few skills besides bedside manner and the willingness to do basic and often unpleasant caregiver tasks.

Example of (1) ✅, (2) ✅, (3) ❌

Consider artisanally-produced goods, whose low productivity is part of their appeal (meets 1 and 2). But the market for them is limited to the small subset of people who can afford to buy them (fails 3).”

I encourage you to read that amazon book review in its entirety. Totally worth every second you will spend reading it.

The Missing Piece of the Puzzle

Jefferson:
Uh! Our debts are paid, I’m afraid
Don’t tax the South cuz we got it made in the shade
In Virginia, we plant seeds in the ground
We create. You just wanna move our money around
…….
Hamilton:
A civics lesson from a slaver. Hey neighbor
Your debts are paid cuz you don’t pay for labor
“We plant seeds in the South. We create.”
Yeah, keep ranting
We know who’s really doing the planting

Hamilton by Lin Manuel Miranda

So far the discussion has focused on technological disruption, globalization, automation, shifting labor markets etc. as the primary movers causing dynamic changes in the equation between work, education and productivity. But this is not the only reason why general assumptions about this equation are at odds with realities of actual value generation in society.

The other primary reason is a social one.

“What work is valuable to society?” This question has always been answered only by a select group of people who represent only a certain demographic of humanity. Historically that demographic has always been powerful men. (And they haven’t gotten it right enough, ever). (And they have extensive mechanisms to keep the output of that value concentrated with themselves). Most social unrest across the world has originated from groups who did not have a say in this answer trying to fight for one. As automation increasingly catches up with aspects of work and society which were considered “valuable” by only this particular demographic, we are in turn periodically forced to reconsider what the term “valuable” means actually.

For example one such dissonance is in the work that the different genders do. As I argue in my essay on the origins of Wonder Woman:

“The work that women do in shaping the future of humanity (literally by raising children) and safeguarding the health of families has no formal recognition in that ever enigmatic metric of honor, GDP”.

Aaksha Meghawat: “The Myth of Wonder Woman”

This is not a plea for more consideration. Creating structural disadvantages for participants of this important aspect of society that moves it forward makes all of us the worse for it. This value creation shows up in other ways. For example, 70% of top male earners in the US have a spouse who stays home. The women in these households are creating value which goes on to create and preserve family legacies but they usually have little ultimate control over the return of that value. Modern economic society has no framework to deal with the value provided to it in rearing nourished, psychologically stable, positively contributing humans. It is probably the reason school teachers earn abysmally low amounts as well.

If we wake up Keynes from his grave today and ask him to imagine a household where both husband and wife are high skilled workers, he would probably describe a scenario hypothesized in Ryan Avent’s essay:

“Each may opt to work 35 hours a week, sharing more of the housework, and ending up with both more money and more leisure.”

Further, regrettably 

“that didn’t happen. Rather, both are now more likely to work 60 hours a week and pay several people to care for the house and children”.

Ryan Avent: “Why we work so hard?”

The exponential trajectory nature of knowledge economy jobs has something to do with this. However the artificial social constructs which decide whose labor is valuable and whose is not irrespective of how they benefit society also has a lot to do with it. These artificial constructs are what prevent that “sharing of housework” (and are problematic to say the very least).

“Those most at risk of technological disruption are men in blue-collar jobs, many of whom reject taking less ‘masculine’ roles in fast-growing areas such as health care”.

Equipping people to stay ahead of technological change

After a lot of efforts from many people across countries and cultures, the only recognition of the labor needed in raising children is parental leave. Beyond this the so called economic wisdom offers nothing for such an important aspect of society. (That silence or inadequacy is taken over routinely by even more archaic power structures such as Religion who attempt to undermine abortion, Planned Parenthood and other medical organizations, sabotaging efforts offering women more control over this process).

And this is not a new problem.  As the fictional rap exchange between Jefferson & Hamilton demonstrates, denying people the true value of their labor and boosting economic metrics has happened time and again. For example consider this short recap of Manumission in the US, capturing how the rise and decline of labor intensive crops directly affected society’s appetite and rules for freeing slaves. This is Goodhart’s law playing out at its worst and it has real damning implications for our lives.  

Goodhart’s Law: When a measure becomes a target, it ceases to be a good measure.

Marilyn Strathern

The culmination of all this is a ‘winner-takes-most’ setup. What I talk about next is what this really means for us as a society. (And no, its not a trope on anti-trust laws, capitalism etc. I hope to offer something different, hopefully more useful).

The Future of Work, but first a History

Posted on by

COVID-19 has forced us all to rethink many aspects of our lives. Job losses, stimulus checks, 0% interest rates and stock market turbulence dominate conversations when avoiding the more morbid topics of death and disease. Most large scale systems are struggling to deal with this pandemic in a coherent way and these struggles offer a rare lens into what we value as a society. As I work longer and longer hours, and fight with a variety of electronic screens to protect my cognitive real estate, I’ve been compelled to analyze the relationship between Work, Education and Value (Productivity).

“Never let a good crisis go waste.” Sometimes the music is not in the notes but in the spaces between them. Taking this pandemic as that silence between the notes, in a series of blog posts I examine deeply the relationship between the 3 pillars of the knowledge economy: Work, Education and Value.

In this post, I briefly tour history to understand how common ideas about Work, Education & Value (Productivity) became ‘common’.

In Future of Work, but first Now and the Near Future I talk about why those ‘common’ ideas are not a reflection of reality any more (pandemic notwithstanding). I try to understand reality and where we are headed in the near future if nothing is done about fixing the gap between that reality and the common ideas/assumptions.

In Future of Work: A Vision, I first propose a hypothesis of the crux of the gap problem. And then, based on all of this, I present a vision for the future of work. Most importantly for the reader, in Future of Work: An Individual Perspective I describe a decision framework at the individual level about how to prepare for that future of work and the changing dynamic between Work, Education and Value (Productivity).

The Common Idea

We have been raised with the idea that Education will make “better” Work accessible to us. Our Education is a means to an end. This is not a judgement on the nature of education. It is an observation, almost an unstated fact. A crude blueprint of this system which happens to be in most people’s minds is:

“Our Education is a means to an end.”

Let me unpack that statement a little bit.

The “Means” need to transform us into a “productive” entity as defined by a Keynesian economic idea of “valuable”. 

The “End” is a realization of this “productivity” in terms of some form of “value”. 

“Value” is usually a combination of money and a lifestyle.

This idea did not just randomly take birth in society. It has very interesting origins which hold grave implications for the future and what we value as society. Prior to these ideas, most of the older generation passed on specialized knowledge to the younger generation via the system of apprenticeships.

The “means to an end” approach for education is an approximation of the commonly understood relation between Education, Work and Productivity (Value). However, in real life the relation between these 3 aspects is not at all obvious or clear. This common understanding of the relationship is often inconsistent with the rapidly changing needs of society, often structurally disadvantageous to some forms of real value generation/productivity over others.

So before jumping to a hypothesis of the future, I want to give you a brief tour of the origin of this idea.

History of Work: A Brief Tour

After most societies of the world had been sufficiently robbed of their ‘Produce’ by the European colonial powers, thanks to the “world” wars in Europe and subsequent independence wars in the rest of the world, modern political states were formed and there was a brief window to rethink what they valued as a society. A brief window to restructure the equation of work, value and education.

The conventional advice we receive today stem from ideas /assumptions developed about work and productivity in the few years following this period. The post-war/post-independence concept of work was largely shifting from work-for-survival to work-for-comfort for an increasing number of people. Not a significant proportion of society but an increasing one. More people were making the transition from ‘Basic Needs’ to upper levels of ‘Psychological Needs’ of Maslow’s Hierarchy.

Maslow's Hierarchy of Needs
Maslow’s Hierarchy of Needs

Several events in the western world set the tone for what conventional ideas surrounding work would soon look like. Labor unions began mounting political pressure in the 1870s for 8 hour-work-days.

Henry Ford re-organized the manufacturing process of his car factory by breaking it up into small, specialized tedious parts which meant that a worker needed only minimal training to contribute to the process. This resulted in huge efficiency gains. Subsequently he was one of the first industrialists who supported a 40 hour work week. It was more or less passed into law in the US in 1940s.

Similarly Germany set the retirement age at 65 in the late 1800s as a populist political move (why 65? Simply because people were not expected to live beyond that age at the time). The trend spread among other governments with the US government finally passing it into law in 1935.

Ryan Avent in his brilliant essay “Why we work so hard?” beautifully articulates that the post-war concept of work was a

“…means to an end…it was something you did to earn the money to pay for the important things in life….the working class had become a leisured class. Households saved money to buy a house and a car, to take holidays, to finance a retirement at ease…work was never supposed to be the centre of one’s life…”.

The industrial revolution was reaching its zenith with increased automation and innovation. There was an idea prevalent in American society at the time that the trend of automation might continue free-ing many more people from the need to work 40 hours per week.

“Keynes extrapolated in 1930 that a century hence, society might be so rich that the hours worked by each person could be cut to ten or 15 a week”.

This extrapolation might seem strange or illogical today but at the time

“productivity rose across the western world, hourly wages for typical workers kept rising and hours worked per week kept falling – to the mid-30s, by the 1970s”.

The extrapolation at the time seemed pretty reasonable, so what happened? Why aren’t you and I looking forward to a 20 hour work week (after graduation) with loads of free time to binge watch the whole of Netflix or work on a startup idea or produce and raise children or compose the next Hamilton?

While the 40 hour work week was on its way to becoming the norm in the western world, the supposed automation that was required to sustain the productivity levels of the typical worker did not actually happen. The productivity level fell out of sync with the automation needed to sustain it and a 40 hour work week was no longer viable. This paved the way for exporting manufacturing jobs to cheaper markets which did not have such labor-market restrictions, like China.

Wherever it was possible to simplify and atomize manufacturing steps, the efficiency gains in the short run resulted in higher wages for workers but also made them more susceptible to automation in the medium run. Another school of thought also points out that all of this was happening against the backdrop of a decline in the bargaining power of labor unions and the welfare state.

“Less-skilled workers found themselves forced to accept ever-smaller pay rises to stay in work. (They willingly or unwillingly worked fewer hours).”

I want to present another key point about automation which is often missed in such analyses. After having built machine learning products for over 5 years overseeing algorithms & pipelines to automate and enable business decisions, a fundamental idea of automation is that the moment you automate something well enough, the user’s expectation quickly adapts to the automation and demands the next bit of automated personalized tweak. The assumption that the user’s demand will stay the same or even grow at a steady pace wherever automation becomes available is a poor one.

Just as in my day job, I must develop models that “evolve” in the right direction with parameters which adapt to the situation, ironically the idea of a static “40” hour work week is practically irrelevant to the dynamic changes in the value creation chain today. Most high skill and low skill jobs require longer hour-work-weeks to justify the cost of the employee to the organization.

So while conventional ideas about work paint a picture of 40 hour, 5-day work weeks, 9am-5pm jobs, with expected retirement at age 65, all of this is becoming rapidly irrelevant. So what does it actually look like? What is the reality? What is it going to look like in the near future? This is what I talk about in The Future of Work, but first Now & the Near Future.

Assault

Posted on by

An unfeeling world
A cage
Could you escape?

Conform conform conform
To an older man’s importance
Could you escape?

Some want sex
Some want beauty
And some want duty

Could you escape?

Tie you in shackles of righteousness
If you tried
But who should you abide?

That people exist with no questions,
Sheep
That people exist unanswered,
f&#$!! (bleep)
That rebels exist,
wallowing, deep

Why must some enjoy their disgusting lust?
Why must some innocence go bust?
Why must you unfeeling living dead go on as if nothing happened thus?

You sad pieces of living s*?!t
Know this
Someday it will bury you
In a heartless pit
Destroy your meaningless bliss

With every breath
You sleep in your living death

Not even a semblance of worth
A weight upon centuries of earth

~Aaksha Meghawat

Dedicated to