Artificial Intelligence and Machine Learning in the world of Agile
By: Luba S., Director of Agile Transformation
A good friend and mentor of mine, mentioned casually over lunch the topic of Artificial Intelligence (AI). While he was searching for a new opportunity and had some free time, he took a six week course on this popular topic. This made a huge impact on him and he spoke excitedly about the future of AI. This is a person who had a huge impact on my career choices and I made a mental note to look into it. Little did I know that within a few weeks, I would be reading Artificial Intelligence articles the same way many read Mysteries of Agatha Christie (you know, staying up late because you don’t want to put the book away?). It is a truly fascinating topic and I believe Artificial Intelligence must become part of strategic planning for all organizations.
How it all began
That same night after a nice lunch with my friend, my Facebook feed showed an advertisement for MIT Management Executive Education. They were offering a six week course on Artificial Intelligence. Was it a coincidence? Was it a sign? Was it magic? Did Facebook get so sophisticated that it could actually read your mind? Scary! Well, the common sense and the little knowledge that I had about AI took over and I was pretty sure it had something to do with me searching the topic on Google earlier. Maybe there was no connection between the two, but long story short, the timing of the course turned out to be perfect and after a little more research, I signed up and my journey began!
1 week into the course…
I was excited. Week one of the course was pretty simple: write a quick intro, meet people in your class, setup your profile and make sure your Online Campus works as expected. Week two got real. It turned philosophical. “What does intelligence mean to you?” was the first question. After many postings of opinions, we got the official definition, along with hours of additional material on the topic. Part of the first weeks homework was to write a quick essay about your organization and the current use of Artificial Intelligence. A few weeks into the course, one of my co-workers posted a very interesting and relevant blog on this topic, but at the point of first assignment, I knew very little and so I turned to my friends at work.
Jerry C., Software Director of Engineering on the Analytics team and Tyler L., Director of Application Architecture allowed me to pick their brain, I was able to submit the following,
About the Organization: The organization I work for, Cengage, provides technology and content for higher education, K-12, library markets, as well as career schools. It is a book publishing company going through a digital transformation.
Recently, we announced a new model for students, where they can take advantage of all the content for a low subscription cost. Although we are trying to disrupt the industry and provide an affordable access to unlimited content, we don’t rely on cost leadership (which means no frills). Regardless of the fact that we sell solutions to instructors and institutions, we focus primarily on our students, their learning objectives and their achievements.
Current state: We have a team that collects unique data points and generates reports answering questions such as “What content is accessed more frequently?”, “How many users login?” or “How many users login more than once per week? This is called Descriptive Analytics.
There are a few other types of analytics, such as Predictive and Prescriptive. As you go through the initiatives below, think about whether they are descriptive, predictive or prescriptive.
A major challenge organizations face is to identify and collect data consistently. Some teams at Cengage work on different hypothesis e.g. “Is content X effective?”, “Do students engage with multiple pieces of courseware per session?”. Having multiple platforms and erratic data collections streams from numerous sources isn’t consistent. The team that gathers all the data also handles the data clean up. We do still have a long way before we are able to test out our hypothesis, determine probable outcomes and predict measurements, but we are definitely on the right track!
The maturity of the organization will come in phases. We have to first implement consistent event tracking (logins, learning path customization, content launches, assessment submittals, assessment scores…). When we create content and assessments, we need to have analytics on top of our minds, just like accessibility. Building a profile on every student, so we can understand their interests, study habits would be essential in applying AI, but we need to pay close attention to how we handle it to make sure we are compliant with the General Data Protection Regulation (GDPR), which is designed to enable individuals to better control their personal data.
Side Note: It was mentioned in some of the reading material that due to the GDPR regulations in Europe and non existing in China, China might soon get much further in their development of AI. Not to deviate too much, but it is worth mentioning that in an article on AI regulation, it says that in order to be successful, AI regulations needs to be international. “If it’s not, we will be left with a messy patchwork of different rules in different countries that will be complicated (and expensive) for AI designers to navigate. “
Back to AI at Cengage. As mentioned earlier in this blog and in detail in my co-workers blog, a lot of focus at Cengage right now is on the Student. As part of the course, the final project is to come up with a roadmap of different AI initiatives that would be closely aligned to the business and/or IT overall strategy. As I was learning about different aspects of AI and different applications of it, I was thinking about what initiatives I should propose. We had to answer questions such as “who from the organization, would we need to involve?”, “what technical challenges might we face?”, etc.
After a lot of consideration, I decided to primarily focus on the initiatives that are within my area of expertise, which is Scrum Framework, as well as Agile Values and Principles. It is worth mentioning that, to my surprise, it turned out there isn’t that much information out there combining AI and Agile. When I asked a question on LinkedIn (to a large community of Agile Coaches) in regards to who experimented with AI, I received a bunch of likes but no actual comments. Agile and Artificial Intelligence are two very popular topics on their own these days. Digging into combining them both seemed challenging and intriguing, so I am going for it!
2 weeks into the course…first set of proposed AI initiatives
- Improving facilitation of Scrum Ceremonies using Cogito’s real-time emotional intelligence software, not just for facilitators, who are expected to have very high emotional intelligence, but the rest of the participants. If every participant had software to coach them through conversations, we would improve effectiveness and efficiency of our meetings tremendously.
- Using AI to encourage team member to pick up tasks to learn new skills needed on the team, rather than always working on tickets they are comfortable with. A team member in Scrum ideally is capable to pick up any work item from the top (most valued) of the backlog.
- Using AI and ML to predict the Success of a Sprint in Scrum. Not an easy task but we do have data, such as status of work items in Jira, total work left to do, peoples capacity and available skills. We would need to categorize skills and tag each work item, as well as skills on the team. We could take it a step further and predict success of longer releases.
- Along the same lines, during a Retrospective, a lot of feedback is given in regards to process improvement. Often (although facilitators try to handle it), there is finger pointing as to why something hasn’t gone well. If we could capture what is being said and pay attention to words like “waiting for QA” or “Jira was down” or whatever other reasons, we might be able to see patterns
Optional…only if the VP of Customer Support decides to lead this effort…
- Customer Support support is the area of high interest to me because the success and happiness/satisfaction of our students heavy depends on our customer support agents. I would also propose using Cogito’s real-time emotional intelligence software, especially since this is the area it was specifically designed for.
WARNING: These were just random ideas that would require additional investigation, feedback from experts, as well as experimentation.
3 weeks into the course…second set of proposed AI initiatives
- I work with a team of Agile Coaches and we organize a lot of meetings. We also organize meetings sometimes with over a hundred of attendees. I would experiment with the x.ai
- My company periodically goes through acquisition process and I believe using software that is pre-trained with a set of clauses to watch out for in the contracts could be very beneficial to our legal team.
- Quill software seems like something totally applicable in our world. In the following video, at about minute 6, it talks about the software giving feedback to students taking online classes on how they are doing and what they should study to improve, https://vimeo.com/130711623
- The demo below, which shows one aspect of NLP, where you can run some txt through the ML model and it spits out a sentiment, along with examples of how they categorized emotion…I was wondering if it could be used to scan a team slack channel to assess their happiness and job satisfaction, https://natural-language-understanding-demo.ng.bluemix.net/?cm_mc_uid=61913652150015064305656&cm_mc_sid_50200000=1508796017&cm_mc_sid_52640000=1508796017
- Along the same lines, we take lots of Survey data. Running quick sentiment analysis on it will allow us to focus on the negative or angry sentiment versus the positive . It allows you to quickly decide which of 10,000 surveys to read first.
- https://aws.amazon.com/deeplens/community-projects/DEAR/
I would love to use the Deeplens to capture faces during a meeting to see how well the meeting has gone and what was the level of engagement/boredom/excitement
From the list above, #1, #4 #5 and #6 are something that falls within my direct responsibility, so I would primarily focus on those.
I would learn more about #2 and #3, so I could recommend it to others, but I wouldn’t directly be involved.
4 weeks into the course…third set of proposed AI initiatives
- To help with on boarding, we could use a robot programmed with frequently asked questions about the company. Overtime, it would learn more and more. It is a repeatable task, so it would work.
- Ability to get a self-driving car to get you from your current location, drive you to your conference room and then going back for recharge until next call could be really cool and very useful. It is easy to get lost in our current building.
- Often people are either busy in meetings or just too much into their work when the hunger takes over. Being able to ask a robot to bring you food (call via an app), and get food from a pre-packed vending machine would come in handy.
- As a facilitator of certain recurring meetings that are part of Scrum Framework, you often find yourself reminding folks that it is time to join the meeting. We have reminders via slack as well, but would be helpful to have a pre-programmed robot to announce in a friendly manner, so you as a facilitator, don’t come across as nag.
Data Science
During the week 4 of the MIT course , General Assembly happened to be hosting a one hour course on “Data Science”. Not only they managed to give a very interesting overview of what it take to be a Data Scientist, they covered the set of questions you will find helpful asking yourself when you are given a data related task…
Identify the problem:
What are you trying to answer?
Acquire the data :
Is the data you need available?
How is it stored?
Is it public or proprietary ?
Can it be supplemented?
What tools will you need to import it ?
What tools will you need to work with it?
Parsing :
What documentation is available?
Were you able to import the data?
What did you learn from initial exploratory analysis?
Is the data, in fact, sufficient?
How much munging will it require?
Mine (Wrangling/munging/mining – different names…80%!of work occurred here):
What sampling methodology will you use?
What secondary features need to be derived?
Refine:
What new trends appear?
Any notable outliers?
Notable results from applying derived features?
Are you documenting your thought process and findings?
Model:
What model or models are most appropriate for the data and the problem?
If the data in the appropriate format for the desired models?
How did the initial model perform?
Any symptoms of over-fitting?
Present: what narrative do I want to tell?
What assumptions we are making?
What inherent limitations should be disclosed?
Was success criteria met?
Instructor made a very interesting comment in regards to defining success criteria. He said that unless you write it down and can refer back to it, it is very easy to deviate from the original question you were trying to answer and justify answering something else marking it as success. What you answered could also be extremely useful but staying true to the success as it was defined originally takes discipline. I will make sure to keep that in mind when drafting set of initiatives for the future of AI at Cengage Learning!
5 weeks into the course…forth set of proposed AI initiatives
A lot of the discussions about Ethics this week…
Most of the initiatives I proposed over the course of this class had to do with having AI as a peer to an Agile Coach, Engineering Manager and in the case of a robot helping with on boarding, a peer to HR. I don’t see any risk of unemployment. I wouldn’t trust a robot (just yet) doing performance reviews or deciding whether to give someone a bonus or a raise. I can see though how if done smart, a robot could be conducting exit interviews gathering insights on different reasons why people are leaving…
Another example of where a human cannot be completely replaced is in the area of analysis itself. Here is great quote from our Data Scientist, Doug Needham, who lives in the trenches of Data Science, Artificial Intelligence and Machine Learning, “AI/ML are great tools, but a Human needs to actually look at the data (at least initially), then work to refine the statistical model that is driving the actual process of Analysis. This human required effort needs to be taken into account, scheduled, and be treated as a first class citizen as part of the process. “Analytics” is not a silver bullet. We as an industry are still maturing, and the days of “fire and forget” are a long way away”.
6 week into the course – The Future of AI at Cengage
My organization has been going through a digital transformation for at least three years now. There has been a lot of focus on behaviors, values and certain principles that would allow people to work together more effective and more efficient. “We are one team”, meaning we don’t throw stuff over the wall, your problem is OUR problem, you are not done until the SOLUTION is done, not just your part…We have a team of 12 Agile Coaches in the company whose primary focus is on how people work together, how they communicate with each other. Respect and empowerment is valued a lot more in my organization than command and control. There is a lot of investment in the leadership skills among managers, directors and VPs to ensure the desired behaviors don’t fall out of the window the second something doesn’t go according to a plan. I think we are in good shape when it comes to soft skills in my organization, although there is room for improvement in that area for most individuals, including myself :).
We have a lot of Data Science, Artificial Intelligence and Machine Learning enthusiasts beside the few people whose primary job it is to improve students outcomes through DS, AI and ML. What we need to do in order to achieve competitive advantage is the start tagging and collecting the data in a consistent manner. It is already on many right people’s radars, so we are heading towards making progress in the AI space. When I asked Doug Needham (whose recent project was to come up with a method for classifying both students and courses that consistently shows a strong relationship between patterns of platform feature usage, and range of final score) to review this blog, one of the comments he made was worth quoting here, “It seems to me sometimes that people think AI/ML/DS is a magic bullet. Some of it appears to be magic, but the magic is really hard work, research, lots of study, and plenty of late nights :)”. It is important to keep that in mind when selecting initiatives to implement in the organization.
As the final homework for this course, we had to aggregate all that we had learned and proposed in the previous homework and come up with a roadmap of AI initiatives. Although many of the AI initiatives are super cool, we always have to think about return on investment. Do we REALLY need a robot who would fetch you a beer or bring you lunch from a vending machine?. I don’t think so :). In fact, I would argue that it is good for anyone who has been sitting for a long time to get up and take that walk. On the other hand, having a Slack bot remind an employee that it might be a good time to get up and take a break could go a long way!
In Conclusion
One of the main learning outcomes of this course was to make sure that if you are afraid of AI and think it will soon take over the world, you calm down and realize it is not happening :). Well, I am definitely not afraid of the robots and love the promise of more interesting jobs added and boring tasks taken care by computers. Taking this course allowed me not only to learn about many aspects of AI, including ethics, it allowed me to have some great conversations with my co-workers, learn about the current state of my organization and come up with some initiatives on how AI can be implemented in my organization as a peer to an Agile Coach or Engineering Manager, NOT A REPLACEMENT of that role. It also confirmed my believe that soft skills and human touch cannot be easily replaced by a machine.
