Singh / Ramasubramanian / Shivam Building an Enterprise Chatbot

Work with Protected Enterprise Data Using Open Source Frameworks
1. ed
ISBN: 978-1-4842-5034-1
Verlag: Apress
Format: PDF
Kopierschutz: 1 - PDF Watermark

E-Book, Englisch, 399 Seiten

Explore the adoption of chatbots in business by focusing on the design, deployment, and continuous improvement of chatbots in a business, with a single use-case from the banking and insurance sector. This book starts by identifying the business processes in the banking and insurance industry. This involves data collection from sources such as conversations from customer service centers, online chats, emails, and other NLP sources. You'll then design the solution architecture of the chatbot. Once the architecture is framed, the author goes on to explain natural language understanding (NLU), natural language processing (NLP), and natural language generation (NLG) with examples. 
In the next sections, you'll design and implement the backend framework of a typical chatbot from scratch. You will also explore some popular open-source chatbot frameworks such as Dialogflow and LUIS. The authors then explain how you can integrate various third-party services and enterprise databases with the custom chatbot framework. In the final section, you'll discuss how to deploy the custom chatbot framework on the AWS cloud.
By the end of Building an Enterprise Chatbot, you will be able to design and develop an enterprise-ready conversational chatbot using an open source development platform to serve the end user.
What You Will LearnIdentify business processes where chatbots could be usedFocus on building a chatbot for one industry and one use-case rather than building a ubiquitous and generic chatbot Design the solution architecture for a chatbotIntegrate chatbots with internal data sources using APIsDiscover the differences between natural language understanding (NLU), natural language processing (NLP), and natural language generation (NLG) Work with deployment and continuous improvement through representational learning
Who This Book Is ForData scientists and enterprise architects who are currently looking to deploy chatbot solutions to their business.


Abhishek Singh is on a mission to profess the de facto language of this millennium, the numbers. He is on a journey to bring machines closer to humans, for a better and more beautiful world by generating opportunities with artificial intelligence and machine learning. He leads a team of data science professionals solving pressing problems in food security, cyber security, natural disasters, healthcare, and many more areas, all with the help of data and technology. Abhishek is in the process of bringing smart IoT devices to smaller cities in India so that people can leverage technology for the betterment of life. He has worked with colleagues from many parts of the United States, Europe, and Asia, and strives to work with more people from various backgrounds. In 7 years at big corporations, he has stress-tested the assets of U.S. banks at Deloitte, solved insurance pricing models at Prudential, and made telecom experiences easier for customers at Celcom, and core SaaS Data products at Probyto. He is now creating data science opportunities with his team of young minds. He actively participates in analytics-related thought leadership, authoring, public speaking, meetups, and training in data science. He is a staunch supporter of responsible use of AI to remove biases and fair use of AI for a better society. Abhishek completed his MBA from IIM Bangalore, a B.Tech. In Mathematics and Computing from IITGuwahati, and a PG Diploma in Cyber Law from NALSAR University, Hyderabad.
Karthik Ramasubramanian has over seven years of practice and leading Data Science and Business Analytics in Retail, FMCG, E-Commerce, Information Technology for a multi-national and two unicorn startups. A researcher and problem solver with a diverse set of experience in the data science lifecycle, starting from a data problem discovery to creating a data science prototype/product.

On the descriptive side of data science, designed, developed and spearheaded many A/B experiment frameworks for improving product features, conceptualized funnel analysis for understanding user interactions and identifying the friction points within a product, designing statistically robust metrics and visual dashboards. On the predictive side, developed intelligent chatbots which understand human-like interactions, customer segmentation models, recommendation systems, identifying medical specialization from a patient query for telemedicine, and many more.

He actively participates in analytics related thought leadership, authoring blogs & books, public speaking, meet-ups, and training & mentoring for Data Science. Karthik completed his M.Sc. in Theoretical Computer Science at PSG College of Technology, India, where he pioneered the application of machine learning, data mining, and fuzzy logic in his research work on the computer and network security.
Shrey Shivam extensive experience in leading the design, development, and delivery of solutions in the field of data engineering, stream analytics, machine learning, graph databases, and natural language processing. In his seven years of experience, he has worked with various conglomerates, startups, and big corporations and has gained relevant exposure to digital media, e-commerce, investment banking, insurance, and a suite of transaction-led marketplaces across music, food, lifestyle, news, legal and travel. He is a keen learner and is actively engaged in designing the next generation of systems powered by artificial intelligence-based analytical and predictive models. He has taken up various roles in product management, data analytics, digital growth, system architecture, and full stack engineering. In the era of rapid acceptance and adoption of new and emerging technologies, he believes in strong technical fundamentals and advocates continuous improvement through self-learning. Shrey is currently leading a team of machine learning & big data engineers across the US, Europe, and India to build robust and scalable big data pipelines to implement various statistical and predictive models. Shrey has completed his BTech in Information Technology from Cochin University of Science Technology, India.

Autoren/Hrsg.

Singh, Abhishek

Ramasubramanian, Karthik

Shivam, Shrey

Inhaltsverzeichnis

1;Table of Contents;5
2;About the Authors;12
3;About the Technical Reviewer;15
4;Acknowledgments;16
5;Introduction;18
6;Chapter 1: Processes in the Banking and Insurance Industries;20
6.1;Banking and Insurance Industries;20
6.2;A Customer-Centric Approach in Financial Services;25
6.3;Benefits from Chatbots for a Business;28
6.4;Chatbots in the Insurance Industry;29
6.4.1;Automated Underwriting;31
6.4.2;Instant Quotations;32
6.4.3;AI-Based Personalized Experience;32
6.4.4;Simplification of the Insurance Buying Process;32
6.4.5;Registering a Claim;32
6.4.6;Finding an Advisor;32
6.4.7;Answering General Queries;33
6.4.8;Policy Status;33
6.4.9;Instant Notifications;33
6.4.10;New Policy or Plan Suggestions;33
6.5;Conversational Chatbot Landscape;33
6.6;Summary;36
7;Chapter 2: Identifying the Sources of Data;38
7.1;Chatbot Conversations;38
7.1.1;General Conversations;39
7.1.2;Specific Conversations;39
7.2;Training Chatbots for Conversations;40
7.2.1;Self-Generated Data;41
7.2.2;Customer Interactions;42
7.2.2.1;Phone;43
7.2.2.2;Emails;43
7.2.2.3;Chat;43
7.2.2.4;Social Media;43
7.2.2.5;Customer Self-Service;44
7.2.2.6;Mobile;44
7.2.3;Customer Service Experts;44
7.2.4;Open Source Data;45
7.2.5;Crowdsourcing;45
7.3;Personal Data in Chatbots;46
7.4;Introduction to the General Data Protection Regulation (GDPR);48
7.4.1;Data Protected Under the GDPR;48
7.4.2;Data Protection Stakeholders;49
7.4.3;Customer Rights Under the GDPR;49
7.4.4;Chatbot Compliance to GDPR;51
7.5;Summary;52
8;Chapter 3: Chatbot Development Essentials;53
8.1;Customer Service-Centric Chatbots;53
8.1.1;Business Context;54
8.1.2;Policy Compliance;56
8.1.3;Security, Authentication, and Authorization;57
8.1.4;Accuracy of User Input Translation to Systems;59
8.2;Chatbot Development Approaches;60
8.2.1;Rules-Based Approach;61
8.2.1.1;Advantages of the Menu-Based Approach;62
8.2.1.2;Disadvantages of the Menu-Based Approach;63
8.2.2;AI-Based Approach;63
8.2.2.1;Advantages of the AI-Based Approach;64
8.2.2.2;Disadvantages of the AI-Based Approach;65
8.3;Conversational Flow;65
8.4;Key Terms in Chatbots;67
8.4.1;Utterance;67
8.4.2;Intent;68
8.4.3;Entity;68
8.4.4;Channel;69
8.4.5;Human Takeover;69
8.5;Use Case: 24x7 Insurance Agent;70
8.6;Summary;71
9;Chapter 4: Building a Chatbot Solution;72
9.1;Business Considerations;72
9.1.1;Chatbots vs. Apps;73
9.1.2;Growth of Messenger Applications;74
9.1.3;Direct Contact vs. Chat;74
9.2;Business Benefits of Chatbots;75
9.2.1;Cost Savings;75
9.2.2;Customer Experience;76
9.3;Success Metrics;77
9.3.1;Customer Satisfaction Index;77
9.3.2;Completion Rate;77
9.3.3;Bounce Rate;78
9.4;Managing Risks in Chatbots Service;78
9.4.1;Third-Party Channels;78
9.4.2;Impersonation;79
9.4.3;Personal Information;79
9.4.4;Confirmation Check;80
9.5;Generic Solution Architecture for Private Chatbots;80
9.5.1;Workflow Description;81
9.5.2;Key Features;84
9.5.3;Technology Stack;85
9.5.4;Maintenance;85
9.6;Summary;86
10;Chapter 5: Natural Language Processing, Understanding, and Generation;87
10.1;Chatbot Architecture;89
10.2;Popular Open Source NLP and NLU Tools;92
10.2.1;NLTK;93
10.2.2;spaCy;93
10.2.3;CoreNLP;95
10.2.4;gensim;96
10.2.5;TextBlob;97
10.2.6;fastText;98
10.3;Natural Language Processing;98
10.3.1;Processing Textual Data;99
10.3.1.1;Reading the CSV File;99
10.3.1.2;Sampling;100
10.3.1.3;Tokenization Using NLTK;101
10.3.2;Word Search Using Regex;101
10.3.3;Word Search Using the Exact Word;102
10.3.4;NLTK;103
10.3.4.1;Normalization Using NLTK;103
10.3.4.2;Noun Phrase Chunking Using Regular Expressions;104
10.3.4.3;Named Entity Recognition;108
10.3.5;spaCy;110
10.3.5.1;POS Tagging;110
10.3.5.2;Dependency Parsing;112
10.3.5.3;Dependency Tree;113
10.3.5.4;Chunking;115
10.3.5.5;Named Entity Recognition;116
10.3.5.6;Pattern-Based Search;118
10.3.5.7;Searching for Entity;120
10.3.5.8;Training a Custom NLP Model;120
10.3.6;CoreNLP;122
10.3.6.1;Tokenizing;123
10.3.6.2;Part-of-Speech Tagging;123
10.3.6.3;Named Entity Recognition;124
10.3.6.4;Constituency Parsing;124
10.3.6.5;Dependency Parsing;126
10.3.7;TextBlob;126
10.3.7.1;POS Tags and Noun Phrase;127
10.3.7.2;Spelling Correction;127
10.3.7.3;Machine Translation;128
10.3.8;Multilingual Text Processing;129
10.3.8.1;TextBlob for Translation;129
10.3.8.2;POS and Dependency Relations;129
10.3.8.3;Named Entity Recognition;131
10.3.8.4;Noun Phrases;132
10.4;Natural Language Understanding;132
10.4.1;Sentiment Analysis;133
10.4.1.1;Polarity;133
10.4.1.2;Subjectivity;134
10.4.2;Language Models;134
10.4.2.1;Word2Vec;135
10.4.2.2;Neural Network Architecture;137
10.4.2.3;Using the Word2Vec Pretrained Model;138
10.4.2.4;Performing Out-of-the-Box Tasks Using a Pretrained Model;142
10.4.2.4.1;Word Pair Similarity;144
10.4.2.4.2;Sentence Similarity;144
10.4.2.4.3;Arithmetic Operations;145
10.4.2.4.4;Odd Word Out;146
10.4.2.5;fastText Word Representation Model;147
10.4.3;Information Extraction Using OpenIE;149
10.4.4;Topic Modeling Using Latent Dirichlet Allocation;152
10.4.4.1;Collection of Documents;152
10.4.4.2;Loading Libraries and Defining Stopwords;153
10.4.4.3;Removing Common Words and Tokenizing;153
10.4.4.4;Removing Words That Appear Infrequently;153
10.4.4.5;Saving the Training Data as a Dictionary;154
10.4.4.6;Generating the Bag of Words;155
10.4.4.7;Training the Model Using LDA;155
10.5;Natural Language Generation;157
10.5.1;Markov Chain-Based Headline Generator;158
10.5.1.1;Loading the Library;159
10.5.1.2;Loading the File and Printing the Headlines;159
10.5.1.3;Building a Text Model Using Markovify;160
10.5.1.4;Generating Random Headlines;160
10.5.2;SimpleNLG;161
10.5.2.1;Loading the Library;162
10.5.2.2;Tense;162
10.5.2.3;Negation;163
10.5.2.4;Interrogative;163
10.5.2.5;Complements;164
10.5.2.6;Modifiers;164
10.5.2.7;Prepositional Phrases;165
10.5.2.8;Coordinated Clauses;165
10.5.2.9;Subordinate Clauses;166
10.5.2.10;Main Method;167
10.5.2.11;Printing the Output;167
10.5.3;Deep Learning Model for Text Generation;168
10.5.3.1;Loading the Library;171
10.5.3.2;Defining the Training Data;171
10.5.3.3;Data Preparation;172
10.5.3.4;Creating an RNN Architecture Using a LSTM Network;178
10.5.3.5;Defining the Generate Text Method;180
10.5.3.6;Training the RNN Model;181
10.5.3.7;Generating Text;184
10.6;Applications;184
10.6.1;Topic Modeling Using spaCy, NLTK, and gensim Libraries;185
10.6.1.1;Tokenizing and Cleaning the Text;186
10.6.1.2;Lemmatization;187
10.6.1.3;Preprocessing the Text Method for LDA;187
10.6.1.4;Reading the Training Data;188
10.6.1.5;Bag of Words;189
10.6.1.6;Training and Saving the Model;189
10.6.1.7;Predictions;190
10.6.2;Gender Identification;191
10.6.2.1;Loading the NLTK Library and Downloading the Names Corpus;191
10.6.2.2;Loading the Male and Female Names;192
10.6.2.3;Common Names;192
10.6.2.4;Extract Features;193
10.6.2.5;Randomly Splitting into Train and Test;193
10.6.2.6;Training the Model;194
10.6.2.7;Model Prediction;194
10.6.2.8;Model Accuracy;194
10.6.2.9;Most Informative Features;195
10.6.3;Document Classification;195
10.6.3.1;Loading Libraries;196
10.6.3.2;Reading the Dataset into the Categorized Corpus;196
10.6.3.3;Computing Word Frequency;197
10.6.3.4;Checking the Presence of Frequent Words;198
10.6.3.5;Training the Model;199
10.6.3.6;Most Informative Features;199
10.6.4;Intent Classification and Question Answering;200
10.6.4.1;Intent Classification;200
10.6.4.1.1;Setting tensorflow as the Back End;201
10.6.4.1.2;Building the Model;201
10.6.4.1.3;Classifying the Intent;203
10.6.4.2;Question Answering;204
10.6.4.2.1;Building the Model;204
10.6.4.2.2;Context and Question;204
10.6.4.2.3;Serving the DeepPavlov Model;206
10.7;Summary;207
11;Chapter 6: A Novel In-House Implementation of a Chatbot Framework;209
11.1;Introduction to IRIS;210
11.2;Intents, Slots, and Matchers;211
11.2.1;Intent Class;213
11.2.2;IntentMatcherService Class;214
11.2.3;The getIntent Method of the IntentMatcherService class;217
11.2.3.1;Intent Classification Service;220
11.2.3.2;General Query Intent;220
11.2.4;Matched Intent Class;221
11.2.5;Slot Class;223
11.3;IRIS Memory;228
11.3.1;Long- and Short-Term Sessions;228
11.3.1.1;Long-Term Attributes;228
11.3.1.2;Short-Term Attributes;229
11.3.2;The Session Class;229
11.4;Dialogues as Finite State Machines;235
11.4.1;State;237
11.4.2;Shields;238
11.4.3;Transition;239
11.4.4;State Machine;240
11.5;Building a Custom Chatbot for an Insurance Use Case;246
11.5.1;Creating the Intents;249
11.5.1.1;CustomNumericSlot;250
11.5.1.2;BooleanLiteralSlot;254
11.5.1.3;AccTypeSlot;255
11.5.1.4;IPinSlot;256
11.5.1.5;AlphaNumericSlot;257
11.5.2;IrisConfiguration;259
11.5.2.1;Adding States;261
11.5.2.2;Shields;263
11.5.2.2.1;DontHaveAccTypeShield;263
11.5.2.2.2;DontHaveQuoteDetailsShield;264
11.5.2.2.3;HaveAccTypeShield;265
11.5.2.2.4;HaveClaimIdShield;265
11.5.2.2.5;HaveQuoteDetailShield;266
11.5.2.3;Adding Execute Methods;266
11.5.2.3.1;Exit State;267
11.5.2.3.2;FindAdvisorState;267
11.5.2.3.3;GetAccountBalanceState;268
11.5.2.3.4;GetAccTypeState;269
11.5.2.3.5;GetClaimIdState;271
11.5.2.3.6;AskForQuote State;272
11.5.2.3.7;GetQuote State;275
11.5.2.3.8;Start State;278
11.5.2.3.9;GeneralQuery State;278
11.5.2.4;Adding State Transitions;281
11.5.3;Managing State;287
11.5.4;Exposing a REST Service;289
11.5.4.1;ConversationRequest;289
11.5.4.2;ConversationResponse;290
11.5.4.3;ConversationService;290
11.5.4.4;ConversationController;293
11.5.5;Adding a Service Endpoint;293
11.6;Summary;294
12;Chapter 7: Introduction to Microsoft Bot, RASA, and Google Dialogflow;296
12.1;Microsoft Bot Framework;296
12.1.1;Introduction to QnA Maker;297
12.1.2;Introduction to LUIS;305
12.2;Introduction to RASA;307
12.2.1;RASA Core;309
12.2.2;RASA NLU;310
12.3;Introduction to Dialogflow;311
12.4;Summary;316
13;Chapter 8: Chatbot Integration Mechanism;318
13.1;Integration with Third-Party APIs;318
13.1.1;Market Trends;319
13.1.2;Stock Prices;325
13.1.3;Weather Information;331
13.2;Connecting to an Enterprise Data Store;336
13.3;Integration Module;340
13.4;Demonstration of AskIris Chatbot in Facebook Messenger;353
13.4.1;Account Balance;353
13.4.2;Claim Status;354
13.4.3;Weather Today;355
13.4.4;Frequently Asked Questions;356
13.4.5;Context Switch Example;357
13.5;Summary;359
14;Chapter 9: Deployment and a Continuous Improvement Framework;360
14.1;Deployment to the Cloud;360
14.1.1;As a Stand-Alone Spring Boot JAR on AWS EC2;361
14.1.2;As a Docker Container on AWS EC2;364
14.1.3;As an ECS Service;367
14.2;Smart IRIS Alexa Skill Creation in Less Than 5 Minutes;372
14.3;Continuous Improvement Framework;383
14.3.1;Intent Confirmation (Double-Check);384
14.3.2;Predict Next Intent;386
14.3.3;A Human in the Loop;388
14.4;Summary;390
15;Index;391