Python Pickle: Problem in Modifying the program

[zak100]

Hi,

I am trying to modify the following Python Pickle program. It always starts by generating a ">". I have tried to use "print" statements but it is ignoring them. Kindly guide me how to modify the program:

import pickle
#trainer program
b=open('war&peace.txt')
text=[]
print("Testing the code")
for line in b:
    for word in line.split():
        text.append (word)
b.close()
textset=list(set(text))
follow={}
print("Testing")
print(text)
for l in range(len(textset)):
    working=[]
    check=textset[l]
    for w in range(len(text)-1):
        if check==text[w] and text[w][-1] not in '(),.?!':
            working.append(str(text[w+1]))
    follow[check]=working
a=open('lexicon-luke','wb')
pickle.dump(follow,a,2)
a.close()

Somebody please guide me.

 

Zulfi.

Hi,

I have got some idea. Please guide me what is the meaning of following code:

 

if check==text[w] and text[w][-1] not in '(),.?!':

I think its checking that 'check' is not equal to (),.?!. Any logic why its checking only 2 substrings of 'text'?

 

Zulfi.

 

[Ghideon]

3 hours ago, zak100 said:

I am trying to modify the following Python Pickle program. It always starts by generating a ">". I have tried to use "print" statements but it is ignoring them.

Which debugger have you tried? By generating a ">" do you mean an input prompt or that program prints ">" or something else? Which line in the program generated ">"? 

If the program you posted does not print anything, not even "Testing the code", it probably means the program failed or stopped before the first print statement.

 

3 hours ago, zak100 said:

Any logic why its checking only 2 substrings of 'text'?

If the code is not yours maybe you could provide a source where it comes from?

Also note that you open a local file 'war&peace.txt'. It may be easier to help if you post an example that does not rely on local data? Are you sure the program finds and is capable of opening war&peace.txt? Note that some systems may have issues with a file name containing "&". 

 

Edited November 29, 2020 by Ghideon
note on file systems and "&" character

[zak100]

Hi,

Thanks for your information. You are right,  ">" means input. Actually there is another file which runs first and that file contains the ">", symbol. The code from that file is:

import pickle,random
a=open('lexicon-luke','rb')
successorlist=pickle.load(a)
a.close()
def nextword(a):
    if a in successorlist:
        return random.choice(successorlist[a])
    else:
        return 'the'
speech=''
while speech!='quit':
    speech=input('>')
    s=random.choice(speech.split())
    response=''
    while True:
        neword=nextword(s)
        response+=' '+neword
        s=neword
        if neword[-1] in ',?!.':
            break
    print (response)

I am able to understand this code.

1a=open('lexicon-luke','rb')//opens a file
2successorlist=pickle.load(a)//retrieve the data stored in the pickle and assign it to successor
3a.close()//close that file

Then jump to the following code:

4speech=''//set speech to null value
5while speech!='quit'://loop runs until we type quit
 6   speech=input('>')//displaythe input prompt
  7  s=random.choice(speech.split())//??????
   8 response=''//create another variable response and assign it a null value
   9 while True://loop starts
    10    neword=nextword(s)//invoking the function nextword and pass it a random word retrieved from the input???
     11   response+=' '+neword//This will continue until we get a question mark, comma or a period
      12  s=neword
       13 if neword[-1] in ',?!.':
        14    break
    15print (response)

 

100if check==text[w] and text[w][-1] not in '(),.?!':

Please guide me the operation of line#7, line#10 and the earlier statement i.e. numbered as 100.

 

 

 

Zulf

[Ghideon]

21 hours ago, zak100 said:

You are right,  ">" means input. Actually there is another file which runs first and that file contains the ">", symbol.

Ok, that means my answer above does not apply. I answered regarding the code you actually posted together with the initial question.

Please answer: Which debugger have you tried? 

Please answer: Provide a source where the code comes from.
edit: found it*: https://stackoverflow.com/questions/55042/how-can-i-program-a-simple-chat-bot-ai/55053. end of edit.

Emphasis mine:

21 hours ago, zak100 said:

Actually there is another file which runs first and that file contains the ">", symbol. The code from that file is:

That code contains 

a=open('lexicon-luke','rb')

Which seems to rely on the other program that dumps the content into the file:

a=open('lexicon-luke','wb')
pickle.dump(follow,a,2)

Running the programs in the order you describe is probably not what the author intended. The "trainer" needs to run before the "bot".

I think you need to clarify the questions and what kind of help you require.

 

*) There are other locations, some full of spam stuff, I choose not to link to any those.

Edited November 30, 2020 by Ghideon
found source

[zak100]

Hi,

Thanks for your response.

Looks like my guesses to line#7 and line#10 are correct that's why you have not provided any answer to my questions.

 

Zulfi.

[Ghideon]

38 minutes ago, zak100 said:

Looks like my guesses to line#7 and line#10 are correct that's why you have not provided any answer to my questions.

I have not yet looked at those questions. I wait for your answers to my questions (the behavior of the code, and hence my answer, may depend on context). 

Edited December 1, 2020 by Ghideon
Grammar.

[zak100]

Hi,

Answers to your questions:

 

Which debugger have you tried?

PyCharm

By generating a ">" do you mean an input prompt or that program prints ">" or something else?

You are right, its input

Which line in the program generated ">"? 

This was in another file:

You are right, I discussed this problem here also and provided the link:

You also posted the link

<Also note that you open a local file 'war&peace.txt'. It may be easier to help if you post an example that does not rely on local data? Are you sure the program finds and is capable of opening war&peace.txt? Note that some systems may have issues with a file name containing "&". >

 

It opens the file crrectly.

 

I have gone ahead of this problem and its working fine but just developed only for one particular problem.

 

Zulfi.

 

[Ghideon]

1 hour ago, zak100 said:

I have gone ahead of this problem and its working fine but just developed only for one particular problem.

Ok! That means you have solved the questions? (line#7 and line#10) 

[zak100]

Hi,

Yes. I have completed one part. I now know how to modify the program. If I don't forget, I share my code and ask about your comment. Maybe today. 

Thanks a lot for keeping in touch.

Zulfi.

[zak100]

Hi,

 

Following is my output:

/home/zulfi/PycharmProjects/chatbot/venv/bin/python /home/zulfi/PycharmProjects/chatbot/MarkovBot.py
>professor not emailing
:professor not emailing
responseBot=  the following information: Your Course#,
>2216
:professor not emailing:2216
 the following information: Your Professor’s Name,
>ABCD
:professor not emailing:2216:ABCD
 the following information: Your Course’s Name,
>First Course
:professor not emailing:2216:ABCD:First Course
 the following information: Your R#,
>12345
:professor not emailing:2216:ABCD:First Course:12345
 the following information: Your Email address,
>zulfi@scienceForums.net

 

I was just in a hurry so I dont have greetings message. The first message is generated by Bot using Markov chain. The remaining messages are generated programatically. The code for triggering this particular method handling "professor email problem" is:

    NEmailCWPmust=["email"]
    NEmailCWPany1 = ["professor", "dr.", "dr", "teacher"]
    NEmailCWPany2 = ["contact","contacting", "reply", "replying", "response","responding", "answer", "answering", "emailing"]
    NEmailCWPany3 = ["no", "not"]
    if "email" in tSpeech:
        #print("Found1")
        if any(x in tSpeech for x in NEmailCWPany1):
            #print("Found2")
            if any(y in tSpeech for y in NEmailCWPany2):
                #print("Found3")
                if any(z in tSpeech for z in NEmailCWPany3):
                    #print("Before Entring prob_no_email bFistMesg=",bFirstMesg,"respBot=",responseBot,"totalResBot=",totalResponseBot,"newCnt=",newCnt)
                    if(not response_string_generated):
                       s = random.choice(speech.split())#Makov Code for generating firts message
                       responseBot=''
                       while True:
                          neword=nextword(s)
                          responseBot+=' '+neword
                          s=neword
                          if neword[-1] in ',?!.':
                             break
                       print("responseBot=", responseBot)#First Message generated by Markov Code
                       totalResponseBot = totalResponseBot + responseBot
                       list = prob_no_email_contact_with_professor(bFirstMesg, responseBot, totalResponseBot, newCnt)#generates the remaining messages

 

 

The contents of my training file are:

#training file starts

Kindly provide us following information: Your Professor’s Name, Your Course’s Name, Your Course#, Your R#, Your Email address.

Don’t worry, we try to contact with your professor but we need the following information:Your Professor’s Name, Your Course’s Name, Your Course#, Your R#, Your Email address.  

Not a problem, Kindly tell us: Your Professor’s Name, Your Course’s Name, Your Course#, Your R#, Your Email address.

Okay we would help you, we need: Your Professor’s Name, Your Course’s Name, Your Course#, Your R#, Your Email address.

#training file ends

Thanks all for helping me, God blesses you.

 

Zulfi.

Edited December 6, 2020 by zak100

[Ghideon]

10 hours ago, zak100 said:

I share my code and ask about your comment.

What kind of comment you are looking for? Since tSpeech is undefined I guess you shared a part of a program. 

As far as I can tell the the programs posted above provides random answers based on input and training data, potentially providing quite humorous responses (which may be the intention). It also illustrates typical issues with a naive approach towards NLP; that insight may be useful when studying more useful methods.

Edited December 6, 2020 by Ghideon
grammar

[zak100]

Hi,

<What kind of comment you are looking for? Since tSpeech is undefined I guess you shared a part of a program. >

If you look at the stack exchange code, it has Speech variable which takes input from the user but I am combining all the user inputs and storing it into "tSpeech".

<As far as I can tell the the programs posted above provides random answers based on input and training data, potentially providing quite humorous responses>

No I have tried to get rid of humorous type of stuff which you last time termed as "joke". You can see that the first time, I got :

"the following information: Your Course#,"

so bot is asking to get input from the user, we want to store all the user information and then a human would use this information to contact with the user. So I am systematically colectig the information. If the Markov Bot will not display something like the above, it would keep trying. Once it displays something like the above or like:

""the following information: Your Email address,"

its job would end and then the programming will come into play to retrieve the remaining information information about the user. So my approach is focussed, but you are right initially its random and this was necessary otherwise I won't be able to have AI part.

 

Okay can you tell me something better for this from your experience:

 NEmailCWPmust=["email"]
    NEmailCWPany1 = ["professor", "dr.", "dr", "teacher"]
    NEmailCWPany2 = ["contact","contacting", "reply", "replying", "response","responding", "answer", "answering", "emailing"]
    NEmailCWPany3 = ["no", "not"]
    if "email" in tSpeech:
        #print("Found1")
        if any(x in tSpeech for x in NEmailCWPany1):
            #print("Found2")
            if any(y in tSpeech for y in NEmailCWPany2):
                #print("Found3")
                if any(z in tSpeech for z in NEmailCWPany3):

 

This means I have several MarkovBots as said by PoetheProgrammerand I am directing the control to a specific MarkovBot using the above code but do you have a better strategy?

 

Zulfi.

 

 

 

 

 

Edited December 6, 2020 by zak100

[Ghideon]

Your description does not match the code provided so I can't really comment. Or maybe the crucial parts are somewhere in the code not shared? I can't tell.

 

1 hour ago, zak100 said:

I have tried to get rid of humorous type of stuff which you last time termed as "joke"

From the code posted above the following dialog seems possible:

Input: "Professor ABC is not replying to my email!"

Bot response: "Don’t worry"

It looks unintentionally humorous to me. (The output generation you have provided stops on "," as far as I can tell)

 

1 hour ago, zak100 said:

Okay can you tell me something better for this from your experience:

That is a large, complicated and interesting question. Answering the question would require many pages, I'll have to provide something short that allows for a discussion. Hardcoding a limited user input will not work in a real situation. I would use more powerful preprocessing; selecting a proper framework depends on many parameters not in the scope for this thread. I would use my experience to look at the architecture and overall design and not only parts of the coding. 

Edited December 6, 2020 by Ghideon

[zak100]

Hi,

I think you have talk some thing very useful but you forget to provide some examples. You have some terms in your answer but I can't understand unless you provide some examples if you have time. This would be useful for others also. This is what I need:

I'll have to provide something short that allows for a discussion.  (please provide an example, what is short you are talking about?)

 

Hardcoding a limited user input will not work in a real situation. (what else you want? do you want more hard coded data or do you want deviation from hard coding?)

 I would use more powerful preprocessing; (for example?)

selecting a proper framework depends on many parameters not in the scope for this thread. (please provide more details, what you mean by framework? Are you talking about ML (machine learning), but every body is doing that)

 I would use my experience to look at the architecture (again please elaborate what you mean by this?)

and overall design (it would be good if you tell us more about it) and

not only parts of the coding. (sorry please  provide more description).

If you discuss more this would be very useful for improving this Markov ChatBot.

 

God blesses you.

Zulfi. 

[Ghideon]

If you use the quote function on this forum it will be easier to understand what you mean. Imagine some other member wanting to participate and trying to 

8 hours ago, zak100 said:

I think you have talk some thing very useful but you forget to provide some examples.

I did not forget, I choose to not to include any examples. It is not clear what kind of examples that would be appropriate and that would allow for fruitful discussions. I get from the questions in your reply that you did not seem to understand my comments. No problem, I'll try another approach:

What is your purpose and the ambitions with the code and questions?
A: Is your goal is to understand the basics of a Markov chain and generate random texts with a simple program for personal use*?
or
B: Do you intend to actually use the program in a production environment where users will expect useful responses and help to solve real tasks?

(Or some other goal? A and B are examples)

Reason for asking: The examples I have in mind depend heavily on what the goal is, A, B or something else.

 

*) like a homework.

Edited December 7, 2020 by Ghideon

[zak100]

Hi,

<What is your purpose and the ambitions with the code and questions?
A: Is your goal is to understand the basics of a Markov chain and generate random texts with a simple program for personal use*?
or
B: Do you intend to actually use the program in a production environment where users will expect useful responses and help to solve real tasks?

(Or some other goal? A and B are examples)>

As far as your reply (i.e. part A) is concerned, I am able to understand  Markov  chain to some extent.

I had to develop it as a project but one of the objective of the project is to use the project for departmental purpose but I think I am out of that race because my prototype  focussed only on one case. But the advantage of my technique is that it is not wholy based on AI which gives odd and unpredictable results. 

But forunately I showed it to my friend and he says that it has high demands  for physians because they look for something like a pager and my application is similar to that. So kindly show me both the aspects as my implementation would depend on simplicity and my understanding. So without seeing both the approaches, I can't say which one is eay for me to implement.

 

Thanks for your time. 

 

God blesses you.

 

Zulfi.

 

[Ghideon]

6 hours ago, zak100 said:

So kindly show me both the aspects as my implementation would depend on simplicity and my understanding. So without seeing both the approaches, I can't say which one is eay for me to implement.

Thanks for the reply. Your request seems outside the scope of the forums and seem to require more time than I wish to invest in this specific topic. Maybe you could get a mentor familiar with the project and its objective?  Maybe you could post more specific questions?

Here is a quick example to highlight. Let's assume I would be mentoring or hired as a consultant / architect. I would ask very different questions when applying my experience in case A) and B) in my previous post. Note: there is no need to try to answer the questions below, they are examples and I have no use for the answers.

A) Why is there "," in the training data when "," is considered end of sentence? That seems to be a bug or a miscommunication regarding the design.

B) Is there a ballpark number for the project budget? Do you have an organization in mind regarding the maintenance and re-training of the bot? Do you have architectural or technical constraints such as usage of certain vendors? Have you sorted out any legal issues such as GDPR or Privacy Shield? If not, should we arrange a workshop with your legal department? Do you have a test environment or tech-lab or similar where we can do some quick prototyping and get stakeholder and end user feedback?

 

6 hours ago, zak100 said:

But the advantage of my technique is that it is not wholy based on AI which gives odd and unpredictable results. 

Yes the code to have presented is intended to give odd and unpredictable results. How is that an advantage? 

 

6 hours ago, zak100 said:

But forunately I showed it to my friend and he says that it has high demands  for physians because they look for something like a pager and my application is similar to that.

Good to hear that there is a demand. I do not see a the connection between physicians' demands and the random text generation program you have posted.

[zak100]

Hi,

I think you have provided a valid point related to the budget. It requires a team to develop this sort of project. Also I have to interact this python project with php. This is another problem. Okay thanks for your reply.

 

Zulfi.