Running Multiple thread

[zak100]

Hi,

I have two sorting programs. I want to concurrently execute them. I have written the following code using help from this forum:

        pid = os.getpid()  # current process
        print("start thread")
        psutil.Process(pid).cpu_percent(interval=0)
        th1 = threading.Thread(target=BubbleSort, args=(k, arr2))
        th2 = threading.Thread(target=selectionSort, args=(k, arr2))
        th1.start()
        th2.start()
        while th1.is_alive():
            print("CPU usage:" + str(psutil.Process(pid).cpu_percent(interval=1)))
        print("thread1 ended")
        while th2.is_alive():
            print("CPU usage:" + str(psutil.Process(pid).cpu_percent(interval=1)))
        print("thread2 ended")
        print("all done")

 

Following is the output. This clearly shows that the execution of two threads is overlapping . Initially thread 1 starts prints the sorted list and then control goes to thread2 which prints first 4 numbers and then the control again goes to thread1 which prints the message " thread 1 ended". I want to run the thread 1 to completion and then to start the thread2. In Java we have "synchronize" command. Please guide me how to achieve mutual exclusion in Python.

 

The complete code is:

import random
import psutil
import os
import threading

class SSSort:
    arr = []  # class var access by className.arr

    def __init__(self, n):
        self.n = n

    def generate1000_5digit_RandomNum(self):
        for i in range(self.n):
            num = random.randint(10000, 99999)
            SSSort.arr.append(num)

        for j in range(self.n):
            print("random {0}".format(SSSort.arr[j]))

    def cpu_util_info(self):  # make sure different than posted
        k = self.n
        arr2 = SSSort.arr[:]

        def find_the_min_element(k, i, arr2):
           min = i
           for j in range(i + 1, k):
               if (arr2[j] < arr2[min]):
                   min = j
           return min

        def BubbleSort(k, arr2):
            for i in arr2:
                for j in range(len(arr2) - 1):
                    if (arr2[j] > arr2[j + 1]):
                         temp = arr2[j]
                         arr2[j] = arr2[j + 1]
                         arr2[j + 1] = temp

            for i in enumerate(arr2):
                print(i)

        def find_the_min_element(k, i, arr2):
           min = i
           for j in range(i+1, k):
              if (arr2[j] < arr2[min]):
                 min = j
           return min

        def selectionSort(k, arr2):
            for i in range(self.n):
                min = find_the_min_element(k, i, arr2)
                # swap min and ith element of array
                temp = arr2[min]
                arr2[min]= arr2[i]
                arr2[i] =temp

            for i in enumerate(arr2):
                print(i)


        pid = os.getpid()  # current process
        print("start thread")
        psutil.Process(pid).cpu_percent(interval=0)
        th1 = threading.Thread(target=BubbleSort, args=(k, arr2))
        th2 = threading.Thread(target=selectionSort, args=(k, arr2))
        th1.start()
        th2.start()
        while th1.is_alive():
            print("CPU usage:" + str(psutil.Process(pid).cpu_percent(interval=1)))
        print("thread1 ended")
        while th2.is_alive():
            print("CPU usage:" + str(psutil.Process(pid).cpu_percent(interval=1)))
        print("thread2 ended")
        print("all done")



    def main(self):
         self.generate1000_5digit_RandomNum()
         self.cpu_util_info()
        

if __name__ == "__main__":
    objSSSort = SSSort(10)
    objSSSort.main()

Zulfi.

Quote

start thread
(0, 50735)
(1, 52575)
(2, 58431)
(3, 63366)
(4, 75071)
(5, 77648)
(6, 79961)
(7, 83715)
(8, 89010)
(9, 93857)
(0, 50735)
(1, 52575)
(2, 58431)
(3, 63366)
thread1 ended
(4, 75071)
(5, 77648)
(6, 79961)
(7, 83715)
(8, 89010)
(9, 93857)
CPU usage:0.0
thread2 ended
all done
memory usage=  15552512
disk usage=  sdiskusage(total=982900588544, used=141445500928, free=791455002624, percent=15.2)
virtual mem=  12050952192
rss=  15552512

Somebody please guide me, how to solve this problem.

 

Zulfi.

Edited March 26, 2020 by zak100

[Ghideon]

1 hour ago, zak100 said:

Following is the output. This clearly shows that the execution of two threads is overlapping . 

Having the threads do work in parallel is kind of the point having multiple threads.

1 hour ago, zak100 said:

I want to run the thread 1 to completion and then to start the thread2

Then make the program single threaded? Less room fo errors, easier to debug etc?
Anyway, a basic way to fulfil your requirements (two threads that does never run in parallel) is to delay the start of the second thread until the first thread is complete:

        th1 = threading.Thread(target=BubbleSort, args=(k, arr2))
        th1.start()
        while th1.is_alive():
            print("CPU usage:" + str(psutil.Process(pid).cpu_percent(interval=1)))
        print("thread1 ended")             
        th2 = threading.Thread(target=selectionSort, args=(k, arr2))
        th2.start()
        while th2.is_alive():
            print("CPU usage:" + str(psutil.Process(pid).cpu_percent(interval=1)))
        print("thread2 ended")

 

 

1 hour ago, zak100 said:

Please guide me how to achieve mutual exclusion in Python.

I'm not sure this forum is the best place for me to write a long set of posts about multi threading required to cover this topic in enough detail, a blog would probably be better. Have you tried some other online sources or books?

 

[Sensei]

2 hours ago, zak100 said:

I want to run the thread 1 to completion and then to start the thread2. In Java we have "synchronize" command. Please guide me how to achieve mutual exclusion in Python.

Read about lock, rlock, condition, event objects:

https://docs.python.org/3/library/threading.html#lock-objects

 

e.g. make event object in the main thread, pass to child thread and set it at the end of execution. In the main thread wait till event is triggered.

54 minutes ago, Ghideon said:

Then make the program single threaded? Less room fo errors, easier to debug etc?

In majority of languages and OSes during writing app with GUI you should delay any long time (>50-100 miliseconds) taking operation to separate threads, and not block UI/the main thread, even if it is some basic stuff.

In Java on Android it is even forbidden to call e.g. network or database function/method on UI thread and OS immediately throws an exception.

[zak100]

Hi,

Thanks for your response.

Okay let's suppose we want to work on two case:

1) Concurrent Execution, may produce jumbled output

2) Running two threads accessing the data at the same time but their output is separate, i.e. we can avoid the jumbled output which I am seeing right now

Also for the both the above cases how can I get a combined CPU Usage ?

<Have you tried some other online sources or books? . >

Not in this case but for CPU usage I did a search. But I would do it now. Thanks.

Zulfi.

Edited March 26, 2020 by zak100

[Sensei]

1 minute ago, zak100 said:

How can I get a combined CPU Usage in the above case?

e.g. while loop checking whether any of two threads is alife (rather than just for one). Use logical or operator.

 

[Ghideon]

15 minutes ago, Sensei said:

In majority of languages and OSes during writing app with GUI you should delay any long time (>50-100 miliseconds) taking operation to separate threads, and not block UI/the main thread, even if it is some basic stuff.

In Java on Android it is even forbidden to call e.g. network or database function/method on UI thread and OS immediately throws an exception.

I definitely agree. When multi threading and/or multi processing is required for a task, then I use it. I do prefer to use a simpler approach, such as a single thread, when that will achieve the same result.  

 

2 hours ago, zak100 said:

In Java we have "synchronize" command.

In Java I prefer, when suitable, to use higher level libraries or packages such as java.util.concurrent https://docs.oracle.com/javase/8/docs/api/index.html?java/util/concurrent/package-summary.html

 

8 minutes ago, zak100 said:

How can I get a combined CPU Usage in the above case?

How do you define "combined"? 

[zak100]

Hi,

while th1.is_alive(): 
   print("CPU usage:" + str(psutil.Process(pid).cpu_percent(interval=1))) 
print("thread1 ended") 
while th2.is_alive(): 
   print("CPU usage:" + str(psutil.Process(pid).cpu_percent(interval=1)))

Thanks for your response.

Quote

How do you define "combined"? 

Right now its showing CPU usage for only one thread, I have to modify the above code to see for which code its printing CPU usage.

 

Yes, its possible to add the values of two CPU_percent(..) values.

Quote

e.g. while loop checking whether any of two threads is alife (rather than just for one). Use logical or operator.

Good idea, but first I want to see using large values whether its printing two cpu usage percentages or just one.

I would try and then get back to you people.

Zulfi.

[Ghideon]

2 minutes ago, zak100 said:

Yes, its possible to add the values of two CPU_percent(..) values.

Can you provide something more rigorous? 

 

Learning to express and/or understand (in plain english, using math, pseudocode or other suitable method) the requirements or needs is in my opinion central to solving programming tasks. I recommend using your questions as exercises.

 

[Sensei]

34 minutes ago, Ghideon said:

I definitely agree. When multi threading and/or multi processing is required for a task, then I use it. I do prefer to use a simpler approach, such as a single thread, when that will achieve the same result.  

I prefer to have result fast.. As long as algorithm allows for easy migration to MT. Thanks to multithreading you can have results 8x/12x/.../48x/96x faster than on single thread.. depending how many cores/cpus are available.

Yet another level of making code harder to maintain and to read is migration to multi thousands GPU cores..

 

22 minutes ago, zak100 said:

Right now its showing CPU usage for only one thread, I have to modify the above code to see for which code its printing CPU usage.

 

Yes, its possible to add the values of two CPU_percent(..) values.

I was thinking about single while loop for both threads with separate cpu usage for each thread.. If thread is alife get cpu usage of thread, repeat with second thread, merge strings, print.

Edited March 26, 2020 by Sensei

[Sensei]

This might be interesting to you. CPU usage per thread.

https://stackoverflow.com/questions/26238184/cpu-usage-per-thread

Just use if thread th1 is alive then th1.get_cpu_percent(), render to string. Repeat with th2. Print final string.

Edited March 27, 2020 by Sensei

[zak100]

Hi,

 

Thanks for your useful suggestion and links.

Sorry, I could not understand the code provided at:
 

Quote

 

This might be interesting to you. CPU usage per thread.

https://stackoverflow.com/questions/26238184/cpu-usage-per-thread

Just use if thread th1 is alive then th1.get_cpu_percent(), render to string. Repeat with th2. Print final string.

 

Instead I used a simple solution which i discussed earlier and I found the link also:

https://www.bogotobogo.com/python/Multithread/python_multithreading_Identify_Naming_Logging_threads.php

 

My output is this:

Quote

(141, 20322) t2
(142, 20335) t2
(143, 20348) t2
(144, 20425) t2
(145, 20426) t2
(146, 20477) t2
(147, 20511) t2
(148, 20587) t2
(149, 20613) t2
(150, 20849) t2
(151, 20855) t2
(152, 20919) t2
(153, 21241) t2
(0, 10174) t1
(1, 10293) t1
(2, 10467) t1
(3, 10591) t1
(4, 10614) t1
(5, 10700) t1
(6, 10707) t1
(7, 10718) t1
(8, 10943) t1
(9, 10953) t1
(10, 10990) t1
(11, 11026) t1
(12, 11053) t1
(13, 11098) t1
(14, 11209) t1
(15, 11346) t1
(16, 11378) t1
(17, 11405) t1
(18, 11420) t1
(154, 21314)

Though the output is still garbled but not I know which thread is printing what.

 

God bless you guys.

 

My problem is solved now.

 

Zulfi.

[Ghideon]

14 hours ago, zak100 said:

My problem is solved now.

Good to hear that.

14 hours ago, zak100 said:

Instead I used a simple solution which i discussed earlier and I found the link also:

https://www.bogotobogo.com/python/Multithread/python_multithreading_Identify_Naming_Logging_threads.php

If you scroll down on that page you will find and intro to logging in Python. Generally logging* is an important part in any programming and useful, even in basic cases. Here is a quick example where thread name etc is printed in a consistent fashion by calling logging functions. Note the initial configuration that is done only once:  

import logging

logging.basicConfig(level=logging.DEBUG,
                      format='%(asctime)s [%(levelname)s] (%(threadName)-9s) %(message)s',)

logging.debug('start')
logging.debug('end')

2020-03-28 14:20:31,219 [DEBUG] (MainThread) start
2020-03-28 14:20:31,220 [DEBUG] (MainThread) end

 

Bold by me:

On 3/26/2020 at 9:41 PM, Sensei said:

I prefer to have result fast.. As long as algorithm allows for easy migration to MT. Thanks to multithreading you can have results 8x/12x/.../48x/96x faster than on single thread.. depending how many cores/cpus are available.

Yet another level of making code harder to maintain and to read is migration to multi thousands GPU cores..

Spot on. I think most of my experiences comes from being involved in projects where the bold part above did not apply or had to be weighed against other factors, hence my point of view above (default single thread, MT when required or a good option). Discussion in detail when to use MT vs other means of delivering fast results would be really interesting but unfortunately of topic  

 

*) My opinion is strongly biased by my profession regarding this subject.