Progress track thread pool rather than each process

Heads up! You've already completed this tutorial.

Thomas_Parker | 2020-08-03 06:21:33 UTC | #1

Hiya,

Amazing book and site. By the way just quickly before I ask this question you have mistakes on your 1 to 1 live tutoring page. The 30 min session says 2 hours and is 190 dollars and then you have 2 other sessions each 1 hour at 100 dollars. Doesn't seem right.

Anyway I am building an inference tester for a tensorflow model I have built and trained. I have written it using the skeleton of your "progress_many" example. However in that example you create a for loop within the thread to simulate progress and send it back to the progress bar.

Inference happens so fast there is no point in that for my use. But I do need to know how long the whole batch took. I have two types of workers. One performs predicitons on an image parsed to it, the other is then signalled to run a worker that copies that file into the appropiate folder.

EDIT: I have set it up so I only click the start button once and it does the whole lot. I don't click it to start each thread.

I need to know how long all of the inference threads took as well as the file copying.

I'm not sure how to track the workers and do this. Do you have any quick examples please?

Thank You!


martin | 2020-09-07 12:25:59 UTC | #2

Hey @Thomas_Parker welcome to the forum

By the way just quickly before I ask this question you have mistakes on your 1 to 1 live tutoring page. The 30 min session says 2 hours and is 190 dollars and then you have 2 other sessions each 1 hour at 100 dollars

You're quite right -- thanks for spotting that! Fixed now.

Inference happens so fast there is no point in that for my use. But I do need to know how long the whole batch took. I have two types of workers. One performs predicitons on an image parsed to it, the other is then signalled to run a worker that copies that file into the appropiate folder.

If you want to track the duration of time taken for each worker, I would make each worker calculate it's own time and emit the duration at the end (when it is complete). You can collect these up together, and display. A simple way to do this would be to use Python time() to get the milliseconds before and after the run (from within the runner's run method) and emit this as a duration at the end.

python
import random
import sys
import time
import uuid

from PyQt5.QtCore import QObject, QRunnable, QThreadPool, QTimer, pyqtSignal, pyqtSlot
from PyQt5.QtWidgets import (
    QApplication,
    QLabel,
    QMainWindow,
    QProgressBar,
    QPushButton,
    QVBoxLayout,
    QWidget,
)


class WorkerSignals(QObject):
    """
    Defines the signals available from a running worker thread.

    progress
        int progress complete,from 0-100
    """

    duration = pyqtSignal(str, int)
    finished = pyqtSignal(str)


class Worker(QRunnable):
    """
    Worker thread

    Inherits from QRunnable to handle worker thread setup, signals
    and wrap-up.
    """

    def __init__(self):
        super().__init__()
        self.job_id = uuid.uuid4().hex  # <1>
        self.signals = WorkerSignals()

    @pyqtSlot()
    def run(self):
        start_time = time.time()
        total_n = 1000
        delay = random.random() / 100  # Random delay value.
        for n in range(total_n):
            time.sleep(delay)

        end_time = time.time()
        self.signals.duration.emit(self.job_id, end_time-start_time)
        self.signals.finished.emit(self.job_id)


class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()

        layout = QVBoxLayout()

        self.duration = QLabel()

        button = QPushButton("START IT UP")
        button.pressed.connect(self.execute)

        layout.addWidget(self.duration)
        layout.addWidget(button)

        w = QWidget()
        w.setLayout(layout)

        # Dictionary holds the duration of current workers.
        self.worker_duration = {}

        self.setCentralWidget(w)

        self.show()

        self.threadpool = QThreadPool()
        print(
            "Multithreading with maximum %d threads" % self.threadpool.maxThreadCount()
        )

        self.timer = QTimer()
        self.timer.setInterval(100)
        self.timer.timeout.connect(self.refresh_duration)
        self.timer.start()

    def execute(self):
        self.cleanup() # Clear existing durations.

        for n in range(0, 4): # 4 per batch
            worker = Worker()
            worker.signals.duration.connect(self.update_duration)

            # Execute
            self.threadpool.start(worker)

    def cleanup(self):
        # Remove all previous durations.
        self.worker_duration = {}
        self.refresh_duration()

    def update_duration(self, job_id, duration):
        self.worker_duration[job_id] = duration

    def calculate_duration(self):
        if not self.worker_duration:
            return 0

        return sum(ms for ms in self.worker_duration.values())

    def refresh_duration(self):
        # Calculate total duration.
        duration = self.calculate_duration()
        self.duration.setText("{} ms ({} complete)".format(duration, len(self.worker_duration)))

app = QApplication(sys.argv)
window = MainWindow()
app.exec_()

If you're only running one batch at a time this should be enough. If you're going to be doing multiple concurrent batches, you could also pass the "batch id" into the workers and have them emit it along with the duration so you can group the times for each batch separately.


The complete guide to packaging Python GUI applications with PyInstaller.
[[ discount.discount_pc ]]% OFF for the next [[ discount.duration ]] [[discount.description ]] with the code [[ discount.coupon_code ]]

Purchasing Power Parity

Developers in [[ country ]] get [[ discount.discount_pc ]]% OFF on all books & courses with code [[ discount.coupon_code ]]
Well done, you've finished this tutorial! Mark As Complete
[[ user.completed.length ]] completed [[ user.streak+1 ]] day streak

Progress track thread pool rather than each process was written by Martin Fitzpatrick .

Martin Fitzpatrick has been developing Python/Qt apps for 8 years. Building desktop applications to make data-analysis tools more user-friendly, Python was the obvious choice. Starting with Tk, later moving to wxWidgets and finally adopting PyQt.