Thomas_Parker | 2020-08-03 06:21:33 UTC | #1
Hiya,
Amazing book and site. By the way just quickly before I ask this question you have mistakes on your 1 to 1 live tutoring page. The 30 min session says 2 hours and is 190 dollars and then you have 2 other sessions each 1 hour at 100 dollars. Doesn't seem right.
Anyway I am building an inference tester for a tensorflow model I have built and trained. I have written it using the skeleton of your "progress_many" example. However in that example you create a for loop within the thread to simulate progress and send it back to the progress bar.
Inference happens so fast there is no point in that for my use. But I do need to know how long the whole batch took. I have two types of workers. One performs predicitons on an image parsed to it, the other is then signalled to run a worker that copies that file into the appropiate folder.
EDIT: I have set it up so I only click the start button once and it does the whole lot. I don't click it to start each thread.
I need to know how long all of the inference threads took as well as the file copying.
I'm not sure how to track the workers and do this. Do you have any quick examples please?
Thank You!
martin | 2020-09-07 12:25:59 UTC | #2
Hey @Thomas_Parker welcome to the forum
By the way just quickly before I ask this question you have mistakes on your 1 to 1 live tutoring page. The 30 min session says 2 hours and is 190 dollars and then you have 2 other sessions each 1 hour at 100 dollars
You're quite right -- thanks for spotting that! Fixed now.
Inference happens so fast there is no point in that for my use. But I do need to know how long the whole batch took. I have two types of workers. One performs predicitons on an image parsed to it, the other is then signalled to run a worker that copies that file into the appropiate folder.
If you want to track the duration of time taken for each worker, I would make each worker calculate it's own time and emit the duration at the end (when it is complete). You can collect these up together, and display. A simple way to do this would be to use Python time() to get the milliseconds before and after the run (from within the runner's run method) and emit this as a duration at the end.
import random
import sys
import time
import uuid
from PyQt5.QtCore import QObject, QRunnable, QThreadPool, QTimer, pyqtSignal, pyqtSlot
from PyQt5.QtWidgets import (
QApplication,
QLabel,
QMainWindow,
QProgressBar,
QPushButton,
QVBoxLayout,
QWidget,
)
class WorkerSignals(QObject):
"""
Defines the signals available from a running worker thread.
progress
int progress complete,from 0-100
"""
duration = pyqtSignal(str, int)
finished = pyqtSignal(str)
class Worker(QRunnable):
"""
Worker thread
Inherits from QRunnable to handle worker thread setup, signals
and wrap-up.
"""
def __init__(self):
super().__init__()
self.job_id = uuid.uuid4().hex # <1>
self.signals = WorkerSignals()
@pyqtSlot()
def run(self):
start_time = time.time()
total_n = 1000
delay = random.random() / 100 # Random delay value.
for n in range(total_n):
time.sleep(delay)
end_time = time.time()
self.signals.duration.emit(self.job_id, end_time-start_time)
self.signals.finished.emit(self.job_id)
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
layout = QVBoxLayout()
self.duration = QLabel()
button = QPushButton("START IT UP")
button.pressed.connect(self.execute)
layout.addWidget(self.duration)
layout.addWidget(button)
w = QWidget()
w.setLayout(layout)
# Dictionary holds the duration of current workers.
self.worker_duration = {}
self.setCentralWidget(w)
self.show()
self.threadpool = QThreadPool()
print(
"Multithreading with maximum %d threads" % self.threadpool.maxThreadCount()
)
self.timer = QTimer()
self.timer.setInterval(100)
self.timer.timeout.connect(self.refresh_duration)
self.timer.start()
def execute(self):
self.cleanup() # Clear existing durations.
for n in range(0, 4): # 4 per batch
worker = Worker()
worker.signals.duration.connect(self.update_duration)
# Execute
self.threadpool.start(worker)
def cleanup(self):
# Remove all previous durations.
self.worker_duration = {}
self.refresh_duration()
def update_duration(self, job_id, duration):
self.worker_duration[job_id] = duration
def calculate_duration(self):
if not self.worker_duration:
return 0
return sum(ms for ms in self.worker_duration.values())
def refresh_duration(self):
# Calculate total duration.
duration = self.calculate_duration()
self.duration.setText("{} ms ({} complete)".format(duration, len(self.worker_duration)))
app = QApplication(sys.argv)
window = MainWindow()
app.exec_()
If you're only running one batch at a time this should be enough. If you're going to be doing multiple concurrent batches, you could also pass the "batch id" into the workers and have them emit it along with the duration so you can group the times for each batch separately.
Also available from Leanpub and Amazon Paperback
