Monitoring a Running Background Process from a PyQt6 Application — How to connect your GUI to an external program that's already running

2021-04-30T09:00:00+00:00

I followed the tutorial about using QProcess to launch an external script/program. This allows me to start the program FROM a PyQt application and monitor and get results from it. But what if I need to connect to an external program that is already running in the background 24/7? Is there a way to code that program so that it is outputting stuff all the time, and then when I connect with a PyQt GUI it gives me a view on it?

QProcess is designed to start a new process and communicate with it through standard input/output pipes. If a process is already running, QProcess can't attach to it — those pipes only exist when QProcess is the one doing the launching. So you need a different approach.

The good news is there are several practical ways to bridge a running background process and a PyQt6 GUI. The right choice depends on your situation: how much data you're dealing with, whether you need real-time streaming or periodic snapshots, and whether you have control over the background process's code.

In this article we'll walk through three approaches, from simplest to most flexible:

File-based communication — the background process writes to a file, and the GUI reads it.
Socket-based communication — the background process acts as a server, and the GUI connects as a client.
SQLite as shared storage — useful when you need structured data lookups.

File-Based Communication

The simplest approach is to have your background process write its output to a file, and then have your PyQt6 application watch that file for changes. This works well when you just need to see the latest output and don't need to send commands back to the process.

The background process

Here's a small example script that simulates a long-running background process. It writes a line of data to a file every second:

python

# background_process.py
import time
import datetime

OUTPUT_FILE = "process_output.txt"

while True:
    timestamp = datetime.datetime.now().isoformat()
    with open(OUTPUT_FILE, "a") as f:
        f.write(f"{timestamp} — Sensor reading: {time.time() % 100:.2f}\n")
        f.flush()
    time.sleep(1)

Start this running in a separate terminal (python background_process.py) before launching the GUI.

The PyQt6 GUI

On the GUI side, we can use a QTimer to periodically check the file for new content. We keep track of where we last read up to, so each time we only read new lines:

python

# file_monitor.py
import sys
from PyQt6.QtWidgets import (
    QApplication, QMainWindow, QPlainTextEdit, QVBoxLayout, QWidget
)
from PyQt6.QtCore import QTimer


class FileMonitorWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("File Monitor")
        self.resize(600, 400)

        self.text_display = QPlainTextEdit()
        self.text_display.setReadOnly(True)

        layout = QVBoxLayout()
        layout.addWidget(self.text_display)

        container = QWidget()
        container.setLayout(layout)
        self.setCentralWidget(container)

        self.file_path = "process_output.txt"
        self.last_position = 0

        # Check for new data every 500 milliseconds.
        self.timer = QTimer()
        self.timer.timeout.connect(self.read_new_data)
        self.timer.start(500)

    def read_new_data(self):
        try:
            with open(self.file_path, "r") as f:
                f.seek(self.last_position)
                new_data = f.read()
                if new_data:
                    self.text_display.appendPlainText(new_data.rstrip("\n"))
                    self.last_position = f.tell()
        except FileNotFoundError:
            pass  # File doesn't exist yet, nothing to read.


app = QApplication(sys.argv)
window = FileMonitorWindow()
window.show()
app.exec()

This uses f.seek() to jump to the position where we last finished reading, so we only pick up new lines each time. The timer fires every 500 milliseconds, which gives a near-real-time feel without hammering the filesystem.

File-based communication is straightforward and reliable. The downside is that the file grows indefinitely unless you manage rotation or truncation, and it only supports one-way communication (process → GUI) without extra work.

Using QFileSystemWatcher

Instead of polling with a timer, you can use Qt's QFileSystemWatcher to get notified when the file changes. This is slightly more efficient since it relies on operating system file-change notifications rather than repeatedly opening and reading the file:

python

# file_watcher_monitor.py
import sys
from PyQt6.QtWidgets import (
    QApplication, QMainWindow, QPlainTextEdit, QVBoxLayout, QWidget
)
from PyQt6.QtCore import QFileSystemWatcher


class FileWatcherWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("File Watcher Monitor")
        self.resize(600, 400)

        self.text_display = QPlainTextEdit()
        self.text_display.setReadOnly(True)

        layout = QVBoxLayout()
        layout.addWidget(self.text_display)

        container = QWidget()
        container.setLayout(layout)
        self.setCentralWidget(container)

        self.file_path = "process_output.txt"
        self.last_position = 0

        self.watcher = QFileSystemWatcher([self.file_path])
        self.watcher.fileChanged.connect(self.on_file_changed)

        # Do an initial read to catch anything already in the file.
        self.on_file_changed()

    def on_file_changed(self):
        try:
            with open(self.file_path, "r") as f:
                f.seek(self.last_position)
                new_data = f.read()
                if new_data:
                    self.text_display.appendPlainText(new_data.rstrip("\n"))
                    self.last_position = f.tell()

            # Some platforms remove the file from the watch list after
            # a change, so re-add it.
            if self.file_path not in self.watcher.files():
                self.watcher.addPath(self.file_path)

        except FileNotFoundError:
            pass


app = QApplication(sys.argv)
window = FileWatcherWindow()
window.show()
app.exec()

One thing to be aware of: on some platforms, QFileSystemWatcher may stop watching a file after it's modified (depending on how the writing process handles the file). That's why we re-add the path after each change event. This is a well-known quirk and the re-add is a standard workaround.

Socket-Based Communication

If you need two-way communication — or if you want real-time streaming without writing to disk — sockets are a better fit. The background process runs a small TCP server, and the PyQt6 GUI connects to it as a client.

This approach also works across machines on a network, which is useful if your background process runs on a different computer.

The background process with a socket server

This background process listens on a TCP socket and sends data to any connected client:

python

# background_server.py
import socket
import threading
import time
import datetime

HOST = "127.0.0.1"
PORT = 65432

clients = []
clients_lock = threading.Lock()


def handle_client(conn, addr):
    """Keep the connection open and let the broadcaster send data."""
    print(f"Client connected: {addr}")
    with clients_lock:
        clients.append(conn)
    try:
        # Keep connection alive until client disconnects.
        while True:
            data = conn.recv(1024)
            if not data:
                break
    except (ConnectionResetError, BrokenPipeError):
        pass
    finally:
        print(f"Client disconnected: {addr}")
        with clients_lock:
            clients.remove(conn)
        conn.close()


def broadcast_data():
    """Periodically send data to all connected clients."""
    while True:
        timestamp = datetime.datetime.now().isoformat()
        message = f"{timestamp} — Sensor reading: {time.time() % 100:.2f}\n"
        with clients_lock:
            for conn in clients[:]:
                try:
                    conn.sendall(message.encode("utf-8"))
                except (ConnectionResetError, BrokenPipeError):
                    clients.remove(conn)
        time.sleep(1)


# Start the data broadcasting in a background thread.
threading.Thread(target=broadcast_data, daemon=True).start()

# Accept incoming connections.
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server.bind((HOST, PORT))
    server.listen()
    print(f"Server listening on {HOST}:{PORT}")
    while True:
        conn, addr = server.accept()
        threading.Thread(
            target=handle_client, args=(conn, addr), daemon=True
        ).start()

This server accepts multiple clients and broadcasts data to all of them once per second.

The PyQt6 GUI client

On the GUI side, we can use QTcpSocket from QtNetwork to connect to the server. QTcpSocket integrates with Qt's event loop, so incoming data triggers a signal — no threads or timers needed:

python

# socket_monitor.py
import sys
from PyQt6.QtWidgets import (
    QApplication, QMainWindow, QPlainTextEdit, QVBoxLayout,
    QWidget, QPushButton, QHBoxLayout, QLabel
)
from PyQt6.QtNetwork import QTcpSocket, QAbstractSocket


class SocketMonitorWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Socket Monitor")
        self.resize(600, 400)

        self.text_display = QPlainTextEdit()
        self.text_display.setReadOnly(True)

        self.status_label = QLabel("Disconnected")

        self.connect_button = QPushButton("Connect")
        self.connect_button.clicked.connect(self.toggle_connection)

        button_layout = QHBoxLayout()
        button_layout.addWidget(self.status_label)
        button_layout.addStretch()
        button_layout.addWidget(self.connect_button)

        layout = QVBoxLayout()
        layout.addLayout(button_layout)
        layout.addWidget(self.text_display)

        container = QWidget()
        container.setLayout(layout)
        self.setCentralWidget(container)

        self.socket = QTcpSocket()
        self.socket.readyRead.connect(self.on_data_ready)
        self.socket.connected.connect(self.on_connected)
        self.socket.disconnected.connect(self.on_disconnected)
        self.socket.errorOccurred.connect(self.on_error)

    def toggle_connection(self):
        if self.socket.state() == QAbstractSocket.SocketState.ConnectedState:
            self.socket.disconnectFromHost()
        else:
            self.socket.connectToHost("127.0.0.1", 65432)
            self.status_label.setText("Connecting...")

    def on_connected(self):
        self.status_label.setText("Connected")
        self.connect_button.setText("Disconnect")

    def on_disconnected(self):
        self.status_label.setText("Disconnected")
        self.connect_button.setText("Connect")

    def on_error(self, error):
        self.status_label.setText(f"Error: {self.socket.errorString()}")

    def on_data_ready(self):
        data = self.socket.readAll().data().decode("utf-8")
        self.text_display.appendPlainText(data.rstrip("\n"))


app = QApplication(sys.argv)
window = SocketMonitorWindow()
window.show()
app.exec()

Start the background server first, then launch this GUI and click Connect. You'll see live data streaming into the text area. Click Disconnect to stop, and Connect again to resume — exactly the "connect when you want a view" behavior described in the original question.

The QTcpSocket.readyRead signal is emitted whenever new data arrives, so the GUI stays responsive without any polling. This is similar to how signals and slots keep your application responsive by reacting to events rather than blocking.

Using SQLite as Shared Storage

When your background process produces structured data that you need to query — filtering by time range, looking up specific records, aggregating values — a shared SQLite database is a practical middle ground between files and a full database server.

The background process writing to SQLite

python

# background_db_writer.py
import sqlite3
import time
import datetime

DB_PATH = "process_data.db"

conn = sqlite3.connect(DB_PATH)
conn.execute("""
    CREATE TABLE IF NOT EXISTS readings (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        timestamp TEXT NOT NULL,
        value REAL NOT NULL
    )
""")
conn.commit()

while True:
    timestamp = datetime.datetime.now().isoformat()
    value = time.time() % 100
    conn.execute(
        "INSERT INTO readings (timestamp, value) VALUES (?, ?)",
        (timestamp, value),
    )
    conn.commit()
    time.sleep(1)

The PyQt6 GUI reading from SQLite

python

# sqlite_monitor.py
import sys
import sqlite3
from PyQt6.QtWidgets import (
    QApplication, QMainWindow, QTableWidget, QTableWidgetItem,
    QVBoxLayout, QWidget, QPushButton, QHBoxLayout, QLabel
)
from PyQt6.QtCore import QTimer


class SQLiteMonitorWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("SQLite Monitor")
        self.resize(600, 400)

        self.table = QTableWidget()
        self.table.setColumnCount(3)
        self.table.setHorizontalHeaderLabels(["ID", "Timestamp", "Value"])

        self.status_label = QLabel("Watching database...")

        refresh_button = QPushButton("Refresh Now")
        refresh_button.clicked.connect(self.load_recent_data)

        button_layout = QHBoxLayout()
        button_layout.addWidget(self.status_label)
        button_layout.addStretch()
        button_layout.addWidget(refresh_button)

        layout = QVBoxLayout()
        layout.addLayout(button_layout)
        layout.addWidget(self.table)

        container = QWidget()
        container.setLayout(layout)
        self.setCentralWidget(container)

        self.db_path = "process_data.db"
        self.last_id = 0

        # Auto-refresh every 2 seconds.
        self.timer = QTimer()
        self.timer.timeout.connect(self.load_recent_data)
        self.timer.start(2000)

        self.load_recent_data()

    def load_recent_data(self):
        try:
            conn = sqlite3.connect(self.db_path)
            cursor = conn.execute(
                "SELECT id, timestamp, value FROM readings "
                "WHERE id > ? ORDER BY id ASC",
                (self.last_id,),
            )
            rows = cursor.fetchall()
            conn.close()

            for row_id, timestamp, value in rows:
                row_index = self.table.rowCount()
                self.table.insertRow(row_index)
                self.table.setItem(
                    row_index, 0, QTableWidgetItem(str(row_id))
                )
                self.table.setItem(
                    row_index, 1, QTableWidgetItem(timestamp)
                )
                self.table.setItem(
                    row_index, 2, QTableWidgetItem(f"{value:.2f}")
                )
                self.last_id = row_id

            if rows:
                self.status_label.setText(
                    f"Loaded {len(rows)} new rows "
                    f"(total: {self.table.rowCount()})"
                )

        except sqlite3.OperationalError:
            self.status_label.setText("Waiting for database...")


app = QApplication(sys.argv)
window = SQLiteMonitorWindow()
window.show()
app.exec()

The GUI queries only for rows with an ID greater than the last one it saw, so it efficiently picks up new data without re-reading everything. SQLite handles concurrent reads from multiple processes gracefully, as long as writes use short transactions (which is what conn.commit() after each insert gives us). For more complex table displays, you might want to explore the Model/View architecture with QTableView, which gives you more control over how data is presented.

Choosing the Right Approach

Here's a summary to help you decide which method fits your situation:

Approach	Best for	Direction	Real-time?
File + QTimer/QFileSystemWatcher	Simple logging, one-way output	Process → GUI	Near real-time (polling interval)
TCP sockets + QTcpSocket	Live streaming, two-way communication	Both directions	Yes
SQLite shared database	Structured data, queries, historical lookups	Process → GUI (or both)	Near real-time (polling interval)

For the scenario described in the original question — connecting to an already-running background process to get a live view — sockets are the most natural fit. The background process runs a small server, and your GUI connects and disconnects at will. When connected, data streams in live. When disconnected, the process keeps running undisturbed.

If you don't have control over the background process's code (so you can't add a socket server to it), redirecting its output to a file and using the file-monitoring approach is the most practical path. If you do need to launch and manage processes directly from your application, see the tutorial on running external programs with QProcess. For keeping your GUI responsive while performing long-running work, you may also want to look into multithreading with QThreadPool.

For an in-depth guide to building Python GUIs with PyQt6 see my book, Create GUI Applications with Python & Qt6.

Python GUIs - ipc

Monitoring a Running Background Process from a PyQt6 Application — How to connect your GUI to an external program that's already running

File-Based Communication

The background process

The PyQt6 GUI

Using QFileSystemWatcher

Socket-Based Communication

The background process with a socket server

The PyQt6 GUI client

Using SQLite as Shared Storage

The background process writing to SQLite

The PyQt6 GUI reading from SQLite

Choosing the Right Approach