tl;dr - Today I shipped the ability to see the desktop screen in VR (while using Freebird). And fixed a few user-reported bugs in Freebird.
Performance
The performance is still a bit laggy. The actual screencapture code now runs in a separate process, and copies data over a SharedMemory buffer (which works pretty well for sharing data between two separate processes). That helps avoid Python's GIL while performing numpy operations on large arrays.
But the main performance bottleneck is the inability to update an existing texture using Blender's gpu module. The current implementation in Blender forces me to create a new texture each frame, which is pretty slow. And this has to be done on the main thread, otherwise Blender crashes with a null context error.
So I update the screen just two times per second. For now, the usability is adequate IMO. But if it becomes important, I can maybe look at the Blender codebase and see if there's anything that can be proposed as a solution. Like right now, doing video is impossible at decent framerates using the gpu module. Maybe I missed something, but I really searched a lot (and tried a lot of approaches).
Illusion of responsiveness
To create an illusion of responsiveness, the mouse cursor is updated every frame (72fps), while the actual screen updates twice per second (2fps).
Custom code to avoid dependencies
And thanks to ChatGPT/Claude, I got a custom Windows-only implementation for taking screenshots and getting the mouse cursor location. It's a bit faster than mss/pyautogui and works fine for my purpose.
The main reason was to avoid depending on external libraries (tricky to install them, being a blender addon). With AI codegen tools, it might actually be a good tradeoff, since the generated code is reasonably straightforward and maintainable (and uses well-supported Windows APIs). And I didn't spend much time on them.
Let's see how it performs during actual usage. It's now in early-access on Freebird. Hopefully it won't crash.