Frame Sync Loss

nyaaaaaaatalie · November 30, 2023, 9:44pm

Hi all,

Used the Pi image successfully but need some more compute power for a stationary project. I started with the setup script on a fresh Ubuntu 22.04 x86 VM. I can see the 5 RTL-SDRs with lsusb but for some reason I get Frame Sync: LOSS in the web GUI. I thought it was something with VMware, but I tried on my laptop and got the same issue - fresh 22.04 install as well on my Thinkpad. What am I doing wrong?

krakenrf_syed · November 30, 2023, 10:44pm

How are you powering the KrakenSDR?

nyaaaaaaatalie · November 30, 2023, 10:57pm

Via a connection to a dedicated PCIe USB controller on the server, from a dedicated 12v USB adapter in the car on my laptop, and a good quality USB A-C cable that worked in the car with my Pi. Overall the laptop setup is exactly the car setup just swapping the Pi for my laptop. I can get a USB power monitor if you suspect USB power issues are afoot.

krakenrf_syed · November 30, 2023, 11:09pm

If it’s a known-good power source, then there is no need to measure the voltage output of the power source.

krakenrf_syed · November 30, 2023, 11:15pm

@krakenrf_carl What are your thoughts?

krakenrf_carl · December 1, 2023, 12:53am

It hasn’t been tested with VMWare, only VirtualBox, can you try that instead?

In theory VMWare should also work, but you might need to tweak the USB settings like we do in VirtualBox. For whatever reason in VirtualBox only the USB3.0 controller emulator works fast enough, even though it’s a USB 2.0 device. Maybe there are similar settings in VMware?

Please also remember to take a look at anything that might be in the log files in

hemidall_daq_fw/Firmware/_logs

and in

krakensdr_doa/ui.log

nyaaaaaaatalie · December 1, 2023, 1:07am

Okay, thanks for the notes.

I’m actually not using USB passthrough, I am using PCIe passthrough - passing a whole Renesas USB controller to the VM through ESXi. I have done this before with other high-sample rate SDRs and had good luck.

Log files might lead somewhere - the rtl_daq file is filled with rtlsdr_demod_write_reg failed with -4. It also said to increase the usb buffer size, which I tried and it didn’t do anything other than not throw the error in the logs anymore.

Interestingly I think it’s able to start up but then something happens. In the web GUI I can see it goes through the calibration and the latency numbers are updating then all of a sudden it stops. I observed this behavior on both my laptop and the server. So I’m unsure where to go from here, I looked up the RTL error -4 and didn’t find much.

krakenrf_carl · December 1, 2023, 2:10am

Not sure about PCIe passthrough - I would expect it to work but Kraken can be a little picky when it comes to USB since it needs to align the USB samples on all 5 channels. If there is excessive latency on the USB due to too much USB congestion or latency from a VM layer then it won’t work.

But rtlsdr_demod_write_reg failed with -4 seems to indicate that something else is wrong. If you run rtl_test on each channel simultaneously, does it work or do you also get errors?

nyaaaaaaatalie · December 1, 2023, 2:38am

Both laptop (bare metal) and server (virtualized) are exhibiting the same symptoms so I don’t think it’s the virt layer or PCIe passthru but I will keep that in mind if I start seeing weirdness on the VM.

I let rtl_test -d [0-4] burn for about 5 minutes and got no errors on any of the instances so I think the USB layer is stable. And that’s as my user not root so it shouldn’t be a permissions issue either.

krakenrf_carl · December 5, 2023, 8:36am

With all rtl_test’s running at the same time?

The other remaining things to check then

Try different cables for the USB port
If possible try other USB ports

Please also upload all the log files somewhere.

nyaaaaaaatalie · December 5, 2023, 9:16pm

Yes with all of them running at once. I tried a different cable and port and no dice. Last 100 lines of each Heimdall log file:

==> decimator.log <==

==> delay_sync.log <==
INFO:__main__:IQ adjustment vector: abs:[1. 1. 1. 1. 1.]
INFO:__main__:IQ adjustment vector: phase:[0. 0. 0. 0. 0.]
WARNING:shmemIface:Shared memory not exist
WARNING:shmemIface:Shared memory not exist
WARNING:shmemIface:Shared memory not exist
WARNING:shmemIface:Shared memory not exist
CRITICAL:__main__:Failed to acquire new data frame, exiting..
INFO:shmemIface:Terminate signal sent
/home/natea/miniforge3/envs/kraken/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 6 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/natea/miniforge3/envs/kraken/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/delay_sync_hwc_B': [Errno 2] No such file or directory: '/delay_sync_hwc_B'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/natea/miniforge3/envs/kraken/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/delay_sync_hwc_A': [Errno 2] No such file or directory: '/delay_sync_hwc_A'
  warnings.warn('resource_tracker: %r: %s' % (name, e))

==> hwc.log <==
Traceback (most recent call last):
  File "/home/natea/krakensdr_doa/heimdall_daq_fw/Firmware/_daq_core/hw_controller.py", line 700, in <module>
    HWC_inst0.start()
  File "/home/natea/krakensdr_doa/heimdall_daq_fw/Firmware/_daq_core/hw_controller.py", line 413, in start
    active_buff_index = self.in_shmem_iface.wait_buff_free()
  File "/home/natea/krakensdr_doa/heimdall_daq_fw/Firmware/_daq_core/shmemIface.py", line 188, in wait_buff_free
    signal = unpack('B', os.read(self.fw_ctr_fifo, 1))[0]
struct.error: unpack requires a buffer of 1 bytes
/home/natea/miniforge3/envs/kraken/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 2 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

==> logs <==
Realtek data acquistion module : rtl_daq.log
Sample delay synchronizer      : sync.log
Sample rebuffer                : rebuffer.log
Squelch module                 : squelch.log
IQ sample decimator            : decimator.log
Delay synchronizer module      : delay_sync.log
IQ Server                      : iq_server.log
Hardware controller module     : hwc.log

==> rebuffer.log <==
03:28:35 FATAL rtl_daq.h:62: IQ header read error

==> rtl_daq.log <==
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_read_reg failed with -4
r82xx_write_arr: i2c wr failed=-4 reg=17 len=1
rtlsdr_demod_write_reg failed with -4
r82xx_write_arr: i2c wr failed=-4 reg=05 len=1
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
r82xx_write_arr: i2c wr failed=-4 reg=05 len=1
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
r82xx_write_arr: i2c wr failed=-4 reg=0a len=1
ERROR setting I2C register 0x0A to value 00 with mask 0F
r82xx_write_arr: i2c wr failed=-4 reg=0a len=1
rtlsdr_demod_write_reg failed with -4
ERROR setting I2C register 0x0A to value 00 with mask 0F
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_write_reg failed with -4
rtlsdr_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
r82xx_write_arr: i2c wr failed=-4 reg=05 len=1
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_write_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
r82xx_write_arr: i2c wr failed=-4 reg=0a len=1
ERROR setting I2C register 0x0A to value 00 with mask 0F
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_write_reg failed with -4
rtlsdr_write_reg failed with -4
rtlsdr_write_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_demod_write_reg failed with -4
rtlsdr_demod_read_reg failed with -4
rtlsdr_write_reg failed with -4
rtlsdr_write_reg failed with -4

And ui.log:

INFO:quart.serving:Running on http://0.0.0.0:8080 (CTRL + C to quit)
gpsd Available
 * Serving Quart app 'maindash'
 * Environment: production
 * Please use an ASGI server (e.g. Hypercorn) directly in production
 * Debug mode: False
 * Running on http://0.0.0.0:8080 (CTRL + C to quit)
[2023-12-01 01:03:55,536] Running on http://0.0.0.0:8080 (CTRL + C to quit)
INFO:quart.serving:10.0.100.199:62738 GET /config 1.1 200 661 26211
[2023-12-01 01:03:55,651] 10.0.100.199:62738 GET /config 1.1 200 661 26211
INFO:quart.serving:10.0.100.199:62738 GET /config 1.1 - - 28530
[2023-12-01 01:03:55,654] 10.0.100.199:62738 GET /config 1.1 - - 28530
INFO:quart.serving:10.0.100.199:62741 GET /_push 1.1 101 - 3721
[2023-12-01 01:03:56,111] 10.0.100.199:62741 GET /_push 1.1 101 - 3721
INFO:quart.serving:10.0.100.199:62743 GET /config 1.1 200 661 5710
[2023-12-01 01:04:16,757] 10.0.100.199:62743 GET /config 1.1 200 661 5710
INFO:quart.serving:10.0.100.199:62743 GET /config 1.1 - - 6678
[2023-12-01 01:04:16,758] 10.0.100.199:62743 GET /config 1.1 - - 6678
INFO:quart.serving:10.0.100.199:62744 GET /_push 1.1 101 - 3728
[2023-12-01 01:04:16,936] 10.0.100.199:62744 GET /_push 1.1 101 - 3728
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/natea/miniforge3/envs/kraken/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/natea/krakensdr_doa/krakensdr_doa/_signal_processing/krakenSDR_signal_processor.py", line 303, in run
    if self.module_receiver.get_iq_online():
  File "/home/natea/krakensdr_doa/krakensdr_doa/_receiver/krakenSDR_receiver.py", line 192, in get_iq_online
    active_buff_index = self.in_shmem_iface.wait_buff_free()
  File "/home/natea/krakensdr_doa/krakensdr_doa/_receiver/shmemIface.py", line 196, in wait_buff_free
    signal = unpack("B", os.read(self.fw_ctr_fifo, 1))[0]
struct.error: unpack requires a buffer of 1 bytes
/home/natea/miniforge3/envs/kraken/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 2 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/natea/miniforge3/envs/kraken/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/delay_sync_iq_A': [Errno 2] No such file or directory: '/delay_sync_iq_A'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/natea/miniforge3/envs/kraken/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/delay_sync_iq_B': [Errno 2] No such file or directory: '/delay_sync_iq_B'
  warnings.warn('resource_tracker: %r: %s' % (name, e))

krakenrf_carl · December 8, 2023, 9:59am

Hmm yeah rtlsdr_demod_write_reg failed with -4 is the core error, and all the other errors stem from that.

Unfortunately at this stage I’m just not sure, seems like it could be a specific issue with your laptop. A bit odd that rtl_test simultaneously works fine though. rtl_daq is doing essentially the same thing by running all the RTL-SDRs at the same time.

Do you happen to have any other Linux laptops you can try?

Maybe try recompiling daq_core.

Go into

~/krakensdr_doa/heimdall_daq_fw/Firmware/_daq_core/

and type make

You could try and edit the Makefile too and turning off optimizations just in case something is going wrong on your platform.

edit Makefile and remove the -O2 from CFLAGS=-Wall -std=gnu99 -march=native -O2 -I.

nyaaaaaaatalie · December 9, 2023, 10:05pm

I have no idea why, but I rebooted and it came up fine. I am getting solid green across the board now. I’ll update if I have any more issues. Thanks for your help Carl.

nyaaaaaaatalie · December 9, 2023, 11:26pm

So that was a lie.

I got it to work for all of a few minutes, then when I went to turn down the data block length, I restarted the DAQ chain and it restarted but didn’t come back up again. Then, I went to put it back to defaults and it is stuck on Restarting.

I rebooted the machine and it’s doing the same thing as before - it goes through about 96 calibration frames then gets stuck on Dummy, when I hit refresh it goes back to frame sync: loss. I’m not sure what’s going on with this, does anybody have any ideas?

krakenrf_carl · December 11, 2023, 12:14am

Can you try with the default data block length again?

nyaaaaaaatalie · December 11, 2023, 8:30pm

I made sure to set it back to the default and I’m still encountering this error.

krakenrf_carl · December 19, 2023, 5:47am

At this stage I’m unsure sorry. All I can do is suggest trying some known working systems. If it’s just more computer power you are after, a Pi 5 or Orange Pi 5 might do the job nicely.

It may just be that your machines don’t play nicely with the KrakenSDR. Sometimes it’s just a matter of a USB chip being incompatible. But I have not personally encountered any machines like that in a while.