Sensor Connectivity (which steps to perform when a sensor seems to be down) has been presented during the Cobundu Summer 2021 Sessions, this is the recording: Summer 2021 Session: Sensor Connectivity
Understanding the flow
In general, /wiki/spaces/WM/pages/492666 can be depicted as this: sensors send data to a platform/data cloud (eg LoRa sensors send their data through a LoRa hub to the LoRa Sensor Cloud, whereas proprietary sensors send their data to their platform), and Cobundu picks up the data from that platform.
It's true that between the raw data entering the data cloud and the data being visible in a Cobundu touchpoint (Studio, GO, Workplace App, Kiosk App, ...), the data moves through a couple of Spacewell internal platforms before being stored in timeslots in the data warehouse.
And that it also moves through some more tools before it's available in the Cobundu Dashboards (just think of the overnight loading).
This means that there are many stages in which data can get "stuck" or not end up in Cobundu.
Below questions will help you understand where the issue is when you're missing data, and what you could do to solve this.
Questions to ask when you're missing data
Question 1: Do you notice data is not coming into Cobundu during an installation?
Did you follow the instructions on Installation & Quality Assurance (https://confluence.mcs.be/display/SUM/Quality+Assurance) ?
- Hub installed and confirmed working before sensor activation?
- Look into hub Troubleshooting (Depending on the kind of hub, select the correct one from page LoRa Gateways / Hubs installation guides)
- Sensor activation in a moment where hub is reachable?
- Correct mapping
- No typo’s in the sensor ID’s
- …
- Yes
- Hub offline (if only 1 site) (check ports, network and switch, physical hub)
- Look into hub Troubleshooting (Depending on the kind of hub, select the correct one from page LoRa Gateways / Hubs installation guides)
- Global problem (check other tenants)
- Hub offline (if only 1 site) (check ports, network and switch, physical hub)
- Partial
- Hub offline (check ports, physical hub)
- Look into hub Troubleshooting (Depending on the kind of hub, select the correct one from page LoRa Gateways / Hubs installation guides)
- in order to determine of the issue might be a Sensor vendor problem (is data coming into sensor platform, eg TTN, CPMS?), see question 3
- Hub offline (check ports, physical hub)
- No
- Sensor specific, see question 3
Question 3: At which stage is the data missing? (more detailed info see below)
(for some examples/concrete cases we had at other customers, please check chapter "Sensor connectivity guide")
- Dashboards / live floorplans
- Check the config and location grouping of the sensor in Studio
- Check the floorplan and location master data in Studio + IWMS
- Check Dashboards Space category mapping in Studio (for Space monitor)
- Check installation of sensors + customer environment
- See troubleshooting list Dashboard activation
- Studio
- Check the configuration of the sensor in Studio (New Studio)
- Check status of sensor (New Studio: KPI or on Locations > Floorplan)
- Check the state of the sensor listener (Studio 1.0 > Settings)
- Sensor cloud
- Check if the sensor is provisioned/available on the relevant sensor platform (TTN, CPMS, Airthings)
- Battery level (tabs) (see Battery Lifecycle Measurement (Check Payload)(Spacewell internal))
- Physical location
- Check physical installation on site at the customer (ref Tabs (Tracknet) troubleshooting guide)
Check the data on the GO live floor plan
- Go to https://go.cobundu.com
- Login with tenant specific credentials
- Select the affected floor
- Select the affected sensor type in "View By"
- Check the state of the floorplan
- Check if the color of the location corresponds with the color of the data values of this sensor type (open legend; verify colors in Live floorplan settings)
- Check dashboards
- See troubleshooting list Dashboard activation
Check the state of the STUDIO listeners (currently not available)
- Go to https://studio.cobundu.com/
- Login with your credentials
- Select the affected tenant
- Select the settings
- Go to the jobs tab
- Check the state of the jobs
Check the sensors in Studio
- Go to https://studio.cobundu.com/
- Login with your credentials
- Select the affected tenant
- Select Devices
- Search the affected sensor
- is the sensor ID correct (in hexadecimal?) and complete? Is a dash separating every 2 characters? (ref Tabs (Tracknet) configuration guide)
- Check the sensor ID
- Check the last seen
- Open the Device Detail Page
- Check the channel
- Check the type based on the sensor models
- Check the Location
- Check the attribution of the location to Location Grouping (zones or departments)
Check the data on the sensor cloud
All tracknet/tabs sensors connect to TTN.
All Pointgrab sensors connect to CPMS platform.
All Airthings sensors connect to Airthings platform.
For Tabs: A sensor that sends joining messages is trying to join our LoRa network. Due to the way that LoRa works it can take quiet a few tries before everything is synchronized.
If however the sensor is still sending joining messages after 1 day it is best to reset the sensor by removing the battery for a couple of hours and reinserting it.
If you're interested in knowing the battery health for Tracknet tabs, you can check this page: Battery Lifecycle Measurement (Check Payload)(Spacewell internal)
Cases
Below you find some cases that we encountered at our customers. The questions the consultant should ask are a combination from the above Connectivity Guide and the Tabs (Tracknet) troubleshooting guide
Customer A: Sensor health yellow on alternating sensors → Hub offline due to changes to the switch
At customer A, we noticed that a large part of sensors was not interacting with Cobundu as it should. As you can see on the below sensor health plan: The sensors in the yellow highlighted locations had difficulties sending data to Cobundu.
Analysis:
- Question 1: Do you notice data is not coming into Cobundu during an installation? NA
- Question 2: Are all sensor affected? Only partially
- Question 3: At which stage is the data missing? Sensors not connecting as frequently to Sensor platform/Cobundu as we would like to
Solution:
We noticed that 1 of the 2 hubs that were installed on that site was offline.
After a physical check by the customer, it was confirmed that due to changes to the switch, one of the hubs was offline. So, sensors had difficulties reaching the other hub and therefore they were not offline (red) but were able to connect to the remaining hub once in a while (hence the yellow To Check status).
The switch was modified, the hub confirmed working again and the sensors were able to connect to the nearest hub again.
Customer B: Sensor health yellow on alternating sensors → Hubs needed reboot
At customer B, we noticed that a large part of sensors was not interacting with Cobundu as it should: Different sensors were going into "to check" status, then coming back online; only to find other sensors now in "to check status".
Analysis:
- Question 1: Do you notice data is not coming into Cobundu during an installation? NA
- Question 2: Are all sensor affected? Only partially
- Question 3: At which stage is the data missing? Sensors not connecting as frequently to Sensor platform/Cobundu as we would like to
Solution:
After checking if the hubs were still online, some turned out not to be connecting to TTN: so we reached out to the customer to understand what had changed on their side: were the hubs still all in place and plugged in? Had there been any changes to the switch? (We knew the ports were still open on the network, since other hubs on site did manage to connect to TTN.)
After a physical check by the customer, they confirmed that the lights on the hubs that we told them were not working, were different than the lights on the ones that were... A reboot of these hubs provided the solution: the hubs came back online and the sensors were able to connect to the nearest hub again.
Customer C: Sensor health red on all sensors → Change of internet provider
At customer C, we noticed that all sensors were red on the Cobundu sensor health plan.
Analysis:
- Question 1: Do you notice data is not coming into Cobundu during an installation? NA
- Question 2: Are all sensor affected? All
- Question 3: At which stage is the data missing? Sensors not connecting to Sensor platform
Solution:
After checking if data was coming on the sensor platform TTN (which is was not) and if the hubs were still online (which they were not), we reached out to the customer to understand what had changed on their side: were the hubs still all in place and plugged in? Had there been any changes to the switch? Were the ports still open on the network?
After a physical check by the customer, hubs and switch were still OK. They did not make any changes to the network... Or did they? They realized that around the time that data had stopped coming into Cobundu, they had changed internet provider - and had not communicated the necessary ports to open.
Once the ports were communicated and the new internet provider took action, the hubs and sensors came back online.
Customer D: Sensor health red on many sensors → Battery change
At customer D, we noticed that many sensors were turning red on the Cobundu sensor health plan.
Analysis:
- Question 1: Do you notice data is not coming into Cobundu during an installation? NA
- Question 2: Are all sensor affected? Many
- Question 3: At which stage is the data missing? Sensors not connecting to Sensor platform
Solution:
After checking if data was coming on the sensor platform TTN (which is was not) and if the hubs were still online (which they were), we knew we had to take a closer look at the sensors.
We performed a Battery Lifecycle Measurement (Check Payload)(Spacewell internal) and noticed that the battery for all the relevant sensors was depleted.
After having instructed the customer on how to perform a battery change (take out the old battery, then wait 3h for the capacitor to deplete, before putting in the new battery), the sensors came back online.
Customer E: Sensor health red on some sensors → Config error / typo in sensor ID
At customer E, we noticed that some sensors stayed red / did not turn green on the Cobundu sensor health plan after installation.
Analysis:
- Question 1: Do you notice data is not coming into Cobundu during an installation? Yes, this is visible because Last seen date is "never"
- Question 2: Are all sensor affected? Only some
- Question 3: At which stage is the data missing? Sensors connecting to Sensor platform, but not to Studio
Solution:
Sensor ID’s that were red had a typo in them (one dash - too many in the ID). After having removed the typo on these devices, the sensors started sending data to Cobundu.