The Power of APIs

APIs are reshaping how companies do business. Once solely the domain of software engineers, they have grown to affect all levels of company.

What is an API?

An API is a set of clearly defined methods of communication between various software components. A good API not only makes it easy to get at your data but also combine it with additional data.

Why are APIs important?

Screen Shot 2017-08-24 at 10.42.35APIs are essential for interpretability. Deployed sensors and sensor networks will be part of the infrastructure of buildings for the foreseeable future and beyond, a trend that has gathered pace over recent years and is expected to continue for a long time to come. If the data only goes from the sensor into a black box, there is no built-in flexibility. Not only can you now see the visualizations clearly and easily but you can also integrate the data with the existing CAFM System. The same data can also be integrated into the existing systems already in place and familiar to the client whether meeting room booking systems or building security systems ensuring maximum accessibility and control

Benefits of an API:

  • Freedom from vendor lock-in – easy independent access to data
  • Better security – the more people who can see and test a set of code, the more likely any flaws will be caught and fixed quickly.
  • Customizability
  • Flexibility
  • Interoperability – our list of current integrations is always growing. Contact us to see if your system can be linked together
  • Auditability
  • Try before you buy – why not do a pilot of 5 devices before you rollout to a large scale deployment?

A closed platform can be damaging long-term and expensive. Your data becomes locked up and customers can find themselves very much at the mercy of the vendor’s changing vision, requirements, dictates, prices, priorities, and timetable.

More about the OpenSensors APIs

cropped-os_badge_logoOpenSensors API’s enable you to extract data by a variety of means; project, location, organizational departments, types of sensors, types of messages etc. We have integrated with many CAMF systems, meeting room booking systems, and building security systems. It is a constantly reviewed and expanding list with many other integrations planned on our road map. It is not only possible to build integrations with the visualization systems you already have in place but also use our canned visualizations. The data can also be seen within our systems, but it’s really targeted at enabling flexibility and multi-purpose use of the data.

OpenSensors has an open data mission

Hardware is expensive. Implementation can often be difficult to reduce the data as much as possible. Unless the data is used by multiple systems effectively you are going to have to have a different use for the same data within the building. With closed systems, the same sensors will have to be redeployed for each specific set of data required. This does not make sense to us. Our basic mission is to make these data sets easily accessible. Whether that’s private data or open data, we want you to be able to collect the information and make it immediately reusable and interoperable with all the different systems that you are already or will be using.

Our systems are similar to the electrical voltage standards that allow you to purchase any appliance and plug it in delivering power to your device. OpenSensors allow you to plug in the different devices as needed. Our API allows your applications to connect and extract the data when and how it is needed. The data is no longer locked away in a proprietary system holding you to the ransom of one vendor and puts you back firmly in control.

By harnessing the power of data aggregation across smart building ecosystem, every stakeholder wins. Over and over again, we’ve seen that enriching the experience of all parties (business – agent – customer) generates a higher degree of success and satisfaction

About OpenSensors

OpenSensors aggregates data from a variety of sensors for the next generation of smart Building Management Systems. Our dashboards help you make the most effective use of your office space. With experience in helping more than 100 companies to combine data from new workplace sensors seamlessly, interoperating it with existing and familiar mobile and desktop systems, we aim to give you a fully comprehensive overview.

Lessons Learned From First Generation IoT Installations

At first glance, Wi-Fi-based sensors seems like a good choice for a non consumer facing sensor network, however, we have discovered that Wi-Fi has some significant drawbacks.

Access

One of the biggest drawbacks to Wi-Fi enabled sensors in a corporate environment at many of the companies is gaining access. Corporate IT often has valid security concerns of hundreds if not thousands of sensors joining the network and have deployed corporate firewalls that block any access. Often this means that we are not allowed to spin up our own Wi-Fi network in order to have a gateway for a customer’s IoT sensor network. If IT has already deployed a Wi-Fi network they are rarely willing to provide the passwords to allow the IoT network devices and gateways to take advantage of it. Relying on corporate Wi-Fi can make on-site installations and maintenance extremely complex and painful. The whole project becomes dependent on the goodwill of a network administrator for access every time maintenance needs to be performed.

Power

Wi-Fi has good transmission range but that comes with a cost of high power usage. With a short battery life, maintenance costs for Wi-Fi sensors are higher than low-power alternatives. One wireless protocol that is we see in many successful deployments is LoRa because it offers long transmission range at a much lower battery usage than Wi-Fi.

Moving to LoRa and other long range protocols

If you follow our blog and publications, you will notice we have been talking a lot about network technologies, this isn’t a coincidence. We have spent a long time evaluating and piloting these stacks with our community.

Network access and battery constraints are driving the move to long range networks and off WiFi for many IoT installations. LoRa is working well for us so far for a number of use cases most of our customer spin up a private network. The ecosystem of providers is maturing and we are finding a lot of companies who are adopting existing sensors for their networks Gateway providers such as Multi Tech provide good support for the long tail of small scale (> 250 sensor installs) hardware providers to thrive.

LoRa is a wireless protocol that promises between two and five kilometers transmission range between sensors and gateway, if you haven’t already done so please read our introduction to what it is. With a separate LoRa network, facilities and/or operations can install and manage the whole operation without the access and security issues of using the corporate Wi-Fi network. A typical network will have hundreds of sensor devices sending messages to a gateway. The LoRa gateway is a self contained system, we can have the LoRa network sit completely outside of the corporate firewall (GSM) and minimize IT security concerns.

One LoRA gateway can normally cover an entire real estate. This can significantly reduce infrastructure, deployment, and administration costs compared to other shorter range wireless options like Zigbee or Bluetooth that requires complex installs. Our aim is to have a system that non technical engineers can roll out and support, more on how to do this on later blog posts, but in most cases the OpenSensors team is the equivalent of ‘2nd line support’ to the onsite team who have intergrated our apis to their helpdesk ticketing systems etc.

LoRa networks can be public or private. An example of a public network is The Things Network, we continue to work with and support that community. Most current commercial projects are running private networks at this time but will be interesting to see how that evolves over time.

To conclude, LoRa is working well for us at the moment but we will keep researching other networks to enable us to understand the pros and cons of all the network providers. Sigfox is a very interesting offering that we will properly test over the next few months, for example.

Getting to Grips With IoT Network Technologies

How sensors communicate with the internet is a fundamental consideration when conceiving of a connected project. There are many different ways to connect your sensors to the web, but how to know which are best for your project?

Having just spent the better part of a week researching these new network technologies, this brief guide outlines the key aspects to focus on for an optimal IoT deployment:

Advanced radio technology

  • Deep indoor performance – networks utilising sub-GHz ISM (industrial-scientific-medical) frequency bands such as LoRaWAN, NWave and Sigfox are able to penetrate the core of large structures and even subsurface deployments.
  • Location aware networking – a variety of networks are able to track remote sensors even without the use of embedded GPS modules. Supporting sensors moving between hubs – with advanced handoff procedures and innovative network topologies mobile sensors can move around an area and remain in contact with core infrastructure without disrupting data transmission. Intelligent node handoff is also crucial for reducing packet loss, if the connection to one hub is hampered by passing through particularly chatty radiowaves, the node can switch to a better placed hub to relay it’s crucial payload.
  • Interference resistance – the capability of a network to cleave through radio traffic and interference that would ordinarily risk data loss.

Low energy profiling

  • Device modes – LoRaWAN is a great case and point with three classes of edge node: the first, Class A, allows a brief downlink window after each uplink upload i.e after having sent a message, the sensor listens in for new instructions; a Class A node appoints a scheduled downlink slot, the device checks in at a certain point; and the last, Class C type nodes, listen for downlink messages from LoRaWAN hubs at all times. The latter burns considerably more power.
  • Asynchronous communication – this enables sensors to communicate data in dribs and drabs where possible, services do not need to wait for eachother thereby reducing power consumption.
  • Adaptive data rates (ADR) – depending on the quality of signal and attenuation, modern networks are able to dynamically allocate data rate depending on interference, distance to hub etc. This delivers real scalability benefits, frees up space on the radio spectrum (spectrum optimisation) and improves overall network reliability.

security

  • Authentication – maintains data integrity by ensuring the sensor which is publishing that mission critical data really is that sensor and not an impostor node. Ensures information privacy.
  • End to end encryption (E2E) – prevents tampering and maintains system integrity.
  • Integrated security – good network security avoids potential breaches and doesn’t place the onus on costly, heavily encrypted message payloads.
  • Secure management of security keys – either written remotely on the initial install or embedded at manufacture, security keys are fundamental to system security. ZigBee’s recent security issue shows how not to manage security keys, by sending them unencrypted over-the-air to devices on an initial install.
  • Receipt acknowledgement – ensures mission critical data is confirmed received by network or device.

Advanced network design

  • Full bidirectional comms – enables over the air (OTA) updates, enabling operators to push new firmware or system updates to thousands of remotely deployed sparse sensors at the push of a button. This is critical to a dynamic and responsive network. As with device modes mentioned previously, bidirectionality allows deployed devices to function as actuators and take action (close a gate, set off a fire alarm etc) rather than just one-way sensors publishing to a server.
  • Embedded scalability and consistent QoS – as load increases on a network so too does the capacity of the network. This takes the form of adaptive data rates, prevention of packet loss by interference and channel-blocking, the ability to deploy over-the-air updates and ensuring the capability to add nodes, hubs and maintain existing assets without impacting on overall network service, perhaps through automatic adaptation.

There are also a number of legal, cost, market and power focused aspects worth considering that I shall not cover here. But, critically, it’s worth mentioning that the majority of these technologies operate on ISM (industrial – scientific – medical) frequency bands, and as a result are unlicensed. These bands are regulated and there are rules, however anyone operating on these bands can do so without purchasing a license. Notably, you don’t have sole ownership of a slice of the spectrum, you don’t get exclusive access. Therefore, with a variety of other vendors blasting away across the radio waves, these technologies encounter significantly more interference than the licensed spectrum. However, the new networks, LoRa, Sigfox, NWave etc are based on protocols and technologies designed to better sort through this noisy environment, grab a channel and send a message.

Understanding that the airwaves are a chaotic mess underlines the importance placed on features such as adaptive data rates, node handoff and power saving methods such as asynchronous communication. Wired networks do not have to consider such things. But for most it’s not just a case of who shouts loudest wins. The majority of wireless protocols ‘play nice’ opting for a polite “listen, then talk” approach, waiting for a free slot in the airwaves before sending their message.

Some protocols such as Sigfox forego such niceties and adopt a shout loud, shout longer approach, broadcasting without listening. A typical LoRaWAN payload takes a fraction of a second to transmit, Sigfox by comparison sends messages 3-4 seconds in length. However, if you just broadcast without listening, Sigfox must therefore operate with severe cycle duty limitations, which translate into a limited number of messages sent per device per day and severe data rate limitations.

These choices also translate into varying costs, and critically, into battery life limitations and gains, the crux of any remote deployment.

See this link for a matrix of the major technologies currently vying for network domination.

Taking the Air at the Turk’s Head

5258374476_2ce1b88c66_b

OpenSensors are pioneers in open data and the internet of things, surfacing a wide range of data sets for open analysis. As an open data aggregator we deliver content over a common infrastructure; whether air quality or transport data, you only have to think about one integration point. Future cities need low data transaction costs for friction free operation, bridging technical gaps slow progress, so keeping the number of integration points low makes sense everybody.

Our journey starts here, as we build out our open data content expect to see more stories, more insight and hopefully some catalysts for positive change.

Before our first story, consider what will make open data and the Internet of things useful.

We must bridge the gap from data to information, allow consumers to abstract away the complexity of IoT to ask questions that makes sense to them.

Take data from the London Air Quality network (LAQN), the network is sparse so it’s improbable our need maps directly to a sensor. By coupling some simple python code with OpenSensors data we’ll mash some LAQN data together to get some insight about air quality in wapping.

In this story I’ll show how we can bridge the information gap with some simple code, yielding valuable insight along the way!

Chapter 1: OpenSensors.io Primer

First a quick primer on how data is structured in OpenSensors.io (for more detail check out our forum and glossary of terms)

  • Devices – Each connected ‘thing’ maps to a secured device, things map one-to-one to a device
  • Topics – Data is published by devices to topics, a topic is a URI and is the pointer to a stream of data
  • Organisations (orgs) – An organisation owns many topics and is the route of an orgs topic URI
  • Payloads – Payloads are the string content of messages sent to topic URI’s, typically JSON

Also check out our RESTful and streaming APIs on the website for more background and online examples.

Chapter 2: Putting JSON to Work

You can use the OpenSensors REST API to gather data for research, but it comes in chunks of JSON which isn’t great for data science. For convenience i wrapped up some common data sources for London into a python class. Since IoT data is rarely in a nice columnar form it’s valuable to build some simple functions to shape the data into something a bit more useful.

Chapter 3: Introducing the Turks Head

I’m fortunate to spend a lot of time in Wapping, in and around the community of the Turk’s Head Workspace and Cafe, but unfortunately we don’t have a local LAQN sensor. With a bit of data science and OpenSensors.io open data we can estimate what NO2 levels might be around the cafe and workspace.

A simple way to estimate NO2 is a weighted average of all the LAQN sensors, in this case we derive the weights from the distance between the sensor and our location. Since we want to overweight the closest sensors we can use an exponential decay to deflate towards zero for those far away.

For the Turks Head sensors in Aldgate, Southwark and Tower hamlets and the City are the closest and have the biggest impact on our estimate.

Chapter 4: Getting into the Data

With our air quality time series, and our weights we can dig into what our estimates for the Turks Head look like (NO2 * weight). Here’s the series for NO2 over the last 20 days, it looks like the peaks and troughs repeat, and the falling or rising trend is persistent in between.

Trend followers in finance use moving averages to identify trends, for example the MACD indicator (moving average convergence divergence). MACD uses the delta between a fast and slow moving average to identify rising or falling trends, we’ll do the same. For our purposes we’ll speed the averages up using a decay of 3 and 6 periods (LAQN data is hourly and we are resampling to give estimates on the hour).

What can we conclude from the charts for The Turks Head? From the left hand chart we can see the data is little noisy, with a flat line showing some missing or ‘stalled’ data. Looking at the 3 and 6 period decayed averages the data is smoother, with the faster average persistently trending ahead of the slower one.

Even with fast moving decays the averages cross only a couple of times a day, showing persistence when in trend. So using a simple trend indicator and the LAQN we can build a simple air barometer for the Turks Head.

Good 3 period exp average < 6 period average (green) Bad 3 period exp average > 6 period average (red)

This is helpful because, given a persistent trend state, where we have a ‘good’ air now, we’ll probably have ‘good’ air for the following hour.

Chapter 5: What’s the trend across London?

So we now have means of defining how NO2 levels at the Turk’s Head are trending, but is the trend state predictable over a 24 hour period?

Remember we define good or bad air quality trend as:

Good ‘fast’ average < ‘slow’ average = falling NO2 Bad ‘fast’ average > ‘slow’ average = rising NO2

If we aggregate data into hourly buckets we can visualise how much of the time, over the past 20 days, a sensor has been in a up trend (‘good’) for a given hour.

x = hour of the day y = percentage of bucket that is in a ‘good’ state

We can see that for each 1 hour bucket (24 in total) there is a city wide pattern; if we aggregate across the city (using the same measure, the percentage of sensors in up or down trend) we get an idea of how NO2 trends over a typical day.

Our right hand chart shows the percentage of ‘good’ versus ‘bad’ NO2 sensor states across London over the past 20 days (collected from about 80 sensors over 20 days)

Now this is a really simple analysis but it suggests the proportion of ‘good’ trends across London is high before 7am, and then falls away dramatically during the morning commute. No surprises there.

But the pattern isn’t symmetrical; after peaking around lunchtime, when only ~20% of the cities sensors having improving NO2, NO2 falls throughout the afternoon. From a behavioural standpoint this makes sense; there is a more concentrated morning commute relative to the evening. Most of us arrive at the workplace between say 8 and 9am, but in the evening we may go to the gym, we may go out for dinner, or just work late. The dispersion of our exits from the city is wider than when we enter.

Chapter 6 – PM versus NO2

So we have considered NO2 as our core measure, in part because there are more sensors in the LAQN delivering this data than particulates. But let’s consider particulates for a moment, LAQN deliver PM10 and PM2.5 measures, the definition can be found here.

Our temporal curves for particles differ from NO2 taking longer to disperse during the evening rush hour (remember we are measuring percentage of sensors in a ‘good’ state). As a measure of air quality NO2 builds up faster, and decays faster once peak traffic flows have completed, whereas particles linger only fading deep into the night (on average).

Closing Thoughts

In our data set, NO2 and PM measures differ in their average behaviour over a typical 24 hour period.

  • Behavioural interventions will need to consider whether particulates or N02 are the most impactful.
  • How can we communicate air quality to our citizens, and relate their personal needs to the measures most impactful on their lives?
  • Do we need additional sensors to create a more dense air quality resource? How can we allocate funds to optimally support network expansion and air quality services?
  • Knowing the characteristics of a sensor (location, calibration, situation (elevated, kerb side, A or B road) will improve estimates, how can we deliver this meta-data?

Plenty of food for thought…………..information

Notes and Resources

Our stories are quick and dirty demonstrators to promote innovation and should be treated as such. All data science and statistics should be used responsibly 🙂

All of the code supporting this can be found on github with data sourced from opensensors’ LAQN feed, and I use a postcode lookup to get long/lat locations for wapping. I’ve also taken some inspiration from https://github.com/e-dard/boris and https://github.com/tzano/OpenSensors.io-Py so thanks for their contribution!

http://www.londonair.org.uk/ https://www.opensensors.io/