Is this platform really stable?​

When Google tells us the IoT Core is going to be shutdown, anyone who is running on the "cheap, efficient, secure" Cloud IoT services need to ask that question. I am not only blaming Google. Public cloud providers are harsh. They provide redundant features with a low price to attract customers to make their stock prices great; but when the economy is not as good, such services which only burns cash will be thrown away. Customer will suddenly realize their IoT devices need a new home, just like tenants being kicked out of rental apartments by the landlord. Is there a way to enjoy great features without worrying about such disappointing scenarios? Yeah we have that.

Shifu

Right, it is a new IoT platform.

Shifu has the following features that can replace what you enjoy in Google IoT Core as easily and fast as turning a switch:

1. Device Management and Control: Shifu provides an HTTP interface for users to interact with the devices via REST API. You can just forget about any protocols else because Shifu is always compatible.

2. Security: Shifu can be fully on-prem with a highly secure user authentication system. If you don't trust too much about the internet, alright you can use Shifu without any internet connection!

3. Telemetry Collection: It is one of Shifu's flagship features. You can specify anything that you want to collect by just telling Shifu the name of the telemetry and the destination of the data output, then you can just sit and wait.

OK it is how Shifu can be a great alternative to Google IoT Core. But it only makes it yet another new IoT platform. I know it is not enough. So here are the features turning Shifu into an Iot platform that goes beyond Google:

4. Digital Twin: each device is in fact a containerized service running in Shifu. So Shifu is the most decoupled IoT platform in the world - there is never a single point of failure and each device is as independent as it has already been in real life.

5. Flexibility and Scale: Shifu is making use of Kubernetes to ensure all system resources are managed efficiently and ready to scale.

6. Easiest to Setup: Shifu is the easiest to use IoT platform in the world. You don't need to register an account (well if you like you can still do) to start using Shifu. Everything is already containerized and you just need literally one single action to get everything running; if you don't like writing one line of command, there is a fully functioning Web UI waiting there for you to play with.

7. Multiple Clouds: Besides enjoying the on-prem Shifu, if you still like to have Cloud features, we have that. Shifu Cloud is public cloud agnostic - no single cloud provider is a single point of failure to Shifu.

Shifu is giving you back freedom while still maintaining a high standard that is pain-free, easy-to-use, on-prem, and secure. With all the mist eliminated by Shifu, your vision of the IoT planet will be as clear as, well, the lakes you have seen in those romantic movies, especially the ones shot in New Zealand.

As an IoT enthusiast, you have used millions of device managers. They are all great at doing their job - managing your devices. However, you have also got millions of complaints - they are just too complicated to use.

This complication is actuallly reasonable. A typical IoT environment will have tons of devices, and tons of devices indicate easy-to-mess-up memories, as well as easy-to-scramble threads. That is the curse of a monolith. You had better decouple it - OK, then it makes the coding complex. So now the programmers are facing the trade-off between easy-to-code vs. easy-to-use, and of course, programmers won't make their own life hard.

But should the users always carry the burden? Before now, the answer could be yes as there aren't good solutions to this question; but now, the burden is fully distributed and dissolved - because of Shifu.

Podding your devices​

You may already get it - Shifu is using Kubernetes. K8s is a great devops tool for managing system resources and requirements across multiple pods. As we have already learned from all those large scale clouds, K8s is perfect for decoupling - with its powerful container support, every pod is now completely separated, yet still coordinated together - perfect for the management of the Internet of Things.

Device management is a combination device and management. You have devices already there, now you just need management - and K8s has devoted its entire life into management. Now you just need to tell K8s "lad you need to manage those" to let everything start running. How? Use Shifu.

Shifu makes K8s an IoT native platform. You just need to give Shifu what the device is, the name of protocol it is using, and the list of operations it supports; all those three types of information will be written in a simple configuration file, and then it is done - Shifu will take care of what remains: reading the configuration, building the supported drivers, generating containers, and creating pods and letting K8s know. So K8s has no idea what it is managing - they are still pods, but Shifu knows that each pod is actually a virtual device, entangled with the real device.

This out-of-box experience is what you deserve after you have suffered so many different "advanced device managers" - you should manage your devices to the level above all - Pods.

Introduction​

You may think that the name MQTT stands for "MQ Telemetry Transport". Well, it was, but now not any more. Currently MQTT is just MQTT, not an acronym of anything. Note that MQ is not an acronym either - it is the name of an IBM product, and it does not stand for "Message Queue".

The reason behind is that this protocol has gone beyond telemetry transport. The major focus of MQTT is now the entire IoT space. On its path to becoming the golden standard of IoT communication, MQTT attracts a growing number of users across the world. However, there are only a few resources online that can give us a complete guide to this protocol. And that is what this article series aims to do - to provide a detailed introduction to MQTT for anyone who would like to dig deeper into the IoT planet:

1. Overview
2. How Pub/Sub works
3. Ensuring the QoS
4. Connect MQTT to Kafka
5. Use MQTT to interact with real devices

This article is the first of the Intro to MQTT series. On this page we are going over some basic terms of the MQTT for a quick overview.

A Very Brief Explanation of How MQTT works​

Suppose we have this setup: Publisher: a thermometer in our room, publishing "temperature" topic with tag "room1. Subscriber: a laptop in our room, subscribing "temperature" topic with tag "room1". Broker: running on a server in our room. Topic: "temperature". Tag under "temperature" topic: "room1".

A complete path to the temperature message is:

1. The thermometer publishes the temperature message of topic "temperature" and tag "room1" to broker.
2. The broker checks who is subscribing to the topic and tag, then routes the message to the laptop.
3. The laptop gets the message.

Pub/Sub​

Classic Client-Server pattern requires the client and server know each other. This is an example of strongly coupled system. The issue of that is, the server itself needs to maintain a registry of clients, making it complex, error-prone and somehow single point of failure.

**MQTT uses a Pub/Sub pattern which decouples this system. **

The high level architecture is consisting of three components: publisher - broker - subscriber. Publisher publishes messages of some topics, subscriber subscribes to some topics, and the broker is in between - it filters the messages and ensures the subscriber only gets the messages it wants. We also call publisher and subscriber "clients" of MQTT broker, indicating that they are both the clients of the broker; therefore, in this definition, the "server" is the broker.

The broker filters messages via topic and the tags under a topic. A topic is a "type" or a "class" that can be used to group messages, and tags are "subtype" or "subclass" of a topic. Once a publisher publises a message under a topic and/or tags to a broker, only the subscribers subscribing to the same topic and/or tags will be able to get the message from broker. In this case, both publisher and subscriber need to know the topics and/or tags they are handling.

Message Queue​

MQTT can be considered as a message queue fully tailored for IoT messaging. Comparing to traditional message queue systems in distributed systems like Kafka, MQTT is light-weight and faster in both space and time. MQTT strongly prefers memory to store messages with very low throughput and asks any implementation to support dynamically adding/removing topics, to achieve ultra-low latency and high flexibility.

You can always connect MQTT with other message queues like Kafka to make use of both.

Quality of Service (QoS)​

By its name, QoS defines the quality of the broker's work serving the clients (publisher and subscriber). The clients need to tell the broker what the QoS level is when communicating with the broker.

There are 3 QoS levels:

0. or "at most once" - message can be delivered or not, but there should be no duplicates:

• publisher's message should be delivered to a broker one or zero times;
• subscriber will get a message from a broker one or zero times;

1. or "at least once" - message has to be delivered, but can tolerate duplicates:

• publisher's message should be delivered to a broker one or more times;
• subscriber will get a message from a broker one or more times;

2. or "exactly once" - message has to be delivered, and must be delivered only once:

• publisher's message must be delivered to a broker once;
• subscriber must get a message from a broker once.

The word asynchronous is the core of a pub sub system. Decoupling the message sender and receiver, this system enables the non-waiting nature: each service now just needs to do its own job. Happens to be a perfect explanation of the relationships among people in a tech company, or a financial institution, or a government department, or a football team, or basically every group with more than one human.

Pub​

MQTT broker does not care about what the data is (i.e., payload), as long as it contains the topic so the broker can forward it to the subscribers.

PUBLISH message​

A typical MQTT message of PUBLISH has the following fields: packetId: the identifier of the packet. When the message is configured to always delivered (i.e., qos > 0), the broker needs this identifier to locate the message to retry.

topicName: the name of the topic, it is formatted similar to a directory in a linux file system. For example, we can have "shimokitazawa/beast/tadokoro" as a topic.

qos: level of the quality of service. As mentioned in the previous article, we have 3 levels: 0 (at most once), 1 (at least once), and 2 (exactly once).

retainFlag: if set to true, the broker will retain the last message of a topic; when a new subscriber comes, the broker will automatically send the last message of this topic to it.

payload: the payload. It could be anything in any format.

dupFlag: if set to true, then this message is a duplicate. Used by broker when we have qos > 0 in the retrying scenario.

Broker's job​

It is simple - the broker gets the message, reads the qos level, and sends the message to the subscribers according to the qos. Note that the publisher's work is done when the message is delivered to broker; and any work remaining is none of the publisher's business.

Sub​

Er, it is easier to explain what the subscriber does: subcriber tells the broker "hey buddy I am the subscriber of this topic, if you receive the messages of the topic just give it to me".

SUBSCRIBE message​

A typical MQTT message of SUBSCRIBE has the following fields:

packetId: still, it is the identifier of the packet of the message.

And a list of combinations of a topic and a qos: qos_n: the expected qos of topic_n delivering to this subscriber. This qos serves as the max qos of the topic for a subscriber, for example, if the message has a qos of 2 but the max qos of the topic in subscriber_a is 1, then the message will be delivered to this subscriber with qos being 1 instead of 2.

topic_n: the topic this subscriber subscribes to.

Broker's job​

Upon receipt of the SUBSCRIBE message, the broker will send back a SUBACK message to the subscriber with the metadata of the subscriber (i.e., the max qos). Once the SUBACK is received by the subscriber, the communication is established.

Unsubscribe​

To unsubscribe a topic, the subscriber just needs to send an UNSUBSCRIBE message with the topic to be unsubscribed (of course it also needs a packetId), and the broker will return an UNSUBACK message to the subscriber.

Conclusion​

This is how MQTT publisher, broker and subscriber work in this Pub Sub relationship. Because MQTT is just a protocol, we need to define how the broker identifies the subscriber - you can just maintain an in-memory table to do it, or you can use a more sophisticated way to do it, like using multiple ECC memories to handle a replicated containerized service protected by a distributed locking mechanism to maintain an in-memory table.

You already have a vague impression about the QoS of MQTT - it instructs how is the message delivered; and we have 3 QoS levels and 2 types of messages.

Quick Review

3 levels of QoS: 0 - at most once 1 - at least once 2 - exactly once

2 types of delivery:

1. publisher to broker
2. broker to subscriber.

Also, in the last article we have discussed the SUBSCRIBE message sent from subscriber to broker; and it defines the maximum QoS level that the subscriber can accept, and any message with higher level will be reduced to the maximum level of the subscriber subscribing to that.

0 - At Most Once​

You can always know what it behaves by its name. So if you have QoS level 0, the message will either be delivered once, or not delivered - just try once and no retry. Send and forget about it. Wow, much async.

1 - At Least Once​

I want deliver this message! And I will retry like crazy. Whoever sends the message will keep retry sending the message until it receives PUBACK (consisting of packetId and that is it) from recipients.

Remember we have the dupFlag field in PUBLISH message? Here is how we use it - in every retry following the initial try, this dupFlag is set to true, informing the recipients that this could be a duplicate message.

2 - Exactly Once​

The message is ensured to be delivered, and ensured to be delivered only once. This is the slowest and most complicated QoS level. Here is the entire lifecycle of a QoS 2 message:

First, the sender (publisher or broker) sends the PUBLISH message to receiver (broker or subscriber). "Knock knock!"

Second, once the receiver gets the PUBLISH message, it sends the PUBREC message to the sender, indicating that "I recognize this message, oh it is QoS 2, so take my word as a delivery proof, stop harassing!". Before receiving PUBREC, the sender will keep retrying to send PUBLISH message with dupFlag == true to the receiver.

Third, upon the receipt of PUBREC, the sender learns that it has been delivered, and because it knows the message QoS is Exactly Once, it will throw away the message and stop retrying like crazy. The sender sends PUBREL to the receiver saying "I promise I won't knock again with this message."

Fourth, when the PUBREL arrives, the receiver will send PUBCOMP back to the sender to complete the life cycle.

It is very similar to how we handshake in TCP - using some additional steps to ensure all stakeholders are aware of current status.

If the receiver is not reachable, the sender will keep the message locally in a queue so it can send it when the receiver is online.

In today's article we will go directly into what the topic is structured and how we can define a good topic.

Definition of Topic

A topic is defined as a string and formatted in a linux file system way. In common practice, from left to right, we describe multiple levels of one topic, from highest to lowest, like this:

"earth/antarctica/elderthings/shoggoth"

and "earth", "antarctica", "elderthings", "shoggoth" are four levels of this topic.

Subscription

The subscriber needs to tell the broker what topic it is subscribing to. MQTT gives us freedom to use wildcard characters to match multiple topics at once:

+: matches any single level

#: matches one or more levels (can only be placed at the end)

Let's say we have the following topics:

• "earth/antarctica/elderthing/shoggoth"
• "earth/antarctica/worker/shoggoth"
• "earth/antarctica/migo"
• "earth/antarctica/cthulhu/starspawn"
• "yith/greatrace"

If subscriber A subscribes to "earth/antarctica/+/shoggoth", then it can receive messages from "earth/antarctica/elderthing/shoggoth" and "earth/antarctica/worker/shoggoth".

If subscriber B subscribes to "earth/antarctica/#", then it can receive messages from all four topics under "earth/antarctica".

If subscriber C subscribes to "#", then it can receive messages from all five topics above.

$SYS

"SYS" after a dollar sign is a special "keyword" used by the broker to track and maintain system statistic data. Broker publishes to this topic and no one else can publish to it.

Remember that it is your or whoever implementing the protocol's job to make your MQTT service compatible with the standards above, and by default, MQTT allows you to use almost anything in the topic string, including non-ascii characters.

RTSP is used by most camera in the world, as it is efficient and classic. Well then why the hell is TikTok using RTMP to deliver us real time trolling videos? It is because Abode, the supreme leader of trolling media creation, developed it. Today we are going to have a high level overview of what RTSP is and what it does. First, one thing you should know is that to stream a video, RTSP (Real-Time Streaming Protocol) is not the only protocol you will need to have - RTP (Real-Time Transport Protocol) and RTCP (Real-Time Control Protocol) are important too.

Roasted Turkey with Sweet Potato

RTSP is used to handle the request and request acknowledgement between client and server, just like HTTP.

You send a request in this format:

<method> <url> <version>
CSeq: <seq>
<content>

For example, we can have this:

DESCRIBE rtsp://114.514.19.19:810 RTSP/1.0
CSeq: 2
Accept: application/sdp

And you can expect a response in this format:

<version> <code>
CSeq: <seq>
<content>

For example, we can have this:

RTSP/1.0 200 OK
CSeq: 2
Content-Length: 155
Content-Type: application/sdp
....

In the example above, we send a DESCRIBE method to the rtsp server, asking for the description of all streams on the server; and the response contains an SDP file describing what we just asked, containing a list of descriptions of all streams, like the address, type, communication protocol, encoding, etc. We will talk about the SDP file later in this series.

For a compelete receipe about what each method does, take a look at this official menu: https://www.rfc-editor.org/rfc/rfc2326#page-29.

Note that RTSP itself is nothing about transmitting videos - it just tells the stakeholders that "I'm gonna do this".

The real work is done by RTP and RTCP.

Red Trout Poached

RTP takes care of transmitting the data of video. It uses UDP by default (of course for a real time data), packages the video data (video and audio) and transmits them.

It's header is like this:

Woah too many new terms! But you just need to know that it is a typical header of RTP to identify the data being transmitted. We will go over the details in the future article about RTP in this series.

Resins and Tomato Cooked in Pumpkin

RTCP is all about synchornizing. While RTP is transmitting videos, RTCP is sending the status/metadata of the data being transmitted by RTP. You use that for monitoring the video quality, controlling the load, throttling, etc.

It has five types of messages: Sender Report (SR), Receiver Report (RR), Source Description Items (SDES), BYE, and APP. Well, many of them are self-explanatory as those are acronyms. BYE is just to say this is the end of the transmission; and APP can be considered as "customized", as it is telling us the format and type are not registered and are experimental, so techinally you won't need it.

We will discuss more about each type in the following article about RTCP in this series.

Millions of questions must have arisen. What the hell is SDP? What does an RR message look like? How on the earth can I build an RTSP server? Can I stream myself playing Genshin Impact so I can collect enough donations to pull my next waifu? Just wait a little bit for the next articles...

IoT application development and management platform Shifu (https://github.com/Edgenesis/shifu) is officially open source, entering a new phase of open source collaborative iteration. Developers can login to GitHub and search Shifu for repository information, and click star to support the project or fork the project. Shifu provides users with a transparent framework for full-scene device hosting and integrated software development. By using Shifu, developers can connect, monitor and control any IoT device more simply.

The innovative advantage of Shifu is to empower devices with a thinking "digital brain" through the digital twin technology within the transparent framework. The digital twin will reflect the real-time state of the device, and it is the same to operate on the digital twin and on the device itself. Standardized interfaces will be generated when IoT devices are integrated into Shifu to enable remote interaction. Shifu can achieve northbound data collection and southbound command control for all devices in the scene through the platform layer.

Shifu provides a bridged device interconnection solution with a microservice architecture that makes device capability modules invocable and reusable, with the goal of achieving easy integration to various heterogeneous devices through configuration files. Currently, Shifu has achieved access to IoT devices via HTTP, MQTT, TCP Socket, RTSP, OPC UA and other protocols, and has also integrated Siemens S7, Hikvision (HIKVISION) and other devices that communicate via private protocols.

As a cloud-native framework, Shifu extends the resources of K8s through the CRD function of Kubernetes to achieve high availability, static domain name, service management, etc. Shifu can support any kind of configuration for any device. When a physical device is connected, Shifu recognizes and launches that device's digital twin deviceShifu as a K8s Pod. By accessing the deviceShifu interface, developers can access all the features of an IoT device, while programmatically defining features that the device would not otherwise have.

The cloud-native Shifu makes system operations and maintenance much less difficult, and application developers can manage operations and maintenance through one set of Kubernetes infrastructure. Shifu will turn Kubernetes to the standard underlying architectural for IoT development, and bring container orchestration technology to the IoT software development ecosystem.

Yongli Chen, project founder of Shifu expressed his gratitude to the open source community at home and abroad for their strong support of Shifu open source. He said, "We hope to make Shifu a universal open source base for IoT development, so that IoT developers can also enjoy the benefits Android and iOS bring to mobile development through Shifu. We can't wait to create the era of the Internet of Everything with Shifu's supporters!"

Wei Guo, Apache Foundation Member, Mentor of Apache SeaTunnel PMC of Apache Dolphin Scheduler, and founder of ClickHouse China Community said that the advent of the cloud-native era has reconfigured the infrastructure of all enterprises, and the open source of Shifu will further help enterprises with their IoT management. He sincerely hopes that Shifu project will excel itself in the future.

Tina Tsou, Chairman of LF Edge Board said that LF Edge pays close attention to promising edge computing projects around the world. Shifu's cloud-native architecture is based on Kubernetes, which is very innovative, further unleashing the control and management capabilities of edge devices and providing application development engineers with a unified base for cloud edge collaborative management. They will continue to pay attention to the open source process of Shifu, and expect Shifu to drive edge computing technology toward more efficient scenario implementation.

Kevin Wang, CNCF Ambassador and Huawei Cloud Native Open Source Lead said that edge computing has become one of the most critical directions for cloud native development. We are glad to see the open source of Shifu project, which enriches the ecology of cloud-native and IoT, and wish the Shifu open source community better and better.

Linbo He, AliCloud technologist and OpenYurt startup member said that the cloud-native edge computing space is gaining momentum and is favored by more and more companies. Shifu takes cloud-native as the entry point and introduces declarative API abstraction IoT lifecycle management, which elegantly solves the traditional IoT industry's painful problems of long development cycle, complex large-scale deployment and high O&M costs. At the same time, Shifu takes the vision of letting IoT practitioners experience the fun of technology and biold a software-defined world. At the point when Shifu is officially open-sourced to the industry, we highly recommend practitioners in cloud-native and edge computing industries to experience and use Shifu. We wish Shifu contributors and participants to have fun, enjoyment and satisfaction in the open source community.

Pengcheng Zhao, Senior Solutions Architect of Microsoft IoT said that as IoT, edge computing, digital twin and other application scenarios gradually mature, cloud-native technologies are also exerting increasingly important impact in IoT-related solutions. Shifu is such an IoT edge computing framework based on Kubernetes technology, and has very cleverly integrates the concept of digital twin and container technology, though this set of platform seems slightly green at the moment, the overall architecture is very flexible. We expect the open source Shifu will continue to broaden its influence, accelerate code and product iterations, perfect itself in industrial scenarios. So as to become the trusted open source solution for IoT in the industry.

Jianhui Tao, founder of TDengine, the world's leading open source timing database, congratulated Shifu on its official open source, saying that choosing to open source is a key move for China's infrastructure software and middleware to establish a leading position in the industry, and a brave choice for the global market. Open source means taking the initiative to accept the attention of peers in the community, while accepting the opportunities that openness brings, so that the technology can really be implemented in real-life scenarios. Shifu's cloud-native architecture supports containerized deployment more perfectly. He hopes that Shifu could explore more possibilities in IoT scenarios with TDengine 3.0 in the future.

Fahua Jin, co-founder of EMQX, a modern data infrastructure provider for IoT and a leading global MQTT Broker open source community, congratulates Shifu on its official open source. Shifu supports the development and deployment of IoT applications through cloud-native architecture, providing a new innovative possibility for digital transformation of the Internet of Things. It is hoped that the Shifu open source community, together with EMQ's open source family of products, will build distributed cloud-native IoT solutions from cloud to edge for community users.

In the future, Shifu will support automatic deviceShifu generation, declarative API, advanced Shifu controllers, device grouping, multi-layer packaging, etc. We expect open source contributors to provide Shifu with more functional iterative directions and real scenario requirements. Shifu was created with the intent of equipping every IoT device with a Shifu, letting software define the world, solving infrastructure problems, and making developers and O&M staff happy again in the IoT world. Shifu open source community will work with global developers for more IoT devices to better serve humanity.