Data Handling in Manufacturing | Jeremy Theocharis

Chris Engelbert: So with that kind of data, I guess people are tend to self host probably not a very typical, like, Hey, we have a SaaS platform, just store everything. Jeremy Theocharis: It's only the control plane. It's only that allows people or services to access the devices. So we never store any of the customer data on our servers. Everything is in motion. This is a pay patent pending end-to-end encryption. So the user, we're using the management console. It's end-to-end encrypted with the devices. Chris Engelbert: You actually made a good bridge because you already mentioned Kubernetes and you mentioned Timescale. So how is the infrastructure build up, especially if it's not a cloud infrastructure, but set up by customers? Jeremy Theocharis: We wouldn't start as a system integrator and we needed to install this type of tools again and again. So we decided, okay, what about we bundle it together? Chris Engelbert: Hello everyone. Welcome back to this week's episode of Simplyblock's Cloud Commute Podcast. This week, you know the spiel, another great guest. I say that every single week and it's true. But this week actually a little bit of a different thing. Somebody, I think at least we haven't met before. We talked a few times but so far the meeting is still ongoing, which is weird because we're just living away, like, what is it, 20 kilometers or stuff for the people in the States, that is less than 10 miles. Jeremy, thank you for being here. Maybe just give us a quick introduction. Jeremy Theocharis: Yeah, thank you. Thank you for allowing me to be here. I'm Jeremy, co founder and CTO of the Unite Manufacturing Hub. And I'm responsible at the UMH, how we call ourselves, for product engineering and also developer advocacy and community. Chris Engelbert: All right. So you have a full day job or a full time job, more than one. Jeremy Theocharis: Yes, it's startup. You, you also worked previously in startup. So, you know, it's way more than a full-time job. I just remember, I think it was this Monday where I, don't know, I worked like 4:00 AM in the morning just to get some features done. I mean, it was the topic was always my passion. So I basically made my passion and my hobby made it to my work. So for me, it's absolutely fine to program a night through. However, it always has some downsides. When you code the entire night through, you can't be at 9am in the check in. Chris Engelbert: As long as there's pizza. Jeremy Theocharis: Didn't order pizza. I think that's a little bit cliche for coding nights. Chris Engelbert: Is it though? Jeremy Theocharis: My friends, a colleague of mine, they love to put like a pizza in the oven when they're coding. Maybe at least for me. Chris Engelbert: All right. Fair enough. Yeah. I know what you're talking about. I mean, Simplyblock is still very small. We're 18, 19 people. Three people, not in engineering including me being the one man, almost one man marketing team. So yeah, I, hear you. Small startups. It's fun. It's great. I always think it's the good type of stress and the good type of pressure. The one that doesn't make you run into a burnout. But it is stressful. All right. You said UMH, the United Manufacturing Hub. Maybe tell us a few words about it. Jeremy Theocharis: Yeah. So UMH is a central platform that provides all the data in your factory in a single point. And it extends the so called automation pyramid with a new data layer. So how can you think of it? I assume most of the people who are watching this or hearing this, they come from pure IT. So how does it work? How does manufacturing work? Things are digitalized in manufacturing, but they are still in the nineties. So for example, if you have a production machine, let's take- what do I have here? Yeah, a machine that produces these bottles. This is one of our- this is our mascot Otto. So a machine that produces one of these bottles. So you take metal sheets in, I don't know, I already studied mechanical engineering so currently I'm also thinking about how you would produce it, but you would probably just missing the English terms. However, you would have like a single metal sheet coming in, then you have some bending. I'm using wrong English word, but nobody cares. Form it into this. And it is obviously not done manually. So there is electric stuff in there. There is even a controller in there that you program. However, this control is called PLC. But you can't program it in C or any other IT language that you know. Do you know PLCs? Chris Engelbert: I do. I do. Jeremy Theocharis: Yeah. So it is digitized. And you can also, so for example, for a simple thing like this, you would have maybe two machines connected to each other. One that prepares like the metal sheet and moves it, the transport band. The one that does it, and one that maybe puts it somewhere else. And that would be also connected with some kind of the so called field buses. So you can connect machines with each other, but it's not IT. It's OT. They kind of split up somewhere in the 90s. So you have all of this digitization. You have these, it's called automation pyramid, on the bottom you have sensors, then you have PLCs, then you have a lot, was this on top of it, every layer can just talk with the layer directly above it, or directly under it, but now if you could think about Industry 4.0, if you have a question like, okay, but how many of these bottles were now produced? It gets quite difficult because every time you move a layer up, the data is always down sampled, so you don't have data from all the layers in the same place. It's always the sensors on the bottom have a very high frequency. PLC is also more frequency, but then on top, maybe you have only the total amount of produced bottles at the end of a month or something like this. So what you do is we extend it. We keep the control that you have there. And we send with a new data layer on the site. So that you can take all the data out there and work with it. So that's kind of a basic introduction. Chris Engelbert: Right. For the people that are just listening to the audio, he's actually pointing at, what is it, a warming bottle, like a metal sheet warming bottle. Which might also be CNC drilled. I don't know. Probably not. It would be a lot of waste. Jeremy Theocharis: Yeah, it would be a lot of waste and it would take, I don't know, like two hours or something to drill it with probably metal sheets, that gets bended. Chris Engelbert: Right, right. So that means you're collecting the actual metrics or whatever from the machines itself, and you store those. What are use cases for that? I can imagine that for certain industries like auto manufacturers or car manufacturers, auto car manufacturers, or companies, in that kind of space that you might have to give, or you might have to store this information for a long period of time. In the past, I think, people call both systems like historian databases. So I think that is one use case, but it's probably not the only one. Jeremy Theocharis: The biggest part is actually to have the data available in real time. This alone is it's not like typical business use case, but it's the most, it's called, currently people call it Unified Namespace. It's the biggest value proposition of us. It sounds super stupid. You have to have it in real time in a single place, but no, it is really helpful. If you haven't had that so far, if you don't have a real time visualization of the data, then you don't know what you're producing. Then you're missing out on a lot of things. And from this real time thing, data is then, can also be stored in a, we call it a store in, but in the end it's a TimescaleDB, so Postgres with a time series add on. the use case that you build on it, various ones. So like, a common one, for example, energy monitoring. Like what I said with the automation pyramid, it's hard now to do like deep dive analysis in there. So you need to get sensors, but you also don't want to mess with existing sensors. So you add like new sensors to it and you send the data to a message broker. So that, and then part of the data gets stored, or maybe even processed with stream processing. So that you have a dashboard with the energy consumption of your factory. I know this sounds super, super basic and what it's like, wait, this doesn't exist yet? No, it doesn't. And there's this year there was like a regulation coming, I'm missing, Energieversorgskassette or something like this in Germany, which forces companies to do this in the next couple of years. So if you need to put a law, then you don't have it. This alone isn't that helpful because you could say, yeah, just buy any new sensors and you now have a nice dashboard. The real value comes when you start adding more and more use cases to it. So, imagine now you have the energy data and now you also have the data from the machines, whether it's running or not. And suddenly you can do also do energy optimizations. Like why is the machine running? Sorry, why was the machine stopped? But the energy consumption was still high. Did maybe someone forgot to put the heater off? Whatever. Just so, so one single use case isn't why you would use UMH or buy UMH. The real value comes by having a lot of cases there and start combining the data across of it. And then I think I could mention a lot of more use cases, but in general, it's always about this, I would say as an IT person, kind of like this very obvious monitoring, alerting. Yeah, you can also do like more complex stuff. You could actually also send data back and you can control the production, et cetera, but for most companies, it's just about this basic monitoring orders and to try to correlate all the data. Chris Engelbert: Yeah, I see. It sounds a little bit like we've, or that I've done with a couple of other people, like years and years ago, where we actually used a Siemens SPS for an injection mold machine just to figure out when it runs out of flakes and stuff like that. So you have like, as he said, real time alerting, like, Hey, the machine is empty. Go. Refill it now. Like every second that it stops costs a lot of money. Jeremy Theocharis: Yeah. Yeah. And to combine it. I mean, this, every use case you can do also like, separately. You could now also do it in the classic automation pyramid. You could configure the PlC.So SPS is the German, Speicherprogrammierbare Steuerung. And PLC is the English one. So we're talking here about the same thing. You could actually program it to send directly data REST call to somewhere. But you want to be flexible about it. You also want to- you want to have first a message broker and now you can subscribe to it. And maybe you want to trigger multiple things. The ejection molding machine is stopped. Maybe you want to show it on the dashboard, but maybe you also want to have it historically stored, but also you want to have it alerted. You also want to alert maybe multiple people or multiple systems. Chris Engelbert: Maybe you want to fire somebody because you let the machine run empty. Jeremy Theocharis: This is always the topic that people kind of can't hear, but, you know, how the machines are set up is, or the system is set up, you never have a one on one tracking of the person behind it. This is not allowed in a lot of European countries. You can only do it if you have a group, I think of at least six people that could have worked theoretically on the machine. Yeah. It's exactly the point I'm- important thing to say how you can actually do it. Chris Engelbert: Yeah. I think it's an important thing and an important point that people are not allowed to track that, for exactly all the reasons you would might have in countries that don't have like, strict or good labor laws, right? The other thing you pointed out is like, that people might be surprised that is not a thing or that is a very new thing. And just because before we started the recording, we were chatting a little bit about what I've done before. I actually joined Simplyblock and before that Timescale because I had my own startup in the animal husbandry space. And it's kind of the same thing. When we showed the first, like, prototype, not even an alpha version, but first prototype of like real time monitoring, 24/7 monitoring, the farmers were just like mind blown. It was like, what do you mean? You can buy like an IOT smart thermometer, whatever, like in every single, I don't know, supermarket these days. It was all new to them. Anyway, sidetracked, by the way, if you are interested in time series and, specifically Timescale, we talked to Mike just a few weeks ago. I'll put the link somewhere, show notes, the I, whatever. Jeremy Theocharis: Sort of brought my TimescaleDB. I think I have a couple of TimescaleDB merch here. Chris Engelbert: Me too. So, with that kind of data, I guess people are tend to self-host. It's probably not a very typical like, Hey, we have a SaaS platform. Just store everything. Jeremy Theocharis: Yes. So everything, also how you use UMH, also how it's set up. Even though we have a SaaS type of solution, it's called the management console, but it is only for, it's only the control plane. It's only that allows people or services to access the devices because, in the end, how it's all set up, it's, there are multiple DMZs behind it. So a tool like this is helpful. And how we did it is there is a backend running. And for a lot of companies, this is now critical. Oh no, you can now access everything. How we set it up is by disallowing ourselves from doing anything. So we never store any of the customer data on our servers. Everything is in motion. And additionally, this is a patent pending end-to-end encryption. So the user, when using the management console, it's end to end encrypted with the devices. So even though we, how hosted, I mean we have some type of information similar to Bit one. Bit One also has your phone number and I don't think even the name, but has like the IP address and how many messages you're sending, whatever. Similar to that, we have no clue what's going on. So it's important that everyone self hosts it, because all the companies they want to have in their data infrastructure. And for us is we encourage it because we don't want to have to do anything with the data because it's too important. And all the companies want to self host it. But then with our management console, you can actually control those instances and manage it. Chris Engelbert: And I think from the customer premise, you basically only have outgoing connections, so you never can connect to anything. It's only outgoing and connecting to your management console. Jeremy Theocharis: Exactly. So there's this multi layer approach. So you have an instance and there are like sub instances that connect to it because of demilitarized zones, etc. So, everything's just outgoing connections. There's a lot of the failure modes that I've seen very often with these type of IOT platforms, whatever. Yeah, just open firewall rules. So it works quite easy in the proof of concept, but then, It fails at scaling. Chris Engelbert: I hear you. I hear you. Especially with multiple layer DMZs, you have like six, seven firewalls in between, like, yeah, so much fun. Been there, done that. Jeremy Theocharis: And with the architecture we have, it's not a big problem. We even set up everything that you get also the updates and everything that you get or that you need is just the access to a single domain. So that all the updates that you get, everything that's, because in background, it sets up Kubernetes, it does a lot of things in the background, it's entire- It's not like a monolith. It's an entire Kubernetes K3S application with a lot of microservices. They all require updates. And this is what we do also with the management console. It's all available under a single domain, so you only need to whitelist a single outgoing port, of port 443 to management.umh.app, and you have everything settled. You can remotely control it. You can get all the updates. And this makes it now really, really easy to integrate it. Like, opening one single firewall port, nobody asks questions about it. If you need outgoing, incoming, over multiple zones, it's, but with this, it's heavily- Chris Engelbert: So how do you do that? Is that MQTT? Is it WebSocket? Jeremy Theocharis: It's actually HTTP calls as HTTPS. Chris Engelbert: Right, right. Jeremy Theocharis: You know, ignore the encryption, obviously encrypted, but- and then Chris Engelbert: you just keep it open. Jeremy Theocharis: I'll encrypt it, but it's just simple. Get or post command. Yes, it is slower, but with this we know because a lot of manufacturing companies, they have like weird firewalls, and we kind of fear by introducing WebSocket that they might do packet inspection, they do like weird stuff in it. So we started with the most basic setup that you can have, HTTPS connection, GetPost. Chris Engelbert: So is it even long polling or is it literally just like, once in a while you poll and see if something happens? Jeremy Theocharis: Yeah. Well, once every second or there's a back off for slow. So connections obviously, but yeah, just every second or up to five seconds, it fetches data. It makes it all a bit slower because now you're from the front end, you click something and it takes some time. But now we could pulled up on that. So just during the lunch break, we discussed, okay, what about that we try to additionally open a WebSocket connection. If it doesn't work, we'll have the fallback. So it's very reliable. But if it works, you have even a quicker system. So that when you press the button, it only takes, I don't know, 200 milliseconds, whatever, to arrive at the target destination and 20 milliseconds back. So you have, a four milliseconds delay, which is, I would say, what the user still feels as real kind of real time. It's a moment. Chris Engelbert: I agree. Okay. we, we almost crossed by 20 minutes already. Jeremy Theocharis: Oh, wow. Chris Engelbert: So, the- you actually made a good bridge, because you already mentioned Kubernetes and you mentioned Timescale. So like, I mean, you're storing quite a, lot of time. I think substantial amount of data. You have to process it probably in some way. Like how is the infrastructure built up, especially if it's not a cloud infrastructure, but set up by customers? Jeremy Theocharis: We originally started with the Ming stack, Mosquito, InfluxDB, Node RED, Grafana. It's very popular for home automation. But there are a lot of issues with that. We wrote a lot of blog articles about it. One of them was being scalability, or if you- Fun fact, if you Google TimescaleDB versus InfluxDB, my post, depending on the country that you're in, will be in the top one. Oh, and now, I'm lost. What was the question again? Chris Engelbert: How's the stack? Jeremy Theocharis: So we would have started as a system integrator and we needed to install this type of tools again and again. So we decided, okay, what about we bundle it together? And we first with Docker Compose, but this also has a little bit, you can't configure it properly. So we actually chose Kubernetes K3S or it's a bit for the ability of Helm charts that you can take a lot of existing components in a single YAML file and configure them. So the infrastructure itself, it's K3S. As brokers, there's a MQTT broker, HiveMQ, and Redpanda's Kafka. There is TimescaleDB for Postgres and time series data. And, I think at the end of this week or next week, we release, we switch out all the bridges between it, and move 100 percent to Ventos, Redpanda Connect, it's also called. It's a very good, if you don't know it yet, it's a super nice tool for stream processing. It's lightweight. It's simple configuration file. It's not this super big scaling, because very often you don't need it. But it is like in the sweet spot. If you just have a couple thousand messages per second, you can put it in there or it scales up to 100,000, 1 million. If you don't need the Uber, Netflix, ad hoc technologies, it's a perfect fit for it. And that's kind of like the core technology in there. Chris Engelbert: Right. I guess the simplest version is probably like a Terraform recipe or anything like that to just kick it off. Jeremy Theocharis: So we're also in the middle of switching things out. In the core, it's a Helm chart, which combines everything. But we're currently moving slower towards a Kubernetes operator. Because we're all getting at the limits of Helm charts. So you install. And in the end it is an install script which then sets up K3S. So it's actually, you'd solve with a, sorry, a bash script. It's all K3S. It pre configures K3S for the typical workload that you have in manufacturing. And then it takes in all the Helm chart with all the software pre configured to the typical, with standard data models, pre configured to the typical workload that you have in manufacturing. So it's not just, yeah, startup Grafana. No, it's like you start it up and then you just need to send a message by MQTT with a specific topic payload and it will automatically be stored in the database. So full end to end throughput call. Chris Engelbert: Right, right. So from a scalability point of view, I guess you can start fairly small and you would grow it over time? Jeremy Theocharis: So everything is built on IT and OT best practices. OT, operations technology stuff on the top floor. And it is a fundamentally different approach from a lot of other vendors in the market in this field. We start with, yeah, let's make it look very easy. Then they start to put IT/OT best practices on that. We started the other way around. We started with IT/OT best practices and now make them easier to apply and easier to use. So from scaling, I mean, it's K3S, scaling it three nodes. It's not self service yet, but you can always do that. Yes, there are some difficulties, but with etcd, whatever, we will put like a standard configuration in within the next one or two, three years that you can do it by default. Then also about scaling, for MQTT, you actually to buy Hyphenq if you want to scale it up. But for Redpanda, you can easily scale it. I mean, we're using the Helm chart slash operator of them, scale it up to three nodes. You have a Kubernete on three nodes. Need to fine tune a little bit with replicas, et cetera, what we already did. Now database, you scale it up with- Usually don't need a scalable data writing because, even if you take the biggest amount of possible data and put it, makes it times 10, you still are fine with a single instance. So for writing data, so there, what we use is one main write, and then a couple of read replicas. I also over. And that is- Chris Engelbert: So you, basically have like the single rider, but you use for the, potentially just slightly off, read replicas to scale out queries down. Yeah. Jeremy Theocharis: And the read load and then if there's a failover, or so if OneNote fails, then another can take over. Just had discussion yesterday. I think you can solve, you only need scaling cross nodes for high availability. I don't think you need it for scaling because you have a manufacturing. Chris Engelbert: Yeah. Yeah. It sounds like a very similar setup to what we had on my own startup, which was also the biggest workload was live dashboarding. And that means your active data set is most probably still in memory anyway. So, whatever. And the rest is written down to disk and you have some processes somewhere. Doing stuff with that, but it's all not mission critical and not time critical. So there's, before we go to the two last questions that I basically ask everyone, like, UMH has two different versions. The enterprise version is obvious to me. I understand why people would use that. But what do you use the community version for? Jeremy Theocharis: So actually we have three versions. So there's the open source version where every- And this is important for us. All the core basics of UMH are always open source. So all the connectors to Siemens S7, et cetera, we call it BentoSUMH. It's an open source project. The Helm chart itself is also open source again. When we move to the operator, the core operator functionality will also be open source. So, this is the core. Then, when you want to remotely manage them, the management console, there is a free community edition. There is at the moment no fixed limit on it yet, except to currently the users. So enterprise users can already have multiple users, but a community user can only have one. But otherwise there's a feature parity between enterprise and community definitely, just in terms of features at this moment, besides amount of instances and amount of users. This will change over time. Also, the what's most important is that you have a guaranteed support level, a guaranteed service contract. Not a service contract, but guaranteed SLAs. This is kind of the main difference of the enterprise version. You can use multiple users because one user, you can't do something with it. You have all the SLAs for it. And, but as a community edition, you can still do everything. You can still try it out. It's a small company, you can even use the community permanently. You don't have any guarantees from us. You can use it for free, which is important for us because we want to have all the feedback. Chris Engelbert: So is it private people or is that like just everyone? Jeremy Theocharis: What do you mean? Chris Engelbert: Yeah. Do people use that for, I don't know, like home automation things? Jeremy Theocharis: Some use it. We come actually from that space. However it is, for most home automation people, that is always overkill and the specs are also too big for it. Like you always need four CPUs and the one with 16 gigabyte of RAM. You don't have that lying around. It doesn't work on a Raspberry Pi. So it makes it a bit hard to use, but if you, some people have like VMs at home, they- Normally, the normal user is an enterprise or someone in an enterprise who wants to use it, that just installs it there on a VM, plays around with it, connects machines, and at one point is like, hey, this is exactly what I need, and then contacts us, and is like, hey, can I get the enterprise versions? I want to include my team. I want to do things like single sign on, I need service contracts, et cetera. Chris Engelbert: All right. Yeah. Okay. That makes sense. So it's a little bit like a marketing tool, and a brand marketing. Yeah. Yeah. That makes sense. Jeremy Theocharis: Feedback. This is something that you can't, because most of the vendors there, they always have small user group. And therefore the product are only tested with the actual people, not even the people who use it, but for the people who buy it. Just also a couple of weeks ago, I wrote an article about this. By providing a free community edition and going this open source stream, we actually need to make the product good for the users. And we have already people who use it, so, and they're not forced to use it. They decide because they like it. So we get their feedback. This is what I would say this community open source approach is what makes us a better product because we have way more feedback that goes into it, way more edge cases, way more tutorials than you typically would find in the space. How you basically use it, how you get to know it, from in IT. In IT, it's normal to have a lot of tutorials. Manufacturing, it's not. It's all, yeah, call me for a quote. Chris Engelbert: I know, yeah, yeah, you get a machine, 400 contacts. Like, screw contacts. You have no idea what is happening, but you call them and they send you over the technician to set it up. Same thing. All right. So, because time, what do you think is the next big thing? It could be in, whatever, manufacturing, could be in cloud, in scalability. Jeremy Theocharis: So what's currently coming up and it's gaining more and more and more traction in manufacturing is this concept of the Unified Namespace. It is, it sounds super simple, like just have a message broker. But in manufacturing, it's a huge game changer. As I said at the very beginning, you have the data available in real time. It's totally- mind is blown. This is the next big, big, big thing there because it will enable, sorry, it will enable so many other use cases. Everything else that you, at least for manufacturing this, all LLMs, et cetera, if you don't have access to the data, they're all worthless. So, having this, at least in manufacturing, is the next big thing. I know it doesn't sound so exciting for IT people. But literally in our industry, it is. Chris Engelbert: Yeah. I think Dominik from HiveMQ was also talking about that. And I think he also brought this up as like the big thing right now, for him specifically in MQTT. But also in general. All right. Then last question, is there anything else you want the audience to know about? Jeremy Theocharis: If you are in manufacturing, or also building automation or anything that has to do with PLCs, and what I said sounds nice, we need your feedback, so you can just go on management.umh.app and try out the free community edition. And give us feedback. That would be really helpful. Chris Engelbert: Perfect. Thank you very much for being here then. Jeremy Theocharis: Yeah, thank you for the invitation. Chris Engelbert: And for the audience, you know how it works. Next episode, next week. I hope you listen again and see you soon.