Badcaps.net Forum
Go Back   Badcaps Forums > Troubleshooting Hardware & Devices and Electronics Theory > General Electronics Technical Discussion
Register FAQ Calendar Search Today's Posts Mark Forums Read

 
Thread Tools Display Modes
Old 01-15-2020, 01:20 AM   #1
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Talking Let's talk about SCADA

Good day folks. As some of you may know from some of my stories here, my boss always likes to take on new and interesting projects he comes across, especially if there's a lot of money involved, so he ropes in average Joes like yours truly for the job, even though they've never done said thing before, so you only have like 1 week to learn it

This year's trend and topic seems to be all about IoT and SCADA systems....big topic, so I really don't know where to start with this. I know what it is and I understand the concept, so I guess the best place to start is the beginning: what we want to achieve.....we're not entirely sure yet, so this is mostly going to be a speculative discussion, discussing the good, the bad, the ugly, the dos and don't, ins and outs of SCADA.

From what I understand, the end goal of all this hype is going to be for us to be able to take over the maintenance and operation of some floodgates....yes, that's right ! Not some small insignificant prototype device on a workbench, but actual goddamn floodgates, complete with ships passing through and everything ! We're not going to go into management stuff here because I don't (want to) know that part myself, but from what I gather, the contract between this compound and the company currently in charge of support is about to expire and of course our boss wants to take over at that point....cool

It's very early to know anything about the systems or particular equipment they're using, though I HAVE been in the control room once and saw the monitors showing the schematic representation of the compound and the different parameters and stats of the valves, sensors, pumps, servos, along with the main gates themselves. Aside from the sheer scale of the thing we're talking about here, it's not really that spectacular in the grand scheme of things: a PC interfaces with some widgets via (what seems to be called) a "HMI" and back....that's SCADA.

Now to make our own I found a variety of resources online, ranging from small DIY stuff based on Arduino, to proper industrial-grade devices, which is what these guys actually use. The software used also seems to be diverse, again, ranging from stuff some chap wrote himself in Visual Studio, to industry-standard programs that comes with the hardware used there. I'm not a software buff as you may know, so this is where I'd like some help first.....suggestions ? What should we use ? Simple-Scada looks like a nice pre-made piece of software I found, though it seems to be in Russian, and of course we can go full-DIY by using Visual Studio itself to at least play with some pre-made programs, since I'm not sure I'd be able to learn C# like that

If I understand correctly, it works like this: the PC program with its visual interface the user clicks on spits out some commands over a COMx port via a MODBUS (?) protocol. The Arduino set to the matching COM port and baud rate like any serial device, with a MODBUS library loaded onto it, responds accordingly to either read something, or issue a command to an output...plenty of stuff to talk about here...
__________________
Wattevah...
Dannyx is offline   Reply With Quote
Old 01-15-2020, 04:35 AM   #2
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Thumbs up Re: Let's talk about SCADA

THIS is a good example that comes fairly close.

I'm already thinking of implementing something like this to monitor those sewage treatment stations I talked about in my PLC discussion so we wouldn't have to go in the field to do it. Connectivity will be an issue, so I also have to come up with some sort of GSM/Cellular connection for them to report back to the main office or monitoring station...
Dannyx is offline   Reply With Quote
Old 01-15-2020, 05:01 AM   #3
sam_sam_sam
Badcaps Veteran
 
Join Date: Jul 2011
City & State: Sunny Jacksonville FL
My Country: USA
Line Voltage: 120 Volts 60 HZ
I'm a: Knowledge Seeker
Posts: 1,868
Default Re: Let's talk about SCADA

We have a machine where I work that uses this system

https://infosys.beckhoff.com/index_en.htm
__________________
9 PC LCD Monitor
6 LCD Flat Screen TV
30 Desk Top Switching Power Supply
10 Battery Charger Switching Power Supply for Power Tool
6 18v Lithium Battery Power Boards for Tool Battery Packs
1 XBox 360 Switching Power Supply and M Board
25 Servo Drives 220/460 3 Phase
6 De-soldering Station Switching Power Supply 1 Power Supply
1 Dell Mother Board
15 Computer Power Supply
1 HP Printer Supply & Control Board * lighting finished it *


These two repairs where found with a ESR meter...> Temp at 50*F then at 90*F the ESR reading more than 10%

1 Over Head Crane Current Sensing Board ( VFD Failure Five Years Later )
2 Hem Saw Computer Stack Board

All of these had CAPs POOF
All of the mosfet that are taken out by bad caps
sam_sam_sam is offline   Reply With Quote
Old 01-15-2020, 05:44 AM   #4
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Default Re: Let's talk about SCADA

Beckhoff...interesting. There, see ? Plenty of different solutions out there. They're probably proprietary too, so it involves knowing exactly which one to pick up and learn....the more, the better of course.
Dannyx is offline   Reply With Quote
Old 01-15-2020, 12:33 PM   #5
sam_sam_sam
Badcaps Veteran
 
Join Date: Jul 2011
City & State: Sunny Jacksonville FL
My Country: USA
Line Voltage: 120 Volts 60 HZ
I'm a: Knowledge Seeker
Posts: 1,868
Default Re: Let's talk about SCADA

Yes and from the little bit that I have read on this system it is quite involved when I have problems with this machine I have to call tech support for them to help me with it

One note

This machine is back in the 2000s I think

Last edited by sam_sam_sam; 01-15-2020 at 12:34 PM..
sam_sam_sam is offline   Reply With Quote
Old 01-15-2020, 01:08 PM   #6
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Default Re: Let's talk about SCADA

Everything DIY seems to revolve most around Visual Studio, so it looks like the perfect opportunity for me to learn something new
Dannyx is offline   Reply With Quote
Old 01-15-2020, 02:17 PM   #7
stj
Great Sage 齊天大聖
 
stj's Avatar
 
Join Date: Dec 2009
City & State: Europe
My Country: some shithole run by Israeli agents
I'm a: Professional Tech
Posts: 22,168
Default Re: Let's talk about SCADA

dont get involved.
one word: LIABILITY.

soon as this fucks up and floods someone or damages something your boss will be on his way to tel-aviv or the nearest border leaving you and your workmates in the shit.

i'd be curious how the hell he got the contract with no prior work to show!!
stj is offline   Reply With Quote
Old 01-15-2020, 02:20 PM   #8
stj
Great Sage 齊天大聖
 
stj's Avatar
 
Join Date: Dec 2009
City & State: Europe
My Country: some shithole run by Israeli agents
I'm a: Professional Tech
Posts: 22,168
Default Re: Let's talk about SCADA

incidentally, critical monitoring such as water or gas distribution does not use public networks - they use a private radio frequency and commercial transceiver modules - i'm sure you can work out why.
stj is offline   Reply With Quote
Old 01-15-2020, 02:43 PM   #9
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Thumbs up Re: Let's talk about SCADA

Ok, no, don't worry, it's not AS crazy as it sounds: we won't dive head-first into this on our own, with nobody there but us newbies - there's plenty of staff there who already handles all the critical work, plus I won't actually DO anything, other than maybe inspect and report defects to the higher-ups.

This perfectly reminds me of the Chernobyl series on HBO, where untrained staff were forced into doing stuff they were aware was dangerous and would lead to failure, but they HAD to comply.....

Quote:
Originally Posted by stj View Post
i'd be curious how the hell he got the contract with no prior work to show!!
Also one word: CONNECTIONS
Dannyx is offline   Reply With Quote
Old 01-15-2020, 04:42 PM   #10
redwire
Badcaps Veteran
 
Join Date: Dec 2010
City & State: Alberta
My Country: Canada
Posts: 1,643
Default Re: Let's talk about SCADA

You start with the I/O (point) count - how many digital in's/out's, analog in's/out's. This tells you how big the control system need to be. If the site wiring diagrams and floor plan are up-to-date, that helps too.

Usually a PLC is used for local control (i.e. stopping motors due to limit switches) etc. so the system can work safely even if comms is down. Yes an Arduino and custom I/O board can be made and work... but you are the only person who knows how it works and if you get hit by a bus - then nobody can make a program or hardware change easily.
Coding it (controls) on a PC is silly because they aren't reliable enough for anything past the HMI/logger. Windows update? Anti-virus update? Your program gets overruled and big drama results.

A big decision is if you are doing controls (outputs) over the Internet. Easy SCADA is "eyes only" where you just watch inputs and never start/stop a pump or something. But if you are remotely turning on equipment, there is always a safety hazard somebody gets squished or hackers get in and cause trouble. You need AUTO/MAN/OFF and LOCAL/REMOTE switches as a minimum on both the HMI and at the PLC.

As far as the HMI, it's always basic MODBUS comms to the PC and HMI software running on that. Take a look at DaqFactory it's low cost.
redwire is offline   Reply With Quote
Old 01-15-2020, 06:05 PM   #11
stj
Great Sage 齊天大聖
 
stj's Avatar
 
Join Date: Dec 2009
City & State: Europe
My Country: some shithole run by Israeli agents
I'm a: Professional Tech
Posts: 22,168
Default Re: Let's talk about SCADA

indeed, i wouldnt trust an o.s. if i could avoid it - just have the code running stand-alone on the hardware.
and write in in c or assembly - not some convoluted bullshit that when compiled has no resemblance to what you wrote.
because you will never find a compiler bug!
stj is offline   Reply With Quote
Old 01-15-2020, 07:16 PM   #12
sam_sam_sam
Badcaps Veteran
 
Join Date: Jul 2011
City & State: Sunny Jacksonville FL
My Country: USA
Line Voltage: 120 Volts 60 HZ
I'm a: Knowledge Seeker
Posts: 1,868
Default Re: Let's talk about SCADA

To me any system that is critical systems should not any access to the outside world you are just asking for trouble that just my thought about this subject

What gets me with this country is that are electrical grid and water supply system and the like are all accessible from the internet really ( you can automation with out being on the internet to be done remotely really get off your ass go to the site and fix the problem)

All you need is a good hacker to cause you nothing but grief

What has surprised me that we have not had major issues with this kind-a thing yet

The views that are in this post are my own and do not get me wrong I not saying that it should not be access by tech support when you the owner or operator turn this feature on so things can fixed once things are taken care of turn this feature off when not in use and not need

Do not make things easy for good hackers

Last edited by sam_sam_sam; 01-15-2020 at 07:18 PM..
sam_sam_sam is offline   Reply With Quote
Old 01-15-2020, 11:51 PM   #13
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Default Re: Let's talk about SCADA

I think punctuation got lost somewhere along the way

There's actually 2 of these compounds and a head office all connected via fiber and a backup radio. The fiber's been down for a solid year now, since it's been ripped and torn to pieces in various places along the canal, so in reality communications is only by radio ATM, since it's us, our company, who is in charge of it, apparently. Some of the boys just recently froze their asses off to install some masts and antennae in the middle of nowhere for this purpose. Of course, I don't actually KNOW what info the compounds exchange about one another, or IF they do at all, or if one can be remotely controlled by the other or from the head office, but I imagine they at least report their status back and forth - makes sense.

Off the shelf equipment is most likely what these guys use - there are some racks with some pretty serious looking equipment in that control room there and the control PCs boot Linux to run that control interface ! Pretty intimidating stuff...I can't imagine what it'd be like if something went wrong there ! :|
Dannyx is offline   Reply With Quote
Old 01-17-2020, 04:48 AM   #14
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Thumbs up Re: Let's talk about SCADA

I reckon those Xinje PLCs I messed around with in the past when I was told to rethink those water treatment stations some of you may remember are also capable of being part of a SCADA system. According to page 163 of their manual, they use MODBUS as well, so the C# software everybody seems to recommend can interact with these instead of the Arduino...it's a good starting point because I already have a couple of these PLCs still doing nothing, so it would be a hands-on experience. The interface would still be serial RS232 from the back of the PC on my desk, so I'd need to figure out a way to make this remote when it's time to deploy these in the middle of nowhere....
Attached Files
File Type: pdf xc_plc_v2.5.pdf (3.00 MB, 4 views)
Dannyx is offline   Reply With Quote
Old 01-17-2020, 01:10 PM   #15
redwire
Badcaps Veteran
 
Join Date: Dec 2010
City & State: Alberta
My Country: Canada
Posts: 1,643
Default Re: Let's talk about SCADA

The hard part is understanding the old control logic and wiring.
Is this a split bridge? There's extra code to deal with one end malfunctioning (thanks for wrecking my yacht) and collisions, power failures etc.

If you need to go low-tech on the communication link, modbus over TCP/IP or even RS-232 over TCP/IP. I don't how the bridge could have ethernet?
Some sites I use private 900MHz radio (serial) links for two PLC's to talk to each other over a short hop.
Attached Images
File Type: jpg trust that plc.jpg (85.0 KB, 7 views)
redwire is offline   Reply With Quote
Old 01-17-2020, 02:12 PM   #16
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Default Re: Let's talk about SCADA

No, it's not quite a split bridge like the one pictured, though one of the compounds DOES have a single-leaf (?) drawbridge as part of its system. This is actually a water-lock: it lowers or raises the water level depending on where the ships come from - either upstream or downstream. The waterway is split into two right down the middle, so there's one water lock on either side, that means 4 individual gates, plus other devices to regulate pressure and water level and stuff. The control tower sits on an island in the very middle of this whole setup, between the 2 locks. The bridge I mentioned is used to allow vehicles to get to the middle part if need be. Having visited this place many many times so far for other duties, I noticed this bridge is kept raised almost all the time because vehicles need to cross over only on very rare occasions, so it makes most sense to just keep it out of the way all the time, to decrease the waiting time for the ships. Personnel, on the other hand, crosses from the control tower in the middle to the banks and back by walking across the top part of the gates themselves, where a pedestrian bridge forms once they close. That's just backstory to better explain this whole compound. It's all academical really, because there's no way you can be expected (make that "ALLOWED" ! !) to run all this willy nilly like that or even get close to the heavy duty machinery that runs it all. It WILL happen though, slowly but surely
Dannyx is offline   Reply With Quote
Old 01-17-2020, 02:51 PM   #17
redwire
Badcaps Veteran
 
Join Date: Dec 2010
City & State: Alberta
My Country: Canada
Posts: 1,643
Default Re: Let's talk about SCADA

It sounds like a good project, it could be fun. Although water sensors are not very reliable. Some SCADA projects work terrible only because the sensors are terrible. I usually upgrade any sensor they have had ongoing problems with. Like a level sensor.

I would start with tons of photos of the site, and get the I/O point list, or "taglist" as it's called in SCADA. If you know how much I/O plus spares for future additions, then you can size the PLC and HMI. If you choose too small a PLC it's a disaster.
redwire is offline   Reply With Quote
Old 01-17-2020, 03:25 PM   #18
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Default Re: Let's talk about SCADA

Well the compound is currently fully operational and will most definitely remain so in the future as well, so we don't actually need to reinvent the wheel here and deploy any complex equipment of our own or implement anything from scratch. We DO need to get a good understanding of the existing setup - good or bad - and, at the very most, suggest improvements where needed/possible.

That doesn't mean I can't do small-scale prototyping on the bench to familiarize myself with all these concepts, like MODBUS, hence why I wanted to build a bench-sized SCADA of my own. Once that's done, it would be MUCH easier to move on to the real deal and it would look more professional if we know what we are doing instead of just stumbling around wondering what's what. Sure, the floodgates probably don't run off an Arduino, but the concept's the same in the long run. I think it may actually be MORE challenging to design your own HMI than using a pre-made one !
Dannyx is offline   Reply With Quote
Old 01-17-2020, 04:42 PM   #19
redwire
Badcaps Veteran
 
Join Date: Dec 2010
City & State: Alberta
My Country: Canada
Posts: 1,643
Default Re: Let's talk about SCADA

I used GE Cimplicity HMI, I think $5,000 for a dongle. The GE ecosystem is crazy expensive, especially for the nuclear power/coal plant DCS. Very hard to program too, it's all quirky, nothing is straightforward.

In MODBUS, there is a Force Multiple Coils (Function Code 15) command which activates multiple outputs all at once. I'm not sure what it's good for. You always use Function code 05 to turn on a single output.

Guess what, the crappy GE Cimplicity program defaults to send FC 15!
So a coworker programmed up the HMI and later a tech went to turn on one output... It shut off power to the entire town, every breaker switched off at once. What a fiasco.

Point is, MODBUS is simple and easy to use but because the standard is poorly defined, companies do their own thing with it and that can be hard to troubleshoot.

We always set up testbenches in the office, with an entire RTU/PLC and HMI to do programming and test. It's too costly and dangerous to debug on site.
redwire is offline   Reply With Quote
Old 01-19-2020, 03:22 AM   #20
Dannyx
CertifiedAxhole
 
Dannyx's Avatar
 
Join Date: Aug 2016
City & State: Constanta
My Country: Romania
Line Voltage: 230VAC 50Hz
I'm a: Hardcore Geek
Posts: 2,743
Default Re: Let's talk about SCADA

Good to know. These guys may very well be willing to invest an insane amount of cash in this project like that in the long run, hence why it'd be ideal to keep costs to a minimum so the bulk of cash goes to us as profit...if that makes any sense.

I've been learning about MODBUS lately and trying to figure out how to implement it on those Xinje PLCs outside its own little piece of software. The manual has quite terrible English translations, no doubt from Chinese, so I could not figure out how all the bits and bites work to do what. All the video tutorials I've watched so far make it quite clear that in order to tell a PLC to do something, you have to use proprietary commands, since they differ between brands, so they all direct you to the product's manual or datasheet. If this were Siemens or other proper brand, I'd no doubt find this info somewhere, but I'm a bit stuck ATM. Sure, I can control the PLC with its own software, but what if I want to send it commands through a third party piece of software which has a graphical interface and all that ? THAT'S what I'm struggling with ATM.
Dannyx is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump



Badcaps.net Technical Forums 2003 - 2020
Powered by vBulletin ®
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
All times are GMT -6. The time now is 10:38 AM.
Did you find this forum helpful?