Category: Uncategorized

Ok in my post on the whole situation with routers in the US I mentioned something about supply chain attacks and as someone who works in the software world those things are absolutely terrifying and I think that it deserves a bit of a deeper dive.

So, for people that work in software, probably going to be a bit boring here, but for people that aren’t in software for a living you need to understand how software works when people are building products for you to use.

Software can be complex, so complex that in a lot of cases people will re-use code to prevent having to rewrite things from the start each time they go to do something. This can come in a bunch of different forms but it usually winds up as being a dependency that comes into play when you go to use a product.

So let’s say that I’m building software that tracks inventory. Details don’t matter much but at some point I’m going to have a requirement to store data for use of the product, and chances are that I’m going to be using some type of database engine to allow that to happen. So I can go and write my own database engine from the ground up and go through all the work of getting that developed, performance tuned, and ready to go.

Or I can just install PostgresSQL, a FOSS project that already does this. It has solid documentation, community support, and a history of being able to scale in a way that’s probably in excess of what I’m going to require.

The same thing applies to my user interface. I can build my own from the ground up, or I could use something like Gradio that provides me with a lot of building blocks to make building my UI quicker so that I’m not having to reinvent the wheel.

Now here’s where things start to get messy. While our application only has two dependencies each of those products have their own dependencies, and each of those dependencies have their own dependencies, and each of those dependencies have potentially more dependencies that come into play. How deep is that rabbit hole?

You can get to the bottom of that rabbit hole, and I’m sure that there are companies out there that will spend the time to do so. However I’ve worked around software developers for a long time and as sure as I am that there are companies that handle this properly I’m just as sure that there are many that do not.

So why is this important and how does it become an attack vector?

Let’s say that my inventory application becomes something crazy good, winds up getting used all over the place, even to the point where it’s now in government systems being used to track things that are considered very important. Let’s even say that whatever the application is tracking is important enough that there are people who are now very interested in breaking into those systems and getting access to the data that they contain.

So now my little application is a target. Potentially one that’s worth some money to somebody who is able to break into it.

So now that I’m a target people are poking at my software, looking anything that would allow them to get into my product and mess around. However, let’s say that for purposes of this post that I’m reasonably competent and that my app is solid enough that attackers can’t get in by going after my code. So now what do they do?

Well my product has some dependencies, and those products have dependencies, and lets say they walk down that chain and eventually come across something used by one of my dependencies – and now by extension used by my product. Now the objective is to get control over whatever that is and try to bend it to their will.

Perhaps they submit some helpful bugfixes that hide some malicious code. Perhaps they bribe or buy the control of that dependency from the person who is in control of it now. In either case they now have control over that dependency and by extension can now get into my chain of dependencies and start looking at ways to cause problems for my product. Perhaps they write code that steals passwords and other secrets. Perhaps they find a way to build code that lets them copy data out of my product. In either case the attacker has managed to breach my product.

Now let’s say that I’m really good that what I do and I maintain a full software build of materials and I know that this dependency exists and is something that I have to keep track of and watch out for problems with. In a lot of cases dependencies are just used by the people building the code without a huge amount of review of the actual code involved. That type of review is something that’s fairly labor intensive. So even if I am crazy good and willing to put the time in to review the code there’s still a chance that it might be outside of my area of expertise and I might not catch what was being done and flag it as being malicious.

Let’s say that I’m as good as possible, and I’m paranoid as hell and decide to fork whatever I’m using and freeze things so that I’m not pulling in the changes from the products once I have things working. At some point there’s still going to be some problem that I run into that will require me to update the dependencies that I’m working with either because of a vulnerability or functional bug that needs to be fixed. At that time there’s still a chance that things that I don’t want get into my product.

The software industry as a whole has processes in place to try to catch this stuff, but there’s still a lot that just seems to run on trust, and some programs pull in a lot of libraries for simple things. Hell, a few years ago a small utility (11 lines of code) in NPM was pulled from their repository over a dispute and managed to cause enough chaos in the course of a couple of hours that NPM wound up restoring the thing from backups to keep the world working properly. In this case the supply chain attack wasn’t being done to break into systems or steal data, it was a protest of sorts but the end result was very disruptive.

So, even doing everything right, and being very good about the security of my product there’s still a very real possibility that I’m getting breached, and if I’m not paying attention and happen to catch what’s going on this type of breach could run for a very long time before it’s caught, patched, and no longer a threat.

That’s why these supply chain attacks are terrifying. And if you have coworkers or other folks who don’t seem to see this as a issue that’s a pretty big red flag to watch for.

So a couple of days ago the FCC updated a list of banned telecomunications equipment to include, and I’m quoting here;

Routers produced in a foreign country, except routers which have been granted a Conditional Approval by DoW or DHS.

If you want to read it’s all linked on the FCC site linked here.

Now, from what I’m aware there are no domestic manufacturers of routers in the USA. So what they have done is ban the import and sale of any new equipment that does not already have a FCC approval tagged to the device.

If there was a legitimate concern about the security of routers that are deployed in the world why are only new devices being targeted? I would assume that the decision to ban these things is based on some legitimate history of security issues or a history of operating in bad faith on the part of these manufacturers. So if that is the situation why are the existing devices not getting flagged as a problem? Why are we not being told that it’s time to replace those devices?

So if there isn’t a history of bad behavior what is this about? The argument as I understand it is that there are concerns about the security of the devices and their potential to be used as an attack vector rather than any indication that they have been used as such.

Is that legit?

Arguably yes, but without a history of bad behavior this is either the US Government pressuring the hardware vendors to move manufacturing back to the US or it’s a breakdown in the chain of trust that has allowed us to take advantage of offshore manufacturing for as long as we have.

If you look at it every device that you use establishes a chain of trust, regardless of if you realize it or not. Let’s look at your phone, say a iPhone of some generation.

First of all you are trusting Apple, since they built the device and the operating system that it’s running. Implicit in that is that you are also trusting everybody that Apple has trusted as part of their development and supply chain on both the hardware and software side of things. This includes the folks that manufacture the screen, storage, and the developers that write the software that makes up iOS – including any libraries or tools that they use to build the operating system.

You would think that this is fairly simple but the supply chain for software and hardware gets really complicated, really quickly. If you look at the news there have been all sorts of supply chain attacks showing up in the news recently like the one below;

https://snyk.io/articles/poisoned-security-scanner-backdooring-litellm/

The general idea is to look at software libraries and service providers that your providers make use of and attack those instead of coming after you directly. The impacted software library that I linked above is downloaded somewhere around 3.4 million times per day, and this attack was live for about three hours. Assuming a even distribution of downloads that would mean that the people that got away with this hit 500k downloads while this was live, and who knows where they were able to get from there.

So obviously we have to draw a line and work on the assumption that Apple is doing what’s right our example and that they have done their due diligence on things further down the chain.

So if we take this new ban at face value the US Government has some trust issues with the router manufacturers and is taking steps to try to address those by forcing manufacturing of the devices into the hands of domestic companies that they can regulate and mandate some level of security. However the cynic in me is wondering what hardware vendors – if any – are already going through the process of greasing palms to get exemptions for their hardware in place under this program.

And to be clear it’s going to have to be some level of exemptions going through here – it’s going to take a long time for someone to gear up to build out routers domestically in the US for consumer use considering the number of these things that are sitting in peoples homes, offices, and in datacenters.

One of the things that I have spent some time playing with is storage solutions for deployment at home. My primary use for a storage server is to store backups from my other systems, media for streaming in my house, and a general dumping ground for files that need to move around machine to machine.

The obvious solution was to go pickup something from Synology, Qnap, or one of the other vendors who build solutions purpose built for this type of thing and deploy that. However a lot of those solutions were at the time priced more for business use rather than deployment at home. As well I got bit in the past with a Western Digital MyCloud that had somewhat lacklustre firmware updates over time resulting in my decision to go shopping around for something a bit more robust for a solution.

Ruling out a purpose built device means that I’m building my own server and the choice comes down to what software is going to be backing the thing once it’s deployed. The only package that I found at the time that would handle the mix of random drives that I had sitting around at the time was Unraid. Unraid’s method of data protection uses a non-standard type of RAID that uses the largest disk in your array to provide protection in the event of a disk failing in the array. Unraid has a good description of the protection they have in place in their user documentation.

Looking at TrueNAS the ZFS pools that it uses have advantages with from a data resiliency standpoint, from a performance standpoint, as well as the ability to pull off some neat tricks with snapshots. However it doesn’t really handle the mixed pile of disks that I was working with at the time, something that was fairly important since I didn’t want toi go out and buy a pile of new disks when I had plenty sitting around collecting dust.

Another thing that people will probably be quick to point out is that Unraid is not a free solution. When I bought the software it was a single purchase based on the number of drives that you are looking to attach to the server. Pricing is similar now but unless you purchase the “Lifetime” version it doesn’t come with lifetime updates, so that’s something that you should be aware of. From my viewpoint I personally prefer free solutions wherever possible however paying out for Unraid got me up and running quicker than I would have been able to if I was going to try to roll my own solution.

Getting Unraid installed on a USB key is fairly straightforward and once you boot the machine that you are going to use as a storage server from the USB device you just select the drives to add to your array, setup your shares, and you are basically up and running. You can get more stuff running as you start bringing containers into the picture but for basic file services it’s not much more complicated than that. Swapping out drives to increase capacity or when one dies is dead simple, provided that your parity devices are working properly.

I’ve run this Unraid server now for a number of years and it’s done pretty much everything that I have needed for a long while, however there are a couple of things that might be nice to have that TrueNAS does do out of the box. The ability to replicate snapshots between two servers, as well as the ability to upload copies of the data to a cloud storage provider makes backing up things much simpler than what’s in place in my Unraid server right now. And yes, there are a number of plugins and other things that can make this type of function happen on a device running Unraid, but none of them are things that have been baked into Unraid by the developers at this point, so I’m reliant on somebody else maintaining and keeping those plugins updated and functional as Limetech does their thing.

So now to the question of what someone reading this blog should do. That’s kind of going to depend on what you need the storage to do for you and what you happen to have laying around.

If you don’t have a existing system that you can reuse as a server then I would have a look at a pre-built solution like what Ugreen, Synology, or Qnap offer. The pricing on the 2 and 4 bay solutions has dropped over the years to a point where you could potentially get a cheaper system to act as a host but it might not really be worth it given the extra work in stuffing the disks into a used machine that wasn’t meant to take them.

If you have a spare machine sitting around with enough SATA ports and NVME sockets to make a usable server and you are going to be purchasing new drives to work with I would be looking at TrueNAS, if you have a pile of mismatched drives that you want to reuse then I would still go with Unraid as long as the drives that you are working with are large enough to offset the price of the Unraid license that you have to purchase instead of putting that money towards a matching set of drives.

Frankly, I’m nobody in the grand scheme of things and you probably shouldn’t really care what I have to say about anything posted here. This blog is going to be a place for me to toss down random comments on things that I’m interested in and anything here is my opinion, nothing more. 

Feel free to read through as I’m posting things here and if something that I post makes sense or resonates with you that’s great to hear. Keep in mind however that I’m just another random voice on the internet and anything that I’m posting here should be taken with a grain of salt and you should be doing your own homework to make sure that I’m not completely off my rocker. 

That said, I’m interested in Technology, Photography, and I have been trying my hand at learning Brazilian Ju Jitsu for the last several years. 

On the technology side I like playing around with different types of technology, but I also don’t get super excited about things unless there’s a actual reason or application for the technology that I can make use of. So while I’m really interested in 3d printing the current excitement over the sate of “AI” stuff does not really catch my eye as much. 

That whole AI thing is a discussion for another post since I can rant about that specific piece of tech all day long. 

I have spent a lot of time playing around with networking equipment, servers, laptops, desktops, mobile devices, and the management of all those devices. I have opinions on all of them, some more informed than others, and I’m sure that we will get into the topic of managing hardware at some point. 

Photography is something that I’ve been doing off and on since I was young. My first Camera was a Canon T60 that I was given as a Christmas present in high school and I’ve been taking shots ever since. Around 2006 I made a switch over to using a Digital SLR and transitioned away from shooting film. As much as I liked shooting film I don’t really miss the work in the darkroom or the cost of getting film processed if I didn’t feel like doing it myself. 

Most of what I shoot winds up being scenery, landscapes, things like that. I’m not huge on taking photos of people since people can in many cases object to having photos taken and I try to respect that. 

On the sporting side I do find myself taking part in Brazilian Ju Jitsu – it wasn’t something that I sought out but I wound up taking some classes and it’s stuck with me for a bit. A bit being long enough for me to get my belt awarded last year and for me to start the process of training for the IBJJF worlds with a eye to going and actually trying my hand in competition this year. 

So I’m no expert, and I wouldn’t call myself a competitor but I’m probably a bit more than what you would consider a casual player in the sport if that makes any sense. 

As far as politics or religion, I’m going to try to keep that out of this space. I did used to try to talk through this type of thing with people but the current situation in the world makes that difficult to say the least. A lot of people are just taking the stance that if you don’t have the exact same views they do that you are their enemy. So In general I don’t actively go looking for that type of conversation any more.

Anyhow, that’s a bit about me. Stay tuned for more as I get around to typing up stuff here.

Back in 2015 I originally started this up as a place to rant, toss messages around, and in General just mess about with things. Honestly for the most part it was hard to bother with keeping things updated since none of what was posted was actually anything that I saw value in. 

So, second round, let’s give this another try and see if something of value can’t be put on this page.