This post is a project in progress, as such the build isn’t complete, the final build pictures will be posted when this article is finished (expected finish date is around June / July) as I’m only putting aside £100 or so per month on this project
Another project I’m starting on, since migrating my internet to Youfibre. Is building a custom 1u router, powered by pfSense.
I’ve been thinking of calling this project Frankenstein, simply because of how many modifications I’ve done to things and the amount of mish mash components and setups inside the chassis
Desired specifications on my 1U Pfsense Router
- 10 Gig network cards (quad port ideally), both RJ45 and SFP for flexibility
- Modern multicore CPU that supports AES-NI for encyrption and VPN (ideally 4 cores or more)
- Approx 16GB RAM
- NVME Hard Disk (for speed and lower power)
- 1U Chassis with Flex PSU (250mm deep max)
Build budget for my pfsense router
I’m aiming for a target of around £350 for building my Pfsense router, whilst this is overkill and doesn’t need to be this much I can reuse the equipment for another build should I decide to change this around or go for lesser spec equipment
Build log for my 1U Pfsense router
Entry #1 Pfsense build
As this project has only just started, there’s not mainly much to report as it was just in progress this month. Currently I have purchased two network cards on eBay which I think should work. I read about recommended network cards on the servethehome blog post, and network cards seem to be a bit of a pain unless you’re getting genuine Intel ones or recommended network cards for pfsense builds. I did a search on eBay and managed to find a Quad Port RJ45 Chelsio T540-BT which is 4 x 10GB ports that can be used for a NIC for a cheapish prices of around £40. The only thing I’m unsure about is most cards seem to be branded by manfuacturers like Dell or HP. Not sure if this is an issue, I’ll find out.
Whilst I’m not expecting line rate speeds to be achieved, I also don’t want to hit bottlenecks, so just want to minimise it. I also added extra ports for flexibility. I also luckily (whilst searching the above card), managed to blag a Quad Port SFP Chelsio-T540-CR fibre card, which is the Fibre version of the above card I purchased (for around £60). As a bonus it also came fitted with 4 transceiver modules to save me probably another £20 or so.
That’s the network side of things sorted, next months purchase will probably be either the case. Or the hardware components perhaps a motherboard, then CPU/RAM in the month after. As I need to save pennies first.
The biggest problem I’m going to encounter I think in my DIY pfsense build is the chassis size. Due to me having two network cards, I need to have a motherboard that supports dual PCI-express slots. The problem is the cases I’m looking at (250mm deep), are generally recommended for Mini-ITX motherboards, and for dual PCI slots, you normally need a Micro ATX motherboard (244mm x 244mm). Now, in “theory” a board should fit, although I’m probably going to have to do some magic to my pfsense build. I’m thinking about buying a black case, and then drilling and cutting all the holes myself.
As a quick side note, I know I can go deeper for MicroATX chassis, however I’m trying to get this to fit inside my Electronics workbench rack portion and it’s only about 12″ deep, so 250mm is probably my max
Power for the pfsense server build won’t be an issue, I’ll just use a flex PSU for that in my build which will supply all the power I need. The next big step after this would be cooling the server. I think I’m going to be really limited on space if I manage to honk in a MicroATX board, so may have to custom 3D print a shroud I can whack over the heatsink, or direct CPU), and get some fans to cooling the 1u router build.
Entry #2 Pfsense build
I extended the build further this month, I was looking around eBay every now and then for the off chance something cropped my eye. Whilst I was browsing, I came across a Gigabyte Gaming 5, MicroATX motherboard which came with an Intel i5-4690k processor, and 16GB of DDR3 RAM. Whilst it looks nice, it’s also more modern than my existing computer. It had a built in NVME M.2 slot on the motherboard, and 3 x PCI-e 16x slots. When populated on two slots they run in 8x mode (which is great because the network cards are x8). Great. I decided to purchase it, cost me £100. I didn’t want to go overboard on the hard disk, but wanted something to support the M.2 slot. So purchased a P320 Intel NVME drive. Perfect. It is 256gb in size, and fast enough for my requirements. The M2 NVME cost me £15 for my pfSense router.
The next months steps will be buying a 1U rackmount case (250mm deep) and a flex PSU for the rack equipment. Rather than a pre-made one (which is designed for wrong size ATX board). I’m going to buy a blank chassis, then cut slots, and drill holes to my own specification. After this, I’m going to 3D print a shroud of some sort (or maybe a copper heatsink), then attach it to the board and improve the cooling for the case.
Once this is done, it’s a case of drilling holes necessary for the airflow and go from there.
Entry #3 Pfsense build
I ordered myself a 1U Flex PSU for the build this month from a reliable brand for the server, I chose a Seasonic SSP-300SUB Flex PSU, which is more than enough power for the build. I’m having trouble finding a 1U chassis currently without going over my 1U height specification, I need to find a case design that has around 40mm of internal space to play with or go custom / larger. If I go custom, it makes it easier to fit but I lose space. I can also get a custom made one with benefits of choosing my own colours.
At the moment the case design I’m looking at is probably something like a Logic Case LC-1250-BL chassis. As the metal design doesn’t seem to interfere with the internal height looking at the photos they have online. The one I looked at had a different case design where it wasn’t flat/flush with the mount so you lose around 8mm for the case internally (there was only 36mm of internal room within the case). I’d rather have a case from a manufacturer where it’s attached on the sides rather than the front. That way I can maximise space.
Anyhoo I’m going to wait until I get a response then try it out
i’ve also purchased 32GB of RAM for the server instead of the 16GB it came with. I know this is already overkill, but I wanted low profile room. The RAM I purchased for the pfSense build, is also ECC Registered and low voltage (1.25v I think from memory). Hypertec brand.
The other thing I have my eyes on is a CPU Fan (1U blower style), low profile. As I know I’m going to be limited on space.
I’m doing a lot of experimentation with the build because of how little space I have to work with. Last things I purchased were PCI-e riser cables, as I know they won’t be able to fit in the factory slots. Although it did give me an idea about designing some boards for it. The next point will be insulation of the board from the bottom of the case to prevent it shorting. I’m thinking along the lines of Kapton Tape, Foam, or some form of rubber membrane. It doesn’t have to be thick, just enough to stop metal on metal
The next thing I’m considering is changing the server so it runs proxmox rather than just pfSense. The main reason for this is the specification is way overpowered for pfSense, and it would be better running a virtual machine on it with lots of environments.
Just for the moment I’m going to run pfSense, then if I like the configuration, I’ll probably buy another M.2 SSD, and then simply install proxmox under a new configration from this and set it up scratch again
Entry #4 Pfsense build
I purchased the Logic Case, they’re a brand of servercases. it was mainly down to internal height (it’s 42mm), another brand I was looking at was 36mm internally which wouldn’t give me the room I required to fit the board. Postage fee sucked though, had to pay £17 postage, + VAT so on top of my order it cost me another £40
I’ve already begun modifying the case using my Milwaukee diegrinder. I’ve ground out all the stand offs, and removed the hard disk caddy since I won’t need it.
If I fit the motherboard in the sense where the power is located, I basically loose two things. the front cooling fans, and the front switches usb and power. I also have a bit of a chew when it comes to orientation of the expansion slots for the network cards. Ideally I wanted my Fibre card somewhere near the front left, but that option goes out the window in this orientatation. Mainly because it’s near the front corner so I could run a short uplink cable to the switch.
What I’m currently thinking of doing, is mounting the board upside down inside the chassis, and then modifying the rear cage for installation. cooling is another issue, but one challenge at a time. Upside down allows me to retain the front switches, and mount the hardware where I want it (I prefer the power being rear left, rather than rear right). I still lose the cooling fans, but I can work around that.
Whilst it may seem weird having a motherboard inside a case upside down because you know, hot air rises. As long as there’s sufficient cooling, it shouldn’t generally matter in the pfSense build. Plus I’m not building this pfSense server as something to admire, it’s something I’m building to sling it inside a cubby hole and forget about it whilst I run other services on my home network.
Entry #5 Pfsense build
The first hiccup!
Why oh why didn’t I check sooner.
Looks like the motherboard I bought was either borked, or I broked it along the way with my work. There’s about 6 or 7 damaged CPU pins, they’re bent I straightened them out. A couple were missing, I replaced using RJ45 wire but this didn’t seem to do much
Basically when I switch it on, the CPU fan just spins up, beeps constinously, and then spins down, no post. Then cycles again. I’ve removed all the ram (as PSU / RAM are suspects for the constant beeping), but I think it’s the CPU housing borking it.
Simply because before I got to this stage, I noticed some bent pins. Had the same problems however it wouldn’t light up the LED lights on the motherboard, yet after I straightened them it did.
Realistically I’m thinking I’m going to have to get a new motherboard, but whatever. I have the CPU and RAM now anyway, so I’ll look for another cheapo board that supports (2 x PCI-e X8) slots and 1 x M2 slot so that I can use the hardware on my build. Whilst this is inconvenient, I can at least order a new CPU 1150 housing from somewhere like Aliexpress, and then get it delivered. You’re meant to use a board preheater, but I reckon I can probably get away with my hot air station and a heat gun. Besides the board is dead anyway
Another issue I had was the cards, I’m thinking of making my own extenders using FPC cables, as ribbon ones are too thick. It only has to carry a signal really so the wires should be more than enough for that. I did have an issue fitting the board, and the front mount panel but I sorted that (basically fitted it upside down). OK the power/reset and USB parts are the wrong way around but who cares. It still functions.
Entry #6 Pfsense build
OK got my new motherboard, also discovered that I could fit a full size ATX motherboard into my case, and it actually makes the design easier. Whereas the MicroATX design 9.6″ x 9.6″ fits into the case, and leaves room at the edges, everything was compact, and I had to ditch the cooling fans which I would have liked, and rotate the front power supply.
I trial fit the motherboard using my Gigabyte one I had lying around using the standard ATX one (I believe it’s something like 12″ x 9.6″ and it fits the case perfectly, and the gap between the power supply and motherboard is almost perfect for another set of 40mm fans. Since cooling is going to be a problem for me I think, I wanted to find ways to cram in as much as possible.
This design allows me to use the factory fans in their original placements too, and the factory controls.
What I’m currently thinking of at the moment, since I’m currently: a) Using full height PCI-e cards, and b) Don’t have room. I’m having to think on the fly. What I’m working on is having the WAN port copper (RJ45) at the rear, and have the Fibre card internally within the case itself. Then running a cable internally and coming out the front via a grommet. OK, it’s not how a router should look, but this case is also custom. I’m working with the room I have for the spec.
I’ve tested out the motherboard, bonus it came with a i3 CPU aswell which I wasn’t expecting. Powered it up, and all is good. It registers the CPU and RAM in the BIOS, so feel like I’m at a good starting point now and bench tested it.
The 2nd hiccup!
Minor one, not sure about the RAM, when I was bench testing, the 32GB kit I bought stopped it booting, but the 16GB kit I received on the original board worked fine. So whatever, that’s what I’m using now. It’s enough for the router
In terms of the motherboard, it’s pretty similar to the one I purchased. I swapped over the CPU from the other motherboard which I’m going to repair. The eBay seller was kind enough to offer me a refund, but I turned it down. Not sure if they busted it during a reinstall, or it was me. I’m thinking me realistically, you learn from mistakes.
Still, the next steps for the build is going to be installing the NVME drive, and then a build of pfSense for a USB install. I’m probably going to do that this weekend when I have more time. I’ll install the network cards too, so that I can verify they actually work and are detected.
After this, the next steps will be working on installation of the network cards inside the case, and going from there. I’ve ordered some FPC cables from eBay to try out a new fitment, so I’ll feedback after this too
I’ve also received my temperature sensors, so once I’ve built the main router, and got it working. I’ll look at troubleshooting and setting up an ESP32 temperature monitoring with webpage, that integrates into my sensors. I may need help on this. Arduino stuff isn’t my strong point
Entry #7 Pfsense build
Currently got the machine up and running in testing purposes. Still working on the NIC’s for my pfSense build but the following are completed
- Rear shield cut to allow the StandardATX board to fit into the case
- Drilled and tapped the new fitment holes
- Made sure 40mm fans fir both side and front of the motherboard (I’m going to need to cool as much as poss)
Still to do
- Add a non conductive coating to the base to prevent anything shorting out
- Buy more 40mm fans
- Develop cooling solution for the NIC’s
- Make up the PCI-e riser cables using FPC cables
- Setup cooling sensors with ESP32 to monitor the board and chassis and all else
- Develop basic webpage for sensor monitoring (maybe with graphics)
Entry #8 Pfsense build
Haven’t finished yet, but still updating this post. Progress is going on. I’ve also got loads of pictures, but obviously going to add these when finished as I’m concentrating on finishing the build first before updating this post.
Actions completed on to do list for last list. The current progress is I’ve bought some adhesive rubber membrane to add to my casing as insulation. If it ends up being too thick. I’ll try and find a spray adhesive, I just need to make the steel non conductive. Experimenting currently, once I know it’s not conductive, I’m going to then test it with multimeber and use Prime95 on the build to stress test it for heat output
I’ve also purchased more 40mm fans, as I think I’m going to fill the machine with all the empty room, I’ve got the temperature sensors I need for cooling solution. Had a minor set back with the FPC cables as they aren’t good with soldering iron. Next best thing I can think of is using superglue, or stripping back the wire insultion on an FPC cable, and soldering the cable itself. I purchased some UV Confonrmal coating, to see if I can bond it to the board.
The main major problem is the fibre card. I can easily install a single card in the chassis. When I install both cards, I’m limited on orientation, and space. I can fit the 2nd fibre card IF, I mount it directly above the CPU. In order to do this however, I need to remove the CPU cooler. So that’s probably going to be the next challenge, or figure out another way to mount it all
Offtopic, this router is heavy, so glad it’s being rack mounted
Entry #9 Pfsense build
My main problem is noted earlier, the wires joining the PCI-e connectors to the edges. I can trim down the connectors so everything fits. My big main problem however is the wires, can’t solder the FFC cables it melts them. Don’t know what dissolves the plastic so I can solder directly
Not sure what I’m doing with the cables yet, whether it be 3D printed. At the moment I’m thinking if you ever solder something like FFC cables, you need something like a miniature spot welder or wire bonding machine. They’re small sizes, and probably the best as you dump an instant amount of heat to create a bond.
At the moment spot welding is looking like the candidate. I could make one or buy one, since I want to work with LiIon batteries anyway in projects. The rest of the components are pretty much done. I’ve fit the rubber to insulate it, I know where I can mount all the heat sensors and this will be “relatively easy” compared to the rest. As for mounting I can 3d print something to hold the cards in place. Still determined to win this one
What problems building the pfSense router
The main problem is space, a 250mm deep is the maximum I could get for a rack case. I can actually go deeper, but I want the router in a specific location in my workbench. I didn’t want to use a PicoATX motherboard, because it lacks M.2 and PCI-express ports due to space constraints.
MicroATX motherboards “can” fit a 250mm deep case, but there’s very little wiggle room around 5mm. So I plan on purchasing my own case and modifying it to my own design to suit.
I’m also not sure about the gap between the base of the case, and the motherboard itslef, even with low profile stand offs the room is going to be limited I think. I may have to switch to a 1.5u case design, one of the benefits with this is the extra spac,e and I have room to put in two full profile height PCI-e cards if I want to install them at the rear I/O. I’d much rather have a 1u case though, I may have to come up with some magic to fit everything inside the case. I’m now more interested in this as a challenge aswell. I’ll buy the 1U first, and if needed I’ll buy the 1.5U then just keep the 1U for something else
Cooling the pfSense router
Cooling is obviously going to be a big problem for my custom built pfSense router build. There’s a few reasons for this.
- Space inside the case is tight
- Minimal room for cooling
- Rear of case is tight
- High ouput from heat internally from devices like NIC, CPU and RAM
As a base build, I’m looking at fitting maybe a low profile CPU cooler inside the case. In order to test the build I’m planning on running Prime95 stress test seperately on the build itself once it’s built up. This will be a good way to stress the equipment and see what kind of temps I will hit and see if anything is worrying me.
I am planning on building a PWM Fan Controller and fitting a few 40mm fans inside the case. May aswell make use of any space that can be used for cooling, as cooling it is more of a problem than exhausting the heat for me.
The network cards appear to use ASIC chips and these run incredibly hot. The fitment into my case, the only method (because they’re full height) not low profile PCI-e cards the ones I have are removing the heatsinks, and putting them face own. Now as long as I can find a way to keep them cool, it doesn’t matter as that’s all that matters!
Might end up making a custom heatsink for it, one that’s smaller perhaps than the existing one, but made of copper or something to improve the cooling. Thinking cap going on
Cooling and Monitoring the pfSense router
Whilst cooling is mentioned above, I need a way of monitoring it. As ideally I want to see what the internal temperatures are like for multiple zones in the server case. Mainly because of it’s compact fit and space I have to work with at the rear.
Ideally I’d love a multi channel display however these are too big, and also too expensive. Currently what I’m thinking of doing is I’ve purchased 10 x 1-wire temperature sensors (I only need something basic after all). I can then hook these up to an arduino/ESP32 micro controller and write code for a webpage which displays the results externally. I can then review this data on the local lan or via external monitor. I’ve already made a simple set of cables (power, ground, signal) using RJ45 jacks and about 15″ of wire. This is so I can power and route the sensors where I want on the board to monitor about 10 different locations of various stages
These are the areas I’m thinking of monitoring :
PSU Front, PSU Rear, Chassis Rear, Chassis Front (Ambient temp), RJ45 NIC, Fibre NIC, Motherboard front, Motherboard rear.
The above zones should cover all the areas that could potentially get hot inside the case. I can also attach the sensors to the boards and points directly inside the case with an adhesive of some sort. Plus I imagine there’s a way to monitor sensors with pfSense too for extra output.
Cooling wise I’ve noted above I’m going to have plenty of cooling inside the case (although I’d like low-RPM) fans to reduce the noise. I’ll stress test the final build with Prime95 before I release it into live environment as the equoipment is going to be left running continously unattended, so obviously I don’t want any fire hazards (which shouldn’t happen as thermal shutdown would occur) always best to be safe though.
I don’t forsee an issue with cooling, but I’m just documenting everything as I go as some may be interested in this build
Power saving on DIY pfSense router
One of the things I’ve considered will considered when building the router is power saving functionality. The CPU supports AES-NI, so I’ll turn this on, but I’m going to look at turning on any little feature that needs it, and disabling anything I can to save power. Onboard audio, don’t need that. Anything which isn’t functional to the build can be turned off to me. I’ll be documenting each change I do and recording the results so that people can see real world figures over the base line build.
Whilst I can afford it, with the current price of utilities I want this machine to use as little power as possible. I’ll do testing with a Killawatt style power monitor (or metered PDU) so that I can see real time draw statistics. I’m not sure if there’s anyway to load test a pfSense server so I’ll have to look this up too
Off hand, I’m consdering removing a RAM module, my motherboard has two sticks and each one is 8gb. This is more than enough for a pfSense router build. I can easily whack in more if needed. I considered changing to Low voltage ram. I have my eyes on some sticks at the minute as due to the space I have available I want to maximise the airflow internally too.
I’d like a passively cooled build, but I think reasonably this won’t happen. I’m going to look at fitting a Flex PSU to the build maybe a higher quality one (with lower wattage) so it’s running an optimum efficiency if 80+ specification. Not sure how mcuh power supply fans add onto the build, but I’ll look at these two. I want the build to be as silent as possible.
The overall goal for the pfSense router
I want more control over my home network and technology that I can use to explore IT knowledge. The problem with ISP equipment is that it works, or is severely limited and hinders the equipment, I don’t want any hindrane on my network setup and also plan on receiving a Cat6a or Cat7 certified network cable, that is guaranteed 10Gbe so I can run it between the ONT and my new pfsense router build when the build is finished.
Plus it’s fun
I’m well aware the specs for this build are completely overkill and would never get used. Overall I’m not really bothered, just wanted something custom built as it’s been a while since I had chances to build up a PC.
I’m going to use a certified 10Gb network cable so that I know it’s guaranteed to perform. This is going to run straight from the ONT, the RJ45 output port so that it becomes the WAN input on the pfSense DIY router. I plan on using PCI risers with the WAN card at the rear, then using the SFP card as an uplink port to my 10Gb switch that I’m going to be fitting to. I’d love to get line rate speeds on my router, but I’ll see what possible options I can do with the configuration.
I’ll still need to buy myself a Wifi router until I can get my Wifi6/7 access points to provider WiFI networking, but primarily the design is for me to play about with home networks and learn.
Final build pictures for 1u pfSense DIY router
Pictures will be added after the final build is complete
