Hi again :), ok so lets break this into two issues, syncing and performance related by the looks of it.

1. Syncing - each node records information about its synchronisation and rendering, this data is logged and available for inspection in the Crowdrender panel. You can view this information by clicking the speech bubble icon next to a node.

If you like you could try saving the output for the nodes that wont sync and send them to us so we can take a look :).

Other things to look out for would be using different versions of blender and/or crowdrender, this tends to cause sync failures.

2. Performance, ok, those are some pretty decent rigs you have by the way. When it comes to balancing it might help to think that rendering is not just one operation, there are actually a few things that happen that are different with respsect to how the hardware is used and how best to optimise them.

1. Starting background process - Crowdrender starts a separate process to render your file. This means that blender, and all your addons must be loaded into memory on each computer. We've had some folk try some exotic things like host blender or their addons on a remote folder on their network. This does have a purpose, it means that all nodes will always use the same version of the addons and blender, which is a key requirement for things working. If you used this approach then the speed of your network is a key factor in the time each frame takes to render since we use a new process for each frame that is rendered. I am guessing this might not apply to you though, if you install blender locally on each node, then starting the background process is really fast and probably won't differ that much between your nodes. Unless you have lightening fast SSD's in some and old platter drives in others, that might make a difference.

2. Loading your project - Once the process for rendering starts, it immediately loads your project file. This phase of the render can be impacted by the size of your file and whether your nodes all have similar speeds for disk access/read. What may be a major factor is if you decide to share things like textures/models/sim cache from your local node. These files can be significant and since the local already has them it doesn't have to wait to transfer them across the network, so all the other nodes on your network will have a time delay until they even start rendering since they have to transfer the files from the local node first.

3. Synchronising the render kernel - The next stage after loading the project is when the render engine begins to process your scene and load its representation of your scene into memory prior to starting to draw pixels. This speed of this phase mainly depends on how fast your CPU and RAM are, and a little on the speed at which you can transfer data to your GPU (if you're using GPU rendering). Things that can impact this phase are not having enough system RAM to complete the sync phase. If your system gets short on system RAM it will start paging to the hard drive, using it as a much slower type of RAM so it can continue the calculations without crashing. It really hurts the performance though if this happens. You seem to have plenty of system RAM though, so this might not apply to you.

4. ACTUALLY RENDERING!!! - When the render engine starts drawing, this is mainly the speed of your render device now. This is the part of the rendering where you likely judge how fast each system is. Its not the whole story though. Especially for renders where the drawing phase might be short compared to the other parts. If that is the case, then the other phases of rendering might start to dominate the actual optimised values you should be using for each node.

5. Collecting the render tiles - Once the render finishes, the finished tiles of the frame are collected from each render node. The time taken for this process can vary since it depends on the speed of your network, the size of the tiles, and the compression codec used (you can change this codec, see below, the default is lossless but can result in tiles of over 100MB, the more layers and render passes you choose, the bigger these tiles get, the longer they take). Since the local doesn't need to do this part, it has a speed advantage. Choosing a lossy codec can reduce the advantage a bit.

In summary, optimising a render requires understanding the data flow and the hardware involved. Happy to discuss further if all of that didn't help. It was rather a lot looking back. But thats the nature of the beast :)

I did some tests with and without CPU and it was faster with it selected on all my nodes. I've have had success speeding things up by deleting app data on my nodes and reinstalling the plugin, but syncing is temperamental, with it frequently failing. I'm getting this error when I close Blender on each of the nodes:

WARNING ; CRmain.handle_disc_req: ; got a response that wasn't in the correct format :connect_error

I'm still convinced that I'm not getting the most out of the farm. Currently I'm shaving 12 seconds off a 60 second render by balancing around 80, 10, 10. Admittedly, my local machine has an RTX 3090 and more cores, but I'd expect something closer to 60, 20,20, at the very least.

Are there any other potential bottleneck's I could look into?

Hi, ok, one thing that would be helpful to know is if you're using CUDA, OPTIX or CPU rendering?

The large difference on the local should not be occurring since crowdrender uses blender in background mode to render, which if anything is usually the same if not a little quicker.

One possibility is that if you are rendering with a GPU, you could have the CPU active as well. This tends to lag on some systems.

I've personally seen renders last a lot longer when using CPU, particularly if you use large render tile sizes. A CPU can get a tile that is highly detailed, and spend a long time finishing it, the GPU might have already finished and the CPU is where the delay is.

So I'd highly recommend turning off the CPU in Crowdrender's settings for each node if you experienced this lag trying to use the GPU. Let me know if that has the desired effect?

James