So I had a chance to test Crowdrender (CR) this morning and had a few things that came to mind.
Some if it you already mention in the setup video, but I wanted to give you my full set of findings. :)
- Live Tile Updates: When test rendering, it's so important to see the rendered tiles. I stop a lot of renders after a few tiles because I'm just figuring out how much noise is acceptable. If I'm waiting for the full image every time, I'm actually working slower. Updates before the tiles are done would be even better. Most of the distributed rendering I've done with other packages supports this, so it would be nice to see CR do the same thing. (There could be a bunch of extra options you could enable / disable depending on your network speed in the next points, "live" tile updates could also be one of them.)
- Custom Working Directory: Having CR dump everything in to my home folder on Linux is something I'd like to avoid. My machines have a custom scratch disk set up for this kind of thing. It would be nice if CR offered one of two options (or both). A) Be able to define a custom working directory for in in the Add-on prefs. B) Use Blender's temporary directory as a starting point.
- GPU Load Balancing: I'm sure this is something you guys are working on. ;) But I have one computer with three gpu's and one with two, yet the one with two seems to finish a lot faster every time. Maybe look into a benchmark scene you can optionally run the first time you setup CR that can give you some good info for load balancing? No issues mixing AMD and Nvidia so far though, so that's good! Or maybe even progressively smaller tile sizes to keep both machines working until the end?
- Background Sync / Auto Sync: I have a dedicated wired network for my work machines, so it wouldn't be an issue if I had the option to Auto Sync / Background Sync while I'm working. Maybe something simple like an interval setting, or a "Sync on render start" checkbox so I don't get half a pink render when I forget to sync before I render. Again, it's all about time-saving.
- DWAA EXR's: The VFX TD in me has a small aneurysm when I see lossy compression being used inside an EXR. I get that the file size is something to consider, but it would be nice to have the option the use lossless compression instead. For AOV's it's vitally important that the data doesn't get messed with. Another checkbox could do the trick to enable it. (Again, wired network etc.)
- Locked UI / Fullscreen Rendering: I generally render in fullscreen mode with a locked UI, but crowdrender doesn't update the top text then it seems, so there's no visible feedback. Not a huge deal, but might be something to consider.
This all being said, the Alpha runs pretty smooth so far for me. You guys are doing a great job! A good distributed render platform for Blender has been something that I've been looking for for quite some time now and what is here so far feels really nice already. :)
For reference I'm using two machines bot with multiple gpu's, one with Nvidia cards, one with AMD cards. Both running Xubuntu 16.04.3 and syncing over a local wired network.
Hi Daniel :) Yes, that is currently the expected behaviour, but the next pre-release for 0.1.3 will store the optimisation data so that each time you load the scene or resync, it will still be as fast as it can be :D. Stay tuned for the announcement as to when this version will be released to the early access testers. Should be in a week or so from now.
James
Hey guys,
i tried the load balancing on our gpu machines, and noticed, that the rendertime will get lower on every rendering. One machine with 2 and the other with 4 gpus.
Problem is, after recyncing the rendertime is back to high.
Cheers Daniel
Hi again! No worries as we say down under :D
Directory: I'm creating a task in our backlog now ref CR-227
Load Balancing: It should use CPU as a starting point, but then adjust on each subsequent render based on actual performance, so if you can verify you have the correct setup as per instructions above, try rendering a few times with a simply scene (for the sake of an expedient experiment!) to see which set of cards is actually quicker. Also as I said, Jez is working on making all the render settings update in real time. This will make the whole process easier. You'll be able to change the device used in the render settings panel to GPU/CPU without needing to re-sync your files. You'll just need to set the user preferences on each machine you use first to GPU and enable the cards you want to use. Then save. Sorry, its a bit fiddly right now :) We're working on making it better.
Sync: Great to hear that the pink went away after a resync. That check box doesn't sound too hard to implement. Let me have a think about this. On the surface it appears easy enough to implement, just want to have a good old think about what could go wrong!
EXR's: I'm creating a task for this ref CR-226
Lock: I gave this a go, locked UI indeed! Seems it stops everything when rendering with crowdrender. I guess this is a result of the way our addon works. I will need to do some further investigation. For now its safe to say its probably not a good idea to lock the UI and then render with crowdrender unless you enjoy clicking on paintings.
GPU and CPU: There has been much interest in us developing this already and so we've got an issue in the backlog already (CR-212 just in case anyone would like to know where its up to!). Stay tuned as we experiment with this. There are some extra things we need to add to the code to make this happen, first of all is that we'll need a way to adjust the tile sizing so its different for GPU/CPU since they won't like to share tile size, in my experiments GPUs work much better with larger tiles, which CPUs struggle to cope with.
Then there is the problem of how to make configuring all that nice and easy and not a terrible point and click adventure. Its in progress right now, you liely won't see it in the 0.1.2. release but it is def on the drawing board.
Awesome sauce!
Thanks for all the info! I'm glad some of the things I mentioned make sense to you guys. :)
Directory: I figured the custom directory wouldn't be terribly hard to realize, and I agree with putting it in the add-on preferences panel, seems like the most logical place following the example of other add-ons.
Side note: It would be nice if CR respected the directory set for rendering and not create it's own. Or if it has to do that it at least cleans up after itself. :) But I'm guessing that stuff is just there for debugging purposes currently. Especially, for bigger projects I try to keep a tight file-naming convention. If things need to be automated, I'll always know exactly where they are.
Load Balancing: It's a tricky issue indeed, one of the machines has a 4-core cpu and 2 gpus and the other is an 8-core with 3 gpus, so what I was seeing in terms of balancing would make sense if it's basing everything off cpu at the moment. But leveraging both AMD and Nvidia cards from both machines at the same time is interesting, to say the least. Once you have command line rendering setup, let me know as well. It's generally how I batch render scenes overnight, and it's faster than rendering in the UI.
If you'd like a big production scene that requires a lot of resources to test, let me know, I'd happily supply one. Once I get your email I'm sure we can chat more about it.
Sync: Yeah, I had a pink render, but after syncing it was fine. The point is just that I don't want to waste time clicking sync, or forgetting it when I pop off a render before going to bed and waking up to half a pink render. An "Auto Sync Before Rendering" check box would be enough to alleviate that I'd say.
EXR's: Thanks, a menu would be more than satisfactory! ;)
Lock: Yeah, it's a small button indeed. It's just a personal preference on my end, but knowing what's going on in the background would be nice.
Like I said though, it's looking pretty impressive already. You guys are on to something here! :D
And one other thing, how possible would it be to leverage both GPU and CPU power simultaneously, seeing that you're already starting multiple threads in the background? That would be insane.
Hi there :)
So, lots of great feedback there, thank you Mr Mantissa! I'm glad that overall you liked it :D. Now lets dive in!
Live updates
These are in the pipeline for sure. They're part of a planned overhaul of our core application that will resolve both integration with the compositor and drastically reduce overhead.
Custom Working Directory
Ok great :) this could be implemented by giving you an option in the addon's user-preferences panel. I'm thinking similar to how blender currently has settings for things like paths to scripts in the user_preferences. If users are used to going there for configuring where blender stores files, then this option might be best placed there. Putting it in the render settings panel might cause clutter since once you set this, I doubt you'd need to set it each time you use crowdrender, so it would be of little value having immediate access to.
Not a big edit for us really, the cr path is currently hard coded to be the expanded user path. Its easily switched to something else, I'd need to edit the layout of the user preferences panel for our addon.
GPU Load Balancing
Hmmm, ok this is a little odd, I'd like to look into this, I think I am going to send you a separate e-mail, but for the benefits of others that may read this thread, here's how the load balancer works, and how to get crowdrender to actually render with GPUs, since we haven't released a how to on how this works yet :D
First, the load balancer initially calculates the workload of each machine based off CPU cores. This is due for replacement in the overhaul, but for now its used as the starting point, and for GPUs its not relevant. The first time you render, esp using GPUs, it won't be well optimised. However each time after you render (from the start of your session) the load balancer calculates the relative processing speed for each machine. It then uses this to figure out how much work each machine can do and finish at the same time as each other.
Second, we're working on better support for GPU rendering, I think we're getting there. But for now, you'll need to follow these steps
First, you need to make sure that GPU is saved in your user preferences-> system as the compute device you want to use.
Then you also need to make sure that GPU device is also selected in the render properties panel.
Connect to your machines using crowdrender.
If these computers you are connecting to have already once been connected to using this blend file, you should do a resync. The GPU device setting is not yet tracked and uploaded automatically so if you originally used CPU as the device then all the computers you connect to will use CPU and your local machine GPU. The same goes for tile sizes. I know I know its annoying, that is why Jez is fixing this so all render settings will be updated live, including tiles and the Device used in the render settings panel.
Background Sync / Auto Sync
Hmmm, so you'd be ok with doing a complete sync before rendering, am I reading you correct? I get the feeling you have tried something and it hasn't worked resulting in a pink render! At least judging by that comment anyway! So, what is supposed to happen is that we track all the data in the scene to make sure that if there is a difference that would result in a pink render, the computers in that state would signal this by showing "Sync fail" as their status in the crowdrender panel.
If all nodes are showing "synced" and you render and get missing textures or other pink bits then there is a problem with the tracking system and I'd like to know more so I can fix it :)
DWAA EXR's
Yeah I hear you! So I can certainly make it so you have other options available to you. And by now I am starting to feel very jealous that you have a wired network :D. Yep, a menu for this would be no hassle, just where to have it would be the trick, I am leaning more towards having it in the crowdrender panel proper so its readily available.
Locked UI / Fullscreen Rendering
I am a little embarresed to admit I had to look this up, and then had to stare hard at the render panel for a bit until I saw the lock icon. This is why we do community testing, there are always features used by artists that we've never used (or sometimes sheepishly admit that we've never heard of).
I will test with this setting and see what happens on my system since I am curious about it now I know its there!