Sanity- it's overrated!

Nothing wrong with ad-hoc; it just needs someone sitting in front of each node ;-)

Thanks for the excellent tips; it's sort of what I had guessed from a cursory read.

I think a good first step would be to create an example, simple blend file, with all the above features.

Then make some basic test scripts that verify all the needed assets (and paths) can be discovered and accessed via Python.

This will make it easy/easier to raise bugs on Blender.

That's quite a real-world, contained starting point for someone.

(Doing that would also help me figure out why just moving one of my projects to a different directory, that's supposed to be all relative, ends up with pink bits!)

I'll try to get to this basic project this week and put it on GitHub. Will post back here- if you have time to take a look and see if it's missing any features that generate local files :-)

Hi Julian, well my approach may be a little ad-hoc, but I change then via the scripting interface and really I just go hunting for them by hand, I haven't really put in the time to figure out a robust way of changing them.

The bpy.utils.blend_paths operator does provide a list of the external references in a blend file, but, it doesn't actually give you a mapping of what data blocks those paths belong to (really annoying!).

So remapping is still hard, in that there's no convenience function I have found that will do this for you.

Some methods for remapping do exist, though see my recent reply to your other question in the forum on creating a cross platform render system. I think you might be able to avoid remapping in blender, and create a workaround where you only need to create mounted share volumes in linux and then use symbolic links to make the assets of your projects appear along side the cr\server\UUID\svr.blend file.

If you still think that it would be a good lesson in life how programming in blender can make you want to scream, then for sure, I would point the students to go look at

1. bpy.data.libraries - where all linked/appended assets are referenced. each library should have a filepath attribute. I say should....

2. bpy.data.images - where all images that are used in textures, env maps etc are, they too should have a filepath attribute.

3. particle cache - ummmm, this doesn't even show up if you do bpy.utils.blend_paths, even though its an externally saved cache ( I tested this, I can see there is a folder created on my hard disk to store the cache, but the folder path never shows up, other than a single blank entry ['']).

4. simulation caches - just had a look at one of my projects, you can find the cache directory using D.objects['fluid domain object name'].modifiers['Fluid sim name'].domain_settings.cache_directory.

I warn you, be prepared to lose sanity and hours of your life digging through blender's python API looking for stuff!

and thanks for the tip on WSL :)

all the best

James

Thanks James! I have a couple of teens interested in doing some development, so may have some interested volunteers :-)

Can you give any pointers to how you change the paths in a blend file? I'm assuming the easiest way will to be load the blend file in blender, and run a script that tweaks the paths and then saves the blend file? Something like bpy.utils.blend_paths(absolute=False,sacked=False,local=False) ?

I can probably debug/work on this occasionally, more so if this does pique these young devs, but the closer you can point me to what functions are involved, the more time I'll be able to focus on giving it a try.

ASIDE: WSL2 might be an interesting twist in the next year (https://docs.microsoft.com/en-us/windows/wsl/wsl2-faq - mentions GPU access...)

Hmmm... the WSL might be an interesting angle for running blender-CPU only.... will have to think on that too.

Thanks for all your quick and interesting feedback!

Julian

Hi Julian,

Ok, you can either;

1. Relocate/replicate the cache, faster as no network overhead at render time, but more work if you change the simulation.

2. Share the cache over the network.

1. Relocation/replication, you need to find the corresponding project folder on the render nodes and copy the cache folder so that it is in the same relative position on the render node's file system as the masters and ensure you have used relative paths on your master. To find the project folder, this is simple, on each node you'll be able to find the Users\Your_user\cr\server folder. Open the project on your master and then do a quick test render, you can cancel it once you see the render nodes have started to render. Each render node will now have modified files in the right folder, just search for the most recently updated folder in \cr\server. This is the clone of your master's blend file. Locate the cache folders in this folder. Lastly, if you changed the paths to relative or at all in fact, make sure you do a resync before rendering as changes to paths and the relative/absolute setting are not tracked and so won't be synced.
   

2. Share the cache. This pretty much will only work (I say only, but you can hack if you feel brave) if you have the same OS type, Simply relocate the cache, and then go to blender and in your file where you have the simulation, change the path for the cache to point to the network location, using an appropriate path that works on each render node. You can test if it works by copying the path and then simply using it in either a file browser or terminal. Again, this works only if all nodes have the same OS. Once again, after changing the path, resync your file with all your nodes connected so that the new path is synced to each render node. Gotchas with this technique are many and varied, but just remember that you might need to authenticate the user on each render node to access the network share, this video again goes through one method on windows to get around this https://youtu.be/TOUSCaduE1s Feeling hacky? What I said about not being able to use the same path different OS types is true, but nothing stops you from manually editing the paths (or scripting this) once you have found the svr.blend file on each render node (though remember this gets overwritten if you resync, if you change paths, resync before you try this hack). This allows you to change the path on each node to one that will work and get around the limitation of blender (and admittedly us not supporting this in code yet) not translating paths. Since you are a human (assumed) you are intelligent and can use your judgment to correctly set the paths that actually work on a per node basis. Tedious? Yes, works? Should do, but not tested, Worth it? Maybe.... If the render times can be brought down by using more nodes, then sure, the longer the render job, the more worth it i think it will be. Of course you have to weigh up your time commitment to trying this to just using the nodes that work right now. You could try a wait calculation to see at what point you should quit stuffing around and just render already :) but I thought it might be useful to at least mention this hack in case you were eager to try getting cross OS up and running.

May the silicon gods be with you

James

Aha- this is starting to make sense! Please do send me some info re locating/syncing sim caches. Though if they're relative to the blend file and relative paths can be made to work... might we get that for free?

And if not, seems like not to bad to create an rsync or similar script to update these when they're re-baked.

Is there a way to hook an external script to the sync/re-sync action?

Very cool stuff. Learning lots about blender!

Packing assets doesn't pack your simulation caches :( you'll need to share those in the same way you share other types of asset. Shout out me here if you'd like more guidance than this :P