New neural lv2 plugin from Aida DSP based extensively on existing NeuralPi, reduced to the bone

madmaxwell · July 8, 2022, 8:49am

Hello, a few months ago I’ve put myself into this crazy neural stuff for audio dsp. Since I was not fully satisfied with NeuralPi plugin (build issues due to JUCE, built-in effects that are not needed on mod platform) I’ve pushed this

Since I have Neural DSP Archetype Plini and I think it sounds amazing, I’ve trained a model over Electric Sunrise: Riff preset without parametrizations (you can’t change gain and volume of the model). This is a very high gain preset for killer riffs.

The result is really promising although cpu consumption for now is really high. By switching to 1024 as buffer size for jackd I was able to hear it and it sounds amazing like the original.

Of course this is a lot subjective, so what I like of the neural approach is that is has ESR measurements embedded into the training workflow, so we can let the numbers talk.

The images below show the response of the trained network against an input which it has never seen (test)

we of course have audio too, see online archive here, go to AI dir and the pick latest training. You can do audio A-B comparison between:

aidadsp-1-target.wav in Data/test dir which is the original aidadsp-1-input.wav (dry) material rendered through the Neural DSP plugin (I’ve used Reaper DAW on Windows 10)
in Results dir test_out_final.wav which is the output from the trained neural network

The Dataset used is an original one I’ve put together since I was not fully satisfied with the stock one proposed by original developers.

I’ve also forked Automated-GuitarAmpModelling repo to inject changes needed for using this dataset

I’ve still have a Colab subscription so if you’re interested I can train other models, you just need to record properly the thing you want to model. I can share details of the procedure in a separate thread.

For the lv2 plugin code all the credits go to the original authors @keyth72 @chowdsp and others. I’ve just used LSTM class which let me came with a simple lv2 plugin with 4-5 calls to this class and it’s done. Of course like I said before some development work is needed to shrink cpu usage, not to mention integration with mod-ui filemanager and neural models json files selection, which I would like to add in the future.

OOT: still a little nightmare is mod-sdk, I’ve pulled official docker hub image and also builded latest source in a 18.04 ubuntu container and always getting thumbnail/screenshot deploy error, so @falkTX if everything else is ok I would ask for those two images thanks!

Azza · July 8, 2022, 10:33am

Amazing !!!

spunktsch · July 8, 2022, 3:53pm

really nice stuff - although I understood only half of it. Do the audio samples of results have cab emulation on?

spunktsch · July 8, 2022, 4:08pm

please do. If I want to be able to control gain and volume how much more is there to do?

madmaxwell · July 8, 2022, 4:20pm

No, they don’t and they shouldn’t, at least if one want to capture only the amp’s tone. Usually the cab is skipped since you may want to apply your cab sim in cascade. Althought given the technology, why put limitations…we could train models with and without the default cab used in the preset…

For the parametric models: the work need to be done only in the forked Automated-GuitarAmpModelling that to simplify is the (training) script that produces the lstm-model.json. @keyth72 already implemented parametrization in his own fork of Automated-GuitarAmpModelling, I need to cherry-pick and adjust his commits on top of my changes. I think should be feasible…

spunktsch · July 8, 2022, 5:21pm

ok, thought so but good to know. If you need help with the gui stuff just hit me up.

madmaxwell · July 9, 2022, 8:46am

For the graphics I would need to finish the current package with thumbnail and screenshot, but I’m not able to export them from mod-sdk, using official docker container…thanks!

[Update] trying to solve high cpu usage issue with original developers, bumped latest RTNeural and following their suggestions in next branch. Will do a recap after the tests

[Training] while I’m preparing the dedicated thread, a few infos. On Colab Pro with nvidia Tesla with current training script it takes 2h to finish the training (with no parametrization), plus the time to register the tracks. Currently I use two tracks, bass and guitar, and to register it is sufficient to click “render” with the given plugin settings. Is better to render instead to record it since no latency is involved, this is super important the input and target tracks are time aligned! Of course a whole different story is if analog outboard needs to be recorded. In this case we rely on DAW’s latency comp afaik.

falkTX · July 9, 2022, 9:39am

phantomjs might be missing from the container.

if you have a duox or dwarf, you can install the developer images which contain mod-sdk with a working screenshot generator.
https://wiki.moddevices.com/wiki/Developer_Images

Or ignore that for now, we can help with the screenshot later on.

micahvdm · July 11, 2022, 8:53pm

Hey @madmaxwell i tried to build this on a pi 3, but it just errors out. Are there any specific things I need to do when building?

madmaxwell · July 11, 2022, 9:46pm

@micahvdm can you check out https://github.com/MaxPayne86/meta-moddevices-contrib/blob/dunfell-hotfix-1.11/recipes-plugins-extra/aidadsp-lv2/aidadsp-lv2.bb?

I’m using yocto 64bit OS dunfell branch so I have GCC 9.3.0 and I’m compiling with aarch64. I have an alternate build system with yocto sdk I’ve generated that potentially is more similar to RPi3

1. go to sdk download dir and type ./poky-glibc-x86_64-aidadsp-sdk-image-aarch64-nanopi-neo2-toolchain-2.1.15.sh you can provide install dir on the cmd line
3. source environment-setup-aarch64-poky-linux
4. git clone https://github.com/AidaDSP/aidadsp-lv2.git && cd aidadsp-lv2
5. mkdir build && cd build
6. cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DRTNEURAL_XSIMD=ON -DRTNEURAL_DEFAULT_ALIGNMENT:STRING="16" ../
7. cmake --build .
8. make install DESTDIR="absolute path of a local dir on your Pc" (this is just for deploy this dir to the target device)

Problems: in my setup it does compile & run, just cpu consumption is high and force me to use 1024 buffer size in jackd. If I compile with the CXXFLAGS and LDFLAGS commented in the recipe, it does compile but became a nice white noise generator…

madmaxwell · July 11, 2022, 10:00pm

Below my docker mod-sdk recreation, step by step (where thumbnail/screenshot deploy doesn’t work, just like the official docker hub mod-sdk). Phantomjs seems installed properly.

docker pull ubuntu:18.04
docker run -it -d --network host -v ~/Stuff/lv2:/lv2 --name "mod-sdk" ubuntu:18.04 bash
docker exec -it mod-sdk bash
apt-get update
apt-get install build-essential liblilv-dev phantomjs python3-pil python3-pystache python3-tornado python3-setuptools python3-pyinotify
apt-get install git
cd /home
git clone https://github.com/moddevices/mod-sdk.git
cd mod-sdk
git checkout e0a9cf283982d16ced2c01fd2cbd0de77c1b8640
python3 setup.py build
export LV2_PATH=/lv2
./development_server.py

brummer · July 12, 2022, 9:52am

You may try this patch for webserver.py:

github.com/moddevices/mod-sdk

make_screenshot ain't work with tornado > 5.0

opened 06:59AM - 26 Nov 18 UTC

brummer10

Hi After a recent update on debian/sid I notice that the screenshot generatio…n ain't work any more. I found that's because in the function `loop.add_handler(proc.stdout.fileno(), proc_callback, 16)` the proc_callback get never called. (with tornado > 5.0) So I've replace that with a watcher thread to make it work for me. Here is a patch to fix the issue with tornado >5.0 Additional I've added a signal handler to catch the KeyboardInterrupt and exit clean. regards hermann ``` diff --git a/modsdk/webserver.py b/modsdk/webserver.py index 5e32201..ffc73e2 100644 --- a/modsdk/webserver.py +++ b/modsdk/webserver.py @@ -3,12 +3,18 @@ import json import os +import sys import random import re import shutil import subprocess import pyinotify +import asyncio +import signal +from threading import Thread +from queue import Queue +from pydispatch import dispatcher from base64 import b64encode from PIL import Image from time import time @@ -511,27 +517,35 @@ class Screenshot(JsonRequestHandler): tmp_filename = ''.join([ random.choice('0123456789abcdef') for i in range(6) ]) return '/tmp/%s.png' % tmp_filename + def proc_callback(self, message): + fh = open(self.fname, 'rb') + os.remove(self.fname) + self.handle_image(fh) + + def proc_watch(self,out, queue): + asyncio.set_event_loop(asyncio.new_event_loop()) + for line in iter(out.readline, b''): + queue.put(line) + out.close() + dispatcher.send(message='screenshoot is ready', signal=self.signal, sender=self.sender) + sys.exit() + def make_screenshot(self): - fname = self.tmp_filename() + self.fname = self.tmp_filename() + self.sender = 'watcher' + self.signal = 'ready' + dispatcher.connect(self.proc_callback, signal=self.signal, sender=self.sender) proc = subprocess.Popen([ PHANTOM_BINARY, SCREENSHOT_SCRIPT, 'http://localhost:%d/icon.html#%s' % (PORT, self.uri), - fname, + self.fname, self.width, self.height, ], stdout=subprocess.PIPE) - - def proc_callback(fileno, event): - if proc.poll() is None: - return - loop.remove_handler(fileno) - fh = open(fname, 'rb') - os.remove(fname) - self.handle_image(fh) - - loop = ioloop.IOLoop.instance() - loop.add_handler(proc.stdout.fileno(), proc_callback, 16) + self.q = Queue() + self.t = Thread(target=self.proc_watch, args=(proc.stdout, self.q)) + self.t.start() def handle_image(self, fh): screenshot_path = self.data['gui']['screenshot'] @@ -786,10 +800,15 @@ def run(): if not check_environment(): return + signal.signal(signal.SIGINT, signal_handler) lv2_init() welcome_message() make_application().start() lv2_cleanup() +def signal_handler(signal, frame): + print(' Thanks for using the MOD-SDK, bye bye.') + sys.exit(0) + if __name__ == "__main__": run() ```

Azza · July 12, 2022, 1:26pm

Hi everyone,

would you mind if I move this thread to the developer section of the forum? I think it would fit better there.

madmaxwell · July 13, 2022, 12:17am

Tried again the whole procedure with your patch but id didn’t solved. For this I think better to continue in another thread. We could gather somewhere python3-tornado 4.3 and use it in the container

madmaxwell · July 13, 2022, 12:34am

UPDATE: after some big support from @chowdsp @keyth72 I’ve switched to models with hidden_size=16 and measured a cpu usage drops by 50%. The hidden_size is specified in the config file of the training, in NeuralPi hidden_size = 20 but in new DarkStar plugin is 16 for reference. In the next days I will focus on cxxflags and ldflags to further improve performance, but the hidden_size seems to play a major role. I will also study for drawbacks in quality when reducing this parameter.

madmaxwell · July 27, 2022, 9:12pm

Guys…it’s done! I’m running a decent archetype-you-know preset with 128 buffer…

since I used cmake everything should be portable to Mod Duo build system effortless. Then several stuff, depending on how much you are interested, can happen. We would need to support these models in file explorer for example. In addition to that I will now dedicate in creating new models to came with a decent workflow for registering trainable datasets.

redcloud · July 27, 2022, 9:15pm

Shut up and take my money!

spunktsch · July 27, 2022, 9:20pm

exactly.

Avery_Devine · July 28, 2022, 1:45am

This is huge! NeuralDSP archetype plugins would be my go to VSTs if this platform supported them. So this news is amazing!

madmaxwell · July 28, 2022, 6:03am

Please beware that I’m not running the original neural vst! With this plugin we can load models trained over existing hardware. I have trained over a neural dsp plugin since it’s way faster than actually registering a real amp and honestly the result surprised me.