Serial MIDI

Contribute Feature Requests
#1

UPD: Tons of text below here are result of my belief that requested feature is already here.

Hello. I would like to ask about Serial MIDI capability like ttymidi/hairless midi.

Context, why I am interested

I would like to explore some possibilities to additionally control MDX device with ESP32, that should be connected via USB to the MDX. For example:

  • Try out creating a very lightweight web interface using ESP32 with independent WIFI hotspot to invoke CC commands in browser or via rest API for easy integration with python code or whatever would be convenient to implement remote automation or control surfaces.
  • Try out ESP32 MIDI BLE library as a workaround for old bluez stack on MDX, to offload BLE MIDI capabilities to ESP32 in attempt of making some kind of CME-alike dongle out of ESP32 device, to be able to control MDX from phone or M-Wave chocolate that I am waiting to be delivered.
  • I also have TTGO-WATCH 2020 chinese ESP32 based smartwatch for nerds (basically an esp32 with touchscreen and a couple of sensors) that probably could provide some additional lightweight remote control device with a touchscreen/accelerometer axis and stream data via BLE to a regular ESP32 Wroom connected to the MDX as a receiver.

All these random ideas to play with are limited by the fact that ESP32 lacks USB MIDI class compliant capabilities and provides only serial output.

Why I was thinking it should work

At this point I have some blurry memories of noticing Serial Midi port in web interface of Mod Duo X some time ago, when connecting my homebrew STM32 based midi pedalboard prototype, that also has serial output in addition to showing up as midi device. Feels like I’ve noticed it in the list of ports briefly, said “huh, interesting” and forgot about it until now. I even not sure if it is not a fake memory actually.

I’ve found that MOD GitHub has ttymidi in sources, that make me think that I am not crazy with this whole idea:

If it is a really existing feature - it will be much easier for me to try to play with a checklist about, as I will skip the part where I need an additional bridge between esp32 and MDX.

** What I’ve actually got**

However, I do not see this type of port anymore while connecting the same device:
image
While serial port is to by added to the system properly:

image
image835Ă—159 7.73 KB

jack_lsp shows some ttymidi port (however even with my serial device not plugged in, probably has to do something with a ControlChain?)
image

I even tried to create a new port on my own:
image

But it still does not appear in web interface in the list of midi ports.
(Probably there is something to do left in the jack routing, but I am not that familiar with jack yet).

So, finally here is a question.

Are my memories correct about seeing Serial midi port existing on MDX as a feature? If so, was it removed on purpose, or was there a regression, or there are some additional steps to make it work?

1 Like
#2

It would be interesting for you

2 Likes
#3

Yep, that’s seem to be cool and capable project, probably I would give it a shot under a better circumstances. Unfortunately I do not have access to anything else but a a small stack of microcontrollers currently (arduino micro, esp32 and stm32) that I could take with me, and unlikely will have stable access to any DIY tools and components soon, while Pedalino requires to find and solder some connectors and resistors for DIN MIDI due to lack of usb connection.

1 Like
#4

i’d suggest that too. There codebase does most of the stuff you are looking for (and can be adapted or copied) for creating something similar.

If you can get your hands on a Arduino pro micro or some 220 Ohm resistors its pretty easy to build a midi interface. Did it with both for the dwarf.

image
image2945Ă—1593 172 KB

3 Likes
#5

Oh, I did not notice that they use pro micro as a middleman, I just have one or two of those. Probably I need to have a closer look to the sources, might be a feasible plan B to s̶̶t̶̶e̶̶a̶̶l̶̶ borrow, once it is already written. Even with the current limitations it should be still possible for me to solder esp32 to pro micro, it is still easier than to search for DIN connectors in Tbilisi.

3 Likes
#6

Still curious about making ttymidi-port visible in UI though, Might be a dead end however, I see a bunch of xruns happening when trying to run ttymidi against my usb serial port, and it can ruin any ttymidi related attempts.

Probably I could also try hijacking into ControlChain - it seems to be listening for /dev/ttyS1
image

image
Looks like it is possible to replace ttyS1 with ttyACM0 and mimic controlchain device from esp32 via usb serial instead of real control chain port, lol. However it is unlikely expecting that underlying vfs device will be plugged in or plugged off during runtime.

UPD: ControlChain arduino library seem to rely on low level data transmission heavily, it looks like it is not a simple task to switch to usb serial connection. Bad news for someone who does not want to do a real physical shield.

3 Likes
#7

Just for history - I was able to rebuild mod-ttymidi from sources using mod-plugin-builder docker image

Had to add

${BR2_MAKE} jack2mod

into the end of bootstrap.sh before building image, otherwise jack will not be found during build.

Build was as easy as

source local.env modduox

in mod-plugin-builder, and running make debug in the mod-ttymidi sources.

As result, I’ve obtained a debug version of ttymidi, that provides some verbose output.

image
image702Ă—308 3.12 KB

As far as I can see, this debug ttymidi is perfectly able to parse notes and CC from serial input at least. However, there is still not a clue why xruns happen, and why despite of the new ttymidi instance being connected to jack according to jack_lsp - UI seems to be completely unaware about it, probably will have to look into mod sources once again, or I just do not understand jack connections enough yet.

4 Likes
#8

Further digging.

Looks like ttymidi generates midi messages fine as well.
I’ve made a virtual port with jack_midi_dump monitoring, connected it with jack_connector to the arduino’s ttymidi, and it prints incoming serial midi messages coming from arduino just fine.

However, midi display plugin (effect_1) ignores this input, while is connected the same way, probably due to some unknown additional filtering.

image
image1231Ă—659 209 KB

Meanwhile, the same effect_1 midi display plugin sees usb midi keyboard’s messages if I connect it by manual jack_connect invocations.

2 Likes
#9

Okay. After reading mod-ui sources, mod-host sources and playing with API’s, there is some stupid progress on my topic.

If instead of trying to make additional ttymidi instance to work, I will override MDX’s setting of the internal serial port responsible for hardware DIN midi input as below:

[root@modduox ~]# cat /etc/mod-hardware-descriptor.env | grep ACM0
  MIDI_DEV_SERIAL_PORT=ttyACM0

I am able to sacrifice hardware midi DIN ports and instead use arduserial-based arduino as input with no xruns.

It works only if arduino is plugged in before powering on MDX, and replugging it on the fly will not work.

To my understanding, working with ttymidi is hardcoded in mod-ui and below, and it was never assumed that two ttymidi instances will ever appear.

It is unlikely that I can do all the necessary changes in underlying Python/CC+ code and patch my MDX without breaking everything apart.

However, as soon as it is more or less proved that mod-ttymidi is already capable of parsing data produced by Arduserial library, it should be possible for the dev team to implement that.

I will make a feature request from this.

2 Likes
#10

Feature request itself:

My suggestion is to make a last step to implement compatibility with ArudMidi library, to make Mod even more friendly to geeks, controlling MOD with just spitting out some serial data.

Context

Currently, if you are willing to create a peripheral control device for MOD you already have some interesting choices:

  1. ControlChain - this requires investing into a physical shield, and it seems to be limited in microcontroller choice of only a couple of arduino controller currently. It is not something easy or friendly to start with writing some code instantly. Porting to other microcontrollers does not seem easy at this point to me.

  2. Hardware DIN inputs - that also will require some additional hardware, connectors and resistors and additional powerline. Feasible, but still not something where you can start from the software.

  3. USB Midi device emulation - this one is most “friendly” approach currently, as for a start you need only a microcontroller and plug it to usb port, and you already have a lot of things to play with in software, before you have to start soldering something. To my knowing, USB midi is supported by Teensy, Ardunio Pro/Micro and STM32 with maple core, with existing libraries and examples. None of these can into wireless, as far as I know, unfortunately.

Suggestion

There could be a 4th option. The are other microcontrollers, that lack USB midi support, like ESP32 wroom/wrover devices. These are interesting by their on-board bluetooth and wifi stacks, that may engage developing wireless controls for Mod devices, with communication as easy as throwing data to serial according to the already existing protocol:

Communication with these devices could be done by using ArudMidi (~ttymidi-team/ttymidi/trunk : files for revision 31)
that encodes midi messages into a regular serial communication.

There are also a lot of usb-serial capable, but not usb-midi capable devices - cheap arduino nanos for example that can benefit from this extension.

As far as I can see, MDX already has ttymidi as a part of the firmware, and is already used to parse messages from the hardware DIN midi port, and according to my PoC it was capable of parsing midi data from usb serial port of my arduino as well.

Would it be possible to extend existing ttymidi usage to an additional serial ports, in a way where additional USB serial ports could appear in UI?.

3 Likes
#11

How do these connect to the MOD units? Is this USB serial? then MIDI data flowing through it?

If that is the case, perhaps some automated udev rule that loads up ttymidi against that device could work

2 Likes
#12

Yes, that’s right, I mean this approach. Looks like ttymidi’s initital purpose was that - providing workaround midi-through-serial capabilities for devices that are not capable of real midi.

P.S.

At this point, after reading source code again and again, I’ve understood that in MDX ttymidi should not be launched standalone, but it’s proper use is to be launched as a plugin by jack.

With this information in mind I was able to set working communication without xruns as below:

mount / -o remount,rw
cp /usr/lib/jack/ttymidi.so /usr/lib/jack/ttymidi2.so
mount / -o remount,ro

USB_SERIAL=`ls /dev/ttyACM* -1 | head -n1`
echo $USB_SERIAL

jack_load   -a ttyarduino ttymidi2  $USB_SERIAL

For reasons that I do not understand yet (however suspecting some semaphore-like stuff), I have to launch this second ttymidi listener from copy of ttymidi.so binary, otherwise communication stucks completely, like I cannot use same binary twice.

New midi devices do not appear in UI ( because it is not real midi device, heh), but xruns are gone and midi display lv2 plugin sees messages that are sent through usb serial port by my microcontroller.

Looks like I will be able to acheive my goal in “userspace” after all, after adding some udev rules.

Possibility to repeat the same with some kind of UI would be nice, I guess, but as for me - I am happy enough right now.

I am really amazed about the level of customization achievable with MOD, if you are stubborn enough.

3 Likes
#13

Yet another update on this edge case.
All my testing was done using arduino pro micro, and it’s usb serial was supported out of the box. But suddently I’ve found that ESP32 (that was the main reason for all this trouble) uses ch341 or CP2104 drivers for serial communication (maybe there are others out there, but mine used these two) , and these are not included into MDX firmware.

Had to build ch341 driver module on my own and to inject it into MDX approximately as below:


#>>>on MDX: >>> 

# -> obtain current  MDX kernel configuration from, for example:
 zcat /proc/config.gz  
# ->  save whole output somewhere. I used WinSCP for that actually.

#>>>On docker-host machine  >>> 

#build image

git clone --depth 1 https://github.com/moddevices/mod-plugin-builder
cd mod-plugin-builder/docker
docker build -t modduox-plugin-builder --build-arg platform=modduox .

# run plugin builder docker container as a build environment
docker run --rm -it --entrypoint bash modduox-plugin-builder
 
# -> put config extracted from config.gz into running container, I used docker cp.
#  docker cp /tmp/config  2e0af363aa14:/home/builder/

#>>> Inside container >>>

#load environment variables stuff that points us to toolchain location

cd ~/mod-plugin-builder
. local.env modduox
cd

# obtain current kernel sources
git clone https://github.com/moddevices/linux-rk3399
cd linux-rk3399
git checkout -b rt176 origin/develop-4.4-rt176 # seems like it is current kernel for modduox

#  insert config that we got from MDX
cp  /home/builder/config  .config

# enable CH341 module in config

echo CONFIG_USB_SERIAL_CH341=m >> .config
cat .config  | grep -i ch341

#try to configure kernel build using that config we got from mdx
make olddefconfig
 
# build module
 
make ARCH=arm64 CROSS_COMPILE=aarch64-mod-linux-gnueabi-   scripts prepare modules_prepare
#  some of the flags we loaded from local.env  do break the build, unsetting them back:
unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE`
make ARCH=arm64 CROSS_COMPILE=aarch64-mod-linux-gnueabi- drivers/usb/serial

#locate kernel module.

find  . | grep ch341.ko
# and somehow transfer it to MDX.
 

#>>>on MDX: >>> 
 mount / -o remount,rw
# put ch341.ko  file that we built to /lib/modules/4.4.167-rt176-duox/
# ...
 mount / -o remount,ro

depmod 
modprobe ch341   # at this point module is loaded and ch341-based  ESP32  starts working fine.

Really not sure if messing with building kernel modules is a good idea, and what things I could do wrong, but module seems to work, and ESP32 started sending serial-midi to the pedalboard.

UPD:

Udev rules to reconnect automatically:

/etc/udev/udev.rules/94-ch431.rules

ACTION=="add",     SUBSYSTEMS=="usb-serial",  ENV{ID_VENDOR_ID}=="1a86", ENV{ID_MODEL_ID}=="7523", RUN+="/data/plug_ttyUSB.sh"
ACTION=="remove", ENV{ID_MODEL}=="USB_Serial",   ENV{ID_VENDOR_ID}=="1a86", ENV{ID_MODEL_ID}=="7523", RUN+="/data/unplug_ttyUSB.sh"

scripts: (require chmod +x)

/data/plug_ttyUSB.sh

#!/bin/bash

USB_SERIAL=`ls /dev/ttyUSB* -1 | head -n1`
jack_load   -a ttyarduino ttymidi2  $USB_SERIAL

/data/unplug_ttyUSB.sh

#!/bin/bash

jack_unload   ttyarduino
3 Likes
#14

Congrats to your stubbornness! I love hearing a good developer story that might very well have ended in a dead end. :+1:

4 Likes
#15

ezgif-3-e845edcc64

So here is a first thing to share that is not just a wall of text or speculation. What’s happening in this gif:

  1. Midi commander (android app) sends CC midi data from XY control surface to ESP32 as to regular MIDI BLE device.
  2. ESP32 powered by MIDI BLE library receives the data, and uses ardumidi to throw this data to MDX via usb serial
  3. MDX extended by ch431 driver, sees this data in /dev/ttyUSB0
  4. additional ttymidi instance sends CC data to the pedalboard.

As result, I am controlling pedalboard parameters with my phone.
MIDI Berry on pc seem to work fine with this ESP32 module as well.

Looks like ESP32 as a poor man’s WIDI is possible, will try to use it with Artiphon Orba 2 controller and M-Vave Chocolate, once I’ll have them in my hands.

11 Likes
#16

that’s awesome! Great work :slight_smile:

1 Like
#17

ttwatch

Here how it works in pair with TTGO Twatch2020v1 (ESP32 based chineese smartwatch) with a BLE MIDI sending firmware, that maps accelerometer tilt data to 4 CC signals.

I think that after some development it could be used as a something like a Source Audio Hot Hand controller.

10 Likes