Marvin Haagens Tech Blog

I have written a Javascript function that will parse the updated HTML code attached to a received SSE event into DOM elements and replace all elements that match the XPath given in the received event against the DOM elements specified in the received SSE event.

I will now work on sharing the SSE connection accross multiple browser tabs.

I managed to add Thymeleaf to my project, created a Server Sent Event endpoint and made it to call the Thymeleaf template engine from code.

The client registers to that SSE endpoint using pure Javascript. When the client connects to the endpoint, the controller calls the Thymeleaf template engine which processes a specified fragment in a specified template and returns the finished HTML code as a String. That String is then sent by the controller to the client as a custom event using the registered Server Sent Event endpoint, where the Javascript message handler receives the event with the HTML and displays it for demonstration purposes as a pop up message using window.alert().

The current status of the project can be seen here:
https://gitlab.marvin-haagen.de/software/oskar/-/tree/23-migrate-to-vaadin?ref_type=heads

My next step is to extend the Javascript code on the client side so that it replaces the DOM subtree of the component addressed by the received message with the HTML specified in the message received from the SSE endpoint. I need to write that function in such a way that I can reuse it for updating other components on the webpage without having to modify the Javascript code.

Also I need to find a way to share that SSE connection between multiple browser tabs because there is a low limit to the amount of SSE connections that can be active at the same browser for the same domain.

I tried out using Vaadin, but I decided to not using it. My main point for deciding against Vaadin is that when Javascript on the client is disabled then no content at all is shown at the page, not even the static parts of the page. I instead want my webfrontend to be as usable as possible even when Javascript is disabled.

So I will instead try a first simple demo implementation using Thymeleaf and Server Sent Events. My idea is to generate the HTML of a page on the server using Thymeleaf, containing all the dynamic information that is available on the frontend at that time and embedding a simple Javascript function into the distributed HTML that registers to a Server Sent Event endpoint on the frontend where updated parts of the HTML are pushed to the client when information has changed.

That way, with Javascript enabled the web client is receiving real time updates from the server as soon as new information is available, but if the client can not execute Javascript the user just has to refresh the page to see new information, because the client still receives a whole updated HTML page on the pages refresh.

In the last days I improved my client application, so that source informations that have been received via the newly added WebSocket connection are communicated to all relevant components.

I also did a heavy rework of the client code itself, to a more modular, component-based approach.

In the next days I will try if it would be viable to implement a web frontend using Vaadin to replace the Java client.

I now have added a raw WebSocket endpoint that distributes the current SourceInfo to the client that initiated a connection. In the next step I will connect the client application with this endpoint and register a listener which updates all source related things in the client when a SourceInfo message is received on that endpoint.

When this is done, I will improve the backend so that every time when a Source or its status is changed the current SourceInfo of all sources will be distributed to each client currently listening on that endpoint.

After that I will remove the REST endpoint that currently is used for polling the current Source status.

The source code of the project can be found here:
https://gitlab.marvin-haagen.de/software/oskar

Alternativly you can also find the source code on my old gitlab.com account, where the projects main branch is being duplicated to:
https://gitlab.com/marvinh/oskar

Currently I’m working on a fully automated trading system. The project is already running for a long time because I had taken a lot of heavy architectural changes since the start and the project had some larger development breaks in between. It started as a micro service architecture using plain Java servlets, which was quite a mess since I had to write a lot of boilerplate code for registering the REST endpoints and mapping messages. Later in development I learned about Spring Boot and decided to do a rewrite for a more simple code base. But I was not happy after that since I noticed that with the micro service architecture a lot of heavy communication between the parts of the system would be necessary. For example every Source (which had its own service at that time) had to register at the database. Every database node would need to register to the backend and if a new price occurs that price would have been needed to be synchronized across all running database nodes. How to detect which data row between mismatching databases is correct? Maybe using some kind of versioning tool like Git? But that would have meant more complexity, more possible error sources and a lot more development time. So I did another rewrite for a more monolistic approach. It now consists of a backend which holds the database, runs the Sources and handles requests and a client application which allows the user to interact with the system.

Currently the user can create Sources by uploading a JAR plugin that can query price information for a specified product on a specified market from a given data source. The data source can be a website which HTML tree is crawled or any other source that provides price information. After creation, the source can be configured and activated or deactivated. When activated, the source provides information to the system either by polling or by waiting for messages from its underlying data source.

At the moment I’m working on storing the collected prices to the database and communicating new collected price data to the client application. When that works, I will show the price data in the client application using charts. I’m currently experimenting on using WebSockets to communicate the prices to the client so that I hopefully can avoid polling against the database for new price information. Originally I thought about using RSocket for this, but WebSocket is better supported and allows me to maybe later add a web interface to the trading system for users that don’t want to use the dedicated client application.

The source code of the project can be found here:
https://gitlab.marvin-haagen.de/software/oskar

Alternativly you can also find the source code on my old gitlab.com account, where the projects main branch is being duplicated to:
https://gitlab.com/marvinh/oskar

Today I noticed that fail2ban has blocked someone from trying passwords via SSH on my Gitlab server.

Since this has been the second time of someone trying I decided to increase my servers security by setting up two factor authentication via the SSH. I also have created a script that will notify me via email any time someone successfully manages to login via SSH.

I have updated my tutorial on How to secure a root server accordingly and added a tutorial on How to setup two factor authentication via SSH. You can find them here:

• How to secure a root server
• How to setup two factor authentication via SSH

Recently, while working on a project I have found out that the Classloader of my plugin framework does not find resources contained in the plugin JAR file. I fixed this problem by overriding the getResourceAsStream method of the PluginClassLoader. Actually there are 2 or 3 related methods which also should be overriden by the PluginClassLoader, but currently I don’t need to use that other methods and it would cost me valuable time, so I decided to just fix the getResourceAsStream method and fix the related methods later when I maybe need them.

In the last days I have rebuild my Gitlab and Gitlab Runner servers from scratch, because I was annoyed with them being really slow and while this opportunity I did upgrade both of my servers to Debian 11 („Bullseye“). I have set up the Gitlab on the new server without the Docker Layer and although it took me quite a few hours, now my new Gitlab runs smooth and the pipeline executes fast.