I have been using Hermes agent now for two months. The experience has been .. mixed. I am not sure if the results would have been substantially better if I had made some other choices than I did, but this post is not about that.

To recap, the earlier episodes were:

Me and my agent was the start of the journey. Me and my agent - part 2 (of N) was bit over two weeks in, about infra around it and setup of the agent itself. Me and my agent - part 3 (of N) was about LLMs I use with it.

So far the main use for the agent has been simply internet searching and using web browser for me. To do that, I created two new (vibe coded) hobby projects:

Project 1: remote-chromium

https://github.com/fingon/remote-chromium is the first one. While I looked at existing alternatives, I could not find an existing container which did exactly what I wanted it to:

  • Decent base distribution with, with chromium updates based on its updates (Debian stable in this case)
  • Chrome DevTools Protocol (CDP) access for the agent
  • VNC access for me
  • HTTP server to VNC for clients without VNC (again, for me)
  • ( later on, tab auto-close N minutes after CDP sessions become idle )

Anyway, if interested, its auto-updating container (latest only, as security concerns make stale images worthless) is here: https://github.com/fingon/remote-chromium/pkgs/container/remote-chromium

While Hermes by default has built-in browser, in my case, its problem is that it lives in a container, and therefore my ability to give it access to things is limited. remote-chromium addresses this though, as I can log in to it to see what is going, and give it persistent logins to places I need for the agent to use it.

Project 2: firecrawlish-extractor

https://github.com/fingon/firecrawlish-extractor is my approach at solving web search for Hermes. Web search for an agent consists of two distinct problems:

  • how to get search results
  • how to get machine readable versions of the search results’ pages

FireCrawl seems like one reasonable solution to both of them - there is number of other solutions as well. It can be even self-hosted, but its infrastructure is bit on the heavy side. Hermes supports it natively, which is nice.

So I set out to implement its API subset, which

  • uses SearXNG for actually searching (which uses variety of search engines without actually logging to them or leaving client traces to speak of)
  • uses Trafilatura Python HTML extractor library to produce markdown of the search results.

Just like project 1, this one is also latest-only container which gets updated as the deps change (but as it does not contain anything critical, I’m the meat proxy in the middle pressing dependabot buttons for the content in Python lock file).

What does it look like in terms of architecture?

One word: Messy.

  graph TD

mss[Screen Sharing] -- vnc -----> rc
mc[Matrix clients] ---> Hermes


Safari --> Hermes
Safari --> gomodel
Safari --> rc
Safari --> OpenWebUI --> Hermes





Hermes -- openAI API --> gomodel -- OpenAI models --> goap[go-openai-proxy] --> oa((OpenAI))

gomodel -- local models --> oMLX

Hermes -- firecrawl API --> fce[firecrawlish-extractor] --> SearXNG --> ddg((DuckDuckGo etc))

Hermes -- browser_exec ---> rc[remote-chromium] --> i((Internet))

It does not even have quite everything (OpenWebUI has really gomodel in the middle on its way to Hermes too, and probably something else I forgot too)

Use cases

I have so far logged in to:

  • Facebook: Hermes composes me daily digest of FB posts/notifications. I don’t want to see any ads, and these days if I choose no targeted policy, I can see one feed item before having to watch some obnoxious video ad. No thanks.
  • LinkedIn: Hermes provides a weekly summary of what is going on with my direct contacts and notifications (I am not that interested in the rest of the feed)
  • Untappd: Beer list export for free ( as opposed to pay for it if you do it by their official export feature )
  • k-ruoka.fi ( Finnish grocery chain we order our groceries from )

However, the main use case is simply unauthenticated internet use. A lot of it is search powered, but there are also some ‘regular’ sites my agent uses a lot:

  • foreca.fi ( Finnish weather forecasts )
  • hintaseuranta.fi ( Finnish price comparison website )

Some final thoughts

I will write about details of the agent use one of these days, but so far I am really ambivalent about usefulness of my current setup.

While the Hermes is ’nice’, and has replaced most of my web searching needs (I ask something, switch tab, and go back eventually to get nicely referenced summary of what I wanted) and some of the day-to-day context switching (it automates some things, monitors others), the amount of time spent fiddling with it has been nontrivial.

At the heart of the matter are two aspects I find troubling:

  • (lack of) capability - the 5.6 Luna model I am stuck with currently (even at max thinking level) is not great for many things (more about this later, perhaps)
  • privacy - as a lot of the content goes through OpenAI servers, I am not willing to put really private things within Hermes yet

I am hoping these both can be solved with local hardware and some decent model in the future, but we shall see. Or perhaps with better agent software, as I have some ideas (and I feel Hermes is not really fully aligned to what I want).